Remove webhook after retries

Maybe I'm missing something obvious here but I can't find out how to remove webhooks once they have timed out or are gone.
Tried to implement an IWebHookSender but the methods in IWebHookStore all require a user argument to remove a webhook. The user is not available in the workitem.
I can pause or remove all filters from the in-memory webhook but they will come back once the store is refreshed.
private sealed class MyWebHookSender : DataflowWebHookSender
{
    private readonly IWebHookStore _webHookStore;

    public MyWebHookSender(ILogger logger, IWebHookStore webHookStore, IEnumerable<TimeSpan> retryDelays, ExecutionDataflowBlockOptions options)
        : base(logger, retryDelays, options) {
        _webHookStore = webHookStore;
    }

    protected override Task OnWebHookGone(WebHookWorkItem workItem) {
        await _webHookStore.DeleteWebHookAsync(** how to find user **, workItem.WebHook);
    }
}

It's as little tricky - I agree. You can find the offending WebHook using the IWebHookStore.QueryWebHooksAcrossAllUsersAsync where you pass "*" as the action (or you get the specific action from the first NotificationDictionary from the WebHookWorkItem) and then the check the WebHook ID in the predicate. Once you have that, you can delete it.

Thank you for your reply.
Unfortunatly I don't understand what you mean.
There are only two delete operations in IWebHookStore: DeleteWebHookAsync and DeleteAllWebHooksAsync. Both requires a string user.
I already got the WebHook and Id from the WebHookWorkItem but how do I get the user?

Thanks, I missed that the predicate included the user.
For anyone else needing it, here is my code
internal sealed class WebHookSender : DataflowWebHookSender
{
    private readonly IWebHookStore _store;

    public WebHookSender(ILogger logger, IWebHookStore store)
        : base(logger) {
        _store = store;
    }

    public WebHookSender(ILogger logger, IWebHookStore store, IEnumerable<TimeSpan> retryDelays, ExecutionDataflowBlockOptions options)
        : base(logger, retryDelays, options) {
        _store = store;
    }

    protected override Task OnWebHookFailure(WebHookWorkItem workItem) {
        return DeleteWebHook(workItem.WebHook.Id);
    }

    protected override Task OnWebHookGone(WebHookWorkItem workItem) {
        return DeleteWebHook(workItem.WebHook.Id);
    }

    private async Task DeleteWebHook(string webHookId) {
        string webHookUser = null;

        await _store.QueryWebHooksAcrossAllUsersAsync(new[] { "*" },
                                                      (webHook, user) => {
                                                           if (webHook.Id == webHookId) {
                                                               webHookUser = user;
                                                           }

                                                           return false;
                                                       });
        // Multiple workitems can run in parallel so not finding it in the store is not an error
        if (webHookUser != null) {
            await _store.DeleteWebHookAsync(webHookUser, webHookId);
        }
    }
}


systemd-timesyncd fails at step STATE_DIRECTORY on semi-clean install

Issue description
On a semi-clean install (where no stateVersion is set since upgrade is immediate and no state is expected/important), systemd-timesyncd fails due to https://github.com/NixOS/nixpkgs/pull/61321
Other reports: https://github.com/NixOS/nixpkgs/issues/31540#issuecomment-506936820, https://github.com/NixOS/nixpkgs/issues/63688
Steps to reproduce

install nixos 19.03
immediately nixos-rebuild switch to nixos-unstable
reboot

Discussion
https://github.com/NixOS/nixpkgs/pull/61321 includes a fix that is only run on stateVersion < 19.09, but most installs with no user-generated state do not have stateVersion set, so another solution might be needed.
cc:@andir

Any reason to NOT set the state version? If the Filesystem would be really stateless the problem should not be there. I initially added that check since I did not want to keep those lines in all activation scripts until the sun erupts. Seeing that a few people are running without a stateVersion set is surprising.

The state version is not set by default, it's not obvious that it should be set when instantly upgrading an image from nixos-19.03 to nixos-unstable.

The standard installer sets it to the current stable release. Which kind of
images are you referring to?
On Tue, 16 Jul 2019, 19:35 Yorick<EMAIL_ADDRESS>wrote:

The state version is not set by default, it's not obvious that it should
be set when instantly upgrading an image from nixos-19.03 to nixos-unstable.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/NixOS/nixpkgs/issues/64922?email_source=notifications&email_token=AAE365HNB45YOF4MCNCFE2LP7YBGBA5CNFSM4IEC2A5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2BS4XI#issuecomment-511913565,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAE365CSBPHS7PQ2ETTC2XLP7YBGBANCNFSM4IEC2A5A
.

Well, there are 9 reports so far. It seems some people are running the .iso. I'm referring to the packet.com nixos-19.03, which I think are installed using nixos-install, and do not include a state version.

I remembered Packet.com's installers configuring the stateVersion, so I provisioned an instance to check. Indeed, it does:
[grahamc@test:~]$ cat /etc/nixos/configuration.nix
# Edit this configuration file to define what should be installed on
# your system.  Help is available in the configuration.nix(5) man page
# and in the NixOS manual (accessible by running ‘nixos-help’).

{ config, pkgs, ... }:

{
  imports =
    [ # Include the results of the hardware scan.
      ./hardware-configuration.nix
      ./packet.nix
    ];

  # Use the GRUB 2 boot loader.
  boot.loader.grub.enable = true;
  boot.loader.grub.version = 2;
  # boot.loader.grub.efiSupport = true;
  # boot.loader.grub.efiInstallAsRemovable = true;
  # boot.loader.efi.efiSysMountPoint = "/boot/efi";
  # Define on which hard drive you want to install Grub.
  # boot.loader.grub.device = "/dev/sda"; # or "nodev" for efi only

[ snip a bunch of commented stuff ]

  # This value determines the NixOS release with which your system is to be
  # compatible, in order to avoid breaking some software such as database
  # servers. You should change this only after NixOS release notes say you
  # should.
  system.stateVersion = "19.03"; # Did you read the comment?

}

Well, there are 9 reports so far. It seems some people are running the .iso.

I am not questioning the relevance of this bug. I am just trying to figure why nobody I talked to so far has ever done that. With the .iso you are refering to the ISOs from the website? nixos-generate-config definitely adds a stateVersion. My main concern right now is that I do not see how people end up in that situation when following the standard installation procedures so we can properly test it now and in the future.

Maybe state version should be mandatory. I might still have set it to 19.09, since I never considered some timesyncd symlink "state" (and generally expect state version issues to result in missing data, not failing units).

My config is missing stateVersion, and I believe that's because it was generated way too long ago (before installer was generating it)... though I don't think that would be the most common reason nowadays.

@andir We're discussing making stateVersion mandatory in above PR, but it would be great if this wouldn't be necessary. In particular the changes you made in https://github.com/NixOS/nixpkgs/pull/61321/commits/024a383d64036dab02157927369ca680427aa61d don't seem like they need stateVersion as mentioned by @oxij in https://github.com/NixOS/nixpkgs/pull/65314#issuecomment-520118767. stateVersion should only really be used for things that have to be migrated manually. In this case I'd think this can be done automatically, which would fix this problem for a lot of people that for some reason have a messed up stateVersion (which is not optimal, but that's the current state). Another report of this problem just rolled in on IRC.

How does the automatic migration fit in with the rollback story of NixOS?

Good point @grahamc, maybe automatic migration shouldn't be done after all because of this, meaning stateVersion might be needed after all for this..

in the absence of stateVersion, setting it to 19.03 should be enough to deal with this, however, an in-place NixOS install from another distribution will cause the generated configuration file to use whatever stateVersion currently set on nixpkgs, causing this error until it is adjusted to use the latest release stateVersion.

@grahamc systemd will automatically migrate the other way. They recently added code to migrate the other way around. https://github.com/systemd/systemd/issues/12131. Maybe we should backport those patches and scrap the stateVersion migration thing.

I am happy that there is now a patch and backporting that seems like the best option we have right now. The patch is also not that complicated but I didn't check if any previous patches are a prerequisite for it.

Does this migration mean a rollback won't be smooth anymore back to generations without it? Or can systemd handle it? Because if not this will cause some trouble for almost every NixOS user wanting to do such a rollback

systemd can roll back this migration, according to their issue about this

Could we automatically perform the migration if stateVersion is unset? As long as it doesn't error if you are already migrated, I think this should be safe.

Nevermind, I understand why this is necessary. stateVersion has a default to the current NixOS version when unset, so we have no way to distinguish between "stateVersion = "19.09";" and "unset stateVersion". Letting systemd handle this is much better.

The patch looks like just https://github.com/systemd/systemd/commit/5c6d40d13238e56699074fc1b01f4ac929ba62b8 ?

... we have no way to distinguish between "stateVersion = "19.09";" and "unset stateVersion".

There, kinda, is. See highestPrio use in 1f0b6922d3c2de3da235a7075d0d3cb9255b7cd7.

The upstream patch is in systemd version 243. The PR for that just landed in staging #68098.
Would be nice to know if it works for your scenarios (that do not set a stateVersion).

@yorickvP is this still an issue now that we have the migration patch on our systemd branch?

I would assume not

Apparently people are still reporting this issue: https://github.com/NixOS/nixpkgs/issues/69258#issuecomment-534030279

The fix should be in our systemd, anyone have any idea why it's not running?

@yorickvP if you need any particular information on this, I can reproduce it on every nixos-build switch I do on my system currently.  I haven't dug into the cause due to lack of time though.

@alunduil  The easiest method would probably stepping through systemd in gdb to see why it's not triggering the migration code, but it would take time to do.

@yorickvP that doesn't sound undoable, but I would need some instructions on how to do that (it's been almost a decade since I even looked at gdb).  Is there an easy way to tell if I have debugging symbols so I can find the code in consideration quickly?

Another thing you could do is set the SYSTEMD_LOG_LEVEL=debug environment variable on the timesyncd setvice. (I might have made a typo of the env var. Im on mobile).  It will then log more and journald will have code line information for each log entry. This might help seeing what's going wrong if systemd logs sufficiently

@arianvp adding that logging didn't seem to get anything extra in the journal but I do have a paste with my current output (all of it so it's a bit long): https://gist.github.com/alunduil/8c3a8ad7ccc679fd12c20171377ddf54
If I can figure out the gdb debugging process reasonably well, I'll try that next but I will be pretty busy until next week.

Removing from 19.09 milestone since it's not a blocker for the release anymore.

For anyone else who hits this, the steps to recover are:

set system.stateVersion = 19.03
sudo nixos-rebuild switch
remove system.stateVersion = 19.03
sudo nixos-rebuild switch

You should see no more units listed in systemctl --failed.
With these instructions available, I'm comfortable marking this issue resolved, but I'll leave it up to others that are watching this issue.

Do leave stateVersion set however! Not setting it is what caused the original issue, and might very well cause future ones too.

I removed mine and subsequent updates worked as expected.  If I need to permanently set the stateVersion then I'm worried about moving forward with the distribution.  Is it assumed that when a new version is released I have to completely reinstall?

No need to fear getting stuck, as almost nothing changes based on it.  Feel free to git grep stateVersion yourself.

Is there a set of best practices for stateVersion?  Specifically, it'd be nice to know if it should always be set and should be part of the release channel update process.  I haven't seen this recommendation in the past, but I'll look at the current upgrade documentation.  It sounds like clarifying the use of this variable is the missing piece of this and related issues.

It's generated into your (first) configuration according to the current release.
{
  # This value determines the NixOS release from which the default
  # settings for stateful data, like file locations and database versions
  # on your system were taken. It‘s perfectly fine and recommended to leave
  # this value at the release version of the first install of this system.
  # Before changing this value read the documentation for this option
  # (e.g. man configuration.nix or on https://nixos.org/nixos/options.html).
  system.stateVersion = "20.03"; # Did you read the comment?
}

My general installation practice writes over /etc/nixos/configuration.nix with a git checkout so, yes, I did miss this.  Could we add this to the description on NixOS options?  In that description it says this doesn't need to stick around and can be deleted (at least that's how I read it).  This is also getting off-topic so let me know if another issue or other media would be appropriate.

I run in the issue at the end of nixos-rebuild switch. Putting system.stateVersion = "20.03"; into /etc/nixos/configuration.nix doesn't help.
nixos-version
20.03.2648.69af91469be (Markhor)

@p-alik if you originally installed 19.09 or earlier, use "19.09" for stateVersion.

@lheckemann, my system was on 20.03 since April, but I run in the issue for the first time yesterday.

As suggested in https://github.com/NixOS/nixpkgs/issues/31540#issuecomment-598838035 I've solved the issue by rm -f /var/lib/systemd/timesync

I'm closing this as the issues we've seen have workarounds or resolutions.


Added size information for webviews

It can be useful to know the content size of a webview after it has finished loading.  With this commit I have added the ability to get to this information via the onNavigationStateChange function.  I also added more documentation for that function to explain what properties are available.

Can you explain what is your use case?

In our app we use a web view to display a banner at the top of the screen.  We need to know the size of the banner in order to define the correct size for the native web view within the style.  The only way to know the height of the banner is to check the clientHeight after the web page has finished loading.
Right now there is no way to know what the size of the content of the web view is, so we have to resort to hard coding a value for the size of the web view.  This looks odd since with that approach the web view is bigger / smaller than the banner.

I think that instead of hardcoding this specific situation, we should provide a way for the page inside of the webview to communicate with React Native.
Maybe something like
// React Native
<WebView onMessage={function(message) { ... }} />

// Inside of the webview
top.postMessage(document.body.clientWidth, document.body.clientHeight);

The reason why I'm reluctant to get this change in is that reading from document.body.clientWidth forces a reflow and may hurt the webview performance. And, the way it is setup right now, it is going to do it every time, regardless if you actually use those dimensions or not

I didn't know about the reflow issue, learn something new everyday.  You're right, this wouldn't be a good solution due to that issue.
What you propose sounds like a solution similar to message handlers which WebKit supports and this library for UIWebViews.  I agree, a more general way to message between the web view and react would be helpful.  That goes beyond the scope / idea of what I had in mind with this PR, perhaps a new issue should be opened?

Something along those lines would be great indeed. If you don't mind I'll close this pull request. would be nice if you could open an issue to create that API :) Thanks!

cc @ide

@vjeux Why could not you change the android API.
In the file [ReactWebViewManager.java], Each event don't carry the following properties :

contentHeight
contentWidth


Run microservices

deploy locally with Visual studio - cannot start the microservices as well - any leads on this ?

environment variables set correctly -

env file:
SETX PCS_KEYVAULT_NAME "keyvault-prdya"
SETX PCS_AAD_APPID "322c4311-5435-42cb-b7ed-6e8ebaf23249"
SETX PCS_AAD_APPSECRET "72e85120-8108-11e9-a7bd-93bd1241aff0"
any more changes here ? please advise, thanks

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 19039245-3c01-72b1-b90a-de21be4d688e
Version Independent ID: c0af7f85-3a2a-ae20-93b3-cbf2b2851230
Content: Deploy the remote monitoring solution locally (via Visual Studio IDE) - Azure
Content Source: articles/iot-accelerators/iot-accelerators-remote-monitoring-deploy-local.md
Service: iot-accelerators
GitHub Login: @avneet723
Microsoft Alias: avneets

@ac2707  Thanks for the feedback! We are currently investigating and will update you shortly.

@ac2707 Please ensure that your env file should be in the location of .pcs<solution name>.env
Also, check if you are using the correct subscription where your resources are created.

@ac2707 I am able to complete the deployment by following the documentation steps without any issues. Also, try creating new resources and see if it works as expected. I am now closing this thread. If you have further questions, please tag me in the comments and I will gladly continue the conversation.

@AshokPeddakotla-MSFT  env variables are at the correct location and with the correct file name.. created new resources for this with the correct subscription as well. Microservices don;t seem to start . screenshot attached earlier - can you provide more details for the microservices? and also, the env file has 3 entries as sgown in the screenshot attached earlier - anything else that should be added manually ?

as per documentation - "Each web service opens a command prompt and web browser window. At the command prompt, you see output from the running service, and the browser window lets you monitor the status. Don't close the command prompts or web pages, this action stops the web service." But while running, these command prompts and web pages close on their own - cmd screenshot included in the first thread. Please advise

@ac2707 This would require further investigation to find out the root cause. I suggest you contact technical support for deeper analysis of the issue. Refer How to create an Azure support request. If you don’t have a support plan, please send an email to<EMAIL_ADDRESS>with the below details to investigate this further.
Thread URL:
Subscription ID:

Created a new support request Case<PHONE_NUMBER>00876
Will be really helpful if you can add detailed steps and screenshots for running the microservices.
Thanks


LanguageExtensions and Serverless-2016-10-31 conflicts - Support Intrinsic Function in DependsOn

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about the bug
AWSTemplateFormatVersion: "2010-09-09"
Transform:
  - AWS::LanguageExtensions
  - AWS::Serverless-2016-10-31
Parameters:
  paramEnvironment:
    Type: String
    Description: Which environment do you want to deploy to? (local,dev,stage, or prod)
    AllowedValues:
      - local
      - dev
      - stage
      - prod
    Default: local
Resources:
  resApiGateway:
    Type: AWS::Serverless::Api
    Properties:
      OpenApiVersion: 3.0.1
      StageName: !Ref paramEnvironment
  resUsagePlan:
    Type: AWS::ApiGateway::UsagePlan
    DependsOn: resApiGatewayStage
    Properties:
      Description: Unlimited usage plan
      ApiStages:
        - ApiId: !Ref resApiGateway
          Stage: !Ref paramEnvironment

It appears that if AWS::LanguageExtensions is removed, paramEnvironment is treated as an intrinsic function, then the AWS::ApiGateway::Stage resource gets correct generated with the name resApiGatewayStage.
However, if we add AWS::LanguageExtensions, paramEnvironment is a fixed value after AWS::LanguageExtensions transform, then the resource name for ApiGateway Stage becomes resApiGatewaylocalStage.
Code for the above logic can be found here
However, if there is another resource say resUsagePlan that depends on the generated resource AWS::ApiGateway::Stage, this would be a deadlock for the customer because they don't know the exact generated name (depending on paramEnvironment), and AWS::LanguageExtensions doesn't allow intrinsics function in DependsOn.
Additional context
I do NOT think it's a bug from either SAM Transform or AWS::LanguageExtensions transform itself. However, when they both exists in a template like described above, customers can only resolve this issue by using intrinsic function in DependsOn, which is not supported by  LanguageExtensions

This issue is also reported in SAM repo, however, there is no way we can resolve this without making a backward incompatible change. The only possible workaround is to have AWS::LanguageExtensions to support intrinsic function in DependsOn
https://github.com/aws/serverless-application-model/issues/2778

Thanks for raising this issue @GavinZZ. We'll triage it and will post updates here.

So, just want to confirm that I'm able to reproduce the problem. When LanguageExtensions is used then the template is passed to SAM with StageName: "local". When LanguageExtensions is not used then the template is passed with StageName: !Ref paramEnvironment. And as you point out, SAM treats these two cases differently.
I want to make sure I understand your suggestion regarding the intrinsic function @GavinZZ. Are you suggesting something like this?
DependsOn: !Join
  - ''
  - - 'resApiGateway'
    - !Ref paramEnvironment
    - 'Stage'

Thanks for help investigating it. I'm suggesting exactly like you mentioned.

Just to update the status here. Thanks @GavinZZ  for the suggestion. What you propose is more of a feature than a bugfix, but I also couldn't think of a simpler way to address the issue. We will discuss internally if this is a feature we want to build and if so if there is dev capacity to build it.

I am facing this same issue, in my template I am crating dynamically some Alarms that are attached to a CompositeAlarm,
I want to add a dependsOn section in my CompositeAlarm to mention the logicalIds of my Alarms but since the alarms are created dynamically I am not able to use !Sub or !Join to reference them.
For the Alarm creation I am using a Count to loop throw my ressources and create an alarm for each one:
Transform:
  - Count
...
Resources:
  Alarm:
    Count: 2
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName:
        !Join
          - ""
          - - !Select [ 0, !Ref MyList ]
            - "-my-log-alarm"
...

  CompositeAlarm:
    Count: 2
    #the Count value needs to be manually adjusted to your project's number of modules
    Type: AWS::CloudWatch::CompositeAlarm
    DependsOn:
      - !Join
          - ""
          - - !Select [ 0, !Ref MyList ]
            - "-my-log-alarm"

When I check the cloudFormation ressources, I was that the alarms are created with logical Ids : Alarm1, Alarm2 ...
So when I switch the Depends on section like bellow, it works but It's not very clean as a solution since I am giving the logical Ids list manually, for same cases I have many alarms (a) for many ressources (r) then I have to add a x r lines
    DependsOn:
      - Alarm1
      - Alarm2

is there any clean solution for this please?

@MalikAtalla-AWS Any update on this feature? I've a number of reports that customers are stuck due to this conflict.

I’ve found a workaround for cases where one resource depends on another. You can split the dependent resource into a separate stack, and then deploy the stacks sequentially to ensure the proper order
Something like this should work.
Before
#Stack 1
  MyApi:
    Type: AWS::Serverless::Api
...
  ApiUsagePlan:
    Type: AWS::ApiGateway::UsagePlan
    DependsOn: MyApiStage
    Properties:
      ApiStages:
        - ApiId: !Ref MyApi
          Stage: "PROD"

After:
#Stack 1
  MyApi:
    Type: AWS::Serverless::Api

Outputs:
  MyApiId:
    Value:
      Ref: MyApi
    Export:
      Name: MyApiId

#Stack 2
  MyUsagePlan:
    Type: AWS::ApiGateway::UsagePlan
    Properties:
      ApiStages:
        - ApiId: !ImportValue MyApiId
          Stage: "PROD"


Ionic 2 scroll freezes in iOS Safari.

Ionic version:  (check one with "x")
[ ] 1.x
[X] 2.x & the recent RC versions
I'm submitting a ...  (check one with "x")
[X] bug report
[ ] feature request
[ ] support request => Please do not submit support requests here, use one of these channels: https://forum.ionicframework.com/ or http://ionicworldwide.herokuapp.com/
Current behavior:
Scroll freezes in iOS wkWebView and Safari. See video here... https://youtu.be/OtSY6v9ftKU
Expected behavior:
Scroll shouldn't freeze
Steps to reproduce:
I can recreate this issue on any ion-content view on iOS 10 in platform browser or ios. What you can do is once you have enough content to scroll vertically, you can scroll the content to the bottom of the view, tap once on the bottom of the screen, then start scrolling like crazy. It immediately goes into an unresponsive frozen like state. This is a little hard to reproduce and its gotten better with releases. But its still there.  It also seems to happen more when going out of safari and coming back in then immediately scrolling.
Related code:
All ion-content views with scrolling
Other information:
Ionic info: (run ionic info from a terminal/cmd prompt and paste output below):
Cordova CLI: 6.5.0
Ionic Framework Version: 2.0.1
Ionic CLI Version: 2.1.18
Ionic App Lib Version: 2.1.9
Ionic App Scripts Version: 1.0.0
ios-deploy version: 1.9.0
ios-sim version: 5.0.12
OS: macOS Sierra
Node Version: v7.3.0
Xcode version: Xcode 7.3.1 Build version 7D1014

BTW, I can help track down the bug and fix it so I'm also looking for some ideas as to where to look to fix it.

Hmm, this is an interesting issue. If you use our fork of the wkwebview plugin do you run into the same issue? UiWebView has alot of weird scrolling edge cases that we are constantly battling with but wkwebview is much more stable in this area.

Hello! As it seems it has been a while since there was any activity on this issue i will be closing it for now. Feel free to comment if you are still running into this issue. Thanks for using Ionic!

@jgw96 hey man, thanks for the help.  Definitely not finished, this is an ongoing issue over here.  Is there any point where they turn off scrolling?  I read somewhere its been added for jank?  I'd like to optimize it or turn it off.  Cause I can get it into this state every time... https://www.youtube.com/watch?v=OtSY6v9ftKU

@jgw96 Hey some more color as I'm diving into the Ionic code, I have a custom app where we have a custom wkWebView wrapper.  Is it possible that I'm missing something like the stopScroll code in the wkwebview plugin?  What does that do anyway?  Is there a chat where I can talk to someone about this?

Hi @jgw96, I have a similar issue.
I have an web-app with ionic (so no native app o what so ever) and I'm doing this test on Safari (latest iOS, iPhone 6).
When I have a long page the inertial scrolling is working fine and when reaching the bottom it will bounce a bit further and then smoothly going back into place (normal and correct behaviour). Then if I'm in the bottom (or the top) of the page and I try to scroll further to the bottom it won't do anything (normal and correct behaviour), but if I quickly scroll to the top (or the bottom) after that I will have a stuck behaviour till I release the scrolling. Starting to scrolling again to the top (or the bottom) now it will work fine.
I can reproduce this every time I want, only on iPhone, not on Android.

@jgw96 I think this problem happen because the body use the style position: fixed, and I don't know why ionic choose to use fixed rather than the absolute. @teknekale @jodybrewster you can try, it becomes better after I replace it with the absolute. btw,  when reaching the bottom, you must wait till the page bounce back and then continue scroll the content.

We're having the same issue, easily reproducible in iOS safari with the conference app. Happens both at top and bottom of a scrollable list.

This only happen in Safari because the focus of the scrolling events goes to the  tag instead of ion-content.
Fortunately this does not happen when building the app with Cordova and WK!

@manucorporat so no workaround when using ionic only as a web app?

I also encountered this issue when running my ionic app in safari or chrome browser on iOs. But it seems to me the issue affects any action elements (i.e. buttons) which also sometimes stop responding :S

I also encounter similar issues on iOS (safari and chrome). I use a ion-navbar and below an iframe (width/height 100%) inside ion-content.
The page within the iframe jumps back to the top after one full "viewport-scroll"... if I scroll down again the scroll freezes..
Anybody any luck to get around these iframe issues on iOS?
Thank you,
Arno

@markholland I also encounter similar issues on iOS safari，how to resolve the issue?

I tried chrome on ios devices, it happened too. Maybe Chrome on ios is also safari browser core.
Although my solution will cause losing the bouncing effect, you can try it.
ionic Content component has a method called 'enableJsScroll', which will disable the native scroll of safari.
import {Platform,Content} from 'ionic-angluar'
export class MyPage{
    @ViewChild(Content) content: Content
    ngOnInit() {
	if (this.platform.is('mobileweb') && this.platform.is('ios')) {
		this.content.enableJsScroll()
	}
    }
    constructor(private platform:Platform){}
}

I'm having the same problem: scroll freezes on iOS with wkwebview like in @jodybrewster's video. Did you ever manage to solve this?

I am also experiencing this in iOS chrome and Safai.  My app is based off of the Conference template.  Seems to happens when I navigate back from a page off the root.


Enhancement ideas taken from the ursa module

openssl is currently broken in atom-shell, so I'm in the hunt for alternatives.  I'm a big fan of your ths module, so I thought I would take a look at node-cryptopp.
As you are probably aware the ursa module only exposes the RSA functions of openssl, so the fact that this module expands the available encryption options is good.
Looking at the RSA functions available in node-cryptopp I'm not seeing (or overlooking) some nice features found in the ursa module.

ability to read/write PEM encoded strings key storage and public key exchange.
ability to specify the encoding of ciphertext for encoding/decoding operations.  It appears that hex is the only encoding available right now.
ursa provides methods/functions on a 'key' object for encryption/decryption/signing instead of passing in the various components of the key.  I wonder if some of these convenience methods could be provided by the node wrapper.

I also appreciate that you have given consideration to protecting the key information in-memory.

Thanks for your interest :)
Answering your points one by one :

Indeed, PEM encoding/decoding is not currently available
When you are not using the KeyRing object, the only output/input encoding is indeed the hexadecimal encoding. However, if you do use the KeyRing, you can specify either hex or base64 encoding for each method call (in a very unclear way I must admit).
If you use the KeyRing, the objects wraps the keypair and only provide methods for each operation where your private key is involved. For public key operations, as of now there is no other way to use the module aside passing each key component one by one.

I know this rather (very) badly built and thought out, as this was my first nodejs module and among my first real projects in C++. node-cryptopp needs a major redesign, hopefully leading to something more consistent like node-sodium, where buffers are used a default data encoding to begin with.
Just out of curiosity, have you tried node-cryptopp inside atom-shell yet? Does it work?


Move retriever / notifier / exporter on their own packages.

Description
:warning: Breaking changes  :warning:
This version change the way to do your SDK initialization in particular when it comes to notifier, provider and exporter.
Why are we doing this breaking changes?
This change is needed because we are adding more and more retriever / exporter and notifier, and with that we are also adding more dependencies to the project.
And since all the dependencies are in the same package we were increasing the size of your build even if you were not using the new providers.
The best example is when we introduce the kubernetes provider, it has added ~20Mo in your application because the client-go from kubernetes is huge.
What did we change?
We have moved all retriever / exporter and notifier in a dedicated package for each of them.
With this new organisation when building your app we will use only the dependencies related to your configuration.
We also change the way to deal with the notifiers to align it with the way it work for other extension.
How to migrate?
Edit your init function of go-feature-flag and replace your retriever, exporter, notifier following this.
Retrievers
ffclient.FileRetriever          ->      fileretriever.Retriever
ffclient.GithubRetriever        ->      gcstorageretriever.Retriever
ffclient.GithubRetriever        ->      githubretriever.Retriever
ffclient.HTTPRetriever          ->      httpretriever.Retriever
ffclient.KubernetesRetriever    ->      k8sretriever.Retriever
fflcient.S3Retriever            ->      s3retriever.Retriever

Exporters
ffexporter.File                 ->      fileexporter.Exporter
ffexporter.GoogleCloudStorage   ->      gcstorageexporter.Exporter
ffexporter.Log                  ->      logsexporter.Exporter
ffexporter.S3                   ->      s3exporter.Exporter
ffexporter.Webhook              ->      webhookexporter.Exporter

Notifiers
Since in this PR we change the way to work with notifier, you have more impact when configuring them.
Before this PR to configure a notifier you had something like that.
_, err := ffclient.New(ffclient.Config{
    // ...
    Notifiers: []ffclient.NotifierConfig{
        &ffclient.WebhookConfig{
            // ...
        },
    },
})

With this PR we had remove the ffclient.NotifierConfig struct to use directly the notifier them self.
It means that now ffclient.Config.Notifiers has the type []notifier.Notifier.
So it will look like:
_, err := ffclient.New(ffclient.Config{
    // ...
    Notifiers: []notifier.Notifier{
        &webhooknotifier.Notifier{
            // ...
        },
    },
})

To follow the same pattern has the retrievers and exporters we also have changed the struct you were using.
notifier.LogNotifier        ->      logsnotifier.Notifier
ffclient.SlackNotifier     ->      slacknotifier.Notifier
ffclient.WebhookConfig      ->      webhooknotifier.Notifier

Changes include

[x] Breaking changes (change that is not backward-compatible and/or changes current functionality)

Closes issue(s)
Resolve #256
Checklist

[x] I have tested this code
[x] I have added unit test to cover this code
[x] I have updated the documentation (README.md and /docs)
[x] I have followed the contributing guide

Thank you, the PR description is very detailed.
I only skimmed over the changes, but LGTM. Only nit is that the placing of the testdata feels a little non-idiomatic. As the exporters/retrievers/notifiers are more isolated now I would normally expect the testdata right next to each exporter. Like exporter/file/testdata and so on.


[v1.13] health: only launch /hello after host datapath is ready

Backporting some host endpoint-related fixes to hopefully address HOST_EP_ID 65535 being rendered into bpf_lxc's ep_config.h.
 27392 32521 32439

/test-backport-1.13

/test-1.16-4.19

/test-1.16-4.19

/test-backport-1.13

/test-backport-1.13

Blocked on https://github.com/cilium/cilium/pull/33376 to fix a couple of workflows.

/test-backport-1.13

/test-1.17-4.19

/test-1.16-4.19

/test-1.20-4.19

/test-1.17-4.19

/test-1.17-4.19

/test-1.17-4.19

/test-1.17-4.19


Change "Proposed Start Date" to "Required Completion Date"

We've been advised that all CWU opportunities should have a completion date set - even if it can't be met 100% of the time. It might be better for us to bake this into the form rather than make sure issuers state a completion date in the free text describing the opportunity.
And, when I think about it, having a "we need this work completed by..." is much more meaningful for viewers of the opportunity than saying "we want you start working on this on 'x' date..." (which is kind of random...)

This has been moved to JIRA


Optimise lookup for singulo so it uses collision for dynamic + static and lookup for anything not covered.
Add EntityQuery around for SPEED.

Requires https://github.com/space-wizards/RobustToolbox/pull/5416
:cl:

fix: Fix AI eye getting deleted by singulo.

Looking at that square comment... Wonder how relevant it is to "singulo can grow in a corner and escape"? Though probably some other system interacting funky...

Looking at that square comment... Wonder how relevant it is to "singulo can grow in a corner and escape"? Though probably some other system interacting funky...

When it was written entitylookup was still only doing worldAABB checks but now uses box2d intersections so it should be unnecessary (short of the singulo moving into a field generator inside of 1 tick at a high velocity).

There seems to be something wrong with SingularityLayer. The singulo can no longer eat some entities like carpets or posters.


ValueError: invalid literal for int() with base 10

Hi there,
given links for Word2Vec are not working (get_word2vec.sh fails downloading) so I built a copy from this repo which is the only resource available at the moment (the Google repo seems to be down).
As a standalone program, word2vec appears to be working correctly, but when combined to the preprocess.py in this project, I cant manage to make it work.
It generates the .txt file, but the process crashes before outputting the hdf5 one with a:
ValueError: invalid literal for int() with base 10: '\x7fELF\x02\x01\x01'
when trying to preprocess any of the given datasets (i.e. MR).
I've tried with various word2vec build but the result is always the same.

After some tests (and more reading), I finally understood that the .bin file the readme refers to is not the word2vec executable binary file, it's a "Pre-trained word and phrase vectors" as the official word2vec project page states.
Still, I think it would be nice to modify the readme stating explicitly this common misconception.


Implemented functionality to display movie names to buttons.

Have implemented a shuffle of the movie array which will then take 4 movies at random and add them to a new array. The movies from the new array are then displayed across the 4 options buttons using a new function optionsButtons.
The application then picks 1 movie from the new array and queries the OMDB API and the Giphy API.

Have now fixed the functionality which checks if an answerOption button has the correct movie name or the incorrect one.


Trying to get rid of inline oncontextmenu="return false;"

Hi Malibu,
I'm trying to get rid of all my inline javascript from the app I'm developing and the last thing left is some of the markup from your plugin, namely:
_scrollButtons=function(){//...}
_pluginMarkup=function(){//..}
I wanted to override the functions that do this, but since they are used internally via the variables, that's no longer an option. Is there any chance this would be refactored in a future version? Or do you think of any way I can achieve this without "hacking" into the original .js file and modify it manually?
Best of wishes,
giryhepai

In version 3.1.5 I have removed oncontextmenu="return false;"


[Smalltalk] add support for Dolphin Smalltalk

Reasons for making this change:
Add support to Dolphin Smalltalk, the template only have support to Pharo and Squeak.
Links to documentation supporting these rule changes:

https://github.com/dolphinsmalltalk/Dolphin
http://object-arts.com/gettingstarted.html

Hi @shiftkey do you need more information about this changes?

@rejurime no, I'm good. Thanks for working through it with me!


Release manager for Rakudo 2019.12 and onward

To make it clear, I'm still here, I just want to focus on other raku things.
It was a wild ride, but after 2+ years I no longer have enough energy for new releases, and rakudo definitely deserves more.
To clarify, @kawaii previously agreed to take over, but lately they have been busy with other real life stuff. So it seems that we still need a new release manager.
Please leave a comment if you're interested! Note that it's a good opportunity for those who haven't heavily participated in rakudo development, it's definitely one of the ways to get more involved :)

Just wanted to thank you for your work so far.
I've already said so many times, but for me, the first part of every year has a lighter workload. That means that from February on I could take over if you'd take me, but definitely not now (and I would still have to leave in September, when new students come my way)

Great job, @AlexDaniel . You've always been someone very pleasant to work with!

@AlexDaniel - Thanks for all the hard work!

@AlexDaniel take my appreciation too! Thanks for the great job!

Thanks a lot for the insane amount of work put into this! You are awesome.
I would like to propose myself for the role of new release manager.
Pros be (the order is random):

For some time I am doing the same work with Comma (changelogs, talking with people about releases, testing, doing builds, updating metadata for the website etc), just at a smaller scale, of course.
Due to my work, I am pretty tied to Raku community, and it is unlikely for me to suddenly disappear.
I would like us to have high paced releases, which means bringing back monthly releases. This can be considered a cons, but I really think that we need some freshness and ship fixes and improvements more often, shortening the feedback cycle.

Cons be:

I am not especially familiar with rakudo internal details, so will ask about better decisions WRT to blockers and issues of this sort. Please, be gentle.
Right now, I don't have a setup to run Blinn, it is close to being impossible. @AlexDaniel++ said we'll have a nice setup for this with @rba++ help. This is more of a requirement than a cons, I think.

I need to do a PR following the standard procedure, right?

I think @Altai-man is a great candidate. From what I've seen they're very responsible and willing to do their best. Not knowing a lot about Rakudo internals is not an issue, if we had that requirement it would narrow down the search to existing rakudo developers and I'd much rather not distract them from their work. To be honest I never had that knowledge too.

I need to do a PR following the standard procedure, right?

I don't think it's needed. @jnthn, do we want to have the release manager engraved in the problem-solving repo? I fail to find a good reason to do that. It made sense for infrastructure stuff, but it's much simpler with release management.
I think a PR tweaking this document is enough: https://github.com/rakudo/rakudo/blob/master/docs/release_guide.pod#planned-future-releases (put your name inside the parens).
Also, feel free to add planned dates for upcoming months if you strongly feel about doing monthly releases.

@jnthn, do we want to have the release manager engraved in the problem-solving repo? I fail to find a good reason to do that.

No, I don't think that's required; just sign up in the mentioned document. :+1: from my side on returning to monthly releases, if time/energy allows.

:+1: for @altai-man

@Altai-man and @JJ would be great release managers. Looking forward to package their releases!

BTW, I think @nxadm have a good point here. I see no good reason of not having two release managers. If once can't do the job for any reason, the other one can pick it up the task.

I think there's enough responsibility on @JJ's shoulders. Note that anyone can help the release manager by contributing to the changelog, helping find the regressions, resolving blockers, etc. If someone does that enough then surely they can be mentioned in the release guide, but I don't think we should appoint more than one person.

My point was in having a backup variant only. I.e. @JJ wouldn't be doing a release most of the time if ever. And only if he applies voluntarily for this role, for sure!

@AlexDaniel great work. I thank you.
👍 for @Altai-man
Off-topic @Altai-man: Regarding Blin: @maettu and then Konrad, dig deep to find some machines. Looks like with the current amount of modules in ecosystems allows to have Blin runs 2x a day on the hardware we found. Will come back to you in December with this, to be able to run Blin regularly and put the results somewhere automatically.

The contributor license agreement for Rakudo is signed and confirmed by Coke.
I'll update the schedule after getting rights to commit to the repo.
In the meanwhile, I am having some steady progress with exploring the release script. My gut feeling is I'll spend some time to make it even more awesome.
The planned release date is 22th, I hope:

By the end of this week I'll sort out the releaseable usage.
Next week I'll dedicate at looking at commits.
At the beginning of the third week I am assuming that 1)Blin is working; 2)I am starting to look at possible blockers, write the changelog, bump things, heading towards the weekend.

@Altai-man I think you've got a good grip on things now. Feel free to ping me if you need any info.
I think this issue is now resolved.


[WIP] 98/build form front

[x] Submit
[x] Review

@dgrcode Everything looks great!
Let me know when the review part is ready so I can merge it.

To make things easier to read, I propose to update the following property names of the builds

address -> userAddress
branch -> branchUrl
desc -> description
image -> imageUrl

(I'd keep id, isDraft and name the same)
Issues:

[ ] When a build is rejected, a null build is created. This happens locally, haven't checked in firebase
[ ] Handle builds without images


Add copy proxies to new session

Hi, I just needed to use proxies with this library and say an OpenPR on adding this functionality.
I figured there are some reviews needed to be resolved but minimal support to coping proxies to new session would solve my problem.
I've been setting proxies in my code like:
L.context._session.proxies = dict(http='[PROXY]', https='[PROXY]')

then I saw the copy_session function used under the hood
I think this quick solution would be nice :+1:

This change, at least in itself, has no function.
Instaloader already supports proxies by setting environment variables. The sessions.proxies value is overridden by those same environment variables.
A meaningful proxy implementation would have to have added value over the existing one. It could do this, for example, by not using the proxies set by environment variables. One approach to this would be to pass the proxy parameter in the requests call instead of through the session object.


Add release notes for DK 3.12.10

Signed-off-by: Tim Smith<EMAIL_ADDRESS>

Docs are complete. Just needs a docs team review and we can merge


make dask, pyarrow deps explicit

Pretty sure these are already included in the environment as deps but make them explicit as they are core packages.

Same for Numba.

Same again for scikit-learn, etc.


Created Loading Screen for the App

Description:
Created a loading screen for the app to use when waiting for pages to load. I have also added a path to the loading screen in the routes file so the loading screen can be navigated to. This is related to issue #135 (an issue we inherited).
The previous loading screen was "Loading ..."
The new loading screen now is the following gif.

All Submissions:

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same update/change?

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[x] Breaking change (fix or feature that would cause existing functionality to change)
[x] My code follows the code style of this project
[x] My code has been commented
[ ] My change requires a change to the documentation
[ ] I have updated the documentation accordingly
[ ] I have added tests to cover my changes
[ ] All new and existing tests passed

This is connected to Pull requests #192 and #185 which were not passing due to the downstream branch having the same name as the upstream branch.

LGTM. Great work on the View controller design. The design looks really good to me :)


remove eslint-plugin-regex

and most likely we might add our regex help tool, but it doesn't need to have autofix. just check if it's got some bad regex pattern.
some eslint plugin regex repo:

https://github.com/ota-meshi/eslint-plugin-regexp
https://github.com/BrainMaestro/eslint-plugin-optimize-regex

if you think that it's better to create our own, refer to this documentation about custom rules: https://eslint.org/docs/developer-guide/working-with-rules
I'm relying on you to decide this, you can ask Farhan if you're not too confident about it. @elianiva

kelihatannya lebih promising yg https://github.com/ota-meshi/eslint-plugin-regexp
tapi keknya mending @farhan443 aja deh yg ngurusin ginian wkwkw
gw mana ngerti regex

kelihatannya lebih promising yg https://github.com/ota-meshi/eslint-plugin-regexp

Couldn't agree more.  The documentation is well-written and the rules are reasonable.

bentar bentar.. bukannya sekarang udah pake yg https://github.com/ota-meshi/eslint-plugin-regexp?

@elianiva ngga akurat, liat aja flourite jadi ancur


RFE: Subscribe in Bridge to "built in" cloudevents (not just tasks)

Describe the solution you'd like
Ability to create subscriptions in the bridge that do not have a task name in the shipyard.
For example, subscribing my custom service to any of the "built-in" cloudevents such as action.triggered.
Usecase

Ability to perform remediation with my own Keptn service (eg. Job Executor service listens for action.triggered and runs a helm container to scale a deployment)
Easier SLI retrieval: Ability to subscribe job executor service to get-sli.triggered. JES then runs a Python script and triggers a get-sli.finished event back to Keptn. SLI retrieval now happens from Python for any tool and doesn't require a heavy custom Keptn service.

@StackScribe this is the missing piece for 0.16 self-healing that doesn't need the helm-service.

Ignore this. My mistake. All tasks are listed in the shipyard anyway so this is not required.


refactor: dependent fold for Fin

Define dfoldl and dfoldr from their monadic versions to avoid code duplication. The cost is that they now require Type _ instead of Sort _. However, induction principles are already available to deal with Prop so there is no overall loss.

Sorry for the wait! Looks good to me.


Update to 2.0.0

See http://scitools.org.uk/iris/docs/v2.0/whatsnew/2.0.html for more details.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@pelson and @djkirkham I'm merging this to start using iris 2.0 :smile:
Please let me know if you have further comments/fixes and I'll do them in another PR.

Awesome! Thanks @ocefpaf! 🎉


video_player plugin: Duration sometimes reported as 0 on iOS

import 'dart:async';

import 'package:flutter/cupertino.dart';
import 'package:flutter/material.dart';
import 'package:video_player/video_player.dart';

void main() {
  runApp(
    Demo(),
  );
}

class Demo extends StatefulWidget {
  Demo();

  @override
  State<StatefulWidget> createState() {
    return _DemoState();
  }
}

class _DemoState extends State<Demo> {
  void test() async {
    VideoPlayerController controller;

    while (true) {
      controller = VideoPlayerController.network(
          'https://flutter.github.io/assets-for-api-docs/assets/videos/butterfly.mp4');
      await controller.initialize();
      print(controller.value.duration);
      if (controller.value.duration == Duration()) {
        print('!!!! DURATION WRONG !!!!');
      }
      await controller.dispose();
      await Future<void>.delayed(Duration(milliseconds: 200));
    }
  }

  @override
  void initState() {
    super.initState();
    test();
  }

  @override
  Widget build(BuildContext context) {
    return Container();
  }
}

Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel beta, v1.0.0, on Mac OS X 10.14.2 18C54, locale de-CH)
[✓] Android toolchain - develop for Android devices (Android SDK 28.0.3)
[✓] iOS toolchain - develop for iOS devices (Xcode 10.1)
[✓] Android Studio (version 3.3)
[✓] Connected device (1 available)

• No issues found!

Can't reproduce the issue with the current plugin version.

Can't reproduce the issue with the current plugin version.

Sorry I need to reopen this. While in the test code I was not being able to reproduce, in a full app I was. Need to check why.

I'm having the same issue. Duration is sometimes reported as 0 on iOS. It's very noticeable when it happens because the VideoProgressIndicator widget will throw an exception:
flutter: ══╡ EXCEPTION CAUGHT BY WIDGETS LIBRARY ╞═══════════════════════════════════════════════════════════ flutter: The following UnsupportedError was thrown building LinearProgressIndicator(100.0%, dirty, flutter: dependencies: [Directionality], state: _LinearProgressIndicatorState#12843(ticker inactive)): flutter: Unsupported operation: Infinity or NaN toInt

@mattkj while the fix I proposed in https://github.com/flutter/plugins/pull/1094 might not be ideal, it fixes the issue reliably for us.

Thanks, I'll take a look.

Fixed in https://github.com/flutter/plugins/pull/1554/files#diff-e534c460662d39cc715c4eed77a31c3fR308


[IPWare] Lodash not installed

I recently wanted to use @fullerstack/nax-ipware package, however it ended up saying that lodash is not installed.

The solution for me was to manually npm install lodash

@John4266 Thanks for your ticket, the fix should be pushed out soon.

@John4266  - Please remove lodash from your workspace package.json, then up version nax-ipware to the latest. It will install lodash deps automatically moving forward.  Thx


switch on terminal shortcut for Explorer-add on

Description of the new feature / enhancement
It would be great to expand short cut capabilities in windows explorer through the explorer - add on. In particular it would be great to have a programmable shortcut to open the windows terminal in the folder you're in at the moment.
Scenario when this would be used?
I had to script it with auto hot key as I use it everyday. I have all kind of bash/python script in every folder so what I do is enter the folder I am working on, shortcut (F1 in my case) to enter the shell, run script in command line to give arguments
Supporting information
No response

I feel this would be a feature better for Terminal vs PowerToys.  If we were to do this, it would be implemented via the macro request for keyboard manager
You can do this today also via Shift+F10 then T as it is in the right click context menu


generate: improve out-of-project usage, resolve ambiguous bundle flags

Description of the change:

internal/cmd/operator-sdk/generate: improve flag help text, enforce correct flag semantics, and use a bare CSV base if one is not present at the expected path
internal/generate/collector: add single-dir update helper
internal/registry: return typed error when annotations were not found

Motivation for the change: This PR improves handling of outside-of-project use of generate bundle and generate packagemanifests, and disambiguates --input-dir vs. --deploy-dir vs. stdin generate bundle usage modes. --input-dir can't actually be used to generate a bundle from an existing bundle because the manifest information used to generate the CSV was "lost".
Closes #4503
Closes #4442
/kind bug
/kind documentation
Signed-off-by: Eric Stroczynski<EMAIL_ADDRESS>Checklist
If the pull request includes user-facing changes, extra documentation is required:

[x] Add a new changelog fragment in changelog/fragments (see changelog/fragments/00-template.yaml)
[x] Add or update relevant sections of the docs website in website/content/en/docs

/cc @camilamacedo86

/retest

/retest

bunch of the comments in internal/cmd/operator-sdk/generate/bundle/bundle.go could be turned into tests which would be nice

seems decent, although some tests might be nice :P
other than that there's really just one part that I really don't understand the help text you've added. other than that looks good to me

@jberkhahn I cleaned up the help text, hopefully its more clear. This code will be much easier to test once the fake generator is done, but that doesn't mean tests can't be added now. I may add them in a follow-up though.

/lgtm

@camilamacedo86 this PR adds examples for non-SDK usage of operator-sdk generate bundle, which is the title and focus of #4442. If you'd like additions of non-SDK usage in the OLM integration docs, then open another issue for that specifically.


Codes ran correct on Ubuntu and didn't run on MacOS

The codes are as follows:
EOS = require("eosjs");
eosconfig = {
    chainId: "aca376f206b8fc25a6ed44dbdc66547c36c6c33e3a119ffbeaef943642f0e906",
    httpEndpoint: 'https://mainnet.genereos.io',
};
const eos = new EOS(eosconfig);
eos.getProducers(true, undefined, 200).then(result => { console.log(result) });

The version of Nodejs/npm/eosjs are the same.

Instead of new EOS use EOS()


Add CLI for extract_implications

Adds a CLI to lake exe extract_implications

[ ] depends on: #93

awaiting-review

Please merge from main and the CI checks will pass.

awaiting-review

do-not-merge

This is now subsumed by #126


⚠ GitLab Server has degraded performance

In 9d12d53, GitLab Server (https://gitlab01.its-telekom.eu) experienced degraded performance:

HTTP code: 200
Response time: 13507 ms

Resolved: GitLab Server performance has improved in 4e0ed6f.


Better syntax highlight for global imported component

Hi Chu,
Is there a way we could enhance this part to show a different color for global imported component?

Keep up with this great work!

You could use GlobalComponents interface to define it, see https://github.com/johnsoncodehk/volar#using.

You could use GlobalComponents interface to define it, see https://github.com/johnsoncodehk/volar#using.

That's cool!  It would be even better if we could skip it. 😋
Love this plugin and thanks!

hmm it's a tricky one, the library I am using which does not export the class.

It seem you already import it as Dropdown, just use typeof Dropdown.
declare module 'vue' {
  export interface GlobalComponents {
    Dropdown: typeof Dropdown
  }
}

Thanks!


Add benchmarks for ResponseHeaders sets

         Method |                 Type |          Mean |     StdDev |           RPS | Allocated |
--------------- |--------------------- |-------------- |----------- |-------------- |---------- |
 BenchmarkAsync |               Common | 1,629.2051 ns | 27.7161 ns |    613,796.25 |       0 B |
 BenchmarkAsync | ContentLengthNumeric |   112.6076 ns |  3.0445 ns |  8,880,393.09 |      32 B |
 BenchmarkAsync |  ContentLengthString |    81.6983 ns |  1.9815 ns | 12,240,157.06 |       0 B |
 BenchmarkAsync |            Plaintext |   192.4687 ns |  2.2937 ns |  5,195,649.48 |       0 B |
 BenchmarkAsync |              Primary |   328.8702 ns |  4.7975 ns |  3,040,713.69 |       0 B |
 BenchmarkAsync |              Unknown |   694.0381 ns |  9.6555 ns |  1,440,843.10 |       0 B |

Superseded by benchmarks in https://github.com/aspnet/KestrelHttpServer/pull/1308


.drift parsing pointing error in the wrong place

Consider the following .drift file:
CREATE TABLE SyncChanges (
  table           TEXT      NOT NULL,
  pk              TEXT      NOT NULL,
  data            TEXT,
  lastUpdateTime  DATETIME  NOT NULL,
  isDeleted       BOOLEAN   NOT NULL,

  PRIMARY KEY(table, pk)
) AS SyncChange;

For this table, drift error is:
[INFO] Running build...
[WARNING] drift_dev:analyzer on lib/data/sync_changes.drift:
line 2, column 3: Expected a column name
  ╷
2 │   table      TEXT      NOT NULL,
  │   ^^^^^
  ╵
[WARNING] drift_dev:analyzer on lib/data/sync_changes.drift:
line 8, column 15: Expected an expression here, but got a reserved keyword. Did you mean to use it as an identifier? Try wrapping it in double quotes.
  ╷
8 │   PRIMARY KEY(table, pk)
  │               ^^^^^
  ╵
[WARNING] drift_dev:analyzer on lib/data/sync_changes.drift:
line 9, column 1: Expected a semicolon after the statement ended
  ╷
9 │ ) AS SyncChange;

The last error isn't really an error (and, as is the last one, I took some minutes to figure out what was going on).
After changing the column name to "table" or collection, all errors are gone.
So,

The last error doesn't really exist, it would be nice if drift parser didn't show it.

Is possible for drift to automatically put all column and table names in " to avoid this error?

The first error has a confusing message, I've fixed that in a988b38ec10f9f1abafe6084afaf35fbc317ed35.
As for the second error, things are a bit complicated. sqlite3 happily accepts it, but it also mentions that reserved keywords should be escaped for them to be recognized. So we're following sqlite3 as described in its documentation, whereas the real sqlite3 uses a ton of lenient backup cases that aren't documented to interpret keywords as identifiers. I think it's better to be consistent than try to play catch with sqlite3 in determining all the exceptions in the parser.

(and, as is the last one, I took some minutes to figure out what was going on

Yeah, I can understand that. Maybe the error message should be improved to make this clearer.

Is possible for drift to automatically put all column and table names in " to avoid this error?

We do this for SQL generated with the Dart API and in generated code too if it's required. But I don't think it's safe to do it automatically for input sources, since there are cases where we really don't know whether you meant to use an identifier or a keyword.


Update workbox-webpack-plugin version

Try add in eslint webpack config {threads: true} (direct in node_modules/create-react-app/...). Build speed up.


fix: Delete the Package.swift in AWSClientRuntime/

Description of changes
Delete the Package.swift in the AWSClientRuntime/ subdirectory & move its function to the main Package.swift.
The AWSClientRuntime library is now available only as a product published by this package's main Package.swift.
This will simplify the package definitions in this project, and allow for editing of AWSClientRuntime source files from within Xcode.
New/existing dependencies impact assessment, if applicable
No new dependencies were added to this change.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

As a retest, I re-generated the releases/ folder from this branch, then integrated the SDK into a CLI application.  It built and ran as expected.

@ganeshnj Please re-review in light of my comments


Provenance of vulnerabilities

The end user needs to see an explanation for how the vuln was detected.
For each vuln, we should store:

which scans detected it
method of detection (for example, through cpe2cve, or which html element wappalyzer scanned)

We should also display this to the end user on the vulnerabilities page.
subdomain.cisa.gov/test has CVE-XXX
cisa.gov (root domain inputted by user) -> findomain found subdomain.cisa.gov -> webscraper scraped subdomain.cisa.gov/test -> wappalyzer detected Apache version 1 -> cpe2cve scan detected CVE-XXX

Draft frontend view:

Still working out bugs to get all information...

@aloftus23 as we discussed today, we should:

further refine out the end state / UI we want, taking what you have as a start
then document (maybe in a word doc?) the changes to models necessary to store the data needed. We should share this doc with the team and get feedback from everyone until we agree on the approach
then, finally, implement this.

Also consider the following things when you create the UI:

We may want to support multiple sources, for example, if a vuln is detected by both shodan and wappalyzer
"cpe2cve" is not that helpful, instead, it would be helpful to show the actual source where the CPE was found (for example: "cpe2cve -> wappalyzer")

Reopening as there's still stuff we can do:

We may want to support multiple sources, for example, if a vuln is detected by both shodan and wappalyzer
"cpe2cve" is not that helpful, instead, it would be helpful to show the actual source where the CPE was found (for example: "cpe2cve -> wappalyzer")


FEAT-#5223: Execute SQL queries on the HDK backend

Depends on #4864

[x] first commit message and PR title follow format outlined here

NOTE:  If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.

[x] passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
[x] passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
[x] signed commit with git commit -s
[x] Resolves #5223
[x] tests added and passing
[ ] module layout described at docs/development/architecture.rst is up-to-date

Since this PR depends on https://github.com/modin-project/modin/pull/4864, should we review and merge #4864 first?

Since this PR depends on #4864, should we review and merge #4864 first?

Yes, #4864 should be merged first


Can't install any plugin - SyntaxError: Unterminated string in JSON at position 49152

Does this issue occur when all extensions are disabled?: Yes

VS Code Version: any, including 1.81.1
OS Version: Linux/irrelevant

Steps to Reproduce:

Garble ~/.vscode/extensions/extensions.json
Try installing any plugin.

2023-08-19 15:08:46.940 [error] Unterminated string in JSON at position 49152: SyntaxError: Unterminated string in JSON at position 49152
    at JSON.parse (<anonymous>)
    at Object.factory (/opt/visual-studio-code/resources/app/out/vs/code/node/sharedProcess/sharedProcessMain.js:82:84553)

I was getting this error for a day and looking at sharedProcessMain.js:82:84553 wasn't very helpful since it's minified. But I figured out it's reading something so I started poking around ~/.vscode and found my extensions.json is garbled. Basically, ENOSPACE must have caused the file to be garbled.
This issue is very hard to diagnose for a regular mortal. The log should indicate what file exactly it has an issue parsing at the very minimum.

Hi. It happens to me too. I cleared the extension.json file to make it recognizable again. I hope it helps you :)

Hi. It happens to me too. I cleared the extension.json file to make it recognizable again. I hope it helps you :)

Can you please share the content of the extensions.json file ?

@sandy081 Take your own extensions.json and remove the } from the end, for example. Or remove half of the content of your extensions.json.

@sandy081 This happens when you hit ENOSPC and the unfortunate file that is affected by it is extensions.json. https://www.gnu.org/software/libc/manual/html_node/Error-Codes.html#index-ENOSPC

This will be fixed by https://github.com/microsoft/vscode/issues/180695 - using atomic write feature. It will avoid writing partial data.

@sandy081 Thank you, much appreciated. I think that's probably half of the fix. The other half would be using one of defensive programming techniques to handle unexpected exceptions with friendly messages, e.g. if JSON is expected in a given file, and JSON won't parse, the error should indicate what file the invalid content came from. In current situation, when you see SyntaxError: Unterminated string in JSON at position 49152, the follow up questions are Where did it come from?.


Escape all shell special characters

Allows only characters which don't require quoting. I realized I was thinking about only the characters used in what's conceptually a string, but there's many more. I consulted https://stackoverflow.com/questions/15783701/which-characters-need-to-be-escaped-when-using-bash/27817504#27817504, which derives its list from what needs escaping with ${var@Q}.

Turned out it didn't work as intended and took me embarrassingly long time to see what was wrong :) fixed in 482c422e57f4041898245d352604a1770cf8f81e

Oops. Sorry, I didn't test it locally since it looked right. I assumed it would be caught with CI

No worries 👍 sadly the current --run-tests-tests does not exercise the cli much. I've thought about improving it somehow, maybe something similar to jq's shtest stuff? or extend the .test-format to be able to run cli commands and assert on expected stdout/stderr output, a bit like fq's tests? 🤔


/upgrade endpoint

Summary
Describe the changes and the issue it addresses (e.g., Fixes #123).
Type
Specify if this is a bug fix, new feature, breaking change, or documentation update.
Context
Add any additional details or screenshots.

Was created for testing the /upgrade endpoint. We discussed as a team whether it is something that belongs in the CLI. The conclusion is that we don't need it.


Fixed Height for Suggestions List Within SingleChildScrollView

Hello,
I am encountering an issue where the size of the suggestions list changes as I scroll the page. I am using the list within a SingleChildScrollView, and I would like to maintain a fixed height for the suggestions list. However, I could not find any parameters related to setting a fixed height in the documentation or the widget properties.
Would love some insights or tips on keeping the suggestions list at a constant height inside a SingleChildScrollView. If anyone's cracked this or knows a workaround to stop the size from shifting when scrolling, I'd be super thankful to hear your thoughts.
Thanks a lot for helping out!

I solved it this way
suggestionsBoxDecoration: const SuggestionsBoxDecoration(
            elevation: 0.0,
          constraints: BoxConstraints(
            maxHeight: 200,
          ),
        ),


pallet-contracts: Use the transactional storage for contracts

This builds upon #3263.

Closing as part of a general PR clean up.

This PR hasn't seen activity in quite a while (>30 days), thus I am closing this for now. This is not a comment on the approach, code quality or whether we like it or not and it would get merged eventually, but it won't get merged as is and seems to be stale. You are welcome to reopen or raise another PR about this any time later.


Monument-reward cards

Heya, is there a plan to add the monument reward Ability cards in? I see they're added to Any2cards
https://github.com/any2cards/gloomhaven/blob/master/data/milestone-ability-cards.js
Images: https://github.com/any2cards/gloomhaven/tree/master/images/milestone-ability-cards/trail-of-ashes
No idea why this is under a ToA folder
Adds a typically VERY nice ability card for each character

Hey, I didnt know these, it seems pretty simple to add, I'll do it

This is awesome, but is there any chance of hiding the card behind some form of spoiler shield or menu? You're supposed to not know about them until you achieve your "monument" and open the packet.

Hi there, thanks for putting the Milestone cards in. But these should be hidden behind some kind of ‘Unlock Milestone Reward for this class’ option…they’re a class-based quest that unlocks them for the whole class (not just character) after completion for the class in question. They didn’t ship with 1st printing Crimson Scales, but with 2nd printing and the Trail of Ashes expansion.

Added #242 to address unlocking milestones


bedtools closest with -fu/-fd does not return overlapping features

Given a:
chr1    1458885 1458886 .      0.902937412261963       +

and b:
chr1    1455997 1458123 .       0       +
chr1    1458279 1458890 .       0       +
chr1    1458947 1459218 .       0       +

where there is an overlap with the middle interval in b, I would expect to get the same result whether or not I use -fu/-fd (unless -io was used). That is, the following three test cases should give me the same interval:
bedtools closest -t first -s -a <(echo -e 'chr1\t1458885\t1458886\t.\t0.9\t+') -b <(echo -e 'chr1\t1455997\t1458123\t.\t0\t+\nchr1\t1458279\t1458890\t.\t0\t+\nchr1\t1458947\t1459218\t.\t0\t+')
bedtools closest -t first -s -fu -D a -a <(echo -e 'chr1\t1458885\t1458886\t.\t0.9\t+') -b <(echo -e 'chr1\t1455997\t1458123\t.\t0\t+\nchr1\t1458279\t1458890\t.\t0\t+\nchr1\t1458947\t1459218\t.\t0\t+')
bedtools closest -t first -s -fd -D a -a <(echo -e 'chr1\t1458885\t1458886\t.\t0.9\t+') -b <(echo -e 'chr1\t1455997\t1458123\t.\t0\t+\nchr1\t1458279\t1458890\t.\t0\t+\nchr1\t1458947\t1459218\t.\t0\t+')

But they don't:
chr1    1458885 1458886 .      0.9       +       chr1    1458279 1458890 .       0       +
chr1    1458885 1458886 .      0.9       +       chr1    1455997 1458123 .       0       +     -763
chr1    1458885 1458886 .      0.9       +       chr1    1458947 1459218 .       0       +     62

Is this the expected behaviour or a bug? I could not find a way to make an upstream/downstream-specific comparison that allows for overlapping intervals.

I am on the road for the next week. There was a bug related to this in version 2.26 that has been fixed in the current version on the Github repository. Would you mind cloning that and testing ?

Unfortunately the master branch still gives me the same result..

Thanks for checking. I will hopefully get some time to dig into this over the weekend.

Thank you very much. I'm currently using a workaround where I run closest twice, once with -fu/-fd and once without and compare the distances. It seems to work ok for now.

I just looked into this and feel that the result is correct, as 1) the first interval in B is upstream of A, the second interval in B overlaps A (see IGV image of your files), and the third interval in B is downstream of A.
Thus, the default execution should return the middle interval in B because it overlaps A. When using -fu and forcing it to return the closest upstream (non-overlapping) interval in A, it should return the first interval in B. Finally, when using -fd and forcing it to return the closest downstream (non-overlapping) interval in A, it should return the last interval in B.

Without getting into semantics of what's closest, I think it would be useful to have a way to specify upsteam/downstream but also include overlaps. Then using the existing -io option you could exclude overlaps if you wanted to. Otherwise there is no way to do the former in with a single command.


get Error "RuntimeError: The weights trying to be saved contained shared tensors"

I set the following parameters and executed finetune.sh
--bf16 False \
--tf32 False \
--disable_flash_attn2 True

But I encountered the following error when saving checkpoint
  File "/work/aa1600161/Phi3-Vision-ft-0716/src/training/trainer.py", line 131, in _save
    super(Phi3VTrainer, self)._save(output_dir, state_dict)
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 3500, in _save
    self.model.save_pretrained(
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 2616, in save_pretrained
    raise RuntimeError(
RuntimeError: The weights trying to be saved contained shared tensors [{'model.vision_embed_tokens.wte.weight', 'model.embed_tokens.weight'}] that are mismatching the transformers base configuration. Try saving using `safe_serialization=False` or remove this tensor sharing.

@CynthiaChuang For temporary fix, you could change the trainer class.
https://github.com/2U1/Phi3-Vision-ft/blob/960cb9cddf089599b0443f20b44e8fda226a8a5c/src/training/trainer.py#L127-L128
comment the super(Phi3VTrainer, self)._save_checkpoint(model, trial, metrics) line and use pass.
I'll make it work asap.
Thanks for reporting.

I'll download it and give it a try, and I'll report back to you later.

Closing the issue for solved issue.
https://github.com/2U1/Phi3-Vision-Finetune/issues/15#issue-2431532353

Sorry, due to the windstorm disaster in my area, I have been unable to complete the test.
Today, after running the test, the program completed successfully without encountering the same error message. If there are any further issues, I will reach out to you. Thank you!

@CynthiaChuang Thanks for your interest! Always welcome and appreciate reporting other issues. If you need any further adjustments, feel free to ask!


[rom_ctrl,doc] Expand countermeasure list

This is going to be the starting point for both the D2S review and
also any extra testplan items that go into V2S. Try to explicitly list
all the extra tricks we're playing.

I've just force-pushed and the new version adds RTL annotations matching the scheme in PR #10080.

Hi @msfschaffner! Ping :-)


Grid protection checks in the PowerDistributor

What's needed?
The power distributor needs to make sure that any charge/discharge requests won't exceed the Grid/Fuse limit, before sending the requests to the Microgrid API.
For this to work,  the SDK needs to know the grid connection fuse limit,  which will be available from the microgrid API once this is implemented: https://github.com/frequenz-floss/frequenz-api-microgrid/issues/19
Proposed solution
No response
Use cases
No response
Alternatives and workarounds
No response
Additional context

https://github.com/frequenz-floss/frequenz-api-microgrid/issues/19
#322
#360

Closing. Supposed to be covered by an (instantaneous) peak shaving aka grid protection actor.


[ChatQnA] Deploy ChatQnA pipelines with different configurations.

Description
[ChatQnA] Deploy ChatQnA pipelines with different configurations.
Issues
N/A
Type of change
List the type of change like below. Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds new functionality)
[ ] Breaking change (fix or feature that would break existing design and interface)
[ ] Others (enhancement, documentation, validation, etc.)

Dependencies
N/A
Tests
CD

I think it'd be better to contribute to the existing chatqna helm chart

I think it'd be better to contribute to the existing chatqna helm chart

It is used for benchmarking only.
It has applied the no_wrapper feature and aligned with benchmarking manifests. Please follow this template to apply latest features.

Why the benchmarking is based on no_wrapper feature? Does this mean this will be default and we'll recommend users to deploy with no wrapper version?
What's the future of Microservice layers(such as https://github.com/opea-project/GenAIComps/blob/main/comps/llms/text-generation/tgi/Dockerfile) and the API definition(https://github.com/opea-project/docs/blob/main/developer-guides/OPEA_API.rst#opea-micro-service-api)?


AUTHENTICATION_BACKENDS question

Introduction:
I am working on my own fork of FAE, customizing it to my needs, but the majority of my experiences with Django are with this app and so it's difficult for me to evaluate the "correctness" of settings as I just assume the code here is valid...because the application works, or at least it did in December 2019, when I had a developer set it up for me on Google Kubernetes Engine.
This will be the first of at least a few questions, but I will try to limit each question to a single topic to make it more manageable and useful to others; I'll skip the introduction in future questions.
Question:
I was hoping to implement a security-related app, django-axes, it's kinda like Fail2Ban for Django. It tracks login attempts and then blocks users that have repeated, unsuccessful login attempts.
The setup seemed simple and there were no obvious conflicts.
But step 2 on the page linked above says:

Add axes.backends.AxesBackend to the top of AUTHENTICATION_BACKENDS:
AUTHENTICATION_BACKENDS = [
    # AxesBackend should be the first backend in the AUTHENTICATION_BACKENDS list.
    'axes.backends.AxesBackend',

    # Django ModelBackend is the default authentication backend.
    'django.contrib.auth.backends.ModelBackend',
]

When I looked through base.py to find the settings for AUTHENTICATION_BACKENDS that I needed to prepend the relevant code to, I couldn't find it.
The only reference to AUTHENTICATION_BACKENDS that I could see appears to be in the Shibboleth settings and because I'm not affiliated with an academic institution, I'm not--and can't imagine I ever will be--using Shibboleth.
I searched the entire project for auth and found other references to it, and I know from using it that there is authentication in use but I can't figure out what the "default" settings would be or how I would add the code needed for the django-axes app...can you help me figure out how to set this up?

@adaminfinitum
We are using django-registration 3.0 so I am thinking that app is adding the backend.  So you could look at that code to see how it is adding it and maybe create a patch to add the reference.

That's right...it hadn't even occurred to me but I am aware of that app being used...I first started working on FAE2.0 and I think you introduced django-registration in 2.1.
Thanks.


Parsing Refactor

This one's a big one but I am very pleased with how it came out. Still adding some tests but I wanted to flag this as ready for review so we can get the ball rolling.
This fixes some problems in the codebase:

The codebase had a lot of custom tsv parsing. This is a problem because that module handles edge cases nicely.
The codebase had multiple reusable custom parsers for the same file types. In the case of iterate_m8 it contained specific business logic in addition to parsing which wasn't immediately clear.
There were numerous examples of ad-hoc parsing inside particular functions, despite having parsers for them implemented. This increased the complexity of those functions dramatically and it is unclear what file format they expect.
In general, it was extremely difficult to know the format of any file in advance. There were many variants of each file and the schemas were not necessarily strictly enforced and it was unclear why. It was very challenging to determine which fields you could depend on and which you couldn't.
The old schema system relied on dictionary key order, which technically works but isn't standard
There were numerous examples of list order and length being used to hold different data. This made it difficult to track which data was in a particular data structure when it was passed across and around files.
Refactoring parsers to a different file format would have been an extreme challenge due to how heterogenous everything was. (It was a big challenge even with the same format)

There were probably other issues as well but these are the major ones. I know this was a big lift but I really felt this needed to change.
To solve this problem I created a wrapper around python's csv parsers to add support for schemas. I would like to standardize the file formats, add headers, and explicitly add null values in any missing fields but I wanted to leave the file formats themselves unchanged here. So I needed a solution for multiple slightly different schemas. To address this I created a system of different schema variants. So instead of making every value optional, as we did before in some cases, rows must now conform to one of a set of variants. This means we can guarantee the presence of certain values in certain situations, and it is easier to reason about the types of files we may encounter. You can read more about it in the parsing.py file.
This does make the code a bit slower. However, I firmly believe this is worth it. The python code is usually not the bottleneck of our pipeline runs. I think reliable maintainable code is the priority here. In the future, we want to replace this stuff anyway with formats and tools that will probably be even more performant. I think this is an important step on the path to replacement because now all of the call sites are obvious.
Testing
Testing this is a bit of a challenge. The existing logic was not tested or fully documented so the challenge is to ensure that we are producing identical results (this is arguably even more important than producing correct results). Fortunately, our current level of testing offers at least some basic assurances (as you can see by the many failures...). This ensures that everything at least works and produces roughly the same results.
I've also run the pipeline with and without this change and done a detailed comparison of outputs. The outputs actually do slightly differ. In some cases, the spacing is slightly different, and in some cases None is replaced with the empty string. I actually think this is a good thing since these are decisions made by the python csv writer and ensure our outputs conform to standards. I compared the contents of these resulting rows and they were identical and parsable.
It's possible in certain edge cases we may encounter more problems, but I still think it is worth it to merge this and monitor for them. It will ultimately make it easier to detect parsing and schema issues.

Thanks @morsecodist, this is a great effort and I'm very happy to see the direction we're going with this.
Some architectural questions for this before we proceed (would have been great for a design review, but post-hoc design review is fine too):

Why use the abc module instead of plain inheritance?
The csv module has csv.DictReader which is a schematized tabulated format reader. Why not inherit directly from that class? What do you think about having a class for each format, with the schema as a class variable, and init passing that schema to the parent constructor?
Can you add some unit tests for this? Any time we add a new interface (in this case, the new readers) we should have a unit test for the interface. I recognize the difficulties with integration testing these - but we should at least have unit tests that alert us if this code ceases to correctly handle input that we expect the pipeline to give it, since the pipeline may have a more complex behavior that might mask the issue.

Thanks @morsecodist, this is a great effort and I'm very happy to see the direction we're going with this.
Some architectural questions for this before we proceed (would have been great for a design review, but post-hoc design review is fine too):

Why use the abc module instead of plain inheritance?
The csv module has csv.DictReader which is a schematized tabulated format reader. Why not inherit directly from that class? What do you think about having a class for each format, with the schema as a class variable, and init passing that schema to the parent constructor?
Can you add some unit tests for this? Any time we add a new interface (in this case, the new readers) we should have a unit test for the interface. I recognize the difficulties with integration testing these - but we should at least have unit tests that alert us if this code ceases to correctly handle input that we expect the pipeline to give it, since the pipeline may have a more complex behavior that might mask the issue.

@kislyuk thanks for the review, good points. The form things are in now is not what I would have chosen if designing from first principles due to a few compatibility concerns, though maybe we should change more. A design review would have been good but a lot of these decisions came up organically as I was going over the code base and wouldn't have been clear beforehand.

My inheritance structure is more complicated because on short-read-mngs/idseq-dag/idseq_dag/steps/blast_contigs.py:442 we need a list of the fields from one of the schemas. Previously, I just had the schemas as a class property and the other classes just extended the base class and called it's constructor with the schemas. I intended to keep everything in the class's namespace and to do this I needed access to the schema when using static methods on the class, hence the complicated abc structure. I considered construcing an instance with an empty iterable which would work but it feels hacky. I also considered exporting the schema and using it directly which would probably be cleaner now that I think more about it. I liked the idea of the caller not needing to worry about the notion of schemas but I think I did more harm than good here. I will change it to this unless you have another suggestion.
I would like to leverage the DictReader more (if we added headers we could just use it as is) but I needed to add some custom logic to recognize which variant we were dealing with because it is not clear from the code. I think I could slot this into DictReader but with the ammount of custom stuff I felt it wasn't worth it. Now that I've done more testing I probably could figure out which is which at each call site. That would simplify things a lot. I wanted to box how much time I spent here because there is a lot more I want to change but now that I have every usage easily findable I think I can do this pretty quickly so I will set a short time box and see if it is doable.
I will write some unit tests. The main issue I had is that we have a lot of idseq-dag unit tests that are no longer usable and they don't run with CICD so I felt if I put it in there they would get lost. I tried a bit to see if I could get the unit tests into a workable state but it would be too much for this PR. I will leave a comment over them that they are post-transfer and should be preserved but we should look into this going forward.

I will make the changes I discussed and update soon.

@kislyuk thanks for the review, good points. The form things are in now is not what I would have chosen if designing from first principles due to a few compatibility concerns, though maybe we should change more. A design review would have been good but a lot of these decisions came up organically as I was going over the code base and wouldn't have been clear beforehand.

My inheritance structure is more complicated because on short-read-mngs/idseq-dag/idseq_dag/steps/blast_contigs.py:442 we need a list of the fields from one of the schemas. Previously, I just had the schemas as a class property and the other classes just extended the base class and called it's constructor with the schemas. I intended to keep everything in the class's namespace and to do this I needed access to the schema when using static methods on the class, hence the complicated abc structure. I considered construcing an instance with an empty iterable which would work but it feels hacky. I also considered exporting the schema and using it directly which would probably be cleaner now that I think more about it. I liked the idea of the caller not needing to worry about the notion of schemas but I think I did more harm than good here. I will change it to this unless you have another suggestion.
I would like to leverage the DictReader more (if we added headers we could just use it as is) but I needed to add some custom logic to recognize which variant we were dealing with because it is not clear from the code. I think I could slot this into DictReader but with the ammount of custom stuff I felt it wasn't worth it. Now that I've done more testing I probably could figure out which is which at each call site. That would simplify things a lot. I wanted to box how much time I spent here because there is a lot more I want to change but now that I have every usage easily findable I think I can do this pretty quickly so I will set a short time box and see if it is doable.
I will write some unit tests. The main issue I had is that we have a lot of idseq-dag unit tests that are no longer usable and they don't run with CICD so I felt if I put it in there they would get lost. I tried a bit to see if I could get the unit tests into a workable state but it would be too much for this PR. I will leave a comment over them that they are post-transfer and should be preserved but we should look into this going forward.

I will make the changes I discussed and update soon.

@kislyuk tests and a few rough edges coming soon but how do you feel about this for a simpler approach? I like it much better than before.

@kislyuk tests and a few rough edges coming soon but how do you feel about this for a simpler approach? I like it much better than before.

I can't say that I've checked all the logic updates - there's a lot - but I trust you've kept good track of everything and it's a definite improvement.
I would prefer to see unit test coverage with a good and bad row for each reader class, but I won't insist on adding it all now.

I can't say that I've checked all the logic updates - there's a lot - but I trust you've kept good track of everything and it's a definite improvement.
I would prefer to see unit test coverage with a good and bad row for each reader class, but I won't insist on adding it all now.

@kislyuk added the tests, just going to do one more branch test for sanity then merge.

@kislyuk added the tests, just going to do one more branch test for sanity then merge.


fix: make it possible to detect clear button as the source for input …

…and change events
Add property __fromClearButton = true (private API for now) to the input and change events sent by the clear button.
Fixes #368 and unblocks vaadin/vaadin-combo-box#805.

This change is

Travis build failed only because SauceLabs removed support for testing on iOS 9.3 (all other tests pass).


SingleTileLayoutRenderer get versions wrong

The tiles module was built with a false assumptions about resource version strategies.
However
The resource_ids field is not used at all in practice.
We originally came up with three strategies
a) App has fixed resources, so has an immutable version and ships all resources at once.
b) App has dynamic resources which all change on some frequency, bump the version at those points to get fresh resources.
c) App has mostly static resources, that might occasionally change.  Return a versioned resource id e.g. _<updated_at>
However c) is a false assumption given resources are only refreshed when the version changes.
https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:wear/tiles/tiles-renderer/src/main/java/androidx/wear/tiles/manager/TileUiClient.kt;l=153;bpv=1

cc @castedmo @garanj

I'm tempted to introduce a first class VersionStrategy into the horologist APIs, something like
sealed trait VersionStrategy {
  val versionId: String
}

object StaticVersion: VersionStrategy {
  val versionId: String = "0"
}
data class IncrementingVersionStrategy(val versionId: String): VersionStrategy
class CollectedVersionStrategy(): VersionStrategy {
  fun addResource(id: String)

  val versionId: String
    get() = // combine resources
}

Now exposed as
    public open fun getResourcesVersionForTileState(state: T): String = PERMANENT_RESOURCES_VERSION

Apps can decide how to handle.


Token pool page showing transfers for tokens in other pools

Tokens -> Token pools -> click into a token pool. This page shows a list of token transfers but is showing tokens that are not in the current token pool. See screenshots below where a token pool created at 4:50pm has tokens listed that were created earlier.

This page is attempting to filter transfers by "poolprotocolid":
https://github.com/hyperledger/firefly-ui/blob/8cb358c5d1a000cbb7bc3c87a7039f4234ea347b/src/modules/tokens/views/TokenPools/TokenPoolDetails.tsx#L117
This filter was replaced with a "pool" filter instead, which uses the pool ID (not the protocol ID):
https://github.com/hyperledger/firefly/commit/72549d0053e3d52b629ccac48181888a7f337afd

New UI resolves this issue