Add register_action decorator, deprecate bind_key

This is mentioned in #7056 but warrants it's own issue.
With app-model migration, NAP-7 and the over haul of the keybinding implementation we are thinking of adding a register_action decorator, and deprecate bind_key. Reasons

Registering the action with app-model allows the command to be tracked by app model, which will be useful for:

keeping track of keybindings, resolving conflicts and potentially showing them in the shortcuts tab (? not sure on this)
user can also specify keybinding enablement (i.e. when/context)
enable users to use the common API (#7190) for executing commands, which is also useful for future 'redo/undo/workflow' features

Users could add the action to a menu at the same time
Command enablement expression could also be specified

Concerns:

Not declarative

This issue is intended to document everything. We will have to try and get comments from people on zulip/at community meetings on top.
cc @DragaDoncila @psobolewskiPhD

The intention is for users to use this programmatically, and not for plugin users to use this, so you are right that this 'can't' be declarative.
@DragaDoncila mentioned this con, not sure if there was more to it?

There is indeed more to it.
The current discussions about keybindings only considers full commands and keybindings declared in the manifest. I believe that many plugin developers are using the current decorator within more complex widgets, binding keybindings to various sub-widgets and sub-actions.
Such usage of keybindings will never make it to the manifest, because you can't bind the command until the widget object is created etc. In theory, this would be the gap that a register_action decorator would fill. I think this use-case is extremely common, and probably far more common than attaching keybindings to commands/full functions.

We may implement this in this way that it will work only in scripts, in plugins it may print a warning ad do nothing.

I therefore think we do need to implement this decorator for use within widgets etc., but still in a way that allows us to capture it for the shortcuts tab.

Clarification after meeting.
I think that plugins should be allowed only for declarative shortcuts.
However declarative shortcuts are not possible in scripts or notebooks (or napari console).
So we may provide such decorator, but only working outside plugins.
For plugins, we may disable it (for example using context manager) and print a warning with instruction how to declare shortcuts in manifest

Such usage of keybindings will never make it to the manifest, because you can't bind the command until the widget object is created etc.

I don't think this is strictly necessarily true. Since we are the ones creating instances from plugins, plugins can declare keybindings that map to a class method, and that activate only when an instance has been created. We would then find that method within that specific instance and call it.

(And, implied in my comment: I think it should be a design goal that everything a plugin provides is declarative. Maybe sometimes we have to fall short, but let's work towards it.)

Notes from quick chat with @DragaDoncila about declaring keybinding to widget, sub-widget:

Enablement - we need to have a context key so the keybinding is only available when the widget is available. I think this would be when the widget is visible (note that you can 'hide' and 'close' widgets, so I think we only want this keybinding when the widget is visible)
De-register - we will need to de-register the action when the widget is closed. Potentially there could be an event relating to change/closing of widgets

bind_key is really intuitive for what it does and has been around a while. Any way we can change the backend but keep that as the API?

Enablement - we need to have a context key so the keybinding is only available when the widget is available. I think this would be when the widget is visible (note that you can 'hide' and 'close' widgets, so I think we only want this keybinding when the widget is visible)

This would be great I think, it could mitigate some of the conflict issues!
E.g. using an annotation plugin, it could "hijack" 1, 2, 3, etc. from the Layer keybindings for its own purposes when active. Then when closed, the normal keybinds would resume working.
Not sure how to do a nice job in the GUI -- VS Code has things like this, but it's basically "have fun editing some json" for a UI.

bind_key is really intuitive for what it does and has been around a while.

Thanks for the input, its good to know how often things are used 'in the wild'.
The intention with changing it to register_action is for the user to be able to add extra things to the action beyond just a keybinding:

enablement for the function (currently would just be viewer/layer context when's)
ability to add the action to a menu (do you think this would be interesting? I thought of this just because app-model easily offers this)

WDYT?

E.g. using an annotation plugin, it could "hijack" 1, 2, 3, etc. from the Layer keybindings for its own purposes when active. Then when closed, the normal keybinds would resume working.

I like this, and I think it makes sense but I feel like we should see what others think. Conflict handling could get tricky though, because 2 plugins could want to use the same keybinding for their widget BUT they won't necessarily conflict as the 2 wigets aren't necessarily going to be open at the same time. I guess if they are open at the same time we could do the pop-up. But then these are not strict conflicts, users could probably just specify priority. It gets complicated...
Just a note, for me, I even think it makes sense for layer keybindings to take precedence over viewer keybindings (e.g., when an image is active keybinding 'A' does something to the image and when no image layer is active we default back to a 'viewer' keybinding) but I think we didn't like this (see: #7210) .

The intention with changing it to register_action is for the user to be able to add extra things to the action beyond just a keybinding:

Yeah, that totally makes sense, but I suspect a lot of people just want to bind a key to a function, much like they do now. So this extra features is a form of extra complexity that may not always be needed. To be clear, I'm not saying don't implement this, just thinking if we can keep bind_key using the new system for the simple case.

Just a note, for me, I even think it makes sense for layer keybindings to take precedence over viewer keybindings (e.g., when an image is active keybinding 'A' does something to the image and when no image layer is active we default back to a 'viewer' keybinding) but I think we didn't like this (see: #7210) .

There's precedent for this isn't there: we have layer keybinds that do different things on different layers -- so they are contextually aware. I'd be ok with extending that to the viewer, but there are viewer things you want to keep working with any/all layers, e.g. reset view. It would be wierd if that stopped working because you had a Labels layer open or something? I guess layer <-> viewer conflicts we could have the warning include: "It's fine, I want it this way".

I suspect a lot of people just want to bind a key to a function,

Thanks, exactly the kind of info I'm after. I'm okay with supporting both bind_key and register_action but I do think it's less ideal to have 2 ways of doing things that differ only slightly. We could also only have bind key and don't offer for users to add to menu, as you say it may not even be of interest to users.

I guess layer <-> viewer conflicts we could have the warning include: "It's fine, I want it this way".

Definitely an option. I think we can keep things as they are and if there is need from the community we can add the extra complexity.

To be clear, I'm not saying don't implement this, just thinking if we can keep bind_key using the new system for the simple case.

Just thought to clarify; we would do a deprecation cycle with bind_key, supporting both bind_key and register_action for some number of releases and giving a deprecation warning when bind_key is used, asking the user to switch to register_action
I agree that bind_key is more intuitive, in that it is pretty clear what you are doing.
With the switch to app-model though, the bind_key term does not convey well the extra notion that the action will be registered with app-model, you'll be able to execute it via app-model (https://github.com/napari/napari/issues/7190), which also allows us to track command history (useful for future redo/undo/workflow features) etc.

Coming back to this with some thoughts!

we need to have a context key so the keybinding is only available when the widget is available.

agreed, and I we need to think about what happens when a plugin declares a keybinding for a command that is associated with a widget contribution and doesn't add this context key.  Should we just always add this context key to the registered action for the developer? Should we invalidate manifests that don't declare this context key? I'm leaning towards the first option, except that maybe the second option is nice for education purposes and to make the developer explicitly consider what's going on with this keybinding? Or perhaps we add it for them and warn? IDK

we will need to de-register the action when the widget is closed. Potentially there could be an event relating to change/closing of widgets

afaik all qt widgets would emit some sort of close event that we can connect to, so it'll be up to us to make sure we handle that event and de-register the corresponding action, possibly in the dock widget's own closeEvent handler? Only thing I'm not sure about is how to get the action's ID in there, but I guess we could store it as an attribute on the dock widget we build idk.

it could mitigate some of the conflict issues! ... Not sure how to do a nice job in the GUI -- VS Code has things like this, but it's basically "have fun editing some json" for a UI

Yeah the UI is going to be a nightmare. VS Code does have a very nice preferences editor, but it's worth noting that as far as Lucy and I can tell, it doesn't really do conflict resolution. It prioritises extension keybindings at the same level as its own, allows you to set a weight for the keybinding, and then when it comes time to handle a keypress, it just lifts the first registered command that satisfies the context out of its already prioritized queue. In the preferences window, you can set conflicting keybindings to your heart's content. The only utility for managing them is you can right-click on a keybinding and then show all commands with the same keybinding.
For our purposes I think deciding on conflicts when there's no context keys and keeping the current layer and viewer conflicts is ok, but I would be very hesitant to add conflict checking for any more complex contexts than that e.g. different widgets being visible... I would be happy to warn when the user triggers such a keybinding (concrete example, two widgets are open that both declared the same keybinding) and say "hey multiple commands are active for this keybinding and we're running this one. Go to preferences to change".

To be clear, I'm not saying don't implement this, just thinking if we can keep bind_key using the new system for the simple case.

It doesn't seem like that big a deal to me to have the register_action decorator and then also keep bind_key and have it call register_action with relevant defaults under the hood. The only thing we'd have to be clear about is any use cases that are no longer supported I guess. Do we have a concrete list of what those are...?

Should we just always add this context key to the registered action for the developer? Should we invalidate manifests that don't declare this context key? I'm leaning towards the first option, except that maybe the second option is nice for education purposes and to make the developer explicitly consider what's going on with this keybinding? Or perhaps we add it for them and warn? IDK

I would vote for having it automatically added. It reduces burden when dev is writing the plugin and chance of errors when writing the expression. It does also allow us to change how we implement contexts (re #7101 :sweat_smile: ) and thus what the expression would look like. Though now that I think about it, I am assuming we would check in the dockwidgets for a widget of that name, not sure what the 'context expression' would be in this case if we just use a function?
With regard to:

to make the developer explicitly consider what's going on with this keybinding

We can ensure that we document well (add comments to cookie-cuttter, update our plugins docs). I am assuming you are just talking about the fact that the action is only active when the widget is visible?

and warn

warn the user or the plugin dev? At what point?

I'm not sure about is how to get the action's ID in there, but I guess we could store it as an attribute on the dock widget we build idk.

My understanding is that you can close the 'inner' widget and the dock widget, and in both cases you should de-register the actions right?
If you are closing the dock widget, you should just de-register all widget associated functions? Maybe we can just ensure these all have some prefix?
And if you are closing the inner widget, can you get the name of the widget?

It doesn't seem like that big a deal to me to have the register_action decorator and then also keep bind_key and have it call register_action with relevant defaults under the hood.

My concern here is that it makes things complicated. E.g., when you teach, would you  recommend the bind_key decorator but give the caveat that if they want to do more complex things, use register_action? And you would want to clear up potential confusion and make sure users know that both do the same thing, and both will register the action with app-model, register_action just has more functionality. Even with good documentation, there may still be confusion.
But yes your suggestion here is nice in that there would be limited extra maintenance burden. Again I am fine either way.

I would vote for having it automatically added. It reduces burden when dev is writing the plugin and chance of errors when writing the expression.

Yeah that's fine, I think that makes sense. We'd just have to be explicit in the docs that "keybindings added to commands that are methods of widgets will only be active when the widget is visible".

not sure what the 'context expression' would be in this case if we just use a context function

I think you're right that it would be a function, since otherwise we'd need a context key for each named widget which is never going to work.

My understanding is that you can close the 'inner' widget and the dock widget, and in both cases you should de-register the actions right?

I'm not sure what you mean by "dock widget" and "inner widget" here. All "dock widgets" have "inner widgets" consisting of the actual widget the plugin developer created, but I don't think you can close/hide just the "inner widget". Plugin developers can definitely declare tabs and panels within their "inner widget" but I'm not sure whether there's an easy way for us to check their visibility, unless we provide plugin devs with an option for declaring their own context function and I really don't want to go down that path just now.

My concern here is that it makes things complicated. E.g., when you teach, would you recommend the bind_key decorator but give the caveat that if they want to do more complex things, use register_action? And you would want to clear up potential confusion and make sure users know that both do the same thing, and both will register the action with app-model, register_action just has more functionality. Even with good documentation, there may still be confusion.

I don't feel that this is that complicated to be honest. I think if people "just want to add a keybinding" then they won't care at all that the action is registered with app-model, or what app-model is for that matter. They just want the key sequence to have an effect. If you were teaching general plugin development then you'd want to explain the difference between the two, and mention that you can always use register_action, and bind_key is there for convenience and grandfathering purposes. I also think even if we didn't reimplement bind_key, you'd still have the teaching burden of explaining to people who used to use bind_key how to do stuff with register_action and what the differences are, so anyway you're back at square one lol. My biggest concern is with documenting the differences. Is there stuff that you could use bind_key for before, that now just wouldn't work.

Ignore the inner widget thing, I got confused.
I'm fine with the bind_key, you know more. I did just notice that you pick context by using Viewer.bind_key or Points.bind_key, which will just be a bit different via register_action (unless we want to keep it the same?)

My biggest concern is with documenting the differences. Is there stuff that you could use bind_key for before, that now just wouldn't work.

My initial thought was that going forward we may be more restrictive on what is considered a valid keybinding, but I see that Kira already added: validate_key_binding. What kind of differences were you thinking of?

What kind of differences were you thinking of?

I wasn't necessarily thinking of any differences we should impose, I was just wondering whether there would be anything we'd like people not to do. But I think I confused myself between the plugin-declared keybindings (which I think we've agreed won't be using register_action OR bind_key) and the convenience decorators for usage in scripts.


Remove scheduled enqueued time when forwarding received message

Fix #40610
When message is scheduled, received, and then forwarded it retains x-opt-scheduled-enqueue-time property.
The original scheduled time on the ServiceBusReceivedMessage  has Date time resulting in illegal cast exception unless it's explicitly modified by user application.
We should not copy the original scheduled time in the ServiceBusMessage(receivedMessage) since it's already passed and message was already posted to the entity.

/azp run java - sevicebus - tests


git_add and git_commit fail due to pathspec problems and not a git repository

Question Checklist

[X] Updated fastlane to the latest version
[X] I read the Contribution Guidelines
[X] I read docs.fastlane.tools
[X] I searched for existing GitHub issues

My fastlane use case is this:
Commit a file named 'dorelease'
fastlane lane 'prep_release' is executed
Removes this file
Updates a releaseinfo file
Git commits these changes and pushes
PROBLEM:
git_add and git_commit fail when removing 'dorelease' and updating 'releaseinfo'
the errors come as either pathspec problem git doesn't know about these files or folder is not a git repository
Here is some debug output:
INFO [2019-11-04 16:12:52.47]: ------------------------------
INFO [2019-11-04 16:12:52.47]: --- Step: default_platform ---
INFO [2019-11-04 16:12:52.47]: ------------------------------
INFO [2019-11-04 16:12:52.47]: Driving the lane 'ios prep_release' 🚀
INFO [2019-11-04 16:12:52.47]: -----------------------------
INFO [2019-11-04 16:12:52.47]: --- Step: setup_circle_ci ---
INFO [2019-11-04 16:12:52.47]: -----------------------------
INFO [2019-11-04 16:12:52.47]: Not running on CI, skipping CI setup
INFO [2019-11-04 16:12:52.47]: -------------------------------------
INFO [2019-11-04 16:12:52.47]: --- Step: test -e ../../dorelease ---
INFO [2019-11-04 16:12:52.47]: -------------------------------------
INFO [2019-11-04 16:12:52.47]: $ test -e ../../dorelease
INFO [2019-11-04 16:12:52.48]: Preparing to do a release to Test Flight.
INFO [2019-11-04 16:12:52.48]: --------------------------
INFO [2019-11-04 16:12:52.48]: --- Step: last_git_tag ---
INFO [2019-11-04 16:12:52.48]: --------------------------
INFO [2019-11-04 16:12:52.48]: $ git rev-list --tags --max-count=1
INFO [2019-11-04 16:12:52.49]: ▸ e6738ef25adcae5b6a630fac1715b8a3570507f4
INFO [2019-11-04 16:12:52.51]: Last git tag: releases/1.0.0-35
INFO [2019-11-04 16:12:52.51]: -----------------------------
INFO [2019-11-04 16:12:52.51]: --- Step: last_git_commit ---
INFO [2019-11-04 16:12:52.51]: -----------------------------
INFO [2019-11-04 16:12:52.59]: Last git commit: 1d87e29
INFO [2019-11-04 16:12:52.59]: -----------------
INFO [2019-11-04 16:12:52.59]: --- Step: pwd ---
INFO [2019-11-04 16:12:52.59]: -----------------
INFO [2019-11-04 16:12:52.59]: $ pwd
INFO [2019-11-04 16:12:52.60]: ▸ /Users/johndoe/source/cloned-repository
INFO [2019-11-04 16:12:52.60]: ------------------------
INFO [2019-11-04 16:12:52.60]: --- Step: git status ---
INFO [2019-11-04 16:12:52.60]: ------------------------
INFO [2019-11-04 16:12:52.60]: $ git status
INFO [2019-11-04 16:12:52.64]: ▸ On branch master
INFO [2019-11-04 16:12:52.64]: ▸ Your branch is up to date with 'origin/master'.
INFO [2019-11-04 16:12:52.64]: ▸ Changes not staged for commit:
INFO [2019-11-04 16:12:52.64]: ▸ (use "git add ..." to update what will be committed)
INFO [2019-11-04 16:12:52.64]: ▸ (use "git checkout -- ..." to discard changes in working directory)
INFO [2019-11-04 16:12:52.64]: ▸ modified:   proejctdir/fastlane/Fastfile
INFO [2019-11-04 16:12:52.64]: ▸ modified:   projectdir/releaseinfo
INFO [2019-11-04 16:12:52.64]: ▸ Untracked files:
INFO [2019-11-04 16:12:52.64]: ▸ (use "git add ..." to include in what will be committed)
INFO [2019-11-04 16:12:52.64]: ▸ dorelease
INFO [2019-11-04 16:12:52.64]: ▸ no changes added to commit (use "git add" and/or "git commit -a")
INFO [2019-11-04 16:12:52.65]: ------------------------
INFO [2019-11-04 16:12:52.65]: --- Step: git status ---
INFO [2019-11-04 16:12:52.65]: ------------------------
INFO [2019-11-04 16:12:52.65]: $ git status
INFO [2019-11-04 16:12:52.69]: ▸ On branch master
INFO [2019-11-04 16:12:52.69]: ▸ Your branch is up to date with 'origin/master'.
INFO [2019-11-04 16:12:52.69]: ▸ Changes not staged for commit:
INFO [2019-11-04 16:12:52.69]: ▸ (use "git add ..." to update what will be committed)
INFO [2019-11-04 16:12:52.69]: ▸ (use "git checkout -- ..." to discard changes in working directory)
INFO [2019-11-04 16:12:52.69]: ▸ modified:   projectdir/fastlane/Fastfile
INFO [2019-11-04 16:12:52.69]: ▸ modified:   projectdir/releaseinfo
INFO [2019-11-04 16:12:52.69]: ▸ no changes added to commit (use "git add" and/or "git commit -a")
INFO [2019-11-04 16:12:52.69]: ---------------------
INFO [2019-11-04 16:12:52.69]: --- Step: git_add ---
INFO [2019-11-04 16:12:52.69]: ---------------------
INFO [2019-11-04 16:12:52.69]: $ git add ./projectdir/releaseinfo
INFO [2019-11-04 16:12:52.70]: ▸ fatal: not a git repository (or any of the parent directories): .git
WARN [2019-11-04 16:12:52.70]: Lane Context:
INFO [2019-11-04 16:12:52.71]: {:DEFAULT_PLATFORM=>:ios, :PLATFORM_NAME=>:ios, :LANE_NAME=>"ios prep_release"}
ERROR [2019-11-04 16:12:52.71]: Exit status of command 'git add ./projectdir/releaseinfo' was 128 instead of 0.
fatal: not a git repository (or any of the parent directories): .git
✅ fastlane environment ✅
Stack

Key
Value

OS
10.15

Ruby
2.6.1

Bundler?
true

Git
git version 2.21.0 (Apple Git-122)

Installation Source
~/.rbenv/versions/2.6.1/bin/fastlane

Host
Mac OS X 10.15 (19A602)

Ruby Lib Dir
~/.rbenv/versions/2.6.1/lib

OpenSSL Version
OpenSSL 1.0.2r  26 Feb 2019

Is contained
false

Is homebrew
false

Is installed via Fabric.app
false

Xcode Path
/Applications/Xcode.app/Contents/Developer/

Xcode Version
11.1

System Locale

Variable
Value

LANG
en_US.UTF-8
✅

LC_ALL
en_US.UTF-8
✅

LANGUAGE

fastlane files:
`./fastlane/Fastfile`
default_platform(:ios)

platform :ios do

  desc "iOS Fastlane for builds and release"

  before_all do
    setup_circle_ci
  end

  lane :adhoc do
    match(app_identifier: ["", ""], type: "adhoc", readonly: true)
    gym(
      scheme: "ui",
      export_method: "ad-hoc",
      configuration: "Adhoc"
    )
  end

  lane :prep_release do
    sh "test", "-e", "../../dorelease" do |status, result, command|
      unless status.success?
        UI.message "Command '#{command}' did not find dorelease present: (pid #{status.pid}) status returned is #{status.exitstatus}"
        UI.message "Not preparing for a release: aborting"
        next
        raise "Nothing more to do"
      end
        UI.message "Preparing to do a release to Test Flight."
        tag = last_git_tag
        UI.message "Last git tag: #{tag}"
        commit = last_git_commit
        UI.message "Last git commit: #{commit[:abbreviated_commit_hash]}"
        Dir.chdir("../..") do
          sh "pwd"
          sh "git", "status"
          `printf "#{tag},#{commit[:abbreviated_commit_hash]}" > projectdir/releaseinfo`
          `rm dorelease`
          sh "git", "status"
          UI.message git_add(path: "./projectdir/releaseinfo")
          git_commit(path: "./projectdir/releaseinfo", message: "[skip ci] VC-713 add releaseinfo for Test Flight")
          push_to_git_remote
        end
        version = get_version_number(xcodeproj:"./ui/ui.xcodeproj", target: "")
        build = get_build_number(xcodeproj:"./ui/ui.xcodeproj")
        newTag = "#{version}-#{build}"
        UI.message "Creating new tag: #{newTag}"
        add_git_tag(tag: "releases/#{newTag}")
        push_to_git_remote(tags: true)
        next
        raise "Done preparing for release to Test Flight."
    end
  end

  lane :release do
    match(app_identifier: ["", ""], type: "appstore", readonly: true)
    gym(scheme: "ui", export_method: "app-store")
    betweenValues = sh("head", "-n", "1", "../releaseinfo")
    puts betweenValues
    changelog = changelog_from_git_commits(between: "#{betweenValues}", merge_commit_filtering: "exclude_merges", date_format: "short")
    upload_to_testflight
    newBuildNumber = increment_build_number(xcodeproj: "./ui/ui.xcodeproj")
    commit_version_bump(
        xcodeproj:"./ui/ui.xcodeproj",
        message:"[skip ci] VC-713 Incrementing to next build number #{newBuildNumber}"
    )
    push_to_git_remote
  end

end

`./fastlane/Appfile`
app_identifier("") # The bundle identifier of your app
apple_id("") # Your Apple email address

itc_team_id("") # App Store Connect Team ID
team_id("") # Developer Portal Team ID

# For more information about the Appfile, see:
#     https://docs.fastlane.tools/advanced/#appfile

fastlane gems

Gem
Version
Update-Status

fastlane
2.134.0
✅ Up-To-Date

Loaded fastlane plugins:
No plugins Loaded
Loaded gems

Gem
Version

did_you_mean
1.3.0

bundler
1.17.2

etc
1.0.1

forwardable
1.2.0

CFPropertyList
3.0.1

public_suffix
2.0.5

addressable
2.7.0

atomos
0.1.3

babosa
1.0.3

claide
1.0.3

colored
1.2

colored2
3.1.2

highline
1.7.10

commander-fastlane
4.4.6

declarative
0.0.10

declarative-option
0.1.0

digest-crc
0.4.1

unf_ext
<IP_ADDRESS>

unf
0.1.4

domain_name
0.5.20190701

dotenv
2.7.5

emoji_regex
1.0.1

excon
0.67.0

multipart-post
2.0.0

faraday
0.17.0

http-cookie
1.0.3

faraday-cookie_jar
0.0.6

faraday_middleware
0.13.1

fastimage
2.1.7

gh_inspector
1.1.3

jwt
2.1.0

memoist
0.16.0

multi_json
1.14.1

os
1.0.1

signet
0.12.0

googleauth
0.6.7

httpclient
2.8.3

mime-types-data
3.2019.1009

mime-types
3.3

uber
0.1.0

representable
3.0.4

retriable
3.1.2

google-api-client
0.23.9

google-cloud-env
1.2.1

google-cloud-core
1.3.1

google-cloud-storage
1.16.0

json
2.2.0

mini_magick
4.9.5

multi_xml
0.6.0

plist
3.5.0

rubyzip
1.3.0

security
0.1.3

naturally
2.2.0

simctl
1.6.6

slack-notifier
2.3.2

terminal-notifier
2.0.0

unicode-display_width
1.6.0

terminal-table
1.8.0

tty-screen
0.7.0

tty-cursor
0.7.0

tty-spinner
0.9.1

word_wrap
1.0.0

nanaimo
0.2.6

xcodeproj
1.12.0

rouge
2.0.7

xcpretty
0.3.0

xcpretty-travis-formatter
1.0.0

generated on: 2019-11-04

Seems like for some reason running sh 'git' commands returns proper responses for 'status' command so behaves like it is in fact a git repo but git_add and git_commit behave as if they execute in a context where the project is NOT a git repo


MH-12942 Paella player config REST endpoint should be accessible by anonymous user

The paella player itself and the underlying configuration should be public accessible. Otherwise you will get an error on loading the player page. As the configuration does not include any passwords or secret data, it should be ok.

@staubesv do you have any ideas, why the checkstyle have failed?

@staubesv @wsmirnow Travis is seeing network errors connecting to the Maven cache.  I've restarted the builds, which is why it's pending :)

@gregorydlogan Thanks!


feat/s2-deregistrations

Pull Request Checklist
Before submitting this PR, please make sure:

[ ] You have run cargo clippy and addressed any warnings
[ ] You have added appropriate tests (if applicable)
[ ] You have updated the documentation (if applicable)
[ ] You have reviewed your own code
[ ] You have updated changelog (if applicable)

Description
Please provide a brief description of the changes in this PR.
Related Issues
Please link any related issues here

#205 @devwckd


Add doc for @configu-integrations/etcd

Closes #587

Hi @Ritesh-Udgata ,
Here are the details for claiming your Hacktoberfest prizes! 🎉 Check out #465  (comment) for more information or simply fill out the form here: Claim Your Swag.
Remember, you only need to fill out the form once for all your PRs. 😊


Improve readability and formatting of the release notes

Release notes are displayed during installation. They should be displayed in a nicely formatted way.

@EPTamminga take a look at https://github.com/DNNCommunity/starter-module-spa/blob/master/ReleaseNotes.txt.  You can use HTML within the TXT file.


As IDIM, I want to approve BCeID client requests

to be added once we have shared client journey experience


Revert "tests: posix: net: fix missing clock_t and clockid_t with newlib"

This reverts commit 45b3010d74924b4a4b5a72b782abb73dcefe76f3.
Issue introduced in #79884.
Signed-off-by: Anas Nashif<EMAIL_ADDRESS>

#79963

are we playing whack a mole now? a hotfix that needs a hotfix that will eventually need another hotfix?

Doesn't this now return issue #79892

Doesn't this now return issue #79892

Yes. Maybe we should revert #76352 also?

Doesn't this now return issue #79892

Yes. Maybe we should also revert #76352?

Hard no @duda-patryk

Doesn't this now return issue #79892

Yes. Maybe we should also revert #76352?

Hard no @duda-patryk

Why? This is the second time the main branch is broken due to #79892 (first time was #75205). I understand that for the first time there may have been some issues you were not aware of - accidents happen. Issue #75205 contains all information to reproduce the problem and I really don't understand why we are dealing with this again.

Issue #75205 contains all information to reproduce the problem

What problem exactly? Feel free to check out the branch for PR #80006 and be specific about what you feel is broken after that PR is merged.

I really don't understand why we are dealing with this again.

Progress. You're welcome 😃


chore: modify diff plugin options

Summary

Test Plan

Require Documentation?

[ ] No
[ ] Yes, the corresponding rspack-website PR is __

!canary

!canary

!canary

!canary


shim-notifier/listen CANCELLED: Call cancelled

[  shim] 13:27:31 INFO: Bootstrapping shim world and chat...
[  shim] 13:27:31 INFO: Shim chat loaded 0, ready to serve.
[  shim] 13:27:31 INFO: > zRPC listening on port 3102
[  shim] 13:27:31 INFO: shim now running lifecycle:true
[  shim] 13:27:31 INFO: HFC Bootstrap complete, total 0 entities.
[  shim] 13:27:31 WARNING: /shim-notifier/listen CANCELLED: Call cancelled
STILL waiting to be ready: shim (36 seconds elapsed)...
[  shim] 13:27:33 WARNING: /shim-notifier/listen CANCELLED: Call cancelled
[  shim] 13:27:35 WARNING: /shim-notifier/listen CANCELLED: Call cancelled
[  shim] 13:27:36 WARNING: /shim-notifier/listen CANCELLED: Call cancelled
[  shim] 13:27:38 WARNING: /shim-notifier/listen CANCELLED: Call cancelled
STILL waiting to be ready: shim (42 seconds elapsed)...
[  shim] 13:27:40 WARNING: /shim-notifier/listen CANCELLED: Call cancelled
[  shim] 13:27:41 WARNING: /shim-notifier/listen CANCELLED: Call cancelled
[  shim] 13:27:43 WARNING: /shim-notifier/listen CANCELLED: Call cancelled

hello @DevinLeamy
clang --version
Ubuntu clang version 14.0.0-1ubuntu1.1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
wsl ubuntu-22.04
python 3.10.6
node v20.11.0

hello @DevinLeamy   Are you here ?
could you please help me?  If my server not have 60G memory, And could I build local?


Backport Reduce code difference (#670) 2.x branch

Signed-off-by: bowenlan-amzn<EMAIL_ADDRESS>Issue #, if available:
Description of changes:
CheckList:

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Known flaky test https://github.com/opensearch-project/index-management/issues/90#issuecomment-1234916109


Update 04_probability_distributions.ipynb

There was a typo on line 751. "porbabilities" to "probabilities"

Refer to current pull request on the same issue which has more updates.


Reorder zIndexes in MiniMap, RequiredInputPrompts, EDAWorkspaceNavigation

Resolves #46
Checked these changes in local development site servers for ClinEpi and VectorBase.
Screenshot shows the MiniMap rendering beneath the header and the dropdown menu rendering above the workspace nav:

Screenshot shows the map and "required inputs" prompt rendering beneath the workspace nav:

I think there is a simpler solution. If we add z-index: 1 to https://github.com/VEuPathDB/web-monorepo/blob/d94203b46082174b182dcb5c31fb2d6bf80ea68f/packages/libs/web-common/src/styles/main.scss#L6-L7, it should address these issues.
The reason this works is that it will turn the .vpdb-Main element into a new stacking context, at the same "level" as the header element. This will prevent child elements of .vpdb-Main from stacking on top of the header.

I think there is a simpler solution. If we add z-index: 1 to
https://github.com/VEuPathDB/web-monorepo/blob/d94203b46082174b182dcb5c31fb2d6bf80ea68f/packages/libs/web-common/src/styles/main.scss#L6-L7
, it should address these issues.
The reason this works is that it will turn the .vpdb-Main element into a new stacking context, at the same "level" as the header element. This will prevent child elements of .vpdb-Main from stacking on top of the header.

Just pushed this change. Seems to work as expected. Thanks for the simple solution and the explanation!


Iss89

Closes #89

Closing this, will create a new one without the norm code.

Can you add NanoStringGeoMxSet-de.R to collate in description?

@rvitancol it it's too much of a hassle to resolve. Just copy the two files and update with my comments in a new branch off dev.

@NicoleEO this is ready now.


Model design issue

Thanks for your great works!
I have a question about the design of positional vector (x, y, z, r). In most of DETR-like detectors, the positional vector are learnable, such as DAB-DETR, Anchor-DETR. In your design, you detach it from computation graph. Dose learnable positional vector hurt the performance?

In the initial experiments on building AdaMixer, the learnable or fixed positional vector has very similar performances. We are in favor of the latter form for its simplicity.


Fix mem alignment.

Description
This is to fix the memory alignment for the memory allocator. When we allocate memory for MKLDNN, we only need to make the memory address aligned to 64. This change also seems to fix a memory error in the test of https://github.com/apache/incubator-mxnet/pull/10021
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
[x] Changes are complete (i.e. I finished coding on this PR)
[x] All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
[x] Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
[x] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Curious why it was set to 4096?

I didn't know what alignment was required by MKLDNN, so I chose an insanely large alignment. I vaguely remember that 4096 is used by mkldnn for memory allocation. Intel folks told me later that 64 should be enough.

fair enough :)

@zheng-da please run the benchmark_score.py to confirm the performance impact of mem changes.

Also need change here? https://github.com/apache/incubator-mxnet/blob/master/src/operator/nn/mkldnn/mkldnn_base-inl.h#L230

good catch! ok, so then needs to be a constant declaration somewhere — constexpr

I just measured the performance. It's more or less the same.

@cjolivier01 I have updated the code accordingly.

So CPP test should be adjusted accordingly? https://github.com/apache/incubator-mxnet/blob/master/tests/cpp/operator/mkldnn.cc#L44

@TaoLv the test doesn't matter. The test only checks the logic in the custom alignment function. Any alignment is fine.

needs JIRA

should this also be tracked by JIRA? I thought small changes are fine.


[Question] Are tags that are not part of the standard included as elements in the dataset after parseDicom is called?

Hi, I just started using dicomParser with the dicom dump example. I'm working with a derivative standard that has additional tags, and I'm wondering whether the tags not known to DICOM would still show up as elements in the dataset. I'm new to both DICOM and the derivative standard, so I don't know the additional tags yet, and the dump shows elements with tags known only to DICOM. This is also true when console logging the dataset. It's possible that the image I uploaded doesn't include any unknown tags, but I wanted to be sure whether they would show up or not if included.
Thanks.

I believe the answer is yes, but I'm not certain. Easiest thing to do is to find an image and try it in the dicom dump example!

dicomParser parses byte streams that follow the encoding as described in DICOM P5.  As long as your "derivative standard" is compliant with this, the tags will parse.


TypeScript error when using <SliceZone> with Next.js's next/dynamic

Details to be posted later
Issue reported by @kb1995

Versions

@prismicio/react:
react:
node:

Reproduction

Additional Details

Steps to reproduce
What is expected?
What is actually happening?

I was unable to reproduce on the latest version of @prismicio/react (v2.4.2) and next/dynamic.
v2.4.2 included a change to <SliceZone> that relaxed the type requirements which may have resolved the type incompatibility.
It's worth noting that Slice components provided to <SliceZone> must still be typed with SliceComponentProps as those are the only props that will be passed to the component.
See #154 for examples of using SliceComponentProps using different combinations of types.


Unable to undeploy. Update leads to error(104, 'Connection reset by peer')

$ zappa undeploy
Calling undeploy for environment dev..
Are you sure you want to undeploy? [y/n] y
Deleting API Gateway..
Oh no! An error occurred! :(
==============
Traceback (most recent call last):
File "/vol/home/ubuntu/python2713/lib/python2.7/sit
e-packages/zappa/cli.py", line 2084, in handle
sys.exit(cli.handle())
File "/vol/home/ubuntu/python2713/lib/python2.7/sit
e-packages/zappa/cli.py", line 399, in handle
self.dispatch_command(self.command, environment)
File "/vol/home/ubuntu/python2713/lib/python2.7/sit
e-packages/zappa/cli.py", line 475, in dispatch_command
remove_logs=self.vargs['remove_logs']
File "/vol/home/ubuntu/python2713/lib/python2.7/sit
e-packages/zappa/cli.py", line 843, in undeploy
domain_name=domain_name
File "/vol/home/ubuntu/python2713/lib/python2.7/sit
e-packages/zappa/zappa.py", line 1167, in undeploy_api_gateway
was_deleted = self.delete_stack(lambda_name, wait=True)
File "/vol/home/ubuntuF/python2713/lib/python2.7/sit
e-packages/zappa/zappa.py", line 1207, in delete_stack
self.cf_client.delete_stack(StackName=name)
File "/vol/home/ubuntu/python2713/lib/python2.7/sit
e-packages/botocore/client.py", line 253, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/vol/home/ubuntu/python2713/lib/python2.7/sit
e-packages/botocore/client.py", line 543, in _make_api_call
raise error_class(parsed_response, operation_name)
ClientError: An error occurred (InternalFailure) when calling the DeleteStack op
eration (reached max retries: 4): Unknown
==============
Essential changes made:

Using a new virtualenv. Had to delete the old virtualenv
Changed the AWS credentials. used aws configure to use the new AWS_ID, AWS_SECRET

This could be related to the current AWS outage.

I'm pretty much gonna avoid debugging this kinda stuff until everything is green again: https://status.aws.amazon.com/
Suggest you go and drink beers for the rest of the day.

Haha :D Thanks! Great work with zappa by the way. Loving it :)

Try now! Stuff is back!

Yup, Things are back working. Thanks again.


Mariari/artem roots fix

@agureev should reroll

re-rolled
https://github.com/anoma/anoma/pull/1424


Update sbt-scalajs-crossproject to 0.6.1

Updates org.portable-scala:sbt-scalajs-crossproject from 0.6.0 to 0.6.1.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention @scala-steward in the comments below.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [{ groupId = "org.portable-scala", artifactId = "sbt-scalajs-crossproject" }]

labels: semver-patch

Codecov Report

Merging #299 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #299      +/-   ##
==========================================
- Coverage   97.34%   97.34%   -0.01%
==========================================
  Files          24       24
  Lines        1318     1317       -1
  Branches       48       43       -5
==========================================
- Hits         1283     1282       -1
  Misses         35       35

Impacted Files
Coverage Δ

...rc/main/scala/io/iteratee/tests/ModuleSuites.scala
100% <0%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1eaa463...7381e14. Read the comment docs.


numeric values with decimal precision not syncing

We use Synapes Link for Finops.  The "Exchange rate" table does not sync all records in Synapes.  Only when we enter data with 2 decimal places does the data sync.  Any more decimals and Synapes Link does not recognise the change.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: ffe6b0c3-2f94-6afe-7fe9-19cea753d899
Version Independent ID: 51908c65-3195-85c4-28f9-6e4206446541
Content: Limitations and known issues with Azure Synapse Link for SQL - Azure Synapse Analytics
Content Source: articles/synapse-analytics/synapse-link/synapse-link-for-sql-known-issues.md
Service: synapse-analytics
Sub-service: synapse-link
GitHub Login: @jonburchel
Microsoft Alias: jburchel

@BenNichol72
Thanks for your feedback! We will investigate and update as appropriate.

@BenNichol72
I couldn't find your issue in this document could you please provide the document where exactly you found this issue.

@BenNichol72
We are going to close this thread if there are any further questions regarding the documentation, please tag me in your reply and we will be happy to continue the conversation.


Question regarding behavior of AsyncLLMEngine on single batch

We're planning to serve online traffic using AsyncLLMEngine based on example: https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/api_server.py
But our traffic limits to 1x1 traffic, i.e. each HTTP request contains only 1 prompt. Will AsyncLLMEngine dynamically form a larger batch size for incoming requests?
Thru reading the code, it seems it should have the same behavior as offline batch llm engine: on each decoding iteration, the scheduler will automatically create a large enough batch from the requests queue to max out GPU memory utilization. I want to confirm if this is true for online serving.

Yes, if multiple requests come in at the same time, they will be batched


Multiple store queue entries per thread

This could improve performance, as writes often happen back-to-back.
https://jbush001.github.io/2016/11/30/measure-twice-cut-once.html

The Ariane RISC-V processor has an interesting implementation. The store buffer has two queues, one for "speculatively issued" stores and one for committed stores. A signal from the writeback stage moves an entry to the committed queue when the instruction completes:
https://github.com/pulp-platform/ariane/blob/master/src/store_buffer.sv


Flaky test: ApplicationForm test 'acknowledgement page continue is enabled once all checkboxes have been clicked'

The test 'acknowledgement page continue is enabled once all checkboxes have been clicked' seems to fail periodically. I believe this is from how the test was written and needs to be updated.

Running into this while fixing #705


Cover cmd-exclude-prefixes-k8s by integration tests

Description
We've completed cmd-exclude-prefixes-k8s and now we can cover the application by integration tests.
Test scenarios
We need to cover the next scenarios:

Verify that cmd-exclude-prefixes-k8s can correctly read cluster prefixes from kube-system configmap.
Verify that cmd-exclude-prefixes-k8s can correctly collect prefixes if configmap from scenario1 is missed.
Verify that cmd-exclude-prefixes-k8s can correctly read user's prefixes
3.1. Verify deleting the user's prefixes.
3.2. Verify updating the user's prefixes.
3.3. Verify adding the user's prefixes.
Verify that cmd-exclude-prefixes-k8s  can correctly read prefixes from envs.
4.1. Verify that cmd-exclude-prefixes-k8s is working fine with correct prefixes from envs.
4.2. Verify that cmd-exclude-prefixes-k8s is failing with incorrect prefixes from envs.
Verify collision prefixes collision cases.

Testing notes
We have not completed tests for nsmgr + nsc yet, and for first step we can collect prefixes into alpine pod via shared volume. And check file source via kubectl cp, kubectl exec -ti "pod name" -- cat "path".

@tigrulya-exe  cc

closed via https://github.com/networkservicemesh/deployments-k8s/issues/1353


An Error occured in e4mc and floating in black void

https://paste.atlauncher.com/view/527e112b-6c84-436e-8363-6b62236bf8aa

duplicate of #19, I presume


Dockerfile Change

Already working upstream

@jnsdrtlf Are you still working on this branch (dockerfile) or can we delete it as it is unused?


feat(thumbnails, thumbnails tests)

Codecov Report
Attention: 50 lines in your changes are missing coverage. Please review.

Comparison is base (217be75) 84.44% compared to head (a519a5e) 82.79%.

Additional details and impacted files
@@               Coverage Diff               @@
##           development      #19      +/-   ##
===============================================
- Coverage        84.44%   82.79%   -1.65%
===============================================
  Files               89      102      +13
  Lines              823      965     +142
  Branches            46       53       +7
===============================================
+ Hits               695      799     +104
- Misses             112      144      +32
- Partials            16       22       +6

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Implement panel bar

This is not exactly part of the basic user interface but the side-bar navigation on the Materialize website behaves a lot like the side-navigation of the current help section in this materialize-bridge.
Also it's quite similar to the variations section of the SideNav component on Materialize website.
No usage example yet and I'm not sure if it really fits in this early milestone.

The basic user interface (which is to be implemented in Twitter Boostrap for already explained reasons - part of which is that all Aurelia Skeletons are built this way) use the accordion for the side-bar. There is no reason that should prevent you to switch to native Materialize alternative

You're right, i've kind of switched the original meaning a little to "implement for this bridge". But then this is also a good starting point for anyone doing aurelia-xxx-bridge, including me. :smile:

Removed from milestone as this is more of a "bonus" component.

reference:
https://github.com/aurelia-ui-toolkits/aurelia-materialize-bridge/issues/32


Add reset daemon for cameras with gpio buttons

Some cameras have a general purpose reset button attached to a GPIO pin. This script is called in /etc/init.d, and if the reset button GPIO is configured, checks to see if it is pressed 4 times per second.
If the button is counted to have been held for 5 seconds, the firstboot command is called, resetting the camera.

I'm not sure that the interrupt daemon written in pure C will always work and is needed by everyone. It's just that there are two ways to do it. Either way, your script is worthy of being saved and possibly implemented because it will work everywhere. The implementation as in my example at the link sometimes did not work correctly on some systems and caused the boards to freeze, all this needs to be tested.

We use the mdev daemon in the firmware to handle various events when mounting flash drives and cards.
It may be worth trying to implement support for gpio as well. Here are some links for you to think about.
http://derekmolloy.ie/kernel-gpio-programming-buttons-and-leds/
https://stackoverflow.com/questions/25507750/detecting-interrupt-on-gpio-in-kernel-module
https://habr.com/ru/articles/419273/


Same as Disconnect Button

The Multiplayer button makes the same thing of Disconnect, but i want that doesn't make me leave the Server.

Hey there, will consider making a config option later on. Been busy sorry

I have implemented this in 2.0.0 (1.20)


Client render without static contents

When using Preact with server-side rendering we're transferring a parts of the app to the client two times!
Consider following app:
const App = () =>
 alert('hi') }>Lorem ipsum

Server will render the app with the string "Lorem ipsum" in the HTML. The client bundle has the string "Lorem ipsum" in it as well.
Solution
Make a transformer(Babel or TypeScript) that removes static content from client bundle
By removing static content that's possible to recover from rendered HTML we can reduce the bundle size:
const App = () =>
 alert('hi') }>

Make client render use existing HTML to hydrate its virtual DOM.

Lorem ipsum

We might have to put markers to maintain order of children in components. For instance:
const Comp =({children, prop1}) => (

        Static content 1
        {prop1}
        Static content 2

)

Should be transformed into something like this:
const Comp =({children, prop1}) => (

        {preact.staticContent}
        {prop1}
        {preact.staticContent}

)

Recovering more than just strings from HTML.
If we can determine part of component is completely static we can remove it entirely and recover it from HTML:
const Comp =({children, prop1}) => (

Hi

        {prop1}

)

Tranfroms into:
const Comp =({children, prop1}) => (

        {preact.staticContent}
        {prop1}

)

I don't understand Preact internals. Is it even possible? This might be a very tricky problem to solve but I believe it worth it. Our own apps are exploding it bundle size and most of the bundle is just static strings.

Could maybe do this in userland by having a scanner that converts DOM to VDOM and then provides a component that can insert that VDOM in place. Hard to ditch the static via Babel though, JSX isn't very deterministic.

Determining which string can be safely removed will be extremely tricky due to the highly dynamic nature of JavaScript. Whilst this is an interesting idea it's out of scope for us to tackle. For this to work one would need to plugin into one of the existing compilers and extend them.


Viet branch angular log out

Explanation

Checklist

[ ] The PR title starts with "Fix #bugnum: ", followed by a short, clear summary of the changes.
[ ] The PR explanation includes the words "Fixes #bugnum: ...".
[ ] The linter/Karma presubmit checks have passed.

These should run automatically, but if not, you can manually trigger them locally using python scripts/pre_commit_linter.py and bash scripts/run_frontend_tests.sh.

[ ] The PR is made from a branch that's not called "develop".
[ ] The PR follows the style guide.
[ ] The PR is assigned to an appropriate reviewer.

If you're a new contributor, please ask on Gitter for someone to assign a reviewer.
If you're not sure who the appropriate reviewer is, please assign to the issue's "owner" -- see the "talk-to" label on the issue.

Codecov Report

:exclamation: No coverage uploaded for pull request base (develop@95633b0). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             develop       #4   +/-   ##
==========================================
  Coverage           ?   47.51%
==========================================
  Files              ?      478
  Lines              ?    27627
  Branches           ?     4273
==========================================
  Hits               ?    13127
  Misses             ?    14500
  Partials           ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 95633b0...9fddc6c. Read the comment docs.


[WIP] AWS S3 object store support

https://github.com/open-cluster-management/backlog/issues/11490
https://github.com/open-cluster-management/backlog/issues/11301
The fix includes:

upgrade aws sdk to the latest version github.com/aws/aws-sdk-go-v2 v1.3.2 to support both minio and AWS S3 object buckets.

AWS s3 object bucket has to specify the region to the channel reference secret. Minio object bucket doesn't need this info, apply the default minio region info.

If the object buckect fails to connect, save the failure message to the appsub status.

apply non cached local/hub client ONLY for local/hub subscription status update as the latest updated local subscription status won't be fetched rigt away by cached client
As a result, subscription status fast conitinous update could easily fail when using cached client.
Since subscription status update does not usually happen, applying non cached client to it won't impact performance too much.

code enhancments to avoid controller crash when working with AWS S3 object store.

/images

/test images

@mikeshng
It seems the image built in the PR is not pushed to Quay. After the image built task is done, I can see the 2.3 and latest tag in the quay.io got refreshed, but I tried the new image in my env. It doesn't include my new code.
Here is what I grabbed from the image built log. I can see it does download the new version of aws lib and got the binary built, So the new image got built with the new binary. Just not sure where it goes :-)
...
STEP 5: RUN make -f Makefile.prow build
...
go: downloading github.com/aws/aws-sdk-go-v2 v1.3.2

real	0m23.972s
user	3m22.514s
sys	0m15.667s

real	0m2.459s
user	0m4.876s
sys	0m1.016s

....

Successfully pushed image-registry.openshift-image-registry.svc:5000/ci-op-y982xjfm/pipeline@sha256:a4b42df20477e6d0795730e778ccfac028b22ab840908d6e8bca6098bf1e5075
Push successful

Does that mean the image built in PR stage is only used for unit test. It won't be visible in quay until we merge the PR and get the publish task complete?
Also I noticed tons of such warning in the image building.
warning: file /usr/share/zoneinfo/zone1970.tab: remove failed: No such file or directory
warning: file /usr/share/zoneinfo/zone.tab: remove failed: No such file or directory

It seems the tzdata package is not installed correctly. I accessed one subscription container and found there is no /usr/share/zoneinfo

responded to XJ's inquiry in slack.

After the PR merge, need to do some performance tests to see if the noncachedclient is called too often or not

/lgtm


[Needs Review] B07902123 蔡奇夆

A Pull Request (PR) should explain why we need this PR and how you implement it, so others could understand it quickly.
Since this is just a homework, you do not need to explain why and how.
Do NOT change the status to NeedsReview if you do not want to be graded yet.
Please fill in the following questions as feedback. It is totally optional and would not be graded.
What is the most challenging part in this homework?
Is this homework helpful for you?
Yes
Is there any suggestion?

PR readiness check list

[ v ] Did it pass the Flake8 check?
[ v ] Did you test this PR?
[ v ] Did you change the title of this PR to the correct format ([status] student_ID)?

LGTM


Pass geometry to forEachFeatureAtPixel callback

Working on #11769, I realized that when editing a geometry collection, it is useful to not only get the feature that a hit was detected on, but also the exact geometry from the collection. Since this information is available at the time a hit is detected, this change is only about passing that information all the way to the map's forEachFeatureAtPixel() method.

I updated the documentation now at least for internal use according to your suggestions. Thanks, @MoonE.

If this is used with the new layer option of the Modify interaction MutliGeometries will not be supported because there is just one geometry object. Is this expected?

If this is used with the new layer option of the Modify interaction MutliGeometries will not be supported because there is just one geometry object. Is this expected?

I know it is unfortunate, but it is expected. For geometry collections, this change was a low hanging fruit. For multi geometries, this would be much harder to achieve.


Casting non-nullable column should stay non-nullable

For example, give a column path ltree not null, when cast to text during a query, the analyzer thinks it is now becoming nullable
SELECT path::text FROM table_name

The suggested functions should be {reader}.text and {reader}.string instead of {reader}.textOrNone variants

Fixed as of v3.20 🎉


opencv:arm64-uwp build failure

Package: opencv:arm64-uwp
Vcpkg version: 2018.11.23-nohash
This package fails to build with the following error:
c:\program files (x86)\microsoft visual studio\2017\enterprise\vc\tools\msvc\14.16.27023\include\emmintrin.h(24): fatal error C1189: #error:  This header is specific to X86 and X64 targets [C:\dev\work\vcpkg\buildtrees\opencv\arm64-uwp-dbg\modules\core\opencv_core.vcxproj]
Attached build log:
install-arm64-uwp-dbg-out.log

emmintrin.h(24): fatal error C1189: #error: This header is specific to X86 and X64

Did you find any workarounds?

still doesn't work for opencv4:uwp-arm64

We are working on a fix with opencv repo. https://github.com/opencv/opencv/pull/16027


Very hastily implemented poc capability to create speaker cards on the fly from speaker profiles

Visit https://url/sharing/[speaker]

Click Generate Screenshot button:

Right click canvas image to save:

Note:
Preview does not match canvas and extra padding considerations required due to year long outstanding bug in html2canvas:
https://github.com/niklasvh/html2canvas/pull/2938

Deployed preview this is very janky so this only is working for local next dev mode for now:


Can't seem to use it in Android

In iOS all goes fine but in Android when I run the app it shows me the error screen:
TypeError: expected dynamic type 'boolean', but had type 'int64' (constructing arguments for RCTNetworking.sendRequest at argument index 5)

and in the logs:
--------- beginning of /dev/log/system
W/ReactNativeJS( 1346): 'Warning: Native component for "RCTModalHostView" does not exist'
W/ReactNativeJS( 1346): '`ProgressBarAndroid` has no propType for native prop `AndroidProgressBar.progress` of native type `number`'

The build goes
I have RN 0.15 and when I tried to upgrade to 0.16 there was too much hassle regarding dependencies so I had to rollback.

I got it working by changing the version to 0.6.3.
Isn't 0.7 compatible with RN 0.15?

Did you update your gradle files React dependency?
—
Reply to this email directly or view it on GitHub [https://github.com/corymsmith/react-native-icons/issues/152] .[https://github.com/notifications/beacon/AAAckz9Ot-5MdDRJoJArNdGcFTmiid6uks5pOo0LgaJpZM4GzbNu.gif]

I did as instructed in the "Getting started - Android".

Errors like this are usually caused by the Java code and he code getting out of
sync. What version of RN is in your gradle file and what version of RN is in
your package.json.
Thanks,
Cory Smith
On Fri, Dec 11, 2015 at 6:52 AM, corymsmith/react-native-icons
<EMAIL_ADDRESS>wrote:
I did as instructed in the "Getting started - Android".
—
Reply to this email directly or view it on GitHub
[https://github.com/corymsmith/react-native-icons/issues/152#issuecomment-163940987] .[https://github.com/notifications/beacon/AAAckw239rQNYW19SkrSBP4WSbMmMep4ks5pOsy6gaJpZM4GzbNu.gif]

In package json I have
"react-native": "^0.15.0",
in build.gradle I have:
compile "com.facebook.react:react-native:0.15.+"
Should I specify "0.15.0" and react-native:0.15.0 ?

Either should work. Are you still getting the same error?
On Mon, Dec 14, 2015 at 6:43 AM, corymsmith/react-native-icons
<EMAIL_ADDRESS>wrote:
In package json I have
"react-native": "^0.15.0",
in build.gradle I have:
compile "com.facebook.react:react-native:0.15.+"
Should I specify "0.15.0" and react-native:0.15.0 ?
—
Reply to this email directly or view it on GitHub
[https://github.com/corymsmith/react-native-icons/issues/152#issuecomment-164441606] .[https://github.com/notifications/beacon/AAAck0X3xuaJGQwuqySlx4GRzdPw5j5mks5pPr8MgaJpZM4GzbNu.gif]

i have the same issue @corymsmith

@jawadrehman What error are you getting? Also, what version of React Native is in your package.json file & gradle file. And what version of this library?

I would highly recommend updating to 0.16.0. The error above doesn't seem like it has anything to do with this project. The example project uses 0.16.0 and works great

@corymsmith i tried updating to version 16  i got a bunch of other errors about something with UndefinedView .
so had to downgrade to 0.15

i will try to upgrade and resolve the issues.

@jawadrehman @nbastoWM Did you end up sorting this out? I'm hoping to close this issue since I'm fairly certain its not related to this project.

I'm using 0.6.3. I guess 0.7 requires RN 0.16.
Version 0.6.3 seems to work fine for me.
Thanks.

@corymsmith version 0.7 worked fine with 0.16


License

It would be nice to have a license for this package

I agree, I would like to package as dependency for browsr.

And here it is now: https://github.com/darrenburns/rich-pixels/blob/master/LICENSE.


Wand does not work if player has been deleted but not auth

The player's wand is being set to "fly" if the player does not have a last login time, but players whose data has been deleted (e.g. via /remove_player) will not have a "powerup" state set anymore (meta was deleted) but will still have a last login time.  You should probably just be setting the default powerup state based on whether or not the player has a stored powerup state, rather than on whether their auth has previously been used.

Hopefully fixed with https://github.com/regulus79/regulus_mtgamejam_2023/commit/c2b833cdf7e7c9fa52ca81f3b55b3f11d8c76612


QubesOS 4.0 support

this is a rebased + squashed + improved version of #85, main features are to read (and listen for changes) firewall rules from QubesDB; and support of hostnames in firewall rules (using the Dns_client).
A test unikernel is available in the test subdirectory (depending on a slightly modified version of Alcotest).
This project was done by @linse and @yomimono in summer 2019.

Anxiously awaiting a release which respects qubes 4 firewall rules.

Hi @talex5 ,
this PR adds:

support for Qubes 4
dynamic rule updates via GUI (and more complex rules via QubesDB)
rules that include host names for easy reading
based on @talex5 qubes-test-mirage:

a test unikernel with extensive automated testing (tests/test.sh) using echo servers
feedback for tester on how to set up test environment

We look forward to your review, and feedback when testing it (/cc @marmarek @xaki23):
$ git clone -b squash https://github.com/roburio/mirage-qubes-firewall.git
$ cd mirage-qubes-firewall
$ mirage configure -t xen
$ make depend
$ make

Outstanding temporary pins:

dns-client with timeout (https://github.com/mirage/ocaml-dns/pull/223)
alcotest-mirage (https://github.com/mirage/alcotest/pull/238)

We don't solve the virtualization mode issue, and still require PV mode.
We are happy to discuss architecture and rework it in a later PR after this has been merged.
Joint work with @yomimono 🐱

$ git clone -b squash https://github.com/roburio/mirage-qubes-firewall.git

this part is 404ing on me, probably the repo is set to not-public.

Outstanding temporary pins:

meaning i need to pin these in my build env?
which ocaml version should be used?

404

please try https://github.com/roburio/qubes-mirage-firewall.git

meaning i need to pin these in my build env?

no, make depend takes care of them :)

which ocaml version should be used?

4.07 or 4.08 should work.

build through qubes-builder works.
resulting FW deployed as template and running in a minor role (few interfaces, many downstream up/down).
basic functionality works.
i do not have any fancy/dynamic/dnsbased FW rules, so that part is untested.
minor unusual loglines during startup: (the timestamped lines are just for context)
2020-04-17 16:03:48 -00:00: INF [udp] UDP interface connected on <IP_ADDRESS>
Attempt to open(/dev/urandom)!
Unsupported function getpid called in Mini-OS kernel
Unsupported function getppid called in Mini-OS kernel
2020-04-17 16:03:48 -00:00: INF [dao] Watching backend/vif

that role was running a 2020-03-18 build before.
the initial memory footprint looks similar.
for now: +1 for merge (iff that RNG blurb isnt anything important)

i am seeing a problem with the NAT support.

setup is "netvm - miragefw - appvms"
start some long-running tcp connection (like ssh) in appvm A
start some random other appvm B
notice the ssh session locked up

a closer look on the netvm side says the from-port of the NAT mapping changes.
[the ip of the m-fw is <IP_ADDRESS>]
13:05:38.449622 IP (tos 0x0, ttl 63, id 53265, offset 0, flags [none], proto TCP (6), length 40)
    <IP_ADDRESS>.62294 > <IP_ADDRESS>.22: Flags [.], cksum 0xe8d9 (correct), ack 18209, win 997, length 0
[at this point some random other appvm downstream of the m-fw is started]
13:05:44.441748 IP (tos 0x0, ttl 63, id 53266, offset 0, flags [none], proto TCP (6), length 168)
    <IP_ADDRESS>.43179 > <IP_ADDRESS>.22: Flags [P.], cksum 0x4ff0 (correct), seq 705149198:705149326, ack<PHONE_NUMBER>, win 1002, length 128
[note 62294 != 43179]

i am not sure if this means the m-fw is

forgetting the nat-mapping entirely (which breaks the session) AND creates a random new one (which violates tcp state tracking because it means creating a mapping for something that isnt the beginning of the tcp session) or
just randomly changes the existing mapping in place (which just breaks the session).

is there some way to enable full connection state logging?

I think it's the former - connecting an AppVM to a network-providing VM triggers firewall reload (in theory only for that new VM, but the easiest thing to do is to reload everything).

What's the difference between reload and restart?
Do we have to handle a reload event in a different manner?

On firewall update you should preserve NAT table, otherwise changing any firewall rule will interrupt established connections (as @xaki23 demonstrated above).

@marmarek thanks for your reply. what does a reload or update mean? does it entail restart of the firewall VM?
if that is the case, we need a place to persist the data (qubesdb / block device / ...).

No, just updating entries in qubesdb, which (should) trigger reloading the rules.

@xaki23 With the last commit, the NAT table is only reset when the rules have actually changed. Can you give us some feedback on whether this fixes your issue? Thanks!

@xaki23 With the last commit, the NAT table is only reset when the rules have actually changed. Can you give us some feedback on whether this fixes your issue? Thanks!

sadly, this does not seem to fix the issue.
is there some way to enable debug-logging?
what i am currently seeing is:
2020-04-23 13:38:42 -00:00: INF [client_net] add client vif {domid=85;device_id=0} with IP <IP_ADDRESS>
2020-04-23 13:38:43 -00:00: INF [qubes.db] got rm "/qubes-iptables-domainrules/"
2020-04-23 13:38:43 -00:00: INF [qubes.db] got update: "/qubes-iptables-header" = "# Generated by Qubes Core on Thu Apr 23 15:38:43 2020\n*filter\n:INPUT DROP [0:0]\n:FORWARD DROP [0:0]\n:OUTPUT ACCEPT [0:0]\n-A INPUT -i vif+ -p udp -m udp --dport 68 -j DROP\n-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT\n-A INPUT -p icmp -j ACCEPT\n-A INPUT -i lo -j ACCEPT\n-A INPUT -j REJECT --reject-with icmp-host-prohibited\n-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT\n-A FORWARD -i vif+ -o vif+ -j DROP\nCOMMIT\n"
2020-04-23 13:38:43 -00:00: INF [qubes.db] got update: "/qubes-iptables-domainrules/80" = "*filter\n-A FORWARD -s <IP_ADDRESS> -j ACCEPT\n-A FORWARD -s <IP_ADDRESS> -j DROP\nCOMMIT\n"
2020-04-23 13:38:43 -00:00: INF [qubes.db] got update: "/qubes-iptables-domainrules/81" = "*filter\n-A FORWARD -s <IP_ADDRESS> -j ACCEPT\n-A FORWARD -s <IP_ADDRESS> -j DROP\nCOMMIT\n"
2020-04-23 13:38:43 -00:00: INF [qubes.db] got update: "/qubes-iptables-domainrules/85" = "*filter\n-A FORWARD -s <IP_ADDRESS> -j ACCEPT\n-A FORWARD -s <IP_ADDRESS> -j DROP\nCOMMIT\n"
2020-04-23 13:38:43 -00:00: INF [qubes.db] got update: "/qubes-iptables-domainrules/82" = "*filter\n-A FORWARD -s <IP_ADDRESS> -j ACCEPT\n-A FORWARD -s <IP_ADDRESS> -j DROP\nCOMMIT\n"
2020-04-23 13:38:43 -00:00: INF [qubes.db] got update: "/qubes-iptables" = "reload"
2020-04-23 13:38:43 -00:00: INF [qubes.db] got update: "/mapped-ip/<IP_ADDRESS>/visible-ip" = "<IP_ADDRESS>"
2020-04-23 13:38:43 -00:00: INF [qubes.db] got update: "/mapped-ip/<IP_ADDRESS>/visible-gateway" = "<IP_ADDRESS>"
2020-04-23 13:38:43 -00:00: INF [qubes.db] got rm "/qubes-firewall/<IP_ADDRESS>/"
2020-04-23 13:38:43 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>/0000" = "action=accept"
2020-04-23 13:38:43 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>/policy" = "drop"
2020-04-23 13:38:43 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>" = ""
2020-04-23 13:38:51 -00:00: INF [client_net] Client 85 (IP: <IP_ADDRESS>) ready
2020-04-23 13:38:51 -00:00: INF [ethernet] Connected Ethernet interface fe:ff:ff:ff:ff:ff
2020-04-23 13:38:51 -00:00: INF [qubes.db] got rm "/qubes-firewall/<IP_ADDRESS>/"
2020-04-23 13:38:51 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>/0000" = "action=accept"
2020-04-23 13:38:51 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>/policy" = "drop"
2020-04-23 13:38:51 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>" = ""

somewhere during that, client 82 nat state is lost.
the old state was probably (thats the last block if qubes.db lines in the log):
2020-04-23 13:29:39 -00:00: INF [qubes.db] got rm "/qubes-iptables-domainrules/"
2020-04-23 13:29:39 -00:00: INF [qubes.db] got update: "/qubes-iptables-header" = "# Generated by Qubes Core on Thu Apr 23 15:29:39 2020\n*filter\n:INPUT DROP [0:0]\n:FORWARD DROP [0:0]\n:OUTPUT ACCEPT [0:0]\n-A INPUT -i vif+ -p udp -m udp --dport 68 -j DROP\n-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT\n-A INPUT -p icmp -j ACCEPT\n-A INPUT -i lo -j ACCEPT\n-A INPUT -j REJECT --reject-with icmp-host-prohibited\n-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT\n-A FORWARD -i vif+ -o vif+ -j DROP\nCOMMIT\n"
2020-04-23 13:29:39 -00:00: INF [qubes.db] got update: "/qubes-iptables-domainrules/80" = "*filter\n-A FORWARD -s <IP_ADDRESS> -j ACCEPT\n-A FORWARD -s <IP_ADDRESS> -j DROP\nCOMMIT\n"
2020-04-23 13:29:39 -00:00: INF [qubes.db] got update: "/qubes-iptables-domainrules/81" = "*filter\n-A FORWARD -s <IP_ADDRESS> -j ACCEPT\n-A FORWARD -s <IP_ADDRESS> -j DROP\nCOMMIT\n"
2020-04-23 13:29:39 -00:00: INF [qubes.db] got update: "/qubes-iptables-domainrules/84" = "*filter\n-A FORWARD -s <IP_ADDRESS> -j ACCEPT\n-A FORWARD -s <IP_ADDRESS> -j DROP\nCOMMIT\n"
2020-04-23 13:29:39 -00:00: INF [qubes.db] got update: "/qubes-iptables-domainrules/82" = "*filter\n-A FORWARD -s <IP_ADDRESS> -j ACCEPT\n-A FORWARD -s <IP_ADDRESS> -j DROP\nCOMMIT\n"
2020-04-23 13:29:39 -00:00: INF [qubes.db] got update: "/qubes-iptables" = "reload"
2020-04-23 13:29:39 -00:00: INF [qubes.db] got update: "/mapped-ip/<IP_ADDRESS>/visible-ip" = "<IP_ADDRESS>"
2020-04-23 13:29:39 -00:00: INF [qubes.db] got update: "/mapped-ip/<IP_ADDRESS>/visible-gateway" = "<IP_ADDRESS>"
2020-04-23 13:29:39 -00:00: INF [qubes.db] got rm "/qubes-firewall/<IP_ADDRESS>/"
2020-04-23 13:29:39 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>/0000" = "action=accept"
2020-04-23 13:29:39 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>/policy" = "drop"
2020-04-23 13:29:39 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>" = ""
2020-04-23 13:29:48 -00:00: INF [qubes.db] got rm "/qubes-firewall/<IP_ADDRESS>/"
2020-04-23 13:29:48 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>/0000" = "action=accept"
2020-04-23 13:29:48 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>/policy" = "drop"
2020-04-23 13:29:48 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>" = ""

note the "rm domainrules" part near the beginning, so if the "compare old and new rules" code is checking for object-identity, it is probably considering the "new rules" new even though their content didnt change...

@xaki23 thanks for your feedback. we figured out that we should only drop the connections of the affected client from the NAT table.
we adapted the code (in f087ba1) and tested with a running ssh connection from a VM while restarting a different VM. the ssh connection persisted.
feel free to test and send us feedback.

feel free to test and send us feedback.

no noticeable difference/improvement for me.
and yes, i double-checked i am actualy running a fresh build.
i considered adding the logging from 6c4c859 back in, but aiui the change is not in the "detect if the rules changed" but in the "drop only nat table entries for one ip" direction?

Excuse us, we reverted too much. 🌴 Does it work now?

@marmarek Can you help us to get rid of the "doesn't support firewall" popup in the Qubes Manager that appears when we try to change the firewall rules?
What does the firewall need to do to announce this capability to the system?

Excuse us, we reverted too much. palm_tree Does it work now?

yes, the "loss of connections on other-vm (re)start" seems to be fixed now.
and HA, that would have been sooo confusing if it suddenly worked if i "just added the detect-if-things-changed and logging back"...
deploying this version in a major role and will report anything else i notice.
summary: still/again +1 for merging this.

What does the firewall need to do to announce this capability to the system?

Ideally, support this: https://dev.qubes-os.org/projects/core-admin/en/latest/qubes-features.html
In short: when running in a TemplateVM, at startup write to qubesdb /features-request/qubes-firewall with value 1, then call qubes.Features.Request qrexec service to dom0 (nothing specific to be send, just call the service).
A manual workaround: in dom0 call qvm-features <vmname> qubes-firewall 1

did see two crashes of yesterdays 4fbe779 by now.
both times the fw vm simply stops while trying to bring up a new VM, right after logging the "removed from NAT table" to console.
[...]
2020-04-25 13:51:37 -00:00: INF [qubes.db] got update: "/qubes-iptables" = "reload"
2020-04-25 13:51:37 -00:00: INF [qubes.db] got update: "/mapped-ip/<IP_ADDRESS>/visible-ip" = "<IP_ADDRESS>"
2020-04-25 13:51:37 -00:00: INF [qubes.db] got update: "/mapped-ip/<IP_ADDRESS>/visible-gateway" = "<IP_ADDRESS>"
2020-04-25 13:51:37 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>/0000" = "action=accept"
2020-04-25 13:51:37 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>/policy" = "drop"
2020-04-25 13:51:37 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>" = ""
2020-04-25 13:51:39 -00:00: INF [client_eth:dom120:<IP_ADDRESS>] who-has <IP_ADDRESS>? responding with fe:ff:ff:ff:ff:ff
2020-04-25 13:51:43 -00:00: INF [client_net] Client 135 (IP: <IP_ADDRESS>) ready
2020-04-25 13:51:43 -00:00: INF [ethernet] Connected Ethernet interface fe:ff:ff:ff:ff:ff
2020-04-25 13:51:43 -00:00: INF [client_net] Connections for client <IP_ADDRESS> removed from NAT table

and then the VM stops without any additional hints about why.
only thing that might be slightly unusual is that this was the first start of a "fresh" vm, not sure what it was last night.
another slightly odd detail i noticed ...
2020-04-25 13:57:15 -00:00: INF [qubes.db] got update: "/qubes-iptables" = "reload"
2020-04-25 13:57:15 -00:00: INF [qubes.db] got update: "/mapped-ip/<IP_ADDRESS>/visible-ip" = "<IP_ADDRESS>"
2020-04-25 13:57:15 -00:00: INF [qubes.db] got update: "/mapped-ip/<IP_ADDRESS>/visible-gateway" = "<IP_ADDRESS>"
2020-04-25 13:57:15 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>/0000" = "action=accept"
2020-04-25 13:57:15 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>/policy" = "drop"
2020-04-25 13:57:15 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>" = ""
2020-04-25 13:57:21 -00:00: INF [client_net] Client 139 (IP: <IP_ADDRESS>) ready
2020-04-25 13:57:21 -00:00: INF [ethernet] Connected Ethernet interface fe:ff:ff:ff:ff:ff
2020-04-25 13:57:21 -00:00: INF [client_net] Connections for client <IP_ADDRESS> removed from NAT table
2020-04-25 13:57:22 -00:00: INF [qubes.db] got rm "/qubes-firewall/<IP_ADDRESS>/"
2020-04-25 13:57:22 -00:00: INF [client_net] Connections for client <IP_ADDRESS> removed from NAT table
2020-04-25 13:57:22 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>/0000" = "action=accept"
2020-04-25 13:57:22 -00:00: INF [client_net] Connections for client <IP_ADDRESS> removed from NAT table
2020-04-25 13:57:22 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>/policy" = "drop"
2020-04-25 13:57:22 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>" = ""

this one doesn't involve a crash, everything is working fine, this is the "normal" case.
but note how it logs the "removed from NAT table" three times as part of a regular vm bringup.
trying again with a build that includes 2753e82 now.

the "silent crash on client vm start" is still happening with the 2753e82 build.
i do not have a clear repro or any idea how to get more debug info on this.

silent crash on client vm start

Check /var/log/xen/console/hypervisor.log, maybe it isn't totally silent.

Check /var/log/xen/console/hypervisor.log, maybe it isn't totally silent.

good suggestion!
two "patterns" after grouping by "entry.o#create_bounce_frame" offsets.
three like:
(XEN) Pagetable walk from<PHONE_NUMBER>246ff8:
(XEN)  L4[0x000] =<PHONE_NUMBER>d19067<PHONE_NUMBER>000410
(XEN)  L3[0x000] =<PHONE_NUMBER>d1a067<PHONE_NUMBER>000411
(XEN)  L2[0x001] =<PHONE_NUMBER>d1c067<PHONE_NUMBER>000413
(XEN)  L1[0x046] =<PHONE_NUMBER>f44025<PHONE_NUMBER>000246
(XEN) domain_crash_sync called from entry.S: fault at ffff82d08020fc3e entry.o#create_bounce_frame+0x67/0x142
(XEN) Domain 119 (vcpu#0) crashed on cpu#2:
(XEN) ----[ Xen-4.8.5-15.fc25  x86_64  debug=n   Not tainted ]----
(XEN) CPU:    2
(XEN) RIP:    e033:[<00000000001e76e0>]
(XEN) RFLAGS:<PHONE_NUMBER>010246   EM: 1   CONTEXT: pv guest (d119v0)

and two two of these:
(XEN) d152v0 Unhandled general protection fault fault/trap [#13, ec=0000]
(XEN) domain_crash_sync called from entry.S: fault at ffff82d08020fd0e entry.o#create_bounce_frame+0x137/0x142
(XEN) Domain 152 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-4.8.5-15.fc25  x86_64  debug=n   Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e033:[<00000000002e200d>]
(XEN) RFLAGS:<PHONE_NUMBER>000246   EM: 1   CONTEXT: pv guest (d152v0)

each followed by a pile of register/stack hexdumps that show no obvious pattern, tell me if any of it might be useful or how to translate them to code if needed.

@xaki23 We split out the dns changes into a second commit. Can you reproduce the crash with the first commit only?

checkout cf2b7d9 .... build works ... basic functionality works ... will report on stability in the next hours/days.

running a build with just cf2b7d9 and another that also has fb42d2f ...
... and neither is crashing.
i am not "actively using" the configured-rules or dns-rules functionality, just the plain "outbound nat" usecase.
again/still: +1 for merge

still running the cf2b7d9 and fb42d2f ... and they still didnt crash. they are both up for 5+ days now a have seen about 40+ vm starts each.
actualy not even seen any https://github.com/mirage/qubes-mirage-firewall/issues/35 crashes since then. (but i am only testing this on one machine, and it is not really powerful, so i might simply not be starting enough VMs.)
i am not sure what the force-push to dd5b92b is about, but will build and test that now.

Thanks for testing, @xaki23 ! I'm glad to hear that the version before the force push is stable for you now!
In dd5b92b we adjusted the code to use a new interface to talk to QubesDB. We tried our best to cover all cases to guard against race conditions there. A stability test of this version would be super helpful as a data point!

i am afraid i have an unexpected datapoint already:
seems a switch create 4.08.1 does not work with fedora-32  ( gcc 10.0.1 and system-ocaml 4.10 )
# gcc -shared -o libcamlrun_shared.so interp_bpic.o misc_bpic.o stacks_bpic.o fix_code_bpic.o startup_aux_bpic.o startup_byt_bpic.o freelist_bpic.o major_gc_bpic.o minor_gc_bpic.o memory_bpic.o alloc_bpic.o roots_byt_bpic.o globroots_bpic.o fail_byt_bpic.o signals_bpic.o signals_byt_bpic.o printexc_bpic.o backtrace_byt_bpic.o backtrace_bpic.o compare_bpic.o ints_bpic.o floats_bpic.o str_bpic.o [...]
# /usr/bin/ld: backtrace_bpic.o:/home/user/.opam/mirage-firewall-4.08.1/.opam-switch/build/ocaml-base-compiler.4.08.1/runtime/backtrace.c:31: multiple definition of `caml_debug_info'; backtrace_byt_bpic.o:/home/user/.opam/mirage-firewall-4.08.1/.opam-switch/build/ocaml-base-compiler.4.08.1/runtime/backtrace_byt.c:47: first defined here
# collect2: error: ld returned 1 exit status
# make[1]: *** [Makefile:310: libcamlrun_shared.so] Error 1
# make[1]: Leaving directory '/home/user/.opam/mirage-firewall-4.08.1/.opam-switch/build/ocaml-base-compiler.4.08.1/runtime'
# make: *** [Makefile:323: coldstart] Error 2

switched buildvm back to fedora-31 template for now.

qubes-builder build of dd5b92b on fedora-31 worked, deployed as template, basic functionality works, will report on stability as soon as i have something to report.
noticed something that was new-to-me during the build:
File "client_net.ml", line 109, characters 6-48:
109 |       (function Lwt.Canceled -> Lwt.return_unit)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Warning 8: this pattern-matching is not exhaustive.
Here is an example of a case that is not matched:
*extension*
Matching over values of extensible variant types (the *extension* above)
must include a wild card pattern in order to be exhaustive.

same for line 129.

running a test set with 4702234 next.
the dd5b92b build seemed stable, i did some load-testing with rapid restarts of downstream vms over night, the only crashes were https://github.com/mirage/qubes-mirage-firewall/issues/35 (which happens every 64 downstream vms).

hi @xaki23,
thanks for your extensive testing.

seems a switch create 4.08.1 does not work with fedora-32 ( gcc 10.0.1 and system-ocaml 4.10 )

This looks like an OCaml runtime compilation issue, could you open an issue on the https://github.com/ocaml/ocaml repository, since we cannot reproduce due to lack of fedora-32?

Warning 8: this pattern-matching is not exhaustive.

We fixed this in commit 4702234.

seems a switch create 4.08.1 does not work with fedora-32 ( gcc 10.0.1 and system-ocaml 4.10 )

This looks like an OCaml runtime compilation issue, could you open an issue on the https://github.com/ocaml/ocaml repository, since we cannot reproduce due to lack of fedora-32?

reported there, they suggest to use newer ocaml:  https://github.com/ocaml/ocaml/issues/9540#issuecomment-625132712
is there any chance of m-fw building with OCaml 4.09.1 / 4.10.0 or newer?
workflow question.
i just noticed you changed the depext-ed versions of mirage and lwt in Dockerfile.
they probably also need to be changed in Makefile.builder (though the old versions seem to work?).
usualy i would just file a separate PR against this repo later, but if this PR stays open longer, i would prefer it is included in this PR or i am going to forget making that change again and again and again on test builds.
could one of you make that change+commit and push it here?
or should i file it as a PR against roburio:squash?
or is there some ocamlish way to have these versions specified in only one place for both build pathes? (didnt i ask that before? thinkfrowns)

is there any chance of m-fw building with OCaml 4.09.1 / 4.10.0 or newer?

we just adapted mirage-xen-ocaml to work with newer OCaml versions, see https://github.com/mirage/mirage-platform/pull/213

workflow question.
i just noticed you changed the depext-ed versions of mirage and lwt in Dockerfile.
they probably also need to be changed in Makefile.builder (though the old versions seem to work?).
usualy i would just file a separate PR against this repo later, but if this PR stays open longer, i would prefer it is included in this PR or i am going to forget making that change again and again and again on test builds.
could one of you make that change+commit and push it here?

we added these changes in commit 9ad8b9e.

or should i file it as a PR against roburio:squash?
or is there some ocamlish way to have these versions specified in only one place for both build pathes? (didnt i ask that before? thinkfrowns)

that is a good idea, but we don't know how to do this. @talex5 do you have an idea how to keep Dockerfile and Makefile.builder in sync?

this mornings build of 16d05c9 with 4.08.1 survived a full day of stressing.
moving on to 9ad8b9e (actualy https://github.com/roburio/qubes-mirage-firewall/pull/1/commits/24e7f908dfc5134956bc79d117393a196c237b46 ) with 4.10.0 now in a full main-daily-use fw role.

oh, this looks tasty. i will spare you my last hour of headscratching and get to the point:
2020-05-07 21:31:46 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>/0000" = "action=accept"
2020-05-07 21:31:46 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>/policy" = "drop"
2020-05-07 21:31:46 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>" = ""
2020-05-07 21:31:46 -00:00: INF [client_net] New firewall rules for <IP_ADDRESS>

2020-05-07 21:31:52 -00:00: INF [client_net] Client 723 (IP: <IP_ADDRESS>) ready
2020-05-07 21:31:52 -00:00: INF [ethernet] Connected Ethernet interface fe:ff:ff:ff:ff:ff
2020-05-07 21:31:52 -00:00: INF [client_net] New firewall rules for <IP_ADDRESS>
0   any   accept
2020-05-07 21:31:53 -00:00: INF [qubes.db] got rm "/qubes-firewall/<IP_ADDRESS>/"
2020-05-07 21:31:53 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>/0000" = "action=accept"
2020-05-07 21:31:53 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>/policy" = "drop"
2020-05-07 21:31:53 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>" = ""
2020-05-07 21:31:53 -00:00: INF [client_net] Rules did not change for <IP_ADDRESS>

the important part is the empty line.
my summary would be "starting vm with ip a.b.c.xy wipes rules for vm with ip a.b.c.x".
as in, something is doing an unanchored substring match!
got introduced somewhen in the last week.
the 20200430 build doesnt have this problem, the 20200506 one does.
as in, it is probably part of the "new interface to talk to QubesDB" changes.

thanks for your report, @xaki23. there indeed was an off-by-one in the qubesdb & commit implementation, which should be fixed in https://github.com/mirage/mirage-qubes/pull/52/commits/cf87a2646cb0df09f1416ba12a25d9340aa4c5a4 (which you'll get if you run the builder again, since this qmf branch pins to the mirage-qubes branch).

EDIT: the issue was that rm /qubes-firewall/<IP_ADDRESS>/ removed the entries for /qubes-firewall/<IP_ADDRESS>.

looking at the commit, thats some PHP-quality string manipulation going on there. coughs
new build (with ocaml 10.0.1 on fedora-32 again) deployed in multiple roles, can already confirm it fixes the prefix-matching issue, will report on stability in the next days.
for the record: i am still not testing the "custom/dynamic rules" parts at all, i am using it with plain default configuration.

no crashes.
no new problems.
no unexpected errors.

I've started running this version. It worked for a while, but it just exited when I started a new AppVM. There's no error in the logs; it just ends with:
2020-05-17 21:19:07 -00:00: INF [client_net] add client vif {domid=6;device_id=0} with IP <IP_ADDRESS>
2020-05-17 21:19:07 -00:00: INF [qubes.db] got update: "/mapped-ip/<IP_ADDRESS>/visible-ip" = "<IP_ADDRESS>"
2020-05-17 21:19:07 -00:00: INF [qubes.db] got update: "/mapped-ip/<IP_ADDRESS>/visible-gateway" = "<IP_ADDRESS>"
2020-05-17 21:19:07 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>/0000" = "action=accept"
2020-05-17 21:19:07 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>/policy" = "drop"
2020-05-17 21:19:07 -00:00: INF [qubes.db] got update: "/qubes-firewall/<IP_ADDRESS>" = ""
2020-05-17 21:19:10 -00:00: INF [client_net] Client 6 (IP: <IP_ADDRESS>) ready
2020-05-17 21:19:10 -00:00: INF [ethernet] Connected Ethernet interface fe:ff:ff:ff:ff:ff

I was running with 32 MB. I'll try for a bit with 64 MB.

Take a look at Xen output (xl dmesg), maybe you'll find crash message there.

xl dmesg shows this. There's no timestamp, so I don't know when it was or even if it was the firewall domain (but it says pv guest, which suggests it is the firewall):
(XEN) mwait-idle: max C-state count of 8 reached
(XEN) Pagetable walk from<PHONE_NUMBER>248fa8:
(XEN)  L4[0x000] =<PHONE_NUMBER>20f067<PHONE_NUMBER>00040f
(XEN)  L3[0x000] =<PHONE_NUMBER>210067<PHONE_NUMBER>000410
(XEN)  L2[0x001] =<PHONE_NUMBER>212067<PHONE_NUMBER>000412
(XEN)  L1[0x048] = 001000068aa48025<PHONE_NUMBER>000248
(XEN) domain_crash_sync called from entry.S: fault at ffff82d08020fc3e entry.o#create_bounce_frame+0x67/0x142
(XEN) Domain 3 (vcpu#0) crashed on cpu#3:
(XEN) ----[ Xen-4.8.5-16.fc25  x86_64  debug=n   Not tainted ]----
(XEN) CPU:    3
(XEN) RIP:    e033:[<00000000001f0f58>]
(XEN) RFLAGS:<PHONE_NUMBER>010206   EM: 1   CONTEXT: pv guest (d3v0)
(XEN) rax:<PHONE_NUMBER>000000   rbx:<PHONE_NUMBER>3b3cc9   rcx:<PHONE_NUMBER>000010
(XEN) rdx:<PHONE_NUMBER>000000   rsi:<PHONE_NUMBER>1e8ee8   rdi:<PHONE_NUMBER>2009a1
(XEN) rbp:<PHONE_NUMBER>200977   rsp:<PHONE_NUMBER>248fb8   r8:  00000000ffffffff
(XEN) r9:  00000000ffffffff   r10:<PHONE_NUMBER>000000   r11:<PHONE_NUMBER>3996b8
(XEN) r12:<PHONE_NUMBER>2490d8   r13:<PHONE_NUMBER>3b40bf   r14:<PHONE_NUMBER>249310
(XEN) r15:<PHONE_NUMBER>397e00   cr0:<PHONE_NUMBER>050033   cr4:<PHONE_NUMBER>3426e0
(XEN) cr3:<PHONE_NUMBER>20e000   cr2:<PHONE_NUMBER>248fe0
(XEN) fsb:<PHONE_NUMBER>000000   gsb:<PHONE_NUMBER>3b40d0   gss:<PHONE_NUMBER>000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=0000000000248fb8:
(XEN)<PHONE_NUMBER>000000<PHONE_NUMBER>000000<PHONE_NUMBER>000000<PHONE_NUMBER>000000
(XEN)<PHONE_NUMBER>000000<PHONE_NUMBER>000000<PHONE_NUMBER>000000<PHONE_NUMBER>000000
(XEN)<PHONE_NUMBER>000000<PHONE_NUMBER>000000<PHONE_NUMBER>000000<PHONE_NUMBER>000000
(XEN)<PHONE_NUMBER>000000<PHONE_NUMBER>000bf8<PHONE_NUMBER>249040 ffffffffffffffff
(XEN)<PHONE_NUMBER>000bfc 6d5f666f5f74754f<PHONE_NUMBER>6f6d65<PHONE_NUMBER>000bf8
(XEN)<PHONE_NUMBER>249070 fffffffffffffffd<PHONE_NUMBER>000bfc 6f7272655f737953
(XEN)<PHONE_NUMBER>000072<PHONE_NUMBER>000bf8<PHONE_NUMBER>3b3cc9<PHONE_NUMBER>1f41f4
(XEN)<PHONE_NUMBER>0007fc 000000036c696146<PHONE_NUMBER>000bf8<PHONE_NUMBER>3b3cc9
(XEN)<PHONE_NUMBER>2009ac<PHONE_NUMBER>1fb058<PHONE_NUMBER>3b40bf<PHONE_NUMBER>1f44bf
(XEN)<PHONE_NUMBER>397e00<PHONE_NUMBER>249310<PHONE_NUMBER>3b40bf<PHONE_NUMBER>1fb058
(XEN)<PHONE_NUMBER>2009ac<PHONE_NUMBER>3b3cc9<PHONE_NUMBER>3996b8<PHONE_NUMBER>000000
(XEN)    00000000ffffffff 00000000ffffffff<PHONE_NUMBER>000000<PHONE_NUMBER>000010
(XEN)<PHONE_NUMBER>1e8ee8<PHONE_NUMBER>3b40bf<PHONE_NUMBER>3b3cc9 ffffffffffffffff
(XEN)<PHONE_NUMBER>1e8ee8<PHONE_NUMBER>00e030<PHONE_NUMBER>010016<PHONE_NUMBER>249188
(XEN)<PHONE_NUMBER>00e02b<PHONE_NUMBER>249198 fffffffffffffff1<PHONE_NUMBER>000bfc
(XEN)    61665f686374614d<PHONE_NUMBER>756c69<PHONE_NUMBER>000bf8<PHONE_NUMBER>2491c8
(XEN)    ffffffffffffffef<PHONE_NUMBER>000bfc 766f5f6b7a797877 0100776f6c667265
(XEN)<PHONE_NUMBER>000bf8<PHONE_NUMBER>2491f8 ffffffffffffffed<PHONE_NUMBER>000bfc
(XEN)    636f6c625f737953 01006f695f64656b<PHONE_NUMBER>000bf8<PHONE_NUMBER>249228
(XEN)    ffffffffffffffeb<PHONE_NUMBER>000bfc 665f747265737341<PHONE_NUMBER>6c6961
(XEN) Disabling non-boot CPUs ...

I'm assuming I was running the unikernel from commit  152202b0dea6e1a8ebe3442e6ef0ec7f7bb0fe02 with hash 7a6b003e712256cce7ac8741239f6d8d5a0db4b71656396f7ee734568282c72d. Interesting addresses on the stack based on that image:
00000000001f0f50 <printk>:
  1f0f51:	48 81 ec d0 00 00 00 	sub    $0xd0,%rsp
  1f0f58:	48 89 74 24 28       	mov    %rsi,0x28(%rsp)
  1f0f5d:	48 89 54 24 30       	mov    %rdx,0x30(%rsp)

00000000001f41c0 <do_general_protection>:
  1f41ef:	e8 5c cd ff ff       	callq  1f0f50 <printk>
  1f41f4:	4c 89 e7             	mov    %r12,%rdi
  1f41f7:	e8 e4 f7 ff ff       	callq  1f39e0 <dump_regs>

00000000001f44a7 <error_call_handler>:
  1f44bd:	ff d0                	callq  *%rax
  1f44bf:	eb 5f                	jmp    1f4520 <error_exit>

00000000001e8ec0 <number>:
  1e8ee7:	7a
  1e8ee8:	0f 29 44 24 20       	movaps %xmm0,0x20(%rsp)
  1e8eed:	66 0f 6f 05 8b 6e 01 	movdqa 0x16e8b(%rip),%xmm0        # 1ffd80 <caml_absf_mask+0xda0>

000000000000df60 <camlFirewall__fun_1142>:
    e028:	48 89 fb             	mov    %rdi,%rbx
    e02b:	48 83 c4 08          	add    $0x8,%rsp
    e02f:	e9 1c f6 ff ff       	jmpq   d650 <camlFirewall__transmit_ipv4_126>

Not sure what's going on here. A stack-overflow maybe?


nextPageCursor in runQuery changes every time when last entity gets modified

Hi,
I'm using datastore client library of version 1.3.3
The 'endCursor' in the runQuery result changes when I update the last item of that query result. If I try to fetch the next set of items using the cursor which I got with the previous query, it returns set of entities in which the first entity will be the duplicate of last entity in the previous set.
like
5007404484788224
5289011263307776
5805078561685504
-- new set --
5805078561685504
4785039263924224
4655690686660608

Here is the code I have used to fetch the entities from datastore
query = dataStore.createQuery(Kind);
query.select();
  .order(, {
    descending: true,
  })
  .limit(10);
dataStore.runQuery(query, function(err, entities, cursorInfo) {
  console.log(err);
}

runQuery sometimes fails with error: 'Error parsing protocol message' while fetching entities using the cursor.
Steps to reproduce
Fetch 1 - 10 items. The client holds the cursor
Modify 10th item
Try to retrieve 11 - 20 using the cursor of Step 1. It fails
Please help me in this regard.
Thanks!

I'm going to close this. Please reopen this issue once you add the additional context.


Moved the glyphs for repo stars and forks to the left side of the text for improved readability

Reading the number of stars and forks a repo has is kind of awkward with the glyph in the middle. This PR moves them to the left as below, which is a bit cleaner and separates the stars from the forks better:

Thanks folks - we've gone a different redirection now that Microsoft's repos index is growing so rapidly, so are in the process of rolling out https://opensource.microsoft.com. We're redirecting the GitHub Pages site in the meantime over there. Thanks, and sorry it's taken so long to triage what all is up here.