alpha4 version composer install error on google app engine

I have installed the alpha4 version on local server and it's working fine in my local computer. I had to install aplha4 version due to support for cloud storage but when  i deploy my code on google app engine i get the 'unisharp/laravel-filemanager v2.0.0-alpha4 requires ext-exif * -> the requested PHP extension exif is missing from your system.' error. I checked the php info of my google app engine and the 'exif' extension has been enabled.
I have attached the screenshot for both error message and exif php info.Can anyone help on this?

Composer runs in terminal environment thus uses the php-cli interpreter. Your webserver on the other hand runs php-fpm. You might need to enable exif on php-cli as well. To check if it is enabled please run php —ini  to determine if it is installed.

Hi gwleuverink,
Thank for the quick reply.I just run 'php -ini' command on cloud shell in google cloud console and it retured following info for exif. I think exif is enabled for php-cli?
exif
EXIF Support => enabled
EXIF Version => 7.0.30-0+deb9u1
Supported EXIF Version => 0220
Supported filetypes => JPEG,TIFF
Directive => Local Value => Master Value
exif.decode_jis_intel => JIS => JIS
exif.decode_jis_motorola => JIS => JIS
exif.decode_unicode_intel => UCS-2LE => UCS-2LE
exif.decode_unicode_motorola => UCS-2BE => UCS-2BE
exif.encode_jis => no value => no value
exif.encode_unicode => ISO-8859-15 => ISO-8859-15

It seems like exif is enabled alright. Can you confirm you are running the php —ini command on the same server you are deploying your code to?

The problem still exists.
unisharp/laravel-filemanager v2.1.0 requires ext-exif * -> the requested PHP extension exif is missing from your system.
P.S. The extention is installed it shows that it is enabled.
OS: Windows 10

Problem 1
- Root composer.json requires unisharp/laravel-filemanager ^2.2.0 -> satisfiable by unisharp/laravel-filemanager[v2.2.0].
- unisharp/laravel-filemanager v2.2.0 requires ext-exif * -> it is missing from your system. Install or enable PHP's exif extension.


Add ses-rook (SOC-10065)

Add deploy_ses_rook
Adds a role for deploying ses via rook
Adds the actions deploy_ses_rook and delete_ses_rook to run.sh
Adds disks to the caasp4 workers as they are needed to provision
ceph via rook
Adds testing of the ceph cluster by provisioning a pvc on a ceph
block storageclass
Adds the ses-rook path to jenkins
Adds a new choice to select the normal ses or ses-rook in jenkins

currently ses is properly deployed via rook but airship fails because its looking for some vars only available from normal ses deployment.
As this is a POC Im not sure we want to continue pursuing this or we are ok with this as something to build upon

@toabctl I updated it to move any dependant things on other roles/tasks into the playbook and remove any non-generic calls out, so the role should be pretty much standalone now!

Seems to pass correctly with the latest updates: https://ci.suse.de/blue/organizations/jenkins/cloud-socok8s-jobs%2Fsocok8s-integration/detail/PR-669/21/pipeline
Removing the extra commit for CI + do-not-merge tag

Do we somehow need to lock this to caaspv4 in order to pass ci?

This code path is not executed by CI so the failures are unrelated...

Do we somehow need to lock this to caaspv4 in order to pass ci?

yep, ses-in-rook is only deployable under caasp4, so they go together

Merging, does not affect CI


Container App Update Fails from null secrets

Please provide us with the following information:

This issue is a: (mark with an x)

[ X ] bug report -> please search issues before submitting
[ ] documentation issue or request
[ ] regression (a behavior that used to work and stopped in a new release)

Issue description
There appears to be an issue where a Container Registry secret was created from a previous Container App, but is null and it prevents other Container Apps from being modified.
Steps to reproduce
Create a Container App and specify registry credentials.
Create another container app and don't specify registry credentials.
Try to update the 2nd container app YAML and it fails with secrets being null.
az containerapp create \
  --name $ACA_WEB_NAME \
  --resource-group $RESOURCE_GROUP \
  --image $ACA_WEB_IMAGE \
  --environment $ACA_ENV_NAME \
  --target-port 3334 \
  --ingress 'external' \
  --registry-server $REGISTRY_SERVER \
  --registry-username $REGISTRY_USERNAME \
  --registry-password $REGISTRY_PASSWORD \
  --cpu 0.5 --memory 1.0Gi \
  --min-replicas 1

➜  ACA_NAME=helloworld

➜  az containerapp create \
  --name $ACA_NAME \
  --resource-group $RESOURCE_GROUP \
  --environment $ACA_ENV_NAME \
  --image mcr.microsoft.com/azuredocs/containerapps-helloworld:latest \
  --target-port 80 \
  --ingress 'external' \
  --query properties.configuration.ingress.fqdn
Command group 'containerapp' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
- Running ..
Container app created. Access your app at https://helloworld.livelycliff-917e6386.eastus.azurecontainerapps.io/

"helloworld.livelycliff-917e6386.eastus.azurecontainerapps.io"

➜  az containerapp show -n $ACA_WEB_NAME -g $RESOURCE_GROUP -o yaml > helloworld.yaml

WARNING: Command group 'containerapp' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus

## NOTE.  No changes to helloworld.yaml have been made.  But in the file, there is a refernce to a secret that I don't think should be there from a previous Container App.
"spaghettidetective" is a Container App I created beforehand.

properties:
  configuration:
    registries:
    - passwordSecretRef: spaghettidetectiveazurecrio-spaghettidetective
      server: spaghettidetective.azurecr.io
      username: spaghettidetective
    secrets:
    - name: spaghettidetectiveazurecrio-spaghettidetective

➜  az containerapp update -n $ACA_WEB_NAME -g $RESOURCE_GROUP --yaml helloworld.yaml
Command group 'containerapp' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Invalid Request: Container app secret(s) with name(s) 'spaghettidetectiveazurecrio-spaghettidetective' cannot have a value of null.

Expected behavior
I expect to be able to update an un-modified configuration without error.  Ultimately, I'd like to be able to modify the YAML to add a mount, but this prevents me from getting that far.
Actual behavior
Updating the Container App YAML fails, even if it's unmodified.  I believe it's due to a Contaienr Registry secret from a previous Container App.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Ex. Did this issue occur in the CLI or the Portal?

This issue should be addressed by end of June.

Same issue here.
az containerapp show not contains secret value but only secret name, so we could not reuse exported yaml if container app use secret for below scenario.
https://docs.microsoft.com/en-us/azure/container-apps/storage-mounts?pivots=aca-cli#azure-files

REST API also does not contain secrets[].value
https://docs.microsoft.com/en-us/rest/api/containerapps/container-apps/get?tryIt=true&source=docs#secret

Hey all, sorry about this. The problem here is that the update command uses a GET followed by a PUT to perform the update. Since the GET API doesn't return secrets, the update fails. We currently have a PR in review to change the CLI to use the PATCH API to perform updates with az containerapp update. This change should go out with the next version of the CLI extension

ref https://github.com/microsoft/azure-container-apps/issues/299#issuecomment-1180824629

Hello I have the same problem. Does anyone have an idea for a workaround? When can I expect the next version?

Can not use Azure Container Registry with 'az containerapp update'
Hi ,
I am not able to use Azure Container Registry with 'az containerapp update' command in following way

If Admin enabled in Azure Container Registry - Throws Unauthorized error

If Admin not enabled in Azure Container Registry - Throws Unauthorized error

Is there any way to use Azure container Registry with 'az containerapp update' command?

Hi @ranjeetkpGit please create a new issue if you are experiencing this problem as it is unrelated to the initial report


feat: add winblend option for windows

This adds the :h 'winblend' option to creating windows so we can have psuedo-transparent completion windows.

I think a nice default for the autocompletion winblend config would be :h 'pumblend', but I don't see other options falling back to similar vim options (pumwidth/pumheight) so i've left it at 0.

Looks great, thanks! I agree with not defaulting to pumblend for now. It may be slightly confusing and it's trivial for users to configure themselves


Fixed README. Added LICENSE

Pull Request Test Coverage Report for Build<PHONE_NUMBER>8
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build<PHONE_NUMBER>1:
0.0%

Covered Lines:
52

Relevant Lines:
52

💛  - Coveralls


Better generated client modules

A redesign of how the generated client modules look like, making it possible to have an alternative "full API" module and to initialize all clients at once while still providing the benefits of the fine grained resource specification.
Resolves #77.
Before this change, a client module was defined as something like the following:
type Pods = Has[NamespacedResource[Pod]]
    with Has[NamespacedResourceStatus[PodStatus, Pod]]
    with Has[NamespacedLogSubresource[Pod]]
    with Has[NamespacedEvictionSubresource[Pod]]
    with Has[NamespacedBindingSubresource[Pod]]

// ZIO accessor functions for all the above interfaces

val live: ZLayer[SttpClient with Has[K8sCluster], Nothing, Pods] = // ...

This works well and has the advantage of having nice type signatures like ZIO[Pods with ConfigMaps, ...] while still being able to write generic functions that work on any resource or with any resource capability (such as arbitrary resources that support a given subresource).
The downside is if there are many resource types used in the program, the union environment type will become huge. Also the layer initialization will be huge, although this could be selled as a feature - you see all the resource types your application will use in one place ;)
But I see this as a pain point that can reduce the usability of the library so this PR introduces two major changes.
First, the resource client modules are now "more proper" ZIO modules, with a service trait defined for each. The above example now looks like this:
type Pods = Has[NamespacedResource[Pod]]
    with Has[NamespacedResourceStatus[PodStatus, Pod]]
    with Has[NamespacedLogSubresource[Pod]]
    with Has[NamespacedEvictionSubresource[Pod]]
    with Has[NamespacedBindingSubresource[Pod]]

trait Service
    extends NamespacedResource[Pod]
    with NamespacedResourceStatus[PodStatus, Pod]
    with NamespacedLogSubresource[Pod]
    with NamespacedEvictionSubresource[Pod]
    with NamespacedBindingSubresource[Pod] {

  val asLayer: Pods = // ... (Has(this) ++ Has(this) ++ ... for all the traits
}

final class Live(
  client: ResourceClient[Pod],
  statusClient: ResourceStatusClient[PodStatus, Pod],
  logClient: SubresourceClient[String],
  evictionClient: SubresourceClient[Eviction],
  bindingClient: SubresourceClient[Binding]
) extends Service {
  // delegates
}

// ZIO accessor methods

val live: ZLayer[SttpClient with Has[K8sCluster], Nothing, Pods] =
  // ZLayer.fromServicesMany { ... new Live(...).asLayer }

This alone does not change anything just makes the generated code more complicated. A natural next step would be to drop the compound type alias and use just Pods = Has[Service]. If I do that the generic use cases will no longer work out of the box. For that each resource client service trait defines an asLayer (or could be called "explode") conversion that basically just tricks the invariant Has and explodes it to the union of the base traits. I explain the reason why I did not change the type alias after examining the second change.
The second change in the PR is a new generated module kubernetes that looks like this:
package object kubernetes {
  class KubernetesApi(backend: SttpClient.Service, cluster: K8sCluster) {
    object v1 {
      lazy val pods: pods.Live = // ...
    }
    // ...
  }

  type Kubernetes = Has[KubernetesApi]

  val live: ZLayer[SttpClient with Has[K8sCluster], Nothing, Kubernetes] = // ...
}

So basically this is a layer that initializes (lazily) and provides access to all resource types at once for those who don't want fine grained control in their effect types. To move between the two styles, the above explained .asLayer conversion can be used together with provideCustomLayer.
The reason I left the per-resource type aliases as they are is that this way the two different styles can better solve their own purpose. When using the "fine grained style" the generic functions and the generated accessor functions work well together and no conversion is needed. At the same time if someone needs a lot of resources or just does not care about fine grained control can initialize the client in one step, and use them by getting the unified interface from the environment and calling its nested members. A further step could be in this unified experience to generate a hierarchic set of accessor functions for the unified Kubernetes service but I'm not sure about it, seems a bit overkill.

After talking with @iravid, additional new changes:
The default type alias for the resource client modules has been changed to the "simple" one and an alternative is provided that provides the compound version:
type Pods = Has[Service]
type PodsGeneric = Has[NamespacedResource[Pod]]
    with Has[NamespacedResourceStatus[PodStatus, Pod]]
    with Has[NamespacedLogSubresource[Pod]]
    with Has[NamespacedEvictionSubresource[Pod]]
    with Has[NamespacedBindingSubresource[Pod]]

everything that used Pods in R still do so the simple one became the default.
I renamed asLayer to asGeneric that converts a a pods.Service to PodsGeneric.
Also added a ZLayer extension method that is a bit like narrow in zio-config but does more because currently the members in the unified api class are typed by the Live classes not the service traits.  This looks like
k8s.focus[pods.Service](_.v1.pods)

but most likely it is possible to just change the field types to x.Service instead of x.Live and then it can be completely equivalent to the narrow and the type parameters will not be needed. I'll try to do this immediately.


Use lower case for source dir

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 65.85%. Comparing base (9af61e9) to head (f6142ba).

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #13   +/-   ##
=======================================
  Coverage   65.85%   65.85%
=======================================
  Files          63       63
  Lines         984      984
  Branches       70       70
=======================================
  Hits          648      648
  Misses        295      295
  Partials       41       41

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


REF: Rework concat datasets and make sure the TrainTestDataset also works with numpy arrays

This is done in #112 now right?


Integrated plugin dialog with install/uninstall & remote discovery

Description
As mentioned in https://github.com/napari/napari/pull/1289#issuecomment-640995785, I'm working on a more integrated plugin dialog:

Features include:

finds available plugins by naming convention on PyPI
finds available plugins on github, even without naming convention (such as ome-zarr) based on these conditions (This was the best I could do without a PyPI trove classifier):

they have the literal string "napari plugin" in the readme
they have a setup.py or setup.cfg file that declares a "napari.plugin" entry_point
they are published on PyPI with development status Alpha or greater.

pip install/uninstall in background process with stdout feedback in the GUI. (paths and environment variables preserved so this would work in a bundle)
detects outdated plugins and offers update button.
integrated plugin sorter
integrated error report (plugins with errors show a clickable warning icon that slides out the report).

(The checkbox next to each plugin will be used to activate/deactivate a plugin without actually uninstalling it)
This requires some changes to napari-plugin-engine that are not yet merged, so if you want to test this locally, you'll need to checkout the napari-plugin-engine/refactor-discovery branch
Type of change

[ ] Bug-fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

References

How has this been tested?

[ ] example: the test suite for my feature covers cases x, y, and z
[ ] example: all tests pass with my change

Final checklist:

[ ] My PR is the minimum possible work for the desired functionality
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works

You could try to request a trove classifier to the PyPI team, pytest has one.

You could try to request a trove classifier to the PyPI team

Yeah, I found a comment to the effect that we need ~10 third party plugins before they'll consider adding a Trove classifier. We're almost there but not quite!

This requires some changes to napari-plugin-engine that are not yet merged, so if you want to test this locally, you'll need to checkout the napari-plugin-engine/refactor-discovery branch

hi @tlambert03 I was going to try and give this some review today, but ran into errors on launch with npe 0.1.5 and 0.1.7. I also couldn't find that branch you mention above. If you let me know how to get started with this one I can do a review and hopefully we can get it in for 0.3.8 :-)
Also let me know if there's anything in particular you want feed back on/ want me to check out, sorry it has sat here for so long!!

hi @tlambert03 I was going to try and give this some review today, but ran into errors on launch with npe 0.1.5 and 0.1.7. I also couldn't find that branch you mention above. If you let me know how to get started with this one I can do a review and hopefully we can get it in for 0.3.8 :-)

hmm, that branch has actually been merged and released.  I just tested this again with napari-plugin-engine-0.1.7 and it worked.  can you reinstall npe, making sure you're not on a local install, and then tell me what error you're getting?

hmm, that branch has actually been merged and released. I just tested this again with napari-plugin-engine-0.1.7 and it worked. can you reinstall npe, making sure you're not on a local install, and then tell me what error you're getting?

So turns out I get the same I get the same error with napari --info and I get that with npe 0.1.7 on master too, so looks like something is probably wrong in my environment .... (I only updated from npe 0.1.5 for testing this PR)
Here is error message
ERROR:root:Unhandled exception:
Traceback (most recent call last):
  File "/Users/nsofroniew/GitHub/napari/napari/_qt/qt_main_window.py", line 311, in _show_pip_install_dialog
    self.plugin_dialog = QtPluginDialog(self._qt_window)
  File "/Users/nsofroniew/GitHub/napari/napari/_qt/dialogs/qt_plugin_dialog.py", line 307, in __init__
    self.setup_ui()
  File "/Users/nsofroniew/GitHub/napari/napari/_qt/dialogs/qt_plugin_dialog.py", line 377, in setup_ui
    self.plugin_sorter = QtPluginSorter(parent=self.h_splitter)
  File "/Users/nsofroniew/GitHub/napari/napari/_qt/widgets/qt_plugin_sorter.py", line 274, in __init__
    for name, hook_caller in plugin_manager.hooks.items():
  File "/Users/nsofroniew/opt/anaconda3/lib/python3.7/site-packages/napari_plugin_engine/manager.py", line 876, in __getattribute__
    self._manager.discover()
  File "/Users/nsofroniew/opt/anaconda3/lib/python3.7/site-packages/napari_plugin_engine/manager.py", line 237, in discover
    path, entry_point, prefix
  File "/Users/nsofroniew/opt/anaconda3/lib/python3.7/site-packages/napari_plugin_engine/manager.py", line 191, in iter_available
    include_uninstalled=bool(self.discover_prefix),
  File "/Users/nsofroniew/opt/anaconda3/lib/python3.7/site-packages/napari_plugin_engine/manager.py", line 1085, in iter_available_plugins
    if name == prefix or (not name.startswith(prefix)):
AttributeError: 'NoneType' object has no attribute 'startswith'

Probably something simple, I'll trying looking more into too, sorry!!!

oh, actually, that looks like you have a package in your environment somewhere for which importlib_metadata can't even determine the name.   An unusual situation, perhaps, but a bug in npe nonetheless.  Will fix.

Not sure if it is related to this PR, but when I open the dialog I get the following messages - sounds like it could be related to our error icon PR?
WARNING: Cannot open file ':/themes/dark/warning.svg', because: No such file or directory
WARNING:vispy:Cannot open file ':/themes/dark/warning.svg', because: No such file or directory
WARNING: Cannot open file ':/themes/dark/warning.svg', because: No such file or directory
WARNING:vispy:Cannot open file ':/themes/dark/warning.svg', because: No such file or directory
WARNING: Cannot open file ':/themes/dark/info.svg', because: No such file or directory
WARNING:vispy:Cannot open file ':/themes/dark/info.svg', because: No such file or directory
WARNING: Cannot open file ':/themes/dark/info.svg', because: No such file or directory
WARNING:vispy:Cannot open file ':/themes/dark/info.svg', because: No such file or directory

run rm napari/resources/_qt_resources* to regenerate your resources

Functionality works great, and design looks good. Only comment there is we might want to hide the checkbox on the left until it does something - I know that's waiting on #1183 but for now might as well hide it, then a follow up PR can show it and enable the turning on and off.
I will go through code now, but I think we should be able to merge this soon, which will be a great addition!!
Can you also comment on the one failing test
E       AttributeError: module 'napari.plugins.pypi' has no attribute 'VERSION_CACHE'

I'm agnostic on the presence of the open-ended line edit.  I can see it being useful for the bundle, and potentially problematic (particularly the uninstall part).  If we do keep it though, I think the two dialogs are too similar to have them both in the menu.  Probably rather just add it to the new dialog

I'm agnostic on the presence of the open-ended line edit. I can see it being useful for the bundle, and potentially problematic (particularly the uninstall part). If we do keep it though, I think the two dialogs are too similar to have them both in the menu. Probably rather just add it to the new dialog

Fair enough, agreed two dialogs is probably too confusing. I have a moderate preference towards keeping the lineedit, particularly now at the beginning when we don't know if the discovery mechanism will find everything etc, i could just see it coming in handy. We could add it in between the status and available plugins in this dialog. If it proves to be more trouble than it is worth or not needed then we could remove in the future. Happy to defer to @jni for a third opinion though.

I'd love to get this PR merged as it adds a lot of really nice functionality. I've updated this PR to fix merge conflicts and a test which looks outdated and reviewed the code. It all looks good to me, and the functionality works as expected, but I'm running into one problem which is a seg fault after opening the new dialog only when launching with pythonw on a mac (which for me happens when I just launch napari from the command line)
/Users/nsofroniew/opt/anaconda3/bin/pythonw: line 3: 70931 Segmentation fault: 11  /Users/nsofroniew/opt/anaconda3/python.app/Contents/MacOS/python "$@"

I've confirmed that the problem occurs with the running of
        # fetch available plugins
        self.worker = create_worker(iter_napari_plugin_info)

        def _handle_yield(project_info):
            if project_info.name in already_installed:
                self.installed_list.tag_outdated(project_info)
            else:
                self.available_list.addItem(project_info)

        self.worker.yielded.connect(_handle_yield)
        self.worker.finished.connect(self.working_indicator.hide)
        self.worker.start()

But I've been unable to make further progress. Any ideas or help here would be much appreciated. If we get this figured out then I think we can merge!

hi @tlambert03, thanks for pushing the update! This will now close #1792 I think.
I'm running into an error now though
ERROR:root:Unhandled exception:
Traceback (most recent call last):
  File "/Users/nsofroniew/GitHub/napari/napari/_qt/qt_main_window.py", line 376, in _show_plugin_install_dialog
    self.plugin_dialog = QtPluginDialog(self._qt_window)
  File "/Users/nsofroniew/GitHub/napari/napari/_qt/dialogs/qt_plugin_dialog.py", line 290, in __init__
    self.setup_ui()
  File "/Users/nsofroniew/GitHub/napari/napari/_qt/dialogs/qt_plugin_dialog.py", line 360, in setup_ui
    self.plugin_sorter = QtPluginSorter(parent=self.h_splitter)
  File "/Users/nsofroniew/GitHub/napari/napari/_qt/widgets/qt_plugin_sorter.py", line 279, in __init__
    if not hook_caller.spec.opts.get('firstresult', False):
AttributeError: 'NoneType' object has no attribute 'opts'

which can be fixed by making it  if hook_caller.spec and not hook_caller.spec.opts.get('firstresult', False):, but I'm not sure if that will now lead to the desired behavior. It could be because I have a plugin that has a hook impl which isn't defined in this branch of napari (I'm using it in the very WIP #1800), but I'm not sure.
I'm also still running into the pythonw segfault described above, any tips or advice on how to fix that one would be deeply appreciated!

yeah, i'm still working on it (there are some other remaining issues as well).  will look at the pythonw issue.
your issue above definitely seems unrelated to this and likely related to whatever you're doing with the hookspec (specifically, you might get that error if you're registering a hook implementation that doesn't have a hookspec?)

yeah, i'm still working on it (there are some other remaining issues as well). will look at the pythonw issue.

Thanks!!

your issue above definitely seems unrelated to this and likely related to whatever you're doing with the hookspec (specifically, you might get that error if you're registering a hook implementation that doesn't have a hookspec?)

Yes, I think that's correct, its because i was making a plugin with a new hook implementation to test out #1800, but then when I switched back to this branch the hook spec wasn't there any more, but the plugin which is in another repo was still registered. I know this is a bit of a weird case, and am not sure if we want to raise a dedicated error somewhere else, but we might want to let that pass here somehow. I don't see this as being super common though

I know this is a bit of a weird case, and am not sure if we want to raise a dedicated error somewhere else, but we might want to let that pass here somehow. I don't see this as being super common though

yeah, it's technically allowed within the plugin framework (to have an implementation without a spec) ... so it's a bug in the plugin_sorter and shouldn't raise an error.

cleaned things up, this is ready enough (can polish it up later).  @sofroniewn, I haven't been able to reproduce the segfault on two different computers/environments.  I confirmed that it was indeed using the pythonw executable ... but can't get it to happen.  might need some more info on that.  also, have you tried a clean environment?

cleaned things up, this is ready enough (can polish it up later). @sofroniewn, I haven't been able to reproduce the segfault on two different computers/environments. I confirmed that it was indeed using the pythonw executable ... but can't get it to happen. might need some more info on that. also, have you tried a clean environment?

pythonw is working for me in a clean env, with both pyside2 and pyqt5. I think it's fine to merge and press on, sorry for not checking that sooner!
Ok I think this is good to go, I love the little loading gif, very nice!!

I'll wait till tomorrow morning my time to merge, but let me know if you want more time to do anything else on it etc

but let me know if you want more time to do anything else on it etc

nope, good by me.

oh, one problem I found (for later PR to follow up on): there might be an issue with the subprocess environment when installing plugins that need to build wheels... such as lazy-openslide. (don't know why yet... but wanted to note that)

there might be an issue with the subprocess environment when installing plugins that need to build wheels... such as lazy-openslide. (don't know why yet... but wanted to note that)

Ah yes, i was unable to install napari-lazy-openslide - good to note, can be for a followup


Move jax.ffi submodule to jax.extend.ffi

After a conversation with @froystig, I realized that the new ffi submodule really should have been added under extend, since this is an interface that we're currently iterating on.
cc @andportnoy who will be interested

I can't figure out why this change seems to cause a new failure when testing with a fresh build of jaxlib. I can confirm that the test passes locally with a source install of jaxlib, so I'm sure there's something (probably about the BUILD configuration) that I've missed.
Pinging @hawkinsp and @superbobry in case they have suggestions, but this is low priority.


Remove offset for 3d model

Thanks for this board! This fixes the offset of the 3d model for me.

Thanks for fixing that!


Feature Request: Add fields to the GANTT Display screen

Currently only Tags are enabled but it will be good if we can get Fileds ( which are already part of Query output) to get displayed in the Overview chart.

There is a new version v1.2.0 out there, which enables to display additional fields in the timeline (Area Path, Iteration Path, Parent, Team Project).


Pass kwargs to along to matplotlib functions in fatiando.vis.mpl

While using the mpl module I've noticed that I can't put **kwargs as inputs of some functions. For example, if I want to put a transparency in a contourf with pyplot I just add "alpha=alpha_value" and then I got it. But with mpl I cannot do it, so it forces me to go back to pyplot.
I've started thinking to change it, but when I realised that it is a very easy job I wondered: Is there any reason because it wasn't applied in the first time? If there's none, I would be happy to apply those modifications, initially to the functions:
- contour
- contourf
- pcolor
- points
- paths
- square
- polygon
Greetings!

@santis19 the main reason for not passing along kwargs is that I didn't think of it at the time :-1:
I completely agree with you that this should be norm for all functions that we have overwritten  in vis.mpl. It wouldn't be much trouble I guess to just pass the extra arguments along. You're welcome to start this if you'd like. I'd be very thankful :+1:
This raises a bigger issue that I've had on my mind about the vis package. There are several things that are bugging me about it. I'll open a new issue to discuss this but it is more longterm plans and doesn't impact this issue.


BUG - Missing rows in Payment Intent Table

Are you a current Fivetran customer?
Yes, Ephemeral Tattoos
Describe the bug
Found 5k rows in my raw Payment Intents table that are missing from the staging Payment Intents table
Steps to reproduce
 select count(*) from raw_data.stripe.payment_intent raw
 left join analytics_data.dbt_meera.stg_stripe__payment_intent stg on raw.id = stg.payment_intent_id
 where stg.payment_intent_id is null

Expected behavior
I'd expect no missing rows across my raw data and staging data
Project variables configuration

name: ephemeral

profile: ephemeral

version: '1.0'

require-dbt-version: ">=0.14.0"

source-paths: ["models"]
analysis-paths: ["analysis"]
test-paths: ["tests"]
data-paths: ["data"]
macro-paths: ["macros"]

target-path: "target"
clean-targets:
    - "target"
    - "dbt_modules"

quoting:
  database: false
  identifier: false
  schema: false

models:
  transient: true
  stripe_source:
    +schema:   # Left blank so that will write to schema "dbt"

on-run-end:
  - "{{ grant_select_on_schemas(schemas, 'reporter') }}"
  - "{{ grant_select_on_schemas(schemas, 'transformer') }}"

config-version: 2

vars:
    # Defined variables for Fivetran Stripe package
    stripe_schema: stripe
    stripe_database: raw_data
    using_invoices:        False  #Disabled since we are not using the invoice and invoice_line_item tables
    using_subscriptions:   False  #Disabled since we are not using the subscription and plan tables.

Package Version

packages:
  - package: fishtown-analytics/dbt_utils
    version: 0.6.5

  - package: fivetran/stripe_source
    version: 0.3.1

Warehouse

[ ] BigQuery
[ ] Redshift
[X] Snowflake
[ ] Postgres
[ ] Databricks
[ ] Other (provide details below)

Additional context

Screenshots

Please indicate the level of urgency
Medium-High, on a tight timeline and finding tons of missing rows
Are you interested in contributing to this package?

[ ] Yes, I can do this and open a PR for your review.
[ ] Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this fixed.
[X] No, I'd prefer if someone else fixed this.  I don't have the time and/or don't know what the root cause of the problem is.

Hadn't re-run my table...whoops.


RocketMQ is producing non-standard-compliant JSON

RocketMQ uses fastjson to serialize object to JSON text.
Unfortunately, with the default configuration, the FastJson library does not respect the JSON standard as expected, which imposes an enormous obstacle for other languages to talk to RocketMQ.
For example,
        Map<Integer, String> map = new HashMap<Integer, String>();
        map.put(1, "abc");
        map.put(2, "def");
        System.out.println(RemotingSerializable.toJson(map, false));

generates
{1:"abc",2:"def"}

which does not comply with the JSON standard and may be validated in JSON Lint.
non-standard-compliant JSON can not be parsed by many JSON libraries, say JsonCpp, RapidJson, etc.  It's mandatory to fix this.

Same issue in OffsetSerializeWrapper,
private ConcurrentMap<MessageQueue, AtomicLong> offsetTable = new ConcurrentHashMap<MessageQueue, AtomicLong>();

the key of offsetTable is MessageQueue object, but only string is accepted as object key in JSON standard.

https://github.com/alibaba/fastjson/issues/2078


Add PassiveDNS

What: https://github.com/gamelinux/passivedns
Requested By: @philhagen

Trusty: https://launchpad.net/~sift/+archive/ubuntu/dev/+build/10316978
Xenial: https://launchpad.net/~sift/+archive/ubuntu/dev/+build/10316853

Both have been published to trusty and xenial series in the sift/dev ppa,


Add an admin panel

So that people can view data - consider the following options:
https://www.forestadmin.com/
https://dashibase.com/
https://budibase.com/

@gglucass FYI automation is available

https://glo.budibase.app/app/users#/home
<EMAIL_ADDRESS>is invited as a basic user, which has read-only access to the User table.
@gglucass

Looks like this:


setView and setZoom firing simulateously freezes the map in Safari

Please check the link,
http://stepladderuk.com/dev/chiltern/exp/1/location-map.html
Hover over the logo in the top left. This will run this function;
$('.home').mouseenter(function(){
map.setView( [51.522772, -0.153286] );
map.setZoom( 17 );
});
When the animation is complete, the map 'fixes' for a few seconds and you can't move the map by dragging the mouse.
I think there's an issue with running setView and setZoom simultaneously, If I drop either one of the methods it works ok.
This happens in Safari for mac, ipad and iphone. Works fine in all other browsers.
I disabled all other scripts on the page.
version: leaflet-0.7.3 stable
tested on a mac Safari 7.1.2
Thanks in advance

probably related: https://github.com/Leaflet/Leaflet/issues/2021

Thanks Javi,
I fixed it with this;
$('.home').mouseenter(function(){
setTimeout(function() {
map.setZoom( 17 );
}, 0);
map.setView( [51.522772, -0.153286] );
});

Ok, that wasn't the fix.
I just noticed the problem only happens when the new co-ordinates declared by setView is within the browser view.
In this example, if you move the view outside the original viewport and then fire the function, it works.
You can see how this happens in the link;
http://stepladderuk.com/dev/chiltern/exp/1/location-map.html

Yes, because it tries to animate the position twice and stucks — it's a known issue. Did you try the master version? It should be fixed there as far as I remember.

@stepladder setView also takes a zoom level so you can do
$('.home').mouseenter(function(){
  map.setView( [51.522772, -0.153286], 17);
});

Thanks Mourner, I tried the master version with setTimeout I posted above and it works ok now.

Closing as duplicate of #3249


Non Nullable doesn't work with 2.10 beta version

I installed with homebrew install dart-beta and setting the sdk version constraint to >=2.10.0-110.3.beta <3.0.0 doesn't work.
I then installed the --dev version, and set the version constraint to sdk: ">=2.10.0-0.0.dev and it worked fine.

@enyo To clarify: Do you mean --enable-experiment=non-nullable does not work for you? Or do you expect the experiment to be enabled by-default in 2.10 beta?
/cc @mit-mit Can you clarify from which point on the experiment is available and from which point on it will be on by-default?

No I wasn't talking about --enable-experiment=non-nullable. I was expecting it to work by setting the constraint to >=2.10 and the appropriate settings in analysis_options.yaml.
With the dev version it did work.

I believe you need to change your SDK constraint slightly to sdk: ">=2.10.0-0.0

Closing as I don't think there is an issue here. @enyo if you still see an issue, please reopen and provide concrete repro steps.

Just tested it and it works. Thanks @mit-mit


Bug 2079965: Ignore non-linux pods while applying SCCs

Ignore non-linux pods while applying SCCs

/bugzilla refresh

LGTM. @ravisantoshgudimetla I would only ask you to kindly remove the Bug 2079965:  prefix from the commit message, I don't like external issue tracker references in the commit messages.

LGTM. @ravisantoshgudimetla I would only ask you to kindly remove the Bug 2079965: prefix from the commit message, I don't like external issue tracker references in the commit messages.

Done.

cc @openshift/openshift-team-windows-containers

/lgtm


sparse and other duck array issues

Issues I've run into working with @friedrichknuth at the Pangeo meeting trying to use sparse

.plot() doesn't work. We need to convert to dense before plotting
.values doesn't work. Raises an error about explicitly calling todense()
align doesn't seem to work.

I think it is intentional for automatic coercion to NumPy arrays to fail. Making plot() and .values work should probably require explicitly writing datarray.to_dense() first.

The better way to phrase this is:

Should we formalize the convention that .values always return a numpy array i.e. it will call todense() implicitly and that .data will always return the underlying container: sparse/dask etc.?
Should we call todense() automatically in plot() the same way we do compute()?

Should we formalize the convention that .values always return a numpy array i.e. it will call todense() implicitly and that .data will always return the underlying container: sparse/dask etc.?

I think .values should either return a NumPy array or raise an exception, based upon whether or not the underlying duck array supports coercion with np.array().
We should have a separate API (maybe .to_dense() or .to_numpy_data()?) for explicitly converting into NumPy arrays. This should not be called automatically inside xarray.
Basically, we should leave the  decision about whether automatic coercion is safe up to the authors of duck array libraries.

:+1: I think to_numpy() would be good.

The main downside of  to_numpy() is that it isn't obvious whether the result should be a DataArray containing a NumPy array or a raw NumPy array. In particular, pandas uses to_numpy() for converting into a raw NumPy, which is the opposite of our intent here.

I do think the pandas approach is pretty good here - don't break anything relying on .values - instead have .array to get the backing array (in whatever form it might be), and to_numpy() coercing to a raw numpy array
ref https://pandas.pydata.org/pandas-docs/stable/whatsnew/v0.24.0.html#accessing-the-values-in-a-series-or-index

Xarray's .data is basically the equivalent of pandas's .array.
On Fri, Aug 23, 2019 at 2:44 PM Maximilian Roos<EMAIL_ADDRESS>wrote:

I do think the pandas approach is pretty good here - don't break anything
relying on .values - instead have .array to get the backing array (in
whatever form it might be), and to_numpy() coercing to a raw numpy array
ref
https://pandas.pydata.org/pandas-docs/stable/whatsnew/v0.24.0.html#accessing-the-values-in-a-series-or-index
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/pydata/xarray/issues/3245?email_source=notifications&email_token=AAJJFVUUGARF55CXIKAQHUDQGBD4NA5CNFSM4IOZW5JKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5BJCVI#issuecomment-524456277,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAJJFVWSWKSJFWNUUGRO3ODQGBD4NANCNFSM4IOZW5JA
.

As far as I can tell, the proposal here will require either
s = pd.Series(...)
xr.DataArray.from_series(s).to_series()

or:
xr.DataArray.from_series(s, sparse=True).to_dense().to_series()

For any code that can't guarantee sparse/non-sparse input, the first will fail sometimes, so it will always be necessary to write the latter everywhere, which IMO is unnecessarily verbose.

Do we arrive at the consensus here for API to change the sparse or numpy array?
xref #3542
To make it sparse array, to_sparse() would be better? How about or as_sparse()?

to_sparse() is probably consistent to todense() method
as_sparse() sounds similar to sparse's function, e.g. sparse.as_coo

To change the backend back from sparse array, to_dense() would be better?
FYI, sparse uses todense().
I personally like as_sparse or as_numpy (or as_dense?), which sounds similar to as_type, which gives xarray object not dtype itself.

I weakly prefer following the upstream API: as_sparse and todense.
though to_sparse,to_dense and to_numpy_data would be more consistent?
Also isn't the function astype not as_type?

I like as_sparse and as_dense, because it makes clear that the objects are still xarray objects. I agree with @fujiisoup that to_sparse``todense could be confused to return sparse arrays directly.

it makes clear that the objects are still xarray objects

This is a good point. I'm in favour of as_sparse, as_dense.

Also isn't the function astype not as_type?

Yes, but I think that's mostly works because it's so short. In general the rule is to use underscores when it improves clarity. assparse could read in at least two different ways, and one of which is not really appropriate.

assparse could read in at least two different ways, and one of which is not really appropriate.

brilliant

What is the best way to save sparse array into a disc?
One naive way would be to use stack -> reset_index, but it flattened coordinates and if there is another variable that depends on these coordinates, they will be also flattened and may consume a lot of space.

@dcherian pointed me over here after I raised #4234.

I like as_sparse and as_dense, because it makes clear that the objects are still xarray objects. I agree with @fujiisoup that to_sparse``todense could be confused to return sparse arrays directly.

I very much agree with this reasoning. to_ sounds like it is converting the whole thing to another type. as_ sounds like it is remaining the same but something about it (the underlying array type) is changing.

Basically, we should leave the decision about whether automatic coercion is safe up to the authors of duck array libraries.

I also agree with this reasoning. In cupy many things will break because np.asarray(cupy_array) will raise. So something like cupy_da.plot() will raise. But the exception will be ValueError: object __array__ method not producing an array which isn't particularly user friendly in this case.
For cupy I do think that automatic coercion to numpy for .values and plot() is a reasonable thing to do. But I understand that may not be true for all duck type arrays.
One suggestion is to add an accessor to make things explicit. So it would become cupy_da.cupy.plot(). Where this plot function coerces to numpy and then calls the upstream plot method. However this feels less satisfying.

I am unsure about automatic coercion for .values. I think we've trained users to expect numpy arrays from .values so cupy raising an error is a little unfriendly. OTOH it'd be nice to have something that is equivalent to np.asarray(da.data) but maybe we can expect users who want that to just type it out.
But I think we should automatically coerce for .plot using da.as_numpy().data or something similar. We call .compute() on dask arrays anyway so it's consistent with that behaviour.

Given the current guidance in #4212 is to aim for cupy to mostly be handled via accessors I wonder if xarray could be extended a little further to allow as_ methods to be registered in the same way as accessors.
Currently with the accessor model a user would need to do something like ds.cupy.as_cupy and ds.cupy.as_numpy. When casting a DataSet or DataArray to use cupy under the hood I expect folks would look for a top level as_cupy method. And once it has been cast it would need a cupy specific as_numpy method to be able to cast back.
Enabling accessors to register as_ methods at the top level would really help neaten this up.

we currently discourage this way of using accessors (especially using too many of these, see https://github.com/pydata/xarray/issues/1080#issuecomment-258582609 and https://github.com/pydata/xarray/issues/1080#issuecomment-258680571), but you can use register_*_accessor to register functions:
@xr.register_dataarray_accessor("as_cupy")
def _(da):
    def as_cupy(self, ...):
        # ...
        return da

    return as_cupy

Thanks @keewis. I did consider that but I assumed that kind of thing wouldn't be encouraged. Happy to go down that path, but I wonder if life would be easier if there were an extension API for this purpose.
I was thinking of something along these lines which would help keep things in check.
@xr.register_dataarray_as_method("cupy")
class CupyAsMethod:
    def to_cupy(self):
        """Convert all data arrays to cupy."""

        for var in self.data_vars:
            self.data_vars[var].data = cp.asarray(self.data_vars[var].data)

        return self

    def to_numpy(self):
        """Convert all data arrays to numpy."""

        for var in self.data_vars:
            self.data_vars[var].data = self.data_vars[var].data.get()

        return self

There would need to be some logic around dispatching as_numpy to CupyAsMethod.as_numpy, but only if the underlying data is cupy.
Perhaps some is_numpy, is_cupy boolean attributes would be useful?
I could go as you suggest and use the current accessor tooling to test this functionality out. Then upstream it into something more maintainable if it works out ok?

Sounds good. It might be worth discussing the general strategy for the support of this kind of duck array (sparse, cupy and others) in the dev call on Wednesday.

It seems like general consensus was (please correct me if any of this is wrong):

.values will return np.asarray(obj.data) and respect the underlying library's choices: this will raise an error with cupy and that's OK.
.plot() is a special case and should always coerce to numpy before passing data to matplotlib. plot currently uses .values so we should add a new function   that always returns xarray objects with numpy arrays (with cupy as special case) and use that in plot.
for now we are OK with special as_sparse, as_cupy, as_pint methods in xarray as long as all that logic is contained in one place: as_duck_array.py?

Thanks for writing that up @dcherian. Sounds good to me!

3. for now we are OK with special as_sparse, as_cupy, as_pint methods in xarray as long as all that logic is contained in one place: as_duck_array.py?

+1 for as_numpy(), too
One question about as_numpy: should it convert pint arrays into NumPy by stripping units? Or should it convert the arrays underlying pint and keep the units? I guess the first would probably make more sense for DataArray.as_numpy(). The second behavior could be achieved with DataArray.pint.as_numpy().

One question about as_numpy: should it convert pint arrays into NumPy by stripping units? Or should it convert the arrays underlying pint and keep the units? I guess the first would probably make more sense for DataArray.as_numpy(). The second behavior could be achieved with DataArray.pint.as_numpy().

This brings up a point I was wondering about as well: how should these as_*() methods relate to multiply-nested arrays? (Not sure if best to discuss that here or new issue.)

IMO a good first pass for as_numpy would be to use np.asarray with a special case for cupy where it would use .get.

Agreed! Sparse also needs a special case.
On Wed, Jul 22, 2020 at 6:35 PM Deepak Cherian<EMAIL_ADDRESS>wrote:

IMO a good first pass for as_numpy would be to use np.asarray with a
special case for cupy where it would use .get.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/pydata/xarray/issues/3245#issuecomment-662778974, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJJFVW33SB5YZJWAMIFMR3R46HVVANCNFSM4IOZW5JA
.

Is there any motion on this? This come up in relation to akward array today (see https://github.com/scikit-hep/awkward-1.0/issues/749 ).
If xarray goes with the to_numpy spelling as well then all of {pandas, xarray, akwardarray} have the same spelling which is nice from the Matplotlib side of things.
I think the as_ vs to_ distinction the "to_" makes more sense as it gets you out of xarray land and drops you back to raw numpy land.

My impression was that the as_* methods would return xarray objects. So we could have DataArray methods
def as_numpy(self):
    # needs cupy special handling
    data = self.data
	if isinstance(data, cupy_array_types):
        raise NotImplementedError
	else:
    	return self.copy(data=np.array(data))

def to_numpy(self):
	"""Coerces to and returns a numpy.ndarray"""
    return self.as_numpy().data

I agree, this would be useful. Ideally, that would be implemented by deferring to the data, for example by calling the data's to_numpy method (so duck array libraries or duck array integration libraries can customize that), falling back to np.asarray(data).

I'm trying to implement this (I wanted pint-aware plotting to work), but not 100% sure if I'm doing it right.
Do we want these as_*, to_* methods to live in Variable or DataArray?
What about multiply-wrapped arrays? At the moment I have essentially
def as_numpy(self) -> T_DataArray:
    """
    Coerces wrapped data into a numpy array, and returns it wrapped in
    a DataArray.
    """
    data = self.data
    if isinstance(data, dask_array_type):
        return self.load()
    elif isinstance(data, cupy_array_type):
        return self.copy(data=data.get())
    elif isinstance(data, pint_array_type):
        return self.copy(data=data.magnitude)
    elif isinstance(data, sparse_array_type):
        return self.copy(data=data.to_dense())
    else:
        return self.copy(data=np.array(data))

but do I actually want multiple passes, like
    data = self.data
    if isinstance(data, dask_array_type):
        data = self.load().data
    if isinstance(data, cupy_array_type):
        data = data.get()
    if isinstance(data, pint_array_type):
        data = data.magnitude
    if isinstance(data, sparse_array_type):
        data = data.to_dense()

    return self.copy(data=np.array(data))

@jacobtomlinson currently .values will extract the values of a cupy array by special-casing it, but based on the above discussion do we actually want to break that in favour of getting them via .to_numpy() instead?

Also I can add a .to_pint() method, but I don't really understand what the purpose would be? To be any use to users it would need to accept a units argument, but then it's basically just a less powerful version of pint_xarray's .quantify() accessor method. Can you see any reason to add it @keewis ?

right, I would leave these methods to xarray extension libraries like cupy-xarray or pint-xarray. I assume you meant .as_pint because adding .to_pint does not make much sense: following the naming convention from above, .to_* should return the duck array, which would make it an alias of either .data or the duck array's constructor called on .data.


[BUG] FsWorkbook.fromXlsxFile throws IOException under unknown circumstances

Describe the bug
Some XLSX files let FsWorkbook.fromXlsxFile throw an IOException. The reason behind this (the properties of such files) are currently not known.
To Reproduce
Steps to reproduce the behavior:

Download this file:
errorInducing.xlsx
Use FsWorkbook.fromXlsxFile <pathToThisFile> in a script environment
See error:

Expected behavior
At least a more precise error message.
Cases like these suggest that a readonly version of the function (see #25) is urgently needed.

#37
#40


Features of configuration loader

StyleLint

plugins path look up. returns path string.
extends load and merge
ignoreFiles support

ESLint

plugins / extends path look up or load, returns path string or object.
ignore support

Babel

presets / plugins path look up or load, returns path string or object.
extends load and merge
ignore support

stylefmt

same with StyleLint

I found 4 suitable to use cosmiconfig to replace its own loader project.
If they are using cosmiconfig to load the configuration file, you can solve the common existence of these projects bug. such as; con't work in editor plugin
@ESLint
@Babel

I think that each of the items you've listed are about resolving paths/modules. If cosmiconfig were able to serve each of these projects, I'm thinking it's role would end after resolving those items to absolute paths — then the consumer would decide what to do with those paths.
So the question here is whether the format of these configuration properties and the the means of resolution can be common enough between the projects to justify a shared tool.
I'll detail what stylelint (and stylefmt) does for comparison (cc @jeddy3 if I've made mistakes):

plugins is an array of string locators or a single string locator. Each locator is either

The name of a module in local node_modules* (have not built in the ability to load global node_modules`, though it's been asked for.
An absolute path
A relative path, relative to the referencing configuration file — or, if that fails, relative to process.cwd()

extends is the same as plugins.
ignoreFiles is an array of globs or a single glob. If the globs are relative, we attempt to resolve them relative to the referencing configuration file, and if that fails we try to resolve them relative to process.cwd().

Stylelint has the special situation that you configure a loaded plugin via "rules". Other tools might not have this, so they may need to be able to accept configuration objects alongside lookups in the plugins array.

I think that there are anthor issue.
Babel and ESLint synchronously load config.
If these project use cosmiconfig, there are breaking change.

I don't think this in issue for cosmiconfig. If you'd like to open issues in other projects discussing this possibility, I think that would be more productive..


Footer refactoring

Test URLs:

Before: https://main--moleculardevices--hlxsites.hlx.live/
After: https://footer-refactoring--moleculardevices--hlxsites.hlx.live/

The more icons are not correctly positioned on the right side for

Danaher Life Sciences
Careers

The more icons are not correctly positioned on the right side for..

@lydiapuric I did it on purpose because I thought it would be cleaner, but since you pointed it out as a relevant difference I take it as it should be the same. hence moved them up. Only in case they are external link icons, though, because i think that this is a typical positioning for that particular case. wdyt?
https://github.com/hlxsites/moleculardevices/pull/122/commits/e3002643666d54db5737fdb979d2305fb5414241


Remove Buildkite jobs

Internal/External
Internal
Area
Build/CI
Summary
The Buildkite agents have been decommissioned. Remove all references to Buildkite (scripts and .github)

@sgillespie> can this issue be closed? You've done everything that was required already rigth?

Not until https://github.com/input-output-hk/cardano-node/pull/5498 is completed


Set strict: true in our various tsconfigs

Per Josh Goldberg from his talk on static analysis, we should turn on strict analysis for our typescript so we write more better typescript.

Also extend their stylistic rules and the eslint:recommended rule.


fix focus trap

Initially the focus trap was not working at all on my machine, so this PR was an attempt at a different approach to handling things. After later getting the existing node to work and minor refactor was done instead.


mco puppet count error with no nodes

Status is OK
mco puppet status

No request sent, we did not discover any nodes.

Finished processing 0 / 0 hosts in 0.00 ms

Count shows error
mco puppet count

No request sent, we did not discover any nodes.Total Puppet nodes: 0

The puppet application failed to run, use -v for full error backtrace details: undefined method `fetch' for nil:NilClass

mco puppet count -v
Discovering hosts using the mc method for 2 second(s) .... 0

No request sent, we did not discover any nodes.Total Puppet nodes: 0

The puppet application failed to run: undefined method `fetch' for nil:NilClass

undefined method `fetch' for nil:NilClass (NoMethodError)
	from /opt/puppetlabs/mcollective/plugins/mcollective/application/puppet.rb:348:in `count_command'  <----
	from /opt/puppetlabs/mcollective/plugins/mcollective/application/puppet.rb:370:in `main'
	from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/mcollective/application.rb:293:in `run'
	from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/mcollective/applications.rb:23:in `run'
	from /opt/puppetlabs/bin/mco:33:in `<main>'

cant help with those errors really - issue with upstream agent.
But not discover any hosts is saying you dont have nodes running the puppet agent, if you have one you are sure you have it on, check with mco inventory the.node and then check its log files to see if its moaning about something.
Presumably if you installed with this module it must have it on so pretty weird

Ok I thought it might be todo with your mcollective_agent_puppet module but I couldn't find the git project to log against.
I'm tearing my hair out just trying to get mco inventory working.   If I try to mco inventory  the client from the mcollective server, I can see some nop packets back and forth between the client and server on 4222 but get this back from the request (on the server):
mco inventory dev-saturn.xxx
warn 2016/08/10 11:51:48: client.rb:251:in `rescue in start_receiver' Could not receive all responses. Expected : 1. Received : 0
Did not receive any results from node dev-saturn.xxx

On the client that I'm trying to inventory, I see this corresponding entry in the log:
I, [2016-08-10T11:51:36.908281 #14296]  INFO -- : choria.rb:411:in `valid_certificate?' Verified certificate /CN=infra-jupiter.xxx against CA /CN=Puppet CA: puppet.xxx
W, [2016-08-10T11:51:36.908492 #14296]  WARN -- : choria.rb:301:in `should_cache_certname?' Received certificate name 'infra-jupiter.xxx' does not match (?-mix:\.mcollective$)
W, [2016-08-10T11:51:36.908699 #14296]  WARN -- : runner.rb:206:in `rescue in block in receiver_thread' Failed to handle message: Received an invalid certificate for choria=infra-jupiter.xxx - RuntimeError

W, [2016-08-10T11:51:36.908825 #14296]  WARN -- : runner.rb:207:in `rescue in block in receiver_thread' /opt/puppetlabs/mcollective/plugins/mcollective/security/choria.rb:363:in `block in cache_client_pubcert'
	/opt/puppetlabs/mcollective/plugins/mcollective/security/choria.rb:356:in `synchronize'
	/opt/puppetlabs/mcollective/plugins/mcollective/security/choria.rb:356:in `cache_client_pubcert'
	/opt/puppetlabs/mcollective/plugins/mcollective/security/choria.rb:110:in `decode_request'
	/opt/puppetlabs/mcollective/plugins/mcollective/security/choria.rb:84:in `decodemsg'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/mcollective/message.rb:179:in `decode!'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/mcollective/runner.rb:230:in `receive'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/mcollective/runner.rb:189:in `block in receiver_thread'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/mcollective/runner.rb:187:in `loop'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/mcollective/runner.rb:187:in `receiver_thread'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/mcollective/runner.rb:142:in `block in start_receiver_thread'

This is because I have three puppetservers (infra-saturn, infra-jupiter, infra-neptune) that act as a collective puppet.xxx server, using puppet cert --allow-dns-alt-names sign.  Which is a fairly common setup, I believe.
Is there any way of configuring choria to not get upset by this?  At this point I'm simply just wanting to get the very basic mcollective client/server doing an inventory, nothing more.

Just to clarify, nats replaces activemq entirely, right?

So if I alter mcollective_chria/files/mcollective/security/choria.rb should_cache_certname() to NOT return false for that test, ie. (currently line 302):
        unless certname =~ valid_regex
          Log.warn("Received certificate name '%s' does not match %s" % [certname, valid_regex])
          #return false
        end

then I get further:
I, [2016-08-10T12:39:20.395489 #13050]  INFO -- : choria.rb:411:in `valid_certificate?' Verified certificate /CN=infra-saturn.xxx against CA /CN=Puppet CA: puppet.xxx
W, [2016-08-10T12:39:20.420192 #13050]  WARN -- : choria.rb:301:in `should_cache_certname?' Received certificate name 'infra-saturn.xxx' does not match (?-mix:\.mcollective$)
I, [2016-08-10T12:39:20.420300 #13050]  INFO -- : choria.rb:366:in `block in cache_client_pubcert' Saving verified pubcert for choria=infra-saturnxxx in /etc/puppetlabs/mcollective/choria_security/public_certs/infra-saturn.xxx.pem
I, [2016-08-10T12:39:20.451826 #13050]  INFO -- : choria.rb:147:in `validrequest?' Received valid request 0541a73ae263542190656e1df048335c from choria=infra-saturn.xxx

And on the server that I'm trying to inventory from:
mco inventory dev-saturn.xxx
Failed to retrieve daemon_stats from dev-saturn.xxx: You are not authorized to call this agent or action.

you have to use mco choria request_cert to make a cert

At this point this one and https://github.com/ripienaar/mcollective-choria/issues/37 is the same issue.  So lets stick to this ticket for now.
mco commands are to be run non root, its not safe/advisable to do them as root.  So the choria plugins enforce that more or less.  Do mco choria request_cert as a normal user and it should go better

Oh.  OK. (good tip for the docs!)
[dev@infra-saturn ~]$ mco inventory dev-saturn.xxx
warn 2016/08/10 12:46:56: choria.rb:218:in `block in check_ssl_setup' Cannot find SSL file /home/dev/.puppetlabs/etc/puppet/ssl/certs/dev.mcollective.pem
warn 2016/08/10 12:46:56: choria.rb:218:in `block in check_ssl_setup' Cannot find SSL file /home/dev/.puppetlabs/etc/puppet/ssl/private_keys/dev.mcollective.pem
warn 2016/08/10 12:46:56: choria.rb:218:in `block in check_ssl_setup' Cannot find SSL file /home/dev/.puppetlabs/etc/puppet/ssl/certs/ca.pem

The inventory application failed to run, use -v for full error backtrace details: Client SSL is not correctly setup, please use 'mco choria request_cert'

[dev@infra-saturn ~]$ mco choria request_cert
Requesting certificate for '/home/dev/.puppetlabs/etc/puppet/ssl/certs/dev.mcollective.pem'
Waiting up to 240 seconds for it to be signed

Certificate /home/dev/.puppetlabs/etc/puppet/ssl/certs/dev.mcollective.pem has been stored in /home/dev/.puppetlabs/etc/puppet/ssl
[dev@infra-saturn ~]$ mco inventory dev-saturn.xxx
Failed to retrieve daemon_stats from dev-saturn.xxx: You are not authorized to call this agent or action.

good so now mco ping works yeah?

Oh, yes it does!  Hurrah!
mco puppet status

 * [ ============================================================> ] 2 / 2

    infra-saturn.xxx: You are not authorized to call this agent or action.
   dev-saturn.xxx: You are not authorized to call this agent or action.

ok good, so now you just need to set up poilcies - you may have some left over, but choria sets up a secure by default configuration,
So add a policy via hiera:
mcollective::site_policies:
  - action: "allow"
    callers: "choria=rip.mcollective"
    actions: "*"
    facts: "*"
    classes: "*"

replace rip.mcollective with your dets

I updated the doc to mention the non root user need - there is examples of policies already.
To your question about activemq, yes, NATS will replace ActiveMQ for most setups, I do not think it's really great for long distance separated clusters which some users do with ActiveMQ and there AMQ might still remain a good choice.
But for the general case, I think NATS will be MUCH nicer than AMQ

Ah I did wonder if rip.mcollective needed changing, sorry to be thick but what should be there?  I don't understand what it represents.
My servers are geographically separated within a continent but the latency isn't too bad (<150ms) so I'll give NATS a go.
One problem I've got is that NATS isn't currently routing/replicating the requests across the 3 servers recommended in the deployment guide.  I can mco ping only those clients directly connected to the server that I'm doing the ping from.  In a traditional activemq mcollective cluster you can communicate with clients connected to any of the three servers.

when you did mco request_cert it printed a path to the cert, its from that, it will be echo "${USER}.mcollective"
That's odd about the routing, are you sure they can connect to each other? port 4223 between the brokers

Aha! (again)  Opening port 4223 now routes the requests correct between the brokers, that's fantastic.  I'm getting slightly unreliable results, I'll try increasing the timeouts and see if that helps.
And changing the rip.mcollective to dev.mcollective (user that I'm now using to call mco) has fixed the authorized problem.
I think I'm in business!    I'll carry on with choria and see how I do :)

no worries, appreciate the feedback - this is still early days for these plugins and especially with NATS, so any feed back appreciated.
I am running NATS over a few countries but quite small setup and been happy with it, let me know how you get on.
I'll improve the docs make sure the ports are mentioned

I added some notes about the NATS firewall ports on the wiki.
So I think I'll go ahead and close this one if you're happy? thanks for the help!

Yes thanks again :)

I released the fixes we discussed in the choria module, resolving


feat: :sparkles: Introduce the Environment type.

This factorises the URL/env name management.
Removes EndpointUtil usage. DApp will have to specify the Environment to use every time this is necessary.
That way, DApp can manage the environment selection the way it wants.

Note: Put this PR in major version because of breaking changes