Example of netcdf => xarray => rechunker => zarr?

I saw that there was experimental support for xarray datasets, so I tried this hopefully reproducible notebook that seeks to read from OPeNDAP and write rechunked zarr:
https://gist.github.com/7f3a4367224aeaee227bdfa90620ed96
The code:
max_mem = '4GB'
array_plan = rechunk(ds, chunk_plan, max_mem, target, temp_store=temp)

results in this error:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-11-969522ca9241> in <module>
      1 max_mem = '4GB'
----> 2 array_plan = rechunk(ds, chunk_plan, max_mem, target, temp_store=temp)

~/miniconda3/envs/pangeo/lib/python3.7/site-packages/rechunker/api.py in rechunk(source, target_chunks, max_mem, target_store, target_options, temp_store, temp_options, executor)
    301         target_options=target_options,
    302         temp_store=temp_store,
--> 303         temp_options=temp_options,
    304     )
    305     plan = executor.prepare_plan(copy_spec)

~/miniconda3/envs/pangeo/lib/python3.7/site-packages/rechunker/api.py in _setup_rechunk(source, target_chunks, max_mem, target_store, target_options, temp_store, temp_options)
    381                 temp_store_or_group=temp_group,
    382                 temp_options=options,
--> 383                 name=name,
    384             )
    385             copy_spec.write.array.attrs.update(variable_attrs)  # type: ignore

~/miniconda3/envs/pangeo/lib/python3.7/site-packages/rechunker/api.py in _setup_array_rechunk(source_array, target_chunks, max_mem, target_store_or_group, target_options, temp_store_or_group, temp_options, name)
    464
    465     if isinstance(target_chunks, dict):
--> 466         array_dims = _get_dims_from_zarr_array(source_array)
    467         try:
    468             target_chunks = _shape_dict_to_tuple(array_dims, target_chunks)

~/miniconda3/envs/pangeo/lib/python3.7/site-packages/rechunker/api.py in _get_dims_from_zarr_array(z_array)
    138     # use Xarray convention
    139     # http://xarray.pydata.org/en/stable/internals.html#zarr-encoding-specification
--> 140     return z_array.attrs["_ARRAY_DIMENSIONS"]
    141
    142

AttributeError: 'Array' object has no attribute 'attrs'

Is it expected that rechunker can be used this way with xarray src data?
If not, does someone have an example of using rechunker to write zarr from xarray src data?

@eric-czech , do you by chance have an example notebook using xarray as a source?

Rich, this test might be a good place to start:
https://github.com/pangeo-data/rechunker/blob/9f49f96004c85b2cb54255d42430d70e25686071/tests/test_rechunk.py#L306

I think this is happening because targets chunk values need to be tuples (of ints), not dicts.
So in this test, target_chunks is {'a': (20, 10)}. It fails if it is {'a': {'x': 20, 'y': 10}} (where a is the variable, and x and y are dimensions).
I think this is just a current limitation in xarray support in rechunker - it should be possible to use the dictionary syntax. (And indeed that would probably be preferable since it would avoid dimension ordering issues.)
BTW here is an example (not a notebook) of where Eric uses xarray rechunking: https://github.com/pystatgen/sgkit/blob/master/sgkit/io/bgen/bgen_reader.py#L510. Note that it too uses tuples.

I think this is happening because targets chunk values need to be tuples (of ints), not dicts.

So we have definitely implemented this for Zarr arrays, here:
https://github.com/pangeo-data/rechunker/blob/9f49f96004c85b2cb54255d42430d70e25686071/rechunker/api.py#L465-L466
https://github.com/pangeo-data/rechunker/blob/9f49f96004c85b2cb54255d42430d70e25686071/rechunker/api.py#L137-L140
For xarray datasets, you would just need to translate the dict syntax to a tuple. This could happen right here:
https://github.com/pangeo-data/rechunker/blob/9f49f96004c85b2cb54255d42430d70e25686071/rechunker/api.py#L363-L364

Sorry for the delay @rsignell-usgs but what Tom pointed out are the best examples I have (thank you you @tomwhite!).  I would like to eventually add some though, or contribute to them, as a part of https://github.com/pangeo-data/rechunker/issues/53.

I'm still struggling with this.   I've got a workflow to process National Water Model netcdf files on s3 to Xarray, but I'm getting a error:
ContainsArrayError: path 'time' contains an array

when I run the rechunk command.
I don't have a minimal example, but I do have a reproducible notebook in case it's clear what's going on.
Also @rabernat, you mentioned you don't have an AWS account, but I added you as a user to https://jupyter.qhub.earthmap.org, so you can go to the /shared/users/rsignell/notebooks/NWM folder and run the nwm_dask notebook.

That error usually means that you have not properly cleared the target locations. Can you try explicitly deleting the temp store and target store (e.g. using the CLI) before running your notebook.
I think that rechunker should do this automatically, but for now this would be a useful debugging step.

Okay @rabernat , I'll definitely try that.
(Also, would it be better to have the temp storage locally instead on S3 since the chunks I guess will be tiny?)

(Also, would it be better to have the temp storage locally instead on S3 since the chunks I guess will be tiny?)

I'm not sure. There are a lot of factors. Try it out! Now that async works in zarr the overhead of creating many small chunks should be much lower. However, you need the unreleased master version of zarr to get these benefits.
A problem we don't know how to solve is how to delete the intermediate store efficiently when rechunker is done.

Okay, writing to brand new objects, I now get a different error:
ValueError: A temporary store location must be provided (array=streamflow).

Now that error sounds familiar. I think I got the same thing when I tried.

Aha!  Switching from s3 to local disk for the tmp zarr seems to work.

For now...but will you get the same error when you start writing the target store?
Also, are you using distributed? Do you have a shared filesystem?

Ah, right.  The dask workers don't have access to the local disk.

Did you ever successfully use object storage for the temp zarr file?

Definitely! We are using this regularly in production with raw zarr arrays. Your error is related to groups and / or xarray datasets.

That seems strange, since it seems to be working without changes using a LocalCluster with local storage temp and target zarr files.
I might be tempted to conclude that my Dask workers can't write to the bucket, but I know they can because if I don't use rechunk I can write Zarr to my bucket using the Gateway cluster.

I might be tempted to conclude that my Dask Gateway workers can't write to the bucket

No I don't think that's the problem. It would be great if you were willing to drive into the rechunker code a bit and see if you can get to the bottom of it. The relevant part is here:
https://github.com/pangeo-data/rechunker/blob/17053cc2956ab8d417a666cc8675bcb4df1c5c56/rechunker/api.py#L512-L516
The error is getting raised because temp_store_or_group is None at this point in the code. So somehow, the temp store object is not being passed correctly to the _setup_array_rechunk function. We already have a hint that it depends on the type of store--file path vs. fsspec filesystem.

Okay, I will try.

Besides the need to provide tuples, this functionality is amazing!!! This shaves hours off my preprocessing workflow for CMIP6!
I just rechunked 500GB in 30 minutes on a 40core/192GB node.

Can someone give me a fully working example of an xarray rechunk?

Can someone give me a fully working example of an xarray rechunk?

How's this?  https://gist.github.com/eric-czech/c5628e1f895e7c93b6096ce95862ac78

Excellent, thanks Eric! In #72 I am trying to make this work with dict syntax, i.e.
target_chunks=dict(
        salt={'ocean_time': 2, ...}
    )

Nice, much better that way.


Implement futures and promises with continuations.

Basically these should be an implementation of:
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2015/p0159r0.html

Woo hoo!  This compiles and passes now:
https://github.com/coryan/google-cloud-cpp/blob/9aafe2e0ceb616c7d2f0cb00155294cfcf8d32db/google/cloud/internal/future_generic_then_test.cc#L83
@dopiera @deepankarsharma @houglum @frankyn  in case they wonder where all this template nonsense is headed.

@coryan - This addresses the recent emptiness in my life of not having enough templates :)


sbt netty eviction warning

[warn] Found version conflict(s) in library dependencies; some are suspected to be binary incompatible:
[warn] 	* io.netty:netty-handler:4.1.22.Final is selected over 4.1.13.Final
[warn] 	    +- org.asynchttpclient:async-http-client:2.4.4        (depends on 4.1.22.Final)
[warn] 	    +- com.typesafe.netty:netty-reactive-streams:2.0.0    (depends on 4.1.13.Final)
[warn] Run 'evicted' to see detailed eviction warnings

But isn't that a mismatch pulled in by async-http-client? Here's a searchable tree of depedencies for easier browsing: https://app.updateimpact.com/treeof/com.softwaremill.sttp/async-http-client-backend_2.11/1.2.0-RC2

Got the same issue.

Probably out of date :)


Add support for remark and rehype as external plugins

Summary
When writing blogs and articles using TinaCMS, it would be useful to allow external plugins such as rehype-slug, rehype-prism-plus, remark-gfm, rehype-autolink-headings, etc. for added accessibility for readers.
Expected Results

Code block highlighting

Autolinked headings

and more...
Notes
These markdown processors would provide more flexibility to the developers to alter the output of TinaMarkdown

Where was this completed (https://github.com/tinacms/tinacms/issues/2865#event-7100949409)?


bug fix for epCnt cocurrent race problem, which make epCnt in remote store not right, and there is no need to compare epCnt and len(epl)

when dockerd was restart, the method cleanupLocalEndpoints() would be called, and this method will call method n.getEpCnt().setCnt(uint64(len(epl))), however, setCnt will force update the endpoint_count value in the remote store, and this may cause endpoint_count not right if cocurrently some dockerd on the different host is starting or stopping containers, and the wrong endpoint_count of some globel network will finally make this network cannot be deleted, because before deleting network, dockerd will check whether the value of endpoint_count is {"Count": 0}, however, if this value is wrong because the criteria mentioned above, the network will not be allowed to delete, and we have to manually change the value to be {"Count": 0}, then, the network can be deteted.
this bug has followed docker form 1.9.0 up to now, I hope it can be fixed. thx~
fix https://github.com/moby/moby/issues/19261
Signed-off-by: Xin He<EMAIL_ADDRESS>

The global-scope datastore has been vestigial for years. There are no longer any remote datastores or remote daemons to race.


Make pathname optional in Typescript typings

Pathname is optional in that you can do the following and it will work correctly:
dispatch(push({ query: { something: "value" }));

I've updated the typings to reflect this and to be inline with the typings that were previously available at @types/redux-little-router

@tptee is anyone maintaining this repo any more?
There are a number of useful pull requests sitting open

@tptee @parkerziegler -- Are either of you available to clear out this and the other short PRs like this one?
@boygirl -- If we can, would be wonderful to get a few folks perhaps also pitching in here as we really need the effort and I know Tyler is slammed with two projects at the same time right now. Thanks!

@ryan-roemer I'm happy to take a look at this this afternoon. I can definitely do some redux-little-router work for Sauce but am also pretty slammed with full time client work.

Thank you @hisuwh for the PR! Verified locally and am merging now. I'm going to take a look at additional type related PRs in this repo, get them onto master, and hopefully publish a patch release by beginning of next week with the updated index.d.ts.

Thanks @parkerziegler ! Mind doing the release too? 🙏

Thanks @parkerziegler
Loving redux-little-router - glad you guys are still maintaining it

15.1.2 is published! 🎉 Let us know if you have any issues!


U2-net model for Pytorch Live

Tutorial Select
Prepare Custom Model
Feedback
Hi Pytorch Live community,
I am trying to implement the (U2-net model) instead of AnimeGan model which is provided as an example to test how it works with Pytorch Live. The steps I do:

Conversion of the u2netp model (the small version of the U2-net model) using this manual and the same live.spec.
After the conversion I got this model - https://github.com/lavandaboy/deeplabv3-pytorch-live/blob/master/u2netp_small_live.ptl
I deploy the example with AnimeGan model, and everything works perfectly
I change the link to the model of Animegan in Model.ts file to this (https://github.com/lavandaboy/deeplabv3-pytorch-live/blob/master/u2netp_small_live.ptl)
I deploy everything on my ios device

As the result, nothing happens ("Processing..." is )I receive the following error: Possible Unhandled Promise Rejection (id:0): Object { "message": " ",}
Could you please advise how to implement the U2-net model into Pytorch Live? If you need anything else to provide from me please let me know.
Thanks!

Hi @lavandaboy, I tried loading the model as lite interpreter model in a Google Colab, but get a RuntimeError.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
[<ipython-input-4-86c4d700d487>](https://localhost:8080/#) in <module>()
      1 from torch.jit.mobile import _load_for_lite_interpreter
      2
----> 3 model = _load_for_lite_interpreter('/content/u2netp_small_live.ptl')
      4
      5 print(model)

[/usr/local/lib/python3.7/dist-packages/torch/jit/mobile/__init__.py](https://localhost:8080/#) in _load_for_lite_interpreter(f, map_location)
     46
     47     if isinstance(f, str) or isinstance(f, pathlib.Path):
---> 48         cpp_module = torch._C._load_for_lite_interpreter(f, map_location)
     49     else:
     50         cpp_module = torch._C._load_for_lite_interpreter_from_buffer(f.read(), map_location)

RuntimeError: Format error

Can you provide the steps how you exported the model and saved it for the lite interpreter runtime (e.g., as a Google Colab)? Ideal would be if you could provide an example in the Google Colab how to use the model.

With PyTorch Live and the react-native-pytorch-core release 0.2.0, the live spec (step #1) is no longer needed. The data processing can be done in JavaScript-only.
See above comment about RuntimeError
Are the model inputs and output the same as with the AnimeGANv2 model?

Happy to give it a try if you can provide additional info as mentioned above 🙂

Hey @lavandaboy, I was able to use your exported model and build a simple on-device demo with PyTorch Live. You can find the code in the following repo:
https://github.com/raedle/UNetExample

Can you provide instructions here or as a PR on the repo how you exported the model for PyTorch Mobile lite interpreter? That would be rad!
Thank you!

Hi Roman,
As always, thanks a lot for your help!
The conversion manual is the following:

Clone https://github.com/NathanUA/U-2-Net.git
Open file u2net_test.py
Go to row 57 and keep model_name='u2net' if you want to use the full model or change to model_name='u2netp' if you want to use the small model (less than 5Mb)
Go to row 94 (just below "net.eval()" code)
Add the following code:

scripted_model = torch.jit.script(net)
optimized_model = optimize_for_mobile(scripted_model)
spec = Path("live.spec.json").read_text()
extra_files = {}
extra_files["model/live.spec.json"] = spec
optimized_model._save_for_lite_interpreter("u2net_small_live.ptl", _extra_files=extra_files)

Add the following code to the beginning of the file u2net_test.py:

from pathlib import Path
from torch.utils.mobile_optimizer import optimize_for_mobile

Add live.spec file into the root of the U2net folder with the following code inside:

{
  "pack": {
    "type": "tensor_from_image",
    "image": "image",
    "transforms": [
      {
        "type": "image_to_image",
        "name": "center_crop"
      },
      {
        "type": "image_to_image",
        "name": "scale",
        "width": 224,
        "height": 224
      },
      {
        "type": "image_to_tensor",
        "name": "rgb_norm",
        "mean": [0.485, 0.456, 0.406],
        "std": [0.229, 0.224, 0.225]
      }
    ]
  },
  "unpack": {
    "type": "argmax",
    "dtype": "float",
    "key": "maxIdx",
    "valueKey": "confidence"
  }
}

Upload the pretrained model from U2-net repo into the folder /saved_models/unetp/ if you are using the small version or into /saved_models/unet/ if you are using the full model
Run u2net_test.py
You will see your converted model for PyTorch Live u2net_small_live.ptl in the folder of U2-Net

Hope this manual helps to convert more models to Pytorch Live!
Now I will play with your working example :) Thanks, Roman!


Make use of shouldComponentUpdate

A lot of components are rerendering more times than needed, investigate that

One of the problem that immutable stores could fix. #109

Moving to #121


Query : Get Rewards Status

Purpose
Added support for query GetRewardStatus
Changes
Added support for query GetRewardStatus
Checklist

[x] My code follows the style of this project.
[x] The code compiles without warnings.
[x] I have performed a self-review of the changes.
[x] I have documented my code, in particular the intent of the
hard-to-understand areas.
[x] (If necessary) I have updated the CHANGELOG.

CLA acceptance
By submitting the contribution I accept the terms and conditions of the
Contributor License Agreement v1.0

link: https://developers.concordium.com/CLAs/Contributor-License-Agreement-v1.0.pdf

[x] I accept the above linked CLA.

done

Looks good. One comment about getRewardStatus.

done

Closed via https://github.com/Concordium/concordium-java-sdk/pull/91


Prerelease time

omfg

Genius me forgot to add a toggle for dungeon waypoints

wait yeah i didnt look for that because its something i expected to exist lol


first attempt

Wouter

Not familiar with git, hope the upload was successful..
For the final prediction only the last bit is needed, from  "Train for the test data" on

Thanks very much for taking part Wouter. I was very happy to see someone doing some serious feature engineering — no-one else had done that yet. Thanks for applyig some magic :) Your entry score 0.557, so it's up there in the top half of the table. Nice job! Definitely worth another attempt I think.


No LSP highlighting after calling :colorscheme

nvim --version: v0.5.0-dev+992-gf3a8c930a
nvim-lsp version(commit hash): Can't find this, but cloned latest at 4:31pm GMT 1st Jan 2021.
:checkhealth result: Just missing providers, debug build of nvim, nothing unusual.
Language servers: Both tsserver and hls.
Can you reproduce this behavior on other language server clients: No, coc.nvim was fine.
Operating system/version: Arch Linux latest everything.
Terminal name/version: Alacritty 0.6.0.
$TERM: alacritty

How to reproduce the problem from neovim startup
Add the following nvim config (I believe any plugin manager and language servers will do):
packadd minpac
call minpac#init()
call minpac#add('k-takata/minpac', {'type': 'opt'})

call minpac#add('neovim/nvim-lspconfig')
lua <<EOF
  require'lspconfig'.tsserver.setup{}
  require'lspconfig'.hls.setup{}
EOF

command! PackUpdate call minpac#update()
command! PackClean  call minpac#clean()

Open a file that'll trigger a language server. Make an error, and you'll see an underline underneath it. Now undo the error, the underline will go away. Now run :colorscheme default, and repeat the error.
Actual behaviour
There will be no underline after running :colorscheme.
Expected behaviour
There should still be an underline after running :colorscheme.

This is the repo for language server configuration for neovim's built-in language server client, this seems like an implementation issue in core. FWIW, when switching to a custom colorscheme (one dark) from default, the underline remains intact.

This is the repo for language server configuration for neovim's built-in language server client, this seems like an implementation issue in core. FWIW, when switching to a custom colorscheme (one dark) from default, the underline remains intact.

I would personally recommend adding
vim.cmd([[ autocmd ColorScheme * :lua require('vim.lsp.diagnostic')._define_default_signs_and_highlights() ]])

If this is bothering you. I'm not sure if this should be added as a default to core or nvim-lspconfig though.

I would personally recommend adding
vim.cmd([[ autocmd ColorScheme * :lua require('vim.lsp.diagnostic')._define_default_signs_and_highlights() ]])

If this is bothering you. I'm not sure if this should be added as a default to core or nvim-lspconfig though.

On further investigation, this isn't even an isolated neovim issue. The issue is in vim core.
https://github.com/vim/vim/issues/4405
All we do is call highlight via a vim command, vim is the one erasing highlight groups on colorscheme changes. The best we can do is maybe add a note somewhere.

On further investigation, this isn't even an isolated neovim issue. The issue is in vim core.
https://github.com/vim/vim/issues/4405
All we do is call highlight via a vim command, vim is the one erasing highlight groups on colorscheme changes. The best we can do is maybe add a note somewhere.

Your suggested fix works, thank you. :slightly_smiling_face:
Interesting! I'll keep an eye on this, but feel free to close this at your own leisure.

Your suggested fix works, thank you. :slightly_smiling_face:
Interesting! I'll keep an eye on this, but feel free to close this at your own leisure.

I would personally recommend adding
vim.cmd([[ autocmd ColorScheme * :lua require('vim.lsp.diagnostic')._define_default_signs_and_highlights() ]])

If this is bothering you. I'm not sure if this should be added as a default to core or nvim-lspconfig though.

Hi @mjlbach, looks like the api has changed for nightly as I cannot find the _define_default_signs_and_highlights() function in the diagnostic.lua How can I do this with nightly?

I'm pretty sure this isn't necessary with the diagnostic overhaul on 0.6, these functions were removed. cc @gpanders

Looks like the underlines disappears when I change from nightfox to gruvbox then back to nightfox.

The colorscheme is running :hi clear so the highlight groups are being removed.
Try adding the following to your init.vim:
autocmd ColorScheme * runtime plugin/diagnostic.vim

plugin/diagnostic.vim is the plugin file included in $VIMRUNTIME that defines the highlight groups.

The colorscheme is running :hi clear so the highlight groups are being removed.
Try adding the following to your init.vim:
autocmd ColorScheme * runtime plugin/diagnostic.vim

plugin/diagnostic.vim is the plugin file included in $VIMRUNTIME that defines the highlight groups.

Thank you @gpanders, @mjlbach this worked!

@quantum-booty Note that as of https://github.com/neovim/neovim/pull/15894 that autocommand is no longer necessary and diagnostic highlight groups will always be defined.


[renovate]Allow make force-bump as post update task

Related: OSPRH-8355

This is fairly harmless on its own. So we can land it.

It will be only used by https://github.com/openstack-k8s-operators/openstack-operator/pull/1178 after that lands.


No documentation for keyword autocomplition

There exists documentation for the keywords (displayed in the tooltips) but it's not displayed for the autocomplete. helptext request reports error when it tries to resolve keyword documentation.

I'm neither getting tooltips nor autocomplete, just happened overnight - where do I look for logging info?

Fixed by https://github.com/fsharp/FsAutoComplete/commit/a6fdf54743f6793daba00a3016665fcb88292fdb


[WIP] Split manual and recurring backups

https://github.com/rancher/rancher/issues/18807
https://github.com/rancher/rancher/issues/18793

Addressed review comments, and rebased -  https://github.com/rancher/rancher/pull/19623

Pulled in changes into #19623.


Bar/cursor in front of save message

URL
( was: https://www.openstreetmap.org/changeset/121691997 )
How to reproduce the issue?

(Set German as language)
Make an edit
Save edit
A vertical bar / the cursor shows up in front of the save message

OS: Windows 10
Screenshot(s) or anything else?
Vertical bar/cursor
https://twitter.com/geoObserver_/status/1531238880426528768

Does not happen for me (Chrome, Windows 7)

Which iD Editor versions do you see the issue on?
Released version at openstreetmap.org/edit
Which browsers are you seeing this problem on?
Chrome

I can reproduce this, it's the mouse cursor like you've mentioned in the changeset comment. I don't really see an issue here. Pretty much all text on the site which doesn't happen to be some link or image behaves exactly the same way.


[DOC] Update dapp-frontend.mdx so it includes how to use the StellarWalletsKit to connect the wallets

This change includes a quick guide of how to use the StellarWalletsKit when using Freighter to request the public key and signing a transaction in the dapp-frontend guide: https://developers.stellar.org/docs/build/apps/dapp-frontend
This PR comes to complete this task: https://github.com/stellar/stellar-docs/issues/766

Thank you @earrietadev!

lol I was going through this section right now and was just about to create an issue and say there was some issues with the new version of Freighter and some calls 😅 Awesome I will move to your kit!


[docs] Avatar docs need review/cleanup

Code tab seems like it needs a little design love. Sections are tightly packed with no margin. Please compare with other doc pages and make sure it's not just an update in the CSS to fix them all.
Also review other sections for consistency.
Current tabs: Overview, Style, Guidelines, Code
Previous IA docs update issue #823.
Tag @coreyvickery for final design review.

@markcaron starting to work on this now

Take a look at the docs cleanup checklist
https://github.com/RedHat-UX/red-hat-design-system/wiki/Docs-Cleanup-Q2-2023-Checklist

XD: https://xd.adobe.com/view/6f3078cf-c5d5-4c7b-8e30-509acd11525c-3a63/
Content doc: https://docs.google.com/document/d/1eEnyWKABadmP4RetzXIB7HXX542C9n-Dj_jjQ4uE_XI/edit#
Images: https://www.dropbox.com/sh/77vfksj34wwqcn1/AADuZQdzhWyxJkcTVt27C9h2a?dl=0


feat: adjust ServiceAccounts to the Kubernetetes 1.24 version

Description
Changes proposed in this pull request:

handle new token approach in Service Accounts
add translations
change the useDownloadKubeconfig hook to be more generic
add and adjust tests

Related issue(s)
Closes #1853

/retest

/retest

/retest

/retest


Migrate e2e test for Azure Service Bus to Go

Proposal
Relates to https://github.com/kedacore/keda/issues/2357, https://github.com/kedacore/keda/issues/2737.
Use-Case
No response
Anything else?
No response

This is already done


Fixed the disintegrity of profile card

also added another card with image, github and linked in links for yash soni

your pr is conflicted plz make a new pr, I will accept that .

I have fixed the conflict and created the pr again kindly check.


S3 getObject fails in macOS Mojave

Describe the bug
A have a simple func that downloads a file/object from AWS S3, in macOS Catalina the download works just fine, the problem is with macOS Mojave (10.14.6) where the download will almost always fail with an EOF error(11). Looking into the stacktrace I was able to figure out that the file is actually downloaded as I see the byte count is exactly the same as my file. The weird thing is, on a clean macOS Mojave install, when I run the code for the first time, it will download the file and after that it will always fail no matter what file I try to get.
To Reproduce
Steps to reproduce the behavior:

Create an S3 client in useast1
Use the following code to get an object from a bucket (my bucket was public, but that doesnt seem to affect this behavior)

let getObjectRequest = S3.GetObjectRequest(bucket: "bucket-name", key: "filename")
let req = s3.getObject(getObjectRequest)

req.whenFailure({ error in
   print(error)
})

req.whenSuccess({ response in
    if response.body != nil {
        print("Hey, we got a body here!")
    } else {
        print("Failed to download template document.")
    }
})

Add a Swift Error breakpoint.
Run the code, if the first time succeeds, try to download the file one more time, or try to download a different file.
The code should stop at line 375 in HTTPDecoder.swift where it throws the EOF error.

Expected behavior
The expected behavior is that the getObject request should always succeed like it does in macOS Catalina 10.15.3
Setup (please complete the following information):

OS: macOS Mojave 10.14.6
Version of aws-sdk-swift 4.1.1
Authentication mechanism: hard-coded credentials
Xcode 11.3.1

Additional context
I'm not sure about this bug, looking at the stracktrace is seems to be a Swift NIO issue rather than an AWS-SDK-Swift bug.
Additional context 2
I want to say a big Thank You to the authors of this awesome framework for their amazing work!

HI @andyzaharia thanks for this.
A couple of things you could do that be helpful.

It would be really helpful if could you supply a full callstack for where it crashes?
When you create your S3 client do you supply your own EventLoopGroup. If you don't can you do the following when creating your client. This will force it to use the non Transport Services version of the HTML client and might produce a different result. If you do supply your own EventLoopGroup can you try without.

let eventLoopGroup = MultiThreadedEventLoopGroup(numberOfThreads: System.coreCount)
let s3 = S3(region:.useast1, eventGroupLoopProvider: .shared(eventLoopGroup)

Finally if you have the time could you test this using the async-http-client branch with both the MultiThreadedEventLoopGroup and without.

thanks

Thanks for the response @adam-fowler !

Posting the stacktrace here

* thread #13, queue = 'nio.nioTransportServices.connectionchannel', stop reason = breakpoint 2.1
    frame #0: 0x00007fff635dd300 libswiftCore.dylib`swift_willThrow
  * frame #1: 0x0000000100929f71 S3Demo`BetterHTTPParser.feedInput(bytes=nil, self=0x00007b440001bbc0) at HTTPDecoder.swift:375:17
    frame #2: 0x0000000100932f32 S3Demo`HTTPDecoder.feedEOF(context=0x00007b180000ca20, self=0x00007b200000c980) at HTTPDecoder.swift:603:29
    frame #3: 0x000000010093425d S3Demo`HTTPDecoder.decodeLast(context=0x00007b180000ca20, buffer=NIO.ByteBuffer @ 0x00007e8000620418, seenEOF=true, self=0x00007b200000c980) at HTTPDecoder.swift:630:26
    frame #4: 0x0000000100934b85 S3Demo`protocol witness for ByteToMessageDecoder.decodeLast(context:buffer:seenEOF:) in conformance HTTPDecoder<A, B> at <compiler-generated>:0
    frame #5: 0x000000010078826f S3Demo`closure #1 in ByteToMessageHandler.decodeLoop(decoder=<no summary available>, buffer=NIO.ByteBuffer @ 0x00007e8000620418, decodeMode=NIO.ByteToMessageHandler<τ_0_0>.DecodeMode @ scalar, self=<no summary available>, context=0x00007b180000ca20, allowEmptyBuffer=false) at Codec.swift:565:49
    frame #6: 0x00000001007940cb S3Demo`partial apply for closure #1 in ByteToMessageHandler.decodeLoop(context:decodeMode:) at <compiler-generated>:0
    frame #7: 0x0000000100785d7e S3Demo`ByteToMessageHandler.withNextBuffer(allowEmptyBuffer=true, body=0x0000000100794080 S3Demo`partial apply forwarder for closure #1 (inout A, inout NIO.ByteBuffer) throws -> NIO.DecodingState in NIO.ByteToMessageHandler.(decodeLoop in _F2A740607FEBA425AF6F8C49A24C0AD7)(context: NIO.ChannelHandlerContext, decodeMode: NIO.ByteToMessageHandler<A>.(DecodeMode in _F2A740607FEBA425AF6F8C49A24C0AD7)) throws -> NIO.(B2MDBuffer in _F2A740607FEBA425AF6F8C49A24C0AD7).BufferProcessingResult at <compiler-generated>, self=<no summary available>) at Codec.swift:524:36
    frame #8: 0x0000000100787259 S3Demo`ByteToMessageHandler.decodeLoop(context=0x00007b180000ca20, decodeMode=NIO.ByteToMessageHandler<τ_0_0>.DecodeMode @ scalar, self=<no summary available>) at Codec.swift:558:35
    frame #9: 0x0000000100786a13 S3Demo`ByteToMessageHandler.processLeftovers(context=0x00007b180000ca20, self=<no summary available>) at Codec.swift:535:29
    frame #10: 0x000000010078ac96 S3Demo`ByteToMessageHandler.channelInactive(context=0x00007b180000ca20, self=<no summary available>) at Codec.swift:662:14
    frame #11: 0x000000010078b1cb S3Demo`protocol witness for _ChannelInboundHandler.channelInactive(context:) in conformance ByteToMessageHandler<A> at <compiler-generated>:0
    frame #12: 0x00000001007548a0 S3Demo`ChannelHandlerContext.invokeChannelInactive(self=0x00007b180000ca20) at ChannelPipeline.swift:1329:28
    frame #13: 0x0000000100754947 S3Demo`ChannelHandlerContext.invokeChannelInactive(self=0x00007b180000c7e0) at ChannelPipeline.swift:1331:24
    frame #14: 0x000000010074e1ac S3Demo`ChannelPipeline.fireChannelInactive0(self=0x00007b140001b760) at ChannelPipeline.swift:817:29
    frame #15: 0x000000010074e045 S3Demo`ChannelPipeline.fireChannelInactive(self=0x00007b140001b760) at ChannelPipeline.swift:567:13
    frame #16: 0x0000000100d5cf02 S3Demo`StateManagedChannel.close0(error=eof, mode=all, promise=nil, self=<no summary available>) at StateManagedChannel.swift:212:31
    frame #17: 0x0000000100d2b487 S3Demo`NIOTSConnectionChannel.didReadEOF(self=0x00007b380000ff80) at NIOTSConnectionChannel.swift:785:18
    frame #18: 0x0000000100d27ee0 S3Demo`NIOTSConnectionChannel.dataReceivedHandler(content=8192 bytes, context=0x00007b140004aba0, isComplete=true, error=nil, self=0x00007b380000ff80) at NIOTSConnectionChannel.swift:707:18
    frame #19: 0x0000000100d2cdce S3Demo`partial apply for NIOTSConnectionChannel.dataReceivedHandler(content:context:isComplete:error:) at <compiler-generated>:0
    frame #20: 0x00007fff63876d00 libswiftNetwork.dylib`partial apply forwarder for closure #1 (Swift.Optional<__C.OS_dispatch_data>, Swift.Optional<__C.OS_nw_content_context>, Swift.Bool, Swift.Optional<__C.OS_nw_error>) -> () in Network.NWConnection.receiveMessage(completion: (Swift.Optional<Foundation.Data>, Swift.Optional<Network.NWConnection.ContentContext>, Swift.Bool, Swift.Optional<Network.NWError>) -> ()) -> () + 176
    frame #21: 0x00007fff6387ffb9 libswiftNetwork.dylib`partial apply forwarder for closure #1 (Swift.Optional<__C.OS_dispatch_data>, Swift.Optional<__C.OS_nw_content_context>, Swift.Bool, Swift.Optional<__C.OS_nw_error>) -> () in Network.NWConnection.receive(minimumIncompleteLength: Swift.Int, maximumLength: Swift.Int, completion: (Swift.Optional<Foundation.Data>, Swift.Optional<Network.NWConnection.ContentContext>, Swift.Bool, Swift.Optional<Network.NWError>) -> ()) -> () + 9
    frame #22: 0x00007fff63868248 libswiftNetwork.dylib`reabstraction thunk helper from @escaping @callee_guaranteed (@guaranteed Swift.Optional<__C.OS_dispatch_data>, @guaranteed Swift.Optional<__C.OS_nw_content_context>, @unowned Swift.Bool, @guaranteed Swift.Optional<__C.OS_nw_error>) -> () to @escaping @callee_unowned @convention(block) (@unowned Swift.Optional<__C.OS_dispatch_data>, @unowned Swift.Optional<__C.OS_nw_content_context>, @unowned Swift.Bool, @unowned Swift.Optional<__C.OS_nw_error>) -> () + 88
    frame #23: 0x0000000103234f1b libdispatch.dylib`_dispatch_client_callout + 8
    frame #24: 0x0000000103238cb9 libdispatch.dylib`_dispatch_block_invoke_direct + 289
    frame #25: 0x00000001021595fc libclang_rt.tsan_osx_dynamic.dylib`__tsan::invoke_and_release_block(void*) + 12
    frame #26: 0x0000000102159352 libclang_rt.tsan_osx_dynamic.dylib`__tsan::dispatch_callback_wrap(void*) + 306
    frame #27: 0x0000000103234f1b libdispatch.dylib`_dispatch_client_callout + 8
    frame #28: 0x000000010323d067 libdispatch.dylib`_dispatch_lane_serial_drain + 773
    frame #29: 0x000000010323df4e libdispatch.dylib`_dispatch_lane_invoke + 493
    frame #30: 0x000000010323ce76 libdispatch.dylib`_dispatch_lane_serial_drain + 276
    frame #31: 0x000000010323df4e libdispatch.dylib`_dispatch_lane_invoke + 493
    frame #32: 0x00000001032494db libdispatch.dylib`_dispatch_workloop_worker_thread + 691
    frame #33: 0x00000001032ae015 libsystem_pthread.dylib`_pthread_wqthread + 421
    frame #34: 0x00000001032ade01 libsystem_pthread.dylib`start_wqthread + 13

And a screenshot just in case https://i.imgur.com/u1EpWUv.png

I dont supply my own EventLoopGroup but when I use one, I get a Timeout error which is even more strange... https://i.imgur.com/ULU1EC4.png

Nothing changes when I try the async-http-client branch, it behaves exactly the same with or without my own EventLoopGroup.

Sorry, One last thing can you can you run using the AWSLoggingMiddleware(). Add the following to your client initialisation
let s3 = S3(region: .useast1, middlewares: [AWSLoggingMiddleware()])

And attach the log output in a comment.

Ah, thanks for the idea @adam-fowler I wasnt aware there was a logging middleware :)
Now the sad part :)
In Mojave I get this:
Request:
  GetObject
  GET https://yyy.s3.amazonaws.com/xx/xx/123.oc
  Headers: []
  Body: empty

In Catalina I get this:
Request:
  GetObject
  GET https://yyy.s3.us-east-1.amazonaws.com/xx/xx/123.oc
  Headers: []
  Body: empty
Response:
  Status : 200
  Headers: [
    Content-Type : application/octet-stream
    x-amz-meta-username : Andy
    x-amz-meta- : ["username": "Andy"]
    Connection : close
    Date : Tue, 04 Feb 2020 16:19:11 GMT
    ETag : "a2f822ae49aac15e64565484d15-1"
    x-amz-id-2 :<PHONE_NUMBER>00000_edited_=
    Content-Length : 4780821
    x-amz-tagging-count : 1
    Accept-Ranges : bytes
    Last-Modified : Tue, 04 Feb 2020 16:18:45 GMT
    x-amz-request-id : 752BE6F29A4E52AB
    Server : AmazonS3
  ]
  Body: data (4780821 bytes)

In Mojave the request doesnt even trigger whenFailure which I guess is because Swift NIO throws internally... So pretty much the response fails to get entirely processed.
I am in a bit of time constraint here and can't debug this further, I will get back to it when I have some more time to figure it out.
Thanks a lot for your input @adam-fowler !

@andyzaharia thanks I think that is enough for us to go on. This is obviously as aws-sdk-swift issue not NIO. Not sure what yet though

ps. I forgot to mention that I have edited the https:// part(replaced with yyy), but I guess thats obvious.

Can you verify you are building with version 4.1 of aws-sdk-swift on Mojave because I really can't see any reason why the region is being removed from the Mojave version. Also I know there have been bugs in the past where the useast1 region would disappear from the URL.
Still trying to get a Mojave build up and running at the moment so I am stabbing in the dark

This is pretty much how the Swift package manager dependencies look for the demo project, I was using the latest versions from the master branch.
Screenshot

Are you running with swift 5.0 on Mojave?

I suppose its Swift 5.1, Xcode doesnt indicate if its 5.0 or 5.1. I'm using the latest non beta release of Xcode, 11.3.1 (11C504) which from what I know includes Swift 5.1.
I specially installed Mojave in a Virtual Machine just to separate any potential misconfigurations from my main dev machine. I create the demo project and I can still reproduce the bug.

Yeah ignore that. If you’re running XCode 11 you compiling with 5.1

Hi @andyzaharia are you using aws-sdk-swift for a client side application. Just asking as swift-crypto has just been released and I would really like to use it but it has a macOS 10.15 requirement and I'm not sure I can accept that.

@adam-fowler yup, I'm using it in an internal app on a few machines with Mojave.
I'm still debating actually the removal of the S3 usage from the main application as it has some limitations for my use case.
So, short story... I think you should really go with SwiftCrypto even if that means losing support for older macOS versions.

So I got Mojave running in Parallels. There appears to be an issue with the transport services version. I was wonder could this be related to working in a VM as well.
The reason you are getting a timeout when using the MultiThreadedEventLoopGroup is most likely because you've got a swift breakpoint, are hitting and the code is then taking longer than 5 seconds. Anyway you should be able to fix it by using the MultiThreadedEventLoopGroupand ditching your swift breakpoint. I couldn't get it to crash in this situation.

The trick is that I run into this issue on an actual mojave machine, not in VM / Parallels...  I will give it another shot later today and see if I can get it to work.
Thanks a lot for looking into it!

Ok maybe it is a Mojave issue. But the MultiThreadedEventLoopGroup should fix it.

Managed to reduce the problem down to this https://gist.github.com/adam-fowler/9efe36a8f3641f3a6e020ae1825c86ad

@andyzaharia can you verify that MultiThreadedEventLoopGroup fixes this

Closing soon if no response

Hi @adam-fowler sorry for my really slow response. I tried again the MultiThreadedEventLoopGroup and it doesnt fix the issues, it throws the exception at line 150: in NIO/System.swift
Thank you!

Thanks @andyzaharia

@andyzaharia as an aside I have defaulted Mojave to use MultithreadedEventLoopGroup instead of NIOTSEventloopGroup here https://github.com/swift-aws/aws-sdk-swift-core/commit/b15c12e1946fe4837038c777d518577c54095abe

Thats awesome! Thanks a lot again for you help!


fix AbortDelay.c Test

Fixing Issue with AbortDelay Tests in Release build.
Description
When xStreamBufferStruct gets out of scope it gets delete and results in an error down the line. This is fixed by moving the variable outside the inner scope. The variable does not need to be static because the Buffer is only used once and is deleted afterwards (per functioncall)
Test Steps

Checklist:

[x] I have tested my changes. No regression in existing tests.
[ ] I have modified and/or added unit-tests to cover the code changes in this Pull Request.

Related Issue

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Hi @NikJen26 Thank you for your contribution. Can you please help me understand which test cases are failing without this change?

/bot run uncrustify

In the function prvTestAbortingStreamBufferReceive the call to xStreamBufferReceive results in a assert. This is only the case when i enable -02 and -flto.

@NikJen26 Thank you for the info, i would also like to see what error you are getting. Can you please help in sharing the error information.


fix: Use coverage directory from jest config when available

Fixes #4
Read coverage directory path from the jest config when provided, defaulting to coverage/.

@rbardini Can you take a look?

@rbardini Can you take a look?

Codecov Report

Merging #5 (a01b5b9) into master (676be2f) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##            master        #5   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            7         7
  Lines           67        68    +1
=========================================
+ Hits            67        68    +1

Impacted Files
Coverage Δ

lib/getData.js
100.00% <100.00%> (ø)

lib/index.js
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 676be2f...a01b5b9. Read the comment docs.

Codecov Report

Merging #5 (a01b5b9) into master (676be2f) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##            master        #5   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            7         7
  Lines           67        68    +1
=========================================
+ Hits            67        68    +1

Impacted Files
Coverage Δ

lib/getData.js
100.00% <100.00%> (ø)

lib/index.js
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 676be2f...a01b5b9. Read the comment docs.

Thanks for your contribution, @juzerzarif! 💖

Thanks for your contribution, @juzerzarif! 💖


Update super usage

On Python 3, super can be called without any arguments where as on Python 2, super needs to be called with the class object and an instance e.g. super(DelayedAttr, self). Given that dask does not support Python 2 anymore on master, the super usage can be updated.
Note that this was an automated regex-based search and replace. The regex used was - (super)+\(+(\w+)(,)+\s(self)+\)+. After the automated search and replace happened, each change was individually checked and committed.
Ref : https://docs.python.org/3/library/functions.html#super

[x] Tests added / passed
[x] Passes black dask / flake8 dask

@mrocklin this PR seems to have skipped past your radar.

Thanks @rahulporuri !


Fixed Issue #9401

Issue Reference
Fixed Issue #9401
Description
Validation added in currency modal for min and max 3 digits.

@amit-webkul
Checked and Found that the issue has been fixed and validation has been added. Please find the screenshot here


Failed commands should write an error

Background
Error handling in PowerShell is extremely messy, by far the primary complaint I have heard about the language, and has actually been cited as a reason to not use PowerShell.  The two major complaints I've heard about error handling are:

The two parallel universes of errors
Poor handling of errors from binaries (described in this feature request)

Current behavior
When running a binary command in PowerShell (ex: git) and the binary errors, the command does not always output an error (neither terminating nor non-terminating).  PowerShell clearly detects that the command fails and correctly populates $?, but does not integrate with normal error handling mechanisms.  The result is this cumbersome syntax:
git checkout -b master
if (-not $?) {
    throw "Command failed"
}

Summary of the new feature/enhancement
If a command sets $? to $false, then the command should write an error to the error stream.

I had to do this hack last week, because DevOps agents seem to have inconsistent behavior interacting with Powershell.
$process = Start-Process "cmake" -ArgumentList "-G `"Visual Studio 15 2017`" -T `"LLVM`" -A $Architecture -S `"../..`"" -Wait -NoNewWindow -RedirectStandardError "cmake.log"

if ($process.ExitCode -ne 0) {
    # Do it again to get past openSSL errors.
    $process = Start-Process "cmake" -ArgumentList "-G `"Visual Studio 15 2017`" -T `"LLVM`" -A $Architecture -S `"../..`"" -Wait -NoNewWindow
    Exit 0
}

Binary failures should throw exceptions just like cmdlets so that they can be handled uniformly.

PowerShell clearly detects that the command fails and correctly populates $?

It makes a guess based on the return value, but the return value can only be a number 0-255; it is a convention that 0 means "success" and "1-255" mean "failure", but it's not universal or mandatory.
Robocopy, for one example, reports many different codes for different kinds of "success"; you can robocopy dir1 dir2 /mir and it copies two new files with no problems, and returns 1 and that means $? is False.
Going the other way, PowerShell takes a command writing to stderr to be a problem, and throws a NativeCommandException - but programs like to write license, or information messages to stderr - psexec, and curl do it - and then people get a NativeCommandException, when nothing was wrong. This is also dependent on the PowerShell host, it's inconsistent.
I'm not saying it shouldn't happen, just that it will still be a bit of a guess, and sometimes wrong. It would be a breaking change for every script which runs a native command, and need workarounds for things which sued to work fine but now raise exceptions.

The convention is returning 0 for success, but this is not a standard. The No.1 principle is to not guess. The variable $? is giving you whether the last command returned 0, not whether the last command has an error. Aside from robocopy, fc uses the return value to indicate whether the files are different, in which case automatically writing an error record is an error. The user might want to and could inspect $LASTEXITCODE for error handling.

Returning 0 is a POSIX standard.  I think PowerShell should be aligned with the rules, not the exceptions to the rules.  Aligning to the rules means the command behaves generally according to intuition, whereas aligning to the exceptions means adding an if statement to every shell command, which I think is very cumbersome when one of PowerShell's differentiating features is error handling.
Common utilities that need to play better with PowerShell already have aliases set up (including fc).  PowerShell team could very feasibly provide a similar solution for robocopy, especially considering robocopy is not cross-platform.

@HumanEquivalentUnit, regarding the inconsistent handling of stderr output, see:

#3996
#4002

@zachChilders, as an aside: there is no need to use Start-Process to invoke console/terminal programs such as cmake in a synchronous manner - that's what direct invocation is for.

@chriskuech fc (as well as robocopy) is just an example, and it has no relation whatsoever with Format-Custom. It is also not possible to change the behavior of fc.exe unless you want to break batch scripts on Windows.
The Wikipedia article you cited is about errno, not exit status, which is another concept. I found here and here (page 352).

The value of status may be 0, EXIT_SUCCESS, EXIT_FAILURE, [CX] or any other value, though only the least significant 8 bits (that is, status & 0377) shall be available from wait() and waitpid(); the full value shall be available from waitid() and in the siginfo_t passed to a signal handler for SIGCHLD.

and

Finally, control is returned to the host environment. If the value of status is zero or EXIT_SUCCESS, an implementation-deﬁned form of the status successful termination is returned. If the value of status is EXIT_FAILURE, an implementation-deﬁned form of the status unsuccessful termination is returned. Otherwise the status returned is implementation-deﬁned.

In particular, POSIX standard doesn't say whether exit status 123 means success or failure. Also, the implementation-defined sense of failure of a native command might not be worth an ErrorRecord for the caller in PowerShell.
Aside from that not being a POSIX standard, the methodology of sticking to POSIX standard is good as long as you're only playing with POSIX. However, not all platforms use POSIX. It is much more important to keep consistency among platforms PowerShell would like to support. Moreover, automatically writing error records after native command invocation could create a whole lot of problems for existing scripts who didn't expect them and handled the exit code manually.
Even if we're speculating the exist status and writing an error record. The design such feature is difficult and may lead to more troubles. For example, are you throwing a terminating error or writing a non-terminating error? What if you want to inspect and wrap the error in a more object-oriented way before it is finally written to the error stream? Do you have to $ErrorActionPreference='Stop' and try-catch for every native command invocation?
In general, I suggest wrapping native commands in a separate PowerShell cmdlet/function, which handles all the conversion between the native, untyped, binary, process-level isolated world and PowerShell.
@mklement0 It is a very bad idea to invoke native commands directly given PowerShell's way of handling native pipe. Using Start-Process avoids object-oriented pipe pollution while allowing the native command to print output to the screen.

I don't think there is any dispute that, even if not official, an exit status of 0 conventionally and in most cases means success and non-0 means error.  I understand that there are some corner cases, but ultimately this behavior is counter-intuitive in the vast majority of cases.  If you look at the issue in the "Background" section, I mention another counter-intuitive (and coincidentally improperly documented) error handling behavior (non-terminating errors) that contextualize the justification for this change.

are you throwing a terminating error or writing a non-terminating error?

You should always write a non-terminating error

What if you want to inspect and wrap the error in a more object-oriented way before it is finally written to the error stream

Use normal PowerShell ($Error or $ErrorActionPreference = "Stop" with try-catch, or if implemented -ErrorVariable).  Whatever the programmer chooses, but the programmer should be able to assume that normal PowerShell will apply.

Do you have to $ErrorActionPreference='Stop' and try-catch for every native command invocation?

If the programmer chooses so.  This would mean that if the programmer wants to see why the command failed, they have to add boilerplate.  That would mean less people have to add boilerplate than in the current scenario where the programmer has to add boilerplate just to see if the command failed.
If people in this discussion agree to break backcompat, I definitely think this issue should be addressed.

I don't think there is any dispute that, even if not official, an exit status of 0 conventionally and in most cases means success and non-0 means error. I understand that there are some corner cases, but ultimately this behavior is counter-intuitive in the vast majority of cases.

I think you need some research on how many current scripts break (or show otherwise undesired error messages) if we automatically write an error when a native command returns non-zero value.
I agree that 0 usually means success. But it is quite debatable whether non-0 means error. For example, GNU diff returns 1 for difference found and greater than 1 for errors.
W.r.t. #7774, I will make a separate comment in that thread.
Creating the error record and then catching it is bad for performance, and well-written code should always handle the error and convert them into a readable representation (instead of exit status). The demand to handle them should be prioritized.
In addition, it is always the case people want to know why a command fails. If the command has written to stderr, one already can read it; otherwise, you want to check $LASTEXITCODE.

Made the comment to #7774. And after a second thought, it makes no sense to throw a terminating error here because that would complicate the commands. However, I still consider writing an error for every native command returning non-0 too self-asserting and it complicates the life for actual exit status inspectors/handlers.

Turns out that this request is a duplicate of #3415, which led to lively debated RFC draft https://github.com/PowerShell/PowerShell-RFC/pull/88.
@GeeLaw

It is a very bad idea to invoke native commands directly given PowerShell's way of handling native pipe. Using Start-Process avoids object-oriented pipe pollution while allowing the native command to print output to the screen.

This is a tangent, for sure, but I find your advice highly problematic, so it is worth addressing:
A core function of any shell - including PowerShell - is the synchronous execution of console programs (command-line utilities), with the invoked program's standard streams connected to the calling shell's.
With PowerShell now being multi-platform, this is ever more important, because on Unix platforms there is a wealth of capable - and fast - command-line utilities available.
While PowerShell cannot "speak objects" with native programs, it does have a shared language, and that is text.
The vast majority of command-line utilities "speak text", and in the vast majority of cases you want synchronous execution and either want to capture or suppress text output by such utilities, which calls for direct invocation, not use of Start-Process.
E.g., get the current branch of the Git repository in the current location:
PS> $currentBranch = git symbolic-ref --short HEAD

Even in the rare cases where a utility outputs raw byte streams - which PowerShell's streams cannot handle - you wouldn't use Start-Process - instead, you would synchronously invoke a shell that can handle those byte streams; e.g.:
# Generate a file with 100 random bytes.
# The redirection must be handled by `sh`, because PowerShell would interpret the raw
# bytes as text.
PS> sh -c 'dd if=/dev/urandom bs=1 count=100 > t.txt'

I've summarized the considerations when capturing output from external programs in  this Stack Overflow answer. I encourage you to provide feedback there, if you disagree with the above (the answer was written a while ago, pre .NET Core, so raw byte output isn't covered yet).


hi, Can we remove the backbitten and need to add some other custom button and related function

Please provide you version information :

Dependency
Version

ionic2

angular

ionic-img-viewer

Have you checked...

[ ] That npm install returns no warning ?
[ ] Doing a purge of your node_modules folder ?

I also need some custom button here ,


Soft re-enable of python 3.4

Re-add to tox matrix, and bump the setup.py python_requires
back down. No reason to forcibly exclude 3.4 if it still works.

Codecov Report

Merging #9 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##            master        #9   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            4         4
  Lines           73        73
=========================================
  Hits            73        73

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 01064de...6180f0d. Read the comment docs.


Problem with building Proj4 package

Hi, I'm having problem building Proj4 package, it kept returning the error below:
(@v1.6) pkg> build Proj4

    Building Proj4 → `C:\Users\Yodefia Rahmad\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\6a52f9d893b6ad6f15760990ea8adbabd7b64aa7\build.log`
ERROR: Error building `Proj4`:
ERROR: LoadError: ArgumentError: Will not install a tarball of platform x86_64-w64-mingw32-gcc7 on a system of platform x86_64-w64-mingw32-gcc8-cxx11 unless `ignore_platform` is explicitly set to `true`., override this by setting `ignore_platform`
Stacktrace:
 [1] install(tarball_url::String, hash::String; prefix::Prefix, tarball_path::String, force::Bool, ignore_platform::Bool, verbose::Bool)
   @ BinaryProvider C:\Users\Yodefia Rahmad\.julia\packages\BinaryProvider\U2dKK\src\Prefix.jl:294
 [2] top-level scope
   @ C:\Users\Yodefia Rahmad\.julia\packages\Proj4\RgJ8T\deps\build.jl:83
 [3] include(fname::String)
   @ Base.MainInclude .\client.jl:444
 [4] top-level scope
   @ none:5
in expression starting at C:\Users\Yodefia Rahmad\.julia\packages\Proj4\RgJ8T\deps\build.jl:81

I'm using Julia 1.6.2 on Windows 10. Can anyone help me with this? Thanks!

I don't fully understand, pre-built binaries are provided through our dependency on PROJ_jll. There is no need to run build in the package manager. Quite a while ago it was needed, but not anymore. Which Proj4.jl version are you using?
This is the help for the build command:
(Proj4) pkg> ?build
  build [-v|--verbose] pkg[=uuid] ...

  Run the build script in deps/build.jl for pkg and all of its dependencies in depth-first recursive order. If no
  packages are given, run the build scripts for all packages in the manifest. The -v/--verbose option redirects build
  output to stdout/stderr instead of the build.log file. The startup.jl file is disabled during building unless julia
  is started with --startup-file=yes.

As you can see in this repo there is no deps/build.jl in this repository.
Does pkg> test Proj4 work?


Allow width and height to be set to auto

Allow width and height to be set to auto, and add new test to verify the fix.  Resolves issue #40.

ping @chad3814


[share] having the filename in the url is weird and makes it complicated to use

Since you give each share a UUID, that should be enough.
If the concern is about the filename when saving from a browser, you can use Content-disposition instead I think (https://tools.ietf.org/html/rfc2183#section-2.3).

The filename is part of the share UUID to avoid url confusion.
For browser auto save, use the following query param and value withContentDisposition=true.


New limiter for DG solver for Euler equations

Hello all,
I am trying to implement a new limiter for the DG discretization of Euler equations.  I need to loop over the element neighbors and compute the max and min and averages on the neighbors.  I know that there is a function on gridfunction to calculate the element averages.  Are there similar methods for computing max and min on DG elements ?
I have a couple of more questions

Also, how do I find and loop over neighbor elements ?
How do I change the solution within the timestepper at every stage?  As the limiter needs to be applied at every Runge-Kutta stage. Thanks for your help. I really appreciate it.

Hi @makrandak,
I don't believe similar methods exist for max and min. Do you want the maximum and minimum values at nodal points? Or at certain quadrature points?

You can use the "element-to-element table" (given by Mesh::ElementToElementTable) to find the neighbors of a given element. Be careful in parallel, because the neighbor element may belong to a different processor (indicated by an index that is greater than the number of mesh elements).
The TimeDependentOperator::Mult function is called at every Runge-Kutta stage. Can you write your scheme in terms of a modified RHS function? If so, then you can apply your limiter here. Otherwise, you might want to create a new subclass of ODESolver that implements your specific method.

Hi @makrandak,
There are some methods for both 1. and 2. in Remhos, it might be useful to check it out. For example, a function for computing DG max/min taking into account the neighbors is given here.

Thank you very much both @pazner  and @vladotomov  for your replies.

Do you want the maximum and minimum values at nodal points? Or at certain quadrature points?
This is adjustable in the method.  Ideally both of these set of points.  The reason is that evaluation of max and min on an element depends on how much you sample the function. It is always just an approximate max and min.  If I am able to sample the function more with both quadrature points and nodes the estimate will be better.

You can use the "element-to-element table" (given by Mesh::ElementToElementTable) to find the neighbors of a given element. Be careful in parallel, because the neighbor element may belong to a different processor (indicated by an index that is greater than the number of mesh elements).

I don't quite understand what this means (indicated by an index that is greater than the number of mesh elements)? Index greater than the total no of mesh elements or the no of neighbors?

The TimeDependentOperator::Mult function is called at every Runge-Kutta stage. Can you write your scheme in terms of a modified RHS function? If so, then you can apply your limiter here. Otherwise, you might want to create a new subclass of ODESolver that implements your specific method.

If I understand correctly Mult is essentially calculating RHS by multiplying Operator to the current state.  I just want to limit the state before this Mult happens, this probably could be done inside Mult if I have access to which element I am in when inside Mult and the element neighbors. Maybe there is a different more efficient way, which you may suggest.
Thanks for pointing me to Remhos @vladotomov, Could you please point me to the code for these examples especially for the loop over neighbor elements in Remhos, and the limiting of the solution on RK stages ?

For example, a function for computing DG max/min taking into account the neighbors is given here.
This is a CG element it seems but the computation should not be much different I imagine for the DG element.

Thanks again both of you for your help.

Remhos is not the best example as it has grown quite complicated, but you can use it to see the basic principles.
The limiting is done at every RK stage with this function call. It uses the non-limited solution du_HO and a low-order solution du_LO to compute the limited d_u. This d_u corresponds to the second argument (i.e. the output) of Mult().
The bounds for d_u are computed in the previous two lines. First we compute minimum and maximum over each cell, using the DG function. Then we use auxiliary continuous functions (CG) that take the max/min over overlapping dofs on the faces. Finally we make another pass over the elements and define the bounds for the DG function through the local max/min of the CG function. This is a way to take into account neighbor elements.
I hope this helps,
Vladimir

You can use the "element-to-element table" (given by Mesh::ElementToElementTable) to find the neighbors of a given element. Be careful in parallel, because the neighbor element may belong to a different processor (indicated by an index that is greater than the number of mesh elements).

I don't quite understand what this means (indicated by an index that is greater than the number of mesh elements)? Index greater than the total no of mesh elements or the no of neighbors?

If the element-to-element table indicates that el2 is a neighbor of el1, and el2 >= mesh.GetNE(), then that means that el2 is a "face neighbor" of el1, i.e. it belongs to a different processor.

Hello Will @pazner,
I am struggling to understand the usage of Mesh::ElementToElementTable.  I understand that it is a Table datastructure containing the connectivity.  But, I am not sure on how to use it.  I have been looking at the doxygen documentation but I did not find any notes on usage.  If I am in an element how do I access the neighbors using this? Sorry if this seems naive, but I did not find usage notes in doxygen.
Are there any examples where it is used and I could look at ?
Thanks for your help.

Hi @makrandak ,
Maybe the below code can help you understand all Table datastructure.
#include <iostream>
#include "mfem.hpp"

using namespace std;
using namespace mfem;

int main(int argc, char *argv[])
{
    Mesh mesh(2, 2, Element::TRIANGLE, true);

    const Table& element2edge = mesh.ElementToEdgeTable();
    Array<int> el2e;
    for (size_t i=0; i<mesh.GetNE(); i++)
    {
        element2edge.GetRow(i, el2e);
        Element* el = mesh.GetElement(i); //打印出element的attribute就可以知道这些element确实不包括boundary element
        el2e.Print(cout << i+1 << "-th element, attribute: " << el->GetAttribute() << ", edge indices: ", el2e.Size());
    }

    const Table& element2element = mesh.ElementToElementTable();
    Array<int> el2el;
    for (size_t i=0; i<mesh.GetNE(); i++)
    {
        element2element.GetRow(i, el2el);
        el2el.Print(cout << i << "-th element's neighbor element indices: ");
    }
}

Hello all,
Thank you Will @pazner for introducing me to the Table datastructure and thank you @chang-yiyi  for clarifying it's usage.  It is very useful and versatile, because I have access to it within the Mult function inside the Time dependent operator class.  I wrote my own max and min evaluation functions.
I now understand what Remhos is doing for limiting (combining low order and higher order solutions appropriately), I am using a different strategy with a bound preserving limiter but the place of application is the same if I understand correctly, i.e Mult() method, instead of changing the output I modify the input x  (to limit it) and then let the operator multiply the modified x.  Thanks for your help in pointing these codelines.
I am still working on a general implementation of all this, which is why I am not closing the issue right now.  I will ask if I need some more help.  Thanks for the guidance till now.
-Makrand

Hello all,
I think I have implemented the limiter correctly at least in serial.  I have two questions

How do I handle neighbour elements which are on a different processor? If the ElementToElementTable called on an object of  ParMesh, is the neighbour index correctly identified ? (that it is not greater than the total no of elements) ?

I implemented the limiter to limit the solution which is the input to Mult(x, y)  i.e. x is limited inside Mult (so I just modified x inside Mult) and then the operators are used to calculated y.  However, this does not allow me to limit the solution after the step is done. e.g. for RK solver see below line 16 below.

0 void RK2Solver::Step(Vector &x, double &t, double &dt)
1 {
2     //  0 |
3     //  a |  a
4     // ---+--------
5     //    | 1-b  b      b = 1/(2a)
6
7     const double b = 0.5/a;
8
9     f->SetTime(t);
10   f->Mult(x, dxdt);   ///// Here x is limited
11   add(x, (1. - b)*dt, dxdt, x1);
12   x.Add(a*dt, dxdt);
13
14   f->SetTime(t + a*dt);
15   f->Mult(x, dxdt); ///// Here x is limited
16   add(x1, b*dt, dxdt, x);  /// Here x is not limited, how do I limit x here ?
17   t += dt;
}

Is there a way to get a hook to the solution of the ode at various stages ? Or do I have to write a new odeSolver derived ?
-Makrand

How do I handle neighbour elements which are on a different processor? If the ElementToElementTable called on an object of  ParMesh, is the neighbour index correctly identified ? (that it is not greater than the total no of elements) ?

Hi @makrandak, if the neighbor element is on a different processor, the entry of the table will be the shifted element index, which is greater than or equal to the number of local mesh elements. Denote this index by shifted_idx. We have shifted_idx >= pmesh.GetNE(). We can compute the face neighbor index by subtracting the number of local elements:
int face_nbr_idx = shifted_idx - pmesh.GetNE();

This face neighbor index can be used to get information about this element (e.g. ParMesh::GetFaceNbrElementTransformation, ParFiniteElenentSpace::GetFaceNbrFE, etc.).

I implemented the limiter to limit the solution which is the input to Mult(x, y)  i.e. x is limited inside Mult (so I just modified x inside Mult) and then the operators are used to calculated y.  However, this does not allow me to limit the solution after the step is done. e.g. for RK solver see below line 16 below.
Is there a way to get a hook to the solution of the ode at various stages ? Or do I have to write a new odeSolver derived ?

One option would be to write a new ODESolver. As you can see from the examples built-in to MFEM, this is fairly straightforward. Another option that would work in this case is just to apply one more step of limiting after the time step (since at each RK stage the limiter is already applied). If you use a SSP RK method, perhaps it is not necessary to apply the limiter at the end, because the solution will be written as a convex combination of forward Euler steps.

Hello all,
I was able to get the limiter working in both serial and parallel.  I wanted to limit the solution near the boundary, I was curious if there is a method which returns whether the given element is on the boundary?
-Makrand

Hello all,
On closer look, it seems that the limiter gives different results compared to serial.  My suspicion is that the calculation of bounds and averages on neighbours on other processes is incorrect.
Here is what I am doing to deal with neighbour index of the element
int neighbourFaceIndex = el2el[neighbourIterator] - parMesh_.GetNE();
neighbourIndex = parMesh_.GetFaceNbrElementTransformation(neighbourFaceIndex)->ElementNo;

I don't think this is correct because it turns out el2el[neighbourIterator] and neighbourIndex calculated as above is exactly the same and >= parMesh_.GetNE(). But I am not sure how to fix this.  I am seeking an element index I can use to query gridfunctions to get corresponding elemental quantities like min, max, avgs.
This is a simple sketch of the calculation I am doing.

Calculate ParGridFunctions containing averages and minimas and maximas of each element.   This is done inside the Evolution class.  This is called before the limiting step inside the custom ODE solver.

I access these gridfunctions using block vectors as follows, e.g.
ParGridFunction avgs(&fes_avg_, u_block_avg_.GetData());
ParGridFunction elemMin(&fes_avg_, u_block_min_.GetData());
ParGridFunction elemMax(&fes_avg_, u_block_max_.GetData());

for (int numEq = 0; numEq < inputData_->num_equation; numEq++) {
  elementalAvgPrimitive_i(numEq) = u_block_avg_.GetBlock(numEq).Elem(neighbourIndex);
   elementalMinPrimitive_i(numEq) = u_block_min_.GetBlock(numEq).Elem(neighbourIndex);
  elementalMaxPrimitive_i(numEq) = u_block_max_.GetBlock(numEq).Elem(neighbourIndex);
}

Where u_block_min and u_block_max are BlockVectors which are external data for ParGridFunctions containing max, min and averages.

Calculate the limiter based on these sampled Min, max and averages.  This all happens inside the Evolution class where if I understand correctly vectors are already domain decomposed.

The calculation inside the ODESolver is as follows for example for RK2SSP scheme.
void RK2SolverLimited::Step(Vector &x, double &t, double &dt) {
	//  0 |
	//  a |  a
	// ---+--------
	//    | 1-b  b      b = 1/(2a)

	const double b = 0.5 / a;

	f->SetTime(t);
	f->Mult(x, dxdt);
	add(x, (1. - b) * dt, dxdt, x1);
	x.Add(a * dt, dxdt);

	///Limiting step
	f->calcAveragesAndBounds(x);
	MPI_Barrier(MPI_COMM_WORLD);
	//std::cout<< "Limiting solution Stage 1" <<std::endl;
	f->limitVector(x, x);

	f->SetTime(t + a * dt);
	f->Mult(x, dxdt);
	add(x1, b * dt, dxdt, x);

	///Limiting step
        f->calcAveragesAndBounds(x);
	MPI_Barrier(MPI_COMM_WORLD);
	//std::cout<< "Limiting solution Stage 2" <<std::endl;
	f->limitVector(x, x);

	t += dt;
}

I have noticed that the limiter is different on the processor boundaries and it triggers instabilities.  I am not sure what is wrong with my setup.  I am more confident that the algorithm works fine in serial.  Any help in this regard is much appreciated.
-Makrand

Update:
I still cannot figure out how to sample the gridfunction in parallel with elementToelement Table.  I am now trying to use @mlstowell 's approach in #1700.  I am still getting different results in parallel compared to serial.  It works fine in serial.  Here is the snapshot of the loop I implemented
        ///call for populating ghost elements
	elemMin_.ExchangeFaceNbrData();
	elemMax_.ExchangeFaceNbrData();

        /// local to shared faces mapping lf2sf_ is a member variable
	for (int sf=0; sf< vfes.GetParMesh()->GetNSharedFaces(); sf++)
	{
		int lf = parMesh_.GetSharedFace(sf);
		lf2sf_[lf] = sf;
	}

	ElementTransformation* Tr = nullptr;
	ElementTransformation* TrNbr_it = nullptr;
	for (int i = 0; i < vfes.GetNE(); i++)
	{
		Tr = parMesh_.GetElementTransformation(i);
		IntegrationPoint ip = Geometries.GetCenter(Tr->GetGeometryType());

		avgs_.GetVectorValue(i, ip, elementalAveragesPrimitive);
		avgs_consv_.GetVectorValue(i, ip, elementalAveragesConservative);

		/// Loop over neighbour elements to calculate min and max
		Array<int> faces, ori;
		if (dim == 3) {
			parMesh_.GetElementFaces(i, faces, ori);
		} else if (dim == 2) {
			parMesh_.GetElementEdges(i, faces, ori);
		}
		DenseMatrix elem_min (inputData_->num_equation, faces.Size() + 1);
		DenseMatrix elem_max (inputData_->num_equation, faces.Size() + 1);

                ///----------------------------------For current element -----------------------------------------------------///
		Vector elementalMinPrimitive_0(inputData_->num_equation);
                Vector elementalMaxPrimitive_0(inputData_->num_equation);
		elemMin_.GetVectorValue(i, ip, elementalMinPrimitive_0);
		elemMax_.GetVectorValue(i, ip, elementalMaxPrimitive_0);

		elem_min.SetCol(0, elementalMinPrimitive_0);
		elem_max.SetCol(0, elementalMaxPrimitive_0);

		///------------------------------------------------------------------------------------------------------------///

		FaceElementTransformations *FETrans = nullptr;
		for (int neighbourIterator = 0; neighbourIterator < faces.Size(); neighbourIterator++) {
			Vector elementalMinPrimitive_i(inputData_->num_equation);
			Vector elementalMaxPrimitive_i(inputData_->num_equation);
			FETrans = vfes.GetParMesh()->GetFaceElementTransformations(faces[neighbourIterator]);
			if (FETrans->Elem2) {
				const IntegrationPoint &ip_nbr = Geometries.GetCenter(FETrans->Elem2->GetGeometryType());
				FETrans->Elem2->SetIntPoint(&ip_nbr);

				elemMin_.GetVectorValue(*FETrans->Elem2, FETrans->GetIntPoint(), elementalMinPrimitive_i);
				elemMax_.GetVectorValue(*FETrans->Elem2, FETrans->GetIntPoint(), elementalMaxPrimitive_i);

				//elemMin_.Print(std::cout << i << " " << neighbourIterator << " face neighbor value from other proc ");
			} else {
				std::map<int, int>::iterator mit = lf2sf_.find(faces[neighbourIterator]);
				if (mit != lf2sf_.end()) {
					int sf = mit->second;
					FETrans = vfes.GetParMesh()->GetSharedFaceTransformations(sf);
					const IntegrationPoint &ip_nbr = Geometries.GetCenter(FETrans->Elem2->GetGeometryType());
					FETrans->Elem2->SetIntPoint(&ip_nbr);
					//avgs_.GetVectorValue(*FETrans->Elem2, FETrans->GetIntPoint(), val);
					elemMin_.GetVectorValue(*FETrans->Elem2, FETrans->GetIntPoint(), elementalMinPrimitive_i);
					elemMax_.GetVectorValue(*FETrans->Elem2, FETrans->GetIntPoint(), elementalMaxPrimitive_i);
					//elemMin_.Print(std::cout << i << " " << neighbourIterator << " face neighbor value from other proc ");
				}
			}
			elem_min.SetCol(neighbourIterator + 1, elementalMinPrimitive_i);
			elem_max.SetCol(neighbourIterator + 1, elementalMaxPrimitive_i);
		}
		///-----------------------------------------------------------------------------------------------------------///

		double limiter = calcLimiterPerElem(i, elementalAveragesPrimitive, elem_min, elem_max);

		DenseMatrix xmatCurrentElemLimited(dof, inputData_->num_equation);/// Limited elemental vectors
		///-----------------------------------Get elemental vectors-------------------------------------------------////
		vfes.GetElementVDofs(i, vdofsLim);
		xLim.GetSubVector(vdofsLim, xvalLimited);
		xmatCurrentElem.UseExternalData(xvalLimited.GetData(), dof, inputData_->num_equation);

		///--------------------------------------------------------------------------------------------------------////
		for (int numEq = 0; numEq < inputData_->num_equation; numEq++) {
			///-------------------------------- limiter corrections----------------------------------------------------///
			Vector col(dof);
			for (int d = 0; d < dof; d++) {
				//col(d) = (1.0 - limiter) * col(d) + elementalAveragesConservative(numEq) * (limiter);
				xmatCurrentElemLimited(d, numEq) = elementalAveragesConservative(numEq)
						+ (limiter * (xmatCurrentElem(d, numEq) - elementalAveragesConservative(numEq)));
			}
			///--------------------------------------------------------------------------------------------------------////
		}
		xLimited.SetSubVector(vdofsLim, xmatCurrentElemLimited.GetData()); /// constructing full vector
	}

avgs_, avgs_consv_, elemMin_, elemMax  are ParGridFunctions which are calculated in a separate function and stored as member variables of the class.
Am I missing something in this setup ?  I am seeing correct behaviour in serial, but I see weird behaviour near the processor boundaries in parallel.

int neighbourFaceIndex = el2el[neighbourIterator] - parMesh_.GetNE();
neighbourIndex = parMesh_.GetFaceNbrElementTransformation(neighbourFaceIndex)->ElementNo;

I don't think this is correct because it turns out el2el[neighbourIterator] and neighbourIndex calculated as above is exactly the same and >= parMesh_.GetNE(). But I am not sure how to fix this. I am seeking an element index I can use to query gridfunctions to get corresponding elemental quantities like min, max, avgs.

Hi @makrandak, to access values, etc. of ParGridFunctions on neighboring processors, you have to use the Vector returned from ParGridFunction::FaceNbrData (making sure to call ParGridFunction::ExchangeFaceNbrData before. You can get the degrees of freedom associated with a face neighbor element by indexing into this vector using the DOF array returned from ParFiniteElementSpace::GetFaceNbrElementVDofs.

Hello Will @pazner  Thanks for your reply.  I do call ParGridFunction::ExchangeFaceNbrData.  I don't think I understand, Is FaceNbrData elemental  or local to all the elements in process ?  I am looping through elements and trying to find data of neighbours for each element.
If I understand correctly, you are suggesting something like this
Vector neighbourData = elemMax_.FaceNbrData();
for (int i = 0; i < vfes.GetNE(); i++)
{
             Array <int> nbrDofs;
             vfes.GetFaceNbrElementVDofs(i, nbrDofs);
             Vector dataOfNeighboursOfElement_i;
             neighbourData.GetSubVector(nbrDofs, dataOfNeighboursOfElement_i);
}

Is this correct?
Does this pruned vector dataOfNeighboursOfElement_i contain the data from all neighbours? In that case how do I distinguish between which neighbours data is index where in dataOfNeighboursOfElement_i ?
Thanks for your help and advice.
Makrand

Hi, @makrandak ,
I see one potential issue with your implementation above.  When you call GetVectorValue the integration point that you're using as an argument, FETrans->GetIntPoint(), is a point on the face and not the center of the element.  The FETrans object has three ElementTransformation objects within it; one for the face itself and one for the elements on each side of that face.  Each of these transformation objects could be evaluated at separate locations.  I believe the integration point you want here is ip_nbr or you could use FETrans->Elem2->GetIntPoint().  These should be the same since you just set this.
Incidentally, providing the integration point both within the ElementTransformation object and as a separate argument to calls like GetVectorValue is an unfortunate source of confusion.  We do this for backward compatibility because, apparently, at one time the ElementTransformation didn't have the integration point.
I expect this will change your results, at least by a small amount, both in serial and parallel.  However, I doubt it will bring the two into agreement.
Best wishes,
Mark

If I understand correctly, you are suggesting something like this
Vector neighbourData = elemMax_.FaceNbrData();
for (int i = 0; i < vfes.GetNE(); i++)
{
             Array <int> nbrDofs;
             vfes.GetFaceNbrElementVDofs(i, nbrDofs);
             Vector dataOfNeighboursOfElement_i;
             neighbourData.GetSubVector(nbrDofs, dataOfNeighboursOfElement_i);
}

Is this correct?

No, this isn't quite right. vfes.GetFaceNbrElementVDofs returns the DOFs associated with a given "face neighbor" element, i.e. a single element that is owned by a difference MPI rank. The index you pass should be the "face neighbor index", i.e. what you compute as int neighbourFaceIndex = el2el[neighbourIterator] - parMesh_.GetNE(); You can do something like:
int neighbourFaceIndex = el2el[neighbourIterator] - parMesh_.GetNE();
Array <int> nbrDofs;
vfes.GetFaceNbrElementVDofs(neighbourFaceIndex, nbrDofs);
Vector nbrData;
elemMax_.FaceNbrData().GetSubVector(nbrDofs, nbrData);

At this point nbrData will contain the DOFs associated with the specified face neighbor element.

Hello Will @pazner,  I understand now, I implemented this logic, I think because there are 4 unknowns in the flux vector the nbrData has to be reshaped into a matrix.  I select the first column from this matrix as these ParGridFunctions are piecewise constants so they should be the same whereever within the element they are sampled.
for (int neighbourIterator = 0; neighbourIterator < el2el.Size(); neighbourIterator++) {
			Vector elementalMinPrimitive_i(inputData_->num_equation);
			Vector elementalMaxPrimitive_i(inputData_->num_equation);
			int neighbourIndex;
			bool onDifferentMPIProc;
			if (el2el[neighbourIterator] > parMesh_.GetNE()) {
				onDifferentMPIProc = true;
				int neighbourFaceIndex = el2el[neighbourIterator] - parMesh_.GetNE();
				neighbourIndex = neighbourFaceIndex;
			} else{
				onDifferentMPIProc = false;
				neighbourIndex = el2el[neighbourIterator];
			}
			if (onDifferentMPIProc) {
				Array <int> nbrDofs;
				vfes.GetFaceNbrElementVDofs(neighbourIndex, nbrDofs);
				Vector nbrDataMax;
				Vector nbrDataMin;
				elemMax_.FaceNbrData().GetSubVector(nbrDofs, nbrDataMax);
				elemMin_.FaceNbrData().GetSubVector(nbrDofs, nbrDataMin);
				DenseMatrix neighbourDofDataComnbinedMax(dof, inputData_->num_equation);
				DenseMatrix neighbourDofDataComnbinedMin(dof, inputData_->num_equation);
				neighbourDofDataComnbinedMax.UseExternalData(nbrDataMax.GetData(), dof, inputData_->num_equation);
				neighbourDofDataComnbinedMin.UseExternalData(nbrDataMin.GetData(), dof, inputData_->num_equation);
				elementalMinPrimitive_i = neighbourDofDataComnbinedMin.GetColumn(0);
				elementalMaxPrimitive_i = neighbourDofDataComnbinedMax.GetColumn(0);
			} else {
				TrNbr_it = parMesh_.GetElementTransformation(neighbourIndex);
				IntegrationPoint ipNbr_it = Geometries.GetCenter(TrNbr_it->GetGeometryType());
				elemMin_.GetVectorValue(neighbourIndex, ipNbr_it, elementalMinPrimitive_i);
				elemMax_.GetVectorValue(neighbourIndex, ipNbr_it, elementalMaxPrimitive_i);
			}
			elem_min.SetCol(neighbourIterator + 1, elementalMinPrimitive_i);
			elem_max.SetCol(neighbourIterator + 1, elementalMaxPrimitive_i);

When I run this I run into an assert
Assertion failed: (i>=0 && i<size) is false:
 --> Access element 3328 of array, size = 3328
 ... in function: const T& mfem::Array<T>::operator[](int) const [with T = mfem::Element*]
 ... in file: /home/makrand/Packages/blastfemdg/external/mfem/mesh/../general/array.hpp:718

application called MPI_Abort(MPI_COMM_WORLD, 1) - process 10

Not sure what sampling is going out of range. Do you have any ideas ?

Hello @mlstowell
Thanks for taking a look.

I see one potential issue with your implementation above. When you call GetVectorValue the integration point that you're using as an argument, FETrans->GetIntPoint(), is a point on the face and not the center of the element. The FETrans object has three ElementTransformation objects within it; one for the face itself and one for the elements on each side of that face. Each of these transformation objects could be evaluated at separate locations. I believe the integration point you want here is ip_nbr or you could use FETrans->Elem2->GetIntPoint(). These should be the same since you just set this.

These are piecewise constant ParGridFunction so it should not matter where I am sampling the Vector. But thanks for this tip, it is very useful.
But is the logic for the rest of the loop correct ? Do I need to do anything special for elements on Dirichlet boundaries ?
-Makrand

I haven't looked at your code in detail, but I do notice that el2el[neighbourIterator] > parMesh_.GetNE() should be replaced with el2el[neighbourIterator] >= parMesh_.GetNE().
It may also help to debug your code to get a full backtrace to see where the error is occurring.

Hi @makrandak,
I didn't notice any other issues with the loop.  The faces on Dirichlet boundaries in MFEM are no different than any other boundary faces so they should require no special care in such a loop.
Best wishes,
Mark

Hello Will @pazner ,
Yes, that was the issue, I forgot to add the edge case in the comparison.  It is running now.  I am testing both strategies one with shared faces from #1700 by @mlstowell and the other suggested by you. I will get back to both of you on whether it succeeds.
Thanks for your help, really appreciate it.
-Makrand

Hello Mark @mlstowell
Thanks for looking into it. I will test it further. Thanks for your help, really appreciate it.
-Makrand
-Makrand

Hello Will @pazner ,
I implemented both the approaches and I still see different results in parallel and serial in both of them.  The difference mainly comes on the boundaries of the processors.  I think the sampling the pargridfunction for elements not on the same process is still incorrect.  This is what I am trying.  The results in serial are correct.  I am using a Reimann 2D case to test this.  I can see the shocks are different for parallel compared to serial, I see weird patterns near the shock which align with the boundary of the processors, triggering extra instabilities.
element2element.GetRow(i, el2el);
///----------------------------------------For neighbour elements----------------------------------------------///
for (int neighbourIterator = 0; neighbourIterator < el2el.Size(); neighbourIterator++) {
	Vector elementalMinPrimitive_i(inputData_->num_equation);
	Vector elementalMaxPrimitive_i(inputData_->num_equation);
	int neighbourIndex;
	bool onDifferentMPIProc;
	if (el2el[neighbourIterator] >= parMesh_.GetNE()) {
		onDifferentMPIProc = true;
		int neighbourFaceIndex = el2el[neighbourIterator] - parMesh_.GetNE();
		neighbourIndex = neighbourFaceIndex;
		Array <int> nbrDofs;
		vfes.GetFaceNbrElementVDofs(neighbourIndex, nbrDofs);
		Vector nbrDataMax(dof * inputData_->num_equation);
		Vector nbrDataMin(dof * inputData_->num_equation);
		elemMax.FaceNbrData().GetSubVector(nbrDofs, nbrDataMax);
		elemMin.FaceNbrData().GetSubVector(nbrDofs, nbrDataMin);
		for (int numEqIt = 0; numEqIt < inputData_->num_equation; numEqIt ++) {
			elementalMinPrimitive_i(numEqIt) = nbrDataMin(numEqIt);
			elementalMaxPrimitive_i(numEqIt) = nbrDataMax(numEqIt);
		}
	} else{
		onDifferentMPIProc = false;
		neighbourIndex = el2el[neighbourIterator];
		TrNbr_it = parMesh_.GetElementTransformation(neighbourIndex); ///  should this is be shared face index
		IntegrationPoint ipNbr_it = Geometries.GetCenter(TrNbr_it->GetGeometryType());
		elemMin.GetVectorValue(neighbourIndex, ipNbr_it, elementalMinPrimitive_i);
		elemMax.GetVectorValue(neighbourIndex, ipNbr_it, elementalMaxPrimitive_i);
	}
	elem_min.SetCol(neighbourIterator + 1, elementalMinPrimitive_i);
	elem_max.SetCol(neighbourIterator + 1, elementalMaxPrimitive_i);
}

Hello all,
Thanks for Will @pazner , we found the problem in the data sampling in the case where the elements are on a different processor.
the correct piece of code is as follows,
///----------------------------------------For neighbour elements------------------------------------------------///
for (int neighbourIterator = 0; neighbourIterator < el2el.Size(); neighbourIterator++) {
	Vector elementalMinPrimitive_i(inputData_->num_equation);
	Vector elementalMaxPrimitive_i(inputData_->num_equation);
	int neighbourIndex;
	if (el2el[neighbourIterator] >= parMesh_.GetNE()) {
		int neighbourFaceIndex = el2el[neighbourIterator] - parMesh_.GetNE();
		Array <int> nbrDofs;

		elemMax_.ParFESpace()->GetFaceNbrElementVDofs(neighbourFaceIndex, nbrDofs);

		Vector nbrDataMax(inputData_->num_equation);
		Vector nbrDataMin(inputData_->num_equation);
		elemMax_.FaceNbrData().GetSubVector(nbrDofs, nbrDataMax);
		elemMin_.FaceNbrData().GetSubVector(nbrDofs, nbrDataMin);

		for (int numEqIt = 0; numEqIt < inputData_->num_equation; numEqIt ++) {
			elementalMinPrimitive_i(numEqIt) = nbrDataMin(numEqIt);
			elementalMaxPrimitive_i(numEqIt) = nbrDataMax(numEqIt);
		}
	} else{
		neighbourIndex = el2el[neighbourIterator];
		TrNbr_it = parMesh_.GetElementTransformation(neighbourIndex);
		IntegrationPoint ipNbr_it = Geometries.GetCenter(TrNbr_it->GetGeometryType());
		elemMin_.GetVectorValue(neighbourIndex, ipNbr_it, elementalMinPrimitive_i);
		elemMax_.GetVectorValue(neighbourIndex, ipNbr_it, elementalMaxPrimitive_i);
	}
	elem_min.SetCol(neighbourIterator + 1, elementalMinPrimitive_i);
	elem_max.SetCol(neighbourIterator + 1, elementalMaxPrimitive_i);
}
///--------------------------------------------------------------------------------------------------------------///

where elemMax_, elemMin_, are ParGridFunctions and ExchangeFaceNbrData has been called on them before the above sampling.
Thanks all who wrote to help. Closing this issue.
-Makrand


Failure using Keras' fit_generator attempting to train image classifier

I created a DenseNet model following the instructions in the README.md. I am using Keras with TF for the back-end to train a model (from scratch) to classify 240x240 images into one of 13 classes.
Attached is a simplified script that shows how I set up the model, generators, and optimizer. The code fails when I call the train_generator's fit_generator method. Attached is the output log.
Here are my package versions:

Python	2.7.5
cv2	3.3.0
numpy	1.13.3
Keras	2.0.8
Tensorflow	1.2.1
h5py	2.7.1
NVCC	release 8.0, V8.0.61
tensoflow-gpu	1.2.1

densenet_simplified.txt
densenet_output.txt
Any help in how I might change the configuration to use less memory (if that's what I am doing wrong) would be greatly appreciated.
Thanks,
Michael.

Doesn't the Titan X have 12 GB memory? Also, this is the smallest DenseNet that is in the paper so it shoudn't run out of memory. However, this DenseNet model is used for CIFAR 10/100 models.
Perhaps try with subsample_initial_block=True, bottleneck=True

I was able to get it working by changing the depth. The "default" is depth=40. I changed it to depth=7, and fit_generator is now working. That is, I don't get the memory failure. It's still on the first epoch but the accuracy is going up, so this looks very promising!


[Bug]: Version 5.4.0 missing in Debian Repository

Before you file a bug report

[X] I have checked the issue tracker and have not found an issue that matches the one I'm filing.
[X] This issue is not a troubleshooting question. Troubleshooting questions go here: https://forum.mattermost.com/c/trouble-shoot/16.
[X] This issue is not a feature request. You can request features and make product suggestions here: https://mattermost.com/suggestions/.
[X] This issue doesn't reproduce on web browsers (such as in Chrome). If it does, issue reports go to the Mattermost Server repository.
[X] This issue reproduces on the most recent stable version, or the most recent prerelease version of the Mattermost Desktop App.
[X] I have read the contribution guidelines.

Mattermost Desktop Version
5.3.1
Operating System
Debian GNU/Linux 11 (amd64)
Mattermost Server Version
No response
Steps to reproduce
The client told me, that 5.4.0 is available, but the version is apparently missing in the repo available under https://deb.packages.mattermost.com/.
Expected behavior
Version 5.4.0 should be available in the repo.
Observed behavior
Only the following versions are available: 5.0.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.1.0, 5.1.1, 5.2.0, 5.2.1, 5.2.2 and 5.3.1.
Log Output
# apt-get install mattermost-desktop
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
mattermost-desktop is already the newest version (5.3.1-496).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Additional Information
No response

Looks like our build didn't finish properly for the DEB repo. I've rerun the pipeline and it should be here now:
devinbinnie@devinbinnie-Standard-PC-Q35-ICH9-2009:~$ sudo apt install mattermost-desktop/stable
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Selected version '5.4.0-570' (Mattermost Releases:stable [amd64]) for 'mattermost-desktop'

Thanks for letting us know :)