Loss of performance with series which have null values

Codepen
https://codesandbox.io/s/18vx35nn2q
To reproduce the issue, you need to copy the content of  this file and paste it on the first input. Then, copy the content of  this file and paste it on the second input. This allows you to reproduce the issue with my data.
Here is the exemple :

Explanation
As you can see, once the two series are paste, there is a big lack of performances because of the second serie (it's composed of null values).
If I replace null values by 0, there is no lack of performance anymore.
You can test by replacing the second serie with this file
Any way to prevent this ? (I need the second serie to have null values)
I notice there is nevertheless a loss of performance with only the second series but less than with the two series.

When you have null values in a series, the path is broken into single bits for each data-point as it cannot be drawn as a single path.
If you switch to line chart for such a scenario, you will notice a significant improvement in the performance. Also, changing fill.type: 'solid' will make a slight difference in performance.

I think, instead of breaking path when null values are present, an option should be provided to join the null values. I will think about it.

If you switch to line chart for such a scenario, you will notice a significant improvement in the performance. Also, changing fill.type: 'solid' will make a slight difference in performance.

thank you very much, it works perfectly !

I think, instead of breaking path when null values are present, an option should be provided to join the null values. I will think about it.

Personally, I use the null values so that the line is cut at certain points.
Indeed, it can be a good idea of additional option

I have data points of 1000+ in some scenarios most of the points are null values for their respective dates. I'm using a line chart with time series. performance of the graph is very low while getting more null points. sometimes page went to unresponsive mode.

+1


Check 3d jumps plot

After #228, it seems that there are no periodic jumps, check if that is correct

in #278 a plotly version is added with the periodic jumps, this change should be backported to the matplotlib version


Backport #26434

to fix threejs viewer in jupyter

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.


Removes unused fa icons preloading

Description
Fixes fa-solid-900.woff2 not found 404 error
Notes (out of scope, known isues, hints for reviewing code, ...)  (optional)
...
Test cases

[auto-generated message] E2E test report on: https://algorea-static.s3.eu-central-1.amazonaws.com/branch/bugfix/remove-fa-fonts-preloading/playwright-report/index.html


Refactoring

Description

Reorder imports.
Change to import a named export.
Make ipcWrapper.
Delete unnecessary else.
Refactor error handling.

Use early return and add error logging.
Some edits may improve performance.

Make installation and counting verification common.

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

This PR became too big.
So, I'll open separate PRs and close this PR.


Close connections when either reader done

Try to close connections when either reader from client or db done to read, so the db connection state is clean.

@iporsut Thanks for the PR!


Add 'MatchesAllOf' to make it possible to combine matchers.

MatchesAllOf allows us to combine multiple checkers into one checker to be used with the usual test tooling.

/assign @srinivashegde86


Preview image/pdf on the card

Feature request from the Twitter account: https://twitter.com/gruenler/status/518108481036255232

+1 on the request for an image thumbnail on the kanban card.  This feature is the only reason why I haven't ditched Trello yet.

What do you think about this?

:+1:

       Looks good. Which image formats does it support?

FROM: Timo
SENT: Sunday, 1 March 2015 10:57
TO<EMAIL_ADDRESS>REPLY TO: fguillot/kanboard
CC: Olivier Paroz
SUBJECT: Re: [kanboard] Preview image/pdf on the card (#307)

What do you think about this?
[1]
—
Reply to this email directly or view it on GitHub[2].

Liens:
[1]
https://cloud.githubusercontent.com/assets/1961634/6430242/a233ba2c-c001-11e4-955b-af5bc8e7da7a.PNG
[2] https://github.com/fguillot/kanboard/issues/307#issuecomment-76590221

at the moment kanboard supports image types, that are supported from php getimagesize https://github.com/fguillot/kanboard/blob/master/app/Model/File.php#L122 http://php.net/manual/de/function.getimagesize.php
This php command supports:
[IMAGETYPE_GIF]
[IMAGETYPE_JPEG]
[IMAGETYPE_PNG]
[IMAGETYPE_SWF]
[IMAGETYPE_PSD]
[IMAGETYPE_BMP]
[IMAGETYPE_TIFF_II]
[IMAGETYPE_TIFF_MM]
[IMAGETYPE_JPC]
[IMAGETYPE_JP2]
[IMAGETYPE_JPX]
[IMAGETYPE_JB2]
[IMAGETYPE_SWC]
[IMAGETYPE_IFF]
[IMAGETYPE_WBMP]
[IMAGETYPE_JPEG2000]
[IMAGETYPE_XBM]
[IMAGETYPE_ICO]

I have changed the layout, i think it looks much clearer, but you could see the whole width of the picture, but not the full height.

Hi BlueTeck: I believe the original Twitter message was asking for a thumbnail image to show up "on" the kanban card viewed within the overall board view.  The method Trello uses is to pick the "last uploaded image" as the representative thumbnail to go on the kanban card.
Here is a sample image taken from my Trello board:

I will look into that:

better attachment handling in task view
better popup in task on board
optional display last image on card
thumbnails for better performance

From a usability perspective, I think a general option to set the default behavior of displaying an image thumbnail on all cards would be useful, probably more useful that an option within each card for those of us who use hundreds of cards.
BTW, thank you so much for picking this up BlueTeck

@lawre i thought about an this option per project

I think the general option is a bad idea, i like this:

You could select one image from the attachments as cover. So not every task with images has a cover image.

Fair enough.  It might work out well that way

BlueTek, can you make sure #511 is incorporated in your new feature?  I saw you commenting on #511, so I have my fingers crossed :)

@lawre thumbnails are also available with screenshots

as @fguillot said, its wirking fine with screenshots.
regarding the cover image i am playing around with it, but it isnt finished. https://github.com/BlueTeck/kanboard/tree/cover_image

Hi,
Can we have images smaller that thumbnali not get resized? I have small icons attached to task and thumbnails do not look good :)

It is me again ;)
The coverimage is coming via plugin, first version is a little bit ugly but working via plugin:
https://github.com/BlueTeck/kanboard_plugin_coverimage

Only working with the kanboard dev version or the next release.

First Version released:
https://github.com/BlueTeck/kanboard_plugin_coverimage/releases/tag/0.1.0
Needs Kanboard v1.0.27 or the latest master


Do you have any plans for add TouchScreen swipe (move)?

https://github.com/zhgqthomas/puppeteer/commit/9f54be7bd5937c4224e10554b76270ce7357ef8a

@SleepyF0X, that code is not official. We only implement features implemented if the official repo. I would start asking for swipe support there.


Xgboost regressor training with GPU does not work when python environment does not have "cudf" package

Testing on xgboost==1.7.1
Reproducing code:
# run on a spark cluster configured with GPU.

df_train = spark.createDataFrame([
    (Vectors.dense(1.0, 2.0, 3.0), 0, False, 1.0),
    (Vectors.sparse(3, {1: 1.0, 2: 5.5}), 1, False, 2.0),
    (Vectors.dense(4.0, 5.0, 6.0), 0, True, 1.0),
    (Vectors.sparse(3, {1: 6.0, 2: 7.5}), 1, True, 2.0),
] * 100, ["features", "label", "isVal", "weight"])
df_test = spark.createDataFrame([
    (Vectors.dense(1.5, 2.0, 3.0), ),
    (Vectors.sparse(3, {1: -1.0, 2: 5.5}), ),
] * 100, ["features"])

from xgboost.spark import SparkXGBRegressor
xgb_regressor = SparkXGBRegressor(
    num_workers=2,
    max_depth=5, missing=0.0, use_gpu=True,
    validation_indicator_col='isVal', weight_col='weight',
    early_stopping_rounds=1, eval_metric='rmse'
)
xgb_reg_model = xgb_regressor.fit(df_train)
xgb_reg_model.transform(df_test).collect()

Error:
Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Could not recover from a failed barrier ResultStage. Most recent failure reason: Stage failed because barrier task ResultTask(71, 0) finished unsuccessfully.
org.apache.spark.api.python.PythonException: 'ModuleNotFoundError: No module named 'cudf''. Full traceback below:
Traceback (most recent call last):
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-824c7a6f-bc83-452f-baf0-520e7f6f9371/lib/python3.9/site-packages/xgboost/spark/core.py", line 809, in _train_booster
    booster = worker_train(
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-824c7a6f-bc83-452f-baf0-520e7f6f9371/lib/python3.9/site-packages/xgboost/spark/data.py", line 309, in create_dmatrix_from_partitions
    dtrain = make_qdm(train_data, gpu_id, meta, None, params)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-824c7a6f-bc83-452f-baf0-520e7f6f9371/lib/python3.9/site-packages/xgboost/spark/data.py", line 167, in make_qdm
    m = QuantileDMatrix(it, **params, ref=ref)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-824c7a6f-bc83-452f-baf0-520e7f6f9371/lib/python3.9/site-packages/xgboost/core.py", line 620, in inner_f
    return func(**kwargs)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-824c7a6f-bc83-452f-baf0-520e7f6f9371/lib/python3.9/site-packages/xgboost/core.py", line 1386, in __init__
    self._init(
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-824c7a6f-bc83-452f-baf0-520e7f6f9371/lib/python3.9/site-packages/xgboost/core.py", line 1445, in _init
    it.reraise()
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-824c7a6f-bc83-452f-baf0-520e7f6f9371/lib/python3.9/site-packages/xgboost/core.py", line 488, in reraise
    raise exc  # pylint: disable=raising-bad-type
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-824c7a6f-bc83-452f-baf0-520e7f6f9371/lib/python3.9/site-packages/xgboost/core.py", line 478, in _handle_exception
    return dft_ret
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-824c7a6f-bc83-452f-baf0-520e7f6f9371/lib/python3.9/site-packages/xgboost/core.py", line 534, in <lambda>
    return self._handle_exception(lambda: self.next(input_data), 0)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-824c7a6f-bc83-452f-baf0-520e7f6f9371/lib/python3.9/site-packages/xgboost/spark/data.py", line 99, in next
    data=self._fetch(self._data[alias.data]),
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-824c7a6f-bc83-452f-baf0-520e7f6f9371/lib/python3.9/site-packages/xgboost/spark/data.py", line 85, in _fetch
    import cudf  # pylint: disable=import-error
ModuleNotFoundError: No module named 'cudf'

Seemingly I need to manually build cudf if I don't use conda environment.

@trivialfis Could you release a pypi xgboost package that containing the  cudf lib ?
cudf cannot be installed correctly via pip , and building it is hard.

@WeichenXu123 It is now possible to install cudf using pip. See https://rapids.ai/pip.html

@hcho3
I tried:
pip install cudf-cu11==22.10.0 --extra-index-url=https://pypi.ngc.nvidia.com

it will install a dependency cupy-cuda115, which is only compatible with cuda-11.5
I cannot find a version that is compatible with cuda 11.3 (which is the version currently we use)
@hcho3 Can we make xgboost-spark using "cudf" optionally ? If no cudf installed, fallback to use normal pandas dataframe.

@WeichenXu123 https://rapids.ai/pip.html says that you can install cuPy version for CUDA 11.3:
pip uninstall cupy-cuda115; pip install cupy-cuda11x

Have you tried this?

And I'm not sure if cuDF can be made optional. I'll defer to @trivialfis

@WeichenXu123 https://rapids.ai/pip.html says that you can install cuPy version for CUDA 11.3:

When installing these packages with CUDA 11.2, 11.3, or 11.4, you may experience a “Failed to import CuPy” error. To resolve this error, please uninstall cupy-cuda115 and install cupy-cuda11x:

pip uninstall cupy-cuda115; pip install cupy-cuda11x

Have you tried this?

Yes I tried, then got error:
>>> import cudf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/cudf/__init__.py", line 22, in <module>
    from cudf.core.dataframe import DataFrame, from_dataframe, from_pandas, merge
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/cudf/core/dataframe.py", line 57, in <module>
    from cudf.core import column, df_protocol, reshape
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/cudf/core/df_protocol.py", line 645, in <module>
    _INTS = {8: cp.int8, 16: cp.int16, 32: cp.int32, 64: cp.int64}
AttributeError: module 'cupy' has no attribute 'int8'

@hcho3

And I'm not sure if cuDF can be made optional. I'll defer to @trivialfis

I think it is doable, because when I contributed the xgboost.spark module, it works without cudf, but now it does not work without cudf

@WeichenXu123  The import cudf statement was introduced in #8088, which you approved. It seems like we accidentally introduced cudf dependency there. We should try to make cudf optional.

Just made #8469 as a first attempt to fix.

Hi everybody, is there any solution or we have to wait 1.7.2 patch release?
Thanks guyz

You can use nightly build if you don't want to wait for a patch release.

nightly build?

https://xgboost.readthedocs.io/en/stable/install.html#nightly-build


Pod being OOMKilled

No obvious reason, just not working...

Ok found, the operator seems to require a minimum of 132MB of RAM in order to get it working


Adding option target_region for base profiles

If a target_region is specified in a baseProfile, then all regions that list it as a sourceProfile will automatically use that region.

Any chance of this getting merged in?


Missing git tag for version 0.6.4

I see you published 0.6.4. That's great! Can you create a git tag so that downstream package managers can point at it? 🙇

Thank you for pointing this out. I indeed only published that version on crates.io. I'm going to make a release now.

I just checked back and saw you created the tag. Thank you!


ROCm gfx10 and gfx11 not supported

Looks like only gfx9 and below is supported here ATM.
gfx1010, gfx1012, gfx1030, gfx1100, gfx1101, and gfx1102 are affected.
Here's the error log. I'll link you to the build environment once I submit a PR to nixpkgs.
ginkgo-hpc-hip> In file included from /build/source/build/hip/preconditioner/jacobi_advanced_apply_instantiate.64.hip.cpp:66:
ginkgo-hpc-hip> /build/source/common/cuda_hip/preconditioner/jacobi_advanced_apply_kernel.hpp.inc:84:9: error: no matching function for call to 'tiled_partition'
ginkgo-hpc-hip>         group::tiled_partition<subwarp_size>(group::this_thread_block());
ginkgo-hpc-hip>         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ginkgo-hpc-hip> /build/source/build/hip/preconditioner/jacobi_advanced_apply_instantiate.64.hip.cpp:98:21: note: in instantiation of function template specialization 'gko::kernels::hip::jacobi::kernel::advanced_adaptive_apply<64, 64, 4, float, int>' requested here
ginkgo-hpc-hip>             kernel::advanced_adaptive_apply<max_block_size, subwarp_size,
ginkgo-hpc-hip>                     ^
ginkgo-hpc-hip> /build/source/hip/components/cooperative_groups.hip.hpp:392:5: note: candidate template ignored: requirement '64UL <= kernels::hip::config::warp_size' was not satisfied [with Size = 64]
ginkgo-hpc-hip>     tiled_partition(const Group&)
ginkgo-hpc-hip>     ^
ginkgo-hpc-hip> In file included from /build/source/build/hip/preconditioner/jacobi_advanced_apply_instantiate.64.hip.cpp:66:
ginkgo-hpc-hip> /build/source/common/cuda_hip/preconditioner/jacobi_advanced_apply_kernel.hpp.inc:50:9: error: no matching function for call to 'tiled_partition'
ginkgo-hpc-hip>         group::tiled_partition<subwarp_size>(group::this_thread_block());
ginkgo-hpc-hip>         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ginkgo-hpc-hip> /build/source/build/hip/preconditioner/jacobi_advanced_apply_instantiate.64.hip.cpp:105:21: note: in instantiation of function template specialization 'gko::kernels::hip::jacobi::kernel::advanced_apply<64, 64, 4, float, int>' requested here
ginkgo-hpc-hip>             kernel::advanced_apply<max_block_size, subwarp_size,
ginkgo-hpc-hip>

How did you determine the affected versions? It looks like they still use wavefronts of 64 threads, so there might be something going wrong with the macro we use to determine whether we are compiling for CUDA or ROCm.

Simple. Ran the build like 6 times and added whichever one's I saw was failing in the log to the gpu targets exclude list in my package.

PR: #261155
Relevant files: https://github.com/NixOS/nixpkgs/pull/261155/commits/93464fea98b8b758cfd98eefea22f52e37d566d2

Can reproduce this outside nixos

First observations: This is the failing command
/opt/rocm-5.6.0/bin/hipcc
-c /home/tribizel/ginkgo/build-tmp/hip/preconditioner/jacobi_advanced_apply_instantiate.64.hip.cpp
-o /home/tribizel/ginkgo/build-tmp/hip/CMakeFiles/ginkgo_hip.dir/preconditioner/./ginkgo_hip_generated_jacobi_advanced_apply_instantiate.64.hip.cpp.o
-std=c++14
-DGKO_COMPILING_HIP
--amdgpu-target=gfx1010
-fPIC
-fPIC
-O3
-DNDEBUG
-I/opt/rocm-5.6.0/include -I/home/tribizel/ginkgo/build-tmp/hip/.. -I/opt/rocm-5.6.0/include -I/opt/rocm-5.6.0/include -I/opt/rocm-5.6.0/include -I/opt/rocm-5.6.0/include -I/opt/rocm-5.6.0/include -I/home/tribizel/ginkgo/build-tmp/include -I/home/tribizel/ginkgo/include -I/home/tribizel/ginkgo -I/usr/local/include -I/home/tribizel/ginkgo/build-tmp/include -I/home/tribizel/ginkgo/include -I/home/tribizel/ginkgo -I/usr/local/include -I/usr/local/include -I/opt/rocm-5.6.0/include -I/opt/rocm-5.6.0/include -I/opt/rocm-5.6.0/include -I/opt/rocm-5.6.0/include -I/opt/rocm-5.6.0/include/hiprand -I/opt/rocm-5.6.0/include -I/opt/rocm-5.6.0/include/rocrand -I/opt/rocm-5.6.0/include -I/opt/rocm-5.6.0/include -I/opt/rocm-5.6.0/include

Removing --amdgpu-target (which is a deprecated flag BTW, we should change it) fixes it. hipcccalls two compilation passes (one host, one device), where the second one includes the generated GPU binary blob. With--amdgpu-target, the first compilation already fails. This might have to do with /opt/rocm-5.6.0/amdgcn/bitcode/oclc_wavefrontsize64_off.bc` that is being added for gfx1010.

Related: https://github.com/RadeonOpenCompute/llvm-project/blob/7c3a4e693e3ae2ea69f34f9fdbb84dd9889835cb/amd/device-libs/doc/OCML.md#controls

You mean here? https://github.com/ginkgo-project/ginkgo/blob/2f3720ffb35f9f005ef95a15ea86765f40748afe/cmake/hip.cmake#L201
# `--amdgpu-target` is deprecated
substituteInPlace cmake/hip.cmake \
  --replace "--amdgpu-target" "--offload-arch"

Removing list(APPEND GINKGO_AMD_ARCH_FLAGS --amdgpu-target=${target})  works, but I'm not sure if that should be done.
We specify those flags for a reason right?

Probable fix: https://llvm.org/docs/AMDGPUUsage.html#target-features use -mwavefrontsize64 for those targets. Not sure if it has an effect in targets that support size 64 wavefronts natively.

I'm getting this error when prepending -mwavefrontsize64 to --amdgpu-target.
In file included from /build/source/include/ginkgo/core/base/exception.hpp:41:
ginkgo-hpc-hip> In file included from /build/source/include/ginkgo/core/base/types.hpp:50:
ginkgo-hpc-hip> /nix/store/7y1f77gd62zy52b52ivapa4inkjcb9mq-clr-5.7.0/include/hip/hip_runtime.h:41:2: error: HIP is not supported on the specified GPU ARCH with wavefront size 64
ginkgo-hpc-hip> #error HIP is not supported on the specified GPU ARCH with wavefront size 64
ginkgo-hpc-hip>  ^
ginkgo-hpc-hip> [ 20%] Building CXX object reference/CMakeFiles/ginkgo_reference.dir/solver/cb_gmres_kernels.cpp.o
ginkgo-hpc-hip> 1 warning and 1 error generated when compiling for gfx1010.
ginkgo-hpc-hip> CMake Error at ginkgo_hip_generated_residual_norm_kernels.hip.cpp.o.cmake:200 (message):
ginkgo-hpc-hip>   Error generating file
ginkgo-hpc-hip>   /build/source/build/hip/CMakeFiles/ginkgo_hip.dir/stop/./ginkgo_hip_generated_residual_norm_kernels.hip.cpp.o
ginkgo-hpc-hip>
ginkgo-hpc-hip> make[2]: *** [hip/CMakeFiles/ginkgo_hip.dir/build.make:399: hip/CMakeFiles/ginkgo_hip.dir/stop/ginkgo_hip_generated_residual_norm_kernels.hip.cpp.o] Error 1
ginkgo-hpc-hip> make[2]: *** Waiting for unfinished jobs....
ginkgo-hpc-hip> [ 21%] Building CXX object reference/CMakeFiles/ginkgo_reference.dir/solver/common_gmres_kernels.cpp.o

Thanks for the report, that's disappointing. It should be possible to compile Ginkgo with wavefront size 32, but unfortunately we don't have any gfx10/11 GPUs available right now to test this, and I'm not comfortable claiming support for it without checking that the tests run correctly. Some dependencies (rocBLAS, rocSPARSE, rocFFT) might also not work on them? Though I haven't checked yet.

If you can give me a patch set I can give you at least a preliminary indicator of whether it works or not.
I'm on gfx1030. Theoretically if it works for that it should work for 1010, 1012, 1100, 1101, and 1102.

I just noticed we are doing this almost correctly already. As a first check, you could try
diff --git a/hip/CMakeLists.txt b/hip/CMakeLists.txt
index 21b573b6cd..6740a83d99 100644
--- a/hip/CMakeLists.txt
+++ b/hip/CMakeLists.txt
@@ -69,7 +69,3 @@ endif()

-if (GINKGO_HIP_PLATFORM MATCHES "${HIP_PLATFORM_NVIDIA_REGEX}")
-    set(GKO_HIP_JACOBI_MAX_BLOCK_SIZE 32)
-else()
-    set(GKO_HIP_JACOBI_MAX_BLOCK_SIZE 64)
-endif()
+set(GKO_HIP_JACOBI_MAX_BLOCK_SIZE 32)
 if(GINKGO_JACOBI_FULL_OPTIMIZATIONS)

and check which tests run correctly. I only see a handful that will definitely fail.

Looks like that fixed pretty much everything.
Going to try that test with only one GPU target and see if everything passes.
https://github.com/NixOS/nixpkgs/blob/59dabc593819109a57df9bf03731ecf217beb9dc/pkgs/development/libraries/ginkgo-hpc/default.nix#L175-L188

Note that this is not a complete fix, it's is intended only a workaround for consumer-grade GPUs. Though I'm not sure if we actually do need larger than 32 Jacobi blocks.

ginkgo-test.log
Effectively the same test output using only gfx1030.

That's how it looks when no kernels are being run in CUDA/we compiled for the wrong architecture. We don't seem to be catching errors of that kind explicitly. I guess it's safe to say that for now, we can only support server-grade GPUs (Radeon VII and Instinct series)

Could you also try appending the following change in hip/base/config.hip.hpp?
diff --git a/hip/base/config.hip.hpp b/hip/base/config.hip.hpp
index 8a53037ffa..ea33ce5f09 100644
--- a/hip/base/config.hip.hpp
+++ b/hip/base/config.hip.hpp
@@ -56,7 +56,7 @@ struct config {
      * The type containing a bitmask over all lanes of a warp.
      */
 #if GINKGO_HIP_PLATFORM_HCC
-    using lane_mask_type = uint64;
+    using lane_mask_type = uint32;
 #else  // GINKGO_HIP_PLATFORM_NVCC
     using lane_mask_type = uint32;
 #endif
@@ -66,7 +66,7 @@ struct config {
      * `device_functions.h`.
      */
 #if GINKGO_HIP_PLATFORM_HCC
-    static constexpr uint32 warp_size = warpSize;
+    static constexpr uint32 warp_size = 32;
 #else  // GINKGO_HIP_PLATFORM_NVCC
     static constexpr uint32 warp_size = 32;
 #endif

warpSize to 32 might not be neccessary because AMD should use 32 already for those archs.

@yhmtsai that likely won't make a difference, since uint32 and uint64 are inter-convertible. The use of warpSize is why this issue came up in the first place, otherwise we wouldn't be trying to instantiate the 64 blocksize Jacobi kernels.

The voting function will consider the all bits of mask type.

The result of __ballot gets cast to uint64 from uint32, which means the upper 32 bits are zero

Yes, but it only works for checking any but does not work for all because some of the bits will be zero.
The all will always be false. We did safer operations on SYCL backend but in HIP we assume the mask type bits == the wavefront size.

I think we're safe there:
        if (Size == config::warp_size) {
            return __all(predicate);
        } else {
            return (__ballot(predicate) & data_.mask) == data_.mask;
        }

but you are right, many of the failing tests use either cooperative groups or rocBLAS.

Could you also try appending the following change in hip/base/config.hip.hpp?
diff --git a/hip/base/config.hip.hpp b/hip/base/config.hip.hpp
index 8a53037ffa..ea33ce5f09 100644
--- a/hip/base/config.hip.hpp
+++ b/hip/base/config.hip.hpp
@@ -56,7 +56,7 @@ struct config {
      * The type containing a bitmask over all lanes of a warp.
      */
 #if GINKGO_HIP_PLATFORM_HCC
-    using lane_mask_type = uint64;
+    using lane_mask_type = uint32;
 #else  // GINKGO_HIP_PLATFORM_NVCC
     using lane_mask_type = uint32;
 #endif
@@ -66,7 +66,7 @@ struct config {
      * `device_functions.h`.
      */
 #if GINKGO_HIP_PLATFORM_HCC
-    static constexpr uint32 warp_size = warpSize;
+    static constexpr uint32 warp_size = 32;
 #else  // GINKGO_HIP_PLATFORM_NVCC
     static constexpr uint32 warp_size = 32;
 #endif

warpSize to 32 might not be neccessary because AMD should use 32 already for those archs.

Sorry for the late reply.
With these changes more tests are passing.
ginkgo-test.log

That is encouraging. Based on the test output, I would be so bold to put the blame for all non-MPI failures on rocBLAS, which we use for our GEMM operations, which we use in all tests for the system matrix. Can you try running the rocBLAS test suite?

I assume you're talking about gtest in https://github.com/ROCmSoftwarePlatform/rocBLAS/tree/develop/clients.
It will take me a bit of time since I'll need to rearrange the derivation, but sure.

I'll also have a gfx1102 available soon, so if this is too much effort, I can also take over ;)

I need to rearrange a lot of the rocm derivations to use a layout more like ginkgo in the linked PR.
Basically, I'll get to it. Just might take a while since rocBLAS is one of the later things in the toolchain.
By all means though, if you can get to it faster please do.

I would just build rocBLAS separately outside of nix, since I would assume if it's an issue, it's less likely to be a configuration issue, and more likely AMD not testing rocBLAS on all of their GPUs.

I would just build rocBLAS separately outside of nix, since I would assume if it's an issue, it's less likely to be a configuration issue

While likely not a nix issue, I'm not going to discount it's possibility. I can probably spin up a VM or something and do some A-B testing with each having a  rocBLAS installation to make sure.

and more likely AMD not testing rocBLAS extensively on all of their GPUs

I've packaged most of their software, and I can tell you that's very likely.


Export local backend middlewares in proxy server

Summary

For local_backend users with an existing express server already running, who would like to avoid having to run yet another server on another port to have some parts of the functionality work. With this change, it's possible to use the middleware directly, without creating a whole server in a new process:
const handler = require('netlify-cms-proxy-server/dist/middlewares').localFsMiddleware({
  repoPath: ...,
  logger: console,
})

handler(req, res)

👆 the above works both with express and in a next.js api handler.
Test plan
So, my plan was just to export it to begin with. My hope was that it could be an undocumented part of the API surface of this library and therefore subject to change without a major bump, but it would be useable for those who happen to know it's there.
Having it be an official part of the "real" API surface would be a comparatively change: there would need to be a separate entrypoint for the middleware module part of the library, which doesn't have the side-effect of trying to start a server (and it'd probably be a good idea to generate .d.ts files and write documentation, recommended usage, etc., and maybe get rid of the type dependency on winston in the params). And I can't see the harm in just doing this small change for now.
Checklist

[x] I have read the contribution guidelines.
[x] Code is formatted via running yarn format.
[x] Tests are passing via running yarn test.
[x] The status checks are successful (continuous integration). Those can be seen below.

A picture of a cute animal (not mandatory but encouraged)

(my parents' cat)

Hey @erezrokah, all very fair points. I was taking a shortcut to do the bare minimum possible to get it working with next.js - i.e. in a context where there is no app with .post or .use methods, so traditional middleware basically isn't supported. But yes it does mean there's no schema validation, cors, json-parsing or logging. Those were all ok to miss because respectively, the client is only sending valid requests, we happen to not need cors, next.js has already done json-parsing for us, and we are able to do our own logging.
I think I could probably make it work with the existing exports by passing in a fake express app from the handler, but that would be quite brittle.
An alternative - could the dist folder be generated using tsc rather than webpack? Given it's a node library/CLI I can't see why it would need to be bundled or minified. There'd be a few benefits:

users could import from dist/middlewares/whatever, at their own risk
type declarations would come for free
no need to maintain the webpack config anymore! (This is only a benefit because you already have a tsconfig, otherwise you'd be swapping one for the other)

Update: got it working with the existing exports by passing in a fake app. Behold the hoop-jumping:
export default async (req, res) => {
  const netlifyCmsProxyServer =
    require('netlify-cms-proxy-server/dist/middlewares') as typeof import('netlify-cms-proxy-server/src/middlewares')
  const path = require('path')
  const handlers: Function[] = []
  process.env.GIT_REPO_DIRECTORY = path.join(process.cwd(), '../..')

  const fakeApp = {
    use: () => {}, // skip next.js
    post: (_path: string, handler: Function) => handlers.push(handler),
  }

  netlifyCmsProxyServer.registerLocalFs(fakeApp as any, { logLevel: 'error' })

  for (const handler of handlers) {
    let nextCalled = false
    const next = () => (nextCalled = true)
    await handler(req, res, next)
    if (!nextCalled) {
      break
    }
  }
}

The above could easily break if the library changes some parts of its implementation, so I'd still be in favour of exporting the extra functions. But I might play around with a tsc build alternative instead.

Closing in favor of https://github.com/netlify/netlify-cms/pull/5856. I do believe using tsc is the right approach, given that we don't break existing consumers (we'll need to verify).


version 0.2.0 on npm: skip method absent

0.2.0 from npm seems to expose only elementPlaceholder but no skip method, version 0.2.0 here on git has skip
which is to be considered the latest?
if it's the one here on git how to declare it as an npm dependency?

Skip wasn't added until after the v2 tag. You can check everything in v2 here: https://github.com/google/incremental-dom/tree/0.2.0

I see, so being in the master this is the expected behaviour?
I's there a way i can explicitly require it from npm? if install @latest i'll get 0.2.0
The reason i'm asking is that i'm publishing a little jsonml to idom parser, and this is a beaking change as __placeholder passed as attribute seems not to work anymore. I got no problem refactoring as long as i can put the right dependency in the package.json.
You can take a look at the small library here:
https://github.com/paolocaminiti/incremental-dom-jsonml
Also once done it would be great to see it listed in the readme as possible idom approaches, is there anything specific i need to provide for it?

this is a beaking change as __placeholder passed as attribute seems not to work anymore

Can you provide an example? It should work just the same.

You are right, i double checked my code and the problem was generating the childNodes in the jsonml and not bypassing them explicitly inbetween the elementOpenEnd and elementClose, sorry for wasting your time.
Still i'm a bit puzzled:
elementOpenStart(...)
attr('__placeholder', true)
elementOpenEnd()
elementClose(...)

is equivalent to call elementPlaceholder?
which is equivalent to call skip in between open and close?

elementPlaceholder has an extra assert that makes sure it is used in a safe way that using the attribute directly doesn't. I never officially documented either since I wasn't quite sold on them yet. I think I might keep elementPlaceholder around, but the attribute method will probably go away in a future version.
The thing that skip buys you is that you can take a look at the element and then make your decision if you want to skip rather than needing to know beforehand (say by looking at the state of its parent). As far as modifying the tree goes, they all do the same thing.

Thanks for the detailed explanation, i like the explicit skip call method. I had like to change my code but I'm not sure how to define the npm dependency to get a version having it? Even declaring an " * " version in my package.json keeps fetching 0.2.0 while i'd like to get the updated version even if unstable. Am I doing something wrong or should I just wait?
Also is it ok to open issues on github for questions about the api or are there other preferred ways to seek help? I'm a bit uncertain about the relation between static attributes and the element's key as well.

Yeah, the current NPM workflow is kind of confusing, since the version in master doesn't really correspond to what is in NPM. You can use the github URL as the dependency, which can be useful if you are developing against the library as it develops.
I think if you have a question, it is fine to create an issue. It likely means that some part of the documentation or API itself isn't clear enough, which should be addressed.
For statics - currently the only way to make sure that they are always used correctly, you need to have an associated key. This is because the diff may in some situations re-use nodes and statics are only applied during node creation. For example, if you don't have a key, the following situation will result in incorrect attributes:
if (condition) {
  elementOpen('div', null, ['id', 'foo']);
  elementClose('div');
} else {
  elementOpen('div', null);
  elementClose('div');
}

Without a key if condition starts as true, you will end with

. If it then goes to false, the id attribute will still be present. I'm working on making the key not necessary to use statics, but for now you still need it.

Hi Sparhami, thanks for the details, folowing your adive I reafactored my library to use .skip directly and avoid assigning static attributes (for now).
I'm very happy with the current results sand believe this can be of interest to others willing to experiment with Incremental DOM: how can i list my repo into the README here so that interested people can discover it? Should I submit a pull request of the README containing the link and a snippet?


Automatic PR from RestAPI-PR2039

Generated from RestAPI PR: https://github.com/Azure/azure-rest-api-specs/pull/2039

Codecov Report

Merging #1642 into master will decrease coverage by 0.03%.
The diff coverage is 41.95%.

@@            Coverage Diff            @@
##           master   #1642      +/-   ##
=========================================
- Coverage   56.13%   56.1%   -0.04%
=========================================
  Files        3956    3959       +3
  Lines       94940   95033      +93
=========================================
+ Hits        53296   53318      +22
- Misses      41644   41715      +71

Impacted Files
Coverage Δ

azure-batch/azure/batch/models/task_add_result.py
100% <ø> (ø)
:arrow_up:

.../azure/batch/operations/compute_node_operations.py
70.37% <ø> (ø)
:arrow_up:

azure-batch/azure/batch/models/cloud_pool.py
100% <ø> (ø)
:arrow_up:

...h/azure/batch/models/output_file_upload_options.py
100% <ø> (ø)
:arrow_up:

...atch/azure/batch/models/auto_pool_specification.py
100% <ø> (ø)
:arrow_up:

...ure-batch/azure/batch/operations/job_operations.py
72.32% <ø> (ø)
:arrow_up:

...zure/batch/models/virtual_machine_configuration.py
100% <ø> (ø)
:arrow_up:

...atch/azure/batch/models/container_configuration.py
75% <ø> (ø)
:arrow_up:

...atch/azure/batch/models/auto_user_specification.py
100% <ø> (ø)
:arrow_up:

...batch/azure/batch/models/task_scheduling_policy.py
100% <ø> (ø)
:arrow_up:

... and 33 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 40fa36d...4673c16. Read the comment docs.

Broken PR


modify the /usr/bin/cron-send-docker-metrics path

get +1 from Drew
https://github.com/openshift/openshift-tools/pull/1536


Build related 0.43 tasks

Update Goreleaser with a new set of binaries for Win, MacOs and Linux (for now)
Finish the Docker image for the above
Snap? I assume we leave it as is for now. @anthonyfok
Talk to the Brew people.

Snap? I assume we leave it as is for now. @anthonyfok

I have just started tweaking snapcraft.yaml and experimenting with the extended-snap branch, which https://launchpad.net/~gohugoio/+snap/hugo-dev is now temporarily tracking and making test builds for me.  :-)

Re. snap, is it possible to deliver 2 versions? I assume there will be people wanting a "pure Go" Hugo version and does not need the SCSS/SASS.

Re. snap, is it possible to deliver 2 versions? I assume there will be people wanting a "pure Go" Hugo version and does not need the SCSS/SASS.

I think it is possible.  Would you like two separate executables in the same snap (easier to do), or two entirely separate snaps?
Assuming the first case, how would you like to name the executables?  How about hugo and hugo.unextended?

hugo and hugo.unextended?

... Unextended?
No, I was more thinking some "snap target" (I'm not sure what that is) ... like dev, extended.
If two binaries, then the default should be the one without change, ie. hugo and then hugo-extended.

Note that the binaries on the release page will be named "hugo" and "hugo_extended"; I think I want them to be named the same, but that will have to wait.

No, I was more thinking some "snap target" (I'm not sure what that is) ... like dev, extended.

Snaps can be released into 4 channels, namely "stable", "candidate", "beta" and "edge".  IIRC, hugo-dev is just a random name I picked for the build daemon (?) that tracks the hugo master branch for the most cutting-edge builds, and have them uploaded and published directly into the "edge" channel, but the name hugo-dev itself doesn't actually appear anywhere in the snap itself, and snap has no knowledge of that name.
So, to install what I named hugo-dev on Launchpad, the end user would actually use snap install hugo --edge, and not snap install hugo-dev.
While we can create an entirely new snap called hugo-extended, that seems a bit too complicated to me, especially we would have to maintain an extra branch with a special snapcraft.yaml that says name: hugo-extended on its the first line.
At least that is my (limited) understanding of it.

If two binaries, then the default should be the one without change, ie. hugo and then hugo-extended.

Since most end users do not compile Hugo from source and instead use whatever release binaries are provided to them, I and since Hugo with SCSS support is such a hot feature, I actually think that most people would prefer using the full Hugo (with extended build tag) and wouldn't worry too much about the enbedded C LibSass.  (i.e. most Hugo users do love Go but are not Go purists, or most likely don't even know how to write in Go code.)
For me, with my "end user" hat on, I would rather prefer having one single binary but with a runtime flag to disable the SCSS feature if I were to keep using gulp or something else to build my SCSS files.
But that's just me and my guess.  I could be wrong.  ;-)

Note that the binaries on the release page will be named "hugo" and "hugo_extended"; I think I want them to be named the same, but that will have to wait.

Is that due to Gorelease not yet being ready for them to be named the same?  Or for some other reasons?

Is that due to Gorelease not yet being ready for them to be named the same? Or for some other reasons?

I should have looked more carefully.  You've already opened #4916.  ;-)

I spent 8 hours yesterday to get Windows and Darwin builds from a Docker container, and today has also been a whole lot of other troubles, so I think I will postpone the Snap discussion.

Instead of hugo-extended can it be named just hugo+
Hugo Vs. Hugo-plus benchmark

About snaps: goreleaser can generate them as well, but it doesn't publish them for now... (there is an issue for that)
Once goreleaser can also publish, it would be straightforward to have 2 snaps IMHO...

I spent 8 hours yesterday to get Windows and Darwin builds from a Docker container, and today has also been a whole lot of other troubles, so I think I will postpone the Snap discussion.

I didn't quite realize the full complexity of the using-Docker-for-Windows-and-Darwin-builds when I first read your tweet, but after seeing what you wrote above, and starting to think about it... Wow, huge task indeed!  You are amazing as always!
Meanwhile, the Snap build with two binaries is a lot more trivial and seems to be working.  I'll make more tweeks and try to match the names hugo and hugo_extended as much as possible.

several hours later

Changes to snapcraft.yaml pushed to master in commit e1027c5846b48c4ad450f6cc27e2654c9e0dae39.
Due to snap's design (to avoid namespace collision), hugo.extended can be readily made accessible but not hugo_extended.  The latter will hopefully be available in a week or two, see the "automatic alias" request at https://forum.snapcraft.io/t/hugo-auto-alias-request-for-hugo-extended-hugo-extended/6297

@anthonyfok not sure I understand; In the main release we package it in several archives (one extended, one not), but with the same binary. A user wants one of them, not both. So we don't need (want?) different binary names, that would lead to confusion.

Does the user have to type "hugo_extended ...?"

Yes, with the current snapcraft.yaml that I just committed.
So, would you prefer that Hugo snap users use snap install hugo to install the pure-Go version, and use snap install hugo-extended to install the "extended" version, and in either case, the end users simply run hugo?
If that is the case, I think we'll need to have a separate snapcraft.yaml with the first line name: hugo changed to name: hugo-extended, and probably have that "extended" snapcraft.yaml live on another branch, say "master-extended-snap", and maybe "stable-extended-snap", and probably have these branches automatically rebased and updated from "master" and "stable" with some kind of automated git hook.  I could go and set up a new snap called "hugo-extended" now on the snapcraft dashboard web interface and pursue that route.

I'm into all sorts of "does not work" on CircleCI, which is my third day in a row with nothing but trouble ... I think this is a sign from above that I need a vacation ...

Hang in there!  Getting these CI things are very tricky indeed, not to mention such a complicated cross-platform setup with Docker!    We have full trust in you!
(If you wish, I'd be more than happy to help, though admittedly my experience with Circle CI and Docker are minimal.)

The CircleCI thing was my fault; I had slipped with some Git resets... I'm getting there...

Is the extended edition supposed to be a permanent solution? We're talking 2 MB extra, so I can't at the moment see the benefit of having to editions.
Do you see other features becoming exclusive to the extended edition in the future?

Just a final comment on this issue related to our Snap setup for the "extended" build of Hugo:
The Snapcraft experts recommend the use of tracks to us, which was unbeknownst to me.  (/me embarrassed):
@chipaca (John Lenton) wrote:

hmm, reading that issue, wouldn’t it make more sense to have a single hugo snap, with an extended track ?

@roadmr (Daniel Manrique) wrote:

Hi,
In principle this seems to fit the use case for tracks:
snap install hugo <- non-libSass version
snap install hugo --channel=extended  <- libSass version

Just be aware that you will need to publish the corresponding binaries to the matching track, otherwise > people tracking extended will not receive updates.
Also, each track has its own independent set of risks (stable, candidate, beta, edge), so this also works:
snap install hugo --channel=extended/beta
I’m +1 to giving a track for Hugo to maintain this extended version “in parallel” with the traditional one.
Per the process, there’s a voting period, at the end of which we’ll look at the votes from reviewers (mine included) and act on the track request. Thanks for your patience!

Daniel

I think this is also something that @bep has in his mind in the first place.  And, indeed, the Skype Snap is also using the track feature for its "insider" build.  No need for "two executables" or a separately named hugo-extended snap complexity.
As of today (2018-07-09), we are into the 3-day (or 7-day?) review/voting period for the creation of an extended track for the existing hugo snap.
I'll be tracking this personally over the next few days, and will build the snaps for both the default track and the "extended" track when it is ready.

About snaps: goreleaser can generate them as well, but it doesn't publish them for now... (there is an issue for that)
Once goreleaser can also publish, it would be straightforward to have 2 snaps IMHO...

FWIW, GoReleaser can now publish snaps: https://carlosbecker.com/posts/goreleaser-snap-travis/


wrangler build fails on compilation of rustc-serialize

Hello,
On a Mac running 10.14.2, I get the following when I run wrangler build in the folder generated by wrangler generate without changing anything:

Says it fails on the rustc-serialize function. I'll try to compile this module separately, and I'll file an issue on that dependency if the compilation issues are universal.

Please close and disregard-- turned out there was a dependency in my "clean" folder and that caused the problem!


no dispatch function defined for #{

It's a nice package but some bugs must be resolved for example in sbcl:
(with-c-syntax ()
#{
format (t, "Hello World!");
}#)
debugger invoked on a SB-INT:SIMPLE-READER-ERROR in thread
#<THREAD "main thread" RUNNING {240FE8C1}>:
no dispatch function defined for #{
File-Position: 103
Stream: #<SYNONYM-STREAM :SYMBOL SB-SYS:STDIN {22402939}>
Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.
restarts (invokable by number or by possibly-abbreviated name):
0: [ABORT] Exit debugger, returning to top level.
(SB-INT:SIMPLE-READER-ERROR #<SYNONYM-STREAM :SYMBOL SB-SYS:STDIN {22402939}> "no dispatch function defined for ~S" #{)
0]

Thank You.
I notice I forgot to enable reader-macros on the examples in README.
fixed in 86b4aba1cd080dd575d126b7013d2312c2d1e954

Thanks. This package is awesome really good work.


Archaius version update?

Hystrix is still tied to version 0.4.1 of the archaius library. Can we update to a newer version? I only noticed because that version doesn't correctly expose it's dependencies. In particular it doesn't list commons-lang in its depdencies which gave me an unexpected runtime error (below).
[ERROR][scheduler-3][2014-04-29 08:26:01,901] org.springframework.scheduling.support.TaskUtils$LoggingErrorHandler.handleError(TaskUtils.java:95) - Unexpected error occurred in scheduled task.
java.lang.NoClassDefFoundError: org/apache/commons/lang/text/StrLookup
at com.netflix.config.ConfigurationManager.getConfigInstance(ConfigurationManager.java:125)
at com.netflix.config.DynamicPropertyFactory.getInstance(DynamicPropertyFactory.java:263)
at com.netflix.config.DynamicProperty.getInstance(DynamicProperty.java:245)
at com.netflix.config.PropertyWrapper.<init>(PropertyWrapper.java:58)
at com.netflix.hystrix.strategy.properties.HystrixPropertiesChainedArchaiusProperty$DynamicBooleanProperty.<init>(HystrixPropertiesChainedArchaiusProperty.java:300)
at com.netflix.hystrix.HystrixCommandProperties.getProperty(HystrixCommandProperties.java:384)
at com.netflix.hystrix.HystrixCommandProperties.<init>(HystrixCommandProperties.java:116)
at com.netflix.hystrix.HystrixCommandProperties.<init>(HystrixCommandProperties.java:110)

From looking at public interfaces which deal with configuration, I believe a 2.0 release will be required.  I'll start outlining the required changes here.

archaius 0.4.1's dependency specifically on commons-configuration 1.8 gets it flagged by some security scans because of its optional dependency on commons-collections 3.2.1, which has popular security vulnerabilities.
Seems like this will get resolved only in a 2.x, but there's so little movement on the project lately.. should I stop holding my breath?


Add developer getting started section and content

@pchico83 I believe I've addressed all comments. Will add the changes to the "Deploy Your App" section to a later PR.


read dependencies from *.cabal

With etlas, I am able to build a large directory of pure Haskell code, not otherwise configured for use with Eta. In this case, etlas reads dependencies directly from a pre-existing .cabal file. On the other hand, I believe I can also build the project using gradle-eta so long as I specify all dependencies in build.gradle, e.g. compile eta('text:<IP_ADDRESS>').
This seems like duplicate work, and it will be easy for my package.yaml and build.gradle to become out of sync. In my case, I need to compile the same code via stack and gradle-eta or a custom build script. Can gradle-eta be configured to read from *.cabal?

The solutions suggested by @rahulmutt is to generate a *.gradle file based on the *.cabal; this could then be included in the main build.gradle. This would be a much better solution than redefining dependencies already in the Cabal file (when parts of a project need to remain buildable as pure Haskell).
If this is possible, how hard difficult would it be to detect the Cabal file and generate the *.gradle at build time, writing it to a standard location in the build directory and including it dynamically? I.e. is it possible to minimize or even eliminate extra configuration in build.gradle?
From Gitter:
Rahul Muttineni @rahulmutt 22:59
@joshsh If etlas generated a build.gradle for you from a .cabal file would that be helpful? And we could expose this as a Gradle task via something like ./gradlew generateEta -Pproject-type=gradle
Joshua Shinavier @joshsh Sep 08 23:20
@rahulmutt I think that would work well. The generated *.gradle would have to be included from the main build.gradle (which will have other config for the pure Java components of the project, if any). I think this can be done using the apply from syntax

Moving forward, I'm sure this will be pretty useful to a lot of people, but right now we have our hands full.
@jneira Do you think implementing etlas project --gradle command is something you'd be interested in working on since you've been doing a lot of work on transforming X -> GenericPackageDescription? This would be the inverse - GenericPackageDescription -> X.
If that's done, adding it to gradle-eta should be straightforward.

Not sure if i fully understand the solution,  but it seems the main info that gradle would need from the cabal file are the dependecies. The rest of the GenericPackageDescription info is not essential.
So maybe we could modify etlas deps to output the project dependencies (including eta packages and maven ones). Something like
> etals deps --gradle

compile ('org.bouncycastle:bcprov-jdk15on:1.59')
compile eta('base:[4.8,4.12[')
compile eta('dhall:[1.18.0,1.19[')
compile eta('text:[0.11.1,1.3[')

could it be that output usable in a easy way from gradle plugin?

@jneira Actually, the Gradle DSL supports more than just dependencies - you can specify pretty much everything you could in the .cabal file. Currently, gradle-eta only supports the most frequently used fields, but I'd be happy to support all the fields if this conversion command is implemented.
I imagine etlas project to be a generic querying command for information about the build and etlas project [component] --[format] will output the build component information in the particular format where format can be Gradle DSL, .etlas file, .cabal file, or etlas.dhall file. Will be extremely useful when moving between Haskell/Eta as well as between the different file formats (say you want to migrate from .etlas to etlas.dhall). This command will be incredibly useful in quickly migrating users to the dhall format if they're only had experience with the .cabal format. Given that you have done extensive work on supporting multiple formats, you probably have a better idea of the architecture of how to implement this nicely.

I see, it could be a more general solution to interoperate between formats, i'll try to take a look.
The hard part could be the GenericPackageDescription -> GradleDSL cause it is in pure java/groovy in the gradle plugin, right?
We should implement a builder in haskell for the gradle dsl format, or would be there another option?

@jneira I can handle the GenericPackageDescription -> GradleDSL bit, probably the first iteration will just be concatenating raw Text's. If you can setup the basic framework (basic implementation of the etlas project command), that'll be enough.


Homekit: Test bugfixes for py3.5

Checklist:

[x] The code change is tested and works locally.

Travis error not relate to the homekit component. Issue should be fixed.


Devices.yaml not reading display_name correctly.

When I try to run hyperglass build-ui or hyperglass run I get the error "hyperglass.exceptions.ConfigInvalid: The value field "network: display_name" is invalid: field required"

OS: Ubuntu 18.04
Python Version: 3.6.9

Attached is my devices.yaml

Hi there,
At first glance, I believe this is a syntax error, probably stemming from a documentation error :) Can you change your network definition from this:
routers:
  - name: core1
     network:
      - name: primary
         display_name: AS213242

To this:
routers:
  - name: core1
     network:
         name: primary
         display_name: AS213242

That - is making it a list/array, which is incorrect. I confirmed that’s how it is in the docs, but not how the code accepts it. Please let me know if that’s resolves the issue, and I’ll fix the docs.
Thanks!
-Matt

Yep that fixed it, thanks.


Improve react-components build system

https://www.pivotaltracker.com/story/show/187689236
This PR should have a reviewable diff, as GitHub seems not to show all the moved files. However, if it doesn't work for you, you could review it commit by commit, skipping the chore: move /react-components dir to /rails/react-components commit.
The main idea was to remove the built react-component.js file from the git repository and update the webpack configuration so webpack builds directly into the location in the Rails assets pipeline structure. This way, developers won't have to remember to build react-components before each commit, avoiding merge conflicts, and we can use webpack --watch for very fast incremental builds (10s -> ~1s or less).
@dougmartin , @scytacki, I am not very confident with the Docker setup, so I'd focus especially on changes to Dockerfile, Dockerfile-dev, and docker/dev/run.sh. I guess it is especially important to ensure that the react-components will be built when we release the staging/production Portal. 😉
To achieve this:

I added a modern Node version to our Docker image. Version 18.18 was the latest one I could make work with the Alpine Linux we use. It's sufficient, and I didn't want to spend more time on this, as I wasn't sure why we use that specific Alpine image version.
I removed the checked-in react-components.js and react-test-globals.js files, and added a git-ignored app/assets/javascripts/builds directory where webpack builds the react-components.
I moved react-components to rails/react-component so the source is available in the Docker image and can be built automatically.
I updated the Dockerfile and docker/dev/run.sh to build react-components automatically.
I updated GitHub CI to lint, build, and test react-components.
I removed the empty lint job from GitHub CI (unrelated to other work, but since I was there).

Now, a developer who works with Portal and simply runs docker-compose up doesn't need to do anything else if they are working on Rails code. The react-components will be built automatically at the beginning.
I perform the build just once in docker/dev/run.sh, without starting webpack --watch, as that would require using foreman to run two processes simultaneously -> rails server and webpack --watch. That's what jsbundling-rails does and it's not very complicated, but I think it’s more convenient for a developer working on react-components to open a new terminal and run webpack --watch separately from all the logs coming from Rails, SOLR, ElasticSearch, etc.
I haven't yet tried to extract our assets like CSS or images from the final JS bundle. This works as it used to – absolutely everything is inlined, so there's no regression. I'll try to work on that tomorrow. However, I wanted to open this PR first, as I hope this is a significant enough improvement and is less likely to break things than the CSS and images extraction.

Thanks @dougmartin, I changed npm install to npm ci, it makes sense. Another advantage over jsbundling-rails gem that uses npm install in its rake commands and we wouldn't be able to change it. :)


Shorten jaeger instance names in e2e tests

Resolves https://issues.redhat.com/browse/TRACING-965
Fixes smoke test failures on OCP make e2e-tests-smoke caused probably by https://github.com/jaegertracing/jaeger-operator/pull/891. Other related fixes https://github.com/jaegertracing/jaeger-operator/pull/904 and https://github.com/jaegertracing/jaeger-operator/pull/915
Signed-off-by: Pavol Loffay<EMAIL_ADDRESS>

This is the failing log
Get https://my-jaeger-with-ingress-osdk-e2e-22fc72a5-db25-4707-876c-173229d49b42.apps.02-18-1582017833.devcluster.openshift.com/api/services: dial tcp: lookup my-jaeger-with-ingress-osdk-e2e-22fc72a5-db25-4707-876c-173229d49b42.apps.02-18-1582017833.devcluster.openshift.com: no such host

The instance name is the shortest part of the URL. Truncate should be called on the final URL but we do not have access to it.
This PR fixes the issue though.

The instance name is the shortest part of the URL. Truncate should be called on the final URL but we do not have access to it.

The hostname is made of name and namespace of the route, and we can control the size of the name. This is what the actual code does to prevent creating an invalid route:
https://github.com/jaegertracing/jaeger-operator/blob/6b81f9d23b38396ba4401cf9f10a745672164da0/pkg/route/query.go#L37-L45

I had the same problem as I mentioned in https://github.com/jaegertracing/jaeger-operator/pull/904
This PR seems to fix the issue.

To @jpkrohling note, the test code correctly gets the route and uses the hostname from the route to construct the URL for smoke test.
The problem might by that shortened URL is ending with - which is causing the failure.

The problem might by that shortened URL is ending with - which is causing the failure.

There's a util.DNSName() that could be applied to the name after the util.Truncate() operation.