Running EDnCNN and MLPF: import error

Hi,
I'm facing an issue related to importing the pretrained models.
I downloaded the files and added them to ./modules/net. I also built the package again using
CC=gcc-10 CXX=g++-10 cmake .. -DEMLB_ENABLE_PYTHON=ON -DTORCH_DIR=/home/mohamadhalwani/anaconda3/envs/emlb/lib/python3.8/site-packages/torch/share/cmake/Torch
The output of the build was:

-- EMLB_ENABLE_PYTHON ON
-- EMLB_ENABLE_MODULES ON
-- EMLB_ENABLE_SAMPLES
-- EMLB_ENABLE_LIBTORCH OFF
CMake Warning at /usr/share/cmake-3.22/Modules/FindBoost.cmake:1369 (message):
  New Boost version may have incorrect or missing dependencies and imported
  targets
Call Stack (most recent call first):
  /usr/share/cmake-3.22/Modules/FindBoost.cmake:1492 (_Boost_COMPONENT_DEPENDENCIES)
  /usr/share/cmake-3.22/Modules/FindBoost.cmake:2102 (_Boost_MISSING_DEPENDENCIES)
  /usr/lib/x86_64-linux-gnu/cmake/dv-processing/dv-processing-config.cmake:29 (FIND_PACKAGE)
  /usr/lib/x86_64-linux-gnu/cmake/dv/dv-config.cmake:19 (FIND_PACKAGE)
  CMakeLists.txt:53 (find_package)

-- Linker flag --as-needed enabled
-- Project name is: cuke-emlb
-- Project version is: 1.0.0
-- Build type is: RELEASE
-- Compiler is Clang: FALSE
-- Compiler is GCC: TRUE
-- Compiler is IntelCC: FALSE
-- Compiler is MS VisualC: FALSE
-- OS is Unix: TRUE
-- OS is Linux: TRUE
-- OS is MacOS: FALSE
-- OS is Windows: FALSE
-- System is big-endian: 0
-- Thread support is PThreads: TRUE
-- Thread support is Win32 Threads: FALSE
-- Thread support libraries: -lpthread
-- Current C flags are:
-- Current CXX flags are:   -Ofast -march=native
-- Final install bindir is: /usr/local/bin
-- Final install libdir is: /usr/local/lib
-- Final install includedir is: /usr/local/include
CMake Warning at /usr/share/cmake-3.22/Modules/FindBoost.cmake:1369 (message):
  New Boost version may have incorrect or missing dependencies and imported
  targets
Call Stack (most recent call first):
  /usr/share/cmake-3.22/Modules/FindBoost.cmake:1492 (_Boost_COMPONENT_DEPENDENCIES)
  /usr/share/cmake-3.22/Modules/FindBoost.cmake:2102 (_Boost_MISSING_DEPENDENCIES)
  /usr/lib/x86_64-linux-gnu/cmake/dv/dv-config.cmake:22 (FIND_PACKAGE)
  CMakeLists.txt:53 (find_package)

CMake Warning at /usr/share/cmake-3.22/Modules/FindBoost.cmake:1369 (message):
  New Boost version may have incorrect or missing dependencies and imported
  targets
Call Stack (most recent call first):
  /usr/share/cmake-3.22/Modules/FindBoost.cmake:1492 (_Boost_COMPONENT_DEPENDENCIES)
  /usr/share/cmake-3.22/Modules/FindBoost.cmake:2102 (_Boost_MISSING_DEPENDENCIES)
  /usr/lib/x86_64-linux-gnu/cmake/dv/dv-config.cmake:22 (FIND_PACKAGE)
  CMakeLists.txt:53 (find_package)

-- Found Boost: /opt/inivation/boost/include (found suitable version "1.80.0", minimum required is "1.76") found components: nowide filesystem
-- Filesystem modules installation directory is: /usr/local/share/dv/modules
-- Actual modules installation directory is: /usr/local/share/dv/modules
-- DV_LIBRARIES set to: dv::sdk
-- Found Boost: /opt/inivation/boost/include (found version "1.80.0")
-- TOOLKIT_ENABLE_SAMPLES OFF
-- TOOLKIT_ENABLE_PYTHON OFF
CMake Warning at /usr/share/cmake-3.22/Modules/FindBoost.cmake:1369 (message):
  New Boost version may have incorrect or missing dependencies and imported
  targets
Call Stack (most recent call first):
  /usr/share/cmake-3.22/Modules/FindBoost.cmake:1492 (_Boost_COMPONENT_DEPENDENCIES)
  /usr/share/cmake-3.22/Modules/FindBoost.cmake:2102 (_Boost_MISSING_DEPENDENCIES)
  /usr/lib/x86_64-linux-gnu/cmake/dv-processing/dv-processing-config.cmake:29 (FIND_PACKAGE)
  external/dv-toolkit/CMakeLists.txt:28 (find_package)

-- Found Boost: /opt/inivation/boost/include (found suitable version "1.80.0", minimum required is "1.76") found components: nowide
-- pybind11 v2.11.1
-- Configuring done
-- Generating done
-- Build files have been written to: /home/mohamadhalwani/denoising/src/cuke-emlb/build

Then when trying to run the code, I'm getting an import error as follows:
edncnn     Traceback (most recent call last):
  File "denoiser.py", line 97, in <module>
    model = Denoisor(denoisor, resolution)
  File "/home/mohamadhalwani/denoising/src/cuke-emlb/configs/denoisors.py", line 142, in Denoisor
    return model(resolution, modified_params)
  File "/home/mohamadhalwani/denoising/src/cuke-emlb/configs/denoisors.py", line 48, in __init__
    from modules.python import event_denoise_convolution_network as module
ImportError: cannot import name 'event_denoise_convolution_network' from 'modules.python' (unknown location)

I can't find the module under modules/python.

I see some cmake warning, it seems that you not installed successfully so you may not find any module under modules/python. Maybe fix this issue first:
CMake Warning at /usr/share/cmake-3.22/Modules/FindBoost.cmake:1369 (message):
  New Boost version may have incorrect or missing dependencies and imported
  targets

Thanks for your prompt response.
I fixed the build issue by re-installing libboost
sudo apt-get install libboost1.76-all-dev
Now running the following gave me no warnings or errors.
CC=gcc-10 CXX=g++-10 cmake .. -DEMLB_ENABLE_PYTHON=ON -DTORCH_DIR=home/mohamadhalwani/anaconda3/envs/emlb/lib/python3.8/site-packages/torch/share/cmake/Torch
output:
-- EMLB_ENABLE_PYTHON ON
-- EMLB_ENABLE_MODULES ON
-- EMLB_ENABLE_SAMPLES
-- EMLB_ENABLE_LIBTORCH OFF
-- Linker flag --as-needed enabled
-- Project name is: cuke-emlb
-- Project version is: 1.0.0
-- Build type is: RELEASE
-- Compiler is Clang: FALSE
-- Compiler is GCC: TRUE
-- Compiler is IntelCC: FALSE
-- Compiler is MS VisualC: FALSE
-- OS is Unix: TRUE
-- OS is Linux: TRUE
-- OS is MacOS: FALSE
-- OS is Windows: FALSE
-- System is big-endian: 0
-- Thread support is PThreads: TRUE
-- Thread support is Win32 Threads: FALSE
-- Thread support libraries: -lpthread
-- Current C flags are:
-- Current CXX flags are:   -Ofast -march=native
-- Final install bindir is: /usr/local/bin
-- Final install libdir is: /usr/local/lib
-- Final install includedir is: /usr/local/include
-- Found Boost: /usr/local/include (found suitable version "1.76.0", minimum required is "1.76") found components: nowide filesystem
-- Filesystem modules installation directory is: /usr/local/share/dv/modules
-- Actual modules installation directory is: /usr/local/share/dv/modules
-- DV_LIBRARIES set to: dv::sdk
-- Found Boost: /usr/local/include (found version "1.76.0")
-- TOOLKIT_ENABLE_SAMPLES OFF
-- TOOLKIT_ENABLE_PYTHON OFF
-- Found Boost: /usr/local/include (found suitable version "1.76.0", minimum required is "1.76") found components: nowide
-- pybind11 v2.11.1
-- Configuring done
-- Generating done
-- Build files have been written to: /home/mohamadhalwani/denoising/src/cuke-emlb/build

Then I built the package using:
cmake --build . --config Release
The output
Consolidate compiler generated dependencies of target double_window_filter
[  4%] Building CXX object python/src/denoisors/CMakeFiles/double_window_filter.dir/double_window_filter.cpp.o
[  8%] Linking CXX shared module ../../../../modules/python/double_window_filter.cpython-38-x86_64-linux-gnu.so
lto-wrapper: warning: using serial compilation of 3 LTRANS jobs
[  8%] Built target double_window_filter
Consolidate compiler generated dependencies of target event_flow
[ 12%] Building CXX object python/src/denoisors/CMakeFiles/event_flow.dir/event_flow.cpp.o
[ 16%] Linking CXX shared module ../../../../modules/python/event_flow.cpython-38-x86_64-linux-gnu.so
lto-wrapper: warning: using serial compilation of 5 LTRANS jobs
[ 16%] Built target event_flow
Consolidate compiler generated dependencies of target khodamoradi_noise
[ 20%] Building CXX object python/src/denoisors/CMakeFiles/khodamoradi_noise.dir/khodamoradi_noise.cpp.o
[ 25%] Linking CXX shared module ../../../../modules/python/khodamoradi_noise.cpython-38-x86_64-linux-gnu.so
lto-wrapper: warning: using serial compilation of 3 LTRANS jobs
[ 25%] Built target khodamoradi_noise
Consolidate compiler generated dependencies of target reclusive_event_denoisor
[ 29%] Building CXX object python/src/denoisors/CMakeFiles/reclusive_event_denoisor.dir/reclusive_event_denoisor.cpp.o
[ 33%] Linking CXX shared module ../../../../modules/python/reclusive_event_denoisor.cpython-38-x86_64-linux-gnu.so
lto-wrapper: warning: using serial compilation of 3 LTRANS jobs
[ 33%] Built target reclusive_event_denoisor
Consolidate compiler generated dependencies of target time_surface
[ 37%] Building CXX object python/src/denoisors/CMakeFiles/time_surface.dir/time_surface.cpp.o
[ 41%] Linking CXX shared module ../../../../modules/python/time_surface.cpython-38-x86_64-linux-gnu.so
lto-wrapper: warning: using serial compilation of 3 LTRANS jobs
[ 41%] Built target time_surface
Consolidate compiler generated dependencies of target yang_noise
[ 45%] Building CXX object python/src/denoisors/CMakeFiles/yang_noise.dir/yang_noise.cpp.o
[ 50%] Linking CXX shared module ../../../../modules/python/yang_noise.cpython-38-x86_64-linux-gnu.so
lto-wrapper: warning: using serial compilation of 3 LTRANS jobs
[ 50%] Built target yang_noise
Consolidate compiler generated dependencies of target m_double_window_filter
[ 54%] Building CXX object src/denoisors/CMakeFiles/m_double_window_filter.dir/double_window_filter.cpp.o
[ 58%] Linking CXX shared library ../../../modules/Km_double_window_filter.so
[ 58%] Built target m_double_window_filter
Consolidate compiler generated dependencies of target m_event_flow
[ 62%] Building CXX object src/denoisors/CMakeFiles/m_event_flow.dir/event_flow.cpp.o
[ 66%] Linking CXX shared library ../../../modules/Km_event_flow.so
[ 66%] Built target m_event_flow
Consolidate compiler generated dependencies of target m_khodamoradi_noise
[ 70%] Building CXX object src/denoisors/CMakeFiles/m_khodamoradi_noise.dir/khodamoradi_noise.cpp.o
[ 75%] Linking CXX shared library ../../../modules/Km_khodamoradi_noise.so
[ 75%] Built target m_khodamoradi_noise
Consolidate compiler generated dependencies of target m_reclusive_event_denoisor
[ 79%] Building CXX object src/denoisors/CMakeFiles/m_reclusive_event_denoisor.dir/reclusive_event_denoisor.cpp.o
[ 83%] Linking CXX shared library ../../../modules/Km_reclusive_event_denoisor.so
[ 83%] Built target m_reclusive_event_denoisor
Consolidate compiler generated dependencies of target m_time_surface
[ 87%] Building CXX object src/denoisors/CMakeFiles/m_time_surface.dir/time_surface.cpp.o
[ 91%] Linking CXX shared library ../../../modules/Km_time_surface.so
[ 91%] Built target m_time_surface
Consolidate compiler generated dependencies of target m_yang_noise
[ 95%] Building CXX object src/denoisors/CMakeFiles/m_yang_noise.dir/yang_noise.cpp.o
[100%] Linking CXX shared library ../../../modules/Km_yang_noise.so
[100%] Built target m_yang_noise

Am I doing something wrong ? because I can't see EDnCNN or MLPF during the building process

Great, It's almost done. I notice that the -DTORCH_DIR you use makes a mistake. You need to download libtorch rather than pytorch at https://pytorch.org/, where the former one is designed for C++. Once you download, just unzip it and no installation required. Also pay attention to downloading the correct CUDA version.

I did this step already, but for some reason I'm getting the following warning. Even though I'm sure of the path for libtorch:
(emlb) mohamadhalwani@mohamadhalwani-ASUS-TUF-Gaming-F15-FX506HM-FX506HM:~/denoising/src/cuke-emlb/build$ CC=gcc-10 CXX=g++-10 cmake .. -DEMLB_ENABLE_PYTHON=ON -DTORCH_DIR=home/mohamadhalwani/anaconda3/envs/emlb/lib/python3.8/site-packages/libtorch/share/cmake/Torch
-- EMLB_ENABLE_PYTHON ON
-- EMLB_ENABLE_MODULES ON
-- EMLB_ENABLE_SAMPLES
-- EMLB_ENABLE_LIBTORCH OFF
-- Linker flag --as-needed enabled
-- Project name is: cuke-emlb
-- Project version is: 1.0.0
-- Build type is: RELEASE
-- Compiler is Clang: FALSE
-- Compiler is GCC: TRUE
-- Compiler is IntelCC: FALSE
-- Compiler is MS VisualC: FALSE
-- OS is Unix: TRUE
-- OS is Linux: TRUE
-- OS is MacOS: FALSE
-- OS is Windows: FALSE
-- System is big-endian: 0
-- Thread support is PThreads: TRUE
-- Thread support is Win32 Threads: FALSE
-- Thread support libraries: -lpthread
-- Current C flags are:
-- Current CXX flags are:   -Ofast -march=native
-- Final install bindir is: /usr/local/bin
-- Final install libdir is: /usr/local/lib
-- Final install includedir is: /usr/local/include
-- Found Boost: /usr/local/include (found suitable version "1.76.0", minimum required is "1.76") found components: nowide filesystem
-- Filesystem modules installation directory is: /usr/local/share/dv/modules
-- Actual modules installation directory is: /usr/local/share/dv/modules
-- DV_LIBRARIES set to: dv::sdk
-- Found Boost: /usr/local/include (found version "1.76.0")
-- TOOLKIT_ENABLE_SAMPLES OFF
-- TOOLKIT_ENABLE_PYTHON OFF
-- Found Boost: /usr/local/include (found suitable version "1.76.0", minimum required is "1.76") found components: nowide
-- pybind11 v2.11.1
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    TORCH_DIR

-- Build files have been written to: /home/mohamadhalwani/denoising/src/cuke-emlb/build

How about rm -rf ~/denoising/src/cuke-emlb/build and redo it again?

Still facing an issue with Torch!
CMake Error at CMakeLists.txt:73 (find_package):
  By not providing "FindTorch.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "Torch", but
  CMake did not find one.

  Could not find a package configuration file provided by "Torch" with any of
  the following names:

    TorchConfig.cmake
    torch-config.cmake

  Add the installation prefix of "Torch" to CMAKE_PREFIX_PATH or set
  "Torch_DIR" to a directory containing one of the above files.  If "Torch"
  provides a separate development package or SDK, be sure it has been
  installed.

Considering this situation, maybe you need to set environment variables first in terminal if you have successfully installed CUDA.
export CUDA_HOME=/usr/local/cuda
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH

I had some issues in the compatibility of my GPU with the Cuda at first bur finally its working now.
Final versions that worked for me are (GPU: RTX 3060 on Ubunutu 20):
Cudatoolkit 11.8
torch version: 2.4.1
torch_version_cuda 11.8
Thanks a lot


Consider introducing a Unique constraint

This would allow enforcement of set based uniqueness across domain objects.
I propose that a user creates a UniqueAttribute and decorates any AggregateRoot property with it upon which they want to enforce uniqueness. The persistence layer would the use the same mechanism that is used for NaturalKey uniqueness to ensure that the value is unique across aggregates.
This would fix the "username" problem.

Something like a UniqueKeyViolationException inheriting from PersistenceException should be thrown. But maybe named a bit better.


github-release raising 404 Client Error: Not Found for url on powershell

Hello, when I use scriv github-release on powershell I get this output:
scriv github-release --verbosity debug
debug: Running command 'git remote -v'
debug: Command exited with 0 status. Output: 'origin\thttps://github.com/henriquelino/timer.git (fetch)\norigin\thttps://github.com/henriquelino/timer.git (push)\n'
debug: Running command 'git tag'
debug: Command exited with 0 status. Output: 'v0.1.0\nv<IP_ADDRESS>\nv<IP_ADDRESS>\nv<IP_ADDRESS>\nv<IP_ADDRESS>\n'
debug: Starting new HTTPS connection (1): api.github.com:443
debug: https://api.github.com:443 "GET /repos/henriquelino/timer/releases HTTP/1.1" 301 159
debug: https://api.github.com:443 "GET /repositories/577091747/releases HTTP/1.1" 200 995
Creating release v<IP_ADDRESS>
debug: Starting new HTTPS connection (1): api.github.com:443
debug: https://api.github.com:443 "POST /repos/henriquelino/timer/releases HTTP/1.1" 307 159
debug: https://api.github.com:443 "POST /repositories/577091747/releases HTTP/1.1" 404 None
error: text: '{"message":"Not Found","documentation_url":"https://docs.github.com/rest/reference/repos#create-a-release"}'
Traceback (most recent call last):
  File "C:\Users\henri\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\henri\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "C:\Users\henri\AppData\Local\Programs\Python\Python39\Scripts\scriv.exe\__main__.py", line 7, in <module>
  File "C:\Users\henri\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\henri\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "C:\Users\henri\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\henri\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\henri\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\henri\AppData\Local\Programs\Python\Python39\lib\site-packages\scriv\ghrel.py", line 67, in github_release
    create_release(repo, release_data)
  File "C:\Users\henri\AppData\Local\Programs\Python\Python39\lib\site-packages\scriv\github.py", line 88, in create_release
    check_ok(resp)
  File "C:\Users\henri\AppData\Local\Programs\Python\Python39\lib\site-packages\scriv\github.py", line 24, in check_ok
    resp.raise_for_status()
  File "C:\Users\henri\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://api.github.com/repositories/577091747/releases

I couldn't find anything related neither could trace why this happens on powershell but not on git bash.
I am on Windows 10, python 3.9 and scriv 1.0.0, when running on git bash it works fine.

Interesting! Since it is failing when trying to create the release, I will guess that on git bash, you have credentials in a GITHUB_TOKEN environment variable, but under Powershell you don't.  Is that possible?

Yep that's exactly what was happening, I ran gci env:* | sort-object name and github token wasn't there, then I closed powershell and open again, then ran the command again, poof, github_token now shows and scriv github-release is working, thanks for the insight!
It seems that after changing system environments I should've restarted powershell, so it recognizes the new var.


Carlos Test Creation & Organization

Created test files for each of the backend python files.
Organized backend files into a better structure
Implemented create_prediction_data() in read_data.py (Eric please check this out)

This is what I think create_prediction_data should look like: the role of read_data is just calling the methods and transfer them to app.py. It doesn't do filtering by itself. Could you please write a filter function in a new file? The new filtwer function should input spefici "filters" (such as sex, race, etc) and output the data already filtered. My design idea is that the model is re-run for each filter as the "original dataset" is different under each filter. Here's my modified read_data.py:
`import pandas as pd
import os
from Model import TARGET_VAR, Model
Read the raw dataset from the csv
def read_dataset():
# Temporary
# return {"data": "yes"}
return pd.read_csv(os.path.join(os.path.dirname(file), "data", "transaction_data.csv"))
def create_model(filter):
raw_data = read_dataset()
# filtered_data = filter(raw_data, filtering_factor)  # to be checked with Carlos
filtered_data = raw_data  # temporary
return Model(filtered_data)
Creates the data in the format that VISX needs
def create_formatted_data(filter):
cleaned_data = create_model(filter).get_cleaned_data()
formatted_data = [
{'date': record['Date'].strftime('%Y-%m-%d'), 'value': record[TARGET_VAR]}
for record in cleaned_data[['Date', TARGET_VAR]].drop_duplicates().to_dict(orient='records')
]
return formatted_data
def create_prediction_data(filter, forecast_steps: int):
model = create_model(filter)
forecast_df = model.get_forecast()
forecast_values = forecast_df['mean']
future_dates = pd.date_range(start=model.get_cleaned_data().index[-1] + pd.DateOffset(months=0), periods=forecast_steps, freq='D')
prediction_unformatted =  [{'date': date, 'value': value}
for date, value in zip(future_dates, forecast_values)]
prediction_data = [{'date': record['date'].strftime('%Y-%m-%d'), 'value': int(record['value'])} for record in prediction_unformatted]
return prediction_data
print(create_formatted_data('A'))`
If you have a different design idea plz lmk.
Also, I'll push my changes after you implement the filter, so when you complete implementing the filter plz also lmk.

I wouldn't recommend to write "from backend.Model import something" cuz we shouldn't do this.

I also remember seeing something about it being not recommended. However, Carlos has already made the init.py files to turn all our folders into packages, so just adding the backend.* now works from anywhere within our python environment (*I believe). If you just test it out locally after adding the backend. imports  everything works so I don't see it as a big issue.

For now, I would suggest trying to use absolute path.
See my conversation with ChatGPT for more instructions:
https://chatgpt.com/share/6725bb6b-e600-8011-9a99-c14104e38493

If you think absolute path is a more futureproof implementation then sounds good. Matthew and I tried to get it to work a couple days ago when we were having the Heroku issues and it didnt seem to work.

Also this really should be split into two PR's one about the model changes and one about the backend/backend_test changes

We should decide on one way of doing these imports that we can keep consistent for the entire backend.
I don't think I understand fully what's happening in create_prediction_data(). Maybe you can explain it tdy on call.
Also the backend build is failing from import statements.

I wouldn't recommend to write "from backend.Model import something" cuz we shouldn't do this. I doubt there were issues with CI/CD and have asked Alex and Tara. For now, I would suggest trying to use absolute path.
See my conversation with ChatGPT for more instructions:
https://chatgpt.com/share/6725bb6b-e600-8011-9a99-c14104e38493
I also had a long chat with GPT last night about this whole import mess. Still trying to figure it out. I attempted using an absolute path but it still wasn't quite working. I'm debating just returning the file organization to how it was before.

For create_prediction_data, I don't think we should implement the filtering logic at this stage. Could you add a new file and implement the filtering logic? I think for read_data, its only job is to call functions from elsewhere [let's discuss with Matthew on this later].
@Adventurer-E The reason I implemented the filter function in read_data was because that file also contains a function to read the dataset and another to format it, so it seemed logical that a function to filter data (which is also basically formatting it) would also go here. It would be great if you could hop on a call later today to discuss each of these files (app, Model, read_data) because it is unclear to me what each file is supposed to contain/do. I disagree that we should create a separate file for the data filtering, after all it's not a huge function. It should exist in whatever file we assign to data manipulation/formatting (whether that be read_data or something else). The reason I implemented it in create_prediction_data was because of how I interpreted the comments on the file. I see what you want to do with that function now, and it looks good. Maybe create_prediction_data should be in a file on its own though (let's talk about this today).


make quick-start fails / whisk controller not coming up

On my Ubuntu-VM (hosted by Win10-Virtualbox) using "make quick-start" hangs at the start of the whisk-controller.
The controller log says: Exception in thread "main" pureconfig.error.ConfigReaderException: Cannot convert configuration to a whisk.core.loadBalancer.ShardingContainerPoolBalancerConfig. Failures are:
at 'whisk.loadbalancer':
- Key not found: 'invoker-user-memory'.
at pureconfig.package$.getResultOrThrow(package.scala:138)
at pureconfig.package$.loadConfigOrThrow(package.scala:160)
at whisk.core.loadBalancer.ShardingContainerPoolBalancer.<init>(ShardingContainerPoolBalancer.scala:159)
at whisk.core.loadBalancer.ShardingContainerPoolBalancer$.instance(ShardingContainerPoolBalancer.scala:437)
at whisk.core.controller.Controller.<init>(Controller.scala:117)
at whisk.core.controller.Controller$.main(Controller.scala:258)
at whisk.core.controller.Controller.main(Controller.scala)

[2018-08-26T16:45:36.615Z] [INFO] Cluster Node [akka.tcp://controller-actor-system@controller:2551] - Exiting completed
[2018-08-26T16:45:36.615Z] [INFO] Cluster Node [akka.tcp://controller-actor-system@controller:2551] - Shutting down...
[INFO] [08/26/2018 16:45:36.642] [kamon-akka.actor.default-dispatcher-10] [akka://kamon/user/metrics] Message [kamon.metric.SubscriptionsDispatcher$Tick$] without sender to Actor[akka://kamon/user/metrics#466360108] was not delivered. [1] dead letters encountered.
[ERROR] [08/26/2018 16:45:36.884] [controller-actor-system-dispatchers.kafka-dispatcher-20] [Controller$(akka://controller-actor-system)] [#tid_sid_unknown] [KafkaConsumerConnector] poll timeout occurred. Retrying 3 more times.
"
(Setting DOCKER_HOST_IP as in another ticket resulted in an error message, which comes up wayearlier: "Recreating openwhisk_apigateway_1    ...
"docker inspect" requires at least 1 argument.
See 'docker inspect --help'.
Usage:  docker inspect [OPTIONS] NAME|ID [NAME|ID...] [flags]
Recreating openwhisk_apigateway_1    ... done
"docker inspect" requires at least 1 argument.
See 'docker inspect --help'.
Usage:  docker inspect [OPTIONS] NAME|ID [NAME|ID...] [flags]
Return low-level information on Docker objects
invalid argument "db:" for "--add-host" flag: invalid IP address in add-host: ""
See 'docker run --help'.
Makefile:235: recipe for target 'init-couchdb' failed
make: *** [init-couchdb] Error 125
")

Same here with different working environments.

Mac OS High Sierra
Linux 18.04 LTS VM running on Azure infrastructure.

Logs in tmp/openwhisk/controller/logs/controller-local_logs.log
`Exception in thread "main" pureconfig.error.ConfigReaderException: Cannot convert configuration to a whisk.core.loadBalancer.ShardingContainerPoolBalancerConfig. Failures are:
at 'whisk.loadbalancer':
- Key not found: 'invoker-user-memory'.
at pureconfig.package$.getResultOrThrow(package.scala:138)
at pureconfig.package$.loadConfigOrThrow(package.scala:160)
at whisk.core.loadBalancer.ShardingContainerPoolBalancer.<init>(ShardingContainerPoolBalancer.scala:159)
at whisk.core.loadBalancer.ShardingContainerPoolBalancer$.instance(ShardingContainerPoolBalancer.scala:437)
at whisk.core.controller.Controller.<init>(Controller.scala:117)
at whisk.core.controller.Controller$.main(Controller.scala:258)
at whisk.core.controller.Controller.main(Controller.scala)`

Same problem that #151 is working on fixing.  Need to adapt to an upstream change in the arguments to controller/invoker.

just merged #151 .

@dakoller , @paolasoto88 I'll leave this ticket open so you can confirm it's working fine for you as well.

Great work guys, now the whisk controller is back again. I just pulled recent changes and it worked in both environments. Thanks!

@dakoller can you try pulling the latests code and seeing if it works for you locally?

...it works now..thx for the help!


Why is there no integrated uptick chain?

It is already added to ibc-transfer UI
http://gon.disperze.com/ibc-transfer
I forgot to push latest changes. but now it's done


Add unit/acceptance tests and update README

This adds working unit tests and an acceptance test in beaker to test
the dynect provider.  The README is also updated with examples and
better descriptions of about everything.
After this is merged, I'll be able to look into changing the dynect
provider more easily to use the (new) fog gem for dynect instead of the
manual hacks I use to only implement a subset of DNS records.
https://travis-ci.org/charlesdunbar/puppetlabs-dns/builds/101127203

Looks like it may be worth adding .swp to .gitignore to catch spec/acceptance/.dynect_dns_record_spec.rb.swp.
Otherwise, +1.

Or I should close that file in vim before committing code.  The joy of panels.

+1


Tracking individual - finalized

Finalized ability to control individual log statements and have different controls for different log files

Sorry, I haven't been on top of github stuff, to put it mildly.
Looking at this PR, I see you're adding a separate library, not modifying any existing functionality.
I don't see the need to include it into async: it seems better if you just publish it as a separate library on OPAM.


Releases Freestyle RPC 0.13.2

This release mainly brings support for AvroWithSchema serialization type. See changelog/release tag for deeper information.

👍


Nonce usage while signing

As described here, sig_data should be created by first prepending a 64 bit value to the message, i.e. 64bit_integer_from_unix_epoch. What is the need of this 64 bit value? It looks like you want indeterministic signatures but why?

This RFC was a work in progress that we're opting to deprecate in favor of using JWSs. I just haven't gotten around to updating stuff yet.

@kdenhartog - can this be closed?  Looks like the two of you agree.

yup. thanks for checking I'll close it now.


Output Variables not available as $(variables)

TFS 2015 Update 3.
AzureResourceGroupDeployment Task 1.0.62
Output Variables here it claims that all Resource Template output variables are now created as variables for later use.
My template has 2 output variables, I can see them in deployment history in Azure Portal. However the variables are not being set in my Release.
"outputs": {
        "domainname": {
            "type": "string",
            "value": "[variables('hostDNSName')]"
        },
        "storageAccount_UniqueName": {
            "type": "string",
            "value": "[variables('storageAccount_UniqueName')]"
        }
    },

I was previously (TFS 2015 Update 2) using this powershell to collect the variables as the task didn't support. I have disabled this script as it shouldn't be necessary.
$outputs = (Get-AzureRmResourceGroupDeployment -ResourceGroupName $resourceGroup -DeploymentName $resourceGroup | Select -Last 1).Outputs
$outputs.Keys | % { Write-Host ("##vso[task.setvariable variable="+$_+";]"+$outputs[$_].Value) }

However even if I do enable this script it doesn't work because the Deployment name now changes on each deployment and I don't know how to get it. Previously the deployment name was the same as the resource group name, now it's something like template-20160628-0254

I might be mistaken here, and it could be a different problem that i'm having.

Actually, this is a real problem.

Do you require more information to help diagnose this issue?

@AHoran

Can you modify your script to get things running?
DeploymentName is optional in Get-AzureRmResourceGroupDeployment. We can get all the deployments and then get the latest.
Or Get the resource group by name and fetch the properties you are interested in.

I am sorry that the documentation caused confusion for you. The output variable in the documentation doesn't refer to those in the template. It refers to the Resource Group variable in the Output section of the task. This exposes only IP Address and FQDNs. It doesn't expose the output variables from template.

We will see if a task enhancement is needed to support exposing output variables from template in the future.

I had already updated my script as per your recommendation in point 1.
Regarding point 2. I believe that creating variables for all template output variables would be very useful. As you can guess from my example I am creating a storage account with a unique name within the ARM script, and the best way to discover this name and to use it in the rest of the script is via the template output section. I'm sure there are many other use cases related.

@AHoran Thanks for the feedback
@RoopeshNair to consider/comment on the enhancements

@AHoran,
We have added feature to our backlog wherein CSM output  can be set as Task out put variables. However, it will be a while before we can get to it.


Support creating a client with a custom tls connector

This PR would allow a user to create a client by supplying a native-tls::TlsConnector, rather than just being able to specify either default-tls or rustls.

Thanks for the PR! Do you have some specific case that can't be solved by the existing config options?
I'm wary of exposing such an API as it means that upgrade to a new version becomes a breaking change in reqwest.

For example, setting use_sni or min_protocol_version or max_protocol_version for the native_tls connector. Also I will be working on finishing the ability to specify identities with PKCS8 keys and cert chains, and that and any other changes in native_tls would require new settings here. It seems simpler to me to just pass in the already configured TlsConnector.

I'd also like this feature. For example, I intend to add more advanced certificate validation options to native-tls and this API will probably be a lot bigger than a simple toggle function on the builder. It seems silly to have to add forwarding functions in reqwest for every builder option--however complex they are--that might ever be added in native-tls.

The issue about not being able to upgrade versions internally seems important to me...
Since reqwest can potentially use different TLS backends, it seems reasonable to me to provide TLS options as needed on the ClientBuilder.

Not all configuration options are going to be available on all backends. Clearly, if someone wants to put in their own builder, they've already decided on a particular TLS backend. It's good to have sane & safe defaults for most users, but reqwest has all kinds of functionality that's useful to TLS power users too ((de)serialization, proxy support, etc.).
I think these are the options:

Commit to wrapping all possible TLS backend configuration options.

How would this work when different backends support different things?

Support only a subset of TLS configuration options

This forces people to not use reqwest or use a fork of reqwest.

Make native-tls part of the public API.

You would be able to upgrade native-tls with a major version bump of reqwest.
You could make a crate reqwest-native-tls that publicly depends on native-tls. reqwest would depend on that crate and wrap everything, except for the public native-tls part of the API, which is not exposed. If you want to bump native-tls you do a new (major) version bump of reqwest-native-tls and you can change to this version in reqwest. Users who want to depend on native-tls directly can use reqwest-native-tls. If there's a major version bump to reqwest-native-tls, they would need to update their use of native-tls to the new version anyway, so there's not a lot of extra effort needed for them to update to the new version.
You might be able to build a scheme similar to the previous point using Cargo features.

Abstract over streams using something like hyper::client::connect::Connect.

It's not clear to me that this is going to be possible with dynamic dispatch, which is probably what you want here so you don't have to make everything generic.

As I was curious how curl handles this, they have a CURLOPT_SSL_CTX_FUNCTION option of a callback receiving a void *, and requires the user to unsafely fiddle with the backend. We could provide something similar-ish, though a little safer.
An advanced option passing impl Fn(&mut dyn Any) could be used, and then you can use safe downcasting to fiddle all the toggles...

Do you need a callback? You could also do something like fn builder() -> &mut dyn Any. It's not clear to me this is a semver improvement. While this doesn't break compilation when you change the backend TLS version, it does probably stop code from working at run-time.

As an advanced config option, it allows users to decide what to do if the backend isn't what you wanted. Maybe you can set the advanced options in multiple backends. Maybe the advanced option wasn't critical, just a nice-to-have, and so you do nothing. Or you could choose to just panic...

I think your proposal sounds very similar to this:
impl ClientBuilder {
    /// Returns `false` if `reqwest` doesn't know how to use the provided `connector`.
    fn with_preconfigured_connector(&mut self, mut connector: impl Any) -> bool {
        if let Some(conn) = connector.downcast_mut::<NativeTlsConnector>() {
            self.connector = NativeTls(std::mem::replace(conn, Default::default()))
        } else if ... {
            ...
        } else {
            return false
        }
        true
    }
}

I'd think I'd prefer a compilation error over something that will panic at runtime? I guess one could test for this if they needed it to work. But then I feel that should be clearly documented.

I have updated the PR to reflect Jethro's suggestion, and although it compiles I have not tested the new functionality, it is mostly just a proof of concept at this point.

fn with_preconfigured_connector(&mut self, mut connector: impl Any) -> bool {

Ah, I didn't notice this was inverted to what I was thinking. My initial thoughts were to return &mut dyn Any of the internal connector, so a user could inspect it for multiple different types if need be...

I continued this in https://github.com/seanmonstar/reqwest/pull/606 with some changes and tests.

I'm going to close in favor of #606, though let me know if #606 does or doesn't fit your needs. Thanks for working through the design and rationale here!


Support  with debian/ubuntu builds

GitHub Actions has enabled multi arch support  for Ubuntu causing TileDB builds to fail. This PR allows for  on debian/ubuntu if enabled by passing CMAKE_INSTALL_LIBDIR to ExternalProjects. Also, awssdk was not passing the CMAKE_INSTALL_LIBDIR to its C dependencies, so installing all the awssdk C dependencies explicitly as ExternalProjects.

Codecov Report

Merging #116 (a7a3951) into develop (3c4c0f9) will decrease coverage by 1.04%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop     #116      +/-   ##
===========================================
- Coverage    63.16%   62.12%   -1.05%
===========================================
  Files           59       59
  Lines        17875    17875
===========================================
- Hits         11291    11104     -187
- Misses        6584     6771     +187

Impacted Files
Coverage Δ

core/src/storage_manager/storage_gcs.cc
5.95% <0.00%> (-71.83%)
:arrow_down:

core/include/storage_manager/storage_gcs.h
0.00% <0.00%> (-40.00%)
:arrow_down:

core/src/storage_manager/storage_manager_config.cc
57.83% <0.00%> (-1.21%)
:arrow_down:

core/src/storage_manager/storage_hdfs.cc
64.81% <0.00%> (-0.56%)
:arrow_down:

core/src/misc/utils.cc
76.32% <0.00%> (-0.14%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3c4c0f9...a7a3951. Read the comment docs.


Progress Bar may consume unnecessary memory

Recently cmdlets used WriteProgress() (ex., web cmdlets) can potentially create hundreds of thousands of  ProgressRecord objects.
It looks like undue load on GC.

After Progress Bar performance improvements in #2822 many ProgressRecord objects are simply ignored. It would be great not to create a new object each time, and reuse one.
I suppose it can be done in AddNode() here


Fix cross-platform file type autodetection

Closes #727
Contributor Checklist:

[x] I have added or updated my entry in the creators.json file
[x] I have added a changelog entry for my contribution
[x] I have added/updated documentation for all user-facing changes
[ ] I have added/updated test cases

It seems that this doesn't fix the issue on Python 3.6.6.

Confirmed: The issue appears on Python 3.6.6, but not on Python 3.7

Looks good, merging, thanks!


Expand readme description

Thanks for creating the addon!
I think it would be helpful to expand the readme with the following information

what does ahoy do
when would you use it
give some example of how you use

This issue is just a placeholder / suggestion. Feel free to ignore / close if you wish.

@tyler36 thanks! All of that is really a copy paste from the ahoy cli homepage, which is linked. Will leave it for now and try to come up with something to add, but it's a very small cli binary :) Sometimes there's not a lot to say. I can say, however, why I like it :)

Closing to tidy up.


ROADMAP: Build slim and full Falco container images

Currently the Falco image contains build tools to automatically build the kernel module. In order to allow users to run without the build tools, we should instead build a minimal image.  In the slim image the kernel module is provided via HTTPs, or a mounted volume. The full image would behave as today with the ability to compile the kernel module at container start.

Greetings,
I discussed this with @mstemm and I would like to write down the outcome:

Create a Kubernetes initContainer that runs the image that builds the kernel module
Mount the built kernel module in the light Falco container

Separating build tools from the Falco agent will also remove vulnerabilities that those build tools adds in the current falcosecurity/falco Docker image. Because of those build tools, snyk test --docker falcosecurity/falco:latest currently finds 3 "High" vulnerabilities that we can get rid off :)
If you need some hands to develop this, poke me!

@JPLachance Not sure if you have time to work on this, but your help would be appreciated.
I'd also like to create a mutli-stage Dockerfile that builds Falco in one stage and then installs it in the second stage. Not sure how much work that is but it would make it easier to test developer builds in minikube, etc.

@mfdii building the kernel module in one stage and use it in finale one is exactly what I'm doing in my Dockerfile for building a falco image which is compatible with Docker for Windows.
See : https://github.com/Issif/falco-docker4windows/blob/4.9.125/Dockerfile
The finale image is as big as official, but I can help for start.

I agree with @JPLachance - the idea seems the one we would likely need to implement.
He just wrote me on slack he is not sure whether to use Alpine or Ubuntu as base images.
Wdyt @mstemm @mfdii @leodido ?
Would we have problem with Musl for example? Let's just collect some opinions and make a decision on that so that we can go on and do this

I tend to prefer Alpine because it's smaller and usually faster.
Nevertheless the main point here is about glibc versus musl libc.

Hello!
I don’t think a multi-stage Dockerfile is the solution. Building the kernel module needs to be done at boot of the Falco container when the kernel is unsupported (no builds provided). The multi-stage Dockerfile won’t help because building the kernel module would happen at the build of the container image 🙂
About Alpine VS Ubuntu 18.04, I think it will depend on where is Falco happier. Does Alpine offer all the libraries and packages Falco needs to run without crashing?
I'm sure we all agree we prefer something stable over a super slim image 😄

@JPLachance You're right.
I've bypassed the problem in Docker for Windows by this way : https://github.com/Issif/falco-docker4windows/blob/4.9.125/Dockerfile
I agree it's not native, but it works.

@Issif, it works to build one Falco container image for one specific version of the kernel. If we want to take that approach, we need to build one container image per kernel version we want to support. I'm sure we can do a script that generates and publish a ton of container images but is there where we want to go? 🙂
We could build two container images. One with all the required build tools, another one that is slim and only contains and run Falco. Falco users on systems for which the kernel module is pre-built will have the option to use the slim image. Other Falco users on systems for which we don't have a pre-built kernel module should then have three options:

Use the eBPF probe (if supported by their system)
Use the large image, build the kernel module on boot
Use an init container if they are in Kubernetes, make the init container build the kernel module, mount it in the main Falco container through a volume (could be an emptyDir).

Sounds good?

Yes, I agree with the approach of having an alternate set of container images. We take this approach with our commercial agent, where you can either install a single container that builds the kernel module and contains the agent, or run two containers--one builds the kernel module, and the other only runs the agent.
Alpine would be a good choice for a small image, but we'll need to rebuild falco as alpine linux uses a different libc and it's awkward to get glibc-linked binaries to run cleanly in alpine linux.

Just chipping in here from a possible new Falco/Sysdig user:
We are currently aiming at using Falco as IDS for our Kubernetes clusters on DigitalOcean. As the DigitalOcean K8s VM images do not include kernel headers by default, it is not possible for us to use the "on demand" probe build method inside your current container image.
Pending that you might start building the probe module for the DigitalOcean Kubernetes image based on Debian Stretch: We are now going to follow the strategy to base our own Docker image off your Docker image and include a pre-compiled probe module in that Docker image. So it would be very helpful to get the tiniest base image from your side so that we can simply drop in our precompiled via a very simple Dockerfile of our own.
It will also help if you keep your Docker base image clear of vulnerabilities, as your current Docker image reports various applicable CVEs when checking with Snyk.io:

+1 for a slim image.
There is a trend of moving away from Alpine linux for CNCF project images. While Alpine is small, motivation seems to be partly stability (alpine has shown issues with golang programs in the past) and partly "standardizing" which helps with supply chain auditing.
See this kep for an example: https://github.com/kubernetes/enhancements/blob/master/keps/sig-release/20190316-rebase-images-to-distroless.md

I would prefer to have an image that includes everything it needs and no modifications at boot time.  We are moving towards requiring a readonly file system and no egress for everything and our major offenders preventing us from achieving this right now are our monitoring and security tools.  We would prefer the deploys to hard fail if the image didn't work (ex: wasn't built properly w/ proper module) vs trying to build on demand.  The fewer moving parts the better.

Hello!
A couple of weeks ago we talked about reworking the Falco container image to make it work like the commercial Sysdig Secure image: create an init container image that builds the kernel probe, loads it, then create a slim Falco image that uses the kernel probe and unloads it when exiting.
Where are we in this project? 😄

The list of known vulnerabilities in the Falco container image is only getting longer and longer. It's quite painful to analyze all those and make sure we are not vulnerable.
Thanks for the update!

Yup. As @leodido and @fntlnz are finishing getting onboard with the project, we are beginning to do some release planning that includes this. Thanks for the patience.

@JPLachance please look at #776 and add any comments. I'll get some images published using these docker files for testing.


kubectl CLI should more consistently and concisely convey discovery

Moved from kubernetes/kubernetes#53387

Now that discovery is complete as an API, and CRD and aggregation are in heavy use, we need to make some minor tweaks to kubectl to improve discoverability of apis from clients.
Common issues:

I want to see all resources that I might be able to create
I want to be able to disambiguate conflicting resources
I want to distinguish between cluster scoped and non-cluster scoped resources
I want an equivalent to the all shortcut that is actually all resource types that are namespace scoped

We want to orient brand new users to kubectl from a config perspective into:

what can i get
if i want to create a config resource, how do i specify it (explain, which could be better)
i want to know what a resource is for

Some thoughts:

kubectl api-versions is too simple, and needs to be better or removed
There should be a command that makes it easy to see the list of:

all namespace scoped resources
all cluster scoped resources
the shortcuts of those resources
what the resource does (potentially as a -v option)
a brief discussion of how to deal with version conflicting names (pod v1beta.special.bar and pod v1)
potentially what versions exist for those resources

explain and get should suggest that command when you specify something it doesn't recognize
We could piggyback this into get except discovery resources aren't quite a format API, so it would be somewhat hacky.  However, get and describe already work so a few special cases aren't terrible (we did reserve APIGroupList v1 and APIGroup v1)

Hypothetical
$ kubectl get apigroups
NAME      VERSIONS  RESOURCES
autoscaling v1         horizontalpodautoscaler

$ kubectl describe apis
API Groups
  autoscaling:
    v1:
       Resources: horizontalpodautoscaler
    v2:
       ...

$ kubectl get apis
NAME      VERSIONS  RESOURCES
autoscaling v1         horizontalpodautoscaler

$ kubectl apis
GROUP      VERSIONS  RESOURCES
autoscaling v1         horizontalpodautoscaler

$ kubectl apis --group=autoscaling --version=v1
GROUP      VERSIONS  RESOURCES
autoscaling v1         horizontalpodautoscaler

$ kubectl explain
error: You must select a resource type to explain:

   Nicely formatted two column table - one for namespace scoped and one for cluster scoped

It should be easy to select all namespaced resources from a command - we should add:
$ kubectl get all-namespaced --all
$ kubectl describe all-namespaced
$ kubectl delete all-resources --all

We can't change all because of backwards compatibility (anyone with kubectl delete all --all would be broken if we change what all covers), but also because it's deliberately short to remain easy to type for end users.

This is really annoying now that I'm dealing with more and more groups.
Explain should tell me what group something is
I need to be able to see info on a given resource type in the exact form that I would specify it to get (the so called "resource-arg" form resource.version.group). I.e. "I got an error that told me this resource doesn't do what I think - how do i find out more about it".
Error messages need to do a better job of returning a discoverable type (for 2) - right now we're doing either the struct print for many of them or the GroupVersionResource.String() method which is not an accepted form.
I think this is the #1 issue dealing with extensions - no place to go to find them, no way to know how to use them in kubectl, no way to check that the ones I care about are registered.
Use cases:
When I as a user get errors about mismatches between what I typed and what is available on the server, I need to discover what is on the server
As a user, I expect to get canonical representations of resources so as to avoid having to explain how to transform the representations myself

We already have kubectl api-versions and there was a PR to add kubectl api-resources.  But when I think about the operations that these commands perform, and that I may want to describe the objects they return in more detail, I think it makes more sense to expose these as resources under kubectl get and describe.  That also makes it easier to suggest options:
$ kubectl get
usage: This command expects a resource type and an optional name of a resource to locate.  Use `kubectl get apiresources` to see all valid resource types.

/remove-lifecycle stale

/kind feature
/area kubectl
/priority P2

/assign

We should go through this and take inventory.
@soltysh will look at.


Use GitHub's issue forms to simplify issue triage

What problem does this solve or what need does it fill?
Instead of requiring an organization member to manually apply common labels to issues, it'd be great if the person filing the issue could help label them.
What solution would you like?
https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/syntax-for-issue-forms
GitHub's issue forms should allow asking more targeted questions that we already have in our issue templates, and automate some parts like applying relevant labels to issues. For example, one of the questions may involve asking what kind of bug was seen, so we can ask if the problem is a crash, and it will automatically add C-Crash to the issue without a organization member triaging the issue.
What alternative(s) have you considered?
Leaving it as is, use issue templates as we have been.

Duplicate of #2462?

Yep, looks like a duplicate to me. Whoops.


Async

Hi there! I was wondering if there is a reason why the auth requests are made synchronously rather than async? I am in a situation where I need to make the requests async so wondering if this is deliberate

Hi! Nope, not deliberate; more less random.
To give you a bit of background, this project was somewhat of a saturday-hack where I was learning the details of this ancient model of authentication while seeing if this was actually possible to reimplement it bottom-up. Most browsers have NTLM-challenge response builtin for "normal" HTTP-requests, but I hadn't seen it done with AJAX before. As time went by people started using it, especially developers stuck in an unforgiving domain (I'm looking at you random enterprise product with your own markup and no viable way of customisation except injecting javascript).
I haven't looked at the code in a while, but if I remember correctly it should be straight-forward to asyncify it by simply changing the XHR-requests to async and accepting a callback into the auth method. All PR:s are certainly welcome.


Clipboard is null

Environment
React Native with Expo 39
Platforms
Android
Versions

Android: 10
react-native: 63

Description
...

import Clipboard from '@react-native-community/clipboard';

...

const Component = () => {

  useEffect(() => {
    const check = async () => {
      const clipboard = await Clipboard.getString();
      if (clipboard) {
        setClipboard(clipboard);
        Clipboard.setString('');
    };
    check();
  }, []);

...

This code throws this error: null is not an object (evaluating 'NativeClipboard_1.default.getString')
But strangely, when I test the code with Jest it works... Maybe Clipboard is not yet mounted properly when the effect is triggered.

I had the same issue with Android.
Weird enough I had to do a pod install:
cd ios && pod install && cd ../
after that I runned react-native run-android and it worked.

I had the same issue with Android.
Weird enough I had to do a pod install:
cd ios && pod install && cd ../
after that, I ran react-native run-android and it worked.

Gonna try this too @Jorciney .

@Dheyson good luck!

@Jorciney are you using React Native with Expo (managed project) ?
Unfortunately I think it's not safe to use pod install with Expo (managed project)...

@lapico You are right, I'm not using Expo.

Seems like using the native alternative (deprecated) is the only workaround for now. The community version just doesn't work with Expo. I tried a lot, including using useClipboard hook, but have to now settle with using the deprecated version. At least it doesn't throw that ugly error.

I confirm this, I also use the deprecated version because the community one does not work with Expo.
Strange fact, the community version seemed to work in the tests (Jest).

Unfortunately, this module is not compatible with Expo, since expo only works with preinstalled native modules. If you're using Expo with managed workflow, please use the Clipboard module exported from react-native

I solved it this way： cd android && ./gradlew clean

I solved it this way： cd android && ./gradlew clean

I solved it this way： cd android && ./gradlew clean

This answer is the correct one. thank you!

Stil this is open?


Website wont load, but the controller is online.

I am having trouble with the controller website. I can no longer load https://<IP_ADDRESS>:8443/manage.
I know the service is running because my guest wifi is running and as I understand that doesn't work unless you have the controller online.
I've tried a few different restarts. Any suggestions?

Are you getting a black page? That's what I get, I just installed it for the first time.

Are you getting a black page? That's what I get, I just installed it for the first time.

@bafonso  no, the website doesn't load at all

I found out my issue was that pfblockerNG already uses some of the same ports. This should be checked for in the script and mentioned in the installation notes.

I ended up re-installing. I did lose some settings, but that is on me.


Route53 DNS Entry is not cleaned up.

When tearing down the workshops the Route53 DNS entry remains.

fixed with PR https://github.com/network-automation/linklight/pull/104


Add react-native-callkeep link

To use callkit or connection service with your voip application.

Cheers!


[DRAFT] Improve link resolution for symbol links

Bug/issue #, if applicable: rdar://78518537
Summary
This is a draft PR that aims to improve link resolution for symbol links. The first step towards that is to allow symbol links to be redundantly disambiguated. This opens up for further changes without breaking existing links.
There is still a lot of integration work before this code can be cleaned up, documented, and optimized.
Dependencies
n/a
Testing

[ ] Add testing instructions

Checklist
Make sure you check off the following items. If they cannot be completed, provide a reason.

[ ] Added tests
[ ] Ran the ./bin/test script and it succeeded
[ ] Updated documentation if necessary

This has been very informative. These changes are well worth pursuing.
That said, I don't think this PR as-is is the best way to land these changes. Rather then adding more changes to this PR I'll go through the history to break out smaller changes that can land incrementally to make preparations for running both link resolution systems side by side.
I'll keep this PR open as a Draft until there's a new PR with the new link resolution system that's ready for review.

I'll close this prototype PR now. See https://github.com/apple/swift-docc/pull/337 instead.


Clear Clipboard does seem to be working

The standard "Clear" command clears clipboard, but the new Automation Task, "Clear Clipboard", does not work

The clear commands clear clipboard history. The Automation Task (note the description) removes clipboard contents. Meaning that on ⌘V, nothing will be pasted. This is useful to free up the memory used by a big copy, or to ensure you won’t accidentally paste something you dind’t want.


Tell SchemaCrawler not to get routines

Avoids VerifyMigrationTask emitting "Could not retrieve functions" because SQLite doesn't support routines (stored procedures).
Fixes #1348

dope, thanks

🎉 thanks!


Platformio-ide-terminal stopped automatically and now unable to install back

[ yes ] I have reset Atom to defaults prior to submitting report.
[ ] I have not reset Atom to defaults prior to submitting report.

Description

Steps to reproduce

1.unistall atom on windows 10
2. tried to reinstall platformio-ide-terminal
3.Getting the below message in error log
2130 error code ELIFECYCLE
2131 error errno 1
2132 error<EMAIL_ADDRESS>install: prebuild-install || node scripts/install.js
2132 error Exit status 1
2133 error Failed at the<EMAIL_ADDRESS>install script.
2133 error This is probably not a problem with npm. There is likely additional logging output above.
2134 verbose exit [ 1, true ]
Expected behavior:
The terminal to have install as expected
Actual behavior:
Not able to install back
Reproduces how often:
NA
Versions

> atom --version
 Atom    : 1.52.0
Electron: 6.1.12
Chrome  : 76.0.3809.146
Node    : 12.4.0

> apm --version
apm  2.5.2
npm  6.14.8
node 12.4.0 x64
atom 1.52.0
python 3.7.6
git 2.27.0.windows.1
visual studio

OS name and version: windows 10
Platformio-ide-terminal version: 2.10.0
Additional Information

I tried to install with the master zip file in https://github.com/platformio/platformio-atom-ide-terminal/archive/master.zip
related to  https://github.com/platformio/platformio-atom-ide-terminal/issues/795.
It installed successfully but I am still not able to see the terminal from atom
I tried the command: apm install platformio-ide-terminal
I got the same message in error logs
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>install: prebuild-install || node scripts/install.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:

+1 same on mac.

I have the same problem, also tried to reinstall, but unsuccessful

same here, suddenly stopped working, removed it and now I am unable to install it.
macOS Catalina 10.15.7
Atom 1.52.0

same on windows 2.10 i just get a blank terminal i can't type

Reinstall Atom and it should work again.

tried it.
but not solve that problem.

True, it worked until I rebooted, then it stopped working again.
I just went with Terminus.
https://atom.io/packages/terminus

oh!! thank you!!
The package solved my all the problems.

I reinstalled atom and it didnt work. It seems to be the package issue. The necessary files are not getting installed. I have stared using Terminus for the time being

This package seems to be terminal

Fixed in https://github.com/platformio/platformio-atom-ide-terminal/releases/tag/v2.10.1
Please update platformio-ide-terminal to the latest version.


[Feature Request] use pynput instead of AHK

Hi,
I use Linux and AHK doesn't work (no AHK for Linux, others libraries needed or Wine needed and possibly a lot of headaches).
Could you use pynput ? https://pypi.org/project/pynput/
It works for Linux, MacOS and Windows :
https://pynput.readthedocs.io/en/latest/limitations.html
Regards

Linux support is coming.
2 pull requests added on exDev branch (last one : https://github.com/Deopster/Mercenaries-Hearthstone-game-bot/pull/20)
The next one (the third) should do the work

Hi,
I use Linux and AHK doesn't work (no AHK for Linux, others libraries needed or Wine needed and possibly a lot of headaches). Could you use pynput ? https://pypi.org/project/pynput/
It works for Linux, MacOS and Windows : https://pynput.readthedocs.io/en/latest/limitations.html
Regards

Hey dude, i just returned to work on this project and checked out what exactly you did. And this is such a big rework of my project, thank you very much.

Hey dude, i just returned to work on this project and checked out what exactly you did. And this is such a big rework of my project, thank you very much.

The pleasure was mine :)
Linux is supported; I close the issue.


Add docker image

Thanks for building the docker image!
There are now official docker images on https://github.com/elves/elvish/pkgs/container/elvish though and it's updated every commit, so I'll close this PR.
BTW this doesn't mean I'm not accepting any Docker image into this repo - if the image is sufficiently different from the official image it'd still make sense to include here.


Sankey Crashes

With certain queries, the sankey maps begin to crash. The browser tab takes up an inordinate amount of resources.
Example:
CLPFU a2
Germany
PCM
Energy
Final
IEA
Not including NEU
1970
To look into:

[ ] See if Professor Heun can create the sankey diagram in R given the same query.
[ ] Look for a number of rows threshold that makes the rendering crash.

Possible fix:
- Assuming the cause is too much JavaScript being executed client side at once.
Output diagram as an image and then send that back to the client.
Cons:
Lose interactivity provided by Plotly.

Each plotly plot comes with a chunk of script that actually renders the plot using the plotly library, which is only included once when the whole page loads. From my testing, the issue is in the plot being rendered client-side and the only thing changing client side, besides a few new divs for the plot to live in, is the chunk of javascript being delivered with a new plot.

Here is an example of the javascript delivered with a plot:

To look into:

[ ] See if Professor Heun can create the sankey diagram in R given the same query.
[ ] Look for a number of rows threshold that makes the rendering crash.

After today's meeting we know that the sankeys we currently have are slightly wrong. I don't know or think that it will have a large impact on this problem, but it is a possibility.

With certain queries, the sankey maps begin to crash. The browser tab takes up an inordinate amount of resources.
Example: CLPFU a2 Germany PCM Energy Final IEA Not including NEU 1970
To look into:

[ ]  See if Professor Heun can create the sankey diagram in R given the same query.
[ ]  Look for a number of rows threshold that makes the rendering crash.

Here is the Sankey diagram that I constructed in R for the requested example:

I noticed that it takes a while (22 seconds) to download the DEU, 1971 data from the database. Making the Sankey diagram is very fast (< 1 second).
For posterity, the code I used to download the DEU, 1971 data is
foo <- PFUPipelineTools::pl_filter_collect("PSUT",
                                           countries = "DEU",
                                           years = 1970,
                                           methods = "PCM",
                                           ieamws = "IEA",
                                           last_stages = "Final",
                                           energy_types = "E",
                                           includes_neu = FALSE,
                                           conn = conn,
                                           collect = TRUE,
                                           matrix_class = "matrix")

Another possible (temporary?) solution here would be to create the sankey diagrams in R.
See, for example,

https://christophergandrud.github.io/networkD3/
https://rviews.rstudio.com/2022/05/25/calling-r-from-python-with-rpy2/

After some testing:
The SanKEY.js library does not crash in making this report. Server side it runs in around 0.1 seconds and it renders pretty quick client side--maybe a second or less on my machine on Firefox.


vsc-leetcode-cli: init at 2.8.0

vsc-leetcode-cli: init at X.Y.Z
node-packages.nix: generate

Motivation for this change
Add the vsc-leetcode-cli npm package, which is a fork of the leetcode-cli, and looks
to be maintained.
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

cc @SuperSandro2000

Result of nixpkgs-review pr 119246 at c05f2fea run on aarch64-linux 1

7 packages failed to build:

python39Packages.batchspawner
python39Packages.dockerspawner
python39Packages.jupyterhub
python39Packages.jupyterhub-ldapauthenticator
python39Packages.jupyterhub-systemdspawner
python39Packages.jupyterhub-tmpauthenticator
python39Packages.oauthenticator

2 packages skipped due to time constraints:

iosevka
lumo

13 packages built successfully:

discourse
python38Packages.batchspawner
python38Packages.dockerspawner
python38Packages.jupyterhub
python38Packages.jupyterhub-ldapauthenticator
python38Packages.jupyterhub-systemdspawner
python38Packages.jupyterhub-tmpauthenticator
python38Packages.oauthenticator
vimPlugins.YouCompleteMe
vscode-extensions.matklad.rust-analyzer
vscode-extensions.vadimcn.vscode-lldb
ycmd
zerobin

Note that build failures may predate this PR, and could be nondeterministic or hardware dependent.
Please exercise your independent judgement.

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 119246 run on x86_64-linux 1

1 package marked as broken and skipped:

image_optim

1 package blacklisted:

tests.nixos-functions.nixos-test

1 package failed to build and already failed to build on hydra master:

netlify-cli: log was empty

50 packages built:

antora
balanceofsatoshis
bitwarden-cli
castnow
discourse
emojione
epgstation
fast-cli
flood
gtop
imgbrd-grabber
iosevka
joplin
lumo
mirakurun
pm2
postcss-cli
pulp
python38Packages.batchspawner
python38Packages.dockerspawner
python38Packages.jupyterhub
python38Packages.jupyterhub-ldapauthenticator
python38Packages.jupyterhub-systemdspawner
python38Packages.jupyterhub-tmpauthenticator
python38Packages.oauthenticator
python39Packages.batchspawner
python39Packages.dockerspawner
python39Packages.jupyterhub
python39Packages.jupyterhub-ldapauthenticator
python39Packages.jupyterhub-systemdspawner
python39Packages.jupyterhub-tmpauthenticator
python39Packages.oauthenticator
redoc-cli
sloc
thelounge
vimPlugins.YouCompleteMe
vimPlugins.coc-html
vimPlugins.coc-metals
vimPlugins.coc-prettier
vimPlugins.coc-pyright
vimPlugins.coc-rust-analyzer
vimPlugins.coc-stylelint
vimPlugins.coc-tsserver
vimPlugins.coc-vetur
vscode-extensions.matklad.rust-analyzer
vscode-extensions.ms-python.vscode-pylance
vscode-extensions.vadimcn.vscode-lldb
wasm-text-gen
ycmd
zerobin


testAdd group by aggregate negative case

pick from master #36562
Proposed changes
testAdd group by aggregate negative case

run buildall


Add Test Runner

Add any test runner. Maybe Karma?

@matheuslc What about karma and jasmine? :) #3

Awesome, @drdpedroso! Thank you!


Unable to import Lex bots with fallbackIntent and clarification prompts unchecked in the Error Handling section of the bot

“
INFO:amazon_lex_bot_deploy.amazon_lex_bot_deploy:deployed all intents
ERROR:main:'clarificationPrompt'
Traceback (most recent call last):
File "./build-lexbots.py", line 131, in
check=True)
File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['lex-bot-deploy', '-s', './build/lexbots/YesNoWithFallback.json']' returned non-zero exit status 1.
ERROR: could not deploy ./build/lexbots/YesNoWithFallback.json

Resolved. Checks now if clarificationPrompt exists in the schema and if not does not try to send it to the put_bot command.
Thx for pointing out the issue!


[Feature Request] Support for without xaml

using Avalonia.Controls;

namespace AvaloniaDemo;

public class MainWindow : Window {
    public MainWindow() {
        Title = "Avalonia Demo";
        Content = new TabControl {
            ItemsSource = new TabItem[] {
                new TabItem() { Header = "Main", Content = new TextBlock { Text = "Main Content" } },
                new TabItem() { Header = "Configuration", Content = new TextBlock { Text = "Configuration Content" } },
            }
        };
    }
}

Is it possible to implement a hot restart when modifying the code here?
e.g.
using Avalonia.Controls;

namespace AvaloniaDemo;

public class MainWindow : Window {
    public MainWindow() {
        Title = "Avalonia Demo";
        Content = new TabControl {
            ItemsSource = new TabItem[] {
                new TabItem() { Header = "Main", Content = new TextBlock { Text = "Main Content" } },
                new TabItem() { Header = "Configuration", Content = new TextBlock { Text = "Configuration Content" } },
+               new TabItem() { Header = "Hello", Content = new TextBlock { Text = "Hello Content" } }
            }
        };
    }
}

Probably a lot like #13.

But hot reloading with dotnet watch alone doesn’t seem to update the interface in real time, maybe I’m missing something?

Yeah, that's because it doesn't automatically re-run the code that has already been executed. For that, you can use a trick suggested by AVVI94.

OK, I found MetadataUpdateHandlerAttribute when I was looking for information. Maybe I can simplify the operation with this attribute? But it may require more complicated processing

You can use a metadata update handler for that instead of a crude "Rerender" button, for sure. Just trigger your UI rerender in the UpdateApplication method, there's no real need to actually process the updated types provided to it.

OK, thanks for the help

You are welcome!
I'll close this issue as "not planned", however, if anyone wants to contribute an IHotReloadContext for XAML-less controls to the project, I'm always open to PRs ;)


Updating CameraCaptureUI Spec Doc

This PR includes the following changes:

The change updates the CameraCaptureUI specifications doc to include important notes on usage for PhotoSettings and VideoSettings.
Added the CameraCaptureUIContract in the WinRT contracts as well.
A small update by removing the marshalling=none and threading=STA attributes. These settings were relevant to the UWP version of CameraCapture, but they are no longer applicable in the context of WinAppSDK.

/azp run

/azp run


Make MoshiModule public

Making this available publicly, so that lightweight tests that only need moshi set up and don't rely on more misk modules can depend on it

e: /home/circleci/misk/misk/src/test/kotlin/misk/moshi/MoshiAdapterPrecedenceTest.kt: (37, 7): Unresolved reference. None of the following candidates is applicable because of receiver type mismatch:
public operator fun <T, R> DeepRecursiveFunction<TypeVariable(T), TypeVariable(R)>.invoke(value: TypeVariable(T)): TypeVariable(R) defined in kotlin
:misk:compileTestKotlin (Thread[Execution worker for ':' Thread 2,5,main]) completed. Took 1 mins 28.469 secs.

@acostama, if you still want this PR, please fix the problems and resubmit.

@acostama, if you still want this PR, please fix the problems and resubmit.

Looking at the conflicts/master, I think this is no longer required.  Closing.


Handling button after view submission?

Description
Dear developers, bolt is awesome, thanks for making this available!
I have a question about handling buttons after view submissions. My process is as follows:

Post a message with one action button.
Click the button and open a view.
Submit the view.

In bolt, the app.view construct has an 'ack' function but not a 'say' or 'response' function. How would I update the action button at step (1) to show a user that this view was submitted? I want to disable that button.
I hope you can help me!
PS: this is related (https://github.com/slackapi/bolt/issues/305) but I think we can only use ack() for error handling.
What type of issue is this? (place an x in one of the [ ])

[ ] bug
[ ] enhancement (feature request)
[x] question
[ ] documentation related
[ ] testing related
[ ] discussion

Requirements (place an x in each of the [ ])

[x] I've read and understood the Contributing guidelines and have done my best effort to follow them.
[x] I've read and agree to the Code of Conduct.
[x] I've searched for any related issues and avoided creating a duplicate issue.

I will store the original block in a temporary array and find it again based on an id that I can pass to the view and to the view submission. Then change the button to text and use the chat.update function to update the block.

Hey @robinpapa, I only just saw this issue but I have one more suggestion. You can use the private_metadata property of the modal view to store arbitrary info. In this case, you can store the ts and channel of the message with the button, so that you can use chat.update after submission to update it. This will help you avoid that temporary array, which can get tricky if your app has to restart or if you fail to clean up the entries correctly.

Ah, that's smart, @aoberoi ! Thank you, will try that!

You could read the original blocks by using the conversations.history method. However, I wouldn't recommend this unless your app already needs the channels:history, groups:history, im:history, or mpim:history scopes. These scopes are generally considered by admins to be highly permissive, since they allow apps to read many many messages.
I'd instead recommend just copying the entire message (blocks and all) into the private_metadata of the dialog. When the button in the message is clicked, the action listener will have access to that message in the body argument, as body.message. You can serialize it to a string, and then JSON parse it back on the view submission.

app.action('my_button_action_id', async ({ ack, body }) => {
  // TODO: validate and perform any logic necessary to open the modal
  ack();

  // Generate private metadata for the modal
  let privateMetadata = '';
  if (body.message !== undefined) {
    privateMetadata = JSON.stringify(body.message);
  } else {
    // TODO: log and early return? this would be a logical issue and should never happen
  }

  await app.client.views.open({
    trigger: body.trigger_id,
    callback_id: 'my_modal',
    view: {
      type: 'modal',
      title: { type: 'plain_text', text: 'My Modal' },
      blocks: [ /* ... TODO: some blocks */ ],
      private_metadata: privateMetadata,
    },
  });

  // ...
});

app.view('my_modal', ({ ack, view }) => {
  // TODO: validate inputs
  ack();

  // Read original message from private metadata
  let originalMessage = undefined;
  try {
    const originalMessage = JSON.parse(view.private_metadata);
  } catch (error) {
    // TODO: log and early return?
  }

  // TODO: do the work

  // TODO: use chat.update to update the original message
});

@aoberoi That's the solution! JSONify the string. Thank you!


Include realTimeConfigQuickScan.pl as a part of Musnix(-starter)

Hey there! Yeeeaaah...
Spent the morning of sunday trying to get perl/Tk to build - it did, but little did I realise that the GUI version is still unstable -- and I forgot to test run realTimeConfigQuickScan.pl initially. It works!
I was introduced to realTimeConfigQuickScan.pl through Arch Linux. What it does is check the system for certain settings and outputs a list of good or bad values. As it stands I'm not 100% sure if it outputs correctly. or if it even gives us a set of system settings we want to use. Here's what it gave me and I have musnix-starter.nix imported from /etc/nixos/configuration.nix.
== GUI-enabled checks ==
Checking if you are root... no - good
Checking filesystem 'noatime' parameter... 3.18.3 kernel - good
(relatime is default since 2.6.30)
Checking CPU Governors... CPU 0: 'performance' CPU 1: 'performance' CPU 2: 'performance' CPU 3: 'performance' CPU 4: 'performance' CPU 5: 'performance' CPU 6: 'performance' CPU 7: 'performance'  - good
warning: '/sbin/sysctl vm.swappiness' did not produce a parsable result
Checking swappiness...  - warning
Checking for resource-intensive background processes... none found - good
Checking checking sysctl inotify max_user_watches... < 524288 - not good
increase max_user_watches by adding 'fs.inotify.max_user_watches = 524288' to /etc/sysctl.conf and rebooting
For more information, see http://wiki.linuxaudio.org/wiki/system_configuration#sysctlconf
Checking access to the high precision event timer... readable - good
Checking access to the real-time clock... readable - good
Checking whether you're in the 'audio' group... yes - good
Checking for multiple 'audio' groups... no - good
Checking the ability to prioritize processes with chrt... yes - good
Checking kernel support for high resolution timers... found - good
Kernel with Real-Time Preemption... not found - not good
Kernel without real-time capabilities found
For more information, see http://wiki.linuxaudio.org/wiki/system_configuration#installing_a_real-time_kernel
Checking if kernel system timer is set to 1000 hz... found - good
Checking kernel support for tickless timer... found - good
== Other checks ==
Checking filesystem types...
** Warning: do not use /run/media/thor/Storage for audio files.
   fuseblk is not a good filesystem type for realtime use and large files.
   For more information, see http://wiki.linuxaudio.org/wiki/system_configuration#filesystems
not found.
** Warning: no tmpfs partition mounted on /tmp
   For more information, see:
   - http://wiki.linuxaudio.org/wiki/system_configuration#tmpfs
   - http://lowlatency.linuxaudio.org
** Set $SOUND_CARD_IRQ to the IRQ of your soundcard to enable more checks.
   Find your sound card's IRQ by looking at '/proc/interrupts' and lspci

So why include it in musnix and not nixpkgs? Well it might be usefull for setting up video workstations as well, but it's fairly obvious that this script checks for audio related settings. On top of that there is no build process what so ever. You only need Perl to be able to run it.
Perhaps it could be a musnix module?
Source:
https://github.com/raboof/realtimeconfigquickscan

Hi NixThor (I guess),
Thanks for the reminder. I've been using this for years, it's great.
I'm not really convinced this shouldn't be in nixpkgs though.
I would propose to put it in there, and maybe import it here.

Hi NixThor (I guess),
Thanks for the reminder. I've been using this for years, it's great.
I'm not really convinced this shouldn't be in nixpkgs though.
I would propose to put it in there, and maybe import it here.

Hi NixThor (I guess),
Thanks for the reminder. I've been using this for years, it's great.
I'm not really convinced this shouldn't be in nixpkgs though.
I would propose to put it in there, and maybe import it here.

Hi NixThor (I guess),
Thanks for the reminder. I've been using this for years, it's great.
I'm not really convinced this shouldn't be in nixpkgs though.
I would propose to put it in there, and maybe import it here.

Hi NixThor (I guess),
Thanks for the reminder. I've been using this for years, it's great.
I'm not really convinced this shouldn't be in nixpkgs though.
I would propose to put it in there, and maybe import it here.

Hi NixThor (I guess),
Thanks for the reminder. I've been using this for years, it's great.
I'm not really convinced this shouldn't be in nixpkgs though.
I would propose to put it in there, and maybe import it here.

I agree that it belongs in nixpkgs.
In terms of whether or not the output is accurate: if indeed you are just importing the current musnix-starter.nix into your configuration without changing any musnix settings, and you have rebuilt and rebooted your system, it should be reporting that you have a kernel with real-time preemption and real-time capabilities.  So this script might be making assumptions about the system which don't apply to NixOS, and would need to be patched to give accurate results.
If you do decide to submit it to nixpkgs, we can include it in musnix-starter.
Okay to close?

@magnetophon Cheers. It's pretty cool just as a checklist. If we import it as a module we can modify it to output stuff like machine readable outputs :D :dancer:
@henrytill My problem was originally tied with the fact that I called a linuxPackage AFTER importing musnix, so you can kind of guess that the musnix kernel was ignored in favour of the vanilla kernel. So the script didn't fail; it really didn't find a kernel with PREEMPT.
Other than that I had to change the SwappinessCheck.pm file to call "sysctl" rather than "/sbin/sysctl". I've made an issue on the github page for realTimeConfigQuickScan, but again: it's not hard to patch.
What can I say? It's out of my hands. Do you want it as a sub-module or not? If not I could always try and package it :)

@thorhop In case you're still waiting for an answer: It would be great if you pkg it for upstream!


Looking for a new maintainer to take over this repository

I don't see myself being able to continue working on this project in the future. It''s been abandoned for a while.
It would be great if someone else picked it up. Hare is an awesome upcoming programming language after all!

I would be willing to take it over if you are not able to spend much time on the plugin. I am actively using it for my Hare projects, so it would be pretty convenient for me.

Great!
It's not letting me do so right now since you have a fork of the repository with the same name.
Please let me know when you've either renamed or deleted the repository so I can send you a repository transfer request.

Ok, I renamed my fork. Thanks.

Oops sorry, they want you to delete it:

Ahh, ok. Deleted. Thanks!

Sent the transfer request

Great, looks like it worked. Thanks for starting this minimal Hare plugin @colinkiama; I look forward to taking care of it.


How to do multi-node inference using docker

I see that FasterTransformer seems to support multi-node inference by MPI. If on two cluster machines, each running one docker installed with FasterTransformer, could you provide more details about how to configure the network between dockers? Seems like we need to do ssh from dockerA  running on machine A to  dockerB running on machine B? I am not sure whether this is achievable and how to achieve. Thanks a lot!

The example is in subsection Run gpt on multi-nodes of https://github.com/NVIDIA/FasterTransformer/blob/main/docs/gpt_guide.md#run-gpt


Examples don't have version so buildozer complains.

Tried adding to it the audio and pictures examples and they built ok (though the audio one didn't display the buttons as I guess the layout doesn't work on the smaller screen).
I guess it's not just adding version but checking if they actually work ..

I'm not aware of buildozer requiring version in the main module, is that the case?

Hm, when I remove it and try and rebuild it seems OK... trying one of the other examples now..
Here's the pictures example
http://imgur.com/j6wXrSm

the pics themselves are quite small - should probably be scaled relative to the screen size initially ?

here's the sound example
http://imgur.com/qzSvH0m

can't press any sound buttons as they are offscreen

Ah - found the relevant output (was lost way back in history).   I have no idea if it's needed, as I am super new to it
----------
id: 155 or "extra-google-simulators"
     Type: Extra
     Desc: Android Auto API Simulators, revision 1
           By Google Inc.
           Android Auto API testing simulators
           Install path: extras/google/simulators
----------
id: 156 or "extra-google-webdriver"
     Type: Extra
     Desc: Google Web Driver, revision 2
           By Google Inc.
           WebDriver
           Install path: extras/google/webdriver

# Android packages installation done.
# Check application requirements
# Check garden requirements
# Compile platform
# Distribution compiled.
# Build the application #1
# Package the application
Traceback (most recent call last):
  File "/usr/bin/buildozer", line 9, in <module>
    load_entry_point('buildozer==0.31.dev0', 'console_scripts', 'buildozer')()
  File "/usr/lib/python2.7/dist-packages/buildozer/scripts/client.py", line 13, in main
    Buildozer().run_command(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/buildozer/__init__.py", line 987, in run_command
    self.target.run_commands(args)
  File "/usr/lib/python2.7/dist-packages/buildozer/target.py", line 85, in run_commands
    func(args)
  File "/usr/lib/python2.7/dist-packages/buildozer/target.py", line 97, in cmd_debug
    self.buildozer.build()
  File "/usr/lib/python2.7/dist-packages/buildozer/__init__.py", line 198, in build
    self.target.build_package()
  File "/usr/lib/python2.7/dist-packages/buildozer/targets/android.py", line 583, in build_package
    version = self.buildozer.get_version()
  File "/usr/lib/python2.7/dist-packages/buildozer/__init__.py", line 681, in get_version
    ' (looking for `{1}`)'.format(fn, regex))
Exception: Unable to find capture version in ./main.py
 (looking for `__version__ = ['"](.*)['"]`)

Could you open this against buildozer? https://github.com/kivy/buildozer
Please post this log and a link to the example you have used.


Improve search in "Saved Projects" and "Saved APKs" menu.

If the user has save a project or apk by the name "Test VI" and searches the file by entering "VI" the result is not shown. Basicly the current function checks if the project or APK name starts with the search phase.
Intended - Even if the user enter a part of the filename, the file should be shown in search result.

@opticod @croozeus
Should I make a PR for this issue.

Closed via #261


Rarely no incoming messages and no errors

In our app, in one instance a user has reported that new messages don't show up, etc. Based on the various symptoms and logs it appears that ably messages are not coming in but the device is otherwise able to access the internet. The problem persisted (at least in one instance) if the app is restarted. Our error logging is ignoring subscribe timeouts (ref https://github.com/ably/ably-ios/issues/649) but all other errors are being logged and no other errors are occurring. We're using version 1.0.9. Other devices receive the same messages. My questions are:

Does any state persist across app launches?
Any advice for further debugging this?

Thanks!

For the record re #1: no, state isn't persisted across app launches. The issue is probably not on ably side. Closing.


Add support for offset commit recovery

Add CommitRecovery for defining offset commit recovery strategies. The default strategy only retries RetriableCommitFailedException with a jittered exponential backoff for 10 attempts, switching to fixed rate retries for up to another 5 attempts (see CommitRecovery#Default).
You can override the CommitRecovery using ConsumerSettings#withCommitRecovery. If you would like to go back to default behaviour before this change (no retries), simply use withCommitRecovery with CommitRecovery#None.

This is very nice, 👍 for adding jitter.


Firefox reload hack

@jiwalker-usgs Heres the PR with the app.js and CSS changes, let me know if we find a better solution before this is merged.
@jread-usgs Theres a bit of code for the wallytrends svg but Idk if I should be checking that in or not? Since I don't know how this sandbox area works

@mwernimont can you let me know what the code is and I can add it?


Error on interpolation

When I try the interpolation demo, I have this error
glyph.updateOTCommands is not a function.
It send me to this part of the code (line 20650)
	this.ot.glyphs.glyphs = (
		this.getGlyphSubset( args && args.set ).reduce(function(o, glyph, i) {
			o[i] = args && args.shouldUpdateCommands ?
				glyph.updateOTCommands( null, args && args.merged ) :
				glyph.ot;
			return o;
		}, {})
	);

I'm looking into it right now.

Fixed in 1.0.0 : )


Add getDmsObjectNotes() function in DMS SDK

Hi @andinaether
thank you very much for your contribution. I have reviewed your changes and merged them into the develop-branch. That means your changes are already available via npm i @dvelop-sdk/dms@next.
After some further testing within the next days i will merge them into the main-branch.


Define Memcached and Redis as CustomResource

What are you trying to accomplish with this PR?
Be able to validate the custom resource status for Redis and Memcached resources.
Related PR: cloudbuddies#1783
How is this accomplished?
Define Memcached and Redis as CustomResource and call the deploy_succeeded function to validate the status.
What could go wrong?
Nothing special, this is compatible with the existing CRDs
I tested with and without the kubernetes-deploy.shopify.io/instance-rollout-conditions annotation for both CRDs.

@jerr are you planning on revisiting this PR soon? I see the "buddies v2" project this was part of was moved to Done last week.

@jerr are you planning on revisiting this PR soon? I see the "buddies v2" project this was part of was moved to Done last week.

yes, everything is done on the buddies side and shipped.  I have to find a good way to test the kubernetes-deploy changes.

Test a simple case

create a namespace

$ kubectl create namespace kubedeploy-test
namespace/kubedeploy-test created

create a redis yaml file

$ cat << EOF >> redis1.yaml
{
  "apiVersion": "stable.shopify.io/v1",
  "kind": "Redis",
  "metadata": {
    "name": "test1"
  }
}
EOF

deploy it with kubernetes-deploy

$ REVISION=12345 ./exe/kubernetes-deploy kubedeploy-test minikube  --template-dir=~/tmp/kubernetes-deploy_496

🎉 Works as expected!

Test wrong configuration case

create the wrong configuration

cat << EOF > redis2.yaml
{
  "apiVersion": "stable.shopify.io/v1",
  "kind": "Redis",
  "metadata": {
    "name": "test2"
  },
  "spec": {
    "max_memory": "1Mi"
  }
}
EOF

deploy it with kubernetes-deploy

$ REVISION=12345 ./exe/kubernetes-deploy kubedeploy-test minikube  --template-dir=~/tmp/kubernetes-deploy_496

🎉 Works as expected!

fix the configuration

$ cat << EOF > redis2.yaml
{
  "apiVersion": "stable.shopify.io/v1",
  "kind": "Redis",
  "metadata": {
    "name": "test2"
  },
  "spec": {
    "max_memory": "1Gi"
  }
}
EOF

redeploy

$ REVISION=12345 ./exe/kubernetes-deploy kubedeploy-test minikube  --template-dir=~/tmp/kubernetes-deploy_496

🎉 Works as expected!

Test the pruning case

delete one Redis

$ rm redis1.yaml

deploy it with kubernetes-deploy

$ REVISION=12345 ./exe/kubernetes-deploy kubedeploy-test minikube  --template-dir=~/tmp/kubernetes-deploy_496

🎉 Works as expected!


chore: update sodium-native to latest

Similar reasoning as #337
Updates sodium-native (used by sodium-universal) from 4.0.1 to 4.0.4

the failing CI is a concern here 🤔 seems way flakier than usual - maybe this upgrade does break something??

seems like this is causing some of the discovery/mdns tests to fail. kind of surprised since it doesn't look like they changed the underlying libsodium version that's being used 🤔
@gmaclennan any ideas?
https://github.com/sodium-friends/sodium-native/compare/v4.0.1...v4.0.4

Replaced by #328 which fixes CI errors from tests


Crash on startup - Fabric Server 1.16.5

My server have this mods:

And if I try to startup with CHAS, it crashes. But if I delete the mod, it runs
This is the log:
https://pastebin.com/vzGyEJd6

Okey, I will look into it

Done several different tests and can confirm this mod is not causing any server crashes on its own.
The problem is with on of the other mods that you are using causing the crash in this mod.
This mod has no code at all and is really just a glorified resource pack, so there is actually no way this mod could cause any crashes on its own without some other mods messing with recipes or datapacks with their mixins.
I would suggest looking through the crashlogs and check if some other mods are running some mixins on the crafting recipes or datapacks causing the crash.


Systematic detection of failure to charge for substructure clone

One class of bugs (https://github.com/stellar/rs-soroban-env/pull/1120 identified and fixed several) is failing to meter deep clone when the substructure contains pointers to heap allocated memory.  This type of bug can be recurring e.g. any time a new deep structure is added in the XDR, and hard to identify.
One way to detect this is to write a fuzz/prop test going through all the MeteredClone types and assert for each case "metered MemAlloc bytes >= actual allocated bytes", this will help detecting missed memalloc (not the memcpy part which should be fine).
Ideally we could do this statically, i.e. for every MeteredClone type that is declared "shallow", traverse its substructure and assert they don't include any heap types (i.e. Vec, Map, Box, Rc, the list here is not that long because most of the env "deep" types e.g. xdr::VecM, MeteredVec are made of these).

Moved to v20.x.y P1 for adding potential prop/fuzz test. The static analysis part will be a longer shot.

Closing as not planned.


Modify addon/utils/error.js to only export setErrorTypeThrown() in test mode

Once https://github.com/softlayer/sl-ember-components/issues/1350 has been merged


Add copyWith to SearchQuery

going to need this for the flutter ui kit

Can you give me more info about this method? I also asked that in the PR review so you can answer only here if you want!

since SearchQuery is immutable, we need a way to change certain values in it, which is why the copyWith is introduced.
why this is needed tho can be seen here
https://github.com/meilisearch/meilisearch-flutter/blob/ab97f0eee4fda9512491dae165b28bebbc77ad61/lib/src/widgets/query_builders/offset_based.dart#L99-L122
where we manipulate the input query to automatically progress the search results based on scroll


Change background color of all the countries of a continent

How do I change the color of all the countries of a continent? Simply putting, how to change the color of a specific continent?

You can not, you will have to change each country individually within that continent.
There is no continents map (yet).