adi_update_boot.sh: adding RPI boot files

Whenever the ADI carriers boot files got updated, download also latest
RPI kernels and modules. Otherwise, RPIs won't boot anymore with the
same SD card since boot partition was deleted and RPI kernels don't exist anymore.
Signed-off-by: stefan.raus<EMAIL_ADDRESS>

Do we need to do any checking to make sure we have space? (if the boot partition isn't big enough?)

Do we need to do any checking to make sure we have space? (if the boot partition isn't big enough?)

Yes, we can do that.
I saw also that some default RPI files are still missing (*.elf, config.txt, cmdline.txt). I will do some rework and update the PR.

Updating PR by not deleting everything from boot partition, but letting RPI basic files (bootcode.bin config.txt etc, cmdline.txt, *.elf , *.dat). If rootfs does not have big changes, these files works fine even if they are older.
Anyway, for adding boot files from a newer release (getting 2021_r1 boot files on a 2019_r2 SDcard and for example), there is printed the recommendation that SD card should be created from scratch with new release image and a warning that things may broke.
Regarding available space - archives are initially downloaded in rootfs partition, and only if they download fine, we proceed further with deleting boot files and replace them with extracted ones. A boot partition space check is done at line #156.

A boot partition space check is done at line #156.

it looks like the space check is only done if:
if [ "$current_release" != "$new_release" ];

what happens when someone moves a large file into the boot partition that we don't manage? and the size of the archive on swdownloads grows? You will get random errors in the script further on - won't you?
in theory - we should download the tar.gz, check the uncompressed size (gzip -l), make sure we think we have room, then delete the files, and uncompress/install things.
-Robin

Split in multiple and clearer commits.
Add also a commit that fixes C5SOC preloader and one more commit for handling versal case when restoring boot files.


feat: polishing credential issuance

polish credential issuance cli parameters:

auto-generate credentials id (overridden with --credential-id option)
allow to set both issuer and subject did
improve parameter names for clarity
update seed files

to test:
make start-dev
make seed

this PR is breaking for CLI parameter order


Update ocaml-version

Add base images for OCaml 5.3~alpha1 and 5.4

5.3.0~alpha1+options failed to build.  This is because the beta repo is not included in the Dockerfile.
The decision to include the beta repository is made with this logic:
https://github.com/ocurrent/docker-base-images/blob/28953def3a4eb6bf38635ae086f55fa29b821c3e/src/pipeline.ml#L43-L48
Since 5.3.0~alpha1+options is not included int the list of unreleased_betas, the repository is not added.  Only 5.3.0~alpha1 is included.
https://github.com/ocurrent/docker-base-images/blob/86b55f8d508c0cb8decf6d73c7dd5438efae06af/src/pipeline.ml#L44
The patch treats all +options varients as if they were included in the unreleased_betas list.
There is a larger question of whether we should build the variants of the development versions and unreleased betas.  Dating back to 2019, we have always built the options of the development versions.  Thus, this was continued in the update from earlier this year to build the unreleased_betas.


Allow PeptideShaker mzid files to lookup spectra by spectrum title

https://skyline.ms/announcements/home/support/thread.view?rowId=50606

I'm not too sure about this. MGF does not require spectrum title to be unique. That's why we use index to point into MGF. I would want to be sure that PeptideShaker chokes on an MGF that has non-unique titles, because then we'd be reasonably certain to never expect a PeptideShaker mzIdentML for an MGF with non-unique titles. Even then there's no guarantee that they won't change PeptideShaker later to handle non-unique titles.

I'm not too sure about this. MGF does not require spectrum title to be unique. That's why we use index to point into MGF. I would want to be sure that PeptideShaker chokes on an MGF that has non-unique titles, because then we'd be reasonably certain to never expect a PeptideShaker mzIdentML for an MGF with non-unique titles. Even then there's no guarantee that they won't change PeptideShaker later to handle non-unique titles.

Hmm, didn't know that about MGF. What if we keep using index for MGF but titles for other formats?

PeptideShaker ONLY works with MGF AFAIK: http://compomics.github.io/projects/peptide-shaker#mzidentml-support

PeptideShaker can load results from virtually any identification algorithm in the mzIdentML format as long as the minimal peptide to spectrum match information is present in the file.
The following is required:
Spectrum file format has to be mgf.
Each PSM has a score or e-value as a PSM score CV term.

Weird, on the support request the user shared mzid/mzml results from PeptideShaker. I just tried using msconvert to convert the mzml to an mgf and that seems to work (with indices).

Seems like this was a peptideshaker workflow issue rather than a BiblioSpec issue


Fix event reference to element rotation property. Updated Gitignore

I think the dist/ should be included (although I should update it so it generates just one file).

Updated PR to not ignore dist folder


How to share a file with intent

Hello @darryncampbell can you help me please with some snippets about how to share a file to another app
you can take please the case of an image

What Intent are you trying to send / receive?

Image, Picture, all the medias

Any news?

What specific Intent & payload needs to be send?

maybe I'm being misunderstood, what I'd like is for a third party application to be able to share information such as (text, images, multimedia files) to my application, a bit like facebook and whatsapp do.

Sorry, I am unable to offer generic Android development advice but this Android development guide looks relevant: https://developer.android.com/training/secure-file-sharing/share-file

i want to know please if your plugin can serve for this purpose

Closing due to inactivity


Feature/1287 deploy new branches

#1287
Test if master.permplant.net works
Basics

[ ] The PR is rebased with current master
[ ] I added a line to changelog.md
[ ] Details of what I changed are in the commit messages
[ ] References to issues, e.g. close #X, are in the commit messages and changelog
[ ] The buildserver is happy

Checklist

[ ] I fully described what my PR does in the documentation
[ ] I fixed all affected documentation
[ ] I fixed the introduction tour
[ ] I wrote migrations in a way that they are compatible with already present data
[ ] I fixed all affected decisions
[ ] I added automated tests or a manual test protocol
[ ] I added code comments, logging, and assertions as appropriate
[ ] I translated all strings visible to the user
[ ] I mentioned every code or binary not directly written or done by me in reuse syntax
[ ] I created left-over issues for things that are still to be done
[ ] Code is conforming to our Architecture
[ ] Code is conforming to our Guidelines
[ ] Code is consistent to our Design Decisions
[ ] Exceptions to any guidelines are documented

First Time Checklist

[ ] I have installed and I am using pre-commit hooks
[ ] I am using Tailwind CSS Linting

Review

[ ] I've tested the code
[ ] I've read through the whole code
[ ] I've read through the whole documentation
[ ] I've checked conformity to guidelines
[ ] I've checked conformity to requirements
[ ] I've checked that the requirements are tested

There are formatting changes here, as it seems like the linter / pre commit did not work before for me and is now changing things


Update README.md

@ajfriedman18 Can this be closed? I don't think I have the privilege to close it myself.


Is it possible to send file or picture without Internet connection?

Hi! Is it possible to send file or picture without Internet connection? I noticed that in the demo app I can only send location information.

Yes, Bridgefy can send data only in direct messages, you can use the Message.Builder class to configure the outgoing message.
Note: Bridgefy work with Bluetooth Low Energy, so the bandwidth is limited, the data sent will take time depending on the size, we recommend you to send data of reasonable size.


Fix README typos

Howdy! I found a few typos in the README that I thought I would fix! Cheers.

Thanks!


System assigned and user assigned are mixed

It refers to getting the client Id of the system assigned identity. However the script display its retrieves the appId.
az ad sp list --display-name vm-name --query [*].appId --out tsv
I believe it should display how to retrieve the clientId when using user assgined identity and retrieving the appId in case of system assigned identity. That value is used when adding it in the postgress db. The current document is incorrect and created a bit of confusion.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 3a306eb6-f721-1911-0b9f-c7d5c5de47d9
Version Independent ID: db7b5e6c-2e4b-74db-dd90-ed0b1ffb5a54
Content: Connect with Managed Identity - Azure Database for PostgreSQL - Single Server
Content Source: articles/postgresql/single-server/how-to-connect-with-managed-identity.md
Service: postgresql
Sub-service: single-server
GitHub Login: @sunilagarwal
Microsoft Alias: sunila

#assign: @kabharati

#please-close


removing unused dependencies.

Passes E2E tests

Passes E2E tests

Codecov Report

Merging #552 (b5f4005) into master (65b3785) will decrease coverage by 0.00%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #552      +/-   ##
==========================================
- Coverage   85.56%   85.55%   -0.01%
==========================================
  Files         128      128
  Lines        2964     2962       -2
  Branches      100       99       -1
==========================================
- Hits         2536     2534       -2
  Misses        428      428

Impacted Files
Coverage Δ

...main/scala/hydra/kafka/producer/JsonPathKeys.scala
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 65b3785...b5f4005. Read the comment docs.

Codecov Report

Merging #552 (b5f4005) into master (65b3785) will decrease coverage by 0.00%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #552      +/-   ##
==========================================
- Coverage   85.56%   85.55%   -0.01%
==========================================
  Files         128      128
  Lines        2964     2962       -2
  Branches      100       99       -1
==========================================
- Hits         2536     2534       -2
  Misses        428      428

Impacted Files
Coverage Δ

...main/scala/hydra/kafka/producer/JsonPathKeys.scala
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 65b3785...b5f4005. Read the comment docs.


Website is messy

V-Sekai version
https://v-sekai.org/
System information
Windows 11, Nvidia 3000, firefox
Issue description
The website is messy and not modern.

The text is different sized
Colours are too bright

Steps to reproduce
Open https://v-sekai.org/
Minimal reproduction project
https://v-sekai.org/

Reorganized https://github.com/V-Sekai/V-Sekai.github.io. Needs an rereview.


[Design] Initial rollout backend implementation

Summary
The idea of this document is to describe a feature management system that will answer requests from the wallets with the correct feature flag variable depending on their identifier
Motivation
We need this mechanism to be able to safely roll out the new Wallet Service to the wallets, segmenting users with a controlled percentage so we have more control of exposure
The idea here is to have a simpler version of Optimizely
Guide-level explanation
For an initial implementation, these are the features that we need to implement:

Select a percentage of the user base for each feature flag
Use the user wallet's first_address to uniquely identify users
Return a boolean randomly but respecting the rollout percentage
Return the same value every time the same user (identified by the first_address requests the same feature flag
We need to have an environment for each feature flag, e.g. production, development

The API to request the feature flag value is as described:
GET /features/<environment>/<feature_flag>/<identifier>
Response
StatusCode: 200
Body:
{
  "value": true
}

Response on not found
StatusCode: 404
Body: empty
Decisions:
We will not have an audience table. The identifier of the context will be in the FeatureFlag identifier string. E.g.: mobile-wallet_service_rollout
We will have a version column on the UserFeatureFlag and the FeatureFlag tables that increases every time a FeatureFlag is updated. This is used so we will re-calculate the FeatureFlag for the users (on the stored value) every time the FeatureFlag percentage changes.
Reference-level explanation
Database design
UserFeatureFlag
identifier - User identifier, e.g. first_address
feature_flag - The feature_flag identifier, e.g. mobile-wallet_service_rollout
environment - The environment for this stored feature_flag, e.g.: production or staging
value - The stored value for this feature flag for this user, this is randomized at the first request and then stored so we always respond the same value for the same user
version - The FeatureFlag version of this UserFeatureFlag
FeatureFlag
identifier - The feature flag identifier, as a string. E.g.: mobile-wallet_service_rollout
percentage - The percentage to use when deciding if the user
version - This column is incremented every time the row is updated. This is used so we know when to invalidate the UserFeatureFlag for each user on request.
Architecture design
Lambda
Since this service will be hit every time an user opens an Hathor Wallet, it needs to be scalable enough to handle usage spikes
One potential drawback from using lambda is a risk of the cost getting exponentially high, but that can be prevented by always using hard values on functions timeout configurations.
We can also rate limit the APIs as the wallets will fallback to the old facade if they can not reach our feature management APIs.
Redis
The data we store is small enough to fit in memory, even with a large userbase, so I think Redis is the best option to deliver fast response times.
Redis offers different persistence options (from: https://redis.io/topics/persistence):
RDB (Redis Database): The RDB persistence performs point-in-time snapshots of your dataset at specified intervals.
AOF (Append Only File): The AOF persistence logs every write operation received by the server, that will be played again at server startup, reconstructing the original dataset. Commands are logged using the same format as the Redis protocol itself, in an append-only fashion. Redis is able to rewrite the log in the background when it gets too big.
No persistence: If you wish, you can disable persistence completely, if you want your data to just exist as long as the server is running.
RDB + AOF: It is possible to combine both AOF and RDB in the same instance. Notice that, in this case, when Redis restarts the AOF file will be used to reconstruct the original dataset since it is guaranteed to be the most complete.

I think it is fine to use the RDB persistence as losing UserFeatureFlag records (on a forced restart, for instance) is not a catastrophic failure -- the user will just get another randomly generated response from the server on the next request
Rationale and alternatives
We have previously discussed alternatives in https://github.com/HathorNetwork/internal-issues/issues/11
The conclusion was that we were going to use Optimizely as we already had a PoC working, but we found an incompatibility on our react-native version with their library and decided to take a step back and design this simplified solution to have more information before deciding.
I think that this initial implementation is simple enough to be developed and ready in a single sprint to fulfill our current need -- rollout the wallet service initially.

This issue has been replaced by https://github.com/HathorNetwork/hathor-wallet-mobile/issues/117


Make TaskData's fetch more dynamic

Would be nice to still have an easy way to pass the default, static task.
This feature is useful for scripting languages.

I'd consider this one done since #23 which allows to specify an id.


ddev drush create module puts boilerplate code in ddev docker container instead of in git source controlled code on host

Is there an existing issue for this?

[X] I have searched the existing issues

Output of ddev debug test
Expand `ddev debug test` diagnostic information
[COPY-PASTE HERE THE OUTPUT OF `ddev debug test`]

Expected Behavior
ddev create code to be put in my modules folder of my git source controlled folder
Actual Behavior
robdaviswork@192 fsa-cambridge % ddev drush generate module

 Welcome to module generator!
––––––––––––––––––––––––––––––

 Module name:
 ➤ fsa_populate

 Module machine name [fsa_populate]:
 ➤

 Module description:
 ➤ Calls FSA API to get date to create/update Establishment nodes

 Package [Custom]:
 ➤

 Dependencies (comma separated):
 ➤

 Would you like to create module file? [No]:
 ➤ Yes

 Would you like to create install file? [No]:
 ➤ Yes

 Would you like to create README.md file? [No]:
 ➤ Yes

 The following directories and files have been created or updated:
–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
 • /var/www/html/web/modules/fsa_populate/fsa_populate.info.yml
 • /var/www/html/web/modules/fsa_populate/fsa_populate.install
 • /var/www/html/web/modules/fsa_populate/fsa_populate.module
 • /var/www/html/web/modules/fsa_populate/README.md

robdaviswork@192 fsa-cambridge % git status
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean
robdaviswork@192 fsa-cambridge % cd /var/www/html/web/modules/
cd: no such file or directory: /var/www/html/web/modules/
robdaviswork@192 fsa-cambridge %

Steps To Reproduce
No response
Anything else?
searched for existing issues - could not find similar:
https://github.com/ddev/ddev/issues?q=is%3Aissue+"drush+create+module"+

Closed - no problem. For some reason there appears to be a (small?) delay in writing the created module files to the host. They have now appeared in the host source tree.
A new issue could be created to address the apparent delay I observed. I might come back to this later and would welcome thoughts.

Hi @therobyouknow - I imagine you're using Mutagen (default) on macOS. When large changes are made, there is a minor delay before those are synced to the other side.
The DDEV docs on mutagen may help you understand this behavior.
You can also ddev mutagen sync any time to force the completion of the update.


Issue with CUDA and cuDNN

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): stable-baselines library
TensorFlow version: 1.15.3 also TensorFlow-gpu 1.15.0
Python version: 3.7.7
Installed using virtualenv? pip? conda?: Created conda environment using PyCharm
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version: CUDA release 10.0, V10.0.130 / cuDNN 7.4.2
GPU model and memory: NVIDIA GeForce GTX 1050

I am trying to train models with images as observations. I am using A2C and PPO2 with the Pendulum-v1 environment from OpenAI Gym. I am also using stable-baselines, a reinforcement learning library. Here is the complete code that I am using:
import gym
import numpy as np
from gym.envs.classic_control import PendulumEnv
from stable_baselines.common.env_checker import check_env
from stable_baselines.sac.policies import CnnPolicy
from stable_baselines import PPO2

from skimage import data, color
from skimage.transform import rescale, resize, downscale_local_mean

from gym import Wrapper, spaces

import tensorflow as tf
from tensorflow.compat.v1 import ConfigProto
from tensorflow.compat.v1 import InteractiveSession

class RGBArrayAsObservationWrapper(Wrapper):
    """
    Use env.render(rgb_array) as observation
    rather than the observation environment provides
    """

    def __init__(self, env):
        # TODO this might not work before environment has been reset
        super(RGBArrayAsObservationWrapper, self).__init__(env)
        self.reset()
        dummy_obs = env.render('rgb_array')
        dummy_obs_resized = resize(dummy_obs, (dummy_obs.shape[0] // 10, dummy_obs.shape[1] // 10),
                                   anti_aliasing=True)
        # Update observation space
        # TODO assign correct low and high
        self.observation_space = spaces.Box(low=0, high=255, shape=dummy_obs_resized.shape,
                                            dtype=dummy_obs_resized.dtype)

    def reset(self, **kwargs):
        obs = self.env.reset(**kwargs)
        obs = self.env.render("rgb_array")
        obs = resize(obs, (obs.shape[0] // 10, obs.shape[1] // 10),
                     anti_aliasing=True)
        return obs

    def step(self, action):
        obs, reward, done, info = self.env.step(action)
        obs = self.env.render("rgb_array")
        obs = resize(obs, (obs.shape[0] // 10, obs.shape[1] // 10),
                     anti_aliasing=True)
        return obs, reward, done, info

# tensorboard --logdir=PPO2_IMG_PENDULUM:C:\Users\meric\OneDrive\Masaüstü\TUM\Thesis\Pycharm\pioneer\ppo2_pendulum_tensorboard --host localhost

config = ConfigProto()
config.gpu_options.allow_growth = True
session = InteractiveSession(config=config)

TEST_COUNT = 100

pendulum_env = PendulumEnv()
pendulum_env = RGBArrayAsObservationWrapper(pendulum_env)
check_env(pendulum_env, warn=True)

model = PPO2("CnnPolicy", pendulum_env, verbose=1, tensorboard_log="./ppo2_pendulum_tensorboard/")
model.learn(total_timesteps=100_000, log_interval=10)
model.save("ppo2_pendulum")

sum_rewards = 0
done = False
obs = pendulum_env.reset()
for i in range(TEST_COUNT):
    while not done:
        action, _states = model.predict(obs)
        obs, rewards, done, info = pendulum_env.step(action)
        pendulum_env.render()
        sum_rewards += rewards

    pendulum_env.reset()
    done = False

print(sum_rewards / TEST_COUNT)

And here is the complete output in terminal.
2020-07-08 13:35:50.450192: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

2020-07-08 13:35:53.202978: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2020-07-08 13:35:53.206120: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2020-07-08 13:35:53.236974: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce GTX 1050 major: 6 minor: 1 memoryClockRate(GHz): 1.493
pciBusID: 0000:01:00.0
2020-07-08 13:35:53.237468: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
2020-07-08 13:35:53.242817: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll
2020-07-08 13:35:53.246183: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_100.dll
2020-07-08 13:35:53.247490: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_100.dll
2020-07-08 13:35:53.252276: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_100.dll
2020-07-08 13:35:53.255771: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_100.dll
2020-07-08 13:35:53.268419: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-07-08 13:35:53.268904: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-07-08 13:35:53.903957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-07-08 13:35:53.904131: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0
2020-07-08 13:35:53.904234: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N
2020-07-08 13:35:53.904599: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3001 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050, pci bus id: 0000:01:00.0, compute capability: 6.1)
C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\stable_baselines\common\env_checker.py:25: UserWarning: It seems that your observation is an image but the `dtype` of your observation_space is not `np.uint8`. If your observation is not an image, we recommend you to flatten the observation to have only a 1D vector
  warnings.warn("It seems that your observation is an image but the `dtype` "
C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\stable_baselines\common\env_checker.py:210: UserWarning: We recommend you to use a symmetric and normalized Box action space (range=[-1, 1]) cf https://stable-baselines.readthedocs.io/en/master/guide/rl_tips.html
  warnings.warn("We recommend you to use a symmetric and normalized Box action space (range=[-1, 1]) "
Wrapping the env in a DummyVecEnv.
WARNING:tensorflow:From C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\stable_baselines\common\tf_util.py:191: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\stable_baselines\common\tf_util.py:200: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2020-07-08 13:35:55.069303: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce GTX 1050 major: 6 minor: 1 memoryClockRate(GHz): 1.493
pciBusID: 0000:01:00.0
2020-07-08 13:35:55.069545: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
2020-07-08 13:35:55.069705: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll
2020-07-08 13:35:55.069866: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_100.dll
2020-07-08 13:35:55.070064: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_100.dll
2020-07-08 13:35:55.070230: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_100.dll
2020-07-08 13:35:55.070391: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_100.dll
2020-07-08 13:35:55.070552: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-07-08 13:35:55.070737: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-07-08 13:35:55.070901: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-07-08 13:35:55.071070: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0
2020-07-08 13:35:55.071172: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N
2020-07-08 13:35:55.071344: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3001 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050, pci bus id: 0000:01:00.0, compute capability: 6.1)
WARNING:tensorflow:From C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\stable_baselines\common\policies.py:116: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

WARNING:tensorflow:From C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\stable_baselines\common\input.py:25: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\stable_baselines\common\tf_layers.py:103: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

WARNING:tensorflow:From C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\stable_baselines\common\distributions.py:418: The name tf.random_normal is deprecated. Please use tf.random.normal instead.

WARNING:tensorflow:From C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\stable_baselines\ppo2\ppo2.py:190: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

WARNING:tensorflow:From C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\stable_baselines\ppo2\ppo2.py:198: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead.

WARNING:tensorflow:From C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\tensorflow_core\python\ops\math_grad.py:1424: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\stable_baselines\ppo2\ppo2.py:206: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

WARNING:tensorflow:From C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\stable_baselines\ppo2\ppo2.py:240: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.

WARNING:tensorflow:From C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\stable_baselines\ppo2\ppo2.py:242: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

WARNING:tensorflow:From C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\stable_baselines\common\base_class.py:1169: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

2020-07-08 13:35:55.970675: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll
2020-07-08 13:35:56.208807: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-07-08 13:35:57.076059: E tensorflow/stream_executor/cuda/cuda_dnn.cc:319] Loaded runtime CuDNN library: 7.4.2 but source was compiled with: 7.6.0.  CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library.  If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2020-07-08 13:35:57.079198: E tensorflow/stream_executor/cuda/cuda_dnn.cc:319] Loaded runtime CuDNN library: 7.4.2 but source was compiled with: 7.6.0.  CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library.  If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
Traceback (most recent call last):
  File "C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\tensorflow_core\python\client\session.py", line 1365, in _do_call
    return fn(*args)
  File "C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\tensorflow_core\python\client\session.py", line 1350, in _run_fn
    target_list, run_metadata)
  File "C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\tensorflow_core\python\client\session.py", line 1443, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[{{node model/c1/Conv2D}}]]
	 [[output/strided_slice_1/_9]]
  (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[{{node model/c1/Conv2D}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/meric/OneDrive/Masaüstü/TUM/Thesis/Pycharm/pioneer/pendulum_image_PPO2.py", line 65, in <module>
    model.learn(total_timesteps=100_000, log_interval=10)
  File "C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\stable_baselines\ppo2\ppo2.py", line 336, in learn
    rollout = self.runner.run(callback)
  File "C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\stable_baselines\common\runners.py", line 48, in run
    return self._run()
  File "C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\stable_baselines\ppo2\ppo2.py", line 472, in _run
    actions, values, self.states, neglogpacs = self.model.step(self.obs, self.states, self.dones)
  File "C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\stable_baselines\common\policies.py", line 576, in step
    {self.obs_ph: obs})
  File "C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\tensorflow_core\python\client\session.py", line 956, in run
    run_metadata_ptr)
  File "C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\tensorflow_core\python\client\session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\tensorflow_core\python\client\session.py", line 1359, in _do_run
    run_metadata)
  File "C:\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\tensorflow_core\python\client\session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[node model/c1/Conv2D (defined at \Users\meric\Anaconda3\envs\pioneer\lib\site-packages\tensorflow_core\python\framework\ops.py:1748) ]]
	 [[output/strided_slice_1/_9]]
  (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[node model/c1/Conv2D (defined at \Users\meric\Anaconda3\envs\pioneer\lib\site-packages\tensorflow_core\python\framework\ops.py:1748) ]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'model/c1/Conv2D':
  File "/Users/meric/OneDrive/Masaüstü/TUM/Thesis/Pycharm/pioneer/pendulum_image_PPO2.py", line 64, in <module>
    model = PPO2("CnnPolicy", pendulum_env, verbose=1, tensorboard_log="./ppo2_pendulum_tensorboard/")
  File "\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\stable_baselines\ppo2\ppo2.py", line 97, in __init__
    self.setup_model()
  File "\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\stable_baselines\ppo2\ppo2.py", line 131, in setup_model
    n_batch_step, reuse=False, **self.policy_kwargs)
  File "\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\stable_baselines\common\policies.py", line 602, in __init__
    feature_extraction="cnn", **_kwargs)
  File "\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\stable_baselines\common\policies.py", line 559, in __init__
    pi_latent = vf_latent = cnn_extractor(self.processed_obs, **kwargs)
  File "\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\stable_baselines\common\policies.py", line 25, in nature_cnn
    layer_1 = activ(conv(scaled_images, 'c1', n_filters=32, filter_size=8, stride=4, init_scale=np.sqrt(2), **kwargs))
  File "\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\stable_baselines\common\tf_layers.py", line 107, in conv
    return bias + tf.nn.conv2d(input_tensor, weight, strides=strides, padding=pad, data_format=data_format)
  File "\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\tensorflow_core\python\ops\nn_ops.py", line 2010, in conv2d
    name=name)
  File "\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\tensorflow_core\python\ops\gen_nn_ops.py", line 1071, in conv2d
    data_format=data_format, dilations=dilations, name=name)
  File "\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\tensorflow_core\python\framework\op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "\Users\meric\Anaconda3\envs\pioneer\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()

Process finished with exit code 1

My code was running smoothly before I tried putting cufft64_100.dll, curand64_100.dll, cusolver64_100.dll, cusparse64_100.dll, cudnn64_7.dll in my environment library in the anaconda3 folder.  My goal is to use my GPU during training so it is faster. Any help would be appreciated. I am also okay with deleting my anaconda environment and uninstalling all the CUDA and cuDNN files and starting from scratch if someone could help me on how to do this correctly. I need tensorflow 1.15 any other version does not work for me.

@meric-sakarya
Please share complete code with all dependencies as i ran your code and face different error, please find the gist here.
With respect to the error faced, refer to below issues:
#37725 #34355 #24828 link link1
Also is there any particular reason to be using 1.x when there are new versions of tf.

I am using 1.15 because stable-baselines supports no newer version. You should install stable baselines using this command maybe that is causing your error. pip install git+https://github.com/hill-a/stable-baselines
Also i am using tensorflow-gpu==1.15 and you are only installing regular tensorflow. This is the whole code that I am using. You also should install sci-kit image with pip install scikit-image

This generally happens when CUDA and CUDNN versions are not compatible with each other
Try changing your CUDNN version 7.3 for CUDA version 10.0.130

@meric-sakarya
Please update as per above comment.
I ran the code with the dependencies shared and face a different error, please find gist here

I updated CUDA to 10.1 and CUDNN to 7.6.5 I also downloaded some files from an earlier version of CUDA(10.0) and added them to my NVIDIA GPU Computing Toolkit\bin folder
(link for download:https://drive.google.com/file/d/1GNx7RYty9wq7TJ8UC0A6Pw_nYGQ1qmep/view).
My code does not give any errors anymore. I also added the said bin folder to my PATH. My current problem is the slow training of my model. Unfortunately making these adjustments did not speed up the process. My GPU usage is around %5 only. Any suggestions on that?

@meric-sakarya  Please roll back to cuda 10.0 since you are using TF 1.15
TF 1.15 pre built binary does not support cuda 10.1
Further we should try limiting your gpu memory growth and also reduce batch_size to avoid the error you are facing.
import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
sess = tf.Session(config=config)
 # Rest of your code

Look like you have removed Path or cudart runtime so tf skips GPU(maybe).
You can debug Cuda GPU from https://github.com/dailylifesking/CUDA_not_working/blob/master/cuda%20not%20found.ipynb

Look like you have removed Path or cudart runtime so tf skips GPU(maybe).
You can debug Cuda GPU from https://github.com/dailylifesking/CUDA_not_working/blob/master/cuda_not_found_in_tensorflow.ipynb

How can I debug with this?

@meric-sakarya Does limiting gpu memory help in your case?

I could not debug using the given link. I rolled back to CUDA 10.0 and limited my GPU usage with import tensorflow as tf config = tf.ConfigProto() config.gpu_options.allow_growth = True sess = tf.Session(config=config)

Any help?

Just to clarify, your cuda and cudnn version mismatch issue is solved and now you are not seeing maximum gpu utilization is that correct?
If that is the case can you try manual device placement?
import tensorflow
tf.debugging.set_log_device_placement(True)
# Place tensors on the GPU
with tf.device('/GPU:0'):
  # Your code

Yes I am definitely not seeing max gpu utilization. I tried the code nothing changed.

Yes I am definitely not seeing max gpu utilization. I tried the code nothing changed.

Can you try using the TensorBoard profiler to see if something sticks out?

TensorFlow Profiler: Profile model performance

View on TensorFlow.org

Run in Google Colab

View source on GitHub

Overview
Machine learning algorithms are typically computationally expensive. It is thus vital to quantify the performance of your machine learning application to ensure that you are running the most optimized version of your model. Use the TensorFlow Profiler to profile the execution of your TensorFlow code.
Setup
from datetime import datetime
from packaging import version
import os
The TensorFlow Profiler requires the latest versions of TensorFlow and TensorBoard (>=2.2).
!pip install -U tensorboard_plugin_profile
import tensorflow as tf
print("TensorFlow version: ", tf.version)
Confirm that TensorFlow can access the GPU.
device_name = tf.test.gpu_device_name()
if not device_name:
raise SystemError('GPU device not found')
print('Found GPU at: {}'.format(device_name))
Train an image classification model with TensorBoard callbacks
In this tutorial, you explore the capabilities of the TensorFlow Profiler by capturing the performance profile obtained by training a model to classify images in the MNIST dataset.
Use TensorFlow datasets to import the training data and split it into training and test sets.
import tensorflow_datasets as tfds
tfds.disable_progress_bar()
(ds_train, ds_test), ds_info = tfds.load(
'mnist',
split=['train', 'test'],
shuffle_files=True,
as_supervised=True,
with_info=True,
)
Preprocess the training and test data by normalizing pixel values to be between 0 and 1.
def normalize_img(image, label):
"""Normalizes images: uint8 -> float32."""
return tf.cast(image, tf.float32) / 255., label
ds_train = ds_train.map(normalize_img)
ds_train = ds_train.batch(128)
ds_test = ds_test.map(normalize_img)
ds_test = ds_test.batch(128)
Create the image classification model using Keras.
model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28, 1)),
tf.keras.layers.Dense(128,activation='relu'),
tf.keras.layers.Dense(10, activation='softmax')
])
model.compile(
loss='sparse_categorical_crossentropy',
optimizer=tf.keras.optimizers.Adam(0.001),
metrics=['accuracy']
)
Create a TensorBoard callback to capture performance profiles and call it while training the model.
Create a TensorBoard callback
logs = "logs/" + datetime.now().strftime("%Y%m%d-%H%M%S")
tboard_callback = tf.keras.callbacks.TensorBoard(log_dir = logs,
histogram_freq = 1,
profile_batch = '500,520')
model.fit(ds_train,
epochs=2,
validation_data=ds_test,
callbacks = [tboard_callback])
Use the TensorFlow Profiler to profile model training performance
The TensorFlow Profiler is embedded within TensorBoard. Load TensorBoard using Colab magic and launch it. View the performance profiles by navigating to the Profile tab.
Load the TensorBoard notebook extension.
%load_ext tensorboard
The performance profile for this model is similar to the image below.
Launch TensorBoard and navigate to the Profile tab to view performance profile
%tensorboard --logdir=logs

The Profile tab opens the Overview page which shows you a high-level summary of your model performance. Looking at the Step-time Graph on the right, you can see that the model is highly input bound (i.e., it spends a lot of time in the data input piepline). The Overview page also gives you recommendations on potential next steps you can follow to optimize your model performance.
To understand where the performance bottleneck occurs in the input pipeline, select the Trace Viewer from the Tools dropdown on the left. The Trace Viewer shows you a timeline of the different events that occured on the CPU and the GPU during the profiling period.
The Trace Viewer shows multiple event groups on the vertical axis. Each event group has multiple horizontal tracks, filled with trace events. The track is an event timeline for events executed on a thread or a GPU stream. Individual events are the colored, rectangular blocks on the timeline tracks. Time moves from left to right. Navigate the trace events by using the keyboard shortcuts W (zoom in), S (zoom out), A (scroll left), and D (scroll right).
A single rectangle represents a trace event. Select the mouse cursor icon in the floating tool bar (or use the keyboard shortcut 1) and click the trace event to analyze it. This will display information about the event, such as its start time and duration.
In addition to clicking, you can drag the mouse to select a group of trace events. This will give you a list of all the events in that area along with an event summary. Use the M key to measure the time duration of the selected events.
Trace events are collected from:

CPU: CPU events are displayed  under an event group named /host:CPU. Each track represents a thread on CPU. CPU events include input pipeline events, GPU operation (op) scheduling events, CPU op execution events etc.
GPU: GPU events are displayed under event groups prefixed by /device:GPU:. Each event group represents one stream on the GPU.

Debug performance bottlenecks
Use the Trace Viewer to locate the performance bottlenecks in your input pipeline. The image below is a snapshot of the performance profile.

Looking at the event traces, you can see that the GPU is inactive while the tf_data_iterator_get_next op is running on the CPU. This op is responsible for processing the input data and sending it to the GPU for training. As a general rule of thumb, it is a good idea to always keep the device (GPU/TPU) active.
Use the tf.data API to optimize the input pipeline. In this case, let's cache the training dataset and prefetch the data to ensure that there is always data available for the GPU to process. See here for more details on using tf.data to optimize your input pipelines.
(ds_train, ds_test), ds_info = tfds.load(
'mnist',
split=['train', 'test'],
shuffle_files=True,
as_supervised=True,
with_info=True,
)
ds_train = ds_train.map(normalize_img)
ds_train = ds_train.batch(128)
ds_train = ds_train.cache()
ds_train = ds_train.prefetch(tf.data.experimental.AUTOTUNE)
ds_test = ds_test.map(normalize_img)
ds_test = ds_test.batch(128)
ds_test = ds_test.cache()
ds_test = ds_test.prefetch(tf.data.experimental.AUTOTUNE)
Train the model again and capture the performance profile by reusing the callback from before.
model.fit(ds_train,
epochs=2,
validation_data=ds_test,
callbacks = [tboard_callback])
Re-launch TensorBoard and open the Profile tab to observe the performance profile for the updated input pipeline.
The performance profile for the model with the optimized input pipeline is similar to the image below.
%tensorboard --logdir=logs

From the Overview page, you can see that the Average Step time has reduced as has the Input Step time. The Step-time Graph also indicates that the model is no longer highly input bound. Open the Trace Viewer to examine the trace events with the optimized input pipeline.

The Trace Viewer shows that the tf_data_iterator_get_next op executes much faster. The GPU therefore gets a steady stream of data to perform training and achieves much better utilization through model training.
Summary
Use the TensorFlow Profiler to profile and debug model training performance. Read the Profiler guide and watch the Performance profiling in TF 2 talk from the TensorFlow Dev Summit 2020 to learn more about the TensorFlow Profiler.

Hi There,
We are checking to see if you still need help on this, as you are using an older version of tensorflow which is officially considered end of life . We recommend that you upgrade to the latest 2.x version and let us know if the issue still persists in newer versions. Please open a new issue for any help you need against 2.x, and we will get you the right help.
This issue will be closed automatically 7 days from now. If you still need help with this issue, please provide us with more information.


[Peers] configure voltage levels and termination voltage gRPC example in digitalpattern

What does this Pull Request accomplish?
A C example. NI Digital Configure Voltage Levels and Termination Voltage, is ported to python using the NIDigital gRPC APIs
Why should this Pull Request be merged?
This example can be used to test and understand the generated NIDigital APIs
What testing has been done?
Manual

Extra commits are captured


Improve discussion of df-sb

Add an explicit example to make clear what is being substituted for what.
Also, reorder the explanation to make it clearer.
Basically, describe what substitution does first, then why we
use that notation as opposed to some others, and only then
discuss a technical trick that the definition uses.
A reader will have a harder time understanding the technical trick if he
doesn't first understand what the definition is trying to accomplish.
Signed-off-by: David A. Wheeler<EMAIL_ADDRESS>

Once I added the example to the appendix, I realized the same clarification should go into the body.  This change does that and hopefully improves the description of substitution.

"$y$ is properly substituted with $x$"
should be
"$y$ is properly substituted for $x$"
("with" would mean the opposite.)

  Thanks! Hopefully it's all correct now...

I suggest one final change:
$[ x / y ] z \in y$ is the same as $z \in x$.
change to:
$[ y / x ] z \in x$ is the same as $z \in y$.
because it may be confusing, or at least distracting, to use [y/x] in some places but [x/y] in others.  (In set.mm, [y/x] seems usual, and I'm not sure why elsb4 uses [x/y].)
Aside from that, go ahead and merge.

I can do that, but we normally sort symbols alphabetically left-to-right if there is no reason to otherwise.  Shouldn't this be the default form when there's no reason to do otherwise?:
[ x / y ] z \in y

Here is what "conventions" says:
       Propositional variables (variables for well-formed formulas or wffs) are
       represented with lowercase Greek letters and are normally used
       in this order:
       ` ph ` = phi, ` ps ` = psi, ` ch ` = chi, ` th ` = theta,
       ` ta ` = tau, ` et ` = eta, ` ze ` = zeta, and ` si ` = sigma.
       Individual set variables are represented with lowercase Latin letters
       and are normally used in this order:
       ` x ` , ` y ` , ` z ` , ` w ` , ` v ` , ` u ` , and ` t ` .
       Variables that represent classes are often represented by
       uppercase Latin letters:
       ` A ` , ` B ` , ` C ` , ` D ` , ` E ` , and so on.

Yes, I know the convention says that, although it does say "normally" giving us an out. :)
I believe my original thinking was that we start with a wff ph that has an "x", then we add a prefix that changes the x to a second variable y.  In addition, I think an early notation may have used something like "( ph x | y )" similar to the literature ph(x}y), meaning Curry's "[ y / x ] ph", although I'd have to go back to check very early set.mm's (which may be on old DEC/VMS TK50 backup tapes in my closet that I have no way to read).
(Later:  I found it - see df-sb as "( ph x | y )" in set.mm.1998-03-16-from-vax, which is the very first commit in the set.mm repository.)
In any case, the order isn't that important to me as long as there is consistency, and changing [x/y] to [y/x] would be less work.  If people feel it's important enough I could go the other way, but someone would have to volunteer to do the work.

I believe my original thinking was that we start with a wff ph that has an "x", then we add a prefix that changes the x to a second variable y.

A ph is more likely to start with x (not y), so this is a perfectly rational reason to make an exception.  I just think it needs to be documented in "conventions". E.g., "An exception is when describing substitutions.  Since expressions like ph often have an x to start with, we typically show substitution rules where x is being replaced, and that results in a slightly different order. E.g., a rule might show as [ y / x ] ph , that is, y  is properly substituted for x .


Use !!match rather than match? for Regexp in Rubies before 2.4

Regexp#match? was introduced in ruby 2.4; this change will maintain compatibility with older rubies like 2.3.

Hey @Capncavedan
Thanks a lot for your time!
Sorry, but I don’t see why we need this after all this time.
Especially considering the fact that ruby 2.3 has reached the end-of-life stage https://www.ruby-lang.org/en/downloads/branches/

@m0n9oose I can understand your position; this seems like a straightforward change, though.
Perhaps a note at the top of the README indicating only 2.4+ is supported would be in order.


feat: add preventAutoDelete method to ClassHandle

This PR adds  .preventAutoDelete() method to allow users to explicitly prevent an object from being automatically deleted when .setAutoDeleteLater is set to true. This is quite useful when you need a long-lived handle and want to make sure it doesn't get auto-deleted with other short-lived handles.
const vips = await Vips(options);
vips.setAutoDeleteLater(true);
...
const im = new vips.Image(...);
im.preventAutoDelete();
...
const im2 =  new vips.Image(...);
im2.resize(...).crop(...).gaussblur(...);

// deletes all 4 handles created by im2 chain
vips.flushPendingDeletes();

im.isDeleted(); // false

Without this method the current alternative if you needed a long-lived handle was to .setAutoDeleteLater(false); and call .deleteLater after each operation
const im2 =  new vips.Image(...);
im2.deleteLater()
   .resize(...)
   .deleteLater()
   .crop(...)
   .deleteLater()
   .gaussblur(...)
   .deleteLater();

Hi @kleisauke let me know if you'd like to have this or something similar upstream.

Yes please, that would be great.


Detector status report

Updates the diagnostic models to use the latest configuration data and creates two new reports in the marts folder for BI tool and reporting usage.  Fixes #269.

Created a new branch to perform this work and is not included in #332


Error on invalid commandline argument

Previously the parsing would continue in an endless loop. This behavior is now
removed by erroring on invalid input.

Tried and works.


worker_init_fn is not set to seed dataloaders correctly when using DDP

🐛 Bug
When seed_everything(workers=True) is called, it will set the environment variable PL_SEED_WORKERS=1. Consequently Trainer will set the worker_init_fn for dataloaders to pl_worker_init_function. It seems to me that worker_init_fn is not set when using DDP. The reason is that DDPPlugin.setup_environment() eventually runs reset_seed(), which reads the value of the PL_GLOBAL_SEED environment value and calls seed_everything() with the default argument workers=False.
Please reproduce using the BoringModel
It's not possible to reproduce the issue in colab, since it doesn't support DDP.
To Reproduce
Here's a simple program that demonstrates the issue:
import os

import torch
from torch.utils.data import DataLoader, Dataset

from pytorch_lightning import LightningModule, Trainer
from pytorch_lightning import seed_everything

class RandomDataset(Dataset):

    def __init__(self, size, length):
        self.len = length
        self.data = torch.randn(length, size)

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return self.len

class BoringModel(LightningModule):

    def __init__(self):
        super().__init__()
        self.layer = torch.nn.Linear(32, 2)

    def forward(self, x):
        return self.layer(x)

    def training_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("train_loss", loss)
        return {"loss": loss}

    def validation_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("valid_loss", loss)

    def test_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("test_loss", loss)

    def configure_optimizers(self):
        return torch.optim.SGD(self.layer.parameters(), lr=0.1)

def run():
    seed_everything(1234, workers=True)
    # Sets PL_SEED_WORKERS=1
    print('PL_SEED_WORKERS=' + os.environ['PL_SEED_WORKERS'])

    train_data = DataLoader(RandomDataset(32, 64), batch_size=2)
    val_data = DataLoader(RandomDataset(32, 64), batch_size=2)
    test_data = DataLoader(RandomDataset(32, 64), batch_size=2)

    model = BoringModel()
    trainer = Trainer(
        default_root_dir=os.getcwd(),
        limit_train_batches=1,
        limit_val_batches=1,
        num_sanity_val_steps=0,
        max_epochs=1,
        weights_summary=None,
        gpus=2,
        accelerator='ddp'  # Using accelerator='dp' works
    )
    trainer.fit(model, train_dataloader=train_data, val_dataloaders=val_data)
    # Trainer.accelerator.setup_environment() calls DDPPlugin.setup_environment(),
    # which eventually runs seed_everything(workers=False) that sets PL_SEED_WORKERS=0
    # Consequently dataloader.worker_init_fn is not set.
    print('PL_SEED_WORKERS=' + os.environ['PL_SEED_WORKERS'])

if __name__ == '__main__':
    run()

Expected behavior
I would expect pl_worker_init_function to be called. By printing something from the function, I can see that it's called if I use dp accelerator, but not if I use ddp. I can also notice that the environment variable PL_SEED_WORKERS is reset to 0 during the Trainer.fit() call, but I would expect it to have the value 1 in the end.
I think the correct fix would be to make reset_seed() read the PL_SEED_WORKERS environment variable too and pass the corresponding workers argument to seed_everything(). However, I'm not familiar enough with the code to be sure that this is correct.
Preferably pl_worker_init_function would also display a log message that confirms that the workers are seeded correctly.
Environment

CUDA:

GPU:

NVIDIA Tesla V100-SXM2-16GB
NVIDIA Tesla V100-SXM2-16GB

available:         True
version:           11.0

Packages:

numpy:             1.19.2
pyTorch_debug:     True
pyTorch_version:   1.7.0
pytorch-lightning: 1.4.0dev
tqdm:              4.51.0

System:

OS:                Linux
architecture:

64bit
ELF

processor:         x86_64
python:            3.8.8
version:           #47-Ubuntu SMP Tue May 11 15:51:42 UTC 2021

Additional context
Recently there was discussion about an issue with data loading, where the same NumPy random seed is used across different workers. This causes the workers the use the same random numbers for data transforms. A fix was quickly introduced in PyTorch Lightning that seeds the dataloaders correctly by automatically setting the  worker_init_fn for dataloaders.

Thanks for testing this feature out!
In seed_everything we have this line:
os.environ["PL_SEED_WORKERS"] = f"{int(workers)}"

should we change it to
os.environ["PL_SEED_WORKERS"] = os.get("PL_SEED_WORKERS", False) or f"{int(workers)}"

? And if so would you like to test if this change works for you?

should we change it to
os.environ["PL_SEED_WORKERS"] = os.get("PL_SEED_WORKERS", f"{int(workers)}")

? And if so would you like to test if this change works for you?

os.environ["PL_SEED_WORKERS"] = os.environ.get("PL_SEED_WORKERS", f"{int(workers)}") works. It means that seed_everything() will ignore the workers argument if PL_SEED_WORKERS is set, but it works too (I already tested).

It means that seed_everything() will ignore the workers argument if PL_SEED_WORKERS is set,

Maybe a better fix would be to actually change reset_seed() to call seed_everything with workers=bool(os.environ.get("PL_SEED_WORKERS", False))
so that we don't have to ignore the argument when someone does:
seed_everything(123, workers=True)

seed_everything(123, workers=False) # don't ignore this, turn it actually off

Maybe a better fix would be to actually change reset_seed() to call seed_everything with workers=bool(os.environ.get("PL_SEED_WORKERS", False))

Right. That was my initial thought too. Also, what do you think about writing some log message in pl_worker_init_function() to confirm that the data loaders have been initialized correctly?

would you be interested in sending a PR for the fix, or the log message, or both? :) happy to help wherever.

would you be interested in sending a PR for the fix, or the log message, or both? :) happy to help wherever.

I can do that.

The pull request is created: https://github.com/PyTorchLightning/pytorch-lightning/pull/7942
Didn't have a chance to run the test suite yet.


Remove Vaccinated tooltip body definition

Per decision in copy doc to remove "Percent Vaccinated" definition:
Netlify won't let you add an "empty" required field, and it didn't seem worth updating the plumbing to allow this to be optional throughout the codebase, so I've just edited this manually.
e.g.

instead of:

So to be clear, I think this PR can be dropped.

ah--okay! sounds good, and thanks for taking a look 🙏


Create Client in Hydra-webui

Run hydra-webui
Goto http://<IP_ADDRESS>:3000
Create Client should work

@Luv7804 What are you getting? Is there any error?

504 Gateway Timeout

quick setup of Ory Hydra
https://www.ory.sh/docs/hydra/5min-tutorial


When creating a new organization as a Site Admin, don't require password fields

Summary
The passwords are supposed to be set by the organization when they sign up, initially (via one of our Devise mixins... Inviteable I think?). Our "Add New Organization" (admin/organizations/new) form currently requires we enter a password even though I'm almost certain it does not do anything.
Acceptance Criteria

[ ] Verify that the provided password does nothing and that the new user signs up via a "set your password" form
[ ] Remove the password fields from this page and ensure organizations can be created without a password
[ ] Update the specs for this, or add a new spec for adding a new organization

I'm happy to help with this one!

@AutumnJ Go for it!

@AutumnJ Hello! :) Have you been able to make any headway with this?

@armahillo I have made some progress! I followed the pattern in Devise's documentation to create a user using their invite! method, instead of persisting a user (which requires a password), then sending an invitation, which is what is currently occurring in the organizations controller. Unfortunately, I realized that this would not surface email validation errors to users, even with config.validate_on_invite = true. So the organization would be created but the user creation and invitation silently fail, in the event the email is invalid.
If that would be a problem, we could still remove the password fields and use Devise invitable by following a pattern like this, allowing us to skip the password validation on this form alone, but still surface meaningful errors to users.
Would you prefer one approach over the other?

@AutumnJ For reference, the current workflow, as I recall, is to put junk into the password fields (MATCHING junk), invite the organization, and then they set the password when they sign-in. It might be enough to just not display the password fields at all. Would probably need to override the devise controller for this to work, though. (Either InvitationsController or RegistrationsController I think -- we've got materialized controllers for both) -- the override would just generate a randomized secure password and assign it.
The invite! path you're going down might be the ticket -- the email validation isn't a huge problem; the only people that would be entering emails here are Site Admins (this particular form is on the Site Admin dashboard) so we have some latitude with expected data.

According to @seanmarcia You might be able to just hide the password fields entirely and that might work?

That approach would work, if we used invite! to create the user and send an invitation email, but we're actually creating and persisting the user when the organization is created and saved (prior to sending an invite), which causes the validation to fail with the password fields removed.
I'll use SecureRandom to generate a temporary password on the backend. That seems to be the safest and easiest approach (refactoring to rely on invite! for user creation could introduce other issues). Will have a PR up in the next day or so!


Allow passing args to Order

Allow passing arguments to Order clauses.

What does this address? Also, could you please add a test case?

This addresses that there is currently no way to safely pass arguments to ORDER BY clauses.
An actual use case I have is for a primitive ranking of search results, where results that match name ILIKE '%query' are sorted before results that have the search query elsewhere in the string.

Thank you for the explanation!

If you merge the development branch into yours the tests should pass!

TY!


Manage private repository.

Add GitHub token to main remote if the repository is private.
Fixes #10

@ldez when we gonna see it merged? 👍


YML-based test automation

Triggered by https://github.com/christophwille/dotnet-opa-wasm/issues/3#issuecomment-955959904
The inner-loop tests should definitely remain in the current unit test project. However, supporting running build verification tests using eg https://github.com/open-policy-agent/opa/tree/main/test/cases/testdata is definitely desireable.
Maybe wrap this in a separate test project so they are split from the inner loop tests. How to include the tests is the 1m question... copy is a no-no, on the other hand I don't want to drag in submodules "just" for this. Ideas welcome.

To pick on a specific example: https://github.com/iinuwa/dotnet-opa-wasm/commit/a73dbc76fb8ce33e979708722541795da0479d8c
The C#-implemented SDK builtins should be tested directly from a C# unit test, not indirectly via rego (I would consider the latter an "integration test").

https://github.com/open-policy-agent/opa/issues/3083#issuecomment-954558982

How to include the tests is the 1m question... copy is a no-no, on the other hand I don't want to drag in submodules "just" for this. Ideas welcome.

What do you mean by copying here? Do you mean you don't want files checked into the repo? If so, the npm-opa-sdk CI downloads the opa repo as a tar and then extracts the subdirectory while running the tests.

The C#-implemented SDK builtins should be tested directly from a C# unit test, not indirectly via rego (I would consider the latter an "integration test").

Agreed. Are you saying that you want the YAML-based tests to be unit or integration tests?

Correct, I don't want the files in this repo - the build automation doing the downloading could be an option. And when I am talking about those "third party tests", I consider them "SDK validation tests" (in a sense we are reusing somebody else's unit tests)


Add a loading page

There should be a loading page displayed as soon as someone hits the mybinder.org/repo/*/* end point, while being redirected, until their binder is ready.

It should briefly explain the expected latency and possible necessity of a refresh -- prior to the redirect.
This way the user is forewarned, and hopefully will not dismiss the site altogether.


Subgraph gets stuck when eth_getLogs fails because there are too many results

Do you want to request a feature or report a bug?
Bug
What is the current behavior?
When catching up with historical data, the block stream asks for eth_getLogs on 100,000 blocks at once. It is possible that the eth node (Infura in my case) responds with
  Error { code: ServerError(-32005), message: "query returned more than 1000 results", data: None }

We blindly retry the request, even though there is no chance retrying it will ever succeed.
If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem.
This was observed with the Dice2Win subgraph, with the subgraph at block 6290742, while the ethereum head ptr was at 7271924.
What is the expected behavior?
The code should realize that it has to reduce the size of the range (e.g., by repeatedly halving it) until this error isn't returned any more. When I manually reduced the range to 100 blocks in the code, the block stream started working again.
As a sidenote, the logs should show more details about what exactly went wrong when eth_getLogs fails. Right now it just says something like Trying again after eth_getLogs RPC call failed (attempt #15) which does not have enough detail to understand what's happening.

Interesting. This is long after the node requests eth_getLogs for 100,000 blocks at once. The fast scan algorithm ends at 4,000,000 blocks (by default) and 3,000,000 blocks (hosted service, IIRC).
After that I believe we ask for chunks of 10,000 at a time (with several of such 10,000 block requests running in parallel).
The 1000 results limit must be new. I agree with the proposed solution to retry with smaller ranges. It may also make sense to start out with a smaller block range size in the first place.

The thing that made me mention 100,000 in the above is that things started working again when I changed the 100_000 on this line to 100, after trying 10_0001 and 1_000` to no avail. From what you said, there might be another problem with not backing off large ranges as intended.

That coffee eventually ends up calling https://github.com/graphprotocol/graph-node/blob/a5cad36f3943ed3f7d6401df4c045ed5bc80a7f9/datasource/ethereum/src/ethereum_adapter.rs#L96, which has the chunking logic I talked about.
First I was thinking of baking a range backoff into that log_stream function but that could get pretty messy. I think the best approach might be too wrap the body of log_stream into a back-off-retry future of sorts.

@lutter Have we observed this since?

Nope, haven't seen it since then. Closing it; will reopen if I ever run into it again.

We've now seen this again with Decentraland

Yep. I agree the ideal solution would half the block range size until it either reaches 1 and still fails (in which case we'd either fail the subgraph or keep retrying indefinitely; the former would make it easier for us to catch the problem) or succeeds.
When it succeeds with a certain block range size, we should either increase it again for the next batch of eth_getLogs requests or stay on this size for a certain number of batches until we try the initialize range size again.

I've observed this happening even with no relevant logs in the range. My theory is that the bloom filter has a lot of false positives and Infura is counting those for the 1000 limit.


Add icon only docs for Button

Closes
Add icon-only entry for Button in docs
✅ Pull Request Checklist:

[ ] Included link to corresponding React Spectrum GitHub Issue.
[ ] Added/updated unit tests and storybook for this change (for new code or code which already has tests).
[ ] Filled out test instructions.
[ ] Updated documentation (if it already exists for this component).
[ ] Looked at the Accessibility Practices for this feature - Aria Practices

📝 Test Instructions:

🧢 Your Project:

Noticed that the padding is somehow off on the Events example now: https://reactspectrum.blob.core.windows.net/reactspectrum/eab25a55e7e4b7b5895d0670e92202c95d6b5057/docs/react-spectrum/Button.html#events

+1 to what @ktabors mentioned. I think a specific callout to the "If the label is hidden, an icon is required, and the label will appear in a tooltip" portion of the Spectrum guidelines is appropriate here in case people don't click on that guideline link. It would also be good to have the examples follow this practice as well.

and the label will appear in a tooltip

this is not true of our implementation however.

right, we currently rely on the user to provide the tooltip... Is the tooltip mandatory for icon only buttons? If so, we could wrap it in a TooltipTrigger for the user but we would need them to provide a tooltip text value through a different prop. If it isn't mandatory but strongly recommended IMO we should have the example display that. Otherwise we can keep it as is

I wouldn't add Tooltip to Button. We could add an example though.


For fixing of the bugs in navbar and grammar error

for fixing of the bugs in navbar and fix the grammar ....
please assign this issue

Kindly add any screenshot so that I can understand where is the issue

Correct the spelling in the navigation menu:
Stationary -> Stationery

In the provided screenshot, the phrase "Upto 50% Off" should be corrected to "Up to 50% Off.

Here are the necessary corrections:
"Upto 50% Off" -> "Up to 50% Off"

@SanaAzeem512  kindly find another genuine issue where you can learn something new so that you can mention it in your resume also.
I don't have any issue with assigning you but it is totally worthless
don't mind but I suggest you that find a genuine bug that you can add to your resume.

check whole landing page and also take a look of other page for typo mistake

okay

can you please assign me this issue?


[main] Update actions

Cron -knative-prow-robot
/cc knative/operations-wg-leads
/assign knative/operations-wg-leads
Produced by: knative-sandbox/knobots/actions/update-actions
Details:

/test pull-knative-operator-eventing-upgrade-tests

/lgtm
/approve


Using langchain and LLaMA2 to QA with a large SQL DB

I was working with a sqlite DB that I created for a large dataset (~150k rows).
Code snippets:
db = SQLDatabase.from_uri("sqlite:///MLdata.sqlite")
`SQLITE_PROMPT_TEXT = '''You are a SQLite expert. Given an input question, first create a
syntactically correct SQLite query to run, then look at the results of the query and return
the answer to the input question.
Unless the user specifies in the question a specific number of examples to obtain, query for
at most {top_k} results using the LIMIT clause as per SQLite. You can order the results to
return the most informative data in the database.
Never query for all columns from a table. You must query only the columns that are needed to
answer the question. Wrap each column name in double quotes (") to denote them as delimited
identifiers.
Pay attention to use only the column names you can see in the tables below. Be careful to not
query for columns that do not exist. Also, pay attention to which column is in which table.
Use the following format:
Question: Question here
SQLQuery: SQL Query to run
SQLResult: Result of the SQLQuery
Answer: Final answer here
Only use the following tables:
{table_info}
Question: {input}'''`
`SQLITE_PROMPT = PromptTemplate(input_variables=['input', 'table_info', 'top_k'], template=SQLITE_PROMPT_TEXT)
sql_chain = SQLDatabaseChain(llm=local_llm, database=db, prompt=SQLITE_PROMPT, return_direct=False, return_intermediate_steps=False, verbose=False)
res=sql_chain("How many rows is in this db?")`
Response: 'There are 142321 rows in the input_table of this db.'
Second query
res=sql_chain("Count rows with 'Abdominal pain', VAX_TYPE='COVID19', SEX= 'F' and HOSPITAL= 'Y' is in the input_table of this db")
Response: 'There are 115 rows in the input_table where Abdominal pain is present, VAX_TYPE is COVID19, Sex is Female, and Hospital is Yes.'
Third query I was trying to find the patient ID only instead of the count. But I am not able to get the patient ID.
res=sql_chain("What is the VAERS_ID with 'Abdominal pain', VAX_TYPE='COVID19', SEX= 'F' and HOSPITAL= 'Y' in this db. ")
But the output generated is same as my second query.
Seems like the counting is working fine but nothing more.
Can anyone help me in displaying table like output from sqlDbchain via langchain and llama2?

The below query didnt work and displayed out of memory:
 res = sql_chain("What are the VAERS_IDs for rows with 'Abdominal pain', VAX_TYPE='COVID19', SEX= 'F' and HOSPITAL= 'Y' in this db.")


chore: update build tasks, package.json and changelog

PR Checklist

[ ] Unit Tests have been added for new changes
[ ] API tests have been updated if applicable
[ ] All commented code has been removed
[ ] If you've added a dependency, you've ensured license is compatible with Apache 2.0 and clearly outlined the added dependency.

What are you changing?

Anything the reviewer should know when reviewing this PR?
If the there are associated PRs in other repositories, please link them here (i.e. open-amt-cloud-toolkit/repo#365 )

Fixed semicolon issue. Its ready for review

:tada: This PR is included in version 8.0.2 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


[html] bracket matching in strings

Issue Type: Bug
In HTML the minor sign inside of tags is sometimes recognised as opening a new tag.
For Example:
<div data-ng-if="(something <= 0)" class="alert alert-warning">

VS Code version: Code 1.32.3 (a3db5be9b5c6ba46bb7555ec5d60178ecc2eaae4, 2019-03-14T23:43:35.476Z)
OS version: Windows_NT x64 10.0.16299

System Info

Item
Value

CPUs
Intel(R) Core(TM) i5-3210M CPU @ 2.50GHz (4 x 2494)

GPU Status
2d_canvas: enabledchecker_imaging: disabled_offflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onnative_gpu_memory_buffers: disabled_softwarerasterization: unavailable_offsurface_synchronization: enabled_onvideo_decode: enabledwebgl: enabledwebgl2: enabled

Memory (System)
7.94GB (3.81GB free)

Process Argv

Screen Reader
no

VM
0%

Extensions (3)

Extension
Author (truncated)
Version

python
ms-
2019.2.5558

cpptools
ms-
0.22.1

vscode-spotify
shy
3.1.0

(2 theme extensions excluded)

The problem is that we override the attribute value as 'other' in order to allow quick suggest:
https://github.com/microsoft/vscode/issues/46145

Confirm, seeing the same issue editing Vue templates. A line like
For example:
<template>

 0">

Some content

</template>

Place the cursor on 'colindex' and see that the greater than angle-bracket is incorrectly matched to the opening angle-bracket of the div.
Place the cursor at the end of the line
Change the v-if condition by replacing "colindex > 0" with "colindex == 0" and everything's fixed.
Change the v-if condition by replacing "colindex > 0" with "colindex < 0" also breaks angle bracket matching.

This issue also seems to break editor.action.smartSelect.expand and editor.action.smartSelect.shrink.

Any update on this?


Prometheus metrics?

I realize this is a project from long ago, but ever consider having it do Prometheus metrics?

I wonder why you want to have a complete monitoring tool into the server status page? Even though this is newer than the plain server-status page, but in lua, it is just the apache status page.

I’m looking to have a more native export of Prometheus metrics than the existing Go server-status scripts running another listener. This seemed like a related capability. The existing page output is interesting, but only a tangent to my needs.


Update and rename dockers.md to docker.md

Dockers isn't a word.  Docker is what you mean here.  😄

Changed in Gitbooks: https://app.gitbook.com/@nethermind/s/nethermind/~/drafts/-MQSKAwo8Rr-fErmrOac/@activity
Accidentally part of a JSON-RPC change due to me not understanding how GitBooks works, hopefully not a problem to have both in there at once.

Changed in Gitbooks: https://app.gitbook.com/@nethermind/s/nethermind/~/drafts/-MQSKAwo8Rr-fErmrOac/@activity
Accidentally part of a JSON-RPC change due to me not understanding how GitBooks works, hopefully not a problem to have both in there at once.


chore: pin and update dependencies

Signed-off-by: Jamie Magee<EMAIL_ADDRESS>

Can you rebase and submit again? I merged your other PR so the base package.json is now out of sync with yours.

Closing as I had to manually wrangle this.

Thanks for resolving the conflicts


Ubuntu: Remove gcc and gfortran source repository.

Description
Improvement
In order to keep only official ubuntu repositories on the images, it is required to cleanup repos which were added during the build procedure
Related issue:
https://github.com/actions/virtual-environments-internal/issues/2072
Check list

[+] Related issue / work item is attached
[+] Tests are written (if applicable)

We made a decision do not merge this pull request, due to the possible impact on customers who use this repo.


Issue with ADTs during cycling

I keep encountering an issue while cycling with a 2/3-deg setup we have where it ends up crashing after 3 weeks of simulation time. I tried multiple GEOS settings but after taking a closer look at the increments I figured it might be SOCA related. I'm at a bit of a loss here though.
For these particular plots, I'm using a 24-hr DA window, 2/3-deg tripolar grid, ocean-only assimilation (surface observations only), and using GDAS YAMLs.
I'm calling "unhealthy" increments a the ones behave as spikes in the J_o plots.
A healthy increment (2021-06-09):

An unhealthy increment (2021-06-10) (notice the difference in color scale):

I compared hofx_0 and hofx_1 vs. obs value plots to take a closer look
2021-06-09 (left, healthy)  2021-06-10(right, unhealthy)

So hofx_1 in the unhealthy plot shows analysis is having 2-3m difference and this behavior eventually leads to blow up.
One clear solution is perhaps activating an obs - analysis filter in the ADT YAMLs but I would like to ensure this is not a systemic issue first. It is just puzzling to me this doesn't happen during 1/4-deg runs but I notice some spikes there as well. I will investigate them closely next.
Our first instinct was some sort of a land mask issue but that doesn't seem to be the case, you can see the location of the issue.
I also suspect the correlation lengths might be the culprit here. I used StaticB BUMP for these results and I just used the same correlation lengths from the 1/4-deg setup, obviously had to create new BUMP files for the 2/3-deg grid.
However I also looked at the ADT ufo operator and I was wondering if this might have an impact on the analysis?
https://github.com/JCSDA-internal/ufo/blob/d7a8918003ca5da787c4e459f2928a87e6e50882/src/ufo/operators/marine/adt/ObsADT.cc#L77
P.S.: I see same outliers with SST observations as well but maybe the obs. count prevent the same issue from happening (272k for SST vs. ~9k with ADTs)

FYI @guillaumevernieres

Thanks for sharing @Dooruk . We recently changed the adt obs yamls and the way we assimilate the rads retrieved adt. I'm not sure this is related to your problem though, but here's what changed:
1 - We're no longer blacklisting adt obs in the southern ocean, instead we compute the unbalanced term of ssh at every cycle.
2 - We only have one obs space for adt. This was done to improve the reference that we estimate in the ufo.
A large ssh increment translates into a large temperature and salinity increment. Did you look at those? I suspect something is wrong in your background error for temperature.

A couple of related things.

Guillaume mentions background error. @Dooruk you're using the BkgErrGodas filter for generating parametric background error, right? It's next on my list to rewrite that filter into a proper saber block (https://github.com/JCSDA-internal/soca/issues/1004) and while I'm at it have it optionally save the calculated error so we can look at it.
The line in the ADT operator you note is a problem, but I don't think it's the problem here. An outlier ob would just shift the global bias slightly.

Since the adt hofx_0 looks fine, but hofx_1 is weird, I suspect it is not because of the ADT obs but rather one of the other surface obs, as Guillaume said you should look at what the T/S increments look like at that location

@guillaumevernieres

We're no longer blacklisting adt obs in the southern ocean, instead we compute the unbalanced term of ssh at every cycle.

Sorry, calculating the unbalanced part and incorporating it back is not clear to me. Is this something that can be done via the BalanceSOCA operator? Or by changing h/ssh in the increment file?
Combining ADTs is something I will try, at some point..
@travissluka
Yes, I use the BkgErrGodas. Sounds good on saving errors.

The line in the ADT operator you note is a problem, but I don't think it's the problem here. An outlier ob would just shift the global bias slightly.

Yeah, that makes sense.. I will look at increments closely and conduct more dirac tests while at it. Thanks for the responses, I will look into this issue again in the near future.


Add MaterialStateTextStyle constructor with built-in MaterialPropertyResolver

InputDecorationTheme currently gives several convenient options for customizing the border:
final inputDecorationTheme = InputDecorationTheme(
  errorBorder: border1,
  focusedBorder: border2,
  focusedErrorBorder: border3,
  disabledBorder: border4,
  enabledBorder: border5,
);

Conversely, the labelStyle documentation recommends writing your own MaterialStatePropertyResolver to accomplish the same thing.

This pull request fixes #143488. Originally, I thought about just adding two parameters as requested, but I wanted to cover more use cases:
const inputDecorationTheme = InputDecorationTheme(
  labelStyle: TextStyle(color: Colors.orange),
  focusedLabelStyle: TextStyle(color: Colors.orangeAccent),
  disabledLabelStyle: TextStyle(color: Colors.grey),
  errorLabelStyle: TextStyle(color: Colors.red),
  errorFocusedLabelStyle: TextStyle(color: Colors.redAccent),
  floatingLabelStyle: TextStyle(color: Colors.orange),
  focusedFloatingLabelStyle: TextStyle(color: Colors.orangeAccent),
  disabledFloatingLabelStyle: TextStyle(color: Colors.grey),
  errorFloatingLabelStyle: TextStyle(color: Colors.red),
  errorFocusedFloatingLabelStyle: TextStyle(color: Colors.redAccent),
);

Having all these extra parameters would give easier control over input label appearance, but honestly this is awful to look at.
Instead, this pull request adds a MaterialStateTextStyle factory constructor that doesn't require writing out a MaterialPropertyResolver.
const inputDecorationTheme = InputDecorationTheme(
  labelStyle: MaterialStateTextStyle.fromStyles(
    normal: TextStyle(color: Colors.orange),
    focused: TextStyle(color: Colors.orangeAccent),
    disabled: TextStyle(color: Colors.grey),
    error: TextStyle(color: Colors.red),
    focusedError: TextStyle(color: Colors.redAccent),
  ),
  floatingLabelStyle: MaterialStateTextStyle.fromStyles(
    normal: TextStyle(color: Colors.orange),
    focused: TextStyle(color: Colors.orangeAccent),
    disabled: TextStyle(color: Colors.grey),
    error: TextStyle(color: Colors.red),
    focusedError: TextStyle(color: Colors.redAccent),
  ),
);

Example (flutter/test/material/text_field_test.dart):
before:
TextField(
  style: MaterialStateTextStyle.resolveWith((Set<MaterialState> states) {
    if (states.contains(MaterialState.disabled)) {
      return const TextStyle(color: Colors.red);
    }
    return const TextStyle(color: Colors.blue);
  }),
)

after:
TextField(
  style: const MaterialStateTextStyle.fromStyles(
    normal: TextStyle(color: Colors.blue),
    disabled: TextStyle(color: Colors.red),
  ),
)

Demo

Pre-launch Checklist

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] I read the Tree Hygiene wiki page, which explains my responsibilities.
[x] I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
[x] I signed the CLA.
[x] I listed at least one issue that this PR fixes in the description above.
[x] I updated/added relevant documentation (doc comments with ///).
[x] I added new tests to check the change I am making, or this PR is test-exempt.
[x] I followed the breaking change policy and added Data Driven Fixes where supported.
[x] All existing and new tests are passing.

@justinmc thanks for linking that video—really informative and gave a great explanation of the MaterialStateProperty rationale.
I also received some helpful advice from @QuncCccccc in a previous PR. I thought that it would be appropriate to switch to a factory implementation, but perhaps we should just avoid implementing this constructor altogether.

It's not a huge deal whether .fromStyles() is stored in the Flutter codebase or just in my personal projects, so I'm fine with maintaining status-quo.
I have a few more thoughts I'd like to share. I'll leave this PR open for a few more days to allow further discussion; let's plan on either merging or deleting this branch within the next week.

User feedback is super valuable but is also prone to sampling bias: someone who's having problems is much more likely to reach out than someone who's not having problems.
I personally haven't found it difficult to use theme-swapping widgets, whether they're implemented using MaterialStateProperty classes or not.
And unlike enabledBorder/disabledBorder/errorBorder/focusedErrorBorder, this proposed change has all the configuration for a given property grouped together.
floatingLabelStyle: MaterialStateTextStyle.resolveWith((states) {
  if (states.contains(MaterialState.error)) {
    if (states.contains(MaterialState.focused)) {
      return const TextStyle(fontWeight: FontWeight.bold);
    }
    return const TextStyle(color: Colors.red);
  } else if (states.contains(MaterialState.focused)) {
    return const TextStyle(fontWeight: FontWeight.bold);
  }
  return const TextStyle(color: Colors.black87);
}),

floatingLabelStyle: const MaterialStateTextStyle.fromStyles(
  normal: TextStyle(color: Colors.black87),
  focused: TextStyle(fontWeight: FontWeight.bold),
  error: TextStyle(color: Colors.red),
  focusedError: TextStyle(color: Colors.redAccent),
),

These two snippets produce the same runtime behavior, and when I look at the .fromStyles() constructor, it takes less time for me to understand what's going on.

Much like the Container widget's build() method, this constructor's resolve() method aims to combine each parameter in a logical and intuitive way, but it's a suboptimal use of computing resources when most of the properties aren't set.
If half of developers like using Container and the other half think it's too confusing/inefficient, it might make sense to include it in the framework—this way it can be used, and everyone still has the option to implement custom widgets tailored to their needs.
In my mind, the same concept applies here.

My hope is that long parameter lists such as enabledBorder/disabledBorder/errorBorder/focusedErrorBorder will eventually be deprecated in favor of Material state classes.
Currently, I see very little downside to having both .resolveWith() and .fromStyles() as constructor options.

User feedback is super valuable but is also prone to sampling bias: someone who's having problems is much more likely to reach out than someone who's not having problems.
I personally haven't found it difficult to use theme-swapping widgets, whether they're implemented using MaterialStateProperty classes or not.

That's very true. We get conflicting feedback on MaterialState: both that it's intimidating to use, and that there should be more of it. I have a PR sitting that actually moves MaterialState out of Material to make it available to the broader framework, but there's a disagreement on that PR about whether that should happen which we are still talking through.
I think you bring up good points. If we are already asking people to look at using .resolveWith() maybe giving them access to fromStyles would possibly streamline things. And the ability to umbrella the styles under one property is nice.
I'll look at this more.

Now that a week has passed, I believe it's time to close the PR. I won't delete this branch from my fork, in case we want to pick it back up in the future.

Thanks, it sounds like this is overall issue of MaterialStateProperty is still up for debate, so this may end up being revisited.


issue formatting the volume:

So I have finally gotten around to testing this.  I believe am almost there.  I was able to start the plugin, and have it create the volume.
I see the volume on my nomad client node in vsphere (Hard Disk 2), but I am getting the following error:
I am using ubuntu 22.04
root@nomad-p04:~# nomad volume create csi-volume-demo.hcl
Error creating volume: Unexpected response code: 500 (rpc error: 1 error occurred:
        * controller create volume: CSI.ControllerCreateVolume: rpc error: code = Unknown desc = Exception calling application: Failed to format disk :mke2fs 1.46.2 (28-Feb-2021)
The file /dev/sdb does not exist and no size was specified.)

If I ssh into that nomad host I am able to:
root@nomad-p04:~$ ls /dev/sd*
/dev/sda  /dev/sda1  /dev/sda2  /dev/sda3  /dev/sdb

&

root@nomad-p04:~# mkfs.ext4 /dev/sdb
mke2fs 1.46.5 (30-Dec-2021)
Creating filesystem with 1280000 4k blocks and 320000 inodes
Filesystem UUID: 8af69c8a-b37b-4353-a4a6-e88a4293f02b
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

This is from the docker stderr:
'connect to vhost'
'Load content menu'
'Loading container'
'loading data store'
'Reading hostname'
'Create Disk'
'nyvmlsnmc-p04'
'Creating an image of this many megabytes'
5000
'Finding my vmnyvmlsnmc-p04'
'Preparing disk!'
'scanning devices on the given vm'
'[NimbleVMFS01-Prod] nyvmlsnmc-p04/nyvmlsnmc-p04.vmdk'
'Found empty sd slot 1'
'Running echo "- - -" > /sys/class/scsi_host/host2/scan'
'On production'
'echo "- - -" > /sys/class/scsi_host/host2/scan'
''
''
'0'
'Running echo "- - -" > /sys/class/scsi_host/host0/scan'
'On production'
'echo "- - -" > /sys/class/scsi_host/host0/scan'
''
''
'0'
'Running echo "- - -" > /sys/class/scsi_host/host1/scan'
'On production'
'echo "- - -" > /sys/class/scsi_host/host1/scan'
''
''
'0'
'Running echo "1" > /sys/class/block/sdb/device/rescan'
'On production'
'echo "1" > /sys/class/block/sdb/device/rescan'
''
('/bin/sh: 1: cannot create /sys/class/block/sdb/device/rescan: Directory '
 'nonexistent\n')
'2'
'wait for kernel to finish scanning'
"Disk has been added to virtual host, let's look at our mounts"
'c04ab00a63c7'
['overlay',
 'proc',
 'sysfs',
 'cgroup',
 '/dev/mapper/sysvg-opt',
 '/dev/mapper/sysvg-opt',
 'tmpfs',
 '/dev/mapper/sysvg-opt',
 'udev',
 'devpts',
 'tmpfs',
 'hugetlbfs',
 'mqueue',
 '/dev/mapper/sysvg-opt',
 '/dev/mapper/sysvg-var',
 '/dev/mapper/sysvg-var',
 '/dev/mapper/sysvg-var']
'done connecting'
'formatting disk'
'Running mkfs.ext4 /dev/sdb'
'On production'
'mkfs.ext4 /dev/sdb'
''
('mke2fs 1.46.2 (28-Feb-2021)\n'
 'The file /dev/sdb does not exist and no size was specified.\n')
'1'
ERROR:grpc._server:Exception calling application: Failed to format disk :mke2fs 1.46.2 (28-Feb-2021)
The file /dev/sdb does not exist and no size was specified.

ok - I figured out the issue.   I was hardcoding the value VMWARE_HOSTNAME.  On a cluster it needs to reference the nomad attribute, so it uses itself.  In addition unique.name won't work for me, since I don't use fqdn in vmware names, so I was able to use attr.unique.consul.name and that worked!


Specifying a FileObject.containedIn with a contentUrl of a relative path doesn't work

There's an example in the repo here which leads me to believe that contentUrl can be a relative path from the source FileObject that it's containedIn.
But when I try to build a similar one for a Kaggle dataset, I get:
  File "/usr/local/google/home/goeff/py3/lib/python3.11/site-packages/mlcroissant/_src/operation_graph/operations/download.py", line 175, in __call__
    filepath = get_download_filepath(self.node, self.url)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/google/home/goeff/py3/lib/python3.11/site-packages/mlcroissant/_src/operation_graph/operations/download.py", line 49, in get_download_filepath
    assert url.startswith("data/"), (
AssertionError: Local file "{self.node.uid}" should point to a file within the data/ folder next to the JSON-LD Croissant file. But got: "{self.url}"

For reference, I have an archive that has many files, some of which are CSVs. To build the RecordSets and Fields, it seems like I have to target by source.distribution. So I'm trying to add one FileObject for each CSV to the top-level distributions.

@goeffthomas, I opened https://github.com/mlcommons/croissant/pull/241 to implement this. Notably, you'll find a JSON-LD recipe with a working example of a CSV within a ZIP file. Does this solve your issue?
Thanks!

@goeffthomas, I opened #241 to implement this. Notably, you'll find a JSON-LD recipe with a working example of a CSV within a ZIP file. Does this solve your issue?
Thanks!

Yes, this looks like what I was expecting, thank you! I'll try it out next week on the examples I've been generating.


Halo concentration for Delta = 500-critical

I'm glad that CCL has updated to version 2.2.0, and I can use it to deal with pressure profiles. The GNFW profile uses Delta=500-critical as halo mass definition(M500), but I don't find the compatible mass-concentration relation in this version of CCL. There is one in David's forge version (ConcentrationDuffy08M500c), is it implemented here?

Hi @yanzastro . Feel free to use that c(M) relation, but you should be aware that it's a bit of a fake. It comes from translating between mass definitions assuming an NFW profile, so it's bound to be quite inaccurate. I am not aware of any c(M) relations for M500c, which is sorely lacking.