Recurrent PPO

Description

Experimental version of PPO with LSTM policy.
Current status: usable but not polished, see https://github.com/Stable-Baselines-Team/stable-baselines3-contrib/pull/53#issuecomment-1074371013
Missing:

[x] benchmark https://wandb.ai/sb3/no-vel-envs/reports/PPO-vs-RecurrentPPO-aka-PPO-LSTM-on-environments-with-masked-velocity--VmlldzoxOTI4NjE4 (and models in https://wandb.ai/openrlbenchmark/sb3?workspace=user-araffin )
[x] more tests (reproducibility + identity learning/callback)
[x] reduce code duplication in buffer
[x] document buffer + rest
[x] more testing of the different architecture (shared vs separate lstm for actor/critic)

Known issue: if the model was train on GPU and tested on CPU, a warning will be issued because it cannot unpickle the lstm initial states. This is ok as they will be reset anyway in setup_model() and it doesn't affect prediction.
Context

[x] I have raised an issue to propose this change (required)
closes https://github.com/DLR-RM/stable-baselines3/issues/18

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Documentation (update in the documentation)

Checklist:

[x] I've read the CONTRIBUTION guide (required)
[x] The functionality/performance matches that of the source (required for new training algorithms or training-related features).
[x] I have updated the tests accordingly (required for a bug fix or a new feature).
[x] I have included an example of using the feature (required for new features).
[x] I have included baseline results (required for new training algorithms or training-related features).
[x] I have updated the documentation accordingly.
[x] I have updated the changelog accordingly (required).
[x] I have reformatted the code using make format (required)
[x] I have checked the codestyle using make check-codestyle and make lint (required)
[x] I have ensured make pytest and make type both pass. (required)

Note: we are using a maximum length of 127 characters per line

Hi there, I'm not sure where to mention this, but I'm currently using the work from this branch, and model.save() has an issue where the model saves fine, but it never returns. The result is the code just hangs indefinitely. Thought I'd let someone know.
This is my system:
>>> sb3.get_system_info()
OS: Linux-5.13.0-30-generic-x86_64-with-glibc2.29 #33~20.04.1-Ubuntu SMP Mon Feb 7 14:25:10 UTC 2022
Python: 3.8.10
Stable-Baselines3: 1.4.0
PyTorch: 1.10.0+cu102
GPU Enabled: True
Numpy: 1.22.2
Gym: 0.22.0

Hi there, I'm not sure where to mention this, but I'm currently using the work from this branch, and model.save() has an issue where the model saves fine, but it never returns. The result is the code just hangs indefinitely. Thought I'd let someone know.
This is my system:
>>> sb3.get_system_info()
OS: Linux-5.13.0-30-generic-x86_64-with-glibc2.29 #33~20.04.1-Ubuntu SMP Mon Feb 7 14:25:10 UTC 2022
Python: 3.8.10
Stable-Baselines3: 1.4.0
PyTorch: 1.10.0+cu102
GPU Enabled: True
Numpy: 1.22.2
Gym: 0.22.0

Hello,
I'm back from a 4 week break... could you please provide a minimal code to reproduce the issue? (without a custom env)

Hi there, I'm not sure where to mention this, but I'm currently using the work from this branch, and model.save() has an issue where the model saves fine, but it never returns. The result is the code just hangs indefinitely. Thought I'd let someone know.
This is my system:
>>> sb3.get_system_info()
OS: Linux-5.13.0-30-generic-x86_64-with-glibc2.29 #33~20.04.1-Ubuntu SMP Mon Feb 7 14:25:10 UTC 2022
Python: 3.8.10
Stable-Baselines3: 1.4.0
PyTorch: 1.10.0+cu102
GPU Enabled: True
Numpy: 1.22.2
Gym: 0.22.0

Hello, I'm back from a 4 week break... could you please provide a minimal code to reproduce the issue? (without a custom env)
Note: current version only supports gym 0.19 (gym 0.21 will be supported soon (already on master), 0.22 introduces new breaking changes...)

Apologies, I think it may be an issue on my end regarding the env I'm using. I'm currently using Pistonball of  PettingZoo using this hack. I can't get it to replicate using normal gym, even gym 0.22.

@rvdweerd
Replying to your email here so it serves others:

Is it possible for you to comment on the status of the PPO+LSTM implementation?
I gather it is still in development and not working yet, any chance of a testable version in the near future?

The implementation is working (it was tested on different env, including cartpole without velocity which can be solved only with RNN) but it needs some polishing and additional testing.

Would it also be possible to use action-maskable PPO policies + LSTM?

In theory yes, in practice, I will keep the PPO LSTM implementation as simple as possible, but nothing prevents you from combining the Maskable PPO with it ;)

Would the implementation include Dict observation spaces?

Yes, I think that's one of the thing that need to be done before merging. Contributions are welcomed ;)
And anyway, I would always recommend to start with frame-stacking and feedforward network as a baseline for POMDP, it is usually competitive with LSTM.

@rrfaria This was fixed in a recent PR in SB3 here. Try doing these changes manually and see if that helps.

Hi! When using multiple environments, it does not seem to be running on multiple cores of the CPU. Resulting in very slow training. Exactly the same setup works fine on the normal PPO implementation. Is this something that still has to be addressed?

it does not seem to be running on multiple cores of the CPU. Resulting in very slow training.

could you provide a minimal example?
multiple cores for collecting data or doing gradient update?

Resulting in very slow training.

this highly depends on your hyperparameters, especially the n_steps, LSTM PPO is slower anyway compared to the feedforward one.

For now we have:
model = Model("MlpLstmPolicy",
env=dummy_env,
n_steps=128,
batch_size=14,
gamma=0.5,
n_epochs=5,
ent_coef=0.0,
policy_kwargs=policy_kwargs,
verbose=1,
tensorboard_log=tensorboard_log,
device="auto")
where Model is the RecurrentPPO. While training htop shows that only 1 core is in use except prior to logging the new stats, then it suddenly uses all the cores. With stats i mean:
| rollout/                |             |
|    ep_len_mean          | 252         |
|    ep_rew_mean          | 12.4        |
| time/                   |             |
|    fps                  | 451         |
|    iterations           | 5           |
|    time_elapsed         | 45          |
|    total_timesteps      | 20480       |
| train/                  |             |
|    approx_kl            | 0.021662503 |
|    clip_fraction        | 0.297       |
|    clip_range           | 0.2         |
|    entropy_loss         | -0.907      |
|    explained_variance   | 0           |
|    learning_rate        | 0.0003      |
|    loss                 | 0.336       |
|    n_updates            | 20          |
|    policy_gradient_loss | -0.0208     |
|    value_loss           | 0.56        |
We have 32 threads, so our vectorized environment exists of 32 envs.
env = make_vec_env(StockEnv, env_kwargs={**env_kwargs, 'df': df}, n_envs=args.n_envs, vec_env_cls=SubprocVecEnv)

it does not seem to be running on multiple cores of the CPU. Resulting in very slow training.

could you provide a minimal example? multiple cores for collecting data or doing gradient update?

Resulting in very slow training.

this highly depends on your hyperparameters, especially the n_steps, LSTM PPO is slower anyway compared to the feedforward one.

For now we have:
model = Model("MlpLstmPolicy",
env=dummy_env,
n_steps=128,
batch_size=14,
gamma=0.5,
n_epochs=5,
ent_coef=0.0,
policy_kwargs=policy_kwargs,
verbose=1,
tensorboard_log=tensorboard_log,
device="auto")
where Model is the RecurrentPPO. While training htop shows that only 1 core is in use except prior to logging the new stats, then it suddenly uses all the cores for one sec and goes back to just using 1 afterwards. With stats I mean the little table showing the rollout/ time/ train/ etc in the terminal.
We have 32 threads, so our vectorized environment exists of 32 envs.
env = make_vec_env(Env, env_kwargs={**env_kwargs, 'df': df}, n_envs=args.n_envs, vec_env_cls=SubprocVecEnv)
When using the normal PPO we see usage of all cores in htop. Only when our evaluation callback is called you see a dip to one core because of evaluation. We use the same training loop for PPO and RecurrentPPO. We just initialize, set env as above. and start learning.

Hi folks,  I am looking at using this code, and just confirming a particular of the internals:
When doing the backprop through the action probabilities (here), you just take a single step in the RNN, i.e. the gradient with respect to the RNN parameters given just the previous RNN state and input?
The alternative to this is re-processing a short sequence and backpropping over that (truncated back-prop through time (t-BPTT)).  This will give different gradients on the weights.  This would be analogous to RECURRENT_SEQ_LEN in this implementation.
I could believe that this wouldn't be too difficult to add to the code.  The buffer just generates a sequence of length RECURRENT_SEQ_LEN, instead of calling self.policy.evaluate_actions directly, it calls out to a short subroutine for re-processing the sequence, and still returns values, log_prob, entropy.  Would be happy to contribute to this effort.
More generally, is there a particular consensus on whether (t-)BPTT is required to learn good policies?
Thanks a ton for grafting hard on this recurrent PPO!
A

@andrewwarrington

When doing the backprop through the action probabilities (here), you just take a single step in the RNN, i.e. the gradient with respect to the RNN parameters given just the previous RNN state and input?

Actually not, see the shape in the buffer https://github.com/Stable-Baselines-Team/stable-baselines3-contrib/blob/feat/ppo-lstm/sb3_contrib/common/recurrent/buffers.py#L164
I just pushed proper masking for the padded sequences. We do backprop for the sequences collected during the n_steps of the rollout.

for the time being, is there any way we can feedback the older data back to the trainer as input to mimic a crude version of LSTM? any sample code to do that?

for the time being, is there any way we can feedback the older data back to the trainer as input to mimic a crude version of LSTM? any sample code to do that?

Hello,
you can already use this PR if you need to use recurrent PPO (see install from source in our doc), otherwise you can use frame stacking or history wrapper (see code in the RL Zoo).

when LSTM for A2C?

when LSTM for A2C?

a2c is a special case of ppo ;) (cc @vwxyzjn )

@henrydeclety see https://github.com/vwxyzjn/a2c_is_a_special_case_of_ppo. We have a paper coming out soon...

The preprint of the paper is out at https://arxiv.org/abs/2205.09123 @henrydeclety :)

Hello, you can already use this PR if you need to use recurrent PPO (see install from source in our doc), otherwise you can use frame stacking or history wrapper (see code in the RL Zoo).

I’ll give it a try

How could I configure the maximum sequence length for the LSTM?

@EloyAnguiano As far as I could tell from the code, the implementation in SB3 does not have a sequence length, but saves the hidden state between steps of your environment and then uses it as input. So the maximum sequence length for the lstm would be the number of steps (n_steps) before you update your policy.


[WIP] CFT to PCF conversion

looking to get some feedback on this.
having some issues with dictionary copies when running the tests in sequence. the tests pass individually so i assume i need to add a deepcopy somewhere.

can you also add examples for using the CFT to PCF conversion tool. Wanna try it out

You can look at the sample tests. But yeah can add an example as well.


FIXES ISSUE #407: CODE CHANGE: Complete DarkMode UI Revamp #407

Fixes:  #(issue no.)
#407
Description
I have completely changed the color palette of the dark ui of the website and now the website looks much less eye straining and more professional.
I have also made some animation changes in Travel Itineraries section.
As a whole, the whole website Dark UI is much more eye soothing now.
Type of PR

[ ] Feature enhancement

Screenshots / videos (if applicable)
https://github.com/user-attachments/assets/c6e6f240-b5a2-4f1b-9d26-407f428fdc75
Checklist:

[ ] I have made this change from my own.
[ ] My code follows the style guidelines of this project.
[ ] I have performed a self-review of my own code.
[ ] I have commented my code, particularly in hard-to-understand areas.
[ ] I have tested the changes thoroughly before submitting this pull request.
[ ] I have provided relevant issue numbers and screenshots after making the changes.

I'm a GSSOC'24 Contributor & Hacktoberfest Contributor.
This issue fixed required a lot of hours of work from my end, so i request you to label it as Level 3.
Thank You!

@SagnikGos Please see the conflicts . Resolve the conflicts . Ensure you pull all the latest changes before pushing your feature/bug fixes to the main branch. After resolving the conflicts, please open a new PR for review.
Thank you!


Cannot open same workspace via "Open Recent" with two different profiles

Does this issue occur when all extensions are disabled?: Yes

VS Code Version: 1.78.2 (user setup)
OS Version: Windows_NT x64 10.0.22621

Steps to Reproduce:

Make a secondary settings profile, say we call it "ProfileB".
Make a new window and switch to ProfileB.
In this ProfileB window, open a repo and keep that window open.
From the original window using the default profile, go to Open Recent -> and try to open your recently opened repo.
Observe that VSCode instead brings focus to the ProfileB window, instead of opening the repo in the default profile like expected.

VSCode should not bring focus to an open window when it is not in the same profile as you are opening from. I should be able to open the same repo in the default profile and any secondary profiles I feel like.
I'm trying to investigate another bug for an extension developer and wanted to make a secondary profile to compare extensions and this annoyance is making it quite cumbersome.

Its current design that a folder can be opened only in one window.
@bpasero If there is an issue already for this, please mention.
Thanks.

https://github.com/microsoft/vscode/issues/2686


Additional Symlinks

Is your feature request related to a problem? Please describe.
It would make it much easier if more links were visible in the /barotrauma bind mount:
WorkshopMods/Installed
config_player.xml
Describe the solution you'd like
Add config_player.xml and WorkshopMods/Installed to the /barotrauma bind mount
Describe alternatives you've considered
Bind mounting /game/barotrauma myself and then navigating to the files/directories.
Additional context
N/A

Hi @Dope25 Sorry for the delay, I'm starting work on the config_player.xml mounting

I was not able to add the WorkshopMods/Installed mountpoint. However I think I've found a workaround with the LocalMods folder, I've released 0.4.0 that has both the symlink added.


ISPN-12930 Try to determine current schema during table existence check

...in DB2TableManager
Currently the DB2 manager looks into all schemas to determine if caching
table exists when the schema is not specified.
https://issues.redhat.com/browse/ISPN-12930
https://issues.redhat.com/browse/JBEAP-21362
Work in progress - I would like to have the idea reviewed by someone from Infinispan team. I will add a unit test if this looks OK.

Thanks @TomasHofman . You should however always create PRs against master before they are considered for backport to older branches.

@tristantarrant sure, I created an upstream PR: https://github.com/infinispan/infinispan/pull/9227

Merged, thanks


Installation issues

Hi, I am trying to install the software as described in the README. But I run into the error below when running make -j:
/Users/rajathsalegame/rds/ramulator2/src/dram/impl/DDR4-VRR.cpp:453:51: note: in instantiation of template type alias 'PreqFunc_t' requested here
      m_preqs.resize(m_levels.size(), std::vector<PreqFunc_t<Node>>(m_commands.size()));
                                                  ^
1 error generated.
make[2]: *** [src/dram_controller/CMakeFiles/ramulator-controller.dir/impl/plugin/para.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from /Users/rajathsalegame/rds/ramulator2/src/dram/impl/LPDDR5.cpp:1:
In file included from /Users/rajathsalegame/rds/ramulator2/src/dram/dram.h:11:
/Users/rajathsalegame/rds/ramulator2/src/dram/node.h:219:1: warning: ISO C++ specifies that qualified reference to 'Node' is a constructor name rather than a type in this context, despite preceding 'typename' keyword [-Winjected-class-name]
using RowhitFunc_t = std::function<bool(typename T::Node* node, int cmd, int target_id, Clk_t clk)>;
^
/Users/rajathsalegame/rds/ramulator2/src/dram/impl/LPDDR5.cpp:510:54: note: in instantiation of template type alias 'RowhitFunc_t' requested here
      m_rowopens.resize(m_levels.size(), std::vector<RowhitFunc_t<Node>>(m_commands.size()));
                                                     ^
In file included from /Users/rajathsalegame/rds/ramulator2/src/dram/impl/HBM3.cpp:1:
In file included from /Users/rajathsalegame/rds/ramulator2/src/dram/dram.h:11:
/Users/rajathsalegame/rds/ramulator2/src/dram/node.h:217:1: warning: ISO C++ specifies that qualified reference to 'Node' is a constructor name rather than a type in this context, despite preceding 'typename' keyword [-Winjected-class-name]
using PreqFunc_t   = std::function<int (typename T::Node* node, int cmd, int target_id, Clk_t clk)>;
^
/Users/rajathsalegame/rds/ramulator2/src/dram/impl/HBM3.cpp:386:51: note: in instantiation of template type alias 'PreqFunc_t' requested here
      m_preqs.resize(m_levels.size(), std::vector<PreqFunc_t<Node>>(m_commands.size()));
                                                  ^
In file included from /Users/rajathsalegame/rds/ramulator2/src/dram/impl/DDR5.cpp:1:
In file included from /Users/rajathsalegame/rds/ramulator2/src/dram/dram.h:11:
/Users/rajathsalegame/rds/ramulator2/src/dram/node.h:217:1: warning: ISO C++ specifies that qualified reference to 'Node' is a constructor name rather than a type in this context, despite preceding 'typename' keyword [-Winjected-class-name]
using PreqFunc_t   = std::function<int (typename T::Node* node, int cmd, int target_id, Clk_t clk)>;
^
/Users/rajathsalegame/rds/ramulator2/src/dram/impl/DDR5.cpp:474:51: note: in instantiation of template type alias 'PreqFunc_t' requested here
      m_preqs.resize(m_levels.size(), std::vector<PreqFunc_t<Node>>(m_commands.size()));
                                                  ^
In file included from /Users/rajathsalegame/rds/ramulator2/src/dram/impl/DDR4-VRR.cpp:1:
In file included from /Users/rajathsalegame/rds/ramulator2/src/dram/dram.h:11:
/Users/rajathsalegame/rds/ramulator2/src/dram/node.h:219:1: warning: ISO C++ specifies that qualified reference to 'Node' is a constructor name rather than a type in this context, despite preceding 'typename' keyword [-Winjected-class-name]
using RowhitFunc_t = std::function<bool(typename T::Node* node, int cmd, int target_id, Clk_t clk)>;
^
/Users/rajathsalegame/rds/ramulator2/src/dram/impl/DDR4-VRR.cpp:466:53: note: in instantiation of template type alias 'RowhitFunc_t' requested here
      m_rowhits.resize(m_levels.size(), std::vector<RowhitFunc_t<Node>>(m_commands.size()));
                                                    ^
In file included from /Users/rajathsalegame/rds/ramulator2/src/dram/impl/HBM3.cpp:1:
In file included from /Users/rajathsalegame/rds/ramulator2/src/dram/dram.h:11:
/Users/rajathsalegame/rds/ramulator2/src/dram/node.h:219:1: warning: ISO C++ specifies that qualified reference to 'Node' is a constructor name rather than a type in this context, despite preceding 'typename' keyword [-Winjected-class-name]
using RowhitFunc_t = std::function<bool(typename T::Node* node, int cmd, int target_id, Clk_t clk)>;
^
/Users/rajathsalegame/rds/ramulator2/src/dram/impl/HBM3.cpp:398:53: note: in instantiation of template type alias 'RowhitFunc_t' requested here
      m_rowhits.resize(m_levels.size(), std::vector<RowhitFunc_t<Node>>(m_commands.size()));
                                                    ^
In file included from /Users/rajathsalegame/rds/ramulator2/src/dram/impl/HBM3.cpp:1:
In file included from /Users/rajathsalegame/rds/ramulator2/src/dram/dram.h:11:
/Users/rajathsalegame/rds/ramulator2/src/dram/node.h:219:1: warning: ISO C++ specifies that qualified reference to 'Node' is a constructor name rather than a type in this context, despite preceding 'typename' keyword [-Winjected-class-name]
using RowhitFunc_t = std::function<bool(typename T::Node* node, int cmd, int target_id, Clk_t clk)>;
^
/Users/rajathsalegame/rds/ramulator2/src/dram/impl/HBM3.cpp:406:54: note: in instantiation of template type alias 'RowhitFunc_t' requested here
      m_rowopens.resize(m_levels.size(), std::vector<RowhitFunc_t<Node>>(m_commands.size()));
                                                     ^
In file included from /Users/rajathsalegame/rds/ramulator2/src/dram/impl/DDR5.cpp:1:
In file included from /Users/rajathsalegame/rds/ramulator2/src/dram/dram.h:11:
/Users/rajathsalegame/rds/ramulator2/src/dram/node.h:219:1: warning: ISO C++ specifies that qualified reference to 'Node' is a constructor name rather than a type in this context, despite preceding 'typename' keyword [-Winjected-class-name]
using RowhitFunc_t = std::function<bool(typename T::Node* node, int cmd, int target_id, Clk_t clk)>;
^
/Users/rajathsalegame/rds/ramulator2/src/dram/impl/DDR5.cpp:492:53: note: in instantiation of template type alias 'RowhitFunc_t' requested here
      m_rowhits.resize(m_levels.size(), std::vector<RowhitFunc_t<Node>>(m_commands.size()));
                                                    ^
In file included from /Users/rajathsalegame/rds/ramulator2/src/dram/impl/DDR4-VRR.cpp:1:
In file included from /Users/rajathsalegame/rds/ramulator2/src/dram/dram.h:11:
/Users/rajathsalegame/rds/ramulator2/src/dram/node.h:219:1: warning: ISO C++ specifies that qualified reference to 'Node' is a constructor name rather than a type in this context, despite preceding 'typename' keyword [-Winjected-class-name]
using RowhitFunc_t = std::function<bool(typename T::Node* node, int cmd, int target_id, Clk_t clk)>;
^
/Users/rajathsalegame/rds/ramulator2/src/dram/impl/DDR4-VRR.cpp:474:54: note: in instantiation of template type alias 'RowhitFunc_t' requested here
      m_rowopens.resize(m_levels.size(), std::vector<RowhitFunc_t<Node>>(m_commands.size()));
                                                     ^
In file included from /Users/rajathsalegame/rds/ramulator2/src/dram/impl/DDR5.cpp:1:
In file included from /Users/rajathsalegame/rds/ramulator2/src/dram/dram.h:11:
/Users/rajathsalegame/rds/ramulator2/src/dram/node.h:219:1: warning: ISO C++ specifies that qualified reference to 'Node' is a constructor name rather than a type in this context, despite preceding 'typename' keyword [-Winjected-class-name]
using RowhitFunc_t = std::function<bool(typename T::Node* node, int cmd, int target_id, Clk_t clk)>;
^
/Users/rajathsalegame/rds/ramulator2/src/dram/impl/DDR5.cpp:500:54: note: in instantiation of template type alias 'RowhitFunc_t' requested here
      m_rowopens.resize(m_levels.size(), std::vector<RowhitFunc_t<Node>>(m_commands.size()));
                                                     ^
1 error generated.
make[2]: *** [src/memory_system/CMakeFiles/ramulator-memorysystem.dir/impl/dummy_memory_system.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
1 error generated.
make[2]: *** [src/memory_system/CMakeFiles/ramulator-memorysystem.dir/impl/generic_DRAM_system.cpp.o] Error 1
[ 96%] Linking CXX static library libspdlog.a
[ 96%] Built target spdlog
1 error generated.
make[2]: *** [src/test/CMakeFiles/ramulator-test.dir/test_impl.cpp.o] Error 1
make[1]: *** [src/test/CMakeFiles/ramulator-test.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
1 error generated.
make[2]: *** [src/translation/CMakeFiles/ramulator-translation.dir/impl/no_translation.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
1 error generated.
make[2]: *** [src/frontend/CMakeFiles/ramulator-frontend.dir/impl/memory_trace/loadstore_trace.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
1 error generated.
make[2]: *** [src/base/CMakeFiles/ramulator-base.dir/factory.cpp.o] Error 1
1 warning and 1 error generated.
make[2]: *** [src/dram_controller/CMakeFiles/ramulator-controller.dir/impl/refresh/all_bank_refresh.cpp.o] Error 1
1 error generated.
make[2]: *** [src/frontend/CMakeFiles/ramulator-frontend.dir/impl/memory_trace/readwrite_trace.cpp.o] Error 1
1 error generated.
make[2]: *** [src/dram_controller/CMakeFiles/ramulator-controller.dir/impl/scheduler/generic_scheduler.cpp.o] Error 1
1 error generated.
make[2]: *** [src/dram_controller/CMakeFiles/ramulator-controller.dir/impl/scheduler/blocking_scheduler.cpp.o] Error 1
1 error generated.
make[2]: *** [src/frontend/CMakeFiles/ramulator-frontend.dir/impl/external_wrapper/gem5_frontend.cpp.o] Error 1
1 error generated.
make[2]: *** [src/frontend/CMakeFiles/ramulator-frontend.dir/impl/processor/simpleO3/core.cpp.o] Error 1
1 error generated.
make[2]: *** [src/dram_controller/CMakeFiles/ramulator-controller.dir/impl/dummy_controller.cpp.o] Error 1
1 error generated.
make[2]: *** [src/dram_controller/CMakeFiles/ramulator-controller.dir/impl/scheduler/bh_scheduler.cpp.o] Error 1
1 error generated.
1 error generated.
make[2]: *** [src/addr_mapper/CMakeFiles/ramulator-addrmapper.dir/impl/linear_mappers.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
1 error generated.
make[2]: *** [src/dram_controller/CMakeFiles/ramulator-controller.dir/impl/generic_dram_controller.cpp.o] Error 1
make[2]: *** [src/dram_controller/CMakeFiles/ramulator-controller.dir/impl/plugin/trace_recorder.cpp.o] Error 1
1 error generated.
make[2]: *** [src/memory_system/CMakeFiles/ramulator-memorysystem.dir/impl/bh_DRAM_system.cpp.o] Error 1
make[1]: *** [src/memory_system/CMakeFiles/ramulator-memorysystem.dir/all] Error 2
1 error generated.
make[2]: *** [src/dram_controller/CMakeFiles/ramulator-controller.dir/impl/plugin/twice.cpp.o] Error 1
1 error generated.
1 error generated.
1 error generated.
1 warning and 1 error generated.
1 error generated.
make[2]: *** [src/frontend/CMakeFiles/ramulator-frontend.dir/impl/processor/simpleO3/llc.cpp.o] Error 1
make[2]: *** [src/dram_controller/CMakeFiles/ramulator-controller.dir/impl/plugin/graphene.cpp.o] Error 1
make[2]: *** [src/frontend/CMakeFiles/ramulator-frontend.dir/impl/processor/bhO3/bhcore.cpp.o] Error 1
make[2]: *** [src/dram_controller/CMakeFiles/ramulator-controller.dir/impl/plugin/blockhammer/blockhammer.cpp.o] Error 1
1 error generated.
1 error generated.
make[2]: *** [src/addr_mapper/CMakeFiles/ramulator-addrmapper.dir/impl/rit.cpp.o] Error 1
1 error generated.
make[1]: *** [src/addr_mapper/CMakeFiles/ramulator-addrmapper.dir/all] Error 2
make[2]: *** [src/translation/CMakeFiles/ramulator-translation.dir/impl/random_translation.cpp.o] Error 1
make[2]: *** [src/frontend/CMakeFiles/ramulator-frontend.dir/impl/processor/bhO3/bhllc.cpp.o] Error 1
make[1]: *** [src/translation/CMakeFiles/ramulator-translation.dir/all] Error 2
make[2]: *** [src/frontend/CMakeFiles/ramulator-frontend.dir/impl/processor/simpleO3/simpleO3.cpp.o] Error 1
make[1]: *** [src/frontend/CMakeFiles/ramulator-frontend.dir/all] Error 2
1 error generated.
make[2]: *** [src/dram_controller/CMakeFiles/ramulator-controller.dir/impl/plugin/oracle_rh.cpp.o] Error 1
1 error generated.
make[2]: *** [src/dram_controller/CMakeFiles/ramulator-controller.dir/impl/plugin/cmd_counter.cpp.o] Error 1
1 error generated.
make[2]: *** [src/dram_controller/CMakeFiles/ramulator-controller.dir/impl/plugin/rrs.cpp.o] Error 1
1 error generated.
make[2]: *** [src/dram_controller/CMakeFiles/ramulator-controller.dir/impl/plugin/hydra.cpp.o] Error 1
1 error generated.
make[2]: *** [src/base/CMakeFiles/ramulator-base.dir/config.cpp.o] Error 1
1 warning and 1 error generated.
make[2]: *** [src/dram_controller/CMakeFiles/ramulator-controller.dir/impl/bh_dram_controller.cpp.o] Error 1
make[1]: *** [src/dram_controller/CMakeFiles/ramulator-controller.dir/all] Error 2
make[1]: *** [src/base/CMakeFiles/ramulator-base.dir/all] Error 2
8 warnings and 1 error generated.
8 warnings and 1 error generated.
make[2]: *** [src/dram/CMakeFiles/ramulator-dram.dir/impl/HBM2.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: *** [src/dram/CMakeFiles/ramulator-dram.dir/impl/DDR5.cpp.o] Error 1
8 warnings and 1 error generated.
make[2]: *** [src/dram/CMakeFiles/ramulator-dram.dir/impl/DDR4-VRR.cpp.o] Error 1
8 warnings and 1 error generated.
make[2]: *** [src/dram/CMakeFiles/ramulator-dram.dir/impl/DDR4.cpp.o] Error 1
8 warnings and 1 error generated.
8 warnings and 1 error generated.
make[2]: *** [src/dram/CMakeFiles/ramulator-dram.dir/impl/HBM.cpp.o] Error 1
make[2]: *** [src/dram/CMakeFiles/ramulator-dram.dir/impl/DDR3.cpp.o] Error 1
8 warnings and 1 error generated.
make[2]: *** [src/dram/CMakeFiles/ramulator-dram.dir/impl/HBM3.cpp.o] Error 1
8 warnings and 1 error generated.
make[2]: *** [src/dram/CMakeFiles/ramulator-dram.dir/impl/LPDDR5.cpp.o] Error 1
make[1]: *** [src/dram/CMakeFiles/ramulator-dram.dir/all] Error 2
make: *** [all] Error 2

Re-check your env compiler's version based on the dependencies in README.


okex

Thank you for your analysis.
Can you add the following okex order book and options

This is a very good project

OrderBook of OKEx is supported in this library https://github.com/soulmachine/crypto-crawler
crypto-client aims at placing orders, similar to ccxt. While crypto-crawlers aims at crawling Websocket realtime messages
OKEx options is not supported in both libraries yet, will implement in short future

Thank you...


Unable to install on Ubuntu 18.04

I attempted to follow the instructions to install the azure-cli from https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots=apt
The option 1 installation just threw an error So I then went through the Option 2 installation instructions which reported there was no azure-cli package to install.
To get it to work I then ran the option 1 install and it worked.

add @fengzhou-msft to have a look

@bhagdave what was the error from option 1?

Sorry I dont recall the exact message and did not keep notes that included the error message.  I recall it was around the gpg for package signing.


ICLA Signature Check Fails Due to Multiple ICLA Table Entries

Summary
ICLA Signature Check Fails Due to Multiple ICLA Table Entries
Background
Provide context to the issue - provide steps to reproduce the behavior:

The user makes a commit and creates a PR
The user fails the GitHub EasyCLA check
The user navigates through the signing process, selects ICLA, and decides not to finish the process. Note: a record is created, but signature_signed value is false.
The user changes their mind and decides to go ahead and sign the ICLA - navigates through the process once more and signs the document. Note: a second record is created with the signature_signed value is true.
At this point, we have two ICLA signature entries in the DB:

1 not completed - signature_signed == false
1 completed - signature_signed == true

EasyCLA check queries the database and attempts to pull the latest ICLA document. Since the major and minor versions are the same, the first record appears to be returned in this case.  Since the ICLA is not signed, the check fails.

Expected behavior
The user would pass the ICLA check since they have signed a CLA document.
Screenshots
N/A
Environment
Please complete the following information:

Environment: PROD

** Work Around ***
Manually removing the record where signature_signed == false resolves the ICLA check issue.
Additional Context
Possible Fixes:

Avoid duplicate entries
Adjust document versioning logic to that the latest is used (but the document versioning was intended for something else - like editing that same document, not replacing it)
review the created date/time column and use the latest entry
loop through the ICLA signatures and select the first one that is completed/signed - ignoring the others

Acceptance Criteria
The "done" criteria when this feature or problem is resolved.  Such as:

Unit Tests added and running in CI
Functional Tests updated to cover feature, if applicable
Demonstrate the set of capabilities to the product team while the code is
running in the STAGING environment.

*References
(Optional) Provide any code or specification references that would be helpful
for the developer implementing this feature.

This is still a lingering issue https://github.com/prestodb/tsc/pull/9


Pass video running to a new form.

Is it possible to play the video in a newly created form without creating the vlccontrol again?
My scenario is that in my first Form I have already created a control for the video which is hidden but playing. The reason its hidden because I want to capture the image without displaying it (which is I'm done with that) so I'm silently capturing the img from the video and storing it in the file.
I do not know how to implement the vlccontrol that is already running then play it in the new form.
I do not want to create a new control in the new form because it's like running 2 videos with the same content at once which is not good in memory usage.

I don't really know about how WinForms works, but is it possible to reparent a form?
VLC has an API to change the window handle, but I don't know if it can do that while the video is playing

closing as new feature will not be implemented in Vlc.DotNet, and that's probably not easily doable in any libvlc binding.

closing as new feature will not be implemented in Vlc.DotNet, and that's probably not easily doable in any libvlc binding.


[WIP] Use ed25519::Signature type; impl Signer/Verifier traits

The ed25519 crate provides a common Signature type which can be shared among libraries which implement Ed25519 signing/verification:
https://docs.rs/ed25519
Without common types/traits for Ed25519 signatures, downstream consumer crates which use Ed25519 as part of a high-level protocol implementation which allows consumers of their crates to be able to use their choice of Rust Ed25519 libraries wind up writing their own abstractions to do so on a crate-by-crate basis.
The ed25519 crate, and the signature crate it's built on, provide a common abstraction layer:

ed25519 crate: https://github.com/RustCrypto/signatures/tree/master/ed25519
signature crate: https://github.com/RustCrypto/traits/tree/master/signature

The ed25519 crate has been natively adopted by ed25519-dalek:

Issue: https://github.com/dalek-cryptography/ed25519-dalek/issues/80
Usage example: https://docs.rs/ed25519/1.0.2/ed25519/#using-above-example-with-ed25519-dalek

If sodiumoxide were to adopt these types/traits as well, I'd be happy to add similar documentation to the ed25519 crate for how to use them with sodiumoxide.

FYI, I already maintain a "wrapper crate" which enables the usage of these Ed25519 signature types/traits:
https://docs.rs/signatory-sodiumoxide/0.21.0/signatory_sodiumoxide/
However, I think it'd be great if sodiumoxide added native support the same way ed25519-dalek did.

Note: per #445 I'm definitely still interested in seeing this through. Just wanted to make sure there's actually interest in this PR before completing it.

Yes I am interested in this PR.

The tricky remaining question is what to do about serde support. If you'd like to use the built-in support in the ed25519 crate, it'd require using a hack similar to the one in ed25519-dalek:
https://github.com/dalek-cryptography/ed25519-dalek/pull/124/files#r393865319
If that seems acceptable I can do the same thing, but I just wanted to make sure first.
Note that there's been quite a bit of discussion about serde support in ed25519-dalek, and it seems like the serializers the ed25519 crate provides may differ from the ones you presently define. See this issue for some additional discussion:
https://github.com/dalek-cryptography/ed25519-dalek/pull/146

TBO I personally can't find serde feature useful at all... However we can work on it.

Okay, I can update the PR to do what I did in ed25519-dalek and we can go from there.

I went ahead and rebased this PR, and for now just removed the serde test, as well as WIP.
I think it's pretty problematic to serialize Ed25519 signatures using serde. See the discussion on this issue:
https://github.com/dalek-cryptography/ed25519-dalek/pull/146
Without looping in an additional crate like serde_bytes, the serialization will be bad for at least one of: bincode, CBOR, JSON.
So I left it out for now, but if you'd like me to revisit that I can.

@sodiumoxide/owners  How about breaking serde feature and revisiting in sometimes later when this feature is requested?

Agreed, as it stands now serde support is a misfeature and it will keep
being one until the derive-macros work on large arrays, but that will break
the serialization format anyway.
On Mon, 7 Dec 2020 at 18:06, Roman Proskuryakov<EMAIL_ADDRESS>wrote:

@sodiumoxide/owners https://github.com/orgs/sodiumoxide/teams/owners
How about breaking serde feature and revisiting in sometimes later when
this feature is requested?
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
https://github.com/sodiumoxide/sodiumoxide/pull/434#issuecomment-740051317,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJ5KBKMRDCWOTLNNQ2D6NLSTUDSRANCNFSM4RIRX2DQ
.

bors: r+


add tests for multiple HTTP requests

The HTTP driver can send multiple concurrent requests, but can only
receive one at a time. If JavaScript were ever to get true parallelism,
one could use multiple HTTP drivers to account for multiple physical
network ports that could receive complete HTTP requests at the same
time. This adds tests for sending multiple requests and receiving them
individually.

Codecov Report

Merging #261 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #261   +/-   ##
=======================================
  Coverage   93.35%   93.35%
=======================================
  Files          26       26
  Lines         843      843
  Branches      178      178
=======================================
  Hits          787      787
  Misses         49       49
  Partials        7        7

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6eb9010...d9cb7a7. Read the comment docs.


Fix a potential bug hinted by clang

Originally part of #17382.

I think I see what that was about. Do you have the original Clang warning?

This is what the macros ultimately evaluate to
math_OP_EQUAL_INT: {                                  \
		if (p_b.type == INT) {                   \
		r_ret = p_a._data._int  == p_b._data._int; \
		return;         \
	};   \
		if (p_b.type == REAL) {                   \
		r_ret = p_a._data._int  == p_b._data._real; \
		return;         \
	}; \
		if (p_b.type == NIL) {                   \
		r_ret = !p_b.type == NIL; \
		return;         \
	};                     \
                                                                              \
		{                    \
		r_valid = false; \
		return;          \
	}                                                          \
	};

Due to operator precedence this is what actually happens:
( ! is right-to-left, == is left-to-right)
r_ret = (!(p_b.type == NIL));

Which is what this patch is looking to make explicit here. I'm all for fixing warnings like this! I don't think that this could have caused any bugs though.

Cherrypicked into 3.0.3


Refactor codebase and fix bugs

This pull request includes several code refactorings and bug fixes. The refactorings involve replacing throw statements with new Error instances in multiple files. The bug fixes address issues related to keychain existence, Archethic node reachability, and the instance variable in the ArchethicRPCClient class. Additionally, there are improvements to functions such as uint8ArrayToInt, wordArrayToUint8Array, and sortObjectKeysASC.

Hi, great!
Actually it's because I use VS Code with the SonarLint extension.
Before this PR there are 78 "problems" after which there are only 6 remaining including 3 TODO.


'hash_size' may be used uninitialized in this function

Discussed in https://github.com/DMTF/libspdm/discussions/924

Originally posted by Wu-Sisi June  3, 2022
https://github.com/DMTF/libspdm/blob/8a1054346f049db47c4bcba6c73a92dd74afff81/library/spdm_common_lib/libspdm_com_context_data.c#L1607
'hash_size' may be used uninitialized in this function [-Werror=maybe-uninitialized]

Hi @Wu-Sisi How did you find this issue? Was it with GCC? If so, what version?

After investigation, I think it is false-alarm. hash_size in 1607 is initialized in every path go here.
@Wu-Sisi , would please tell us how you find this issue?
If we really need handle, we should enable the compiler by default, and fix all of them.
Otherwise, I will prefer to close it.


chore: Copy barretenberg.wasm when compiling typescript

Description
This avoids the need to explicitly compile for the wasm target. We use the output from yarn build instead.
Checklist:

[ ] I have reviewed my diff in github, line by line.
[ ] Every change is related to the PR description.
[ ] The branch has been merged with/rebased against the head of its merge target.
[ ] There are no unexpected formatting changes, superfluous debug logs, or commented-out code.
[ ] There are no circuit changes, OR a cryptographer has been assigned for review.
[ ] New functions, classes, etc. have been documented according to the doxygen comment format. Classes and structs must have @brief describing the intended functionality.
[ ] If existing code has been modified, such documentation has been added or updated.
[ ] No superfluous include directives have been added.
[ ] I have linked to any issue(s) it resolves.
[ ] I'm happy for the PR to be merged at the reviewer's next convenience.

superceded by #604


Validation of non-pydantic types?

Hey there - love this package! Quick question - what's the best practice for having dependencies that aren't valid Pydantic types - for example, I'm using it to inject an API client like so:
@inject
async def cancel_running_flow_runs(
    flow_name: str,
    client: PrefectClient = Depends(prefect.get_client)
):

I get the error check that <class 'prefect.client.orchestration.PrefectClient'> is a valid pydantic field type
Now I could remove the type hint and it would work but I'd lose type hints within the function. I might be missing something obvious in the docs but curious if there's a way to either add a new type or allow arbitrary types in arguments?

This is the workaround I'm currently using:

def inject_prefect_client():
    client = prefect.get_client()
    yield client

@inject
@sync_compatible
async def cancel_running_flow_runs(
    flow_name: str,
    client=Depends(inject_prefect_client)
):
    # func body hack
    client: PrefectClient = client
    ...

Hi there! Appritiate your Issue. Indeed, it's my bad.
Right now I release version 1.1.4 to fix this bug.
Now you can use Depends(*, cast=False) to ignore a function parameter by pydantic.
Now your code will works correct way
@inject
async def cancel_running_flow_runs(
    flow_name: str,
    client: PrefectClient = Depends(inject_prefect_client, cast=False),
):
    ...
```


Create AIP16

@Spaydh need the AIP to be in its own directory. Also please copy the latest markdown text from the ARC-16 forum post, I made some styling edits.


Update run dependencies for capabilities() and grSoftVersion()

…tests

Checklist

[x] Used a fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[ ] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

Can you revert the last commit so that we can keep this recipe in sync with defaults?

@isuruf I intended to do 3 commits here

Adding the two functions call capabilities() and grSoftVersion() to tests (done) -> green
Moving libxt from yum_requirements as xorg-libxt to host dependencies (done) -> intended to be red when the two functions as called in tests, but seems to be green
Adding xorg-libxt to run dependencies (not yet pushed) -> intended to be green again.

Background to this is:

https://github.com/conda-forge/r-rmarkdown-feedstock/pull/24
https://github.com/conda-forge/r-rmarkdown-feedstock/pull/23
where it was discovered that xorg-libxt is required and a currently missing dependency to not get a segmentation fault wenn calling capabilities() and grSoftVersion()

@isuruf How do you propose to proceed here? keep libxt-devel in yum_requirementes but still also add xorg-libxt to run dependencies?

@isuruf Now all three commits are there. Can rebase and remove the second one.

Removed second commit "cd3aed4 Move libxt from yum_requirements to host dependencies"

Can you revert the last commit so that we can keep this recipe in sync with defaults?

Alternatively we could update the defaults recipe as well. If we think it is better to rely on the xorg- conda binaries over the RPM binaries, then wouldn't it make sense to make this improvement to both the conda-forge and defaults recipes?

@conda-forge-bot please rerender

@conda-forge-admin, please re-render

@mingwandroid, thoughts?

then wouldn't it make sense to make this improvement to both the conda-forge and defaults recipes?

Anaconda have no plans to use the xorg packages at present. The recipe would have to check channel_targets  and switch dependencies based on that for us to be able to share it. Easy enough to do though and we can do that over at anaconda when we resync.

@conda-forge-admin, please re-render

@conda-forge-admin, please rerender

@dbast I'm not sure why the bot can not update your PR, could you update it please or should I rerender and push to your PR?


[web_socket_link] The link doesn't send StopOperation to the backend

Hi there. Help needed.
I faced an issue that when I close StreamSubscription of a stream WebSocketLink provides, the following code should be called:
 186   response.onCancel = () {
 187     messagesSubscription?.cancel();
 188     _write(StopOperation(id)).catchError(response.addError);
 189     _requests.removeWhere((e) => e.context.entry<RequestId>()!.id == id);
 190   };

And then it should receive the SubscriptionComplete status from the backend and close the StreamController:
 170          } else if (message is SubscriptionComplete) {
 171            response.close();
 172          }

According to this and this test, it should work.
But onCancel callback is not called. hasListener method shows that it still has listeners.
Here's the link I created:
class AuthorizedWebSocketLink extends Link {
  AuthorizedWebSocketLink(this.url);

  final String url;
  gql.WebSocketLink? _client;

  @override
  Stream<Response> request(Request request, [NextLink? forward]) async* {
    final context = request.context;
    final headers = context.entry<HttpLinkHeaders>()?.headers ?? {};

    final controller = StreamController<Response>();
    StreamSubscription<Response>? subscription;

    _client ??= gql.WebSocketLink(
      null,
      initialPayload: {
        'authToken': 'token',
      },
      channelGenerator: () => IOWebSocketChannel.connect(
        Uri.parse(url),
        protocols: ['graphql-ws'],
        headers: headers,
      ),
    );

    // I implemented it in this way to make sure the following code will be called
    controller.onListen = () {
      subscription = _client!.request(request, forward).listen(
            controller.add,
            onDone: controller.close,
          );
    };

    // I can confirm that this will be called in my implementation and it will call subscription?.cancel() as well
    controller.onCancel = () {
      subscription?.cancel();
      controller.close();
    };

    yield* controller.stream;
  }

  void dispose() {
    _client?.dispose();
    _client = null;
  }
}

Client initialization:
    _client = GraphQLClient(
      link: Link.from([authLink, errorLink, authorizedWebSocketLink]),
      cache: GraphQLCache(store: InMemoryStore()),
    );

And the code that tests the implementation:
  StreamSubscription<QueryResult>? _streamSubscription;

  void _subscribe() {
    final vars = generated.RequestArguments(id: 179);
    final subscription = generated.Subscription(variables: vars);
    final options = SubscriptionOptions(
      document: subscription.document,
      operationName: subscription.operationName,
      variables: subscription.variables.toJson(),
    );

    _streamSubscription = _client!.subscribe(options).listen(print);
  }

  void _unsubscribe() {
    _streamSubscription?.cancel();
  }

What am I missing? Thanks

I faced the same issue. Any updates how to fix this?

I am currently busy with work, I might take up two weeks to fix this issue.

Hi @agent3bood, any updates or insights on what's causing the issue?

Hi, are there any updates?

I have not worked on it yet, is anybody is willing to create a fix?


Example tags missing in data/README

The GH-rendered version of data/README.md doesn't show the angle tags in the example data (because the GFM interpreter tries to treat <HIGH-INT> as an HTML tag)

could you show the details of how to run the model ,thanks


ffmpeg merge error to make webm + m4a = mp4 no video

[x ] I've verified and I assure that I'm running youtube-dl 2018.06.19

Before submitting an issue make sure you have:

[x ] At least skimmed through the README, most notably the FAQ and BUGS sections
[x ] Searched the bugtracker for similar issues including closed ones
[x ] Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

[ x] Bug report (encountered problems with youtube-dl)

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:
Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):
C:\Users\[user]\Desktop>youtube-dl.exe -v -f bestvideo+bestaudio --merge-output-fo
mat mp4 --ffmpeg-location C:\Users\[user]\Desktop\ffmpeg-4.0-win64-static\bin http
://www.youtube.com/watch?v=tocKgk76O1g
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '-f', 'bestvideo+bestaudio', '--merge-output-
ormat', 'mp4', '--ffmpeg-location', 'C:\\Users\\[user]\\Desktop\\ffmpeg-4.0-win64-
tatic\\bin', 'https://www.youtube.com/watch?v=tocKgk76O1g']
[debug] Encodings: locale cp950, fs mbcs, out cp950, pref cp950
[debug] youtube-dl version 2018.06.19
[debug] Python version 3.4.4 (CPython) - Windows-7-6.1.7601-SP1
[debug] exe versions: ffmpeg 4.0, ffprobe 4.0
[debug] Proxy map: {}
[youtube] tocKgk76O1g: Downloading webpage
[youtube] tocKgk76O1g: Downloading video info webpage
[debug] Invoking downloader on 'https://r6---sn-ibj-i3b6.googlevideo.com/videop
ayback?gcr=hk&id=o-AFkF4smYiwFySjOUeOK5NrhIew81YCzK7GA3Hg4FJd61&fexp=23709359&c
WEB&fvip=4&gir=yes&signature=A8F02A0741D0573DA1A4787EA309D591463F9722.4F7E5299C
73F2CC407E0D90F2C858DF4BE3C576&ip=<IP_ADDRESS>&requiressl=yes&initcwndbps=13
8750&itag=248&ms=au%2Crdu&mt=1529689401&aitags=133%2C134%2C135%2C136%2C137%2C16
%2C242%2C243%2C244%2C247%2C248%2C278&mv=m&ipbits=0&sparams=aitags%2Cclen%2Cdur%
Cei%2Cgcr%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Ckeepalive%2Clmt%2Cmim
%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Crequiressl%2Csource%2Cexpire&keepalive=yes&mm=31%2C
9&mn=sn-ibj-i3b6%2Csn-i3b7kn76&ei=sjUtW-27CJKdqQHXjIGwBw&clen=457054162&expire=
529711122&mime=video%2Fwebm&pl=18&lmt=1529670873962231&source=youtube&dur=4323.
20&key=yt6&ratebypass=yes'
[download] Destination: 【1pm直播 Live】真係繼續貿戰！舜宇反向路演 + 小米$4444上
市的將來-tocKgk76O1g.f248.webm
[download] 100% of 435.88MiB in 00:48
[debug] Invoking downloader on 'https://r6---sn-ibj-i3b6.googlevideo.com/videop
ayback?gcr=hk&id=o-AFkF4smYiwFySjOUeOK5NrhIew81YCzK7GA3Hg4FJd61&fexp=23709359&c
WEB&fvip=4&gir=yes&signature=7DCE65EF12A873E7D7C35BCCAB326F76FEFDE21E.126A40A25
BBFC0D422FF4FEFD6457CB0E302995&ip=<IP_ADDRESS>&requiressl=yes&initcwndbps=13
8750&itag=140&ms=au%2Crdu&mt=1529689401&mv=m&ipbits=0&sparams=clen%2Cdur%2Cei%2
gcr%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Ckeepalive%2Clmt%2Cmime%2Cmm
2Cmn%2Cms%2Cmv%2Cpl%2Crequiressl%2Csource%2Cexpire&keepalive=yes&mm=31%2C29&mn=
n-ibj-i3b6%2Csn-i3b7kn76&ei=sjUtW-27CJKdqQHXjIGwBw&clen=68663613&expire=1529711
22&mime=audio%2Fmp4&pl=18&lmt=1529669830661680&source=youtube&dur=4323.207&key=
t6&ratebypass=yes'
[download] Destination: 【1pm直播 Live】真係繼續貿戰！舜宇反向路演 + 小米$4444上
市的將來-tocKgk76O1g.f140.m4a
[download] 100% of 65.48MiB in 00:06
[ffmpeg] Merging formats into "【1pm直播 Live】真係繼續貿戰！舜宇反向路演 + 小米
$4444上市的將來-tocKgk76O1g.mp4"
[debug] ffmpeg command line: "C:\Users\[user]\Desktop\ffmpeg-4.0-win64-static\bin\
fmpeg" -y -i "file:【1pm直播 Live】真係繼續貿戰！舜宇反向路演 + 小米$4444上市的
將來-tocKgk76O1g.f248.webm" -i "file:【1pm直播 Live】真係繼續貿戰！舜宇反向路演
+ 小米$4444上市的將來-tocKgk76O1g.f140.m4a" -c copy -map "0:v:0" -map "1:a:0" "
ile:【1pm直播 Live】真係繼續貿戰！舜宇反向路演 + 小米$4444上市的將來-tocKgk76O1
.temp.mp4"
Deleting original file 【1pm直播 Live】真係繼續貿戰！舜宇反向路演 + 小米$4444上
市的將來-tocKgk76O1g.f248.webm (pass -k to keep)
Deleting original file 【1pm直播 Live】真係繼續貿戰！舜宇反向路演 + 小米$4444上
市的將來-tocKgk76O1g.f140.m4a (pass -k to keep)

Description of your issue, suggested solution and other information
Target: download best video with sound
First attempt: use no parameter download command with only URL
Result: only 720p (with sound) download while 1080p available
Second attempt as above
Result: mp4 file with only sound
P.S. the video is originally a live stream.
Or --recode-video needed?

webm+mp4 can't be merged into mp4 container. Don't touch options you have no idea about.
To download maximum quality you don't need to specify any options explicitly.

Please be noticed that i have checked the manual to know default: best quality.
However, it is not working on the link i provided, ref: first attempt: only 720p downloaded while 1080 exists

It is working..
Trying the link with default youtube-dl settings here gave me format code f248 for video, which is the correct one.

@Hrxn
may i know what platform are you on?
C:\Users\family\Desktop>youtube-dl.exe -v https://www.youtube.com/watch?v=tocKgk76O1g
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://www.youtube.com/watch?v=tocKgk76O1g']
[debug] Encodings: locale cp950, fs mbcs, out cp950, pref cp950
[debug] youtube-dl version 2018.06.19
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.17134
[debug] exe versions: none
[debug] Proxy map: {}
[youtube] tocKgk76O1g: Downloading webpage
[youtube] tocKgk76O1g: Downloading video info webpage
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'https://r6---sn-ibj-i3b6.googlevideo.com/videoplayback?key=yt6&ratebypass=yes&pl=18&dur=4323.207&expire=1529780644&signature=D181DE262C76CF3C34B42EFA6E7958428C13A9FE.6A1D88EE153ED73FBF85A775E818DEC1BD4AF72F&ipbits=0&itag=22&mime=video%2Fmp4&requiressl=yes&mm=31%2C29&mn=sn-ibj-i3b6%2Csn-i3b7kn76&c=WEB&mt=1529758906&mv=m&ms=au%2Crdu&source=youtube&ip=<IP_ADDRESS>&sparams=dur%2Cei%2Cgcr%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Cratebypass%2Crequiressl%2Csource%2Cexpire&fexp=23709359&gcr=hk&initcwndbps=1183750&id=o-ANb3mXEkBntAWkR6iGxJMi_2ebEVD1G3GyfXcOztb9Lo&ei=REUuW_yfBYyYgAPjqIfwDw&lmt=1529670085309855&fvip=4'
[download] Destination: 【1pm直播 Live】真係繼續貿戰！舜宇反向路演 + 小米$4444上市的將來-tocKgk76O1g.mp4
Using 2018.06.19 version

Windows 10..
This is the problem:

[debug] exe versions: none

You need FFmpeg, from here, for example: https://ffmpeg.zeranoe.com/builds/
And put the FFmpeg binaries somewhere into your PATH or next to youtube-dl.exe.

thanks.
For others, if not detecting ffmpeg, youtube-dl automatically download "highest" quality video which may be 720 instead of 1080.


rename head check file name

Coverage remained the same at 18.254% when pulling dbb124fc655186d984046823cf131160979162d2 on feature/headcheck into 815d537772b00384d454b82be3d7a913abad3e0c on master.


Distinct variants only

We have a single set of AnVIL project that contain duplicate variants.  Literally:
In [15]: mt.filter_rows(mt.locus.position == 3834167).GT.show()
+---------------+------------+--------------+==================>  (24 + 1) / 25]
| locus         | alleles    | 'sample_id'.GT |
+---------------+------------+--------------+
| locus<GRCh38> | array<str> | call         |
+---------------+------------+--------------+
| chr1:3834167  | ["T","G"]  | 1|0          |
| chr1:3834167  | ["T","G"]  | 1|1          |
+---------------+------------+--------------+

I think also reasonable to just throw an exception here... but that might end up being more work for us in the long run.

I think we should throw an exception here, we don't know what this means and I wouldn't want these counting toward our global AFs. I think if we really start to have too much work communicating errors in peoples VCFs back to them, the solution is to automate that communication, not just accept crappy data.
Which project is this? I'd like to see how its currently working in seqr

unless there was handling for this in the v2 pipeline, in which case I am okay maintaining that behavior

A few things:

Re the v2 pipeline.... we have no 'distinct' handling there from what I can tell.
Only one of the rows will be counted in the global AFs, as we do an inner join, fwiw.

This variant loaded successfully in the v2 pipeline, with GT of 1|1


Is there a way to only sort subgroups?

Is your feature request related to a problem? Please describe.
No
Describe the solution you'd like
Is there a way to only sort subgroups? My imports are all grouped in subgroups which are delimited by a blank line, and I would like to keep them as they are, but would of course like the ability to sort them.

Could you elaborate your requirement, better with some examples?
Thanks!

@paulflo150
Thanks for you feedback!
Is it still a requirement for you? If no, I'll close the issue later.


ListField fix

Fixes ListField when there is a model array difference.

Coverage increased (+0.0002%) to 99.89% when pulling 3121ea05163d3876f161babbbfdeb14c3c7ff193 on ksobolewski-netins:v2 into 9f0cf17df7bae0fa5c266a9fb475f468d6e2e044 on mprzodala:v2.


using fxgl and getting java.lang.IllegalStateException: EntityFactory was not set!

I was following your titorial at https://www.youtube.com/watch?v=yrRfma4a_7U&t=718s and i get that error java.lang.IllegalStateException: EntityFactory was not set! what doexthis mean and how can i fix this.
https://stackoverflow.com/questions/45069531/im-having-an-illegalstateexception-entityfactory-was-not-set-with-fxgl
that is the link to the code on stack overflow you can see it there thanks you for your help.
i have also tried multiple versions from 0.3.0 to 0.3.5 and none have worked...

This has been answered on the stackoverflow link. Let me know if it was fixed.

The issue has been resolved


./setup_p2.sh: line 119: syntax error: unexpected end of file error on mac

Hi, this is a great DL course, thanks very much!
I got "./line 119: syntax error: unexpected end of file" when trying to run the setup_p2 script on a mac machine. I have tried to use dos2unix and the method described in https://github.com/fastai/courses/issues/17
but didn't fix the error.
Have you seen a similar error? Thanks!

I have had the same problem and have not been able to resolve it.

Problem fixed.
See
https://github.com/fastai/courses/issues/21


Do more objective

Add special objects to store points and traces. Do it more easy to use.

Add additional way to add relations:
    new Dijkstra(function($creator){
        $creator->addPoint(1)->addRelation(3, 5);
        $creator->addPoint(2)->addRelation(1, 2)->addRelation(3, 8);
        $creator->addPoint(3);
    });


[EC-56] EVM Block Information opcodes

This PR adds op codes for block info
~It depends on https://github.com/input-output-hk/etc-client/pull/64~

[info] - should disconnect from non-ETC peer *** FAILED *** (3 seconds, 273 milliseconds)
[info]   java.lang.AssertionError: assertion failed: timeout (3 seconds) during expectMsg while waiting for SendMessage(Disconnect(Useless peer))

3 seconds... just taking a note of that.


spyder broken with IPython 4

I get

ShimWarning: The IPython.qt package has been deprecated. You should import from qtconsole instead.

and then consequently:
Traceback (most recent call last):
  File "/usr/lib/python3.4/site-packages/spyderlib/spyder.py", line 3103, in main
    mainwindow = run_spyder(app, options, args)
  File "/usr/lib/python3.4/site-packages/spyderlib/spyder.py", line 2989, in run_spyder
    main.setup()
  File "/usr/lib/python3.4/site-packages/spyderlib/spyder.py", line 830, in setup
    from spyderlib.plugins.inspector import ObjectInspector
  File "/usr/lib/python3.4/site-packages/spyderlib/plugins/inspector.py", line 43, in <module>
    from spyderlib.widgets.ipython import IPythonControlWidget
  File "/usr/lib/python3.4/site-packages/spyderlib/widgets/ipython.py", line 29, in <module>
    from IPython.qt.console.rich_ipython_widget import RichIPythonWidget
ImportError: No module named 'IPython.qt.console.rich_ipython_widget'

/edit: ah, the arch packages just didn’t include qtconsole up until now.
it works, but we should still heed the deprecation warning and try importing from qtconsole first.

Spyder will move to the new convention in the upcoming release, and there might be also a minor release to handle this imports coming soon...

Closing as duplicate of issue #2608 (which is already solved for the stable branch).
qtconsole will be added as a hard dependency for 3.0 beta2. This problem is also solved in the latest master (i.e. Spyder doesn't crash if it can find qtconsole).

great, thank you!


I would like to contribute

I would like to contribute, I can add the code that checks if a single query has multiple plans
Do you agree?

1 no, no issues found. The code is perfect
2 okay, thank you for the advice


Check user/keys of RMDs retrieved from mdJournal

Also put the currentUID and currentVerifyingKey parameters
first consistently (after ctx).

Looks good to me, thanks!


Add network firewall table

Integration test logs

Logs
Add passing integration test logs here

Example query results

select * from oci_network_firewall_firewall where id = ''
and lifecycle_state = 'ACTIVE'
and network_firewall_policy_id = ''
and availability_domain = 'rMTb:AP-SYDNEY-1-AD-1'
and display_name = ''
and compartment_id = ''
select * from oci_network_firewall_policy
select * from oci_network_firewall_policy where id = ''
Results
Add example SQL query results here (please include the input queries as well)

Closing this PR, since the tables were addressed in https://github.com/turbot/steampipe-plugin-oci/pull/507
Thanks @scotti-fletcher for the contribution and the design of the table 👍.
Apologies for the radio silence on the PR.


[Bug] Export assumes temp folder exists

https://github.com/z1pti3/jimi/blob/f707258b14b5de648a9826611e05e0768e2f339e/web/conductEditor.py#L345
Export assumes the data folder contains another folder called temp

Temp folder is part of standard install requirements


Docker Installation Issue on MAc

Just installed Docker and verifying the installation with
docker run hello-world
Expected behavior
hello world image should be imported and docker should say heelo
Actual behavior
Getting following error
Unable to find image 'hello-world:latest' locally
docker: Error response from daemon: Get https://registry-1.docker.io/v2/library/hello-world/manifests/latest: Get https://auth.docker.io/token?scope=repository%3Alibrary%2Fhello-world%3Apull&service=registry.docker.io: dial tcp: lookup auth.docker.io on <IP_ADDRESS>:53: dial tcp <IP_ADDRESS>:53: getsockopt: connection refused.
See 'docker run --help'.
Information about the Issue
I'm getting this issue after installing Docker on Mac
Docker version 1.12.1, build 6f9534c
Steps to reproduce the behavior

Install Docker on Mac
Run command -> docker run hello-world

I tried to import image from kitematic as well but getting same error for any image

Could you open an issue in github.com/docker/for-mac instead?
Thanks


401 on PUT message

Hello, I receive error on Message API CALL :
var ctxioClient = ContextIO({
    key: config.contextio.key,
    secret: config.contextio.secret,
    version: config.contextio.version
  });
  //PUT https://api.context.io/lite/users/ID/email_accounts/LABEL/folders/FOLDER/messages
  ctxioClient.users(config.contextio.userId)
    .email_accounts(config.contextio.label)
    .folders(config.contextio.inboxFolder)
    .messages(message.message_id)
    .put({ new_folder_id: ANOTHERFOLDER })
    .then(function (ctxioClientRes) {
      res.json(ctxioClientRes);
    });

/*
 * The error Stack :
Unhandled rejection StatusCodeError: 401 - {"type":"error","code":105,"value":"Invalid signature \/f9Qf7YEq44qY8CVmpLKtDperlM=. Signature base string should be PUT&https%3A%2F%2Fapi.context.io%2Flite%2Fusers%USERID%2Femail_accounts%2Fcontenu%2540EMAIL%253A%253ASMTP%2Ffolders%2FINBOX%2Fmessages%2F%MESSAGEID%2540%253E%2F&oauth_consumer_key%%26oauth_nonce%%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%%26oauth_version%3D1.0"}
    at new StatusCodeError (/home/charles/dev/komunity-app-web/node_modules/request-promise/lib/errors.js:26:15)
    at Request.RP$callback [as _callback] (/home/charles/dev/komunity-app-web/node_modules/request-promise/lib/rp.js:68:32)
    at Request.self.callback (/home/charles/dev/komunity-app-web/node_modules/request-promise/node_modules/request/request.js:186:22)
    at emitTwo (events.js:100:13)
    at Request.emit (events.js:185:7)
    at Request.<anonymous> (/home/charles/dev/komunity-app-web/node_modules/request-promise/node_modules/request/request.js:1081:10)
    at emitOne (events.js:90:13)
    at Request.emit (events.js:182:7)
    at IncomingMessage.<anonymous> (/home/charles/dev/komunity-app-web/node_modules/request-promise/node_modules/request/request.js:1001:12)
    at IncomingMessage.g (events.js:273:16)
    at emitNone (events.js:85:20)
    at IncomingMessage.emit (events.js:179:7)
    at endReadableNT (_stream_readable.js:913:12)
    at _combinedTickCallback (node.js:383:13)
*/

I've been able to reproduce this one, gonna poke around and see what I can find. Thanks!

We've identified the issue and are working on a fix. Stay tuned!

Did this get fixed?  I'm still running into this issue.

@dado3212 I think this is a separate issue. This one was a signing issue on the server, wasn't actually related to this lib.

You can recreate it by sending new_folder_id in the body parameter of a put request (as described in #43).

Yes, and I am investigating that issue. However, these are not the same.
This issue was a 401 reporting that the oauth signature was invalid.
That issue is a 400 reporting that a required parameter is missing.

The API requires the new_folder_id to be included as a query string parameter.  If it's instead as a form variable, it throws a 401 oauth error.  If there's no form variable but no query string parameter, it 400's.  The issue arises because the default .put({ new_folder_id: folder }) sets the form variable, not the query string parameter.

Let's keep the discussion over on https://github.com/contextio/ContextIO-node/issues/43 for now.


Allow missing or empty clientSecret when not required

Includes case when client_secret is decoded as empty string from an Authorization header.
Solves #403. Includes test that fails on master.

Thanks @mjsalinger just confirmed w v3.0.0.


Reformat everything according to Avast code convention

Currently there are two styles in the library (tabs & spaces). We should unify it using this style before next release:
https://github.com/avast/android-styled-dialogs/blob/master/code-formatting-config.xml

The code was reformatted by @martinpelant, thanks


[3rdparty] remove oneCCL enviroment depency.

@changqi1 Please help to update gitee.diff

link to issue #94

@Duyi-Wang  There are 2 oneccl path?
/3rdparty/oneCCL
/3rdparty/oneccl

@Duyi-Wang There are 2 oneccl path? /3rdparty/oneCCL /3rdparty/oneccl
Do you mean oneCCL in .gitignore?
Because currently, the manually installed directory is named "oneCCL," setting it as "oneccl" can avoid overwriting.

Update README and prepare_oneccl.sh check path.


accessing the $store

i see you can pass in a variable to the options called store where you can access the store to get values(using in fetch data), but i'm getting some weird errors.
Uncaught (in promise) TypeError: Cannot read property 'state' of undefined
at a (liquor-tree.esm.js?6487:6)
at VueComponent.connectStore (liquor-tree.esm.js?6487:6)
at eval (liquor-tree.esm.js?6487:6)
how i'm calling it/configuration:
<tree v-if="websiteId != 0" :options="treeOptions" @node:selected="onNodeSelected" />
treeOptions: {
store: this.$store,
fetchData (node) {
return HTTP.get(/api/websitenavigation?nodeid=${node.id}&websiteId=${this.store.getters.websiteId}).then(function (response) { return response.data })
}
}
if i take out the  store: this.$store and hard code the id (${this.store.getters.websiteId}, i get exactly as i would expect it to work.  any ideas or help?
thanks,
matt

maybe a better question is how can i access a variable on the page or in the store from the fetchdata function?

Probably your case isn't connected to the library.  Check this page:
https://amsik.github.io/liquor-tree/#Integration-with-Vuex


Render groups as bootstrap tabs

Wishing I could use groups to render the form in a different way, like with each section having it's own tab. I could have multiple vfg instances with one form per tab. Or maybe I could use multi attribute to just hide all fields that are not in that tab.
Thoughts on best way to do this?

Or use visibility function that can somehow determine which tab is in use?

This works great:
Take your source schema and insert a function for the visibility field key's value
So if you want to show/hide field(s) based on the value of another field set their visibility to a function like
function(){
   return this.model.fieldA == 'someValue'
}

of course use what ever logic you want now.

For tabs see vue-form-wizard here https://github.com/cristijora/vue-form-wizard

@DelfsEngineering thanks so much for the suggestions! I'll definitely take a harder look at the vue-form-wizard.  Know of any examples of how to make the tabs look like regular bootstrap tabs?  It's possible that I could rethink whether my app should have more of a guided wizard UX, but converting my app over to how it currently works would be great.

@DelfsEngineering also, do you know of any examples of how the multi field and multiple option is used in practice? I see what the documentation says, but I suspect there's a use-case or use-cases behind supporting that and the merge logic that respects multi.  Thanks again for any help!

I suppose the best you could do without forking vue-form-wizard is some injected CSS, or hacking the accompanying CSS.

Hey @ericpyle
For tabs
https://github.com/spatie/vue-tabs-component
https://github.com/cristijora/vue-tabs
For a more strict flow/wizard
https://github.com/cristijora/vue-form-wizard
The wizard supports tab layout, however they do not act like tabs since the wizard has a stricter validation flow (e.g you can't click a tab unless it was visited and validated)
https://jsfiddle.net/bt5dhqtf/99/
If you need tabs use tab components, if you need a strict validation flow use the wizard. Both tabs and the wizard should work fine with form-generator.

Thanks, @cristijora. I'll look into integrating vfg with the other tab components and also ask my team whether we want to move toward a wizard approach.
@DelfsEngineering I still have an outstanding question about use cases for multi and multiple in vfg, if anyone cares to share?

@ericpyle you mean multi select ?

I mean multi field property "if true, it will visible only if multiple is true in component attributes" https://icebob.gitbooks.io/vueformgenerator/content/fields/field_properties.html

@cristijora see reply above. Thanks!

@ericpyle Honestly not sure to what it refers. I've used the multiple property only on multi-select field which has vue-multi-select behind it.
Maybe @icebob could answer to this.

as far as this issue goes, I'm ready to close it. I think if needed, I could have one vfg instance that gets reloaded each time a user switches tabs.
@icebob before I close this, could you please explain what the multi field property is used for in conjunction with multiple vfg option? I've seen the documentation, but I haven't seen working examples to help me understand what it's used for

@ericpyle can we close this?

@zoul0813 yes


scGEN network.batch_removal error

Hi,
any ideas how to solve this issue?
adata format looks like following

Here are commands I ran,
adata_hvg = adata[:, adata.var["highly_variable"]].copy()
adata_hvg = remove_sparsity(adata_hvg)
adata_hvg.X = adata_hvg.X.astype('float32')
network = sca.models.scgen(adata = adata_hvg,
                           hidden_layer_sizes=[256,128])

network.train(n_epochs=epoch, early_stopping_kwargs = early_stopping_kwargs)
corrected_reference_adata = network.batch_removal(adata_hvg, batch_key="sample", cell_label_key="celltype",return_latent=True)
The error occured at the last command.

TypeError                                 Traceback (most recent call last)
Cell In[83], line 1
----> 1 corrected_reference_adata = network.batch_removal(adata_hvg, batch_key="sample", cell_label_key="celltype",return_latent=True)
File ~/anaconda3/envs/scarches/lib/python3.9/site-packages/scarches/models/scgen/vaearith_model.py:147, in scgen.batch_removal(self, adata, batch_key, cell_label_key, return_latent)
127 def batch_removal(self, adata, batch_key, cell_label_key, return_latent = True):
128     """
129     Removes batch effect of adata
130
(...)
145         adata of corrected gene expression in adata.X and corrected latent space in adata.obsm["latent_corrected"].
146     """
--> 147     corrected = self.model.batch_removal(adata, batch_key, cell_label_key, return_latent)
148     return corrected
File ~/anaconda3/envs/scarches/lib/python3.9/site-packages/scarches/models/scgen/vaearith.py:261, in vaeArith.batch_removal(self, adata, batch_key, cell_label_key, return_latent)
259 adata_latent = anndata.AnnData(latent_all)
260 adata_latent.obs = adata.obs.copy(deep=True)
--> 261 unique_cell_types = np.unique(adata_latent.obs[cell_label_key])
262 shared_ct = []
263 not_shared_ct = []
File ~/anaconda3/envs/scarches/lib/python3.9/site-packages/numpy/lib/arraysetops.py:274, in unique(ar, return_index, return_inverse, return_counts, axis, equal_nan)
272 ar = np.asanyarray(ar)
273 if axis is None:
--> 274     ret = _unique1d(ar, return_index, return_inverse, return_counts,
275                     equal_nan=equal_nan)
276     return _unpack_tuple(ret)
278 # axis was specified and not None
File ~/anaconda3/envs/scarches/lib/python3.9/site-packages/numpy/lib/arraysetops.py:336, in unique1d(ar, return_index, return_inverse, return_counts, equal_nan)
334     aux = ar[perm]
335 else:
--> 336     ar.sort()
337     aux = ar
338 mask = np.empty(aux.shape, dtype=np.bool)
TypeError: '<' not supported between instances of 'float' and 'str'

adata_hvg.obs['celltype'] = adata_hvg.obs['celltype'].astype(str)
solved the issue


sv-SE.json

Swedish translation

Du har råkat få med radnumereringen ;)

I'll close this as it isn't fixed yet. Feel free to open a new one if/when you fix it :)


reworked unittest support

please see https://github.com/dcarp/cmake-d/pull/23

nope, still a small mistake in


Use the proper prompt template for Llama2 chat models

The Llama2 chat models need a special prompt template to produce good output, see https://github.com/facebookresearch/llama#fine-tuned-chat-models.
We should add a function that automatically provides the proper prompt template to the model.

The same holds true for Phi-3. That model really requires the correct template.


Live IBKR OptionChainProvider & AddFutureOption not including daily expirations

Expected Behavior
In live trading with IBKR data that includes daily expiring Futures options. OptionChainProvider & AddFutureOption should return all expirations, including daily expiries. I understand that backtest data does not include daily expirations, but live IBKR live data does include daily expirations and is 24 hours for futures.
Actual Behavior
OptionChainProvider & AddFutureOption return only options matching Future expiry date and not daily expiries.
Potential Solution
Lean code may be filtering options based  on Future expiry, with no way to override this filter.
Reproducing the Problem
Run this as back test and also with IBKR live data:
https://www.quantconnect.com/terminal/processCache?request=embedded_backtest_3b5cfe799d9bab4f680cf2268b134831.html
Checklist

[ x] I have completely filled out this template
[ x] I have confirmed that this issue exists on the current master branch
[ x] I have confirmed that this is not a duplicate issue by searching issues
[x ] I have provided detailed steps to reproduce the issue

Lean currently does not support weekly and daily future options contracts.
In order to have this feature in a live IB deployment, the LiveOptionChainProvider would need to take into account all categories returned by CME (future options are source from CME even in an IB live deploment), which include the weekly/daily contracts.
Also, FuturesOptionsExpiryFunctions.GetFutureOptionExpiryFromFutureExpiry would have to be modifed to calculate the proper option expiration date (for instance, for a weekly monday expiration option, given the year, month and week number).
Backtesting would need to have this feature implemented as well.

Sorry, @vguruv we don't have this data. Procuring it is possible but at the moment it's a relatively niche asset class. If more commercial clients come on board seeking this asset we can expand support.

Sorry, @vguruv we don't have this data. Procuring it is possible but at the moment it's a relatively niche asset class. If more commercial clients come on board seeking this asset we can expand support.

Thanks for the attention @jaredbroad. This is only about live trading where IBKR does provide the data, but Lean is ignoring or unable to use it, due to discrepancy in Options vs FuturesOptions implementations. This is not about back testing (sorry if the issue description was not clear about that).

@jhonabreul  @jaredbroad @Martin-Molinero
Following up to ensure that it is clear that this is a live-only trading problem due to mismatch between options and futures options implementation and not a backtest data issue. @jhonabreul  has already identified the location and nature of the change needed to resolve this live-only trading issue.


How to allow to write a filtration condition in one line with WHERE keyword

I use a construction of this type in many places:
WHERE 1 = 1
    AND col1 = 'a'
    AND ...

This is convenient because I can comment out a line or several lines when debugging and nothing will break.
However, sqlfluff requires moving 1 = 1 to a new line. Is there any way I can configure sqlfluff to allow such constructions?
Noqa is not suitable because such a construction is very often used. Ignoring the LT02 rule is also not suitable because other indents issues should not be ignored.

I am not sure, you could ask in the main sqlfluff repo, they know more about configuration than I do.

Okay, thanks


RTP Closed

Given that a respondent goes to the RTP landing page (https://connectivity-rtp-test.apps.silver.devops.gov.bc.ca/)
When they read the page
Then at the bottom of the text in a new paragraph, it says: "Thank you for your interest in responding to the RTP. The response deadline has now passed. Please direct any further questions to<EMAIL_ADDRESS>And the "Login" Button is grayed out.

Work done by PR's #348 #349 - CLOSED

Prod: https://connectivity-rtp.apps.silver.devops.gov.bc.ca/


How can I test ActivitiesApiTest ?

I have generate java project using swagger-generator command
swagger-codegen generate -i https://developers.strava.com/swagger/swagger.json -c config/android.json -l java -o generated/java
How can I add authentication for test case ?
I just run test case I am getting fallowing  error
io.swagger.client.ApiException: Unauthorized
at io.swagger.client.ApiClient.handleResponse(ApiClient.java:926)
at io.swagger.client.ApiClient.execute(ApiClient.java:842)
at io.swagger.client.api.ActivitiesApi.getLoggedInAthleteActivitiesWithHttpInfo(ActivitiesApi.java:876)
at io.swagger.client.api.ActivitiesApi.getLoggedInAthleteActivities(ActivitiesApi.java:859)
at io.swagger.client.api.ActivitiesApiTest.getLoggedInAthleteActivitiesTest(ActivitiesApiTest.java:150)

Hi @ishara, we're doing some cleanup in our github repository.
If you are still having trouble with this, could you please send an email to<EMAIL_ADDRESS>Thanks!


Pando test failure

This test failed on SHA 678484a90a665f663631d7dc9b19d97af4e1ed21.
Symptom:
MongoDriverRecoveryTest > databaseDropped_recovers() > [1] {flushOrWait=FLUSH, parameters=MongoDriverRecoveryTest_1_PandoFormat} FAILED
    org.opentest4j.AssertionFailedError: expected: <TestEntity(id=root, string=after drop, catalog={123=TestEntity(id=123, string=123, catalog={}, listing=/catalog/123/catalog/[], sideTable=/catalog/123/catalog/{}, values=Optional.empty), 124=TestEntity(id=124, string=124, catalog={}, listing=/catalog/124/catalog/[], sideTable=/catalog/124/catalog/{}, values=Optional.empty)}, listing=/catalog/[123], sideTable=/catalog/{}, values=Optional.empty)> but was: <TestEntity(id=root, string=before disruption, catalog={123=TestEntity(id=123, string=123, catalog={}, listing=/catalog/123/catalog/[], sideTable=/catalog/123/catalog/{}, values=Optional.empty), 124=TestEntity(id=124, string=124, catalog={}, listing=/catalog/124/catalog/[], sideTable=/catalog/124/catalog/{}, values=Optional.empty)}, listing=/catalog/[123], sideTable=/catalog/{}, values=Optional.empty)>

logs_367.zip

Stack trace:
MongoDriverRecoveryTest > databaseDropped_recovers() > [1] {flushOrWait=FLUSH, parameters=MongoDriverRecoveryTest_1_PandoFormat} FAILED
    org.opentest4j.AssertionFailedError: expected: <TestEntity(id=root, string=after drop, catalog={123=TestEntity(id=123, string=123, catalog={}, listing=/catalog/123/catalog/[], sideTable=/catalog/123/catalog/{}, values=Optional.empty), 124=TestEntity(id=124, string=124, catalog={}, listing=/catalog/124/catalog/[], sideTable=/catalog/124/catalog/{}, values=Optional.empty)}, listing=/catalog/[123], sideTable=/catalog/{}, values=Optional.empty)> but was: <TestEntity(id=root, string=before disruption, catalog={123=TestEntity(id=123, string=123, catalog={}, listing=/catalog/123/catalog/[], sideTable=/catalog/123/catalog/{}, values=Optional.empty), 124=TestEntity(id=124, string=124, catalog={}, listing=/catalog/124/catalog/[], sideTable=/catalog/124/catalog/{}, values=Optional.empty)}, listing=/catalog/[123], sideTable=/catalog/{}, values=Optional.empty)>
        at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
        at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
        at app//org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
        at app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
        at app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
        at app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1145)
        at app//io.vena.bosk.drivers.mongo.MongoDriverRecoveryTest.testRecovery(MongoDriverRecoveryTest.java:293)
        at app//io.vena.bosk.drivers.mongo.MongoDriverRecoveryTest.databaseDropped_recovers(MongoDriverRecoveryTest.java:137)

I got a failure with debug logs here:
logs_634.zip

I have a branch with some fixes here.

I saw it fail twice in 26 runs, so I'd like to see at least 65 passing runs here before declaring success.
After that, I can clean up the branch and re-enable all tests, and see if I can survive a few dozen runs with no failures. 🤞

Rerunning the single test here, and the full suite on the PR here.

Passed the single test 101 times with no failures. I would have expected about 7 failures if the bug were present, so it's safe to say the bug is fixed.
I'm planning to run the full suite on repeat overnight to ensure I didn't break anything else.

Welp, it didn't run overnight... but it did pass 16 times with no failures. I'm going to call that a victory.


Sharepoint Online REST strange error (400)

Category

[x] Bug

Describe the bug
Since Sep 9, 2020 we started to get strange error on REST API call to get list of sub-sites and their lists:
GET <site url>_api/web?$expand=webs,lists,AllProperties,ThemeInfo,SiteUsers,RoleDefinitions&$select=*,webs/Url,lists/Id,lists/Title,lists/Description,lists/BaseType,lists/BaseTemplate,lists/Hidden,lists/Language,lists/ItemCount,lists/Created,lists/TemplateFeatureId,lists/CurrentChangeToken,SiteUsers/id,SiteUsers/Title,SiteUsers/LoginName,AllProperties/DesignPreviewThemedCssFolderUrl
The API call was working already for few year but something is broken now and in many our customers (not all , but few thousands of Sharepoint online tenants) we see this error.
{"error":{"code":"-2147467261, System.ArgumentNullException","message":{"lang":"en-US","value":"Value cannot be null.Parameter name: key"}}}
Now, we don't ask for any parameter "key", so not sure why it's null, but I found that if I remove the last part
AllProperties/DesignPreviewThemedCssFolderUrl
And request now looks :
GET <site url>_api/web?$expand=webs,lists,AllProperties,ThemeInfo,SiteUsers,RoleDefinitions&$select=*,webs/Url,lists/Id,lists/Title,lists/Description,lists/BaseType,lists/BaseTemplate,lists/Hidden,lists/Language,lists/ItemCount,lists/Created,lists/TemplateFeatureId,lists/CurrentChangeToken,SiteUsers/id,SiteUsers/Title,SiteUsers/LoginName
Then this request is returns fine, without any error
Expected behavior
GET <site url>_api/web?$expand=webs,lists,AllProperties,ThemeInfo,SiteUsers,RoleDefinitions&$select=*,webs/Url,lists/Id,lists/Title,lists/Description,lists/BaseType,lists/BaseTemplate,lists/Hidden,lists/Language,lists/ItemCount,lists/Created,lists/TemplateFeatureId,lists/CurrentChangeToken,SiteUsers/id,SiteUsers/Title,SiteUsers/LoginName,AllProperties/DesignPreviewThemedCssFolderUrl
The original API call should not return that error, or at least notifies how it can be resolved.
Environment details (development & target environment)
MacOS / Linux. Postman as well.
Additional context
This API call was working for few years without any issue (once we got Invalid color XML error, on that call).
Thanks
Please advise.

Same issue in CSOM when trying to load ThemeInfo on SPWeb.
SpContext.Load(spWeb, o => o.ThemeInfo);
SpContext.ExecuteQuery();

Some additional Information to add for triage -
This appears to be a tenant by tenant issue... as this call is working for me on my tenants.

This happens not for all tenants, but I would say for our customers , it's hundreds and hundreds. And even in our private tenant is also happens. I can share tenant id in private if it's needed.
Thanks

@Microsoft - Any progress on this ?

I validated some conditions because I'm experiencing this on different tenants, but not all. Server/farm versions are all identical for both the ones working and not working.
@rjaakke it seems it is not connected to a particular property or object. You experienced this with ThemeInfo and in the SPCB tool it triggered with Site.RootWeb.CurrentUser it seems. And if I leave out this check, then the next statement is on the Web will trigger the error.
Clearly the issue came up after September 9th. See timeline of the issues below.

In my case it REST call, started on Sep 9 as well, but I did a workaround , removed AllProperties/DesignPreviewThemedCssFolderUrl from the select statement

@bramdejager In our Provisioning tool we load a large portion of the available properties (that have a setter) in a single call.
The ThemeInfo object is wat caused the issue there. Because we only needed themeBackgroundImageUri from the ThemeInfo object we now load ThemeInfo.ThemeBackgroundImageUri instead and it solved the issue.
It could be that this is somehow related to expanding of objects in the API. This would explain why we see this issue loading different types of objects.

@andrewconnell @bcameron1231 Is someone at Microsoft can have a look on this issue ? Thanks

@slavag Check out the issue 98 for the SPCB tool referenced earlier in this thread. You see a response from Microsoft in there by Bert. They have an internal thread on this.


Ensure resolution of correct webpack dependencies

Merged to 9.0.x patch branch: https://github.com/angular/angular-cli/commits/9.0.x.


Support add custom CIDR for SSh Ingress Security group rule

It will be good if we can create new parameter for example SshIngressCidrIp instead of hardcoded <IP_ADDRESS>/0 in
https://github.com/buildkite/elastic-ci-stack-for-aws/blob/master/templates/aws-stack.yml#L1081
in our case we want to only allow ssh to agents from within our vpc Cidr
I can help create the PR when I'm free this week
Thanks!

Hi @bhegazy! Thanks for the suggestion. We're a bit hesitant to add more parameters to the stack, since it can already feel a little overwhelming for newcomers. As an alternative if you'd like security groups with custom ingress rules, I'd recommend setting the SecurityGroupId parameter in the stack to SGs you've made in your AWS account already – the one the stack creates is just there to help folks get started more easily.

Hi @bhegazy! Thanks for the suggestion. We're a bit hesitant to add more parameters to the stack, since it can already feel a little overwhelming for newcomers. As an alternative if you'd like security groups with custom ingress rules, I'd recommend setting the SecurityGroupId parameter in the stack to SGs you've made in your AWS account already – the one the stack creates is just there to help folks get started more easily.

Hi @chloeruka sounds reasonable, for our case we have terraform to run elastic-ci cf template so its easy to just include sg to module. However I think we should consider this option as well for anyone who just want to use the cf template directly, as it's not really a good practice to allow 22 <IP_ADDRESS>/0 . Thank you!

Hi @chloeruka sounds reasonable, for our case we have terraform to run elastic-ci cf template so its easy to just include sg to module. However I think we should consider this option as well for anyone who just want to use the cf template directly, as it's not really a good practice to allow 22 <IP_ADDRESS>/0 . Thank you!


refactor: move to functional components

#4, I couldn't actually test the Janus part works or not because the server seems to be inactive, but apart from that, everything seems to be working.

LGTM! Thanks for the contribution!!


Support transformation from Transactions to all Broadcasted Transactions

Currently, we only support Invoke simulate transactions, we would like to support all of them (Declare, Deploy, L1Handler).
First, make sure block_transaction_to_broadcasted_transaction supports all transaction types.
Secondly, make sure to update the rest of the module. For example, Display for transaction results doesn't show helpful information for all the types.

For example, block 642265 fails with the L1Handler not being supported.


[vtkm|vtk|paraview] update ports

[x] Changes comply with the maintainer guide.
[x] SHA512s are updated for each updated download.
[x] The "supports" clause reflects platforms that may be fixed by this new version.
[x] Any fixed CI baseline entries are removed from that file.
[x] Any patches that are no longer applied are deleted from the port's directory.
[ ] The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
[ ] Only one version is added to each modified port's versions file.

closes #36801
closes #36253
closes #36044
Need to look at
#36044
#36345
arm64|x64_osx: missing -lgfortran (need to get trace logs) -> this is wrong due to using pc files for lapack/blas
So blocked on #24327
dbow2 / theia:
CMake Error at D:/installed/x64-windows-static/share/vtk/FindExprTk.cmake:27 (string):
  Syntax error in cmake code at

    D:/installed/x64-windows-static/share/vtk/FindExprTk.cmake:27

  when parsing string

    version = \"(2\.7[0-9.]+)\".*$

  Invalid escape sequence \.

x64-linux: USER_ERROR__inconsistent_build_configuration__see_dlib_faq_1_ <- Inconsistent headers/preprocessor for linux
openmvs: not linking boost program_options and iostreams as it seems.

@BillyONeal: Tell me should i mark VTK as fail in the osx pipelines?
I am not going to solve the issues here since they would have been solved with #24327 already. The problem is dlib looking for lapack (thorugh e.g. pkg-config). Since lapack is still not correctly controlled by vcpkg it picks up lapack-reference instead of the accelerate framework which a) leads to the missing -lgfortran for it and inconsistence lapack usage on osx mixing lapack-reference and accelerate. As such the only way to solve this issue is to either patch dlib (which is not to blame here and thus I won't touch it) or mark the port as a fail in the CI pipeline.

@dg0yt Any Idea why FindPkgConfig used in FindSPATIALITE.cmake within GDAL finds the release versions instead the debug versions in debug builds?

Nevermind found the culprit. VTK is messing with CMAKE_PREFIX_PATH

@BillyONeal: Are the static windows builds of paraview here really running out of disk space or is something else going on here?

@WangWeiLin-MV is there a approximate eta for this PR to be merged, as my team is awaiting on the Paraview update.
related to the following issue: https://github.com/microsoft/vcpkg/issues/36801
fyi: @ jimwang118

@janicebeinrot Use an overlay until this is merged

@Neumann-A  thanks for the suggestion. I'll try that approach,  but what is an overlay specifically?  is there documentation I can read up on it.

@janicebeinrot: https://learn.microsoft.com/en-us/vcpkg/concepts/overlay-ports

https://github.com/microsoft/vcpkg/pull/24327/

#24327

What does this actually mean? That PR seems green except for CI hanging for one triplet for many days.
(#37651 needs the CMAKE_PREFIX_PATH fix which is included here.)

#24327

What does this actually mean? That PR seems green except for CI hanging for one triplet for many days.

I was commenting why I added depends:different-pr , because the description here says:

includes https://github.com/microsoft/vcpkg/pull/24327 to fix dlib linkage in the osx pipelines

that is, this PR implicates the blas/lapack monster.

that is, this PR implicates the blas/lapack monster.

That PR is nearly two years old and might have evolved feelings. Don't call it a monster you might hurt it feelings at this point ;)
I'll resolve the conflicts here if the other PR gets merged someday.

That PR is nearly two years old and might have evolved feelings. Don't call it a monster you might hurt it feelings at this point ;)

The PR isn't a monster. It's an attempt to tame the blas/lapack monster

Thanks for the updates!

Read for review again since rtabmap is baseline.

I'm good to go with these changes once CI is green :)

#38035 broke it

I'm good to go with these changes once CI is green :)

#38035 broke it

🤦♂ sorry about that.

@JavierMatosD CI is green. Ready to merge and fixing the baseline issues.

thanks a lot for your work, when do you think we can merge this one? I am quite depended on it ?