add draw_net

Added feature for drawing neural network architecture (referred this code). Test implementation here in block 10 (draw_to_notebook function)
PS. This is my first contribution, so please feel free to correct my noob mistakes (if any :wink: )
To do:

[ ] How to do a sucessful travis check
[ ] complementary coloring

Hi, this looks pretty cool, we should definitely have something like this. I will have a closer look at the code but here are some questions that popped up:

Could you please add tests, simple ones like the others for visualize.py?
Don't we need pydot as a new dependency?
Does your code produce good results when the network architecture is not linear (e.g. multiple inputs)?
I see that you pick a random color for other layer types via hashing. I'm afraid that this may lead to bad colors (like black, so that the font would be illegible). My proposal would be to choose like 20 good looking colors and map onto those (there are online tools that help you choose nice, complementing colors).

Anyway, I believe this would be a welcome addition.

I would add the tests but the issue is that unlike other functions in visualize (which return a plotted frame) the functions draw_to_file returns a file with specified file format (eg .pdf) and draw_to_notebook returns an IPython Image. So how should I proceed? Also if possible, could you give links to a resources for python testing? (basics and stuff)

Yes this needs pydot as a dependency. It also requires Graphviz for its internal working.

This gives a good results for non-linear graph too (see block 19)

Got it. will work on that.

I have some doubts

Above there's "Travis CI build failed". What does that mean?
In this PR, the nonlinearity layer is drawn as a completely new layer. I particularly liked this implementation (see page 2 model of the paper) where nonlinearity layer is a sub-layer and is smaller. This appears more cleaner and better. But its out of my horizon of coding :disappointed:

One more thing, I'll probably be slow (give or take an hour a day), so please bear with me.
Thanks for hearing me out!

Hi,

Visualizations are not that easy to test; for your functions, it would be sufficient if you show that they work with different settings (i.e. don't raise errors). You could use some of the pre-defined nets, as is done here.

Try to just add pydot to the requirements.txt, this should already fix the travis errors. (Travis is a tool that automatically creates an environment and tests your package.)

This looks good.

Great.

I believe you pasted the wrong link with regard to the nonlinearity layer. Anyway, I would not worry too much about it, there is no harm in fixing this at a later point in time.
Take your time. If you get stuck, upload your code and we may be able to help you out.

For 2.: Should also add it to setup.py.
Also interested to hear about: "Does your code produce good results when the network architecture is not linear (e.g. multiple inputs)?"

Sorry for that... i've edited the link. Will reply for the rest as soon as
I can.

Added tests today, still have to work on them but.
Is there any way to do visualize.py testing locally (what I found that you have to run "$ py.test" but it runs all the tests, instead I only want to run the tests which I created)
There's one problem, there's a python-3 compatibility error. I am still working on that..will reply on how to solve this.

You can run
py.test -k test_visualize
to only run tests within that test module. -k also works with test
functions or classes.

I see that for the tests to pass, we would, in addition to adding graphvis, also need to add IPython. I am a little bit concerned that nolearn might have too many dependencies. @dnouri, what about the possibility to make the addition of the visualization tools optional? They have many dependencies now (and matplotlib/pydot sometimes cause trouble) and many people might not care for visualizations.

I agree that draw_to_notebook is a bit of an overkill, but you can see that it does nothing special. It's just that people (who like me depend on ipython notebook for literally everything) could gain from it.
But I must ask you to keep the visualizations. They are very helpful and intuitive. Moreover the other good libraries (eg caffe, tensorflow) have their own vis tools. It would be great if nolearn could have this. What we could do is refer the implementations of these libraries and incorporate them here.
I am sorry if there are mishaps on my end. I don't have the expertise or resources to do it as optimally as I can. If you could just give me this chance, I swear I would not let you down.

I am sorry if there are mishaps on my end.

Don't worry.  Nothing bad happened.  Happy to have your contributions.

What about the possibility to make the addition of the visualization tools optional?

Yes, let's make these visualization dependencies an extra.  This way people can install nolearn[viz] if they want visualization dependencies.  I'll try to work on this tomorrow.  For now, please feel free to add all the requirements necessary for this PR into the normal install_requires to see if all tests can pass this way.  (Or fine if you can tell me they work locally.)

Tests run sucessfully on local machine

log.txt
network.pdf.zip

#TODO

How to do a sucessful travis check
complementary coloring

@faizankshaikh Would you please rebase this on top of master?  This way you'll get the visualization extras_require, which will allow you to solve your IPython dependency issue.  Also some pep8 checking.

I get this error now,

faizy@faizy-PC ~/workspace/nolearn_contri $ py.test -k test_visualize
usage: py.test [options] [file_or_dir] [file_or_dir] [...]
py.test: error: unrecognized arguments: --flakes --pep8

Sorry about that. Test dependencies have changed. Can you do
python setup.py dev
This should install pytest-pep8 and pytest-flakes. Once you're able to run
tests again, you'll notice that there's a few pep8 errors. Which is why I
added pytest-pep8. Thanks!

Thanks for the reply. I noticed the dependency error and installed them
manually. That is why I deleted the comment, in hopes that it won't disturb
you (guess I was wrong) And yes, there were pep8 errors...cleaned them.
Thanks again.
On 31-Mar-2016 6:53 pm, "Daniel Nouri"<EMAIL_ADDRESS>wrote:

Sorry about that. Test dependencies have changed. Can you do
python setup.py dev
This should install pytest-pep8 and pytest-flakes. Once you're able to run
tests again, you'll notice that there's a few pep8 errors. Which is why I
added pytest-pep8. Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/dnouri/nolearn/pull/231#issuecomment-203934142

Try adding graphviz to the list of packages to install on the Travis box.

YES!!!

Now have to work on the original problem :smile:

Any thoughts on the following would be welcome!
I originally thought (and as @BenjaminBossan suggested) of hardcoding block color and font color. I would suggest that instead of if-else ladder, we could have a data structure (dictionary maybe?) with some complementary colors (I found this site helpful) with (random) runtime allocation. Any tips on this?

Let's add a very prominent comment saying where we lifted the code from.  (Which you did mention in your original comment, but it's not mentioned in the code itself yet.)
I think I'd also be fine to retain the original get_hex_color implementation for now.  Unless @BenjaminBossan has a better idea.

Let me see if I find a better solution for the colors this weekend.

Today I checked with pylint manually and corrected (many!) errors.

I have a proposal for the color picker. I have chosen 4 types of layers (conv, pool, recurrent, other) with each 5 similar colors. Here is the code:
COLORS = ['#4A88B3', '#98C1DE', '#6CA2C8', '#3173A2', '#17649B',
          '#FFBB60', '#FFDAA9', '#FFC981', '#FCAC41', '#F29416',
          '#C54AAA', '#E698D4', '#D56CBE', '#B72F99', '#B0108D',
          '#75DF54', '#B3F1A0', '#91E875', '#5DD637', '#3FCD12']

def get_hex_color(layer_type):
    black = '#000000'
    hashed = int(hash(layer_type)) % 5

    if "conv" in layer_type.lower():
        return COLORS[:5][hashed], black
    if layer_type in lasagne.layers.pool.__all__:
        return COLORS[5:10][hashed], black
    if layer_type in lasagne.layers.recurrent.__all__:
        return COLORS[10:15][hashed], black
    else:
        return COLORS[15:20][hashed], black

On my display at least, the colors look fine. There are some colors that are never used with the current layers ('#3173A2', '#4A88B3', '#98C1DE', '#B72F99', '#C54AAA', '#D56CBE') but that may of course change with the addition of more layers. Maybe you have an idea how to hash onto the colors more evenly.
I also have a proposal: Could you change the code so that make_pydot_graph, draw_to_file, and draw_to_notebook would also accept a NeuralNet instance as first argument? The reason is that none of the other functions accept a list of layers as argument; I could imagine that many nolearn users would find passing the net as argument to be more intuitive.

@BenjaminBossan I like the solution to the color picking.  Two things though:

Can we drop though the second argument we're returning (the black in all cases); that's not used in the original implementation.
Maybe make COLORS a 2d list, this way we can get rid of the slicing.

Could you change the code so that make_pydot_graph, draw_to_file, and draw_to_notebook would also accept a NeuralNet instance as first argument?

Agree that would be useful.
@BenjaminBossan Can we following the merge of this update your notebook to include the rendered network in one of the examples?

@dnouri : The second argument is necessary to change the font color (not in the original implementation, I tweaked it a bit)

The second argument is necessary to change the font color (not in the original implementation, I tweaked it a bit)

OK, do we need it still though?  Seems like font color could always be black?

As per @BenjaminBossan above,

I'm afraid that this may lead to bad colors (like black, so that the font would be illegible)

was a valid point

I agree that we should drop the second color, makes it simpler. With the current colors, black should work. If we ever need other font colors, we could add a similar function that turns layer names to font colors.

Maybe make COLORS a 2d list, this way we can get rid of the slicing.

I thought about a dictionary but then if we ever need the colors for something else, a list might be easier and slicing does not add too much overhead -- but I don't care.

Can we following the merge of this update your notebook to include the rendered network in one of the examples?

I will update it soon, some other parts are not up-to-date either.

Alright then. I'll cook the PR by tomorrow. Thanks for your support @BenjaminBossan @dnouri

Done!

Is there anything left for me to do?

Be patient, I'm sure Daniel will have another look at it and merge it when he believes it's ready.

Thanks @faizankshaikh


Search for gene name is not returning expected results

Searching for a gene name, e.g. KIN10, does currently not return KIN10 as the top scoring gene. The reason seems to be that the knowledge network for KIN10 is huge with over 20k concepts linked to it. As the score of KIN10 is determined by concepts containing KIN10 (5) divide by the total size of the KIN10 network (20k), the score becomes quite small. Other non KIN10 genes receive a much higher score because their knowledge networks are smaller and they interact with KIN10.
There is a work around for this by adding KIN10 to the gene list and sorting the Gene View by column USER, see here:

It would be good to investigate the changes required to the gene scoring algorithm to bump the score in such cases without having any other side-effects.

The gene rank algorithm now also considers the distance to the evidence which might help to rank this cases higher.


Assets are poorly organised

Use Flask-Assets

Also consider moving to SASS for easier maintenance.

Fixing with the new theme.


Add clean-commit-form feature

Basically clean-conversations but for commit forms.
Test URLs
https://github.com/refined-github/refined-github/edit/hotfix/README.md
Screenshot

Before
After

I am not able to find this feature in the Features list in the plugin settings:

Is this an always on feature?

Is this an always on feature?

Yes, it's CSS only

If you want the avatar back, add this to the custom CSS (not restoring the width because it doesn't hurt):
.commit-form-avatar {
    display: block;
}


[BUG]: layer norm error

🐛 Describe the bug
使用huggingface的模型接口，用gpt2的gemini demo跑glm-chinese-10b，修改了tensor_parallel函数和glm的modeling_glm.py来适配colossal，发现layer norm会报RuntimeError: Expected weight to be of same shape as normalized_shape, but got weight of shape [2048] and normalized_shape = [4096]
运行脚本如下：
set -x
# distplan in ["CAI_ZeRO1", "CAI_ZeRO2", "CAI_Gemini", "Pytorch_DDP", "Pytorch_ZeRO"]
export DISTPLAN=${DISTPLAN:-"CAI_Gemini"}

# The following options only valid when DISTPLAN="colossalai"
export GPUNUM=${GPUNUM:-4}
export TPDEGREE=${TPDEGREE:-2}
export PLACEMENT=${PLACEMENT:-"cpu"}
export USE_SHARD_INIT=${USE_SHARD_INIT:-False}
export BATCH_SIZE=${BATCH_SIZE:-8}
export MODEL_TYPE=${MODEL_TYPE:-"gpt2_medium"}
export TRAIN_STEP=${TRAIN_STEP:-10}
# export PYTHONPATH=$PWD:$PYTHONPATH

if [ ${USE_SHARD_INIT} = "True" ]; then
  USE_SHARD_INIT="--shardinit"
else
  USE_SHARD_INIT=""
fi

mkdir -p gemini_logs

torchrun --standalone --nproc_per_node=${GPUNUM} ./train_glm_demo.py \
--tp_degree=${TPDEGREE} \
--model_type=${MODEL_TYPE} \
--batch_size=${BATCH_SIZE} \
--placement=${PLACEMENT} \
${USE_SHARD_INIT} \
--distplan=${DISTPLAN} \
--train_step=${TRAIN_STEP} \
2>&1 | tee ./gemini_logs/${MODEL_TYPE}_${DISTPLAN}_gpu_${GPUNUM}_bs_${BATCH_SIZE}_tp_${TPDEGREE}_${PLACEMENT}.log

traceback如下：

+ export DISTPLAN=CAI_Gemini
+ DISTPLAN=CAI_Gemini
+ export GPUNUM=4
+ GPUNUM=4
+ export TPDEGREE=2
+ TPDEGREE=2
+ export PLACEMENT=cpu
+ PLACEMENT=cpu
+ export USE_SHARD_INIT=False
+ USE_SHARD_INIT=False
+ export BATCH_SIZE=8
+ BATCH_SIZE=8
+ export MODEL_TYPE=gpt2_medium
+ MODEL_TYPE=gpt2_medium
+ export TRAIN_STEP=10
+ TRAIN_STEP=10
+ '[' False = True ']'
+ USE_SHARD_INIT=
+ mkdir -p gemini_logs
+ tee ./gemini_logs/gpt2_medium_CAI_Gemini_gpu_4_bs_8_tp_2_cpu.log
+ torchrun --standalone --nproc_per_node=4 ./train_glm_demo.py --tp_degree=2 --model_type=gpt2_medium --batch_size=8 --placement=cpu --distplan=CAI_Gemini --train_step=10
WARNING:torch.distributed.run:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
environmental variable OMP_NUM_THREADS is set to 20.
environmental variable OMP_NUM_THREADS is set to 20.
environmental variable OMP_NUM_THREADS is set to 20.
environmental variable OMP_NUM_THREADS is set to 20.
[03/01/23 10:39:28] INFO     colossalai - colossalai - INFO:
                             /home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/colossalai/context/parallel_context.py:5
                             21 set_device
[03/01/23 10:39:28] INFO     colossalai - colossalai - INFO:
                             /home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/colossalai/context/parallel_context.py:5
                             21 set_device
                    INFO     colossalai - colossalai - INFO: process rank 3 is bound to device 3
                    INFO     colossalai - colossalai - INFO: process rank 0 is bound to device 0
[03/01/23 10:39:28] INFO     colossalai - colossalai - INFO:
                             /home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/colossalai/context/parallel_context.py:5
                             21 set_device
                    INFO     colossalai - colossalai - INFO: process rank 2 is bound to device 2
[03/01/23 10:39:28] INFO     colossalai - colossalai - INFO:
                             /home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/colossalai/context/parallel_context.py:5
                             21 set_device
                    INFO     colossalai - colossalai - INFO: process rank 1 is bound to device 1
[03/01/23 10:39:31] INFO     colossalai - colossalai - INFO:
                             /home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/colossalai/context/parallel_context.py:5
                             57 set_seed
                    INFO     colossalai - colossalai - INFO: initialized seed on rank 2, numpy: 1024, python random: 1024,
                             ParallelMode.DATA: 1024, ParallelMode.TENSOR: 1024,the default parallel seed is ParallelMode.DATA.
[03/01/23 10:39:31] INFO     colossalai - colossalai - INFO:
                             /home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/colossalai/context/parallel_context.py:5
                             57 set_seed
[03/01/23 10:39:31] INFO     colossalai - colossalai - INFO:
                             /home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/colossalai/context/parallel_context.py:5
                             57 set_seed
                    INFO     colossalai - colossalai - INFO: initialized seed on rank 0, numpy: 1024, python random: 1024,
                             ParallelMode.DATA: 1024, ParallelMode.TENSOR: 1024,the default parallel seed is ParallelMode.DATA.
                    INFO     colossalai - colossalai - INFO: initialized seed on rank 3, numpy: 1024, python random: 1024,
                             ParallelMode.DATA: 1024, ParallelMode.TENSOR: 1024,the default parallel seed is ParallelMode.DATA.
                    INFO     colossalai - colossalai - INFO:
                             /home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/colossalai/initialize.py:116 launch
                    INFO     colossalai - colossalai - INFO: Distributed environment is initialized, data parallel size: 4, pipeline
                             parallel size: 1, tensor parallel size: 1
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
                    INFO     colossalai - colossalai - INFO: /home/colossal-example/glm/./train_glm_demo.py:221 main
                    INFO     colossalai - colossalai - INFO: gpt2_medium, CAI_Gemini, batch size 8
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
[03/01/23 10:39:31] INFO     colossalai - colossalai - INFO:
                             /home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/colossalai/context/parallel_context.py:5
                             57 set_seed
                    INFO     colossalai - colossalai - INFO: initialized seed on rank 1, numpy: 1024, python random: 1024,
                             ParallelMode.DATA: 1024, ParallelMode.TENSOR: 1024,the default parallel seed is ParallelMode.DATA.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
************************************************** load model finish **************************************************
************************************************** start ProcessGroup **************************************************
************************************************** load model finish **************************************************
************************************************** start ProcessGroup **************************************************
************************************************** load model finish **************************************************
************************************************** start ProcessGroup **************************************************
************************************************** load model finish **************************************************
************************************************** start ProcessGroup **************************************************
************************************************** start tensor parallel **************************************************
************************************************** start tensor parallel **************************************************
************************************************** start tensor parallel **************************************************
************************************************** start tensor parallel **************************************************
************************************************** build adam **************************************************
************************************************** build adam **************************************************
************************************************** build adam **************************************************
************************************************** build adam **************************************************
=========================================================================================
No pre-built kernel is found, build and load the cpu_adam kernel during runtime now
=========================================================================================
Emitting ninja build file /home/.cache/colossalai/torch_extensions/torch1.12_cu11.3/build.ninja...
Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module cpu_adam...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/.cache/colossalai/torch_extensions/torch1.12_cu11.3/build.ninja...
Building extension module fused_optim...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module fused_optim...
************************************************** zero model process **************************************************
Loading extension module cpu_adam...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/.cache/colossalai/torch_extensions/torch1.12_cu11.3/build.ninja...
Building extension module fused_optim...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module fused_optim...
************************************************** zero model process **************************************************
Loading extension module cpu_adam...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/.cache/colossalai/torch_extensions/torch1.12_cu11.3/build.ninja...
Building extension module fused_optim...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module fused_optim...
************************************************** zero model process **************************************************
Loading extension module cpu_adam...
Time to load cpu_adam op: 2.284288167953491 seconds
=========================================================================================
No pre-built kernel is found, build and load the fused_optim kernel during runtime now
=========================================================================================
Detected CUDA files, patching ldflags
Emitting ninja build file /home/.cache/colossalai/torch_extensions/torch1.12_cu11.3/build.ninja...
Building extension module fused_optim...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module fused_optim...
Time to load fused_optim op: 0.39287424087524414 seconds
************************************************** zero model process **************************************************
searching chunk configuration is completed in 3.11 s.
used number: 4711.35 MB, wasted number: 28.66 MB
total wasted percentage is 0.60%
************************************************** zero optim process **************************************************
************************************************** load pack parallel **************************************************
[03/01/23 10:41:16] INFO     colossalai - colossalai - INFO: /home/colossal-example/glm/./train_glm_demo.py:307 main
                    INFO     colossalai - colossalai - INFO: the size of testing model size is 4.9B.
load tokenizer
************************************************** zero optim process **************************************************
************************************************** load pack parallel **************************************************
[03/01/23 10:41:17] INFO     colossalai - colossalai - INFO: /home/colossal-example/glm/./train_glm_demo.py:291 main
                    INFO     colossalai - colossalai - INFO: After init optim, GPU memory usage: 0.00 MB, CPU memory usage: 29621.83 MB
************************************************** zero optim process **************************************************
                    INFO     colossalai - colossalai - INFO: /home/colossal-example/glm/./train_glm_demo.py:307 main
                    INFO     colossalai - colossalai - INFO: the size of testing model size is 4.9B.
                    INFO     colossalai - colossalai - INFO: /home/colossal-example/glm/./train_glm_demo.py:308 main
                    INFO     colossalai - colossalai - INFO: After init model, GPU memory usage: 0.00 MB, CPU memory usage: 29621.83 MB
load tokenizer
************************************************** zero optim process **************************************************
************************************************** load pack parallel **************************************************
[03/01/23 10:41:17] INFO     colossalai - colossalai - INFO: /home/colossal-example/glm/./train_glm_demo.py:307 main
                    INFO     colossalai - colossalai - INFO: the size of testing model size is 4.9B.
load tokenizer
************************************************** load pack parallel **************************************************
[03/01/23 10:41:17] INFO     colossalai - colossalai - INFO: /home/colossal-example/glm/./train_glm_demo.py:307 main
                    INFO     colossalai - colossalai - INFO: the size of testing model size is 4.9B.
load tokenizer
Traceback (most recent call last):
  File "/home/colossal-example/glm/./train_glm_demo.py", line 427, in <module>
    main()
  File "/home/colossal-example/glm/./train_glm_demo.py", line 379, in main
    train_step(batch)
  File "/home/colossal-example/glm/./train_glm_demo.py", line 331, in train_step
    outputs = model(**inputs)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
Traceback (most recent call last):
  File "/home/colossal-example/glm/./train_glm_demo.py", line 427, in <module>
Traceback (most recent call last):
  File "/home/colossal-example/glm/./train_glm_demo.py", line 427, in <module>
Traceback (most recent call last):
  File "/home/colossal-example/glm/./train_glm_demo.py", line 427, in <module>
    main()
  File "/home/colossal-example/glm/./train_glm_demo.py", line 379, in main
    main()
      File "/home/colossal-example/glm/./train_glm_demo.py", line 379, in main
main()
  File "/home/colossal-example/glm/./train_glm_demo.py", line 379, in main
    train_step(batch)
  File "/home/colossal-example/glm/./train_glm_demo.py", line 331, in train_step
    train_step(batch)
  File "/home/colossal-example/glm/./train_glm_demo.py", line 331, in train_step
    outputs = model(**inputs)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    outputs = model(**inputs)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    train_step(batch)
  File "/home/colossal-example/glm/./train_glm_demo.py", line 331, in train_step
    outputs = model(**inputs)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/colossalai/nn/parallel/data_parallel.py", line 282, in forward
    return forward_call(*input, **kwargs)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/colossalai/nn/parallel/data_parallel.py", line 282, in forward
    return forward_call(*input, **kwargs)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/colossalai/nn/parallel/data_parallel.py", line 282, in forward
    return forward_call(*input, **kwargs)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/colossalai/nn/parallel/data_parallel.py", line 282, in forward
    outputs = self.module(*args, **kwargs)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    outputs = self.module(*args, **kwargs)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/colossal-example/glm/modeling_glm.py", line 904, in forward
    model_output = self.glm.forward(input_ids, position_ids, attention_mask, mems=mems, **kwargs)
  File "/home/colossal-example/glm/modeling_glm.py", line 793, in forward
    return forward_call(*input, **kwargs)
  File "/home/colossal-example/glm/modeling_glm.py", line 904, in forward
    transformer_output = self.transformer(embeddings, position_ids, attention_mask, mems)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/colossal-example/glm/modeling_glm.py", line 605, in forward
    model_output = self.glm.forward(input_ids, position_ids, attention_mask, mems=mems, **kwargs)
    hidden_states = layer(*args, mem=mem_i)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
  File "/home/colossal-example/glm/modeling_glm.py", line 793, in forward
    return forward_call(*input, **kwargs)
  File "/home/colossal-example/glm/modeling_glm.py", line 425, in forward
    layernorm_output = self.input_layernorm(hidden_states)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    transformer_output = self.transformer(embeddings, position_ids, attention_mask, mems)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/modules/normalization.py", line 189, in forward
    return forward_call(*input, **kwargs)
  File "/home/colossal-example/glm/modeling_glm.py", line 605, in forward
    hidden_states = layer(*args, mem=mem_i)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/colossal-example/glm/modeling_glm.py", line 425, in forward
    layernorm_output = self.input_layernorm(hidden_states)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/modules/normalization.py", line 189, in forward
    outputs = self.module(*args, **kwargs)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/colossal-example/glm/modeling_glm.py", line 904, in forward
    model_output = self.glm.forward(input_ids, position_ids, attention_mask, mems=mems, **kwargs)
  File "/home/colossal-example/glm/modeling_glm.py", line 793, in forward
    transformer_output = self.transformer(embeddings, position_ids, attention_mask, mems)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/colossal-example/glm/modeling_glm.py", line 605, in forward
    hidden_states = layer(*args, mem=mem_i)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    outputs = self.module(*args, **kwargs)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/colossal-example/glm/modeling_glm.py", line 425, in forward
    layernorm_output = self.input_layernorm(hidden_states)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/colossal-example/glm/modeling_glm.py", line 904, in forward
    return forward_call(*input, **kwargs)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/modules/normalization.py", line 189, in forward
    model_output = self.glm.forward(input_ids, position_ids, attention_mask, mems=mems, **kwargs)
  File "/home/colossal-example/glm/modeling_glm.py", line 793, in forward
    transformer_output = self.transformer(embeddings, position_ids, attention_mask, mems)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/colossal-example/glm/modeling_glm.py", line 605, in forward
    hidden_states = layer(*args, mem=mem_i)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/colossal-example/glm/modeling_glm.py", line 425, in forward
    layernorm_output = self.input_layernorm(hidden_states)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/modules/normalization.py", line 189, in forward
    return F.layer_norm(
return F.layer_norm(
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/functional.py", line 2500, in layer_norm
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/functional.py", line 2500, in layer_norm
    return F.layer_norm(
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/functional.py", line 2500, in layer_norm
    return F.layer_norm(
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/functional.py", line 2500, in layer_norm
        return handle_torch_function(
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/overrides.py", line 1498, in handle_torch_function
return handle_torch_function(
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/overrides.py", line 1498, in handle_torch_function
    return handle_torch_function(
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/overrides.py", line 1498, in handle_torch_function
    return handle_torch_function(
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/overrides.py", line 1498, in handle_torch_function
    result = torch_func_method(public_api, types, args, kwargs)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/colossalai/tensor/colo_parameter.py", line 87, in __torch_function__
    result = torch_func_method(public_api, types, args, kwargs)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/colossalai/tensor/colo_parameter.py", line 87, in __torch_function__
    result = torch_func_method(public_api, types, args, kwargs)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/colossalai/tensor/colo_parameter.py", line 87, in __torch_function__
    ret = super().__torch_function__(func, types, args, kwargs)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/colossalai/tensor/colo_tensor.py", line 184, in __torch_function__
    ret = func(*args, **kwargs)
          File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/colossalai/nn/_ops/layernorm.py", line 21, in colo_layernorm
ret = super().__torch_function__(func, types, args, kwargs)ret = super().__torch_function__(func, types, args, kwargs)

  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/colossalai/tensor/colo_tensor.py", line 184, in __torch_function__
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/colossalai/tensor/colo_tensor.py", line 184, in __torch_function__
        ret = func(*args, **kwargs)
ret = func(*args, **kwargs)  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/colossalai/nn/_ops/layernorm.py", line 21, in colo_layernorm

  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/colossalai/nn/_ops/layernorm.py", line 21, in colo_layernorm
    result = torch_func_method(public_api, types, args, kwargs)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/colossalai/tensor/colo_parameter.py", line 87, in __torch_function__
    ret = super().__torch_function__(func, types, args, kwargs)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/colossalai/tensor/colo_tensor.py", line 184, in __torch_function__
    ret = func(*args, **kwargs)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/colossalai/nn/_ops/layernorm.py", line 21, in colo_layernorm
    output = F.layer_norm(input_tensor, normalized_shape, weight=weight, bias=bias, eps=eps)
      File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/functional.py", line 2503, in layer_norm
output = F.layer_norm(input_tensor, normalized_shape, weight=weight, bias=bias, eps=eps)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/functional.py", line 2503, in layer_norm
    return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)
RuntimeError: Expected weight to be of same shape as normalized_shape, but got weight of shape [2048] and normalized_shape = [4096]
    return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)
RuntimeError: Expected weight to be of same shape as normalized_shape, but got weight of shape [2048] and normalized_shape = [4096]
    output = F.layer_norm(input_tensor, normalized_shape, weight=weight, bias=bias, eps=eps)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/functional.py", line 2503, in layer_norm
    return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)
RuntimeError: Expected weight to be of same shape as normalized_shape, but got weight of shape [2048] and normalized_shape = [4096]
    output = F.layer_norm(input_tensor, normalized_shape, weight=weight, bias=bias, eps=eps)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/nn/functional.py", line 2503, in layer_norm
    return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)
RuntimeError: Expected weight to be of same shape as normalized_shape, but got weight of shape [2048] and normalized_shape = [4096]
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 137265 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 137267 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 137268 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 1 (pid: 137266) of binary: /home/.conda/envs/torch-cuda113/bin/python
Traceback (most recent call last):
  File "/home/.conda/envs/torch-cuda113/bin/torchrun", line 33, in <module>
    sys.exit(load_entry_point('torch==1.12.0', 'console_scripts', 'torchrun')())
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
    return f(*args, **kwargs)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/distributed/run.py", line 761, in main
    run(args)
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/distributed/run.py", line 752, in run
    elastic_launch(
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/.conda/envs/torch-cuda113/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
./train_glm_demo.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-03-01_10:41:24
  host      : gzailab-liuzixi01-colossalai2-0
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 137266)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

Environment
conda虚拟环境，python=3.9.13，pytorch=1.12+cuda11.3，显卡Nvidia A30*4

tensor_parallelize修改如下：
def tensor_parallelize(model: torch.nn.Module, pg: ProcessGroup):
    """tensor_parallelize
    Sharding the Model Parameters.
    Args:
        model (torch.nn.Module): a torch module to be sharded
    """
    for mn, module in model.named_modules():
        if mn=='':
            continue
        for pn, param in module.named_parameters(recurse=False):
            # NOTE() a param maybe shared by two modules
            if hasattr(param, 'visited'):
                continue
            # print('*'*50, mn+'--'+pn, '*'*50)
            # if shard init, then convert param to replica and use the dp-only ProcessGroup
            param: ColoParameter = param
            param.set_dist_spec(ReplicaSpec())
            param.set_process_group(pg)

            # shard it w.r.t tp pattern
            if 'mlp.dense_4h_to_h' in mn:
                if 'weight' in pn or 'bias' in pn:
                    split_param_col_tp1d(param, pg)    # colmn slice
                    # keep the shape of the output from c_fc
                    param.compute_spec.set_output_replicate(False)
                else:
                    param.set_dist_spec(ReplicaSpec())
            elif 'mlp.dense_h_to_4h' in mn:
                if 'weight' in pn:
                    split_param_row_tp1d(param, pg)    # row slice
                else:
                    param.set_dist_spec(ReplicaSpec())
            elif 'word_embeddings' in mn:
                if 'weight' in pn:
                    split_param_row_tp1d(param, pg)    # colmn slice
            elif 'position_embeddings' in mn:
                if 'weight' in pn:
                    split_param_col_tp1d(param, pg)    # colmn slice
            elif 'query_key_value':
                split_param_row_tp1d(param, pg)    # colmn slice
            else:
                param.set_dist_spec(ReplicaSpec())
            param.visited = True

I load my model in this manner
with ColoInitContext(device=get_current_device(),
                             dtype=torch.half,
                             default_dist_spec=default_dist_spec,
                             default_pg=shard_pg):
    # model = model_builder(args.model_type)(checkpoint=True)
    model = GLMForConditionalGeneration.from_pretrained('THUDM/glm-10b-chinese', trust_remote_code=True)
    tp_pg = ProcessGroup(tp_degree=2)
    tensor_parallelize(model, tp_pg)

Hi @zixiliuUSC
Could you try `USE_SHARD_INIT: True, TP_DEGREE=1' first and provide a whole file to run your example code?

I upload a repo which is modified from gpt2 example.
https://github.com/zixiliuUSC/colosssal-glm-demo

USE_SHARD_INIT
with USE_SHARD_INIT: False, TP_DEGREE=1, the code can be run normally in two A100 80GB.

Hi @zixiliuUSC
Could you try `USE_SHARD_INIT: True, TP_DEGREE=1' first and provide a whole file to run your example code?
the program got broken when loading model. Part of traceback are below

Traceback (most recent call last):
  File "/home/liuzixi01/colossal-example/glm/./train_glm_demo.py", line 434, in <module>
    main()
  File "/home/liuzixi01/colossal-example/glm/./train_glm_demo.py", line 245, in main
    model = GLMForConditionalGeneration.from_pretrained('THUDM/glm-10b-chinese', trust_remote_code=True)#('/home/liuzixi01/.cache/huggingface/hub/models--BAAI--glm-10b-chinese/snapshots/4e46a55c50884f3df62cb3b550d1b10d4723228e/' , trust_remote_code=True)
  File "/home/liuzixi01/.conda/envs/torch-cuda113/lib/python3.9/site-packages/transformers/modeling_utils.py", line 2478, in from_pretrained
    ) = cls._load_pretrained_model(
  File "/home/liuzixi01/.conda/envs/torch-cuda113/lib/python3.9/site-packages/transformers/modeling_utils.py", line 2844, in _load_pretrained_model
    raise RuntimeError(f"Error(s) in loading state_dict for {model.__class__.__name__}:\n\t{error_msg}")
RuntimeError: Error(s) in loading state_dict for GLMForConditionalGeneration:
	size mismatch for word_embeddings.weight: copying a param with shape torch.Size([50048, 4096]) from checkpoint, the shape in current model is torch.Size([50048, 1024]).
	size mismatch for transformer.position_embeddings.weight: copying a param with shape torch.Size([1025, 4096]) from checkpoint, the shape in current model is torch.Size([1025, 1024]).
	size mismatch for transformer.block_position_embeddings.weight: copying a param with shape torch.Size([1025, 4096]) from checkpoint, the shape in current model is torch.Size([1025, 1024]).
	size mismatch for transformer.layers.0.input_layernorm.weight: copying a param with shape torch.Size([4096]) from checkpoint, the shape in current model is torch.Size([1024]).
	size mismatch for transformer.layers.0.input_layernorm.bias: copying a param with shape torch.Size([4096]) from checkpoint, the shape in current model is torch.Size([1024]).

We have updated a lot. Please check the latest code.
In addition,  you changed the code to another model, so your issue is out of the scope of the general support of the open source community. If you need customized in-depth cooperation or support, please send the details to<EMAIL_ADDRESS>This issue was closed due to inactivity. Thanks.


Ontario - create separate category for case acquisition pending

Create separate stack for pending case acquisition info and recalculate confirmed not-travelled to exclude pending data
Make bar graph horizontal and move map up
Update preview image
Performance improvements

FZHFVEUFJAOI65CH

FZHFVEUFJAOI65CH


API to get at CameraManager or Camera list

For the INDI libcamera driver, we need to get at the camera list before actually opening a camera. Also, we need to be able to Open and Close the camera on a regular basis. Unfortunately, the API that is present needs the camera manager as input.
Can you add a means for that to get the list? Currently, I cloned the code into the driver and  added a method to get a pointer to CameraManager and run the same code that you run for discovering cameras:
https://github.com/indilib/indi-3rdparty/blob/master/indi-libcamera/libcamera-apps/core/libcamera_app.hpp#L118
https://github.com/indilib/indi-3rdparty/blob/master/indi-libcamera/indi_libcamera.cpp#L595C22-L595C22
I can't reuse the one you added to LibCameraApp::GetCameraList() as it takes a std::unique_pointer argument and I can't return the one you created for the instance.
This is of course not very usable, as you'll need to recompile this each time something changes.

Hi, unfortunately it's a bit hard for me to guess exactly what API changes you need here without more context.   If you are able to create a basic PR with the changes, we would be happy to discuss this in more detail and merge the appropriate changes.

Hi, let me jump in with probable guess, anjok may confirm if my suspicions are true.
In INDI we do not know what is the camera is used as many may be used, and INDI is. made as universal driver so we need to get info what model of camera is connected and its properties before we Open the camera, and in current api we need to know Camera before we can start it and use GetCameraList.

The only way to enumerate cameras in libcamera is through the camera manager.
Again, I am a bit unclear on how you want to do things in your INDI framework - I've never used it before. If you could create a PR with some proposed API changes that would work for you we can get something suitable merged for your. The PR does not have to be functionally complete to start with.

Here you go:
https://github.com/raspberrypi/libcamera-apps/pull/528/files
I  have no idea if this is how std:unique_ptr is supposed to get used, though. Since we have an instance of LibcameraApp in the driver, we might as well add an instance method
std::vector<std::shared_ptr<libcamera::Camera>> GetCameras();

which calls the static method with it's CameraManager, but my std:: C++ skills aren't that up to speed.

The PR above is what I added to out local copy of the source files. If you'd add something like this, we don'T have to maintain that copy anymore (provided the user has whatever version of the libcamera-apps lib then contains it)

What timing, I was just going to create an Issue to request a way to query capabilities of the attached cameras.
Our Allsky software calls libcamera-still with options to set shutter speed, contrast, does it support temperature, does it support focus, etc. etc.
We have a front end where users enter those values which our program then adds to the libcamera-still command line.
The front end lists the minimum value, maximum value, and default for all the camera's capabilities that have that info.  For example, contrast goes from 0 to 15.99 with a default of 1.
We currently have that data hard-coded for every RPi or RPi-compatible camera, which is a nightmare and very prone to error.  It also means we can't support new cameras until we add that data.
What I'm requesting is some way to query all that information and have it returned in either a "txt" or "json" format, as is done with the metadata.  I can provide all the info we'd need.

@anjok https://github.com/raspberrypi/libcamera-apps/pull/528 should now be merged.
@EricClaeys there is a way to get this information (with a caveat).  The output of libcamera-hello --list-cameras -v will give you something like what you want as printed below:
pi@pi4:~/libcamera-apps/build $ libcamera-hello --list-cameras -v
Available cameras
-----------------
0 : imx477 [4056x3040] (/base/soc/i2c0mux/i2c@1/imx477@1a)
    Modes: 'SRGGB10_CSI2P' : 1332x990 [120.05 fps - (696, 528)/2664x1980 crop]
           'SRGGB12_CSI2P' : 2028x1080 [50.03 fps - (0, 440)/4056x2160 crop]
                             2028x1520 [40.01 fps - (0, 0)/4056x3040 crop]
                             4056x3040 [10.00 fps - (0, 0)/4056x3040 crop]

    Available controls for 4056x3040 SRGGB12_CSI2P mode:
    ----------------------------------------------------
    AeConstraintMode : [0..3]
    AeEnable : [false..true]
    AeExposureMode : [0..3]
    AeMeteringMode : [0..3]
    AnalogueGain : [1.000000..22.260870]
    AwbEnable : [false..true]
    AwbMode : [0..7]
    Brightness : [-1.000000..1.000000]
    ColourGains : [0.000000..32.000000]
    Contrast : [0.000000..32.000000]
    ExposureTime : [114..674191602]
    ExposureValue : [-8.000000..8.000000]
    FrameDurationLimits : [100000..694434742]
    NoiseReductionMode : [0..4]
    Saturation : [0.000000..32.000000]
    ScalerCrop : [(0, 0)/64x64..(0, 0)/4056x3040]
    Sharpness : [0.000000..16.000000]

The code that retrieves the properties can be found here:
https://github.com/raspberrypi/libcamera-apps/blob/69122be6235f1a757cd16b8ecde367981db4afbd/core/options.cpp#L222
The caveat here is that the settings above are strictly mode specific and not necessarily global for all modes.  So you might have to configure each mode and retrieve the properties individually one-by-one if you need to.

@EricClaeys if you need more help with this feel free to open a new issue.  I'll close this one down now as the original problem is resolved.


Deploy with nginx in front?

Not sure if how common practice that nodejs run alone. but there is some keen discussion on it, namely nginx:
https://groups.google.com/forum/?fromgroups=#!topic/nodejs/gB8veFbcX5g
Is there any support or we need not care if nginx is in front of nodejs?

It's always a good practice to put nginx in front of node on your servers. But it's not relative to shipit, nginx will just forward request to node.
Do you have a question relative to Shipit?

I am setting up grunt-shipit to deploy my keystonjs (on express) now.  I successfully installed grunt-shipit. but the shipit command is not found.
I am still googleing how to deploy via grunt-shipit/shipit. Thanks for your help first.

@arthurtalkgoal you should use Shipit standalone version and not grunt-shipit, it's better. But if you want to use "grunt-shipit", you must use "grunt shipit:env task" to run it.


build_assoc does not carry over the prefix

Environment

Elixir version (elixir -v): 1.5.2
Database and version (PostgreSQL 9.4, MongoDB 3.2, etc.): Postgres 10.1
Ecto version (mix deps): 2.2.6
Database adapter and version (mix deps): postgres 0.13.3
Operating system: OSX and Debian

Current behavior
build_assoc does not preserve prefix information
iex> post = Repo.get(Post, post_id, prefix: "something")
%Post{...}
iex> build_assoc(post, :comments, body: "Some comment") |> Repo.insert()
** (Postgrex.Error) ERROR 42P01 (undefined_table): relation "comments" does not exist

The error is because Ecto forgot the prefix.  It must be supplied again explicitly to Repo.insert for this to work.
Expected behavior
The prefix "something" should carry over into the resulting Comment struct from build_assoc.

I am not sure if I fully agree. If you are using prefixes/schemas for organization purposes, then User could be in prefix "accounts" and Posts would be in another prefix, and you don't want that to propagate. That's why we usually only pass prefixes along if they come from a database operation. The best is for you to call put_meta and set the prefix or pass it on insert. Thanks!

Good point.  I wasn't aware people use schemas in this way.  Thanks for the quick feedback!


Create 752-Open-the-Lock.cs

File(s) Modified:752-Open-the-Lock.cs
Language(s) Used: csharp
Submission URL: https://leetcode.com/submissions/detail/794656721/

Thanks, @agnihotriketan!


Refresh single volume when volume event emitted

This PR fixed part of issue #2245, not totally.
Here is the reason:

When Swarm starts at the first time or restarts, Swarm will fetch all volumes via function RefreshVolumes, and it uses /volumes API against docker engine's which api misses volume's label vesus /volumes/{ID}.
When volumes updated and created during Swarm's running, Swarm will get infos about them via  /volumes/{ID}, which will not miss volume's labels.

This PR did:
1.Refresh single volume when volume event emitted. It will save network package size when dealing with volume.
2.Refresh ONLY the created volume when a volume is created.
This PR is similar to #2255
Some related details:
https://github.com/docker/docker/issues/22838
Signed-off-by: allencloud<EMAIL_ADDRESS>

@allencloud I was looking at it as well. There is the exact same problem for networks by the way. I was not sure that full refresh every time is the way to go. In containers refresh for example we try to reduce that.
PS I will open the issue about networks as well.

@doronp Yes, same case.
API /networks against docker engine will miss field Options of each network, while /networks/{*name} has it.

PTAL, similar to network
@dongluochen @abronan @nishanttotla

LGTM
ping @nishanttotla @dongluochen

ping @nishanttotla @dongluochen

LGTM


how to download ;-;

i might be dumb but im kinda new to github/dont use it alot

Hi, you can click on this link to download:
https://github.com/AbsoluteWinter/more-ore-combo-autoclick/archive/refs/heads/main.zip


Adjusted ActNorm to work as described in the paper

This is a follow-up PR from the discussion sparked in #165.
This PR involves:

Initializing the ActNorm Parameters to unit size in all dimensions except the channel dimension
Drawing the mean and std over the batch dimension, as well as any dimensions beyond the channel dimension
Checking the batch for correct shape before overwriting the parameters
Adjusting the tests to reflect these changes

For free with these changes comes:

A fix for silent device moves in the initialization method
An additional check for an initialization batch that has zero standard deviation in any channel

Please do not yet merge, there appears to be an error with the log jacobian in another test.


Add the ability to specify the host of the redirect url instead of ctx.host

I'm having the same issue as this person here on the original koa-shopify-auth and so am suggesting the exact same thing: https://github.com/Shopify/koa-shopify-auth/issues/11
Overview
The developer should be able to define the host of the redirect URL to be https://{Host} instead of taking the host from context (ctx).
https://github.com/TheSecurityDev/simple-koa-shopify-auth/blob/e113d012a114b9ec3230f687b5ed51cd33a0490f/src/top-level-oauth-redirect.ts#L36
When running createShopifyAuth() there could be an extra optional parameter, host, which if set the above function will use the provided host instead of ctx.host, which can be wrong when using proxies.

For some context, this is the issue I'm facing. Somewhat confusingly it does this about 50% of the time, the other 50% the host is correct and the app works as expected.

This was taking me too long looking for a workaround trying to use koa-proxies and stuff like that. Have added a PR which solves this issue, if you could review https://github.com/TheSecurityDev/simple-koa-shopify-auth/pull/6

Also, have you tried initializing your Koa server like this:
new Koa({ proxy: true });

That's a good idea. Yeah, I am passing that same string as pulled from the .env so that should work in my case.
createShopifyAuth({
      accessMode: "online",
      authPath: "/auth",
      host: config.shopify.HOST_NAME,

Also, have you tried initializing your Koa server like this:
new Koa({ proxy: true });
I had to do this for ngrok and heroku so I could get the client's IP address correctly, so maybe it's a similar situation with getting the host header.

I have not, but we've just resubmitted the app for listing after using the forked version of this package, so just going to wait on a reply from Shopify now before making more changes.

I just published the new package v1.0.5, so I'll go ahead and close this. The new version uses Shopify.Context.HOST_NAME instead, so you don't have to pass in the host name or anything. I suspect the issue would also have been solved by enabling Koa proxy as I mentioned, but this would fix any future issues other people may have like this.


test case  "test heals after a large netsplit (NetsplitTest)" fails

I'm running tests with
ELIXIR_ERL_OPTIONS='-kernel prevent_overlapping_partitions false' mix test
fails with
     test/netsplit_test.exs:56
     Unexpectedly received message {:hello_from, #PID<22941.277.0>} (which matched {:hello_from, _})
     code: refute_receive({:hello_from, _})
     stacktrace:
       test/netsplit_test.exs:73: (test)

To me it looks like the test case is wrong.

Can you elaborate? What is the library doing and what would you expect it to do on this situation?

@derekkraan library is working fine for me.  One of the test cases if failing.  At first glance it looks like the test cases should never pass,  as it never consumes the messages from the cluster creation.


Wrong TTS url

go2rtc version: 1.9.4
webrtc integration: 3.5.2
Home assistant: 2024.8.3
Reolink firmware: v<IP_ADDRESS>08_2407315182
I am trying to send TTS to my Reolink WiFi Doorbell. I already have 2 way audio working fine.
I am running go2rtc 1.9.4 inside my frigate instance. In home assistant, I have the webrtc integration added and it points to my local go2rtc url (http://<IP_ADDRESS>:1984).
In go2rtc, my doorbell stream for sending audio is:
doorbell_send_audio:
      - rtsp://{FRIGATE_REOLINK_USERNAME}:{FRIGATE_REOLINK_PASSWORD}@{FRIGATE_DOORBELL_IP}:554/Preview_01_sub
      - ffmpeg:doorbell_send_audio#audio=opus

In my home assistant configuration yaml, I have:
media_player:
  - platform: webrtc
    name: doorbell_tts
    stream: doorbell_send_audio
    audio: pcma

Next, I use tts.speak to send tts audio to the camera. I am using the cache option so really it is sending a mp3 file from my home assistant server.
When I send TTS, I get no audio at all and see this in the go2rtc logs:
[api] POST /api/streams?dst=doorbell_send_audio&src=ffmpeg:http://<IP_ADDRESS>:8123/api/tts_proxy/b06d94ea6efe2906bfc7273c49f6ea1f4652fc40_en-us_6e8b81ac47_tts.home_assistant_cloud.mp3%23audio%3Dpcma%23input%3Dfile <IP_ADDRESS>:42466

Something is encoding the ffmpeg options at the end of the url (# -> %23, = -> %3D) which seems to be the problem.
If I manually try to POST to this url using curl:
curl --verbose --request POST --url 'http://<IP_ADDRESS>:1984/api/ffmpeg?dst=doorbell_send_audio&file=http://<IP_ADDRESS>:8123/api/tts_proxy/b06d94ea6efe2906bfc7273c49f6ea1f4652fc40_en-us_6e8b81ac47_tts.home_assistant_cloud.mp3%23audio%3Dpcma%23input%3Dfile'

I get the error ffmpeg: unsupported params: #audio=pcma#input=file#audio=auto#input=file
If I change the %23 after mp3 to %2523, it works and I can hear the TTS. (the audio is choppy and cuts out, but atleast I can hear it)
curl --verbose --request POST --url 'http://<IP_ADDRESS>:1984/api/ffmpeg?dst=doorbell_send_audio&file=http://<IP_ADDRESS>:8123/api/tts_proxy/b06d94ea6efe2906bfc7273c49f6ea1f4652fc40_en-us_6e8b81ac47_tts.home_assistant_cloud.mp3%2523audio%3Dpcma%23input%3Dfile'

Or, if I remove the encoding entirely and just put the normal characters back, it also works.
curl --verbose --request POST --url 'http://<IP_ADDRESS>:1984/api/ffmpeg?dst=doorbell_send_audio&file=http://<IP_ADDRESS>:8123/api/tts_proxy/b06d94ea6efe2906bfc7273c49f6ea1f4652fc40_en-us_6e8b81ac47_tts.home_assistant_cloud.mp3#audio=pcma#input=file'

What is causing the ffmpeg parameters to be encoded? go2rtc or webrtc?
Why does changing %23 to %2523 before the ffmpeg parameters make it work?
Not sure if this is a webrtc issue or go2rtc issue (or something else?)

Your IP different - <IP_ADDRESS> and <IP_ADDRESS>
Your API different - /api/streams and /api/ffmpeg. FFmpeg API takes file link as input. Streams API takes stream source as input.

<IP_ADDRESS>:1984 is my go2rtc instance. All /api/ requests go to this address.
<IP_ADDRESS>:8123 is my home assistant. This is where the mp3 file is.
Sorry, I never realized I was mixing up /api/streams and /api/ffmpeg. My mistake.
When curling the /api/streams, I was getting "no consumer found". I think because the codec didn't match. From go2rtc, the codec that shows up as a consumer is "audio, sendonly, PCMU/8000"
In home assistant configuration.yaml, I changed pcma to pcmu.
media_player:
  - platform: webrtc
    name: doorbell_tts
    stream: doorbell_send_audio
    audio: pcmu

Now I am getting audio out using the doorbell_tts.
And it works using manual curl:
curl --verbose --request POST --url "http://<IP_ADDRESS>:1984/api/streams?dst=doorbell_send_audio&src=ffmpeg:http://<IP_ADDRESS>:8123/api/tts_proxy/94aab1c7563c12d0fbba2d2f4d76e16b1caada9e_en-ca_f579540522_tts.home_assistant_cloud.mp3%23audio%3Dpcmu%23input%3Dfile"

It is still choppy/cuts out but that is a different issue.

What about /api/ffmpes? Is it works better with right codec?

I tried the follwing /api/ffmpeg commands:
No extra parameters:
curl --verbose --request POST --url "http://<IP_ADDRESS>:1984/api/ffmpeg?dst=doorbell_send_audio&file=http://<IP_ADDRESS>:8123/api/tts_proxy/b06d94ea6efe2906bfc7273c49f6ea1f4652fc40_en-us_6e8b81ac47_tts.home_assistant_cloud.mp3"

With pcmu parameter:
curl --verbose --request POST --url "http://<IP_ADDRESS>:1984/api/ffmpeg?dst=doorbell_send_audio&file=http://<IP_ADDRESS>:8123/api/tts_proxy/b06d94ea6efe2906bfc7273c49f6ea1f4652fc40_en-us_6e8b81ac47_tts.home_assistant_cloud.mp3#audio=pcmu"

With pcmu and file parameter:
curl --verbose --request POST --url "http://<IP_ADDRESS>:1984/api/ffmpeg?dst=doorbell_send_audio&file=http://<IP_ADDRESS>:8123/api/tts_proxy/b06d94ea6efe2906bfc7273c49f6ea1f4652fc40_en-us_6e8b81ac47_tts.home_assistant_cloud.mp3#audio=pcmu#input=file"

All of these options work.
However, encoding the parameters does NOT work:
curl --verbose --request POST --url "http://<IP_ADDRESS>:1984/api/ffmpeg?dst=doorbell_send_audio&file=http://<IP_ADDRESS>:8123/api/tts_proxy/b06d94ea6efe2906bfc7273c49f6ea1f4652fc40_en-us_6e8b81ac47_tts.home_assistant_cloud.mp3%23audio%3Dpcmu%23input%3Dfile"

ffmpeg: unsupported params: #audio=pcmu#input=file#audio=auto#input=file

/api/ffmpeg uses autoselect for codec. And file flag you already set via file query param.
What about quality?

the audio is choppy and cuts out

I did some more expirementing. The /api/ffmpeg is working very well with the following configuration:
doorbell:
  - ffmpeg:http://{FRIGATE_DOORBELL_IP}/flv?port=1935&app=bcs&stream=channel0_main.bcs&user={FRIGATE_REOLINK_USERNAME}&password={FRIGATE_REOLINK_PASSWORD}#video=copy#audio=copy#audio=opus
  - rtsp://{FRIGATE_REOLINK_USERNAME}:{FRIGATE_REOLINK_PASSWORD}@{FRIGATE_DOORBELL_IP}:554/Preview_01_sub

doorbell is the main stream and RTSP backchannel. I removed doorbell_send_audio because it didn't seem to provide any benefit.
With this setup, I run the following:
curl --verbose --request POST --url "http://<IP_ADDRESS>:1984/api/ffmpeg?dst=doorbell&file=http://<IP_ADDRESS>:8123/api/tts_proxy/b06d94ea6efe2906bfc7273c49f6ea1f4652fc40_en-us_6e8b81ac47_tts.home_assistant_cloud.mp3"

And the audio sounds perfect, there is no choppiness or cut outs.
HOWEVER, I need to make sure that I do not have another process using the backchannel. If I open my frigate card in home assistant and connect the microphone, then try to use the /api/ffmpeg again, the audio becomes choppy and cuts out again.
After I close the frigate card and disconnect microphone access, then /api/ffmpeg gives clear audio again. I guess the audio streams are interfering. I assume reolink only has single backchannel.
Is it possible to resolve this interference some way? Pipe the /api/ffmpeg to an exisiting audio channel? I don't know how difficult this would be. Or maybe there is a better way to set up my configuration.
Thanks for your help with this.

It's very difficult to fix this problem without device


Port binary packages

All binary packages in the official KISS rrpos should be ported over to GKISS.

firefox-bin has been successfully ported.

firefox-esr-bin has been successfully ported.


feat: optimizeDeps.disabled

Description
See https://github.com/vitest-dev/vitest/issues/485#issuecomment-1088664771 for context.
Vitest is currently trying to disable deps optimization by using optimizeDeps: { entries: [] }, but this only avoids scanning.
I choose optimizeDeps: { disabled: true } to avoid a breaking change, but if we want to switch to optimizeDeps: OptimizeDepsOptions | false in ResolvedConfig we could do that instead and push this in Vite 3.0.

What is the purpose of this pull request?

[ ] Bug fix
[x] New Feature
[ ] Documentation update
[ ] Other

+1 for optimizeDeps: OptimizeDepsOptions | false. I don't think it would be a breaking change if we go with this though 🤔

@bluwy we are going to get TS errors like:
(!) Plugin typescript: @rollup/plugin-typescript TS2339: Property 'exclude' does not exist on type 'false | Omit<DepOptimizationOptions, "keepNames">'.
  Property 'exclude' does not exist on type 'false'.
src/node/optimizer/esbuildDepPlugin.ts: (110:55)

110           if (moduleListContains(config.optimizeDeps?.exclude, id)) {

And it actually makes it harder to use. I also like optimizeDeps: false better, maybe it should be false in the User Config, and then undefined in that case in the Resolved config?

Good point, it does make it harder to use internally and externally. What about optimizeDeps: OptimizeDepsOptions | null instead? 🤔 I see build.watch uses this signature too.

I think we could introduce both. optimizeDeps: false for user config, and resolve it to optimizeDeps.disabled: true for ResolvedConfig to consume.

Is it safe for type mismatches between the user config and resolved config? I'm also not sure if mergeConfig handles that well. This may also mean the internal code that uses ResolvedConfig have to check .disabled before using .include (for example), instead of ?.include directly.

Yeah perhaps that is fine now thinking more about it. It feels weird initially as we're deriving a different resolved config shape based on the user config, but there's resolvedConfig.isProduction too so we already have a pattern like this. We could try out Anthony's method then, and maybe the pattern sticks too.

And I think the user could also access include, exclude, and other things independently of the optimizer being enabled or disabled, no? So no need to check for every access

That's true too. I thought there wouldn't be a reason to access include/exclude if optimizeDeps is turned off, but maybe there are usecase out there where this is needed.

Updated, and marked as experimental. @antfu I think we could merge this feature as experimental and undocumented as Vitest is going to be the only consumer until Vite 3.0. And we can discuss in a team meeting before the v3 release to accept or modify this API. What do you think?

This is causing issues in Marko: https://github.com/vitejs/vite-ecosystem-ci/runs/6001834896?check_suite_focus=true
I'm going to revert the OptimizeDepsOptions | false, and only leave the new experimental disabled


Tern: New rules for dataset and update endpoints

Hello all,
This PR contians two main changes:

Update all tern dataset redirections to use Virtuoso sparql_endpoint
Added a new set of rules for a new dataset called NSW-FMIP

Related to the new dataset, I have also applied to the creation of the PID, but I am nor sure about the status of that request:
https://catalogue.linked.data.gov.au/resource/330
Thank you very much for your help.
Best regards,
Javier

Diverted into new local branch.
Go to #69 for the continuation.


Feature/show cursor

Add ability to manually hide or unhide the cursor of the inputfield. Allows to, for example, show cursor on focus.

I have made a slightly different solution for what you proposed so I will be closing this pull request. Please see; https://github.com/rdkcentral/Lightning-ui/blob/main/src/InputField.js
Thanks again for your contributions, I will keep a better lookout at the Pull Requests in the future!

Hi,
One issue with the current implementation is that using the space on line 72 of inputfield (link) to have the cursor position correctly would cause the inputfield to always display a * in passwordmode when empty. My fix to this was this commit where the render was done on firstactive.
There could be other ways to solve this, of course.


API authentication not available via Authorization header: Missing CSRF token

Gitea version (or commit ref): 1.16.0+dev-109-gd17f555fe
Can you reproduce the bug at https://try.gitea.io:

[x] Yes (provide example URL)
[ ] No

Description
Swagger lists the option to authorize via the Authorization: Bearer <token> HTTP header.
Unlike authorization via query string parameter ?access_token=<token>, this always fails with the response:
Missing CSRF token.

I assume this applies to all API routes, I specifically tested with GET /notifications.

Checking some other routes, looks like only GET /notifications is affected. GET /repos/search works fine for example

This is very likely a duplicate where it is due to the fact that you're logged with a session in the web API. If you're logged in the with a session then you need a CSRF to post.
Try replicating your problem using curl on the command line. If you can then this is something different but otherwise it's the above.

Indeed that's it, didn't think of that


Add highway=trailhead pois

Connects to https://github.com/mapzen/vector-datasource/issues/594
@nvkelso, @zerebubuth could you review please?

:+1:


Adds cr-cmark-gfm to readme

Link

Checklist

[x] - Shard is at least 30 days old.
[x] - Shard has CI implemented.
[x] - Shard has daily/weekly periodic builds (ideally with Crystal latest and nightly).

I updated this to pass the CI checks


make sampling still works when the trace ignores plug-in activation

Feature: make sampling still works when the trace ignores plug-in activation

[x] Update the documentation to include this new feature.

[x] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes https://github.com/apache/skywalking/issues/6104.

[x] Update the CHANGES log.

Please don't sync the PR by yourself, we are facing limited CI resources. The PMC will take care of the approved PR one by one.

Please don't sync the PR by yourself, we are facing limited CI resources. The PMC will take care of the approved PR one by one.

Well, the Profiling (Influx) seemed to be failed and I rerun, FYI @dmsolr @mrproliu seems the case is not stable

Well, the Profiling (Influx) seemed to be failed and I rerun, FYI @dmsolr @mrproliu seems the case is not stable


Ensure that Package Manager don't access fils outside of its PackageContainer (SOFIE-3083)

About Me

This pull request is posted on behalf of the NRK.
Type of Contribution
This is a: Bug fix / Feature
Current Behavior

There are cases where a file path might resolve to a path outside the PackageContainer specified path.
New Behavior

paths should always resolve to a path inside the PackageContainer path
Testing Instructions

Other Information
Status

[ ] PR is ready to be reviewed.
[ ] The functionality has been tested by the author.
[ ] Relevant unit tests has been added / updated.
[ ] Relevant documentation (code comments, system documentation) has been added / updated.

There was an issue in an added (unrelated) unit test. I ended up merging only the functionality itself.


Black screen on video if going back to video from search

Should be easily reproducible. Play video, make search, go back.

Fixed.


Parse Server Update Issue - MongoDB Connection

Hi,
I had successfully set up MongoDB and the Parse Server a few days after the service termination announcement & ultimately had one of my apps successfully running in test mode (i.e. just my development version).  I noticed yesterday that a new version of Parse Server is available with Push Notification API included which I need for another app. Hence I used npm update & npm update -g to update my set up.  Now when I start MongoDB in exactly the same way I did prior to the update & then start the Parse Server with npm start, the server does not make a connection to the MongoDB database that it is pointed at.  If I run mongo to make a connection to the same db_uri, it works and everything is exactly the same as it was pre-update.  My mongod.conf file is exactly as it was pre-update also.  The version of MongoDB seems to be the same as pre-update.
Anyone have any ideas what may have happened & where I should look for a change that is causing this?
(I suspect something happened to MongoDB because when I tried to migrate data from parse.com for the 2nd app that uses Push Notifications it kept telling me it  could not find the server even though I was using exactly same procedure that succeeded for 1st app).
Also are there any log files for Parse Server? Can't seem to find any.
Thx for any help.
RB

We reverted a change that url encodes mongo passwords. If you are using certain characters in your mongo password (usually @) you will need to either change your password or wait until we can fix the bug. Please re-open if this is not your issue.
You can turning on some logging with environment variable VERBOSE=1 or by using a logging adapter.

Thanks for your response.  I tried two things:
a) Changing the MongoDB password to a simple combination of lower case letters and numbers & restarting MongoDB with mongo --conf .../mongod.conf & Parse Server with npm start.  Unfortunately, no difference - the server starts fine but does not make a connection to MongoDB.
b) Starting MongoDB without enabling authorization & parse server without username and password in the database_uri.  Unfortunately, this made no difference either.
It seems to be some other issue.  Any other thoughts?  Thx.
RB

Did you capture any logging output from VERBOSE=1?

I did use VERBOSE=1 but could not see anything specific on the Terminal or in the Console.  Is there some other location where I should look for a file?

It would be in the terminal.

Thanks for assisting with this issue which it turns out may have been a non-issue.  Apologies for any time wasted.
After numerous attempts to get the connection between MongoDB & the update Parse Server to work, I uninstalled both & started over with fresh installations of MongoDB 2.6 (apparently no version 3.0 available for Homebrew) and he latest Parse Server (2.1.11 I believe).  Once everything was set up I started MongoDB in a terminal window & parse server in a second window.  Still no indication from MongoDB that a connection was established.  However, for some reason I decided to send a test request using curl & bingo I got back the info requested & the MongoDB terminal echoed the message saying a connection had been established.  With the earlier version of parse server, the message had appeared when the parse server was started.  After updating, the lack of a message had fooled me into thinking there was no connection so I never tried the curl request! Lesson learned.


fix: move content to the middle and redirect to podman desktop on page load event

Fix #40. Cannot avoid opening page right now, but redirect on window load event works.
Open podman desktop on window load
https://github.com/redhat-developer/podman-desktop-redhat-account-ext/assets/620330/3bfddaef-2c64-4af6-93b2-531473c28a7c
Open podman desktop using button on the page
https://github.com/redhat-developer/podman-desktop-redhat-account-ext/assets/620330/194d854b-766e-4459-8e20-bd86b78de408

I did not focus on how the page looks, should be follow up issue. I just moved it to middle page to avoid it to be covered when browser asks about using podman desktop to open the link.

@benoitf can I push it for now, then open all follow ups?

@dgolovin please use podman-desktop:// for now and you can merge with that

Testing now with the latest main from Podman Desktop (i.e., commit 287c15d429c1):
When clicking on sign in, the tab will open in the browser but the focus is still on Podman Desktop not on the tab.
Can others reproduce?

@vrothberg looks like I've got the expected behavior on my side (focus on Chrome then Podman Desktop)
https://github.com/redhat-developer/podman-desktop-redhat-account-ext/assets/436777/e3549e43-62e1-4bc6-8a75-92caf8bce7e6
I used the same sha
$ git log -1 | head -1
commit 287c15d429c1dd104c3451012ccce9e37d50da45

@vrothberg same for me, it works as expected, browser gets focus.


Spp 10189 sml healthcheck clean version

Description
This PR is to add the sml portal healthcheck. This comprises of a rule driven lambda function which checks the portal for expected status codes and text. In the event the lambda fails a cloudwatch alarm state is changed and triggers another lambda sends a message to a slack channel, triggers an SNS topic to send an alert email to the team and changes a route53 health check into an unhealthy state.
Type of change
Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Technical Enhancements

Checklist:
If any of these are not completed, please explain why in the notes.
Definition of Done
Code and merges

[x] Code to be commented on where applicable
[x] Documentation updated where required
[x] Have considered non-functional requirements such as Security, Performance, Scalability, and Fault Tolerance
[x] I have linted the code

Testing

[x] All levels of acceptance test are passing (automated, integration, manual, accessibility, etc.)
[x] I have run the behave command to check the selenium behaviour tests pass locally
[x] Acceptance criteria met

Other Checks

[x] I have performed a self-review of my own code
[x] I have checked for spelling errors
[x] I have commented on my code, particularly in hard-to-understand areas
[x] My changes don't break anything unexpected
[x] I have checked and updated the security.txt file where required
[x] Up to date with the main branch
[x] For a change that needs to be deployed in a release I have a commit with the fix:, feat: or BREAKING CHANGE:
tags and will copy this commit message to the squash
[ ] For a change that does not affect the deployed code I have added a commit with ci:, docs: or test:
depending upon which area of the solution was affected

Note when merging, squash commit must begin with one of the following tags:
"BREAKING CHANGE", "feat", "fix", "ci", "docs", "test"
Usage
BREAKING CHANGE: any change that breaks current functionality and may cause compatibility errors to users upgrading
E.G BREAKING CHANGE: refactored method x to take variable y as input (changed from variable z)
feat: A new feature
E.G feat: adding method x to future methods table
fix: A bug fix
E.G fix: amending page for method x to show correct release version
ci: Changes to our CI configuration files and scripts (Concourse)
docs: Documentation only changes
test: Adding missing tests or correcting existing tests

DEPLOY_URL=https://d1pcrl2ajv6zuh.cloudfront.net/
cloudfront_id=E2CKD0P9TXXX9G

DEPLOY_URL=https://d14bqm7dah5bvn.cloudfront.net/
cloudfront_id=E3UQMR0MYCASA1

DEPLOY_URL=https://ddhmzlpbvbxjx.cloudfront.net/
cloudfront_id=E1OT5R469BG6Y4

DEPLOY_URL=https://d2c542an58okib.cloudfront.net/
cloudfront_id=E3M44FOMIY65ON

DEPLOY_URL=https://d2c542an58okib.cloudfront.net/
cloudfront_id=E3M44FOMIY65ON

DEPLOY_URL=https://d2c542an58okib.cloudfront.net/
cloudfront_id=E3M44FOMIY65ON

DEPLOY_URL=https://d2c542an58okib.cloudfront.net/
cloudfront_id=E3M44FOMIY65ON

DEPLOY_URL=https://d2rc750ddj73bi.cloudfront.net/
cloudfront_id=E12XAE63S90A2U

DEPLOY_URL=https://d2rc750ddj73bi.cloudfront.net/
cloudfront_id=E12XAE63S90A2U

DEPLOY_URL=https://d2rc750ddj73bi.cloudfront.net/
cloudfront_id=E12XAE63S90A2U

DEPLOY_URL=https://d2rc750ddj73bi.cloudfront.net/
cloudfront_id=E12XAE63S90A2U

DEPLOY_URL=https://d2rc750ddj73bi.cloudfront.net/
cloudfront_id=E12XAE63S90A2U

DEPLOY_URL=https://d2rc750ddj73bi.cloudfront.net/
cloudfront_id=E12XAE63S90A2U

DEPLOY_URL=https://d2rc750ddj73bi.cloudfront.net/
cloudfront_id=E12XAE63S90A2U

DEPLOY_URL=https://d2rc750ddj73bi.cloudfront.net/
cloudfront_id=E12XAE63S90A2U

DEPLOY_URL=https://d2rc750ddj73bi.cloudfront.net/
cloudfront_id=E12XAE63S90A2U

DEPLOY_URL=https://d2rc750ddj73bi.cloudfront.net/
cloudfront_id=E12XAE63S90A2U

DEPLOY_URL=https://d2rc750ddj73bi.cloudfront.net/
cloudfront_id=E12XAE63S90A2U

DEPLOY_URL=https://d2rc750ddj73bi.cloudfront.net/
cloudfront_id=E12XAE63S90A2U

DEPLOY_URL=https://dylaxr0c7ecol.cloudfront.net/
cloudfront_id=E2LD1SB4COKDMG

DEPLOY_URL=https://dylaxr0c7ecol.cloudfront.net/
cloudfront_id=E2LD1SB4COKDMG

DEPLOY_URL=https://dylaxr0c7ecol.cloudfront.net/
cloudfront_id=E2LD1SB4COKDMG

DEPLOY_URL=https://dylaxr0c7ecol.cloudfront.net/
cloudfront_id=E2LD1SB4COKDMG

DEPLOY_URL=https://dylaxr0c7ecol.cloudfront.net/
cloudfront_id=E2LD1SB4COKDMG

DEPLOY_URL=https://dylaxr0c7ecol.cloudfront.net/
cloudfront_id=E2LD1SB4COKDMG

DEPLOY_URL=https://dylaxr0c7ecol.cloudfront.net/
cloudfront_id=E2LD1SB4COKDMG

DEPLOY_URL=https://dylaxr0c7ecol.cloudfront.net/
cloudfront_id=E2LD1SB4COKDMG

DEPLOY_URL=https://dylaxr0c7ecol.cloudfront.net/
cloudfront_id=E2LD1SB4COKDMG

DEPLOY_URL=https://db6dsktq0547y.cloudfront.net/
cloudfront_id=EIK98RCGB5TG9

DEPLOY_URL=https://db6dsktq0547y.cloudfront.net/
cloudfront_id=EIK98RCGB5TG9

DEPLOY_URL=https://d29elcgex7e89m.cloudfront.net/
cloudfront_id=E7K1K5G9EJ6WU

DEPLOY_URL=https://d29elcgex7e89m.cloudfront.net/
cloudfront_id=E7K1K5G9EJ6WU

DEPLOY_URL=https://d29elcgex7e89m.cloudfront.net/
cloudfront_id=E7K1K5G9EJ6WU

DEPLOY_URL=https://d29elcgex7e89m.cloudfront.net/
cloudfront_id=E7K1K5G9EJ6WU

DEPLOY_URL=https://d29elcgex7e89m.cloudfront.net/
cloudfront_id=E7K1K5G9EJ6WU

DEPLOY_URL=https://d29elcgex7e89m.cloudfront.net/
cloudfront_id=E7K1K5G9EJ6WU

DEPLOY_URL=https://d29elcgex7e89m.cloudfront.net/
cloudfront_id=E7K1K5G9EJ6WU

DEPLOY_URL=https://d29elcgex7e89m.cloudfront.net/
cloudfront_id=E7K1K5G9EJ6WU

DEPLOY_URL=https://d29elcgex7e89m.cloudfront.net/
cloudfront_id=E7K1K5G9EJ6WU

DEPLOY_URL=https://d29elcgex7e89m.cloudfront.net/
cloudfront_id=E7K1K5G9EJ6WU

DEPLOY_URL=https://d29elcgex7e89m.cloudfront.net/
cloudfront_id=E7K1K5G9EJ6WU

DEPLOY_URL=https://d29elcgex7e89m.cloudfront.net/
cloudfront_id=E7K1K5G9EJ6WU

DEPLOY_URL=https://d29elcgex7e89m.cloudfront.net/
cloudfront_id=E7K1K5G9EJ6WU

DEPLOY_URL=https://d29elcgex7e89m.cloudfront.net/
cloudfront_id=E7K1K5G9EJ6WU

DEPLOY_URL=https://d29elcgex7e89m.cloudfront.net/
cloudfront_id=E7K1K5G9EJ6WU

DEPLOY_URL=https://d29elcgex7e89m.cloudfront.net/
cloudfront_id=E7K1K5G9EJ6WU

DEPLOY_URL=https://d29elcgex7e89m.cloudfront.net/
cloudfront_id=E7K1K5G9EJ6WU

DEPLOY_URL=https://d29elcgex7e89m.cloudfront.net/
cloudfront_id=E7K1K5G9EJ6WU

DEPLOY_URL=https://d21aakc97o0s1o.cloudfront.net/
cloudfront_id=E27NDST043G8EP

DEPLOY_URL=https://d2t4m1wi6kfz1w.cloudfront.net/
cloudfront_id=EK5VGFAI1WJSI

DEPLOY_URL=https://d2t4m1wi6kfz1w.cloudfront.net/
cloudfront_id=EK5VGFAI1WJSI

DEPLOY_URL=https://d1ix3xfmjysgf0.cloudfront.net/
cloudfront_id=E389RUXUG6I6EE

DEPLOY_URL=https://d1ix3xfmjysgf0.cloudfront.net/
cloudfront_id=E389RUXUG6I6EE

DEPLOY_URL=https://d1rtg2vxx9ladf.cloudfront.net/
cloudfront_id=E2XLFAO4KJGKQS


Added port type content filter on port dragging in visual shader

This PR adds filtering to a member dialog on port dragging-in/out to prevent create and connect to nodes with incompatible types (or if ports do not exist at all), using this way. For example, dragging from transform port will show such result:

And dragging from sampler port will be like that:

Note: Custom nodes are ignored and show in any cases, for now

Thanks!


Change/hide location from google search

Is your feature request related to a problem? Please describe.
No it isn't
What package is this feature request for?
the google-sr NPM package
Describe the solution you'd like
Currently, I'm receiving a lot of location-specific results. I'd like for there to be an option to remove my location from the search entirely, and get results that are relevant globally.
Describe alternatives you've considered
I tried setting a 'gl' param with no luck. I haven't tried many solutions yet though.
Additional context
Thanks for building this package! I think you did a great job and I hope this suggestion isn't a burden to implement.

can you give me a specific query that your using to get this result.
the issue is its entirely upto google on what results are to be returned. unless you use a proxy or something i dont see how we can fix this.

okay i did some tests and saw that adding the 'gl' is working as expected (without proxies). can you provide a small code sample on how you added the param?

Thanks for taking the time to look into it!
I used some query like 'best coffee shop', which admittedly is useless without geolocation. I was just hoping to find a way to search with as little bias as possible.
Maybe consider adding the gl param to search()'s options, so it's a little easier to set?

I used some query like 'best coffee shop',

lol that is the same query i used for my test.
the use case seems too specific to add the gl param as a option, since the main reason for requestOptions is to facilitate this

nice lol that's a wild coincidence
Yeah a top-level param option is what I meant, sorry for the !articulate.


Option to append a new line to CSV when saving

Version?

vs code plugin

Feature
Currently last new line of CSV file is removed when saving. This may be the default behavior, but I'd like an option to always append a new line to last line, so that all lines have a newline.

I think the built-in editor settings files.insertFinalNewline and editor.renderFinalNewline are more appropriate for that?

files.insertFinalNewline works perfectly. It absolutely ok, that vscode does this for any file, not just CSV. Thanks for the hint!


VSMac: Exception thrown when editing razor files outside project context

In VSMac, open a razor file (no project/solution opened)
With debugger attached, type < to bring up comletion

Actual: Debugger.Fail is hit due to RazorDirectiveAttributeCompletionSourceProvider
Callstack:
at System.Collections.Hashtable.Insert (System.Object key, System.Object nvalue, System.Boolean add) [0x00199]
at System.Collections.Hashtable.Add (System.Object key, System.Object value) [0x00000]
at System.Collections.Specialized.HybridDictionary.Add (System.Object key, System.Object value) [0x00008]
at Microsoft.VisualStudio.Utilities.PropertyCollection.AddProperty (System.Object key, System.Object property) [0x0002e]
at Microsoft.VisualStudio.Editor.Razor.Completion.RazorDirectiveAttributeCompletionSourceProvider.GetOrCreate (Microsoft.VisualStudio.Text.Editor.ITextView textView) [0x00043]
at Microsoft.VisualStudio.Language.Intellisense.AsyncCompletion.Implementation.AsyncCompletionBroker+<>c__DisplayClass86_2.b__4 () [0x00000]
at Microsoft.VisualStudio.Text.Utilities.GuardedOperations.CallExtensionPoint[T] (System.Object errorSource, System.Func`1[TResult] call, T valueOnThrow) [0x0000f]

Closing as this was VSMac


[KBM] Prevent keyboard manager crash when index is not found

Summary of the Pull Request
What is this about?
This PR prevents Keyboard Manager from crashing in the target app textbox LostFocus handler due to a InvalidPointerRead exception.
To repro the exception easily, add Sleep(5000) in the handler like in this commit https://github.com/microsoft/PowerToys/commit/7064482e989c004170f07180ea83233ac7c449d7 and follow the steps in the gif. When this is done the WinRT error input data cannot be processed in the non-chronological order will also appear in the VS output.

The exception seems to occur if a user manages to press the delete and invoke LostFocus at the same time. Without the artificial Sleep this could happen if someone's system is running slow and hanging for some reason (most likely external to PowerToys).
PR Checklist

[X] Applies to #6483
[X] CLA signed. If not, go over here and sign the CLA
[X] Tests passed

Info on Pull Request
What does this include?

The IndexOf method can fail if the element was not found in the collection. For this we should check the return value, and if it is false that means the element was not found. As mentioned above this could occur if the row got deleted before the LostFocus handler completed execution.
An extra safeguard was added for the part where element in the vector was accessed, however while testing with the above repro the execution never went into that block.
Added similar IndexOf safeguards in the Delete button handlers.

Validation Steps Performed
How does someone test & validate?

Add Sleep(5000) as in this commit https://github.com/microsoft/PowerToys/commit/7064482e989c004170f07180ea83233ac7c449d7 and repro with the steps in the gif, i.e. add 3 remap rows, and set focus to the bottom target app textbox and then press the Delete buttons and other textboxes. With the changes in this PR the app will not crash.

Trying to 'kick' the cla check.


User does not have sufficient permissions for this profile

I've gone through all the setup instructions, and looked through previous Issues here but can't find any solution to getting this working. I'm wondering if there's been Google updates that have broken this? Initially I thought it might be because my Analytics property is GA4, so I created a 2nd one that's GA3/GA4, but the same issue persists.
Console errors:
Uncaught (in promise)
{error: {…}}
error:
code: 403
errors: [{…}]
message: "User does not have sufficient permissions for this profile."

https://content.googleapis.com/analytics/v3/data/ga?start-date=2021-07-16&end-date=2021-08-15&ids=ga%3A_[propertyID]_&dimensions=ga%3Adate%2Cga%3AnthDay&metrics=ga%3Asessions&_src=embed-api%3Av1
Browsing to the above url (ID removed) gives the following output:
{
  "error": {
    "code": 401,
    "message": "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
    "errors": [
      {
        "message": "Login Required.",
        "domain": "global",
        "reason": "required",
        "location": "Authorization",
        "locationType": "header"
      }
    ],
    "status": "UNAUTHENTICATED"
  }
}

It's not that useful a message in the end as any malformed or even missing data seems to produce the same error. The best I can take from that is that none of login credential info is getting through. The instructions for GA_KEY_FILEPATH are a little ambiguous (is this an OS path or partial URL?) - I tried both and also using the GA_KEY_CONTENT to make sure. Same result.
Steps I've followed (and re-followed from scratch to make sure I'm not doing something stupid):

Create new project in Cloud console.
Enable Analytics API
Create service account with Viewer role.
Create key.
Copy service account email address.
In Analytics console, add service account email address to property with Read & Analyse access.
Add relevant entries to local.py using property ID from property settings. I've tried with both GA_KEY_FILEPATH and GA_KEY_CONTENT.

From cloud console:

From analytics console:

3rd time setting up Analytics property and cloud project from scratch, using non-g-suite affiliated google account this time, same result.
Shame, this looked like it would be a good tool.

I ran into this issue today. It works with UA properties but not with GA4 properties. GA4 properties don't have a View ID (the Property ID is not the same, using that leads to the 403 error). I think Wagalytics would need to be refactored to use the new Analytics Data API.