gadgetCPU.gadgetInit()  report an error!

When I run the code:
gadgetCPU = SpeedTorch.DataGadget( 'data.npy',CPUPinn=True)
gadgetCPU.gadgetInit()
report an error like this:
Exception ignored in: <function PMemory.del at 0x7fcef8ca86a8>
Traceback (most recent call last):
File "/usr/local/python3/lib/python3.7/site-packages/SpeedTorch/CUPYLive.py", line 19, in del
AttributeError: 'NoneType' object has no attribute 'runtime'
But it's ok when I run :
gadgetGPU = SpeedTorch.DataGadget( 'data.npy' )
gadgetGPU.gadgetInit()
I can't find  the reason,it confused me .

Hi xscjun,
That is very confusing. I am investigating by trying to recreate the issue. So far I am unable to re-create the issue in Colab.
Here's the code I used
!pip install SpeedTorch
#Always import cupy before SpeedTorch
import cupy
import SpeedTorch
import torch
import numpy as np
import torch.nn as nn

sampl = np.random.uniform(low=-1.0, high=1.0, size=(10, 10, 10, 10))
np.save('data.npy', sampl)
del sampl

gadgetGPU = SpeedTorch.DataGadget( 'data.npy', CPUPinn=True )
gadgetGPU.gadgetInit()

For convenience, here's the colab notebook I used.
https://colab.research.google.com/drive/1TbqKwZ94p_B6q0t_orYObKsWwa7Fg0ld
Are you able to recreate the issue in Colab? If so, link a notebook for further investigation.
If not, it looks like it's an issue with your system. In that case, you could provide your system information and see if we can figure out what is causing the error from that info.

Hi xscjun,
That is very confusing. I am investigating by trying to recreate the issue. So far I am unable to re-create the issue in Colab.
Here's the code I used
!pip install SpeedTorch
#Always import cupy before SpeedTorch
import cupy
import SpeedTorch
import torch
import numpy as np
import torch.nn as nn

sampl = np.random.uniform(low=-1.0, high=1.0, size=(10, 10, 10, 10))
np.save('data.npy', sampl)
del sampl

gadgetGPU = SpeedTorch.DataGadget( 'data.npy', CPUPinn=True )
gadgetGPU.gadgetInit()

For convenience, here's the colab notebook I used.
https://colab.research.google.com/drive/1TbqKwZ94p_B6q0t_orYObKsWwa7Fg0ld
Are you able to recreate the issue in Colab? If so, link a notebook for further investigation.
If not, it looks like it's an issue with your system. In that case, you could provide your system information and see if we can figure out what is causing the error from that info.

Thanks for your reply，It's the python3.7 that report the error, I change the version of python to 2.7,   It's ok now.

Glad you were able to get it working. I am wondering what the cause is; I did all my testing in Python 3.

Glad you were able to get it working. I am wondering what the cause is; I did all my testing in Python 3.

I can't get it working in python3.7 ,it is confusing.

I noticed Colab has Python 3.6.8 by default, so perhaps there is something off about 3.7.
Are you able to recreate the issue in Colab? Is the 'data.npy' the same as in the notebook?

I noticed Colab has Python 3.6.8 by default, so perhaps there is something off about 3.7.
Are you able to recreate the issue in Colab? Is the 'data.npy' the same as in the notebook?

The 'data.npy' is the same as in the notebook. I haven't recreate the issue in Colab

I got OOM error, how can I deel with it? BTY,  how can I load multiple data in one container？
gadgetGPU = SpeedTorch.DataGadget(target_mel) gadgetGPU.gadgetInit() Traceback (most recent call last):   File "/home/zzy/anaconda3/envs/StarGAN-VC/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3326, in run_code     exec(code_obj, self.user_global_ns, self.user_ns)   File "<ipython-input-19-226edd99569f>", line 1, in <module>     gadgetGPU.gadgetInit()   File "/home/zzy/anaconda3/envs/StarGAN-VC/lib/python3.6/site-packages/SpeedTorch/CUPYLive.py", line 265, in gadgetInit     self.CUPYcorpus = cupy.load( self.fileName)   File "/home/zzy/anaconda3/envs/StarGAN-VC/lib/python3.6/site-packages/cupy/io/npz.py", line 71, in load     return cupy.array(obj)   File "/home/zzy/anaconda3/envs/StarGAN-VC/lib/python3.6/site-packages/cupy/creation/from_data.py", line 43, in array     return core.array(obj, dtype, copy, order, subok, ndmin)   File "cupy/core/core.pyx", line 1768, in cupy.core.core.array   File "cupy/core/core.pyx", line 1845, in cupy.core.core.array   File "cupy/core/core.pyx", line 1920, in cupy.core.core._send_object_to_gpu   File "cupy/core/core.pyx", line 134, in cupy.core.core.ndarray.__init__   File "cupy/cuda/memory.pyx", line 540, in cupy.cuda.memory.alloc   File "cupy/cuda/memory.pyx", line 1234, in cupy.cuda.memory.MemoryPool.malloc   File "cupy/cuda/memory.pyx", line 1255, in cupy.cuda.memory.MemoryPool.malloc   File "cupy/cuda/memory.pyx", line 1033, in cupy.cuda.memory.SingleDeviceMemoryPool.malloc   File "cupy/cuda/memory.pyx", line 1053, in cupy.cuda.memory.SingleDeviceMemoryPool._malloc   File "cupy/cuda/memory.pyx", line 775, in cupy.cuda.memory._try_malloc cupy.cuda.memory.OutOfMemoryError: Out of memory allocating 86,528 bytes (allocated so far: 0 bytes).

I got OOM error, how can I deel with it? BTY, how can I load multiple data in one container？
gadgetGPU = SpeedTorch.DataGadget(target_mel) gadgetGPU.gadgetInit() Traceback (most recent call last): File "/home/zzy/anaconda3/envs/StarGAN-VC/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3326, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "<ipython-input-19-226edd99569f>", line 1, in <module> gadgetGPU.gadgetInit() File "/home/zzy/anaconda3/envs/StarGAN-VC/lib/python3.6/site-packages/SpeedTorch/CUPYLive.py", line 265, in gadgetInit self.CUPYcorpus = cupy.load( self.fileName) File "/home/zzy/anaconda3/envs/StarGAN-VC/lib/python3.6/site-packages/cupy/io/npz.py", line 71, in load return cupy.array(obj) File "/home/zzy/anaconda3/envs/StarGAN-VC/lib/python3.6/site-packages/cupy/creation/from_data.py", line 43, in array return core.array(obj, dtype, copy, order, subok, ndmin) File "cupy/core/core.pyx", line 1768, in cupy.core.core.array File "cupy/core/core.pyx", line 1845, in cupy.core.core.array File "cupy/core/core.pyx", line 1920, in cupy.core.core._send_object_to_gpu File "cupy/core/core.pyx", line 134, in cupy.core.core.ndarray.__init__ File "cupy/cuda/memory.pyx", line 540, in cupy.cuda.memory.alloc File "cupy/cuda/memory.pyx", line 1234, in cupy.cuda.memory.MemoryPool.malloc File "cupy/cuda/memory.pyx", line 1255, in cupy.cuda.memory.MemoryPool.malloc File "cupy/cuda/memory.pyx", line 1033, in cupy.cuda.memory.SingleDeviceMemoryPool.malloc File "cupy/cuda/memory.pyx", line 1053, in cupy.cuda.memory.SingleDeviceMemoryPool._malloc File "cupy/cuda/memory.pyx", line 775, in cupy.cuda.memory._try_malloc cupy.cuda.memory.OutOfMemoryError: Out of memory allocating 86,528 bytes (allocated so far: 0 bytes).

Can you make a colab that reproduces this error? That way I can interact with the bug.

BTY, how can I load multiple data in one container？

I haven't put that feature in, but I can put it in. So you would want
ModelFactoryObject.loadCupy( loadFileName) for the first dataset, and then
for new datasets. something like
ModelFactoryObject.appendCupy( loadFileName2)
?

import cupy import SpeedTorch gadgetGPU = SpeedTorch.DataGadget('mel-20170001P00084I0004.npy') gadgetGPU.gadgetInit()
mel-20170001P00084I0004.zip
It seems gadgetGPU.gadgetInit()will result this error no matter which data I load.

It looks like the data format is incorrect. It looks like there's an issue with how you're saving the data, and/or how your zipping the file.
Checkout this notebook, which saves and loads numpy data
https://colab.research.google.com/drive/185Z5Gi62AZxh-EeMfrTtjqxEifHOBXxF
Try using numpy.save to directly save your data into a numpy format file.

It looks like the data format is incorrect. It looks like there's an issue with how you're saving the data, and/or how your zipping the file.
Checkout this notebook, which saves and loads numpy data
https://colab.research.google.com/drive/185Z5Gi62AZxh-EeMfrTtjqxEifHOBXxF
Try using numpy.save to directly save your data into a numpy format file.

It seems hard to change the data formate as my model has trained for a long time... Is there any method I can transfer data from torch to your container? Or is there any method can replace torch.utils.data.Dataloader in pytorch? For example I've already preprocessed my data and saved it in CPU.

Yup, I forgot the exact commands, but you can access your embedding data and mount them to CPU, in numpy form. It looks something like this  YourModel.YourEmbeddingVariable.Weight.data.cpu().numpy()
details
https://discuss.pytorch.org/t/how-to-transform-variable-into-numpy/104/5

Yup, I forgot the exact commands, but you can access your embedding data and mount them to CPU, in numpy form. It looks something like this YourModel.YourEmbeddingVariable.Weight.data.cpu().numpy()
details
https://discuss.pytorch.org/t/how-to-transform-variable-into-numpy/104/5

Can't open the link...

Thank you for replying! I can load files by using speedtorch now. But cupy doesn't support multi-thread, so I have to modify the thread from 8 to 1, after that, the time cost doesn't change......

Yes, as long as the data is saved in numpy format, data gadget can open it, or your could transfer live data onto there. If you give me a colab notebook which loads your data, I can tinker around with it. I think the easiest way to do this to upload your data onto google drive, then use !gdown --id followed by the google drive id to download it directly to your notebook.
There's documentation at the bottom of the readme, and here's a colab notebook which shows how to use the data gadget:
https://colab.research.google.com/drive/1TbqKwZ94p_B6q0t_orYObKsWwa7Fg0ld

Thank you for replying! I can load files by using speedtorch now. But cupy doesn't support multi-thread, so I have to modify the thread from 8 to 1, after that, the time cost is even longer......

How many cores is your CPU? The main speedtorch advantages are for a lower number of CPUs like, 1-4. After that, Pytorches indexing kernals become more efficient.
I would love to see a colab version of your code, maybe i can tinker a bit

Thank you for replying! I can load files by using speedtorch now. But cupy doesn't support multi-thread, so I have to modify the thread from 8 to 1, after that, the time cost is even longer......

How many cores is your CPU? The main speedtorch advantages are for a lower number of CPUs like, 1-4. After that, Pytorches indexing kernals become more efficient.
I would love to see a colab version of your code, maybe i can tinker a bit

My CPU info: 8  Intel(R) Xeon(R) W-2123 CPU @ 3.60GHz
The model I use is based on https://github.com/NVIDIA/tacotron2
And I modified  this line to
melspec = SpeedTorch.DataGadget(full_path) melspec.gadgetInit() melspec = melspec.getData()
(BTW, I don't know how to get full Data, so I modified getData() and remove the parameter index)

How many cores does that CPU have? I can't seem to look it up.
I'm not too familiar with that model. But with a colab notebook perhaps I can tinker around.

How many cores does that CPU have? I can't seem to look it up.
I'm not too familiar with that model. But with a colab notebook perhaps I can tinker around.

4 cores. I'm afraid I can't upload the model on colab, every time I open that link, my computer is about to freeze......


JSON output for CLI

The CLI tool currently outputs key-value, line by line, e.g.
domain: konklone.com
uri: https://konklone.com
government: false
live: true
ssl: true
...

This changes the CLI to output pretty-printed JSON instead:
{
  "domain": "konklone.com",
  "uri": "https://konklone.com",
  "government": false,
  "live": true,
  "ssl": true,
  ...

It's basically just as readable, but can be piped into other things and efficiently saved to disk in a reliably parseable, standard format.
I also:

updated the CLI test to accommodate the new format
bumped the Ruby version up to 2.2.0 to fix the build on Travis, which for some reason says 2.1.1 isn't available anymore
fixed a typo in .travis.yml that shouldn't have had any effect anyway

The build is failing, but for reasons that have nothing to do with my work. Maybe DNSSEC doesn't work from the Travis environment? Either way, the tests pass locally.

If you don't mind changing 2.2.0 to 2.1 (travis shorthand for 2.1.x), more than glad to merge (which may resolve the failing build).

I left the .ruby-version at 2.2.0, but set the Travis build info to use 2.1, which selected 2.1.5 during the build process -- which unfortunately still fails on DNSSec.

I give up. Either way, your PR didn't break it. Will clean up on master. Thanks again!

Thank you!


copr: add non-vectorize JsonValidJson

Signed-off-by: aooohooo<EMAIL_ADDRESS>UCP: #6798
Checklist: #5751

What problem does this PR solve?
Issue Number: close #6798
Problem Summary: add non-vectorize JsonValidJson
What is changed and how it works?
What's Changed: Port the scalar function JsonValidJson from TiDB to coprocessor.
Related changes

No

Check List
Tests

Unit test

Release note

LGTM

/approve

/merge

@return02 Sorry for the delay. Unfortunately non-vec expressions no longer exist, so I'm going to close this PR.
Sorry about that, and thank you for your contribution.


Bundle JS with the Hono middleware

Currently, the Hono middleware uses the CDN version.
But we should bundle the JS as we do for the Fastify plugin.

Out of scope for now


Fix doc expension

The doc argument was not expended and the doc was always
/// doc_arg
or
/// $doc_arg

I'll first fix the CI error and merge soon.

@jeromerobert can you rename your commit following the conventional commit convention? -> fix: doc expansion

The convco checker seems confused (not sure why). I picked your commit in #2 and merged it. Thanks anyway!
I'll publish si-scale 0.2.2 soon.


Ease Github Notifications in Jenkinsfiles

Using https://github.com/jenkinsci/pipeline-githubnotify-step-plugin might allow for easier to read and use Jenkinsfiles.

Obsolete due to #214


believe it does not detect "python -m venv"

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
python by default has its own virtual environment by command
python -m venv [projectname]
once created, it will created several folders inside  [projectname] folder and create file pyvenv.cfg.
I believe this extension does not auto detect it.
Describe the solution you'd like
A clear and concise description of what you want to happen.
it is good if this extension able to auto detect it and have the terminal inside vscode auto start inside this virtual env.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

Hi @msrn1975,
First off I would like to mention that this extension should be seen as an extension to the official Python extension available. It simply does not work at all without it.
Could it be that you missed this requirement?

it is good if this extension able to auto detect it and have the terminal inside vscode auto start inside this virtual env.
This is actually something that the official Python extension already has support for. This extension only tells the official Python extension where the virtual env is located, that's it.

This extension could further be set up to detect further requirement filenames or names of virtual env directories. Have you tried those out yet?

Could you send over some of the logs from the extension logging output?
You can find them here:

I think u are right, the log shows microsoft python ext error.
On Fri, Feb 19, 2021 at 11:55 PM Mattias Edlund<EMAIL_ADDRESS>wrote:

Could you send over some of the logs from the extension logging output?
You can find them here:
[image: image]
https://user-images.githubusercontent.com/2932594/108527887-1ae7d900-72d3-11eb-80d8-a7a753a73cbe.png
—
You are receiving this because you were mentioned.
 Thank you for using it and reporting this issue, and sorry for not being able to continue supporting it.


Update to OWNERS_ALIASES

Update needed so I can use CNCF tools
Pep is no longer assigned to project KubeVirt so I'm removing him as well

/lgtm
/approve

/release-note-none


Body Posture Detection and Prediction

Body Posture Detection and Prediction
Nishtha Pabreja
Nishtha1203 (github)
Linear Regression, KNN, Decision Tree

Share the dataset and approach for this project.

https://github.com/abhisheks008/ML-Crate/issues/111
The issue is already created, why have you created a new issue. No need to create a new issue, you can comment out in the existing issue with your approach to get assigned.


pin a version of serde without intransparent unreproducible binary blobs

Serde is shipping a binary blob in its derive crate, which is highly unexpected and subverts user trust. To make matters worse, the binary is not even reproducible, making the crate largely unauditable and relinquishing the security benefits of open-source software. Build times are not nearly painful enough to justify forcing users to trust binary blobs.

@bors r=RalfJung,oli-obk

:pushpin: Commit 5c4be1a41c3b92fec4eb57a8be420d32e30332d7 has been approved by RalfJung,oli-obk
It is now in the queue for this repository.

:hourglass: Testing commit 5c4be1a41c3b92fec4eb57a8be420d32e30332d7 with merge 531c35e01d4859e3d44f1bd5c599bbe82d0e6b03...

:sunny: Test successful - checks-actions
Approved by: RalfJung,oli-obk
Pushing 531c35e01d4859e3d44f1bd5c599bbe82d0e6b03 to master...

:sunny: Test successful - checks-actions
Approved by: RalfJung,oli-obk
Pushing 531c35e01d4859e3d44f1bd5c599bbe82d0e6b03 to master...


For the STAT-Analysis "filter" job type apply the -set_hdr job commands to the -dump_row output file.

This issue arose when Jonathan ran SWPC data through MET.  This resulted in long strings in the MET output columns.  The strings were too long for METviewer and caused the loader to fail.  To patch the data, Jonathan used sed to shorten the strings.  We tried running a STAT-Analysis filter job with the -set_hdr option to do this, but it didn't work.  The -set_hdr option currently only applies to the output of the aggregate or aggregate_stat job types.
This task is to enable the -set_hdr option for the filter job type when creating the -dump_row output.  BUT it should only apply to the -dump_row output from filter, and not the -dump_row output of the other job types!

Also consider supporting the use of regular expressions for the -set_hdr option throughout stat-analysis.
Currently, the set_hdr value is applied to all of the output for the column.  But this make using the "-by" option less useful.
For example, let's say you have data with FCST_VAR = TMP and UGRD, you could define:
-job aggregate -line_type SL1L2 -by FCST_VAR
-set_hdr FCST_VAR TEMPERATURE 'T.'
-set_hdr FCST_VAR UWIND 'UG.'
So the set_hdr options would only be applied when the current string matches the regular expression listed.

Here's an email to Mallory Row describing the potential for related development:
Mallory,
When used for the aggregate or aggregate_stat job types, the intended purpose of the -dump_row option is for users to be able to see the actual input lines that were used when processing each job.  So it's meant as a sanity check to double-check the filtering logic the user defined.
But for the filter job (which writes it output to the -dump_row file), the intention is a little different.  For filter, stat_analysis is a fancy form of "grep", enabling the user to slice/dice their data however they'd like.
Just earlier today, we talked about enhancing the filter job type to support the "-set_hdr" option.  For example, we have some data with a very long FCST_UNITS string and want to reset that to a shorter string.  So we'd like to run a job like this:
stat_analysis -lookin stat_data -job filter -set_hdr FCST_UNITS TEC -dump_row short_units.txt
But this is not currently supported.  Here's the GitHub issue for this:
https://github.com/NCAR/MET/issues/1129
If -set_hdr is used, this would require STAT-Analysis to actually parse the input lines, update strings, and write it back out.  As long as we're parsing the data anyway, we could also consider updating the version number before writing it to the output.  And in that step, we would, for example, add FCST_UNITS and OBS_UNITS to the output of the filter job.
It seems to me like using "-dump_row" in both contexts is confusing.  Instead, perhaps we should require that the "filter" job use the "-out_stat" job command option to specify its output file?
Would that be a useful solution?  Of course, that would only fix .stat output files.  There is no "filter" job for MODE or MTD output data.

Mallory confirms that this functionality would be useful.  So the changes would be this:
(1) The -dump_row option remains as-is... whatever .stat lines are read as input should be written to the output -dump_row file.  If we're writing the first line of output and the first line read is a header line... dump that to the output file.  All future header lines should be ignored.
(2) For the -filter job, make the -out_stat command line option required.  Regardless of the version of the .stat lines read as input, the output will now be written for the current version number.
Should we buffer all of the lines in one ascii table in memory to get the columns to line up?  Maybe that's overkill and isn't worth the extra memory consumption.

Charge 277047


fix duplicite IDs for steps

Fix duplicite IDs in workflow script.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Please upload report for BASE (main@6460a83). Learn more about missing BASE report.

Additional details and impacted files
@@           Coverage Diff            @@
##             main      #336   +/-   ##
========================================
  Coverage        ?   100.00%
========================================
  Files           ?        18
  Lines           ?       502
  Branches        ?         0
========================================
  Hits            ?       502
  Misses          ?         0
  Partials        ?         0

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Upgraded to Python 3 still could not run udc_test.py?

I upgraded to Python 3 but errors still ... Any parameters need to change in udc_test.py in order to evaluate the model?
Traceback (most recent call last):
File "udc_test.py", line 39, in
estimator.evaluate(input_fn=input_fn_test, steps=None, metrics=eval_metrics)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 356, in evaluate
name=name)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 641, in _evaluate_model
max_steps=steps)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/graph_actions.py", line 773, in evaluate
_write_summary_results(output_dir, eval_results, current_global_step)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/graph_actions.py", line 613, in _write_summary_results
_eval_results_to_str(eval_results))
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/graph_actions.py", line 607, in _eval_results_to_str
return ', '.join('%s = %s' % (k, v) for k, v in eval_results.items())
AttributeError: 'NoneType' object has no attribute 'items'

I just had this strange issue as well. Occurred after calling evaluate on a just-trained Estimator passing in the following metrics dict:
{'auc': <function streaming_auc at 0x49226e0>, 'recall': <function streaming_recall at 0x49225f0>,  'precision': <function streaming_precision at 0x4922578>, 'confusion_matrix': <function confusion_matrix at 0x4730d70>, 'accuracy': <function streaming_accuracy at 0x4922500>}

Traceback is below:
Traceback (most recent call last):
  File "main.py", line 301, in <module>
    tf.app.run()
  File "/usr/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
    sys.exit(main(sys.argv))
  File "main.py", line 220, in main
    eval_steps=1
  File "/opt/projects/tensorflow/cross_validation/evaluate.py", line 54, in evaluate
    metrics=metrics
  File "/usr/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 356, in evaluate
    name=name)
  File "/usr/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 641, in _evaluate_model
    max_steps=steps)
  File "/usr/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/graph_actions.py", line 773, in evaluate
    _write_summary_results(output_dir, eval_results, current_global_step)
  File "/usr/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/graph_actions.py", line 613, in _write_summary_results
    _eval_results_to_str(eval_results))
  File "/usr/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/graph_actions.py", line 607, in _eval_results_to_str
    return ', '.join('%s = %s' % (k, v) for k, v in eval_results.items())
AttributeError: 'NoneType' object has no attribute 'items'

Never mind, this is a direct result from having a model that hasn't been trained, i.e. no weights or other variables saved. This can result from trying to use training files that don't exist when fit is called. If you need help debugging this, use tf.logging.set_verbosity(tf.logging.DEBUG) and look at the logs carefully to see what's going on. The Estimator will not fail due to user misconfiguration

While training the model I am getting the error
InvalidArgumentError (see above for traceback): Incompatible shapes: [80,1] vs. [160,1] 	 [[Node: prediction/logistic_loss/mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](prediction/Squeeze, prediction/ToFloat)]]
I am running it with python3, any idea what might be causing the issue?

same error for me.
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [80,1] vs. [160,1]
[[Node: prediction/logistic_loss/mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](prediction/Squeeze, prediction/ToFloat)]]

Some error for me.
InvalidArgumentError (see above for traceback): Incompatible shapes: [80,1] vs. [160,1]
[[Node: prediction/logistic_loss/mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](prediction/Squeeze, prediction/ToFloat)]]


Remove duplicate SWHS SRS generation

We no longer need to generate the version of the SWHS SRS that does not use RefSec.
Please remove it (you'll likely need to touch multiple files to remove all references to it / remove the second generated files).
When done, SWHS should only generate one copy of SWHS_SRS.html (+.css) and SWHS_SRS.tex. i.e. SWHS_SRS2.* should no longer be generated.

You'll also need to update the makefile to avoid doing diffs on the old duplicate versions.


fix: align with guidance on input type for numbers (NumberField)

Description
Bring the NumberField component inline with the GOV.UK design system guidance on how to ask for numbers.
Update the HTML attribute type from "number" to "text" which stops the browser from restricting which type of characters can be entered into the input - this restriction can be confusing for users, limit the ability to copy & paste and has proven problematic for screen readers.
As the current NumberField component uses joi.number() as the base schema there will already be server side validation ensuring that if anything that can't be parsed by Number() is entered (text or invalid decimals) the user will be prompted to enter a number.

New behaviour: component already validates the input value against Joi number schema, even if the form allows the user to submit

Setting the inputmode attribute to "numeric" will still prompt mobile browsers to provide a keyboard specifically for numbers and decimals (note on the mobile browsers I've tried this change against the keyboard behaviour is currently identical to if the type attribute was set to number).

Current behaviour: mobile keyboard
New behaviour: mobile keyboard

This change does still fallback to the type "number" if the precision option is set on the components schema. Although I wouldn't advise this was used the step attribute which is set on the HTML input will not work if the type is set to "text". Adding custom server side validation to validate and clearly feedback on incorrect decimal precision seemed well out of the scope of this fix and this should allow existing configurations to work as-is. (Note from an early review it looks like setting the precision option doesn't actually appropriately set different granularities of step but I'll look to raise that as a separate issue).
This change aims to bring the NumberField inline with the guidance found here: https://design-system.service.gov.uk/components/text-input/#numbers
Specifically attempting to fix the issues described by: https://design-system.service.gov.uk/components/text-input/#avoid-using-inputs-with-a-type-of-number
A more detailed description of why the guidance was set was published in Feb 2020:
https://technology.blog.gov.uk/2020/02/24/why-the-gov-uk-design-system-team-changed-the-input-type-for-numbers/

Note that because the date part and date time part components use the govuk-frontend macro govukDateInput directly (by passing items into the macro options) those components are already asking for numbers using the appropriate type "text" and inputmode "numeric".
Type of change
Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?
Although the form builder is used in my department this issue was identified during an internal demo between designers and developers. It was not flagged as an issue by end users.
This commit updates to try and align the code to the design system guidance but has not been tried in a real world scenario with users submitting data.
Checklist:

[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas

Thanks @sfount! This looks great. Really appreciate the contribution!
Apologies on my part -- the actions aren't running properly for forks due to our caching strategy. I'll rethink and get this sorted so we can merge.

Thanks @jenbutongit, appreciate you taking the time to look into the caching! There are a few other things I spotted along the way I'd be happy to raise if this is helpful (very small changes with a bit of thinking on the implications)


Linux Foundation (LF) Core Infrastructure Initiative (CII)

Complete CII self certification process.
CII Certification

Updated CII application 4/19/19

Updated to include renamed organization and URLs.

Completed


New version: Amazon.WorkspacesClient version 5.19.0

[ ] Have you signed the Contributor License Agreement?
[ ] Have you checked that there aren't other open pull requests for the same manifest update/change?
[ ] This PR only modifies one (1) manifest
[ ] Have you validated your manifest locally with winget validate --manifest <path>?
[ ] Have you tested your manifest locally with winget install --manifest <path>?
[ ] Does your manifest conform to the 1.6 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

Hi @PckgrBot,
This manifest adds Apps and Features entries that aren't present in previous PR versions. These entries should be added to the previous versions, or removed from this version.
(Automated response - build 857.)

Hi @PckgrBot,
This manifest adds Apps and Features entries that aren't present in previous PR versions. These entries should be added to the previous versions, or removed from this version.
(Automated response - build 857.)

Yes this is correct

From https://github.com/microsoft/winget-pkgs/pull/141633:

@ stephengillie Any idea what is going on with this package? It has deleted the latest version before adding an updated version on several occasions...? Right now the latest version on Winget is 3.1.10 but on the Amazon site it is up to 5.19.0
In the past it has deleted 5.16.0 5 days before adding 5.17.0, same with 5.8.0 and 5.9.0
It keeps deleting the latest before it adds the new which means the only version available in Winget is a 2.5 year old 3.1.10 for several days

Hi @th3nme,
It appears that there are several things happening:

New process is meant to prevent this by holding a removal PR until the complementary addition PR is submitted.

I've been giving this process a timeout of 3 days. So if an addition PR hasn't come up by then, I'll merge anyway to remove from the repo. After this, I'll increase to 7.

Very old version should be removed from the repo.
Apps And Features error here is incorrect, as all 3 versions have them:

3.1.10 very old version has them at the installer level.
5.18.0 previous (removed) version had these at the root level, with ProductCode specified twice.
5.19.0 current (this PR) version has these at the root level, also with ProductCode specified twice.
Guessing this is due to 3.1.10 having it at the installer level.

To summarize: This PR should merge, I should wait a week before closing removals (and check for situations like this first) and a PR to remove 3.1.10 should be created.

[Policy] Reset Feedback


chore: Remove unused org.javassist:javassist

Checklist

[x] the contributor license agreement is signed
[x] commit message follows commit guidelines

Description of change

It would appear this may have just been added from conversion of the old
Ivy setup which didn't do transitive dependencies (so all had to be
managed explicitly in XML files). Having removed it, all still built and
ran via sbt cleanrun.
In response to https://github.com/openequella/openEQUELLA/pull/1698

Most likely a failure due to a flaky test. But the test (com.tle.webtests.test.workflow.rejection.RejectionPointsWithDecisionNodesTest) is not in my list. Rerunning GHA to be sure.


Request: Update tintin to 2.01.1

Tintin++ added ssl support in 2.01.1, their final release. It would be nice to update the install to that version.

their final release

What, did they say this will be their final release? Or do you mean "their latest release"?

Fair point. It's the release from last year which was the only release we've seen from them since 2013. Regardless, getting a version with ssl support into Brew would be nice.

Fair point.

I was not making a point, I was asking a question...
If it's the final release, I would seriously consider boneyarding it. Carrying unsupported software isn't nice, and this one is among the "used by almost no one" camp anyway according to our analytics.
If they did not say it's the final release, then sure.
Can you try to open a PR?

Sorry but I'm not in a position to do the work myself.

Is there any difficulty? It should be pretty easy. Homebrew relies on contributions to thrive; we as a small team can't maintain thousands of packages on our own.

Closing the issue because we don't keep this sort of requests open. Someone, either you or one of us, should open a PR, and that's the only way forward in Homebrew land.


SignalId: add missing methods

Depends on https://github.com/gtk-rs/gir/pull/1060

I think this one is ready for another round of review :-)

Waiting for @GuillaumeGomez now. Before merging this we need to merge the gir PR, then update this here to regenerate with that gir version and then it can be merged.
Also will need to spend some time updating the GStreamer bindings and plugins.

@bilelmoussaoui Then update this PR to latest gir and let's get it merged :)


Some a bit incorrect color and shade gradations

I'm putting some sort of image test to display some hue and value spectrum,
I got some noticeable color banding when using this driver:

I had deployed another technique to actually do the BCM properly, thus, displaying more faithful hue and value spectrum:

(with the cost of sacrificing 1/4 of the total brightness of course, due to limitation of my driver)
Here;s the code of the test image
  char t = millis()>>4;
  for(int x=0;x<128;x++)
    for(int y=0;y<64;y++)
      dma_display->drawPixelRGB888(x,y,
      (uint8_t)((max(min(cosf(2.f*PI*((x-t)/128.f+0.f/3.f))+0.5f,1.f),0.f)*((y+1.f)/64.f))*255.f),
      (uint8_t)((max(min(cosf(2.f*PI*((x-t)/128.f+1.f/3.f))+0.5f,1.f),0.f)*((y+1.f)/64.f))*255.f),
      (uint8_t)((max(min(cosf(2.f*PI*((x-t)/128.f+2.f/3.f))+0.5f,1.f),0.f)*((y+1.f)/64.f))*255.f)
      );

Can you confirm if you get this issue when using version 3.0.0 of the library?
I may have introduced a bug when I fiddled with the way brightness is managed in the most recent version

I had tested the version 3.0.0; it still yields the exact same result. (I can't take the picture I'm sorry).
But I'd think I could try to reproduce how your driver translate the colors:

which was plotted pixel by pixel in python code
r = int((max(min(sin((u/8+0/3)*pi*2)+0.5,1),0)*(1-v)/2)*255)
g = int((max(min(sin((u/8+1/3)*pi*2)+0.5,1),0)*(1-v)/2)*255)
b = int((max(min(sin((u/8+2/3)*pi*2)+0.5,1),0)*(1-v)/2)*255)
r0,r1,r2,r3,r4,r5=r>>7&1,r>>6&1,r>>5&1,r>>4&1,r>>3&1,r>>2&1
g0,g1,g2,g3,g4,g5=g>>7&1,g>>6&1,g>>5&1,g>>4&1,g>>3&1,g>>2&1
b0,b1,b2,b3,b4,b5=b>>7&1,b>>6&1,b>>5&1,b>>4&1,b>>3&1,b>>2&1
r,g,b=r0*6+r1*5+r2*4+r3*3+r4*2+r5,g0*6+g1*5+g2*4+g3*3+g4*2+g5,b0*6+b1*5+b2*4+b3*3+b4*2+b5
sqrt(r/21)*256,sqrt(g/21)*256,sqrt(b/21)*256

The pull request #374 will fix the following artifacts

into

And I already had thorough tests, and only this is was complaint of the driver.


Question: Implementation of mac commands

Hi Orne,
Yesterday I started looking into the MAC commands and here is how I imagine them to work (let me know if that seems OK to you).

LinkCheckReq/LinkCheckAns is somewhat clear (I think that we take lsnr or 0 if lsnr is negative and the number of times the packet was received and construct the answer).
The other MAC commands should be sent by the users (in my opinion through the web interface). It sounds somewhat OK for me if only one set of MAC commands can be saved per device. I would expect that the MAC commands would be more important than the user data if we have to choose between the two (although that might be configurable) as they are supposed to optimize the performance of the end device. I would also expect that in the MAC commands set there are only commands with unique CID. The reason for this assumption is that I did not see any requirement for the order of the MAC commands' answers and if this is the case, then having multiple times the same CID would lead to ambiguous answers. Generally it seems reasonable to be able to send multiple NewChannelReq, but I guess it is not that crucial. The user will be able to see the answers to the last set of MAC commands through the web interface.

I have already implemented the part for LinkCheckReq/LinkCheckAns (as described above), but I was not able to test it today, so I will test it tomorrow I will test it as soon as my end device recovers from whatever happened to it  and push it to my fork. Depending on our discussion I can implement the rest as well and make a pull request.
What do you think about all that?
Thanks,
Ivaylo Petrov

Hi Ivaylo,
Thanks for looking into the MAC commands already! I have to dig a little bit deeper into the MAC commands before I can give a full answer, but as far as I think, there are three types: 1) initiated by the node, 2) initiated by the network server (e.g. frequency related MAC commands, could be based on settings?), 3) initiated by the user (give me the battery power, through the api).
I appreciate your help, but since some of the MAC commands are related to regions, I'm thinking about first implementing support for different ISM bands, or at least cleanup the currently hardcoded frequencies and put them a sane model that can be re-used for different ISM bands (e.g. by using different build flags, one could specify for which ISM band to compile).
As well I have to think about how to model which MAC commands are sent as FOpts (they can go together with an application payload) and which ones need to be sent as FRMPayload. This might be a (per application?) setting too.
Next week Thursday I'm going to present this project to the local Go meetup here in Amsterdam. Until then I'll cleanup some loose ends, fix some bugs and work on my presentation. So probably the week after I'll be working on new features again :smile:

Hi Orne,
My implementation is almost ready and it seems to be working for most of the cases that I have tested it (automatically and with my end-device). I have noticed though that there is some strange error when I was trying to send a couple of mac commands, when the bridge told me that it can not marshal an mac command that I did not sent. When I dug a little bit, I found that it calls txPacket.PHYPayload.MarshalBinary that on its turn, when FPort == 0 tries to marshal the FRMPayload as MAC commands. This fails as I have already encrypted the FRMPayload (as afterwards I don't have the encryption key), so inside FRMPayload there is only 1 DataPayload - the encrypted data. The simplest/cleanest way I see to fix this is to send the PHYPayload directly in binary (possibly base64). Let me know if I did not explain clearly enough the issue or if you have a better suggestion how it should be fixed.

Thanks for your feedback, I think I know what is going wrong! I haven't started yet on the MAC commands yet. I'm currently doing some refactoring work on all three of the projects:

lorawan: getting rid of the uplink argument and implementing a band sub-package containing band specific configurations
loraserver: re-organizing the source so that vendoring becomes easier (no need to vendor the whole loraserver package)
lora-semtech-bridge: send payloads as json over mqtt instead of gob (will add some more visibility)

After that has been completed I'll look into your issue!

Here is my implementation of the mac commands. It still needs some work, but it works fine at least for some of the cases (new channel, duty cycle, dev status have been tested with an end-device). I think that I am not going to change much my implementation , so you can review it. I only want to add the possibility to send mac commands over FOpts  and I will make a pull request if all looks good to you. I might need to rework it in order to for it to match your changes in branch 0_5_dev.

Here is my implementation of the mac commands. It still needs some work, but it works fine at least for some of the cases (new channel, duty cycle, dev status have been tested with an end-device). I think that I am not going to change much my implementation , so you can review it. I only want to add the possibility to send mac commands over FOpts  and I will make a pull request if all looks good to you. I might need to rework it in order to for it to match your changes in branch 0_5_dev.

I've had a quick look at the diff: https://github.com/brocaar/loraserver/compare/master...Acklio:mac-commands?expand=1
First of all, thanks for your efforts!
However, the diff is huge. This makes it really hard to give good feedback and to shape it into a mergable state as it covers a lot of parts of the architecture. As well I'm a bit surprised by the amount of work that you already did. I would have preferred to discuss the right implementation and approach first.

Actually when I started implementing this, it didn't seem as that much work, but after some time I realized this is not going to be the case. Anyway, it was to some extend interesting to write a little bit more Go code, as I am not very experienced with this language so for sure it was useful for me even only in this regard.
For the architecture, I tried to be inspired by the existing code + my sense of good practices. Basically my idea was to have some entities that can be shown to network server administrators for managing mac commands and that those entities will provide a small interface for loraserver.go to transfer and receive mac command requests and answers. It seemed necessary to do a little bit of refactoring in order the code to remain understandable (which contributes to some of the changes in loraserver.go). There is now a send method that receives a reply macPL, rxPacket + others and constructs two txPackets and sends them to the gateway. There is a method that creates the macPL for particular data that needs to be sent. There is a method for handling mac commands and some other helper methods.
The other changes are maybe less significant - nodeSession can store custom attributes for data rates and delays for a particular end-device (loraserver.go had to be changed accordingly - the new send method), app.js knows how to retrieve and update the entities, nodes.html now has very ugly views for the actual visualization.
The more important decisions from my point of view that I had to take were the methods that loraserver.go will access for mac commands manipulations and how/what to keep. Right now I keep the last sent set of mac commands + possibly their replies (if I have them) + mac commands to be sent (right now saved separately and sent only if the last mac commands that were sent are not the same or if we have not received a reply for them).
I hope it will not be that difficult for me to change my modifications

So that it will meet your architectural vision for the project.
So that it will be mergeable on the master branch.
So that it will be easily usable (maybe not exactly the case right now - if you don't know the spec well enough).

So far I have tried to provide a global overview of the change + give some reasons why I implemented all that. Please feel free to provide more details how you would expect this feature to be implemented or to ask for more details, which will make a code review feasible (I do realize it is not easy to see the change and understand it). Also I would not mind at all if I need to make some additional changes (rebase the branch on the new master for example, refactor some things, etc).

I rebased the mac-commands branch on the current master (it seems to me that you have finished with the moving/restructuring). It is still a huge diff, so if you need more explanations, please don't hesitate to let me know :)

Thanks! I won't merge as-is, since I still have to think what the right flow is for each MAC command and I would like to break the implementation of MAC commands into pieces. However, I'll take it as an example and make modifications where needed. This weekend I'll attend IoT Olympics and will try to get some feedback from other people on this.

I've started working on the MAC commands implementation. My idea is that the network-controller will be a component that is not part of LoRa Server and that LoRa Server provides various mqtt topics for communication. Outline of my idea:
There will be a new NetworkController backend which is responsible for the following topics:
application/[AppEUI]/node/[DevEUI]/rxinfo
Is already implemented (in the ApplicationBackend) but this will be moved to the NetworkController backend.
application/[AppEUI]/node/[DevEUI]/mac/rx
Sends received MAC commands to the network-controller
application/[AppEUI]/node/[DevEUI]/mac/tx
Topic to which the network-controller sends MAC commands to be transmitted to the node
application/[appEUI]/node/[DevEUI]/mac/error
Error topic.
LoRa Server will just keep a queue of MAC commands to be transmitted, but does not implement any MAC command logic so it doesn't know what to do with each MAC command. Example flow:

network-controller sends a NewChannelReq to mac/tx
LoRa Server transmits the MAC command when there is a downlink receive window to the node
Node acks the MAC command
LoRa Server forwards the ack to the network-controller
The network controller then performs an RPC call to add the channel to the node-session

What do you think?

Hi Orne,
This generally seems as a good plan which allows a great deal of flexibility. I have some small concerns, but maybe they are due to my misunderstanding and in any case I don't think they are very serious. Here they are:

MAC commands in the queue might overwrite each other. Maybe there should be a way to detect this or at least to cancel a MAC command.
How are you going to handle LinkCheckReq? Through scheduling a reply (most-likely for some of the next uplinks)?
What is the priority of MAC commands in comparison to the data down and do you plan to have a way to say - I want this MAC command in the FRMPayload? Might be useful.
You might occasionally lose a message due to change in the configuration for data down that is not reflected on time (if I understood correctly that loraserver will not wait for a reply to MAC commands ack in order to proceed). This might arise if the RXDelay is changed or RX1DROffset is changed - the server will not send the data correctly in most of those cases.

All those questions could be resolved I think, I mention them here just to make sure they did not escape your attention. Otherwise this seems as a good way to go.
My 2c :)

Yes, however I think it is best that LoRa Server should not know anything about MAC commands. The same issue applies I think for application payloads. Currently there is no way to cancel out a payload that is still in the queue. I think to start with, I'll just implement a FIFO queue. Maybe this can be resolved through the RPC API (eg. the network-controller makes an RPC call to remove a MAC command from the queue)?

When receiving an uplink MAC command, I think it is best to directly use the downlink receive window to sent the MAC response (if the network-controller responds in time to LoRa Server).

MAC commands can be combined with application payloads when FOpts is used. In any other case I think it makes sense to give MAC commands priority over application payloads (to make sure the node stays connected to the network e.g. when using ADR)? LoRa Server can indicate to the node that it should open an other receive window by setting FPending=true in the downlink packet. Regarding FRMPayload, that option is already there in the struct 😄 https://github.com/brocaar/loraserver/blob/network_controller_backend/models/packets.go#L83

In case the network-controller is requesting for example a channel change, I think it must wait for an ack from the node before modifying the node-session properties. Most / all ack payloads indicate if for example the data rate range was ok, channel frequency was ok etc... See for example the NewChannelAns payload.

Something for after the initial MAC command implementation: I'm planning to add an option to the API and web-interface to add proprietary MAC commands (0x80 to 0xFF range). Since all MAC commands are base64 encoded in the LoraServer <-> network-controller communication, I think the only thing needed to realise this is to configure a list of CID + optional payload size, so that LoRa Server is able to split the MAC commands correctly (this will require some changes at the https://github.com/brocaar/lorawan package too, but it will be quite straightforward).

I still have not looked into your implementation, so I am replying only based on your comments and I will provide a second reply when I manage to take a look into the code.

Sounds good to me.
Ok, so loraserver should wait for some time before sending a reply (to give time for the network-controller to reply).
Sounds good :)
Yes, an acknowledgment is required, but my idea was if there is data that will be send to the end node on the downlink receive window right after an ACK was just received, then the loraserver need to wait for the RPC, otherwise the end node will have applied the change, but the loraserver will "notice" the change a little bit later.

Also the idea about proprietary MAC commands sounds very good to me :+1:

I reviewed quickly the changes in for the mac commands and it looks good to me. Good job :+1:

I've just tagged version 0.8.0 with MAC command support :-)
http://loraserver.readthedocs.io/en/latest/changelog/#080
http://loraserver.readthedocs.io/en/latest/network-controller/

Hi @brocaar ,
How did you implement the network-controller.md file please?
And in which file can i find how does my lora-gateway-bridge(or lora-server) do to receive a payload from a gateway?
Thank you in advance


Sighting Search Export

User stories
As a Researcher+exporter or administrator on the sighting search page, I can export the filtered results of the search page as a xls

the export will contain all default and custom sighting and encounter fields

Final design mockup: https://www.figma.com/file/67AYBU2OioLcLpEVdq7uEQ/Codex%3A-support-search-of-all-sighting%2Findividual-content?node-id=0-1&t=JVxfCugm3SBQfd7t-0

need to remove customField. prepend on custom fields

re-closing. going to address customField. prepend as part of separate effort


v21.2.7 optimize migrations

[Current coverage][cc-pull] is 100%

Merging [#545][cc-pull] into [master][cc-base-branch] will not change coverage

@@           master   #545   diff @@
====================================
  Files          28     28
  Lines         439    439
  Methods       111    111
  Messages        0      0
  Branches        0      0
====================================
  Hits          439    439
  Misses          0      0
  Partials        0      0

Powered by Codecov. Last updated by [46596ed...e208872][cc-compare]
[cc-base-branch]: https://codecov.io/gh/eugene-matvejev/battleship-game-api/branch/master?src=pr
[cc-compare]: https://codecov.io/gh/eugene-matvejev/battleship-game-api/compare/46596edfe2f51e0983e62506de986a362d3f1062...e208872d49578bfab1037058050b3c3144567b98?src=pr
[cc-pull]: https://codecov.io/gh/eugene-matvejev/battleship-game-api/pull/545?src=pr


Add usermeta top-level field, optional and ignored by parser.

Addresses https://github.com/vega/vega/issues/1061.
Looks like there are some docs to be updated in vega/vega; not clear if these are generated from the schema or manually updated.
Please let me know if there's anything else that needs to accompany this PR that I missed!
@domoritz please review.

Thanks! I've reviewed and will add schema validation support for usermeta separately, using a simpler approach.
However, this PR also includes usermeta definitions and documentation as a config option. Is including a default, global usermeta object desired as part of the Vega configuration? Currently it might be included as config but this would have no effect whatsoever -- it would not be accessible in any way.
My current understanding is that usermeta should be an optional part of the Vega JSON input but is otherwise ignored. If you want this metadata to some how propagate through (e.g., to be accessible from a view instance), that should be subject to further discussion on a separate issue.

Nope, adding to config was unintentional -- I was following the pattern of background and missed that background is in fact relevant to config. Per the earlier thread, usermeta should exist only at the top level of the spec and nowhere else. I don't see a need for accessing usermeta from within a View instance (though that may be worth floating on the original thread).


Create _export directory in torchtune

Summary: Creates a placeholder directory for adding exportable torchtune components
Reviewed By: larryliu0820
Differential Revision: D65982921

Hey @Jack-Khuu and @dvorjackz, just caught up on the context of this change. Would be awesome if you could detail in the PR summary the motivation for this folder (clean dependency graph for torchchat and executorch for exportable modules) and what the plan is for CI. From what I gathered, the CI will live in torchtune and code quality/updating the CI for _export is largely the responsibility of PyTorch Edge folks?
One concern I have is if an unrelated change in torchtune breaks exportability CI, is it our responsibility to fix this? Is there any way to separate exportability CI from our core CI?

Thanks for checking in, we've been talking with @ebsmothers and the design is in the internal Diff comments, so that it doesn't pop up in OSS. I can add an abridged version to the PR as well

From what I gathered, the CI will live in torchtune and code quality/updating the CI for _export is largely the responsibility of PyTorch Edge folks? One concern I have is if an unrelated change in torchtune breaks exportability CI, is it our responsibility to fix this? Is there any way to separate exportability CI from our core CI?

Correct, PyTorch Edge will own and update the code and CI for these modules. If tune has changes that breaks exportability, Edge will champion the fix (the CI for this will be non-blocking for torchtune)

Cool, thanks for the additional context. I can stamp this once the PR summary is updated.


Enhance slab reallocation for burst of evictions

Extension to the existing slab reallocation model to better cope with eviction patterns that appear in short timeframe bursts.
This means that if more than 25% of the total evictions in the window appear in this slab it becomes a candidate for
needing more pages.

thanks!


Add Octopus Deploy logotype

The logotype is available from the Octopus Deploy website.
https://octopus.com/company/media-resources

Added in d108849295d8d92cef1581142d8b012fe34a8e72


repeated download in utils_mnist.py

In line 31 of utils_mnist.py, os.path.isfile(file_name) is always equal to False, so the datasets downloaded from the Internet would not be saved to the local. Therefore, every time utils_mnist.py is ran, the datasets should be downloaded again.
This code didn't raise any error, but it would be better to avoid repeated download.
Thanks.

Good call. Closed by https://github.com/goodfeli/cleverhans/commit/6ae062639a7baddb1277b7b6bcbd22ca19b844eb


fix: validationTimeout configuration item invalid

It seems that the validationTimeout configured in the configuration file does not take effect. The SocketTimeout of the database connection will always overwrite this configuration item, even if this parameter is not specified in jdbcurl.
This can be verified with a simple springboot project.
application.yaml as follow
server:
  port: 8081
spring:
  application:
    name: hikariTest
  datasource:
    driver-class-name: com.mysql.cj.jdbc.Driver
#    url: jdbc:mysql://localhost/demo?useUnicode=true&characterEncoding=utf8&allowMultiQueries=true&useSSL=false&socketTimeout=100
    url: jdbc:mysql://localhost/demo?useUnicode=true&characterEncoding=utf8&allowMultiQueries=true&useSSL=false
    username: root
    password: root
    hikari:
      validationTimeout: 300

We can see from the following code that the value obtained by Connection#getNetworkTimeout is obviously inconsistent with the validationTimeout in the configuration item.
@SpringBootApplication
public class Main {
    public static void main(String[] args) throws SQLException, NoSuchFieldException, IllegalAccessException {
        final ConfigurableApplicationContext run = SpringApplication.run(Main.class, args);
        final Map<String, DataSource> beansOfType = run.getBeansOfType(DataSource.class);
        final HikariDataSource dataSource = (HikariDataSource)beansOfType.get("dataSource");
        final Connection connection = dataSource.getConnection();
        final int networkTimeout = connection.getNetworkTimeout();
        System.out.println("network timeout is [" + networkTimeout + "]");
        final Class<HikariConfig> hikariConfigClass = HikariConfig.class;
        final Field validationTimeout = hikariConfigClass.getDeclaredField("validationTimeout");
        validationTimeout.setAccessible(true);
        final long validationTimeoutValue = (long)validationTimeout.get(dataSource);
        System.out.println("now validationTimeout is [" + validationTimeoutValue + "]");
    }
}

results are as follows
network timeout is [0]
now validationTimeout is [300]

I think when the socketTimeout value is not specified in jdbcurl, the validationTimeout in the configuration file should be used as this value,instead of being replaced by the default value.
I think the value obtained by Connection#getNetworkTimeout should be consistent with the validationTimeout in the configuration file.
Perhaps the following result should be obtained
network timeout is [300]
now validationTimeout is [300]

By reading the source code, I found that the connection is created through PoolBase#newConnection and the SocketTimeout is set through PoolBase#setupConnection, but the validationTimeout is not set correctly in the setupConnection method. If we configure the validationTimeout.
method as follow
private void setupConnection(final Connection connection) throws ConnectionSetupException
   {
      try {
         if (networkTimeout == UNINITIALIZED) {
            networkTimeout = getAndSetNetworkTimeout(connection, validationTimeout);
         }
         else {
            setNetworkTimeout(connection, validationTimeout);
         }

         if (connection.isReadOnly() != isReadOnly) {
            connection.setReadOnly(isReadOnly);
         }

         if (connection.getAutoCommit() != isAutoCommit) {
            connection.setAutoCommit(isAutoCommit);
         }

         checkDriverSupport(connection);

         if (transactionIsolation != defaultTransactionIsolation) {
            connection.setTransactionIsolation(transactionIsolation);
         }

         if (catalog != null) {
            connection.setCatalog(catalog);
         }

         if (schema != null) {
            connection.setSchema(schema);
         }

         executeSql(connection, config.getConnectionInitSql(), true);
         // must be executed
         setNetworkTimeout(connection, networkTimeout);
      }
      catch (SQLException e) {
         throw new ConnectionSetupException(e);
      }
   }

Because the following line of code will be executed, but networkTimeout has been assigned to dataSource.getConnection().getNetworkTimeout() during initialization.
setNetworkTimeout(connection, networkTimeout);

The Connection of the above method is obtained in newConnection.
private Connection newConnection() throws Exception
   {
      final var start = currentTime();

      Connection connection = null;
      try {
         final var credentials = config.getCredentials();
         final var username = credentials.getUsername();
         final var password = credentials.getPassword();
         // get connection
         connection = (username == null) ? dataSource.getConnection() : dataSource.getConnection(username, password);
         if (connection == null) {
            throw new SQLTransientConnectionException("DataSource returned null unexpectedly");
         }

         setupConnection(connection);
         lastConnectionFailure.set(null);
         return connection;
      }
      catch (Exception e) {
         if (connection != null) {
            quietlyCloseConnection(connection, "(Failed to create/setup connection)");
         }
         else if (getLastConnectionFailure() == null) {
            logger.debug("{} - Failed to create/setup connection: {}", poolName, e.getMessage());
         }

         lastConnectionFailure.set(e);
         throw e;
      }
      finally {
         // tracker will be null during failFast check
         if (metricsTracker != null) {
            metricsTracker.recordConnectionCreated(elapsedMillis(start));
         }
      }
   }

networkTimeout will be assigned in the following method
private void setupConnection(final Connection connection) throws ConnectionSetupException
   {
      try {
         if (networkTimeout == UNINITIALIZED) {
            networkTimeout = getAndSetNetworkTimeout(connection, validationTimeout);
         }
     // ...
    }

private int getAndSetNetworkTimeout(final Connection connection, final long timeoutMs)
   {
      if (isNetworkTimeoutSupported != FALSE) {
         try {
            final var originalTimeout = connection.getNetworkTimeout();
            connection.setNetworkTimeout(netTimeoutExecutor, (int) timeoutMs);
            isNetworkTimeoutSupported = TRUE;
            if (originalTimeout == 0) {
               return (int) timeoutMs;
            }
            return originalTimeout;
         }
         catch (Exception | AbstractMethodError e) {
            if (isNetworkTimeoutSupported == UNINITIALIZED) {
               isNetworkTimeoutSupported = FALSE;

               logger.info("{} - Driver does not support get/set network timeout for connections. ({})", poolName, e.getMessage());
               if (validationTimeout < SECONDS.toMillis(1)) {
                  logger.warn("{} - A validationTimeout of less than 1 second cannot be honored on drivers without setNetworkTimeout() support.", poolName);
               }
               else if (validationTimeout % SECONDS.toMillis(1) != 0) {
                  logger.warn("{} - A validationTimeout with fractional second granularity cannot be honored on drivers without setNetworkTimeout() support.", poolName);
               }
            }
         }
      }

      return 0;
   }

Finally, the value of originalTimeout will be returned, and originalTimeout is directly obtained through dataSource.getConnection().getNetworkTimeout().
That's why validationTimeout is not set correctly in connection.
I'm not sure if this is correct.
If what I said above is not correct, please give me a correct example of setting validationTimeout through the configuration file. Thank you for your time

results are as follows
network timeout is [0]
now validationTimeout is [300]

I think it's correct because validationTimeout is only for validating connection not for default network timeout, maybe I didn't get your point because I didn't read entire post, it's too long.

Sorry, I misunderstood the meaning of validationTimeout, but it seems that hikari does not provide a method to configure the SocketTimeout of the Connection through configuration items. Can you tell me why?

Sorry, I misunderstood the meaning of validationTimeout, but it seems that hikari does not provide a method to configure the SocketTimeout of the Connection through configuration items. Can you tell me why?

I think it's due to not every JDBC driver implements Connection::setNetworkTimeout correctly.


Headless mode improvements

In 2.4.0 there is a great new feature 'set_hide', wich let us use webui for background site testing.
We use webui in headless mode to test pages from other languages, while in order to load the pages themselves, we have to use tricks such as this one:
<script>
function loadData() {
  fetch("linktosomehost.com")
    .then(response => response.text())
    .then(content => {
      document.open("text/html");
      document.write(content);
      document.close();

      const scriptElement = document.createElement("script");
      scriptElement.src = "webui.js";
      document.body.appendChild(scriptElement);
    })
    .catch(error => {
      console.error(error);
    });
}
window.onload = loadData;
</script>

to let webui to load the necessary library code on the browser side to serve events and websocket connection it can be done by dom manipulating.
I certainly understand the true purpose of the library, but since the possibility of working in headless mode has already been implemented, maybe we should think about a more efficient way of working in this mode?
I would like to hear opinions and ideas in this direction.
And this is actual, since webui.js auto inject is removed

fetch() can be blocked by CORS. And bypass it using proxy needs authentication. Using --disable-web-security solves it but it shows a warning message on startup. I suggest using JavaScript like you did for those specific scenarios.

WebUI has control only over the content provided by itself (by CivetWeb http/websocket server).
You can kind of control DOM of other websites with tricks like your your JS, on some sites it will work, but some sites will defend (will you bypass Cloudflare check?). Your URL will always point to localhost, your port will be always different that 80 or 443.
CORS
You can modify your JS to not fetch('http://website') but fetch('/internal'), set up file handler, and on request to /internal download the site you want and return it. CORS will be bypassed.
You can disable CORS using this quick hack if you dont want to modify the lib:
webui_set_profile(window, 'profile', 'profie" --disable-web-security --something "');
File handler
After you set file handler, you will need to download all requests and return them, like a proxy.
Other possibilities
I can see more ways it can be achieved, but these are all hacks and wont be supported by the lib.
eg

start Chromium with a profile that has Tampermonkey installed and configured to include webui.js on every website
maybe there is command line switch to auto load custom extension?

But how to inject the script if you dont know the URL? Its dynamic, on localhost, that you will need to figure out too.
To sum up
WebUI has control only over the content it provided. Its not a web driver.


Add ping-loop plugin

Additionally, seabird should properly shut down after a ping timeout. Error handling will have to be reworked a bit for this to be possible.

Relevant:
https://github.com/jsvana/async-minecraft-ping

@belak do you just mean a plugin to respond to IRC server pings? @NickLanam thanks for the plug :) I'm not sure that this is the same ping, though.

Sort of. We already respond to pings. This would be to send our own pings.

Pings to what? Sorry, not following :)

We need to send irc server pings to make sure the connection is still open

IRC pings are way too tied to the protocol - I think this is a hold over from when this was just an irc bot.

IRC pings are way too tied to the protocol - I think this is a hold over from when this was just an irc bot.


Declare additional namespace prefixes to be used with XPath

Fixes #99
This allows to declare additional namespace prefixes that are not already declared in the document. This is useful to be write XPath expressions without knowing how the document was authored.
Several prefixes can be used for the same namespace, so a node can be queried both using the prefix declared in the document, and the one defined on XMLDocument.
(PS: The unit tests are still working with those changes)

Again thank you so much for helping to make this library better!


Confusing XXS documentation

The XXS documentation says that "DOM interfaces are generally safe to use".
What does generally mean here? Is it safe to use or not? If not, it would be nice if the documentation could explicitly clarify the potential risks and attack vectors when using linkifyjs. Like, What are the potential gotchas? What should I be aware of? What do you recommend?
As someone who is paranoid about security, the wording provokes a fear in me that deters me from using the library.

Hi @Torsteinws, I'm not a web security expert, so I don't make any promises about how "safe" Linkify is to use. The Cross-site scripting page is meant to call out the one obvious potential danger of converting user-provided text input to HTML and outputting it to the page without validation. This is largely left as the end programmer's responsibility, not Linkify's (as stated in the notice at the top).
"Generally" on that page refers to Linkify use cases that, as far as I know, will NOT lead to an XSS attack. However, this does not mean that use case is always safe. For example, a hacker may provide a valid but malicious URL that will hijack your computer if you click it.
I can write something to that effect on that page. Any other suggestions are welcome!

In my research on the subject, I found the https://www.npmjs.com/package/sanitize-html package.


Schema check throwing error in server and CLI if everything is fine after update to latest

Hey,
yesterday, I have done an upgrade to latest hive (self-hosting) (we were ~9 months back). Since then our schema checks and schema publishes fail with:
Failed to check schema
 »   Error: No access (reason: "Authorization token is missing")

The log is showing:
Flag "startTransaction" is disabled but Sentry failed to find a transaction.
Try to create a transaction before GraphQL execution phase is started.
{"level":20,"time":1726219430756,"pid":7,"hostname":"server-867dfdf89f-c9dzd","reqId":"req-7u","msg":"Resolving user"}
{"level":20,"time":1726219430756,"pid":7,"hostname":"server-867dfdf89f-c9dzd","reqId":"req-7u","msg":"Session resolution ended successfully"}
{"level":20,"time":1726219430756,"pid":7,"hostname":"server-867dfdf89f-c9dzd","reqId":"req-7u","msg":"No session found"}
{"level":20,"time":1726219430756,"pid":7,"hostname":"server-867dfdf89f-c9dzd","reqId":"req-7u","msg":"Attempt to resolve an API token from headers"}
{"level":20,"time":1726219430756,"pid":7,"hostname":"server-867dfdf89f-c9dzd","reqId":"req-7u","msg":"Found Authorization header (length=32, token=33f**************************ab7)"}
{"level":20,"time":1726219430757,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Fetching token (token=33f**************************ab7) - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":30,"time":1726219430776,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Checking schema (input={\"service\":\"gateway\",\"github\":null,\"meta\":{\"author\":\"***\",\"commit\":\"a00f33ed931eda3e45b1d6f9e30801a2c95b5105\"},\"organization\":\"0dac3370-08f8-4f10-a949-8ba046cb4235\",\"project\":\"ab6b1c11-e38b-4390-86de-51986b2168f1\",\"target\":\"0322d06b-9941-47eb-9de9-6d5b084ec626\"}) - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219430776,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Fetching token (token=33f**************************ab7) - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219430782,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Fetching target (selector={\"organization\":\"0dac3370-08f8-4f10-a949-8ba046cb4235\",\"project\":\"ab6b1c11-e38b-4390-86de-51986b2168f1\",\"target\":\"0322d06b-9941-47eb-9de9-6d5b084ec626\"}) - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219430782,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Fetching project (selector={\"organization\":\"0dac3370-08f8-4f10-a949-8ba046cb4235\",\"project\":\"ab6b1c11-e38b-4390-86de-51986b2168f1\"}) - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219430782,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Fetching organization (selector={\"organization\":\"0dac3370-08f8-4f10-a949-8ba046cb4235\"}) - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219430782,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Fetching latest schemas (selector={\"organization\":\"0dac3370-08f8-4f10-a949-8ba046cb4235\",\"project\":\"ab6b1c11-e38b-4390-86de-51986b2168f1\",\"target\":\"0322d06b-9941-47eb-9de9-6d5b084ec626\"}) - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219430782,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Fetching latest schemas (selector={\"organization\":\"0dac3370-08f8-4f10-a949-8ba046cb4235\",\"project\":\"ab6b1c11-e38b-4390-86de-51986b2168f1\",\"target\":\"0322d06b-9941-47eb-9de9-6d5b084ec626\",\"onlyComposable\":true}) - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219430782,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Fetching maybe latest version (selector={\"organization\":\"0dac3370-08f8-4f10-a949-8ba046cb4235\",\"project\":\"ab6b1c11-e38b-4390-86de-51986b2168f1\",\"target\":\"0322d06b-9941-47eb-9de9-6d5b084ec626\"}) - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219430782,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Fetching organization (selector={\"organization\":\"0dac3370-08f8-4f10-a949-8ba046cb4235\"}) - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219430782,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Fetching latest schemas (selector={\"organization\":\"0dac3370-08f8-4f10-a949-8ba046cb4235\",\"project\":\"ab6b1c11-e38b-4390-86de-51986b2168f1\",\"target\":\"0322d06b-9941-47eb-9de9-6d5b084ec626\"}) - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219430782,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Fetching latest schemas (selector={\"organization\":\"0dac3370-08f8-4f10-a949-8ba046cb4235\",\"project\":\"ab6b1c11-e38b-4390-86de-51986b2168f1\",\"target\":\"0322d06b-9941-47eb-9de9-6d5b084ec626\",\"onlyComposable\":true}) - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219430782,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Fetching maybe latest version (selector={\"organization\":\"0dac3370-08f8-4f10-a949-8ba046cb4235\",\"project\":\"ab6b1c11-e38b-4390-86de-51986b2168f1\",\"target\":\"0322d06b-9941-47eb-9de9-6d5b084ec626\"}) - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219430782,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Fetching maybe latest valid version (selector={\"organization\":\"0dac3370-08f8-4f10-a949-8ba046cb4235\",\"project\":\"ab6b1c11-e38b-4390-86de-51986b2168f1\",\"target\":\"0322d06b-9941-47eb-9de9-6d5b084ec626\"}) - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219430843,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Fetching base schema (selector={\"organization\":\"0dac3370-08f8-4f10-a949-8ba046cb4235\",\"project\":\"ab6b1c11-e38b-4390-86de-51986b2168f1\",\"target\":\"0322d06b-9941-47eb-9de9-6d5b084ec626\"}) - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219431002,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Executing ClickHouse Query (executionId: getTotalCountForSchemaCoordinates-a979446068cb6):  SELECT SUM(\"result\".\"total\") AS \"amountOfRequests\" FROM ( SELECT SUM(\"operations_daily\".\"total\") AS \"total\" FROM \"operations_daily\" PREWHERE \"operations_daily\".\"target\" IN (['0322d06b-9941-47eb-9de9-6d5b084ec626', '3c71f0f8-c0f8-462d-9535-64afa32e45b3', '84de09df-6771-46e7-ba9b-4a42b65d8339']) AND \"operations_daily\".\"timestamp\" >= toDateTime('2024-08-14 09:23:51', 'UTC') AND \"operations_daily\".\"timestamp\" <= toDateTime('2024-09-13 09:23:51', 'UTC') AND \"operations_daily\".\"client_name\" NOT IN (['integration-tests']) UNION ALL SELECT SUM(\"subscription_operations_daily\".\"total\") AS \"total\" FROM \"subscription_operations_daily\" PREWHERE \"subscription_operations_daily\".\"target\" IN (['0322d06b-9941-47eb-9de9-6d5b084ec626', '3c71f0f8-c0f8-462d-9535-64afa32e45b3', '84de09df-6771-46e7-ba9b-4a42b65d8339']) AND \"subscription_operations_daily\".\"timestamp\" >= toDateTime('2024-08-14 09:23:51', 'UTC') AND \"subscription_operations_daily\".\"timestamp\" <= toDateTime('2024-09-13 09:23:51', 'UTC') AND \"subscription_operations_daily\".\"client_name\" NOT IN (['integration-tests']) ) AS \"result\"  - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219431028,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Finished ClickHouse Query (executionId: getTotalCountForSchemaCoordinates-a979446068cb6, duration=26ms) - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219431028,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Load contract versions for schema version. (schemaVersionId=07eb2984-e21f-4632-849b-6db355d553ab) - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219431032,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"No contract versions found for schema version. (schemaVersionId=07eb2984-e21f-4632-849b-6db355d553ab) - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219431032,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Using STITCHING registry model (version=modern) - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219431032,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Checksum check (existingSchema=yes, existingContractCount=null, incomingContractCount=null) - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219431298,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Schema is modified. - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219431298,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Orchestrator: [object Object] - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219431301,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Composing and Validating Stitched Schemas - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219431872,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"No validation errors - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219431872,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Retrieve previous version SDL. - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219431872,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Return pre-computed SDL. - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":30,"time":1726219431872,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Contract checks: null - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219431982,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Comparing Schemas - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219431999,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Conditional breaking change settings available. - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219431999,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Fetching affected operations and affected clients for breaking changes. - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219432021,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Detected non-breaking changes - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":30,"time":1726219432025,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"diff check status: {\"status\":\"completed\",\"result\":{\"breaking\":null,\"safe\":[{\"id\":\"cd5817edca74ddb2a31e7859647abd14\",\"type\":\"FIELD_ARGUMENT_DESCRIPTION_CHANGED\",\"approvalMetadata\":null,\"criticality\":\"NON_BREAKING\",\"message\":\"Description for argument 'numTypos' on field 'Query.search' changed from 'Number of allowed typos in a field. A single value acts upon all fields, otherwise respects the order of queryBy. Using a negative number defaults this entry. Equivalent to num_typos.' to 'Number of allowed typos in a field. A single value acts upon all fields, otherwise respects the order of queryBy. Using a negative number defaults this entry. Equivalent to num_typo.'\",\"meta\":{\"typeName\":\"Query\",\"fieldName\":\"search\",\"argumentName\":\"numTypos\",\"oldDescription\":\"Number of allowed typos in a field. A single value acts upon all fields, otherwise respects the order of queryBy. Using a negative number defaults this entry. Equivalent to num_typos.\",\"newDescription\":\"Number of allowed typos in a field. A single value acts upon all fields, otherwise respects the order of queryBy. Using a negative number defaults this entry. Equivalent to num_typo.\"},\"path\":\"Query.search.numTypos\",\"isSafeBasedOnUsage\":false,\"reason\":null,\"usageStatistics\":null,\"breakingChangeSchemaCoordinate\":null}],\"all\":[{\"id\":\"cd5817edca74ddb2a31e7859647abd14\",\"type\":\"FIELD_ARGUMENT_DESCRIPTION_CHANGED\",\"approvalMetadata\":null,\"criticality\":\"NON_BREAKING\",\"message\":\"Description for argument 'numTypos' on field 'Query.search' changed from 'Number of allowed typos in a field. A single value acts upon all fields, otherwise respects the order of queryBy. Using a negative number defaults this entry. Equivalent to num_typos.' to 'Number of allowed typos in a field. A single value acts upon all fields, otherwise respects the order of queryBy. Using a negative number defaults this entry. Equivalent to num_typo.'\",\"meta\":{\"typeName\":\"Query\",\"fieldName\":\"search\",\"argumentName\":\"numTypos\",\"oldDescription\":\"Number of allowed typos in a field. A single value acts upon all fields, otherwise respects the order of queryBy. Using a negative number defaults this entry. Equivalent to num_typos.\",\"newDescription\":\"Number of allowed typos in a field. A single value acts upon all fields, otherwise respects the order of queryBy. Using a negative number defaults this entry. Equivalent to num_typo.\"},\"path\":\"Query.search.numTypos\",\"isSafeBasedOnUsage\":false,\"reason\":null,\"usageStatistics\":null,\"breakingChangeSchemaCoordinate\":null}],\"coordinatesDiff\":{\"added\":{},\"deleted\":{},\"deprecated\":{},\"undeprecated\":{}}}} - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":30,"time":1726219432025,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"policy check status: {\"status\":\"skipped\"} - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219432025,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Schema check successful - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219432028,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Translating Organization Clean ID (selector={\"organization\":\"***\"}) - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":20,"time":1726219432028,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Translating Project Clean ID (selector={\"organization\":\"***\",\"project\":\"***\"}) - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
GraphQLError: No access (reason: "Authorization token is missing")
    at new AccessError (file:///usr/src/app/@hive/server/index.js:287289:5)
    at file:///usr/src/app/@hive/server/index.js:289370:15
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async schemaCheck (file:///usr/src/app/@hive/server/index.js:341795:29)
    at async file:///usr/src/app/@hive/server/index.js:269877:16
    at async YogaServer.getResultForParams (file:///usr/src/app/@hive/server/index.js:275312:18)
    at async handle (file:///usr/src/app/@hive/server/index.js:275388:20)
    at async file:///usr/src/app/@hive/server/index.js:360052:24
    at async Object.<anonymous> (file:///usr/src/app/@hive/server/index.js:360049:5) {
  path: [ 'schemaCheck' ],
{"level":20,"time":1726219432381,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"[useResponseCache] Decided not to cache the response because it contains errors - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
  locations: [ { line: 2, column: 3 } ],
  extensions: { code: 'UNAUTHENTICATED' }
}
AccessError [GraphQLError]: No access (reason: "Authorization token is missing")
    at new AccessError (file:///usr/src/app/@hive/server/index.js:287289:5)
    at file:///usr/src/app/@hive/server/index.js:289370:15
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async schemaCheck (file:///usr/src/app/@hive/server/index.js:341795:29)
    at async file:///usr/src/app/@hive/server/index.js:269877:16
    at async YogaServer.getResultForParams (file:///usr/src/app/@hive/server/index.js:275312:18)
    at async handle (file:///usr/src/app/@hive/server/index.js:275388:20)
    at async file:///usr/src/app/@hive/server/index.js:360052:24
    at async Object.<anonymous> (file:///usr/src/app/@hive/server/index.js:360049:5) {
  path: undefined,
  locations: undefined,
  extensions: { code: 'UNAUTHENTICATED' }
}
{"level":20,"time":1726219432381,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"Processing GraphQL Parameters done. - (requestId=wpAMOCe9T++6HsFpKZBnkA/0000000164)"}
{"level":30,"time":1726219432382,"pid":7,"hostname":"server-867dfdf89f-c9dzd","msg":"[200] (<IP_ADDRESS>) POST /graphql"}

I am very confused, as it looks like a schema check is executed. The schema check is also visible in HIVE:

Do you have any ideas what is causing this? As this is currently breaking our pipelines, as we do schema checks and publishes from cli tool.
Another hint:
If the schema check fails, I get a correct error response:

@PlayAnyData did you manage to workaround this? maybe with the latest version of Hive?
Also, what version of the Hive CLI/agent are you using?

@dotansimha Just wanted to upgrade to main, but cannot fetch the manifests anymore, e.g. https://ghcr.io/v2/kamilkisiela/graphql-hive/app/manifests/8b780f9f8e4a77048b7fb09bbc520b37f57d8816 - I get an unauthenticated error.
I will check tomorrow, which version, we are using for CLI. The problem with latest at that point was, that it did not respect self-signed certificates at that point in time and NODE_TLS_REJECT_UNAUTHORIZED was also not taken into account.


Forward signal to process

Command doesn't forward signals to Horizon process to allow Horizon to shut down properly.
Fixes #16

Still not working for me. Same behaviour. On each change, new worker is started and the previous one is left running.

@sdapkus does the process that's left running eventually die?

@sdapkus I've added a wait call to see if that helps. If those processes eventually die, then maybe there's just a delay.
What version of PHP are you using?

I'm getting same error in both commits that were made on Sep 26, 2023
sail artisan horizon:watch

   INFO  Starting Horizon and will restart it when any files change...

PHP Fatal error:  Error installing signal handler for 9 in /var/www/html/vendor/symfony/console/SignalRegistry/SignalRegistry.php on line 37

   Symfony\Component\ErrorHandler\Error\FatalError

  Error installing signal handler for 9

  at vendor/symfony/console/SignalRegistry/SignalRegistry.php:37
     33▕         }
     34▕
     35▕         $this->signalHandlers[$signal][] = $signalHandler;
     36▕
  ➜  37▕         pcntl_signal($signal, $this->handle(...));
     38▕     }
     39▕
     40▕     public static function isSupported(): bool
     41▕     {

   Whoops\Exception\ErrorException

  Error installing signal handler for 9

  at vendor/symfony/console/SignalRegistry/SignalRegistry.php:37
     33▕         }
     34▕
     35▕         $this->signalHandlers[$signal][] = $signalHandler;
     36▕
  ➜  37▕         pcntl_signal($signal, $this->handle(...));
     38▕     }
     39▕
     40▕     public static function isSupported(): bool
     41▕     {

      +1 vendor frames

  2   [internal]:0
      Whoops\Run::handleShutdown()

PHP Fatal error:  Error installing signal handler for 9 in Unknown on line 0

I'm trying this in Laravel sail, here is PHP version:
$ php -v
PHP 8.2.10 (cli) (built: Sep  2 2023 06:59:22) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.2.10, Copyright (c) Zend Technologies
    with Zend OPcache v8.2.10, Copyright (c), by Zend Technologies
    with Xdebug v3.2.1, Copyright (c) 2002-2023, by Derick Rethans

On first commit, I've tried waiting for about two minutes after reload, worker from first horizon instance was still running. Maybe process status output will be helpful, you can see that multiple supervisors are running. I've did three file changes.
sail@cbeea79f3596:/var/www/html$ ps -aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  1.0  32240 25720 ?        Ss   14:46   0:01 /usr/bin/python3 /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf
sail        16  0.1  2.8<PHONE_NUMBER>2 ?        S    14:46   0:01 /usr/bin/php -d variables_order=EGPCS /var/www/html/artisan serve --host=<IP_ADDRESS> --port=80
sail        19  0.0  1.7<PHONE_NUMBER>2 ?         Unfortunately, I am not reproducing the same issue. I am also not using this package directly anymore, so I'm not sure I can help much more here.
I would recommend adding debugging to this command to see what you can learn about how processes are managed in your situation and try some other approaches to fixing this issue. Alternatively, you could revert these changes and use the node-based solution that was present before this PR was merged.

@freekmurze FWIW, the changes in this PR will improve the existing solution. However, you may choose to revert #16 instead.

@sdapkus that's right. that was my observation too, that the signal update I made is to kill the child process when you kill the whole thing. I don't know why calling stop is not actually stopping your processes on restart. You could try adding output in the Process class in your vendor folder to see what the process status is or if something else is going on.
The wait that I added ought to wait for that first process to die since that's what Process::run(...) uses to wait for the process to die too.
You can also test this outside of Sail to see if it's specific to the Docker environment.

But that's the problem, you added trap on horizon:watch, but not on horizon command that is called by watch command.


Error 400

What issue do you have? Please be as thorough and explicit as possible.
Im using the Homebridge-Govee, but im getting the error 400 when using my TV-Strip #6104 over 39%.
It seems like 39% in HomeKit is 100% in Govee - so when I go over 39% it just gives up.
Details of your setup.

Do you use Homebridge (with config-ui?) or HOOBS?
Im using Homebridge - installed directly and NOT through HOOBS

Which version of Homebridge/HOOBS do you have?

To be honest, im not sure what version it is other than its through MacBook Pro.

Which version of this plugin (homebridge-govee) do you have? Has the issue started since upgrading from a previous version?
I just downloaded and installed it - so it would be the latest id think - homebridge-govee v2.2.0

Which Govee devices do you have that are causing issues? Please include product models if applicable.
#4106 - tv strip with camera (Govee ofcourse)

Please paste any relevant logs below.

[12/1/2020, 3:28:02 PM] [Govee] [TV Baggrund] could not be updated and its status will revert soon - Request failed with status code 400.
I hope you have enough to help me. I would really like for it to match the 100% . Other than that, great work!! I can't believe I FINALLY managed to Govee into other places than just Alexa!!

Hi @bauerx-hub please update to v2.2.1 and this should fix the brightness issue :)
Thanks for this report.

Hey!
Thanks for the quick reply!
When im on the homebridge it doesn't look like I can update it? Is there a way to force the update?

Sorry if im asking obvious questions. Im kind of a noob to homebridge :D

A button to update should appear on the Plugins page of the UI:

There is a command to force an update but it differs depending on the platform you are running Homebridge on.
Probably worth waiting for the update to appear in Homebridge, it might take a few minutes

Hi Again!
The update button didn't show up for some reason - I did manage to use the time-stamp on the right side, and just choose the latest version instead of going back!
It works all great now!! Thank you so much.
Ill be donating so you can buy yourself a beer my great man. This is awesome! Good work!

Ah excellent glad it's working for you :) thank you


Argument of '#selector' refers to instance method that is not exposed to Objective-C

Argument of '#selector' refers to instance method 'orientationDidChange()' that is not exposed to Objective-C
Argument of '#selector' refers to instance method 'displayTimerDidFire()' that is not exposed to Objective-C

Add '@objc' to expose this instance method to Objective-C

Still gettin the same error.

Try: pod 'InAppNotify', git: 'https://github.com/lucabecchetti/InAppNotify.git', commit: 'HEAD'
Please update podspec.


Adding EC2 Reboot Action

Per customer request I am adding the reboot ec2 action and placebo tests

Ready for Review and approval.  Thanks! 👍

Looks like its failing tests after I merged the master branch in.  Someone changed a ipv4 test which is breaking the travis ci py3.6 tests

@jtroberts83 ipv4 fixes are coming in #1788, the tox cache was removed when updating to the latest version of botocore which resulted in the error


BroomBooster crafting recipe

Players can't craft the broom booster after the last update.

The recipe did change, players should learn it if they have a broomstick. Are you sure they're trying the new one?
https://i.imgur.com/lX5m9tC.png
And please let me know what server version you're on, I couldn't test them all so it's possible something is broken pre-1.17

They are using the new one. It seems to be only working for me when i'm in creative

The recipe did change, players should learn it if they have a broomstick. Are you sure they're trying the new one?
https://i.imgur.com/lX5m9tC.png
And please let me know what server version you're on, I couldn't test them all so it's possible something is broken pre-1.17

I am not sure then, the recipe changes in 10.0 removed all of the custom crafting ingredients that generally caused issues like this. All the ingredients are vanilla now and so should "just work". Creative mode shouldn't matter at all unless you've got something else going on plugin-wise.
You could try changing the recipe to use some different items, see if that helps 🤷♂
/meditor recipe broom_booster

I took the fireworks out of this recipe, they were causing issues.