DistributionStrategy in tf.keras doesn't support multi input models

Please go to Stack Overflow for help and support:
https://stackoverflow.com/questions/tagged/tensorflow
If you open a GitHub issue, here is our policy:

It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).
The form below must be filled out.
It shouldn't be a TensorBoard issue. Those go here.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Debian and macOS 10.13.6
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): 1.11.0.dev20180805
Python version: 3.6 and 3.5
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:
Exact command to reproduce: See below

You can collect some of this information using our environment capture script:
https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh
You can obtain the TensorFlow version with
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
Describe the problem
@anj-s Huge thanks for adding support for distribution strategies to keras in 92279f8bfa6ce2124439aabfa6db84d722dc2b66 🎉
While using it together with single input models works great, it fails with an uncaught exception when using it with multiple inputs:
Source code / logs
Consider the following unit tests:
import numpy as np

from tensorflow.contrib.distribute.python import mirrored_strategy
from tensorflow.python import keras
from tensorflow.python.data.ops import dataset_ops
from tensorflow.python.platform import test
from tensorflow.python.training import gradient_descent

class TrainingTest(test.TestCase):
  def test_dataset_input_tuples(self):
    with self.test_session():
      a = keras.layers.Input(shape=(3,), name='input_a')
      b = keras.layers.Input(shape=(4,), name='input_b')
      x = keras.layers.concatenate([a, b])
      y = keras.layers.Dense(5, name='dense')(x)

      model = keras.Model(inputs=[a, b], outputs=[y])
      model.compile(loss='mse', metrics=['mae'], optimizer='rmsprop')

      inputs_a = np.zeros((10, 3))
      inputs_b = np.zeros((10, 4))
      targets = np.zeros((10, 5))
      dataset = dataset_ops.Dataset.from_tensor_slices(((inputs_a,
                                                         inputs_b),
                                                        targets))
      dataset = dataset.repeat(100)
      dataset = dataset.batch(10)

      model.fit(dataset, epochs=1, steps_per_epoch=2, verbose=1)

  def test_distributed_dataset_input_tuples(self):
    with self.test_session():
      a = keras.layers.Input(shape=(3,), name='input_a')
      b = keras.layers.Input(shape=(4,), name='input_b')
      x = keras.layers.concatenate([a, b])
      y = keras.layers.Dense(5, name='dense')(x)
      model = keras.Model(inputs=[a, b], outputs=[y])

      optimizer = gradient_descent.GradientDescentOptimizer(0.001)
      strategy = mirrored_strategy.MirroredStrategy(['/device:GPU:1',
                                                     '/device:CPU:0'])

      model.compile(loss='mse',
                    metrics=['mae'],
                    optimizer=optimizer,
                    distribute=strategy)

      inputs_a = np.zeros((10, 3))
      inputs_b = np.zeros((10, 4))
      targets = np.zeros((10, 5))
      dataset = dataset_ops.Dataset.from_tensor_slices(((inputs_a,
                                                         inputs_b),
                                                        targets))
      dataset = dataset.repeat(100)
      dataset = dataset.batch(10)

      model.fit(dataset, epochs=1, steps_per_epoch=2, verbose=1)

if __name__ == '__main__':
  test.main()

While the first test works well, the second will fail with the following error:
Epoch 1/1
2/2 [==============================] - 0s 55ms/step - loss: 0.0000e+00 - mean_absolute_error: 0.0000e+00
.WARNING:tensorflow:You are accessing attribute optimizerof theDistributedCallbackModel that may not have been setcorrectly.
/usr/local/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  return _inspect.getargspec(target)
E.
======================================================================
ERROR: test_distributed_dataset_input_tuples (__main__.TrainingTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tf_test.py", line 57, in test_distributed_dataset_input_tuples
    model.fit(dataset, epochs=1, steps_per_epoch=2, verbose=1)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1366, in fit
    validation_split=validation_split)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 794, in _standardize_user_data
    validation_split=validation_split)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 713, in _distribution_standardize_user_data
    validate_distributed_dataset_inputs(self._distribution_strategy, x, y)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/keras/engine/distributed_training_utils.py", line 212, in validate_distributed_dataset_inputs
    ' are of type {}'.format(type(x)))
ValueError: Dataset input to the model should be tensors instead they are of type <class 'tuple'>

----------------------------------------------------------------------
Ran 3 tests in 0.914s

FAILED (errors=1)

The same is true when using a dictionary as an input.
Since I'm new to using distribution strategies, I'm unsure if this is just a problem with too strict input validation (as in #20753) or if this needs more work to support this use case.
Thanks for the great work!

Thank you lgeiger@ for filing the issue! This error was raised due to support not being added for multiple inputs and targets. I am working on a fix and will update this issue as soon as the fix is in.
You mentioned running into this issue when using dictionary as input. Currently DistributionStrategy requires a Dataset object as input to fit, evaluate and predict. Can you clarify how you were passing a dictionary? Thanks!

@anj-s Awesome to hear that you're working on this.
Sorry for the confusion. I was using a Dataset with a dict instead of a tuple for the first input.
Here is a updated test case, that covers the behavior:
from absl.testing import parameterized
import numpy as np

from tensorflow.contrib.distribute.python import mirrored_strategy
from tensorflow.python import keras
from tensorflow.python.data.ops import dataset_ops
from tensorflow.python.platform import test
from tensorflow.python.training import gradient_descent

class TrainingTest(test.TestCase, parameterized.TestCase):

  @parameterized.named_parameters(
      ('tuple', lambda *x: x, None),
      ('dict', lambda x, y: {'input_a': x, 'input_b': y}, None),
      ('tuple_distribute',
       lambda *x: x,
       mirrored_strategy.MirroredStrategy(['/device:GPU:1', '/device:CPU:0'])),
      ('dict_distribute',
       lambda x, y: {'input_a': x, 'input_b': y},
       mirrored_strategy.MirroredStrategy(['/device:GPU:1', '/device:CPU:0'])))
  def test_multi_input_model(self, input_fn, distribute):
    with self.test_session():
      a = keras.layers.Input(shape=(3,), name='input_a')
      b = keras.layers.Input(shape=(4,), name='input_b')
      x = keras.layers.concatenate([a, b])
      y = keras.layers.Dense(5, name='dense')(x)

      optimizer = gradient_descent.GradientDescentOptimizer(0.001)

      model = keras.Model(inputs=[a, b], outputs=[y])
      model.compile(loss='mse', metrics=['mae'], optimizer=optimizer, distribute=distribute)

      input_a = np.zeros((10, 3))
      input_b = np.zeros((10, 4))
      targets = np.zeros((10, 5))
      dataset = dataset_ops.Dataset.from_tensor_slices((input_fn(input_a, input_b), targets))
      dataset = dataset.repeat(100)
      dataset = dataset.batch(10)

      model.fit(dataset, epochs=1, steps_per_epoch=2, verbose=1)

if __name__ == '__main__':
  test.main()

Thank you for the example! This was very helpful to reproduce. I have a solution and will let you know as soon as the fix is in.

Thank you for the example! This was very helpful to reproduce. I have a solution and will let you know as soon as the fix is in.

Sure, I'm happy to help. Thanks for working on this, I'm looking forward to the feature.

The fix has been checked in. lgeiger@ Can you try running your code again and see if you run into issues?
Thanks!

Nagging Assignee @anj-s: It has been 16 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

@anj-s Thanks for pushing a fix. A quick test revealed a sneaky issue, propably related to incorrect flattening.
While it works when the input names are alphabetically sorted, it fails when they're not. Checkout the following code:
import numpy as np

from tensorflow.contrib.distribute.python import mirrored_strategy
from tensorflow.python import keras
from tensorflow.python.data.ops import dataset_ops
from tensorflow.python.platform import test
from tensorflow.python.training import gradient_descent

class TrainingTest(test.TestCase):
  def test_multi_input_model(self):
    with self.test_session():
      a = keras.layers.Input(shape=(3,), name='aa_input_a')
      b = keras.layers.Input(shape=(4,), name='zz_input_b')
      x = keras.layers.concatenate([a, b])
      y = keras.layers.Dense(5, name='dense')(x)

      optimizer = gradient_descent.GradientDescentOptimizer(0.001)
      distribute = mirrored_strategy.MirroredStrategy(['/device:GPU:1', '/device:CPU:0'])

      model = keras.Model(inputs=[a, b], outputs=[y])
      model.compile(loss='mse', metrics=['mae'], optimizer=optimizer, distribute=distribute)

      input_a = np.zeros((10, 3))
      input_b = np.zeros((10, 4))
      targets = np.zeros((10, 5))
      dataset = dataset_ops.Dataset.from_tensor_slices(({'aa_input_a': input_a, 'zz_input_b': input_b}, targets))
      dataset = dataset.repeat(100)
      dataset = dataset.batch(10)

      model.fit(dataset, epochs=1, steps_per_epoch=2, verbose=1)

  def test_multi_input_model_non_alphabetic(self):
    with self.test_session():
      a = keras.layers.Input(shape=(3,), name='zz_input_a')
      b = keras.layers.Input(shape=(4,), name='aa_input_b')
      x = keras.layers.concatenate([a, b])
      y = keras.layers.Dense(5, name='dense')(x)

      optimizer = gradient_descent.GradientDescentOptimizer(0.001)
      distribute = mirrored_strategy.MirroredStrategy(['/device:GPU:1', '/device:CPU:0'])

      model = keras.Model(inputs=[a, b], outputs=[y])
      model.compile(loss='mse', metrics=['mae'], optimizer=optimizer, distribute=distribute)

      input_a = np.zeros((10, 3))
      input_b = np.zeros((10, 4))
      targets = np.zeros((10, 5))
      dataset = dataset_ops.Dataset.from_tensor_slices(({'zz_input_a': input_a, 'aa_input_b': input_b}, targets))
      dataset = dataset.repeat(100)
      dataset = dataset.batch(10)

      model.fit(dataset, epochs=1, steps_per_epoch=2, verbose=1)

if __name__ == '__main__':
  test.main()

Nagging Assignee @anj-s: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @anj-s: It has been 29 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @anj-s: It has been 44 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @anj-s: It has been 59 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @anj-s: It has been 74 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @anj-s: It has been 89 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

I don't think this is an issue in TF 2 anymore.


bug/issue: infowindow popup displays data from other locations than where clicked in map

Greetings!
We have encountered an issue with the time aware web map template where if the user clicks on the map the infowindow pops up even if there is no data displayed there. Furthermore the user can scroll through all the visible features on the map using the arrows in the infowindow top right corner and the infowindow still does not move to the right feature. This is very confusing for the average user because he does not realize that he is looking at data from completely different locations. A live production sample that we are publicly using can be found here:
http://vedur.maps.arcgis.com/apps/OnePane/azuretime/index.html?appid=d8aa500193ff4590ae241e7884ae008f
The infowindow should not in the first place pop up and give information unless there is a feature to be found at the location the user clicks and secondly it should not show information available for more features at that location unless there are more features within the snapping/sensor range threshold where the user clicked on the map.
Thanks in advance.
Cheers!

Probably a css issue in old version of app. Tested with updated source and not happening.


[Run] I'd like to show only results from a certain plugin

Provide a description of the new feature / enhancement
Currently if I type firefox in PT Run, it just dumps everything (the program, open-in-browser, opened instances, files etc)

Is it possible to just list entries from a certain plugin / source / provider? I'm thinking

open: firefox to start a new Firefox instance
browser: firefox to open a browser searching with the keyword firefox, or open https://firefox/ (if anyone wants that, this entry really should only shows up when entering a valid URL like firefox.com or a known host like localhost, but that would be another issue
running: firefox to show a list of existing Firefox instances

or just add the little grayed description text into the fuzzy finder as well

Think you're looking for Action keys. https://docs.microsoft.com/en-us/windows/powertoys/run#action-keys
You can also not include items in the global results with the plugin manager settings.

FYI, you can change the action keys too

dang, didn't know the listed plugins are clickable :rofl:

What could we have done to help make this more obvious?

Maybe default selecting the first plugin listed would help?

Maybe default selecting the first plugin listed would help?

Wait, does that effectively require doing something in the settings UI source code, instead of just the PT Run sub-module's code?
If that's the case, a line of text is the best solution I can think of.


feature/idea ultimate aarch64 linux

jetbrains.idea-ultimate: add aarch64-linux as a supported platform
Removed unused input
update and fix aarch64 intellij versions

Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[x] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

rebase?

rebase?

Rebased

Merge conflict, please rebase.


Option 2: Support jupyterhub/binder in data_server directly

We can reliably automatically detect when we are
running on JupyterHub / Binder, and can 'do the right thing'.
This allows us to use the same code locally as well as remotely.
Backwards compatibility is preserved by leaving a stub
for data_server_proxied behind.
Alternative to #46. I much prefer this :)

I've tested this too - https://mybinder.org/v2/gh/yuvipanda/altair_data_server/better-hub-support-2. Works fine :)

This is fantastic, thanks @yuvipanda!!! 🎉
I have tested this on our course JupyterHub and can confirm that this works, and it is a very elegant solution to allow the same code in the notebook to be used regardless of whether the notebook is run locally or on a JupyterHub. This was the notebook I tested it on: https://github.com/ttimbers/altair_data_server_reprex/blob/main/reprex/altair_data_server_reprex.ipynb

Gentle ping to @jakevdp who was the last person to touch this repo. I'm happy to assist in maintenance too if necessary. Thank you for your work on altair <3

gentle ping to @jakevdp - would love for this to go in.

Thanks for your PR @yuvipanda. As you have noticed, the altair_data_server repo has not seen development in a while. In Altair 5, the recommended way to work with large datasets is via Vegafusion. The documentation contains a section of how to use the Vegafusion renderer in Altair and if you need additional performance you can use the Vegafusion widget renderer instead.
We are going to archive this repo, so I'm closing all the open issues and PRs before doing so. Try out the new options for working with large data mentioned above and if you run into issues, please open an issue directly in the altair or vegafusion repo.


Add ability to add attributes to underlying inputs for checkbox and other form elements

Is their a possibility to pass attributes to underlying inputs, it would help us in migrating from native inputs to this library.

Hi! Unfortunately no, the elements encapsulated by the shadow dom cannot allow arbitrary attributes.
Can you give more details to what attributes you want to set and why?

Closing since this isn't a planned feature, but we'd love to hear more about your use case!
Generally we recommend treating the element itself as an "<input>" and adding attributes to it as needed.
<md-outlined-text-field label="Label" data-custom-info="foo">

Primary use case for behaving like native input fields is to ease integration with our backend form libraries that only know about html attributes. This can be solved in various other ways though.
Issue is each web component library defining its own api for things like inputs (onchange/ handlechange), making integration complicated unnecessarily.

Issue is each web component library defining its own api for things like inputs

Can you elaborate what APIs are not compatible? Our components aim for parity with platform elements, so <md-outlined-text-field> should have the same API structure as <input>, including events, properties, and attributes.


update to latest asyncpg dependency

Closes #3613
Code Changes

[ ] update to latest asyncpg dependency to avoid build error

Steps to Confirm

[ ] our images build
[ ] no regressions (🤞 for good automated test coverage!)

Pre-Merge Checklist

[ ] All CI Pipelines Succeeded
Documentation:

[ ] documentation complete, PR opened in fidesdocs
[ ] documentation issue created in fidesdocs

[ ] Issue Requirements are Met
[ ] Relevant Follow-Up Issues Created
[ ] Update CHANGELOG.md
[ ] For API changes, the Postman collection has been updated

Description Of Changes
See issue description for a synopsis of why this is just now biting us, it's been lurking for a while!

this is obviously a straightforward change to get us past a build error.
@ThomasLaPiana and @pattisdr just figured you two would be the best suited to know whether updating asyncpg could have some hidden issue, and @allisonking you reported this so also wanted to make sure it looked alright to you!

Also verified that this fixed things, thanks for your work here!

Thanks @adamsachs , I had this error in another project and took me a while to get it was asyncpg.


Is this obsolete?

Is this app still usable? If so, the instructions probably need updating. Of not, perhaps the README should reflect that, and - is there currently any working way to integrate pass with IOS?

I'm using the latest codebase but I haven't tested it on iOS 10.  There's discussion about other options at https://github.com/davidjb/pass-ios/issues/8, though I can't comment on the suggestions there.  My plan is to, if possible, revamp this into a cross-platform solution.  Very much a WIP over at https://github.com/davidjb/passport.
PR etc are welcome here if you're able to contribute; otherwise comment away over at #8.


SCAP scan the docker image for STIG and/or CIS compliance

Security Content Automation Protocol (SCAP) Scan is method for using known standards to run vulnerability and compliance scans.

Blocked by https://github.com/mitre/saf/issues/1163

<EMAIL_ADDRESS>was released which should unblock this effort.

Closed in favor of https://github.com/candrews/jumpstart/pull/1056


go list ./... stop scanning because of subfolders

What version of Go are you using (go version)?

$ go version
go version go1.17.3 linux/amd64

What did you do?
go list ./...
What did you expect to see?
Warning: pattern ./...: open gitea/data/indexers: permission denied
What did you see instead?
pattern ./...: open gitea/data/indexers: permission denied

If the current user can't access any of the subfolders, go list ./... throw pattern ./...: open xxx: permission denied,  and stop the scanning. As a result, the output got empty and this will break the following works.
Maybe throwing a warning is more appropriate for this, because there may be many subfolders, and it is troublesome to verify the permissions of all folders.

There must be some scenarios that you need to set different owner permissions for subdirectories.

use -e (continue on error)
For questions please refer to https://github.com/golang/go/wiki/Questions

However, it doesn't make many difference
The -e flag changes the handling of erroneous packages, those that
cannot be found or are malformed. By default, the list command
prints an error to standard error for each erroneous package and
omits the packages from consideration during the usual printing.
With the -e flag, the list command never prints errors to standard
error and instead processes the erroneous packages with the usual
printing. Erroneous packages will have a non-empty ImportPath and
a non-nil Error field; other information may or may not be missing
(zeroed).

An error still seems appropriate, considering you've asked go to do something it can't do.

Just wanna simply skip the inaccessible folder instead of interrupting the scanning.
Is such an idea mean too much?
Anyway, thank you very much for the explanation.


compose recipe: clickhouse and dagster

minimal example of working dagster + clickhouse.
see readme.md for details

thank you for review. resolved comments


Data Column Summarization Issue After Adding Calculation Groups

Hi Daniel -
When I create a calculation group numerically formatted columns in my model that were previously were set to aggregate, e.g. summarize no longer can be summarized.  More specially, these columns have the sigma sign next to them in PBI desktop and if I were to put the values from that column into a chart the amount would sum up.

That is the behavior prior to adding a calculation group.
After I add the calculation group, the sigma disappears, formatting remains as a decimal, Summarization is still set to Sum and if I add the values to visual that don't sum up.  Is that the expected outcome?

Application specifics

Tabular Editor version: [2.12.4]
Semantic Engine used: [Power BI Desktop Version: 2.85.681.0 64-bit (September 2020)]

Thank you in advance for any insight on this!

This is by design: calculation groups automatically disable implicit measures.
However, using explicit measures is a best practice when you create a model you share with other users.
Documentation: https://docs.microsoft.com/en-us/analysis-services/tabular-models/calculation-groups?view=asallproducts-allversions

Thank you!  The Power BI Community is awesome!  Thank you for being so generous with your time!

Some visuals like Drilldown radial barchart and Impact bubble chart stop working after creating calculation groups even if i use measures.

@kadirkkkk see #380. Long story short, this is an issue in Power BI Desktop.


Add Contact and Terms pages

We no longer have a footer (there's a file, but I think it's empty), which has removed the Contact page (which included the Google Group and the<EMAIL_ADDRESS>info) and the Terms page, which included disclaimers on liability.

Those have been merged to the About page.  Should be easy to switch around if wanted.

@rcheetham mind if I close this now that it's merged into /about?


Generate cli for Pinot REST Api

Having a pinot-rest-cli based on swagger endpoints will be useful. we can eliminate some commands from pinot-admin.sh

https://github.com/swagger-api/swagger-codegen

Hi Kishore, do you have some commands or operations in mind which you would like to see prioritized for this?

Hi @kishoreg, do you have some commands or operations in mind which you would like to see prioritized for this?

I would love to work on this, I can build the CLI using python, it has rich packages for CLIs

I think it would be nice to be able to generate the client once a release is cut. In this way both client and server have the same version. You could use Github Actions to do automate the generation. For example, it is sufficient to do swagger generate client -f ./path/to//swagger.json -a kind -A pinot-sdk and you have it available. Does it make sense?
Maybe the default generated client is the Java one. While, for other languages the community can maintain themselves by still leveraging swagger-codegen or similar.

I think it would be nice to be able to generate the client once a release is cut. In this way both client and server have the same version. You could use Github Actions to do automate the generation. For example, it is sufficient to do swagger generate client -f ./path/to//swagger.json -a kind -A pinot-sdk and you have it available. Does it make sense?
Maybe the default generated client is the Java one. While, for other languages the community can maintain themselves by still leveraging swagger-codegen or similar.

Hello guys, I made some progress on that, fixing some minor swagger issues also, news soon. :)


Date entry while uploading grades via CSV

When a teacher is uploading a grade CSV, the frontend should parse attainment tags from the CSV and allow the teacher to set the date and expiry date for each attainment, which are sent in the body to the server.

Just pinning a comment here to remind about this issue. When importing from CSV, there is currently no way to set the attainment completion date. When uploading a CSV with grades, there could be a dialog that let's the teacher to pick the date.


Optimize SentencePiece tokenizer

Create and configure SentencePieceTokenizer only once during node construction.

Could you please add some information about this operation to the README
I also think, we need an example and tests for this operation.

Can we include exception handling for the situating the input data structure is incorrect? Then the input data is random or incorrectly formatted, the extension is causing a segmentation fault. That might be the case then somebody will use benchmark app without -i param or in OVMS when the data is sent as a vector instead of a string.

Can we include exception handling for the situating the input data structure is incorrect? Then the input data is random or incorrectly formatted, the extension is causing a segmentation fault. That might be the case then somebody will use benchmark app without -i param or in OVMS when the data is sent as a vector instead of a string.

Done

Could you please add some information about this operation to the README
I also think, we need an example and tests for this operation.

Thanks for the comment. I will add documentation but there is a problem with programming of such sub-graph with SentencePieceTokenizer for the test. Not easy to do it because I need somewhere to get sp_model to initialize the tokenizer.
Propose to proceed without test for it and think of it in the next PR. We test it on the MUSE model manually.
Best regards,
Roman

Could you please add some information about this operation to the README
I also think, we need an example and tests for this operation.

Done


Help getting this working

Hi, I have a synology ds1815+ and I’m trying to figure out to change the parameters. I’m able to see MAKEMKV GUI on the webpage but need help getting it to find my optical disc drive.

Found my optical disc drive @ /dev/sg6 but on the webpage I get program can’t find any usable optical drives.

Look at the container’s log: you will see clear messages about which Linux devices need to be exposed to the container.

The log only shows sg6 needing to be exposed and gives a warning that performance might suffer because it doesn't see /dev/sr*.

straight from my log:
[cont-init.d] 95-check-optical-drive.sh: found optical drive [/dev/sr0, /dev/sg10]

You normally have to expose 2 devices.  Can you provide the whole log?

edit the container, in the extra parameters section add:
--device /dev/sr0  --device /dev/sg0 --device /dev/sg1 --device /dev/sg2 --device /dev/sg3 --device /dev/sg4 --device /dev/sg5 --device /dev/sg6 --device /dev/sg7 --device /dev/sg8 --device /dev/sg9  --device /dev/sg10
this will keep you covered over reboots in case it changes

I can provide the whole log a few hours.

@ c4wiz are you using a synology nas? I will be back home in few hours. It would awesome to get this work.

@ c4wiz are you using a synology nas? I will be back home in few hours. It would awesome to get this work.

No, im Using unRAID v. 6.6.6

I started it CLI like this.
docker run -d \

--name=makemkv \
-p 5800:5800 \
-v /docker/appdata/makemkv:/config:rw \
-v $HOME:/storage:ro \
-v $HOME/MakeMKV/output:/output:rw \
--device /dev/sg6 \
jlesage/makemkv

makemkv.html.zip

[0:0:0:0]    disk    HGST     HDN726040ALE614          APGN  /dev/sdc   /dev/sg0
[1:0:0:0]    disk    HGST     HDN726040ALE614          APGN  /dev/sdd   /dev/sg1
[2:0:0:0]    disk    HGST     HDN726040ALE614          APGN  /dev/sde   /dev/sg2
[4:0:0:0]    disk    HGST     HDN726040ALE614          APGN  /dev/sda   /dev/sg3
[5:0:0:0]    disk    HGST     HDN726040ALE614          APGN  /dev/sdb   /dev/sg4
[10:0:0:0]   disk    Synology DiskStation              PMAP  /dev/synoboot  /dev/sg5
[13:0:0:0]   cd/dvd  ASUS     BW-16D1HT                3.01  -          /dev/sg6

Looks like the Linux SCSI Generic (sg) driver is not loaded.  Can you provide the output of the following command:
lsmod|grep sr

It seems that Synology doesn't officially supports optical drives:
https://www.synology.com/en-global/knowledgebase/DSM/tutorial/Compatibility_List/Does_Synology_Product_support_external_DVD_RW_DVD_ROM_CD_ROM_or_CD_RW_drive
But it seems that some users were able to compile proper drivers and make it work.

Do you know what were the permissions and the owner of the device before the change?  This is something that should have been caught by the detection script.

Before crw------- 1 root root
After   crw-rw---- 1 root root

Also no extra parameters section on synology.

Im trying to figure it out now because I keep getting error posix when trying to write to directory.

Just set it 0 to test and it works, will figure out permissions when it finish.

uid=1026(rome1931) gid=100(users) groups=100(users),101(administrators)
I tried 1026 and 100 without success.

Look at the ownership and permission of the folder you mapped to /output.

I had figured it out, but the ripping speed is super slow. 1.1x
When I ripped as root it was 5.5x and I didnt let it finish to see top speed.

It took a while, but rip speed got faster. Thank you

No cdrom module on synology. Your detection script did detect it, right after the [ -, sg6] was group 0. I didn’t know that was the root group.

The permissions kept changing on reboot. I was able to solve that with a udev rule and creating a cdrom group and adding myself to it.
SUBSYSTEMS=="usb", ATTRS{idVendor}=="", ATTRS{idProduct}=="", GROUP="users", MODE="0660"

And yet another new user who can't manage to use the BlueRay drive.
In the meantime I have read quite a few threads about it, but still can't get it to work.
I'm not using Docker on my Synology DS923+ (DSM 7.2.1-69057 Update 3), but the "Container Manager".
JDownloader-2, Handbrake, MKVToolNix work without any problems.
But with MakeMKV I can't get access to my "BUFFALO Optical Drive".
This message appears:
2024/04/03 21:34:19,stdout,[cont-init   ] 54-check-optical-drive.sh: no usable optical drive found.
2024/04/03 21:34:19,stdout,[cont-init   ] 54-check-optical-drive.sh:   --> the host device /dev/sg5 is not exposed to the container.
2024/04/03 21:34:19,stdout,[cont-init   ] 54-check-optical-drive.sh:   --> the host device - is not exposed to the container.
2024/04/03 21:34:19,stdout,"[cont-init   ] 54-check-optical-drive.sh: found optical drive [-, /dev/sg5], but it is not usable because:
2024/04/03 21:34:19,stdout,[cont-init   ] 54-check-optical-drive.sh: looking for usable optical drives...
I tried my admin user (uid=1026, gid=101).
su@NAS:/$ cat /etc/passwd | grep -E 'su|root'
root:x:0:0::/root:/bin/ash
su:x:1026:100::/var/services/homes/su:/bin/sh
Still no access, then i tried uid=0, gid=0 (root). Again, no access to it.
Then i tried like rome1983:
su@NAS:/$ ls -l dev/sg5
crw------- 1 root root 21, 5 Apr  3 21:07 dev/sg5
su@NAS:/$ sudo chmod 660 dev/sg5
su@NAS:/$ ls -l dev/sg5
crw-rw---- 1 root root 21, 5 Apr  3 21:07 dev/sg5
Still no access.
I guess I'll have to keep using the drive on my computer, the DS923+ don't accept it.
Or replace the "Container Manager" with Docker and try that.
But I would like to avoid that if possible.

Good news, sr_mod.ko and cdrom.ko  is compiled and available at
https://synocommunity.com/
https://synocommunity.com/package/synokernel-cdrom
eject now works!!!

Hey fellow Synology users :)
I am trying to get it working and I am running into issues. I managed to get it to work on DSM 7.0 (and then scrapped it) but having problems with getting it to work from scratch on 7.2.1-69057 Update 5.
I am using following yml compose file:
version: '3'
services:
makemkv:
image: jlesage/makemkv
container_name: makemkv
restart: unless-stopped
ports:
- "5800:5800"
environment:
MAKEMKV_KEY: ****
USER_ID: 1026
GROUP_ID: 101
AUTO_DISC_RIPPER: 0
APP_NICENESS: 0
#  WEB_LISTENING_PORT: 5802
MAKEMKV_GUI: 1
volumes:
- "/volume2/docker/MakeMKV-Automation/AppData:/config:rw"
- "/volume1/MakeMKV/AutoMKVStorage:/storage:ro"
- "/volume1/MakeMKV/AutoMKVOutput:/output:rw"
devices:
#- "/dev/sr0:/dev/sr0"
#- "/dev/sg1:/dev/sg1"
#- "/dev/sg2:/dev/sg2"
#- "/dev/sg3:/dev/sg3"
#- "/dev/sg4:/dev/sg4"
#- "/dev/sg5:/dev/sg5"
#- "/dev/sg6:/dev/sg6"
#- "/dev/sg7:/dev/sg7"
#- "/dev/sg8:/dev/sg8"
- "/dev/sg9:/dev/sg9"
#- "/dev/sg10:/dev/sg10"
And I am getting the following:

And no detected drives:

While when I change in yml file:
USER_ID: 0
GROUP_ID: 0
MakeMKV reports the same warning in logs of the container but it sees the drive in the application.
Running command dmesg via SSH (and extracting from it relevant info gives the following two BD drives - I use two drives to rip). Here as you can see they are type 5? While MakeMKV logs report them as type 0? Is it the same thing?
    `Line` 1199: [   25.558557] scsi 11:0:0:0: CD-ROM            HL-DT-ST BD-RE  WH16NS60          1.02 PQ: 0 ANSI: 0
Line 1203: [   25.591509] scsi 11:0:0:0: Attached scsi generic sg9 type 5
Line 1206: [   25.636994] scsi 12:0:0:0: CD-ROM            HL-DT-ST BD-RE BU40N              1.03 PQ: 0 ANSI: 0
Line 1207: [   25.648673] scsi 12:0:0:0: Attached scsi generic sg10 type `5`

Running sudo docker exec makemkv lsscsi -g -k gives the following (I unplugged one of the drives so that is why only one shows in this output).
[0:0:0:0]    disk    TOSHIBA  HDWE150                  FP1R  /dev/sda   /dev/sg0
[1:0:0:0]    disk    TOSHIBA  HDWE150                  FP1R  /dev/sdb   /dev/sg1
[3:0:0:0]    disk    WDC      WD101EFAX-68LDBN0        0A81  /dev/sde   /dev/sg2
[4:0:0:0]    disk    TOSHIBA  HDWE150                  FP1R  /dev/sdd   /dev/sg3
[5:0:0:0]    disk    TOSHIBA  HDWE150                  FP1R  /dev/sdc   /dev/sg4
[7:0:0:0]    disk    WDC      WD101EFAX-68LDBN0        0A81  /dev/sdh   /dev/sg5
[8:0:0:0]    disk    WDC      WD101EFAX-68LDBN0        0A81  /dev/sdg   /dev/sg6
[9:0:0:0]    disk    WDC      WD101EFAX-68LDBN0        0A81  /dev/sdf   /dev/sg7
[10:0:0:0]   disk    Synology DiskStation              DL17  /dev/synoboot  /dev/sg8
[11:0:0:0]   cd/dvd  HL-DT-ST BD-RE  WH16NS60          1.02  -          /dev/sg9
Any ideas why it works when I change it to (from reading this thread looks like I am running it with root privileges) and what to do to be able to go back to 1026 and 101?
USER_ID: 0
GROUP_ID: 0
I can follow instructions and have some very hazy understanding of what I am working with but I am novice linux and linyx like platforms user so please have patience. On the flip side. It will come handy to everyone that follows in my footsteps :)
BTW I have two Synology NASes, both running the same DSM software version (so current DSM that is pushed via update service - I know there is a newer version that can be downloaded manually) and both behave exactly the same way.
In version DSM 7.0 it was working with 1026 and 101...

The permissions kept changing on reboot. I was able to solve that with a udev rule and creating a cdrom group and adding myself to it. SUBSYSTEMS=="usb", ATTRS{idVendor}=="", ATTRS{idProduct}=="", GROUP="cdrom", MODE="0660"

Could you explain how and where to use it in DSM? Thank you.
I assume that I am supposed to enter my VID and PID for my BD drives in place of ** but this command would not work via SSH. Thanks

lsusb gives idvendor an idProduct
152d:0578:0415 00  3.00 5000MBit/s 8mA 1IF  (JMicron USB to ATA/ATAPI Bridge<PHONE_NUMBER>ABCDEF)
udev rule would like this
SUBSYSTEMS=="usb", ATTRS{idVendor}=="152d", ATTRS{idProduct}=="0578", GROUP="cdrom", MODE="0660"
use task-scheduler to copy udev rule on boot
run as root
cp /path/to/edev/rule/51-dvd-rom.rules /usr/lib/udev/rules.d/
udevadm control --reload
I also created a cdrom group and added myself to it.
Hope this helps.

lsusb gives idvendor an idProduct
152d:0578:0415 00 3.00 5000MBit/s 8mA 1IF (JMicron USB to ATA/ATAPI Bridge<PHONE_NUMBER>ABCDEF)
udev rule would like this
SUBSYSTEMS=="usb", ATTRS{idVendor}=="152d", ATTRS{idProduct}=="0578", GROUP="cdrom", MODE="0660"
use task-scheduler to copy udev rule on boot run as root
cp /path/to/edev/rule/51-dvd-rom.rules /usr/lib/udev/rules.d/ udevadm control --reload
I also created a cdrom group and added myself to it.
Hope this helps.

Hey,
Thanks for getting back to me. It does help a lot. I will try it later on today but I am assuming it will not make a difference and the reason is that I have already tried (sg9 is position of one of my BD drives).
sudo chmod 660 /dev/sg9
... and it made no difference and I assume that udev rule only makes it persistent over restarts? Would I be right?
In relation to your tips on how to get the udev rule going. Am I right that I have to create file named 51-dvd-rom.rules and type into it and save SUBSYSTEMS=="usb", ATTRS{idVendor}=="152d", ATTRS{idProduct}=="0578", GROUP="cdrom", MODE="0660" as a sole line in it (of course using my particular VID and PID) and then use the copy command in the boot-up script?
What would happen if would just put the file manually in it's destination? I am guessing that without --reload switch it would not be executed on DSM restart?
BTW - another way to get VID and PID in DSM is to go to system info in CP and hover your mouse over the device you want to get the details for and a pop-up will show with this info.
Thanks

The udev rule be deleted after every dsm update. I uesd task scheduler to automate the proess. I would also create the cdrom group as the udev rule is dependent on it. I have another script that changes ownership of cdrom drive
chown USER:cdrom /dev/sg9

Thank you again. I will be trying all that we have discussed in next few hours and report back afterwards.

How unwise / dangerous it is to leave the container running with root privileges?

Replace USER with your username and judging by your udev rule it would be
sudo chown sebnapower:BDRippers /dev/sg9
after that run
ls -al /dev/sg9

I refined my udev rule to where I only need one. This works if you have the sr_mod.ko and cdrom.ko kernel modules loaded.
The ENV{PHYSDEVDRIVER}=="sr" is nedded to only apply permissions to  devices using that driver.
SUBSYSTEMS=="scsi", KERNEL=="sr*[0-9]|sg*", ENV{PHYSDEVDRIVER}=="sr", ATTRS{type}=="5", OWNER="USERID", GROUP="cdrom"

I refined my udev rule to where I only need one. This works if you have the sr_mod.ko and cdrom.ko kernel modules loaded. The ENV{PHYSDEVDRIVER}=="sr" is nedded to only apply permissions to devices using that driver.
SUBSYSTEMS=="scsi", KERNEL=="s[rg]*", ENV{PHYSDEVDRIVER}=="sr", ATTRS{type}=="5", OWNER="USERID", GROUP="cdrom"

I will be trying suggestions from your both recent replies this evening.
SUBSYSTEMS=="scsi", KERNEL=="s[rg]*", ENV{PHYSDEVDRIVER}=="sr", ATTRS{type}=="5", OWNER="USERID", GROUP="cdrom"
Am I right that:
a) that I only need to replace USERID and cdrom in the quoted line of code with my specific values? (also has that part KERNEL=="s[rg]* correct spelling with the brackets etc. Just in case as it looks a but unusual)
b) I would be putting quoted line of code into 51-dvd-rom.rules file?
Many thanks for refining it and helping with it in general.

KERNEL=="s[rg]*  we are matching against sr* and sg* with * representing a integer. when I connect a drive and run dmesg
1436617.868156] sr 91:0:0:0: Attached scsi CD-ROM sr0
[1436617.868391] sr 91:0:0:0: Attached scsi generic sg9 type 5
If you are not loading the sr_mod.ko and cdrom.ko kernel modules run
udevadm info /dev/sg9
so you can match against the value found
PHYSDEVDRIVER}=="sr"
your updated udevrule would be
SUBSYSTEMS=="scsi", KERNEL=="s[rg]*", ENV{PHYSDEVDRIVER}=="sr", ATTRS{type}=="5", OWNER="sebnapower", GROUP="BDRippers"
or
SUBSYSTEMS=="scsi", KERNEL=="s[rg]*", ENV{PHYSDEVDRIVER}=="sg', ATTRS{type}=="5", OWNER="sebnapower", GROUP="BDRippers"
Powercycle after reloading rules

Ok, so I sat down to test the new options you have kindly prepared for me and I did not have to as it actually is working.
I then realized that I was only using udev --reload command and full NAS restart was what was needed.
Then I came here to let you know about it and I seen this line at the end of your last message :D

Powercycle after reloading rules

At the same time I good few steps yesterday so today I will repeat the optimal one on my other NAS and confirm that it is working,
I believe that your original solution, obviously still works (so with simpler version of UDEV rules). If it is straight forward on the other NAS and works with your simpler solution I will probably stick with it unless there are some other benefits to loading drivers for BD-ROMs other than enabling the auto eject function (would the loading of the drivers allow for higher riping speed for example or less errors etc.? Or is it purely cosmetics other than software eject button?).
Another thing I wanted to ask, which I will test anyway, if I live the original UDAV rules (in my case) and I plug only one of two drives. It would not cause any problems?
I said that if it works on other NAS that I will probably stick with your original solution. But knowing me, once I am done with setting all other switches and configurations of MakeMKV I will probably go back to it and see about your refined solution with inclusion of the drivers loading.
Now lets see if it works straight away on the other NAS.
Thanks

Just wanted to confirm that it was a matter of restarting NAS or possibly it was just enough to unplug and plug again BD drives for UDEV rules to be applied to them.
It went smoothly on my other NAS and working nicely.
Now I am trying to figure out things around using 2 BD drives in parallel. Particularity if two MakeMKV containers can use the same data folder location. That is another topic altogether.

Yes power cycling the drive will trigger the udev rule. I use separate data folders for my 2 drives. Sharing the data folder was causing problems.

Yes power cycling the drive will trigger the udev rule. I use separate data folders for my 2 drives. Sharing the data folder was causing problems.

My apologies for introducing wrong wording into conversation.
I meant config folder as there is no data. I hope that is what you meant as well?
I assume that "storage" and "output" mappings can be shared between the two containers?
      - "/volume2/docker/MakeMKV-Automation/AppData:/config:rw"
      - "/volume1/MakeMKV/AutoMKVStorage:/storage:ro"
      - "/volume1/MakeMKV/AutoMKVOutput:/output:rw"

Thanks

Yes sharing config folders caused problems for me. Yes output folders can be shared.

Set them to two different config folders.
Will start ripping via my NAS in few days but I am sure it will be fine.
Now hoping that the image could be updated to support the newest version of MakeMKV released few weeks ago,


keyphrases in sustained complaints

Zac asked last week if we could check whether there are common phrases present in the allegations that contribute to whether an allegation is sustained.
There's an open source python library, pke, that could be useful for identifying common phrases. I took a pass at it with the TopicRank extractor but found that we may be incorrectly separating allegations that trail onto the next page, so I'll need to fix that issue before continuing to process phrases from the allegations.

We want to ignore field headers that are likely to congest the common string detection, so my thinking was to combine allegations (the cleaned string that follows "SUMMARY OF ALLEGATION(S)") with the text from findings_of_fact.
Both are technically written by the DPA, but the allegations reiterate the complainant's narrative broken out into individual allegations of misconduct. The findings_of_fact discuss the collected evidence and interviews conducted by DPA associated with a given allegation.
I think the findings_of_fact are more likely to be formulaic and contain common phrases (enough that we already have a few indicators set up to detect these phrases, ie. jlp for "justified, lawful, and proper"), but may be important to consider when the summary of allegations is extremely brief.
Open to suggestions! Maybe it's more reasonable to just use the allegations only, since that's more connected the language of the original complaint? Zac was looking for ways to improve the language of submitted complaints based on what has historically received more traction in terms of sustained allegations.

May also be worth noting here that when allegations are added by the DPA (or OCC, since we also have those older records), these tended to be sustained more often due to the nature of how they come to exist.
It might be best to exclude these if our interest is in features of the original complaint/allegation language, though I'm not sure and will process them altogether for now.

My initial thought is we should treat allegations and findings of fact separately rather than combining them. Although both are written by DPA, in terms of strategizing how to word complaints we submit in the future, we'll want to focus on language in allegations. That said, it might be the case that there is language in the findings that points to specific ways of presenting important evidence, that we can adopt for when we submit complaints. I just think it's worth trying it with just the allegations for this purpose (in addition to combined which we can also set up, the code should not be much different).
Ideally, we structure this as a classification problem, where we classify whether the complaint is sustained or not. And we use extracted keyphrases as features, along with: type of alleged misconduct, OCC vs. DPA, and whether or not the complaint was original or added by the agency. Those are the three major features that I imagine we'd want to be able to "control" for in some way (maybe there are others?). From there we can examine variable importance for specific keyphrases, and given your observation here, we should stratify those by whether the complaint was added or not (and perhaps focus on DPA vs OCC). In addition to auto-extracted keyphrases, make sure we include those that Zac has already come up with as features.

That makes sense, I'll work with just the allegations text for now.
I did remember the goal to make it a classification problem and agree that category_of_conduct, report_type, and dpa_added|occ_added plus the extracted keyphrases should be the starting set of features.
I'm fiddling with the model parameters to make sure the extracted keyphrases are useful (top two results are consistently "officer" and "complainant") and I'll go back to confirm which of the existing phrases we indicate were suggested by Zac so we can pull those in.
Thanks for the feedback!


PHP hereDoc and nowDoc incorrect indentation and folding

Issue Type: Bug
Using a hereDoc or nowDoc, the folding and indentations mess up.
$variable = <<<"DOC"
this is a test doc
DOC;
function x()
{
    someFunction();
}
$other_vaiable = <<<"OTHERDOC"
this is another test doc
OTHERDOC;

The folding should fold out the Doc block into this
$variable = <<<"DOC"...
DOC;
function x()
{...
}
$other_vaiable = <<<"OTHERDOC" ...
OTHERDOC;

but it folds like this:
$variable = <<<"DOC"...
DOC; ...
$other_vaiable = <<<"OTHERDOC" ...
 Such extension have a parser and know where the here docs start. There's the folding range provider API that allows the extension to tell where the folding regions are. If no extension provides folding ranges, we fall back to indentation based folding ranges.

Is there anyway I could find out which extension decides formatting and folding for a particular action? anywhere on logs? Truth be told, non of extensions installed that I have listed, mentions formatting.

Formatting and folding are provided in code, so we also don't know beforehand which extension contributes what. We don't have a feature to log what was contributed by who, but that would make sense. Please file an issue.


Create SqlCallTransformer for generic transformations on SqlCall

This PR creates an abstract class SqlCallTransformer for generic SqlCall transformations. SqlCallTransformer can be utilized to facilitate interconversions between languageSqlNode and CoralSqlNode, particularly when information about the operand and operator datatype is necessary for the transformations. It provides following APIs:

protected abstract boolean predicate(SqlCall sqlCall): predicate of the transformer, it’s used to determine if the SqlCall should be transformed or not
protected abstract SqlCall transform(SqlCall sqlCall): implementation of the transformation, returns the transformed SqlCall
public SqlCall apply(SqlCall sqlCall): public entry of the transformer, it returns the result of 2 (transformed SqlCall) if condition(SqlCall sqlCall) returns true, otherwise returns the input SqlCall without any transformation
protected RelDataType getRelDataType(SqlNode sqlNode): helper method to derive the datatype of the input SqlNode with SqlValidator, it’s only used when sqlValidator is provided in the constructor

To use SqlCallTransformer, this PR creates a container class SqlCallTransformers, with which we can create a SqlCallTransformers with of method and call apply to apply all the SqlCallTransformers to the SqlCall.
After defining a set of transformers, we need a SqlShuttle to traverse the SqlNode tree and apply the transformers to the SqlCalls, like HiveSqlNodeToCoralSqlNodeConverter. For Coral-Spark and Coral-Trino, we can add this part in CoralSqlNodeToSparkSqlNodeConverter and CoralSqlNodeToTrinoSqlNodeConverter.
Tests:

Existing UTs, which already contains the tests for array index transformation
i-test on all the prod views, no regression

LGTM except implicit call order assumption in SqlCallTransformer.getRelDataType().
Please file issue for (1) the above and for (2) investigating using SqlBasicCall in SqlCallTransformer then we can merge after @aastha25 and @yiqiangin approval.

LGTM except implicit call order assumption in SqlCallTransformer.getRelDataType(). Please file issue for (1) the above and for (2) investigating using SqlBasicCall in SqlCallTransformer then we can merge after @aastha25 and @yiqiangin approval.

Thanks! Will open issues once this PR is merged. @aastha25 @yiqiangin please take a look if you get a chance.

Also please update the JavaDoc of the current SqlCallTransformer.getRelDataType() with the precise expectation from the caller of this API to behave correctly.

Thanks @wmoustafa @aastha25 @yiqiangin for your review! I have addressed all the comments and kicked off the regression test again, will merge it once the regression test passes.


Remove NextPath from Iterator interface

NextPath is one of the weird hacks that remain in Cayley from old days. It was added to make HasA iterator work properly, but in fact it can be hidden under the flag of HasA iterator.
HasA allows to either iterate over unique nodes that can be found on a specific direction of a quad, or iterate over duplicate nodes in this direction, but also "touch" other paths that can lead to this node.
It always leads to confusion in planner and query optimizer, because it cannot be quite sure if user will call one branch or the other, while backend could in theory build two optimal queries for each case.
It also getting worse when tags are defined on the query tree. It might be optimal to flip the flag on HasA to disable duplicates if no tags are defined in subtree, but again, user may decide to call NextPath anyway and break this optimization.
Splitting an iterator into two, or defining a new option for it will fix an issue and allow to properly optimize these query types. It will also lead to cleaner code.

@dennwc is this part of the iterators refactor planned for this version?


Wrapping up all Sax Parser Handlers

Description

This completes all the available handlers(except Etopo)

Description

This completes all the available handlers(except Etopo)

We do need the etopo datasets. Is there a complication with doing them, or are they just not in this pull?

Was not sure initially how to handle a self closing tag but turns out SAX treats it as both - opening and closing tag.
Pushed Etopo too.
This probably wraps up all the handlers.


User class not working

Hello,
I tried the example in your doc to get information of all users, but it didn't work. The errors are as follows:
Traceback (most recent call last):
File "/home/admin/trial2.py", line 62, in
client.lists()
File "/home/admin/.local/lib/python3.9/site-packages/synapse_admin/user.py", line 117, in lists
resp = self.connection.request(
File "/home/admin/.local/lib/python3.9/site-packages/synapse_admin/base.py", line 626, in request
return request(url, **kwargs)
File "/home/admin/.local/lib/python3.9/site-packages/httpx/_client.py", line 1045, in get
return self.request(
File "/home/admin/.local/lib/python3.9/site-packages/httpx/_client.py", line 821, in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/home/admin/.local/lib/python3.9/site-packages/httpx/_client.py", line 908, in send
response = self._send_handling_auth(
File "/home/admin/.local/lib/python3.9/site-packages/httpx/_client.py", line 936, in _send_handling_auth
response = self._send_handling_redirects(
File "/home/admin/.local/lib/python3.9/site-packages/httpx/_client.py", line 973, in _send_handling_redirects
response = self._send_single_request(request)
File "/home/admin/.local/lib/python3.9/site-packages/httpx/_client.py", line 1009, in _send_single_request
response = transport.handle_request(request)
File "/home/admin/.local/lib/python3.9/site-packages/httpx/_transports/default.py", line 218, in handle_request
resp = self._pool.handle_request(req)
File "/usr/lib/python3.9/contextlib.py", line 135, in exit
self.gen.throw(type, value, traceback)
File "/home/admin/.local/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectError: [Errno -2] Name or service not known

The synapse version is 1.76.0, I see the package is only compatible with version <=1.70, but I'm only trying a basic function.
Also I'm new to this, I didn't use a api.cfg file in the process.
Thank you

Btw, for a side question, I'm using ansible playbook to deploy the synapse, so the server lives in a docker container. And all my current admin api calls have to be made after I enter the docker environment. So is there really no way to use python to make admin api calls in my case?

Absolutely!
Two lines as follows:
client = synapse_admin.User(server_addr="https://matrix.*******.org/", server_port=8008, access_token=params["access_token"], server_protocol="https://")
client.lists()

Does the problem happen all the time or just occasionally? Also, please try the following:
import requests
r = requests.get(
    "https://matrix.*******.org:8008/_synapse"
    "/admin/v2/users?from=0&limit=100&guests="
    "true&deactivated=false&dir=f",
    headers={
        "Authorization": f"Bearer {params['access_token']}",
        "User-Agent": f"matrix-synpase-admin-python/test"
    }
)
print(r.json())

Does it print out a list of users correctly?

The program hangs, I have to kill it with keyboard.
So I guess there is something wrong with forwarding admin api when I set up the synapse.
I'm using ansible, do you happen to know what I might have missed to make admin apis reachable from outside the docker container?
Anyway really appreciated!

I have no idea for now, but I will try to deploy a Synapse instance through ansible to see what's the problem.


⚠ Processors has degraded performance

In 1b55bef, Processors ($STATUS_URL) experienced degraded performance:

HTTP code: 200
Response time: 39 ms

Resolved: Processors performance has improved in 05ba75e.


Refactor check for overriden methods in inner classes

Small follow-up to https://github.com/godotengine/godot/pull/89545.
I didn't get a chance to push this change which I made based on feedback before the PR was merged.

Thanks!


Camera properties use incorrect casing according to glTF 2.0 spec

Hi,
While validating a glTF scene with a camera I encountered the following error:

So in CameraOrthographic.cs and CameraPerspective.cs the written properties are using camel casing. However, xmag, xmag, znear, and zfar should be lower case according to the glTF 2.0 spec.
It might be useful to fix this before merging this camera implementation: (https://github.com/KhronosGroup/UnityGLTF/pull/153).

Fixed in #158. Thanks for pointing this out!


Plot Predicted Glucose w/o Temp Basal

Show predicted glucose without temp basal in a different color. The data is already there in a different json file.

Fixed in #38


EN translation commented out

Why in index.js the EN translation is commented out? I know that the same values are in the code as defaults, but it would be better to use the translation file for EN, so you can adjust your English version if needed and not search and change default labels in the code.

The en translation is the reference, and it is only the same what is the code as default. So to avoid redundancy it is commented out. It is only a collected form to ease the translators work for a new language.


Add Gemini 1.5 Flash Model to Available Model Options in Staging Branch

This PR adds Gemini 1.5 Flash as an available model option as per issue #141 to the staging branch.  Key changes are:

Add Gemini 1.5 Flash to Model Options: Updated constants.ts to include Gemini 1.5 Flash within the model options, making it accessible to users.
Install Dependencies: Added @langchain/google-vertexai to support Gemini 1.5 Flash.
Google Cloud Authentication: Configured with the GOOGLE_APPLICATION_CREDENTIALS environment variable.

Testing and Validation
Testing Confirms:

Gemini 1.5 Flash appears as a selectable model.
API calls to Vertex AI authenticate successfully.
The application runs without errors related to model selection or API integration.

LangSmith Trace Link

Hi Brace,
Thanks for the update and for merging the PR! I really appreciate it.
As for the shoutout, that’s very kind of you! You can find me on Twitter and LinkedIn here:

LinkedIn: https://www.linkedin.com/in/mrjunaid/
X (Twitter): https://x.com/MJunaidshaukat


Port a number of task interfaces from Nipype

Created a Pydra Task interface in a module named Preprocess.py for ANTSRegistration command.

pydra/tasks/ants/utils:  successfully ported and tested ai.py, imagemath.py, multiplyimages.py, thresholdimage.py
pydra/tasks/ants/segmentation: successfully ported and tested atropos.py, n4biasfieldcorrection.py
pydra/tasks/ants/registeration: successfully ported and tested registeration_tested_preprocess

@Mahdieh-dst the tests are failing. Could you please add a file called pyproject.toml to the root directory with the following contents to your branch?
[build-system]
requires = ["setuptools==62", "wheel"]

What's the state of pydra for doing nipype workflows? I see most of the packages like this are stubs. I'd like to implement a pipeline using the new pydra-ants interfaces.

hi @gdevenyi, we are actively working on this and hope to have more than a few stubs available soon. There is a prototype nipype-to-pydra converter you could use if you wanted to try it out to convert the interfaces that you need (https://github.com/nipype/nipype2pydra).
Have a look at https://github.com/nipype/nipype2pydra/tree/main/example-specs/task for some example specs to use as a template.


angular 2 datePicker

hey @behrang , i'm using a angular 2 datePicker with moment calendar in my project, and now i just want add jalali date on this component, can you help me ?
my datePicker is : https://www.npmjs.com/package/ng2-date-picker
in the same vein :
import * as moment from 'jalali-moment';
moment.loadPersian(true);
moment.locale('fa-IR');

but jalali calander not displayed in datePicker (e.g: مهر ، آبان), Now just week days are currect!

Hey @sasanazizi, this library does not change moment, it extends moment with some more methods (like jMonth). So it is not possible to change calendar system of date pickers by just importing this library. A solution is to fork date picker, and change function and method calls to use the appropriate Jalaali version.
Some other date pickers using this library are available on the project home page (react-advance-jalaali-datepicker and ng-jalali-flat-datepicker).


Fix PTY for vast-cloud CLI

Currently, all the commands get a PTY by default. This is not necessary in general and tends to mess up the output (stderr is redirected to stdout).
This change also cleans up the output the the help command.
:memo: Checklist

[x] All user-facing changes have changelog entries.
[x] The changes are reflected on vast.io, if necessary.
[x] The PR description contains instructions for the reviewer, if necessary.

:dart: Review Instructions
There are three aspects to this change:

cleanup the output of the help command by subclassing Program in main.py (this is not directly linked to the PTY fix)
create a thin layer around PyInvoke tasks to allow the vast-cloud entrypoint script to know if a TTY is required. This is implemented in vast_invoke.py
update the vast-cloud entrypoint script to first check if TTY is required before running the actual command
make all the tasks use this wrapper instead of the original PyInvokde @task decorator
also cleaned up per namespace config INVOKE_CONFIG made unnecessary by this change

https://github.com/tenzir/vast/actions/runs/2657913256

https://github.com/tenzir/vast/runs/7307645701

https://github.com/tenzir/vast/actions/runs/2661789468

https://github.com/tenzir/vast/actions/runs/2691557336


No more single file?

this isn't an issue and has no description. Please follow the installation instructions in the read me.


swapping documentation link

/cc @lance @omveda

/cc @Hyperkid123
summary: the link in the documentation does not work, swapping it out with a link that does


Integrate playsound module

Why it is needed to be integrated into tukaan itself? Just run playsound in a separate thread.

Well i just thought it would be useful

Maybe, if you specify what kind of integration do you mean, it would be clearer. From my perspective, it is easy to use playsound as is, as a separate package, without any additional integration.
As an example, what needs some kind of integration is pystray, because there is no clear and obvious way how to add a tray icon when you use tkinter or tukaan. There is some workaround on StackOverflow, but as far as I remember I had some problems with this approach on Linux.

Well, I'm not going to integrate the playsound module, for several reasons.

playsound has only one function, which isn't a big deal to import from there instead of Tukaan

playsound advertises itself as no dependencies required, which is not true. On Linux you need the PyGObject package, which is a big dependency

playsound is dumb. It has only one function, which doesn't allow you to stop the music, edit it, or do anything useful with it.

I'm currently implementing a MUCH better way to play, record, and edit sounds, so playsound is not necessary!
Look at this!
# Play sound
sound_to_play = tukaan.media.Sound("/home/rdbende/Music/OMD/Walking on the Milky Way.wav")
sound_to_play.play()

# Record sound
sound_to_record = tukaan.media.Sound()

with sound_to_record.record():
    tukaan.Timer.wait(10)

sound_to_record.save("/home/rdbende/Music/recorded_with_tukaan.wav")

# Edit sound
# I don't wanna spoiler :D

@insolor I'm not sure in Pystray either.
It's a pain to get it working with Tkinter, since both have their own mainloops that block each other.
I might just implement the tk systray feature that's in Tk 8.7, although it would be nice to find a way to make Tukaan and Pystray work well together.

But mac dosent have Tcl 8.7 unless you use annaconda, right?

I won't add support for Tcl/Tk 8.7 until they release the stable version, so I don't think that's a problem.


bug(COMPONENT): TITLE

Reproduction
Use StackBlitz to reproduce your issue:

Components: https://stackblitz.com/fork/components-issue
Harnesses: https://stackblitz.com/fork/harness-issue

Steps to reproduce:
1.
2.
Expected Behavior
What behavior were you expecting to see?
Actual Behavior
What behavior did you actually see?
Environment

Angular:
CDK/Material:
Browser(s):
Operating System (e.g. Windows, macOS, Ubuntu):

Closing this issue as it does not follow the issue template. Please file a new issue that follows the template, which gives the team the information needed to investigate.


Add ability to subscribe on settings changes

Hi,
I was trying to find event to react on settings change, unfortunately I didn't find it. It would be cool to have an ability to update some data in extension (which depends on settings), when settings have been changed.
To get some context. I'm trying to integrate mp3 player to vs code to listen music inside (https://github.com/acherkashin/yandex-music-extension). I have added setting "Rewind time", to allow user adjust rewind time. Consequently, When user change it I will need update some data (buttons tooltips)  which use settings data.
Besides that, There are credentials "username"/"password". When user change it I need to change playlists and other things which depends on it.
I will really appreciate If you suggest me solution or add such ability to listen settings change. Thanks!
My VS Code Version:
Version: 1.43.0 (user setup)
Commit: 78a4c91400152c0f27ba4d363eb56d2835f9903a
Date: 2020-03-09T19:47:57.235Z
Electron: 7.1.11
Chrome: 78.0.3904.130
Node.js: 12.8.1
V8: 7.8.279.23-electron.0
OS: Windows_NT x64 10.0.18363

Version has been added.

See - https://github.com/microsoft/vscode-extension-samples/tree/master/configuration-sample


Normalise all version of react

Summary
Currently running yarn in the jest repo prints warning<EMAIL_ADDRESS>has incorrect peer dependency<EMAIL_ADDRESS>
Test plan
Running yarn in the jest repo should produce no peer dep warnings.

Codecov Report

Merging #3566 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3566   +/-   ##
=======================================
  Coverage   62.39%   62.39%
=======================================
  Files         181      181
  Lines        6646     6646
  Branches        6        6
=======================================
  Hits         4147     4147
  Misses       2496     2496
  Partials        3        3

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9b157c3...e85dbb8. Read the comment docs.


Usertype can't be found on fresh import

Just imported the package, everything is up to date, no other errors other than from yours, can't seem to find where it comes from, restarting or reimporting doesnt work.
Assets/Pokeyi/UdonSharp/OmniAction/P_OmniAction.cs(13,4): System.Exception: User type P_OmniAction could not be found

Can be solved temporarily by importing VRC-Haptics-Profile first. I will fix this to be a generic behaviour reference instead.

Fixed in v1.0.1.


Problem with normals / smoothing

Hi Torinos,
thanks for making this!
I still have a problem with the normals not behaving correctly.
I made 2 patches that demonstrate the issue.
Basically, the alembic version should look the same as the fbx model,
but as you can see there is something off.
It might be that I missed something in the materials,
in the 1st patch you can toggle between Stride and Fuse.
Maybe it would help to output the result of the _hasNormal test from here:
https://github.com/torinos-yt/VL.Alembic/blob/master/src/native/VL.Alembic.Native/abcrGeom.cpp#L266

if (!N.valid() || N.getNumSamples() <= 0 || N.getScope() == AbcGeo

So any help would be much appreciated, you can also reach me in the Element chat
if I can help with anything (user: rkt).
Thx, and have a great day!

alembic_problem.zip

Thanks for the issue along with the detailed research.
The main problem with this seems to be that the abc file being loaded does not contain normals. I have tried loading this alembic in both cinema4d and houdini, but neither of them read the normal information. Please check your settings when exporting the alembic first.
Also, as a separate issue, VL.Alembic calculates a flat normal from the vertices if the normal is not included in the alembic, but it seems that an inverted normal was set. This is why the inverted scale gives the correct looks.

VL.Alembic calculates a flat normal from the vertices if the normal is not included in the alembic, but it seems that an inverted normal was set.

This is fixed. 9273f30a5a4ebf61d4b4874e0a902048fa90473e
Close the issue once, but feel free to reopen it if it is not resolved.

Thanks for taking the time and the inverted fix!
For anyone else having problems with this:
what seems to help with the models to show up smooth instead of flat shaded is to apply 'Smooth Vectors' in Blender:


Could not resolve entry .tmp\app\main.prod.ts

Note: for support questions, please use one of these channels:
https://forum.ionicframework.com/
http://ionicworldwide.herokuapp.com/
Short description of the problem:
ionic run android --prod make my builld failed with error:
    [10:16:37]  rollup started ...
  [10:16:37]  rollup failed: Could not resolve entry (D:\xx\git\pr\application\xx\.tmp\app\main.prod.ts)
  [10:16:37]  ionic-app-script task: "build"
  [10:16:37]  Error: Could not resolve entry (D:\xx\git\pr\application\xx\.tmp\app\main.prod.ts)
  Error: Could not resolve entry (D:\xx\git\pr\application\xx\.tmp\app\main.prod.ts)
      at error (D:\xx\git\pr\application\xx\node_modules\@ionic\app-  scripts\node_modules\rollup\dist\rollup.js:170:12)
      at D:\xx\git\pr\application\xx\node_modules\@ionic\app-scripts\node_modules\rollup\dist\rollup.js:9284:5

What behavior are you expecting?
The build should work
Steps to reproduce:
1.
2.
3.
package.json
{
  "name": "ionic-hello-world",
  "author": "Ionic Framework",
  "homepage": "http://ionicframework.com/",
  "private": true,
  "scripts": {
    "ionic:build": "ionic-app-scripts build",
    "ionic:serve": "ionic-app-scripts serve"
  },
  "dependencies": {
    "@angular/common": "2.2.1",
    "@angular/compiler": "2.2.1",
    "@angular/compiler-cli": "2.2.1",
    "@angular/core": "2.2.1",
    "@angular/forms": "2.2.1",
    "@angular/http": "2.2.1",
    "@angular/platform-browser": "2.2.1",
    "@angular/platform-browser-dynamic": "2.2.1",
    "@angular/platform-server": "2.2.1",
    "@ionic/storage": "1.1.6",
    "@types/cropperjs": "0.0.22",
    "cropperjs": "^1.0.0-alpha",
    "ionic-angular": "2.0.1",
    "ionic-native": "2.2.11",
    "ionicons": "3.0.0",
    "ng2-translate": "^5.0.0",
    "ng2-ui-auth": "^6.0.0",
    "pouchdb": "^6.0.6",
    "pouchdb-upsert": "^2.0.2",
    "qrcode-generator": "^1.0.7",
    "rxjs": "5.0.0-beta.12",
    "sw-toolbox": "3.6.0",
    "zone.js": "0.6.26"
  },
  "devDependencies": {
    "@ionic/app-scripts": "^1.1.4",
    "rollup-plugin-commonjs": "^6.0.0",
    "typescript": "^2.1.5"
  },
  "description": "xx: An Ionic project",
  "cordovaPlugins": [
    "cordova-plugin-device",
    "cordova-plugin-console",
    "cordova-plugin-whitelist",
    "cordova-plugin-splashscreen",
    "cordova-plugin-statusbar",
    "ionic-plugin-keyboard"
  ],
  "cordovaPlatforms": [],
  "config": {
    "ionic_rollup": "./script/rollup.config.js",
    "ionic_sass": "./script/sass.config.js"
  }
}

Which @ionic/app-scripts version are you using?
latest
Other information: (e.g. stacktraces, related issues, suggestions how to fix, stackoverflow links, forum links, etc)
I went from app script 0.47 to 1.1.4 so maybe I forgot something. I added:
To go to 1.0.0 I added first
npm install sw-toolbox --save --save-exact
Then I did
npm install @ionic/app-scripts@latest --save-dev
I looked at:
https://github.com/driftyco/ionic/issues/8312
So message for ng2-translate has been done
The advice to go to last version (i am on it)
The advice to go to more recent typescript version (2.0.9) already done as I am in 2.1.5
I think I am no more in link with this issue has I removed the app.prod.ts app.dev.ts and created the app.main.ts like it was mentionned in the changelog of the 0.0.47
Here is the content:
import { platformBrowserDynamic } from '@angular/platform-browser-dynamic';

import { AppModule } from './app.module';

platformBrowserDynamic().bootstrapModule(AppModule);

My tsconfig.json is exactly the same as the one mentionned in the changelog 0.0.47
I did not touched to ionic_source_map
Do you maybe have an idea?
For information I also tried to downgrade all angular version to 2.1.0 and also to 2.0.1 (cli to 2.1.0) and I am getting same issue
my npm version 4.0.2
my node version 4.6.0
I created a startapp copied all my files did a npm install but still same issue
I created a startapp and just run ionic run android --prod and everything is working.
From what I can see the version of angular is the same.
So I suspect a code issuue (or maybe a version of typescript 2.1.x vs 2.0.x in the startapp) but I have no logs....

Found the issue!!!!
I overriden the rollup-config.js to handle pouchDB. However it was an old version (before app-script 0.0.47) But as I will remove it to use ionic storage I won't have problem anymore
You can close the issue.
By the way the --prod is wonderfull. Only 5 secondes to start an app with more than 60 pages
Thanks continue like this (3 secondes would be perfect :p)

Please update to a newer version of app-scripts. The .tmp folder went away.
Thanks,
Dan


WIP Docs: Update sample author name and email

Per https://groups.google.com/d/msg/pypa-dev/rUNsfIbruHM/LCEx-CB5AgAJ
the pypa-dev Google Group is now decommissioned.
Using this opportunity to, as in
pypa/sampleproject#118, improve the example author
name and email address.
Signed-off-by: Sumana Harihareswara<EMAIL_ADDRESS>

Since this is an example of an actual API response, we should probably make a new release of sampleproject and then update this to be what PyPI would actually return instead.

That sounds fair.

A new release of sampleproject has been made: https://pypi.org/project/sampleproject/2.0.0/


feat: cleanup HTML file

I delete the script inside the HTML and add a bit of styling for the background, it was quite bright

And now I can delete that because I took your code to my main branch?
Sorry for the question, but I don't have that much experience with this yet!

Your very much welcome, Benedikt. You can also close my branch, by the way, can you check your email, I have something to discuss with you and ofc not over github

I will answer you via email as soon as possible @LaansDole

Now I had time and answered you by mail!


Please use WTRLogger not NSLog

WTRApiClient.m Line 307 is using NSLog it should use WRTLogger

@kaydenthomson  This is fixed. Check latest version.


/opt/newrelic directory is being created owned by newrelic with permissions "0750"

Makes zero sense as I see the mode set to 0775 here:
https://github.com/escapestudios-cookbooks/newrelic/blob/master/providers/agent_java.rb#L34
However, here is what I end up with on my EC2 instances:
[root@ip-10-63-96-33 opt]# ll
total 16
drwxr-x--- 3 newrelic newrelic 4096 Aug 26 12:14 newrelic
[root@ip-10-63-96-33 opt]#

As a result, tomcat fails to start as it cannot even read the jar files.  Is the solution here to set these attributes to tomcat?
default['newrelic']['java_agent']['app_user'] = 'tomcat'
default['newrelic']['java_agent']['app_group'] = 'tomcat'

Hi @eric-tucker,
hmm, why do you see 750 instead of 755?
Yep, changing the app_user and app_group to 'tomcat' rather than 'newrelic' (default) should allow tomcat to be able to read the jar files and start...
Please do let me know how that goes!
Kind regards,
David

So here's what I did:

deleted /opt/newrelic
set the app user and group attributes to tomcat
run Chef

I ended up this time with:
drwxr-xr-x 3 root root 4096 Aug 28 11:56 newrelic

And /opt/newrelic/java:
drwxrwxr-x 3 tomcat tomcat 4096 Aug 28 11:56 java

In order words, it's all good.  The puzzling thing is how on first run the owner is "newrelic".
I actually cannot reproduce this issue after tomcat is installed.  I think what's happening is that if the Java agent is installed (via this cookbook) before Tomcat is installed, the tomcat user does not exist on the system so things get a little "messed" as far as directories go.
I think the issue is with my run list installing the agent before installing tomcat.  I think I can fix the issue by swapping the run list order.
I will close this issue if this proves to work for first time run (when tomcat not installed yet).

I'd suggest to change the order - the server monitor can obviously be installed pretty soon, but the actual application monitor I'd only install once the app server-bits are all in place... (well, that's how we roll anyway ;-) )
Please do keep me updated!

Unfortunately changing the order did not help.  I still get this weird behaviour on the parent directory:
[root@ip-10-63-96-19 ec2-user]# ll /opt
total 16
drwxr-x--- 3 newrelic newrelic 4096 Aug 28 12:20 newrelic
[root@ip-10-63-96-19 ec2-user]# ll /opt/newrelic/
total 4
drwxrwxr-x 2 tomcat tomcat 4096 Aug 28 12:20 java

Any suggestions? It seems like the Chef directory resource doesn't set permissions on parent directories correctly when using recursive true.

Looks like this was complained about before:
https://tickets.opscode.com/browse/CHEF-1621
It just makes no sense that in this case the parent directory is being created 750 and owned by "newrelic" rather than 755 and owned by root.

Thanks for adding in the reference to that Chef ticket here.
The docs of the directory resource also highlight this behaviour: "For the owner, group, and mode properties, the value of this attribute applies only to the leaf directory."...

So in a way it's working as expected.
However, it doesn't explain the parent getting randomly owned by newrelic and set to 750.  :(

FYI: since I use a wrapper cookbook for newrelic, I ended up adding this resource call to my wrapper around the java_agent recipe call (just before):
directory "/opt/newrelic" do
  owner "root"
  group "root"
  mode 0755
end

I got by for now but perhaps this needs to be incorporated into your cookbook when you have the chance.

Just pasting the error that I got in my catalina.out log because of this problem, in case it helps anyone else:
Error opening zip file or JAR manifest missing : /opt/newrelic/java/newrelic.jar
Error occurred during initialization of VM
agent library failed to init: instrument

It almost seems like it's a problem of java itself though, as I see no reason why a higher level directory of /opt/newrelic/java need to be readable, so long as /opt/newrelic/java itself (and that jar file) are readable. Definitely left me spinning my wheels confused for awhile :)

I believe the issue here is that the agent_java provider is using root as hardcoded ownerships, causing tomcat (for example) to not be able to access the file. Though I guess you could change the permissions, I feel its cleaner to change the actual ownership.

Hmm. Does still work for you? /opt/newrelic is owned by newrelic with 750 and there goes app launch... Wrapper cookbook needed still...


(PARTIAL!) Working nocollison tag

This is ported from the C# branch
I cannot tell if there is a performance hit, though I am effectively creating two meshes: one for the visual representation and one for the collision. I will probably have to figure out how to do collision channels in Godot and move the visual one to a different channel and set the line trace you use so that the line trace detects that new channel. If you know how to do that feel free to make the required changes here and then merge them.
Currently, the blocks that are marked with the no collision tag will not interact with the player at all, that includes destroying and replacing them, once they are generated they are there and there is nothing the player can do, hence why I said partial.

Makes sense, much better than using the create_trimesh method (not supposed to be used outside of testing)

I am using that at the moment (the collision is an invisible mesh) that being said I am pretty open to looking for alternatives, the thing is no matter what you do creating collisions at runtime is not cheap, and creating a Complex collision mesh is, even more, the case. Unfortunately, you have no choice in a game like Minecraft but to do such a thing. Terrain generation in general is pretty bad in this regard.

Rather than creating 2 meshes with one that also has the non-collision meshes, wouldnt it be quicker to create 2 meshes where the first one contains blocks which have collisions, and the second mesh has blocks that do not collide? It might also be good to still create a collision mesh for the non-collision blocks, to allow raycasts to collide with them, just make sure its on a layer that doesnt collide with the player. Currently the player can't destroy flowers and stuff.

I have it figured out in the C# branch and will push eventually


Navbar padding of text

This occurs at certain resolutions and is rather annoying. Use a media query or something to fix it.

This was fixed in commit f11c340


Add Danish translation file

Hello, thanks a lot for the danish translations! Merging :)

Hello, thanks a lot for the danish translations! Merging :)

No problem, thanks for being so fast with merging :)


-[UIBarButtonItem tap] passes itself as sender

Fixes crash when the action on the target receives the sender
Looks like the #125 which was merged yesterday crashes in this situation.

Thanks!


Broker_WorldQuests/WorldQuests.lua:272: Usage: HaveQuestData(questID

24x Broker_WorldQuests/24x Broker_WorldQuests/WorldQuests.lua:272: Usage: HaveQuestData(questID
[string "=[C]"]: in function `HaveQuestData'
[string "@Broker_WorldQuests/WorldQuests.lua"]:272: in function <Broker_WorldQuests/WorldQuests.lua:257>
[string "@Broker_WorldQuests/WorldQuests.lua"]:950: in function `UpdateQuestData'
[string "@Broker_WorldQuests/WorldQuests.lua"]:1135: in function `UpdateBlock'
[string "@Broker_WorldQuests/WorldQuests.lua"]:1120: in function `RunUpdate'
[string "@Broker_WorldQuests/WorldQuests.lua"]:1604: in function <Broker_WorldQuests/WorldQuests.lua:1596>

debug settings only give this:
[BWQ] questList.1: ID: nil (mapId: 2024)

WorldQuests.zip
Replace your WorldQuests.lua file with this one (attached) and see if that resolves the issue.   It's involving a past expansion so I don't want to take the time to track down "why" it's happening in that particular zone, etc.   But, this adds a sanity check that should have it skip the routine that it's trying to do when the data isn't there.

https://github.com/smaitch/Grail/issues/365
looks like the upper case change affects various addons on the ptr

Well, you didn't mention you were on the PTR....    In the future, please state this as I am expecting bug reports and other requests to be referencing the Live client.   I would obviously fix it myself once I saw it on the live client.
But, I'll go ahead and look into what's changed with the PTR and make that adjustment so that it's fixed.

As stated above, I'm not supporting the PTR with this addon.   If this issue persists once the changes go Live, I'll fix it at that point.


[Bug]: Browser crashes immediately when running with pm2 in cluster mode

Minimal, reproducible example
// Note: This script won't fail on its own. See https://github.com/ericman314/puppeteer-test

const express = require('express')
const http = require('http')
const puppeteer = require('puppeteer')

const app = express()
const server = new http.Server(app)
server.listen(3010, () => {
  console.log('Server listening')
})

app.get('/hello', (req, res) => {
  console.log('hello')
  res.send('
Hello, world!
')
})

app.get('/render.png', (req, res) => {
  console.log('render')
  doRender()
    .then(image => {
      res.setHeader('content-type', 'image/png');
      res.send(image)
    })
    .catch(err => {
      console.error(err)
    })
})

async function doRender() {
   const browser = await puppeteer.launch({ headless: 'new' })
   const page = await browser.newPage()
   await page.setViewport({ width: 600, height: 400 })
   const url = `http://localhost:3010/hello`
   await page.goto(url)
   await page.waitForNetworkIdle();
   let image = await page.screenshot()
   await browser.close()
   return image
}

Error string
TargetCloseError: Protocol error (Target.setAutoAttach): Target closed
Bug behavior

[x] Flaky
[ ] PDF

Background
I have a server with two endpoints: one that serves a page, and a second that uses puppeteer to launch a headless browser, fetch that page, and take a screenshot. (Link to repo: https://github.com/ericman314/puppeteer-test) All of that works perfectly fine, most of the time.
Where things start breaking is when I run the server on a Ubuntu server, using pm2, in cluster mode. This is a tricky problem because I can't reproduce it on my local development machine (also ubuntu), and it only happens when running using pm2 in cluster mode. If pm2 runs in fork mode, it doesn't break.
Summary of when it breaks:
Running using pm2 in cluster mode on my local dev machine: Works
Running using pm2 in fork mode on my headless server: Works
Running using pm2 in cluster mode on my headless server: Breaks
To be honest, I don't know if the bug is with puppeteer, pm2, or chromium, or if it's a problem with the environment on my ubuntu server. I'm just hoping to get some information from the experts here as to what could possibly be going on. I can provide logs if requested.
Expectation
Render and output this image:

Reality
The browser appears to crash moments after starting:
9|image-server  | TargetCloseError: Protocol error (Target.setAutoAttach): Target closed
9|image-server  |     at CallbackRegistry.clear (/home/dev/puppeteer-test/node_modules/puppeteer-core/lib/cjs/puppeteer/common/Connection.js:138:36)
9|image-server  |     at Connection.#onClose (/home/dev/puppeteer-test/node_modules/puppeteer-core/lib/cjs/puppeteer/common/Connection.js:282:25)
9|image-server  |     at WebSocket.<anonymous> (/home/dev/puppeteer-test/node_modules/puppeteer-core/lib/cjs/puppeteer/common/NodeWebSocketTransport.js:57:30)
9|image-server  |     at callListener (/home/dev/puppeteer-test/node_modules/ws/lib/event-target.js:290:14)
9|image-server  |     at WebSocket.onClose (/home/dev/puppeteer-test/node_modules/ws/lib/event-target.js:220:9)
9|image-server  |     at WebSocket.emit (node:events:390:28)
9|image-server  |     at WebSocket.emitClose (/home/dev/puppeteer-test/node_modules/ws/lib/websocket.js:258:10)
9|image-server  |     at Socket.socketOnClose (/home/dev/puppeteer-test/node_modules/ws/lib/websocket.js:1264:15)
9|image-server  |     at Socket.emit (node:events:390:28)
9|image-server  |     at TCP.<anonymous> (node:net:687:12)

Also a huge core file is output:

Puppeteer configuration file (if used)
No response
Puppeteer version
20.9.0
Node version
16.13.0
Package manager
npm
Package manager version
8.1.0
Operating system
Linux

I checked "flaky" above to satisfy the bot, but the bug definitely happens consistently. The bot probably won't be able to reproduce it.

Thanks for the suggestion!


Inject not wiring

The following code, partially extracted from the docs:
from antidote import implementation, Service, inject

class Database:
    pass

class PostgresDB(Service, Database):
    def __repr__(self):
        return 'Postgres'

class MySQLDB(Service, Database):
    def __repr__(self):
        return 'MySQL'

@implementation(Database, permanent=True)
def choose_db():
    return PostgresDB

@inject
def print_db(db: Database):
    print(db)

print_db()

Throws the following error:

Hello ! It's not a bug, it's a feature. :D
Had it worked, you would have no way of knowing where the Database is coming from when using print_db(). So Antidote forces you to be more explicit by forcing you to specify from where the dependency is actually coming from. Example:
@inject([Database @ choose_db])
def print_db(db: Database):
    print(db)

Or with type hint annotations:
from antidote import From
from typing import Annotated

@inject
def print_db(db: Annotated[Database, From(choose_db)]):
    print(db)

If you use it often, you could also use a type alias:
MyDatabase = Annotated[Database, From(choose_db)]

@inject
def print_db(db: MyDatabase):
    print(db)

Hope that helps ! If not, happy to provide more help.
Could you please tell me how you came up with your example ? I'd love to know which parts of the documentation / readme need to be improved or are missing. Or if you simply didn't read a section because it was too long, simple or complex. :)

Oh I see now the issue with the documentation. Indeed the "How to / Use interface" is wrong. I'm surprised it passes the doctests, I probably mixed things.

Fixed they were two issues:

stable documented seems to have stopped updating at v0.10 for an unknown reason.
the interface recipe in the documentation did need some refresh.

Let me know if this wasn't the source of your issue !

Closing, the issue is fixed.


multiple csv files

Hi, lightning pose team
lightning pose is a great tool. and it is very important to my project.
I used to use deeplabcut before, and now I have multiple videos with multiple csv files.
How can I move on with lightning pose?
should I put all the csv files into one or should I train one model with one csv file?
Hoping for your suggestions.

Hi @Wulin-Tan, thanks for checking out Lightning Pose! If you would like to convert a DLC project to Lightning Pose, please see the docs here: https://lightning-pose.readthedocs.io/en/latest/source/user_guide/directory_structure.html#converting-dlc-projects-to-lightning-pose-format
Please let me know if you run into any issues with this, or if any part of the documentation is unclear.

Hi @Wulin-Tan, thanks for checking out Lightning Pose! If you would like to convert a DLC project to Lightning Pose, please see the docs here: https://lightning-pose.readthedocs.io/en/latest/source/user_guide/directory_structure.html#converting-dlc-projects-to-lightning-pose-format
Please let me know if you run into any issues with this, or if any part of the documentation is unclear.

yes,with the converting script,it works perfect! Thank you!


Enable End to end tests to run against local-test-server, tinylicous, or routerlicious based on config

https://github.com/microsoft/FluidFramework/pull/3612 is a step in the right direction. In it we have copied 3 basic tests and configured them to run against r11s, and have them set up to run in the Client Build CI. My sights are set on doing what this issue says though, and having all e2e tests configurable to run against the different servers we want to ensure support for (including ODSP, which I'm working on now).


Every split gone after BSOD

Got a BSOD Win 10 "memory_management" watching youtube and having ch2 and discord open.
All splits and tabs got deleted

should :b:  fixed

Just happened to me...

Has happened to me multiple times whenever I get a video driver crash. Is there no way to back up your splits?

Thank you @Mm2PL, I missed that in the wiki. That should be a perfect workaround, much appreciated!


Update pinot assembly scripts for release 0.3.0

This is required for release 0.3.0
mvn release command doesn't honor output relocation, so we need to manually specify those in assembly scripts for pinot-plugins.

Whenever we add the new plugin and make it as part of the distribution, we will need to add the line here. We can periodically double check on this as part of the release process (similar that we clean up license for every release).
Can we update the release doc to include this? Otherwise, LGTM! Thank you for working on this.

Whenever we add the new plugin and make it as part of the distribution, we will need to add the line here. We can periodically double check on this as part of the release process (similar that we clean up license for every release).
Can we update the release doc to include this? Otherwise, LGTM! Thank you for working on this.

Yes, also for required for plugin developer, this is how they could package plugin into the distribution. I will add this into not only release process doc, but also dev guide.


Routes display a blank page after set a resource with the same controller before

Hi everyone,
According that we have a resource users in our routes file:
Route::resource('users', 'UsersController');

If we set another routes related to the user resource and use same method on it like that:
Route::get('users/edit', [
    'as' => 'users.self.edit',
    'uses' => 'UsersController@edit'
]);

Route::post('users/edit', [
    'as' => 'users.self.update',
    'uses' => 'UsersController@update'
]);

It's doen't work and display a blank page if these new routes is written after our resource.
But if I write these routes before all things work fine.
Route::get('users/edit', [
    'as' => 'users.self.edit',
    'uses' => 'UsersController@edit'
]);

Route::post('users/edit', [
    'as' => 'users.self.update',
    'uses' => 'UsersController@update'
]);

Route::resource('users', 'UsersController');

Why it would be the intended behaviour? Btw I can't sort properly my routes file.

I believe this is intended behavior as well. In my experience if I ever needed to add extra routes to the same controller attached to a Route::resource, I've had to register them before I've registered the Route::resource. Unfortunately I'm not sure why, but I can't dive into the code ATM

Route::resource('/trainer','TrainerController');
i have same problem here.

Route::resource('/trainer','TrainerController');
i have same problem here.

did you solved the issue coz i am facing also the same issue


Fix generated #line directives on Windows

This avoids inadvertently using the backslash path separator in the generated #line directives on Windows. Using a backslash in a source file directive can cause compilation errors from using incorrect escape sequences and doesn't have the desired effect regardless of whether or not the compiler accepts the input.

Thanks!


Add pipeline to download snapshots

@sbfnk notes that the key thing for now will be to set up an automated download for snapshots (assuming past ones aren't available)

I agree that is indeed the key thing!

Just found this: https://gist.github.com/spoonerf/5a76e743d603c6e8e15f003b14165226.

And this https://github.com/owid/monkeypox, especially the details in the README

The global.health direct data link (from website) is also here: https://mpox-2024.s3.eu-central-1.amazonaws.com/latest.csv – but not sure about update frequency.

@seabbs do you know of similar work for other data pipelines (guessing some exist for covid-19).

I know of internal CFA ones (i.e. not public) and the related ones from Global.health but not otherwise

I only see the snapshots from 2022/23 here: https://7rydd2v2ra.execute-api.eu-central-1.amazonaws.com/web/archive. Not sure they are doing same for the current outbreak.

On a related note, I wondered if there's a package that can handle this e.g. point to a link or folder and automatically version and standardise snapshots? A bit like orderly2 but for data streams.


Fix various astro issues

fix https://github.com/microsoft/typespec/issues/4831

No changes needing a change description found.

You can try these changes here

🛝 Playground
🌐 Website
📚 Next docs