Implement adaptive spin lock on ARM Apple devices

Updated spin lock to use adaptive synchronization as suggested. Normally unfair lock will block, which defeats the purpose of spin locks.
"As it turns out os_unfair_lock doesn’t spin on contention, it makes the calling thread sleep right away when it finds a contended lock."
https://hacks.mozilla.org/2022/10/improving-firefox-responsiveness-on-macos/

CC @stuartcarnie Who added os_unfair_lock

Thanks for the PR, @mrsaturnsan!
Unfortunately, this is a private API, which could affect our iOS deployments and possibly result in rejected builds to the App Store.
Further, the original post was on Hacker News, and a fellow software engineer whom I trust, Saagar Jha posted this comment about the API, and warns against using it, as it is private for a reason. I have previously spoken to Apple engineers about the os_unfair_lock API, and they say it is generally the right approach to replace a spin lock on Apple platforms.

@mrsaturnsan excellent – if you have used it, and Apple has made the API public in the latest OS, that is certainly an endorsement.

@clayjohn LGTM!

AFAIK we support as far back as MacOS 10.13 and iOS 12.0, would using those older version be a problem with this PR?
CC @bruvzg

AFAIK we support as far back as MacOS 10.13 and iOS 12.0, would using those older version be a problem with this PR?

We do, supporting both probably can be done using if (__builtin_available(macOS 15, iOS 18, *)) { ... } (since it's only public since macOS 15 / iOS 18, it probably should be limited to these versions).

The problem with if (__builtin_available(macOS 15, iOS 18, *)) { ... } is that it's a runtime check so using the new function will cause compile errors when targeting older OS versions.

Usually compiling with old compiler is not important, we only want to support running on old OSes. And we already have @available check for macOS 13.4 and iOS 16. But I guess if it's tested on 10.15, available check can be 10.15 as well (not sure which iOS versions are supported).

I think this would be the safest option.

Yes, it should be OK.

Will need to be squashed again

Some time ago, I added a benchmark that works as an indirect test of SpinLock: https://github.com/godotengine/godot-benchmarks/pull/95. I've used it in my own non-Mac spin-lock improvement PR: https://github.com/godotengine/godot/pull/85167.
It'd be amazing if we could get some numbers from a test based on that (maybe with different amounts of threads) with the following SpinLock implementations, on some ARM-based Mac or iPhone:

Current without Apple-specific implementation.
The same, from #85167.
Apple-specific before this PR.
Apple-specific after this PR.

I know it's a lot to ask, so if we could at least compare before'n'after this PR on MacOS, that'd be enough here.

Maybe the TPS demo? https://github.com/godotengine/tps-demo I suspect that it won't have much thread contention as it is mostly single threaded though

Normally unfair lock will block, which defeats the purpose of spin locks.

They don't block if they are not contested, which is probably likely under normal conditions. This is also evident in the single threaded benchmark result, where both are almost identical in performance.
I reformatted your results into a table, so they are easier to read and compare (I don't know the time units):

Benchmark
Before
After
± change

core/object_db/single
18.65
17.71
1.05

core/object_db/2_threads_full_contention
59.71
51.29
1.16

core/object_db/2_threads_half_contention
38.24
34.03
1.12

core/object_db/2_threads_little_contention
22.54
22.25
1.01

core/object_db/4_threads_full_contention
180.6
173.6
1.04

core/object_db/4_threads_half_contention
101.4
95.34
1.06

core/object_db/4_threads_little_contention
35.11
32.9
1.07

core/object_db/4_threads_slope_contention
135.5
127.7
1.06

core/object_db/8_threads_full_contention
516.4
785.5
0.66

core/object_db/8_threads_half_contention
299.4
461.2
0.65

core/object_db/8_threads_little_contention
74.33
102.2
0.73

core/object_db/8_threads_slope_contention
314.4
410.2
0.77

core/object_db/12_threads_full_contention
1317.0
1991.0
0.66

core/object_db/12_threads_half_contention
643.7
1055.0
0.61

core/object_db/12_threads_little_contention
138.5
204.6
0.68

core/object_db/12_threads_slope_contention
586.1
1017.0
0.58

My interpretation is that it is slightly better when there are few threads (2 - 4 threads), but as the thread count increases, the worst case is much worse than the current implementation. This is likely a combination of number of available CPUs, and overall system activity, and so blocking is better, as it forces the thread to yield sooner to give the other threads a chance to make progress.
I suspect if you have a 16 core Mac, you might see the improved performance extend to 8 and possibly even 12 threads.
TL;DR
I'd argue that although best case is slightly better, worst case is significantly worse.


apis.errs.Also does not propagate error level

/area API
/kind bug
Steps to Reproduce the Problem
Using "warning" level error introduced in #2498:
func TestFoo(t *testing.T) {
	var errs *apis.FieldError
	errs = errs.Also(apis.FieldError("foo").At(apis.WarningLevel))
	if errs.Level != apis.WarningLevel {
		t.Errorf("Expected an error at warning level but got level %s", errs.Level)
	}
}

Expected Behavior
The aggregated error has the level that is the highest level of any errors combined with Also.
In the example above, I would expect errs.Level to be Warning, and errs.Filter(apis.WarningLevel).Level to also be Warning.
Actual Behavior
In the example above, errs.Level and errs.Filter(apis.WarningLevel).Level are both Error. (i.e. an error at level "warning" aggregated with no error turns into an error at level "error")
@mattmoor

Thanks for opening this 🙏
cc @dprotaso @evankanderson @vaikas @n3wscott
To reiterate some of the thread in Tekton, this is sort of a reflection of the awkward conflation of apis.FieldError as both an error and an aggregation of errors (via .Also())
In your example, errs.Message won't match the single error it contains either, so I think the breakdown here is more general than the new Level field.  I wonder if for aggregated errors we should set Level to reflect the constituents and perhaps add a Mixed level when multiple levels are present?  This would at least make the cases @lbernick describes a bit more intuitive.  I'd guess it would avoid back-compat problems since nothing is really using anything but error yet, so nothing should get a mixed level with .Also.
What do folks think?

@mattmoor you have a link to Tekton thread handy since it seems to have more context?

I'd expect the "lowest" level when aggregating via Also (since ErrorLevel is 0 and counts upward), but otherwise I agree with the diagnosis.
Given that this code is unlikely to be in a tight loop, I wonder if looping over the set of contained errors to determine the correct collection-level would be correct. You could early-exit the loop if ErrorLevel is encountered, since that's the lowest level. If we imagine a Suggestion level that's less severe than Warning:

Err1
Err2
Alsoed (symmetrical)

error
error
error

error
warning
error

error
suggestion
error

warning
warning
warning

warning
suggestion
warning

suggestion
suggestion
suggestion

The worst case performance would be constructing a long chain of Warning level errors, where you'd end up scanning O(n^2) errors as you add one to the head of the chain, but this is probably acceptable (and maybe we'd cache things so it's actually O(n), I haven't thought too deeply).

/assign

Thanks @evankanderson for the table of the combinations of errs incurred.
I would like to confirm my understanding on the O(n) solution, would  that need to introduce a hash-table in a (possible cyclic?) graph?

You shouldn't need a hash table for caching, as you only need to track the "head" of the chain as long as the cache is correct. Using .Also() in any sort of correct issue should produce a tree, not an arbitrary graph.
If you want, you could use the go benchmark library to compare approaches if you want to learn that...


Feature/116 iam groups

Description
Fixes #116.
Adds the ability to nuke IAM Groups
TODOs
Read the Gruntwork contribution guidelines.

[x] Update the docs.
[x] Run the relevant tests successfully, including pre-commit checks.
[x] Ensure any 3rd party code adheres with our license policy or delete this line if its not applicable.
[x] Include release notes. If this PR is backward incompatible, include a migration guide.

Release Notes (draft)
Added support for nuking IAM Groups and customer managed IAM Policies
Migration Guide

Looks like not just a terrific start, but ultimately pretty close to being merge-able! :clap:

This is pretty non-intuitive, but every time you add a new resource to cloud-nuke, you need to go [here[(https://github.com/ellisonc/cloud-nuke/blob/c75b348b9c33dc6e83ae9d5993a1eb5729feda91/config/config_test.go#L12) and add the exact line ResourceType{FilterRule{}, FilterRule{}}, the same number of times as you added resources.
So, in this case you'll need to add that line twice - otherwise you'll get a warning when trying to run tests about too few resources being defined in the config test.

Did a bunch of smoke testing locally:

Creating and nuking IAM users
Creating and nuking IAM users while excluding a special user via config file
Create and nuking IAM users that are members of groups with and without policies attached
Creating and nuking IAM policies while excluding a special policy via config file

All have worked flawlessly. Well done :+1:

@ellisonc Very nice work. This looks solid to me. Want to resolve the merge conflicts so I can approve?


⚠ Samsung has degraded performance

In 2283dd9, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 1852 ms

Resolved: Samsung performance has improved in bc7e333.


Validate configuration before starting a node

We need to validate configuration before starting a node. We currently have a preliminary version of this. Review and improve it as much as possible.
Check this: https://github.com/buraksezer/olric/issues/9#issuecomment-588569127

Current implementation of Config.Validate, Config.Sanitize and Config.New is good enough to me.  config.New(<ENV>) generates a config for your selected environment with sane defaults. It can be modifiable after created.
If one has any further questions or problems, please comment this.


Changes to support Jubilinux v0.3.0 (stretch)

(Mostly cosmetic changes right now)

works also on jubilinux v0.4.1
:+1:

Is this still a WIP?

I think I'm going to close it in favor of a version of #1241 for dev.

(Superseded by https://github.com/openaps/oref0/pull/1243)


Move the python directory of arrow-ballista to the new arrow-ballista-python repo

We still need to add CI, dev, .github etc and update README.md but at least the python directory has been moved.

@andygrove

I think this is fine to unblock the development of Ballista core, but I we should at least enable the RAT check in this PR to ensure that no code gets checked in without the appropriate license. It would also be good to file an issue for the remaining tasks so that others can pick those up.
Thanks for taking the lead on doing this!


Published license identifier is not compliant with the SPDX license list

As for now the license is getting published as BSD 3 but according to https://spdx.org/licenses/BSD-3-Clause.html the  SPDX-identifier would be BSD-3-Clause.
It would be nice to align it to the SPDX-identifier, because this makes it easier for tools like e.g. Gradle plugins to detect the used license by relying on the SPDX license identifiers.

Fixed in #1538.


Floats and Parsley validation

I try to implement floats and also digits. When I have numbers for the price in USA format it will not be the correct code. How to manage floats or digits. An example:
The price can be 10 USD, 10.55 not 10.5 USD. As I understand it will work required just floats using type="text" and data-parsley-type="number", but how to solve max. two decimals?
Is this working and the correct?
 data-parsley-pattern="[0-9]*(\.?[0-9]{2}$)" using type="text" and data-parsley-type="number"
Need help.

Can be suggested exact demo to work floats? Maybe I missed...

I'm afraid I'll have to close this. First, this is a question and thus does not belong here, second it's not even clear what issue you are having with your solution.


关于Pools 交互问题 和glaxe 任务问题

Describe the bug
在pools交互问题上  没有提示 并且交互gas 到1%  交易时间也非常久远
glaxe的任务 mint the taiko entry pass 打开网页错误

Steps to reproduce
Steps to reproduce here.
Spam policy

[X] I verify that this issue is NOT SPAM and understand SPAM issues will be closed and reported to GitHub, resulting in ACCOUNT TERMINATION.

English only please. Reach out on discord for campaign related problems.


WIP: Use matrix for metal3 image building

Use matrix to avoid repetition of code for the image building pipelines.
TODO:

[ ] Refactor scripts for ubuntu/centos so they take the same arguments

Closing in favor of #599


Max retries error with BigtableClient

For reference, pertains to tf.contrib.cloud.bigtable_api.py and BigtableClient.
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Not really

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A

TensorFlow installed from (source or binary): binary

TensorFlow version (use command below): 'v1.13.1-0-g6612da8951' 1.13.1

Python version: 3.6.4

Bazel version (if compiling from source): N/A

GCC/Compiler version (if compiling from source): N/A

CUDA/cuDNN version: None

GPU model and memory: None

Describe the current behavior
Dataset constructed by way of tf.contrib.cloud.BigtableClient raises error that is primarily the following:
InternalError (see above for traceback): Error reading from Cloud Bigtable: No more retries allowed as per policy.
	 [[node IteratorGetNext (defined at <ipython-input-1-b90b20ec4e01>:10) ]]

This occurs both in graph and eager mode; occurs on a machine where the conventional bigtable python client apis function properly (appropriately credentialed); persists after alternatively activating a service account with the appropriate credentials via gcloud auth activate-service-account ....
Describe the expected behavior
Iterator yields training examples from Dataset when iterated over without error.
Code to reproduce the issue
from pcml.operations.tfrecord2bigtable import BigTableSelection
import tensorflow as tf

data = BigTableSelection(
    project="clarify",
    instance="clarify-cbt-instance",
    table="clarify-cbt-devtabl1",
    prefix="train",
    column_family="tfexample",
    column_qualifier="example")

client = tf.contrib.cloud.BigtableClient(data.project, data.instance)

table = client.table(data.table)

ds = table.parallel_scan_prefix(
    data.prefix, columns=[(data.column_family, data.column_qualifier)])

ds_data = ds.map(lambda index, data: data)

ds_data = ds_data.repeat()

iterator = ds_data.make_initializable_iterator()

next_element = iterator.get_next()

sess = tf.Session()

sess.run(iterator.initializer)

_ = sess.run(next_element)

Other info / logs

WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.

INFO:tensorflow:Entry Point [tensor2tensor.envs.tic_tac_toe_env:TicTacToeEnv] registered with id [T2TEnv-TicTacToeEnv-v0]
WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:1419: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
---------------------------------------------------------------------------
InternalError                             Traceback (most recent call last)
/opt/conda/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
   1333     try:
-> 1334       return fn(*args)
   1335     except errors.OpError as e:

/opt/conda/lib/python3.6/site-packages/tensorflow/python/client/session.py in _run_fn(feed_dict, fetch_list, target_list, options, run_metadata)
   1318       return self._call_tf_sessionrun(
-> 1319           options, feed_dict, fetch_list, target_list, run_metadata)
   1320

/opt/conda/lib/python3.6/site-packages/tensorflow/python/client/session.py in _call_tf_sessionrun(self, options, feed_dict, fetch_list, target_list, run_metadata)
   1406         self._session, options, feed_dict, fetch_list, target_list,
-> 1407         run_metadata)
   1408

InternalError: Error reading from Cloud Bigtable: No more retries allowed as per policy.
	 [[{{node IteratorGetNext}}]]

During handling of the above exception, another exception occurred:

InternalError                             Traceback (most recent call last)
<ipython-input-1-acb694909ee3> in <module>
     29
     30 sess.run(iterator.initializer)
---> 31 _ = sess.run(next_element)
     32

/opt/conda/lib/python3.6/site-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
    927     try:
    928       result = self._run(None, fetches, feed_dict, options_ptr,
--> 929                          run_metadata_ptr)
    930       if run_metadata:
    931         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/opt/conda/lib/python3.6/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
   1150     if final_fetches or final_targets or (handle and feed_dict_tensor):
   1151       results = self._do_run(handle, final_targets, final_fetches,
-> 1152                              feed_dict_tensor, options, run_metadata)
   1153     else:
   1154       results = []

/opt/conda/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1326     if handle is None:
   1327       return self._do_call(_run_fn, feeds, fetches, targets, options,
-> 1328                            run_metadata)
   1329     else:
   1330       return self._do_call(_prun_fn, handle, feeds, fetches)

/opt/conda/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
   1346           pass
   1347       message = error_interpolation.interpolate(message, self._graph)
-> 1348       raise type(e)(node_def, op, message)
   1349
   1350   def _extend_graph(self):

InternalError: Error reading from Cloud Bigtable: No more retries allowed as per policy.
	 [[node IteratorGetNext (defined at <ipython-input-1-acb694909ee3>:26) ]]

Caused by op 'IteratorGetNext', defined at:
  File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()
  File "/opt/conda/lib/python3.6/site-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/opt/conda/lib/python3.6/site-packages/ipykernel/kernelapp.py", line 486, in start
    self.io_loop.start()
  File "/opt/conda/lib/python3.6/site-packages/tornado/platform/asyncio.py", line 127, in start
    self.asyncio_loop.run_forever()
  File "/opt/conda/lib/python3.6/asyncio/base_events.py", line 421, in run_forever
    self._run_once()
  File "/opt/conda/lib/python3.6/asyncio/base_events.py", line 1431, in _run_once
    handle._run()
  File "/opt/conda/lib/python3.6/asyncio/events.py", line 145, in _run
    self._callback(*self._args)
  File "/opt/conda/lib/python3.6/site-packages/tornado/platform/asyncio.py", line 117, in _handle_events
    handler_func(fileobj, events)
  File "/opt/conda/lib/python3.6/site-packages/tornado/stack_context.py", line 276, in null_wrapper
    return fn(*args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 450, in _handle_events
    self._handle_recv()
  File "/opt/conda/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 480, in _handle_recv
    self._run_callback(callback, msg)
  File "/opt/conda/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 432, in _run_callback
    callback(*args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/tornado/stack_context.py", line 276, in null_wrapper
    return fn(*args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher
    return self.dispatch_shell(stream, msg)
  File "/opt/conda/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 233, in dispatch_shell
    handler(stream, idents, msg)
  File "/opt/conda/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 399, in execute_request
    user_expressions, allow_stdin)
  File "/opt/conda/lib/python3.6/site-packages/ipykernel/ipkernel.py", line 208, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "/opt/conda/lib/python3.6/site-packages/ipykernel/zmqshell.py", line 537, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "/home/jovyan/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2843, in run_cell
    raw_cell, store_history, silent, shell_futures)
  File "/home/jovyan/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2869, in _run_cell
    return runner(coro)
  File "/home/jovyan/.local/lib/python3.6/site-packages/IPython/core/async_helpers.py", line 67, in _pseudo_sync_runner
    coro.send(None)
  File "/home/jovyan/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3044, in run_cell_async
    interactivity=interactivity, compiler=compiler, result=result)
  File "/home/jovyan/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3209, in run_ast_nodes
    if (yield from self.run_code(code, result)):
  File "/home/jovyan/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3291, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-1-acb694909ee3>", line 26, in <module>
    next_element = iterator.get_next()
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 414, in get_next
    output_shapes=self._structure._flat_shapes, name=name)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 1685, in iterator_get_next
    output_shapes=output_shapes, name=name)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
    op_def=op_def)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
    self._traceback = tf_stack.extract_stack()

InternalError (see above for traceback): Error reading from Cloud Bigtable: No more retries allowed as per policy.
	 [[node IteratorGetNext (defined at <ipython-input-1-acb694909ee3>:26) ]]

Further suggesting this is a credential issue the following modification to use keys_by_prefix_dataset gives an explicit error about credentials.

import os
import tensorflow as tf
from pcml.operations.tfrecord2bigtable import BigTableSelection

data = BigTableSelection(
    project="clarify",
    instance="clarify-cbt-instance",
    table="clarify-cbt-devtable1",
    prefix="train",
    column_family="tfexample",
    column_qualifier="example")

client = tf.contrib.cloud.BigtableClient(data.project, data.instance)
table = client.table(data.table)
key_dataset = table.keys_by_prefix_dataset("train")
ds = key_dataset.apply(table.lookup_columns(("cf1", "tfexample")))
ds_data = ds.map(lambda index, data: data)
ds_data = ds_data.repeat()

iterator = ds_data.make_initializable_iterator()
next_element = iterator.get_next()

sess = tf.Session()
sess.run(iterator.initializer)
_ = sess.run(next_element)

with error:
UnauthenticatedError (see above for traceback): Error reading from Cloud Bigtable: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
	 [[node IteratorGetNext_2 (defined at <ipython-input-26-f0eff0cf13a5>:36) ]]

This occurring with and without GOOGLE_APPLICATION_CREDENTIALS being set explicitly and verified application-default-credentials are indeed set (gcloud auth application-default print-access-token prints an access token).
As you can tell my intuition is that the issue has either to do with the compiled client op not properly discovering and using credentials or the client machine being mis-configured vis. credentials.

So this appears to be resolved (at least on my end) by resolving issues with both parameterization and environment credentials. In the former case the retries error can be produced if e.g. a non-existent table name or the wrong column family is provided. In the latter case I'm seeing the error occur if I don't unset GOOGLE_APPLICATION_CREDENTIALS which in my case is pointing to a default GCP service account (which in my case does not have the necessary scopes) instead of the service account that is activated and being used otherwise.

import os
import tensorflow as tf
from pcml.operations.tfrecord2bigtable import BigTableSelection

os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = ""

data = BigTableSelection(
    project="clarify",
    instance="clarify-cbt-instance",
    table="clarify-cbt-devtable",
    prefix="train",
    column_family="tfexample",
    column_qualifier="example")

client = tf.contrib.cloud.BigtableClient(data.project, data.instance)
table = client.table(data.table)

# ===
# Option 1
key_dataset = table.keys_by_prefix_dataset(data.prefix)
ds = key_dataset.apply(table.lookup_columns((data.column_family,
                                             data.column_qualifier)))
# ===
# Option 2
ds = table.parallel_scan_prefix(
    data.prefix, columns=[(data.column_family, data.column_qualifier)])
# ===

ds_data = ds.map(lambda index, data: data)
ds_data = ds_data.repeat()
iterator = ds_data.make_initializable_iterator()
next_element = iterator.get_next()

sess = tf.Session()
sess.run(iterator.initializer)
_ = sess.run(next_element)

Closing this issue since its resolved. Feel free to reopen if still having problems. Thanks!


Client->getStories not respecting release

Currently the getStories function does not set the from_release query parameter for requests to Storyblok if the release was set using the setRelease function.
Expected Behavior
The getStories function should add the release parameter correctly if $this->release is set.
Current Behavior
The release parameter is not set when calling the getStories function if it isn't provided using the options parameter
Steps to Reproduce
Not working
$client = new Client('api-key-###');
$client->setRelease('123456');
$options = [
  'by_slugs' => 'test'
];
$client->getStories($options);

Workaround
$client = new Client('api-key-###');
$options = [
  'by_slugs'     => 'test',
  'from_release' => '123456'
];
$client->getStories($options);

I don't mind creating a fix PR, but your contributing guidelines said to open an issue first.

Thank you, @hoersamu, for reporting the issue.
I think that for consistency with other methods like setLanguage() etc is a good option to make the setReleases() work. I see you already opened the PR. I'm jumping on the PR.
Thank you for the feedback!
Roberto


Audio Captcha Solving as an alternative 2Captcha

This is a feature request.
awsapilib can currently solve captchas (e.g. in root user signing and password reset) via 2captcha.
AWS recently added the Audio Captcha functionality to all captcha occurrences (signup, sign-in, password reset).
AWS audio captchas can be solved e.g. via Amazon Transcribe, example code can be found at the superwerker MAVM project.
This would enable awsapilib to be self-contained within AWS without having to provide 3rd party APIs/API keys like 2captcha.

Hi @s0enke , looking into this quite old by now feature request and I am thinking that it would be a chicken and egg problem. The point of this library is asking other things to give root access to an account. That assumes no prior connectivity to the account. How do you imagine the interface to be for the usage of the native services of aws? That would require access to another account for usage of those services which would make things confusing and complex, I think. Any thoughts?

Hi @costastf,
you are correct, using AWS Transcribe would also need valid AWS credentials to work. It doesn't have to be the same account as the one that awsapilib is desired to login as root user - but it can be an arbitrary AWS account with Transcribe permissions.


Cannot read properties of undefined (reading 'toolCalls')

Describe the bug
I have installed qwen2.5-coder. I launched bolt in pinokio. And I'm making a mistake.
Link to the Bolt URL that caused the error
/
Steps to reproduce
bolt launch in pinokio
The choice of ollama - qwen2.5-coder:32b
promt
Expected behavior
start
Screen Recording / Screenshot

Platform

OS: [e.g. macOS, Windows, Linux]
Browser: [e.g. Chrome, Safari, Firefox]
Version: [e.g. 91.1]

Provider Used
ollama
Model Used
qwen2.5-coder:32b
Additional context

this happens when the promt improves.

Describe the bug
I have installed qwen2.5-coder. I launched bolt in pinokio. And I'm making a mistake.
Link to the Bolt URL that caused the error
/
Steps to reproduce
bolt launch in pinokio The choice of ollama - qwen2.5-coder:32b promt
Expected behavior
start
Screen Recording / Screenshot

Platform

OS: [e.g. macOS, Windows, Linux]
Browser: [e.g. Chrome, Safari, Firefox]
Version: [e.g. 91.1]

Provider Used
ollama
Model Used
qwen2.5-coder:32b
Additional context
 this happens when the promt improves.

I have this issue too , hope they fix it

same issue here

can you try other provider?

Same issue here, i have tried using Node.js on local computer

Same issue here, Macbook Pro M1 Pro , macOS Sequoia 15.2

same issue with me

The same issue even if I used other providers.
I tested it using this providers:
TogetherAI
Groq
xAi
Anyway, it worked with local OLlama (Llama 3.1 (8B)).
Device:
MacBook M1 Pro.

happens in two scenario, your models were not selected correctly and its using default model instead of selected model
or you have set the api key wrong
can you set the api keys on the UI and just switch the model dropdown to some other model and then back to the one you want to use, "model" not the "provider"

EDIT: I tried to track NETWORK browser traffic to identify the issue. Anyway, The response of the request is EMPTY, and for the payload I can see that the api key is registered and everything looks fine but the request being responded with error.

were you able to check the selected model? thats in the messages and prefixed in the content of the user message

same happens to me it just stopped working after i finally got it running. I got this in the console:TypeError: Cannot read properties of undefined (reading 'toolCalls')
at Object.flush (file:///C:/Users//Documents/soody/website/bolt.diy-0.0.3/node_modules/.pnpm/ai@4.0.22_react@18.3.1_zod@3.24.1/node_modules/ai/core/generate-text/stream-text.ts:569:33)
at ensureIsPromise (node:internal/webstreams/util:192:19)
at transformStreamDefaultSinkCloseAlgorithm (node:internal/webstreams/transformstream:569:5)
at Object.close (node:internal/webstreams/transformstream:366:14)
at ensureIsPromise (node:internal/webstreams/util:192:19)
at writableStreamDefaultControllerProcessClose (node:internal/webstreams/writablestream:1142:28)
at writableStreamDefaultControllerAdvanceQueueIfNeeded (node:internal/webstreams/writablestream:1222:5)
at writableStreamDefaultControllerClose (node:internal/webstreams/writablestream:1189:3)
at writableStreamClose (node:internal/webstreams/writablestream:699:3)
at writableStreamDefaultWriterClose (node:internal/webstreams/writablestream:1071:10)
at writableStreamDefaultWriterCloseWithErrorPropagation (node:internal/webstreams/writablestream:1063:10)
at node:internal/webstreams/readablestream:1439:15
at complete (node:internal/webstreams/readablestream:1312:9)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
21:38:24 [vite] Internal server error: Cannot read properties of undefined (reading 'toolCalls')
at Object.flush (file:///C:/Users//Documents/soody/website/bolt.diy-0.0.3/node_modules/.pnpm/ai@4.0.22_react@18.3.1_zod@3.24.1/node_modules/ai/core/generate-text/stream-text.ts:569:33)
at ensureIsPromise (node:internal/webstreams/util:192:19)
at transformStreamDefaultSinkCloseAlgorithm (node:internal/webstreams/transformstream:569:5)
at Object.close (node:internal/webstreams/transformstream:366:14)
at ensureIsPromise (node:internal/webstreams/util:192:19)
at writableStreamDefaultControllerProcessClose (node:internal/webstreams/writablestream:1142:28)
at writableStreamDefaultControllerAdvanceQueueIfNeeded (node:internal/webstreams/writablestream:1222:5)
at writableStreamDefaultControllerClose (node:internal/webstreams/writablestream:1189:3)
at writableStreamClose (node:internal/webstreams/writablestream:699:3)
at writableStreamDefaultWriterClose (node:internal/webstreams/writablestream:1071:10)
at writableStreamDefaultWriterCloseWithErrorPropagation (node:internal/webstreams/writablestream:1063:10)
at node:internal/webstreams/readablestream:1439:15
at complete (node:internal/webstreams/readablestream:1312:9)
at processTicksAndRejections (node:internal/process/task_queues:95:5) (x4)

Same issue here, was working fine then I attempted to upload a .py file and then that error occurred. Using Claude 3.5 sonnet.
I also have an open ai api key i tried using, 4o mini worked, but 4o didn't work.

EDIT: I tried to track NETWORK browser traffic to identify the issue. Anyway, The response of the request is EMPTY, and for the payload I can see that the api key is registered and everything looks fine but the request being responded with error.

were you able to check the selected model? thats in the messages and prefixed in the content of the user message

Yes, I can see it there.

[Model: grok-2-1212]\n\n[Provider: xAI]\n\n

any fixes?

As a temporary fix, after you activate the venv, you can do the following to rollback the huggingface_hub package, which is the source of the issue.
pip uninstall -y huggingface_hub
pip install huggingface_hub==0.25.2

I have the same issue when connected with Claude AI, I have saved API key to .env.local but it's still not working. Ollama model running locally works though.

I'm having the same issue. I'm on. a Mac M1  running Sequoia 15.2.
Tried running in Docker and with pnpm. Tried using Chrome Canary, and Brave browser.
Happens with local Ollama and OpenAI with key.  I've tried running from localhost:5173 and <IP_ADDRESS>:5173. Getting same issue across the board.

I have the same problem as all of you I am Sequoia 15.1.1

will look into this

will look into this

Thank you for your efforts.

I read that it was related to docker..i ran docker as per setup instruction but didnt work either

me too, I think it's linked to the M1 chip

worked with llama3(8b) but as visible in the screenshot, it didnt loan in the container

me too, I think it's linked to the M1 chip

Same, I'm using M chips too, probably this is what causing the problem.

I have the same issue, and i do not believe it is related to the M1 chip since i am having it on AMD 7th gen processor

in any case, I can't wait to use it on my computer and for the bug to be fixed.

hope it is fixed soon as well. Is it possible that we messed up with setup and did not set it up properly? I guess it is

I don't think so because I've done all 3 possible installations.

I have M4 an have the same problem (installed via Pinokio)...Anyone have founded a solution?
I have tried to downgrade huggingface-hub but don't work.

Installing dependencies using pnpm instead of npm led to a different problem. Seems like the old is not showing anymore tho.

 worked with llama3(8b) but as visible in the screenshot, it didnt loan in the container

this is llm not following instructions. to open a container the llm has to write the code is a special tags this its not doing it.
i often see smaller llm not able to follow system prompt correctly. try with other models like Togather AI, its free

Looking at the error I believe its related to the vecel api being used for model switching. But its just a theory.

I'm having this issue as well. I've spent days trying to fix it.

Different browsers
Different versions of bolt
Different LLMs
Docker, normal install
Different projects
Troubleshooting different projects, uninstalling, reinstalling dependencies, etc, etc
The list goes on. I start from scratch, get my project set up, I get X part of the way through, then error, troubleshoot, unfixable, start again.

Cannot read properties of undefined (reading 'toolCalls')
at Object.flush (file:///C:/Users/twest/bolt.diy/node_modules/ai/core/generate-text/stream-text.ts:569:33)
at ensureIsPromise (node:internal/webstreams/util:192:19)
at transformStreamDefaultSinkCloseAlgorithm (node:internal/webstreams/transformstream:569:5)
at Object.close (node:internal/webstreams/transformstream:366:14)
at ensureIsPromise (node:internal/webstreams/util:192:19)
at writableStreamDefaultControllerProcessClose (node:internal/webstreams/writablestream:1142:28)
at writableStreamDefaultControllerAdvanceQueueIfNeeded (node:internal/webstreams/writablestream:1222:5)
at writableStreamDefaultControllerClose (node:internal/webstreams/writablestream:1189:3)
at writableStreamClose (node:internal/webstreams/writablestream:699:3)
at writableStreamDefaultWriterClose (node:internal/webstreams/writablestream:1071:10)
at writableStreamDefaultWriterCloseWithErrorPropagation (node:internal/webstreams/writablestream:1063:10)
at node:internal/webstreams/readablestream:1439:15
at complete (node:internal/webstreams/readablestream:1312:9)
at processTicksAndRejections (node:internal/process/task_queues:95:5

rename the env file to just ".env" and set the API key or base URL there. If you are using local model don't use localhost, use <IP_ADDRESS> instead.
Change the "dev" script in the package.json file to "remix vite:dev --host <IP_ADDRESS> --port {your port} --open"

This actually did the job for me and now everything works locally using ollama

@galaridor I just tried this but not working for me :(

@galaridor may be on to something.
Instead of renaming my .env.local, I made a copy of it and named it .env.  I now have both in the root of my project.
After grabbing the latest main branch, and changing my dev script to this:
"dev": "node pre-start.cjs  && remix vite:dev --host <IP_ADDRESS> --port 5173 --open",
I did a pnpm install, and pnpm run dev. I was able to get a response using the Ollama 3.2 model. The Ollama 3.3 I think is timing out, and my OpenAI attempts still don't work.
Running in docker still gives the Cannot read properties of undefined (reading 'toolCalls') popup on all llm's.

i have not tested with docker. It works locally with qwen 2.5 coder using ollama for me but i believe you still have to set some env variables inside the dockerfile or docker-compose.yaml files

will look into this

any update?

can you try this PR https://github.com/stackblitz-labs/bolt.diy/pull/895
this has couple of bugfixes and improved logging

Getting same error even if I run it on Pinokio
M1 chip Mac, OpenAi,


Prevent other gesture recognizers from overriding the tap animation

Fixes #90
Checklist

[ ] Includes UI tests
[ ] Readme is updated

Description
This pull requests fixes an issue caused by having other gesture recognizers in the app that can cause the pan gesture of the tab bar to fail and never get in .ended state. This would lead to the tap animation to be overridden by the swipe animation by mistake. It is especially a problem for those like me who want the tap animation to be disabled.
The PR also updates the example app to show that it's possible to disable the tap animation by setting its duration to 0.

Thanks for the PR @wasp898 I'll take a look at it later today and also check the possibility of setting the transition animation to nil to disable the animation.

@wasp898 I created a new PR (#92) that restores public transitioning properties so they can be set to nil to disable animations. Would you like to take a look at it?

@wasp898 I created a new PR (#92) that restores public transitioning properties so they can be set to nil to disable animations. Would you like to take a look at it?

Ok thanks, I just did.
I do agree that it's better to set the tap animation to nil.
I see that you kept the bug fix I proposed and that's what matters to me, so I'm happy.
p.s. I still have some issues with view controllers going black from time to time though. Hope I can help figuring that out as well so that I can safely adopt this dependency in my app. Would be so cool to have! Thanks again.


How do I build this tool?

Can you please explain how I build this tool?

Did you read the wiki?
https://github.com/sapstern/-edifactconverter-/wiki

No, I was only looking in the README. Thanks!
Do you support OpenTrans 2.1 as well?

No, this is not supported.
But you can do it like this:

Use the converter to transform to EDI-XML
Use XSLT or Java Sax/DOM Parser to transform from EDI-XML to target XML

Use XSLT or Java Sax/DOM Parser to transform from EDI-XML to target XML

Which Java Sax/DOM Parser would you recommend?

I always use SAX:
import org.xml.sax.Attributes;
import org.xml.sax.SAXException;
import org.xml.sax.helpers.DefaultHandler;
Write a Class which extends DefaultHandler.
It fires events while traversing the EDI-XML tree.
Have a look to my code inside the repo:
com.sapstern.openedifact.sax.EdifactSaxParserToFlat

you mean here: https://github.com/sapstern/-edifactconverter-/blob/0357a7a5a9352b725777bfb34a3c4ed9b5e0aebb/openedifact-converter/src/main/java/com/sapstern/openedifact/sax/EdifactSaxParserToFlat.java

How much work do you think it would it take to add support of OpenTrans 2.1 to this tool?

yes, I meant this class, have a look for the @Override annotations.
These are the methods which will be called by the sax parser.
To your last question.
Not insignificant as it looks.
The main task is to apply the process know how to the mappings, I would think.
You need to know which EDIFACT field has its equivalent in the OpenTrans format.

yes corret.
https://github.com/sapstern/-edifactconverter-.git

Would it be easier if we would just start with a couple of sample messages?

Ok, build on Linux was successful! Now I am reading your next steps.

yes, work with sample messages is a good starting point

Ok, I am now at step

Use the converter to transform to EDI-XML

How do I convert to EDI-XML?
At the moment I get:
~/.software/edifactconverter/openedifact-xsd-libs-96A/target> java -jar openedifact-xsd-libs-96A-1.1-SNAPSHOT.jar
kein Hauptmanifestattribut, in openedifact-xsd-libs-96A-1.1-SNAPSHOT.jar

Use openedifact-converter-2.2-SNAPSHOT.jar,  this holds the converter.
You can't  use java -jar, you just have to put the archives on the classpath and wirite your own class which calls the converter.
I think inside the test section of the maven project, there is an example.

This I do not understand from the Wiki:
com.openedi.sax.EdifactSaxParserToXML theXmlParser = EdifactSaxParserToXML .factory("UTF-8"); "<Encoding e.g. UTF-8" );

I am trying the following:
java -jar openedifact-converter/target/openedifact-converter-2.2-SNAPSHOT.jar openedifact-xsd-libs-96A/target/classes/EDIFACTINTERCHANGE_ORDERS_96A.xsd /tmp/SND_PLICA_ORDERS_134747_58408847

you mean here: openedifact-sample-01 - I am looking at the sample now. Thank you!

Example:
EdifactSaxParserToXML theXmlParser = EdifactSaxParserToXML.factory("UTF-8", logger);
theXmlParser.parse(Your Edifact as InputSource)

Write a class with a main method and call the code

In which directory do I find the sample?

Have a look to src/test/java/EDIFACT_2_XML_Test.java
it instantiates and calls the parser.

this one: https://github.com/sapstern/-edifactconverter-/blob/master/openedifact-converter/src/test/java/com/sapstern/openedifact/segmenttest/EDIFACT_2_XML_Test.java

I prefer this one, it is simpler: https://github.com/sapstern/-edifactconverter-/blob/master/openedifact-sample-01/src/main/java/com/sapstern/openedifact/Sample01.java

How do I build this, so I can run it from commandline?
https://github.com/sapstern/-edifactconverter-/blob/master/openedifact-converter/src/test/java/com/sapstern/openedifact/segmenttest/EDIFACT_2_XML_Test.java

I am now trying: mvn package

You need to setup a java project, the whole stuff is meant to be used in calling the converter from your self written java code.
It is not running out of the box, sorry.
If you are not familiar with java, then it will be difficult, I am afraid.

I think the usage of the project would be greatly increased if it would come with a small commandline utility at actually uses the converter as you described above. See #2


feature request: include the browser audio stream in the recorded video

The video recording works great, but I was also looking to get the audio on the browser and couldn't find any way to do it. I guess because this uses screenCapture, it might not be possible, but again I don't know enough.

Thanks @Akarshit, this uses the screencast from chromium devtools protocol, which only has the video frames of the browser and audio is not included.
Using ffmpeg with a screen capture would allow us to get the audio, but that will require that you run the browser on a virtual display which is different across platforms, and would not allow headless mode.
Perhaps it is possible to merge an audio stream from somewhere else, but I am not sure how. I will leave the issue open.

In puppeteer, I have successfully been able to make a headful instance of the browser, so accessing audio via the screenCapture api.
I don’t know if it helps, but the implementation to get audio I use is called html2screen. https://github.com/Ventricule/html2screen
For Linux, when you have a docker instance, installing xvfb gives your server a virtual display. With that, you can run a headful instance in Linux as well. This is helpful starting point if you want to package this as a server container.

@rememberlenny Hey!
I have looked at that package before but was unable to record HD videos using it. Are you able to get HD videos out of it?
If you are only use it for audio, how are you getting rid of sync issues, while merging audio from html2screen and video from playwright?

@Akarshit I didnt end up trying. I've been using html2screen successfully.


Make app_id default to a numeric value instead of a string

@zendesk/vegemite
Hey guys, if we have a line in an app like the following:
  if (this.installationId()) {
    ... blah blah here
  }

if you are running the app from zat (which I think this conditional is checking for), the installationId() method was returning a string '0' by default, which is truthy. I've changed ZAT to only use a default app_id of the numeric 0 where necessary. I suppose I could just do parseInt(this.installationId(), 10) instead, but I wanted to see why this was happening at the source.
I took a look at Thor's Method Options docs as a reference for forcing a numeric type.
Thoughts?

The app should really use this.isZatEnabled() and don't rely on the fact that it is ID 0, which happens to be only true for Zat. Sam for appId()

@ocke yes it should and I have a PR open to do that here: https://github.com/zendesk/timetracking_app/pull/78 but zat should still be consistent here with normal apps.

Isn't it possible to assign app_id to any arbitrary object? I haven't tested- just interested. The possibility of a string showing up was pretty confusing to me when trying to debug the time tracking app 💣

yep, you can currently type whatever you like in the commandline for app_id. Even with this patch it will accept decimals, but you'd have to go out of your way to shoot yourself in the foot that way.

Actually, we shouldn't use isZatEnabled because you might want to test in ZAT against a real app's requirements.

just released this in 1.37.4 sorry for the delay


Do not set reserved to 0

Currently reserved is set to 0 when quota usage is updated.
When expired reservations are removed by Cinder, this may
result in negative reserved values. Since the periodic task
in Cinder takes care of expired reservations and reserved
values, we shouldn't blindly set reserved to 0.

Thanks : )


include cstdlib for atof()

fix for https://github.com/lasalvavida/ahoy/issues/1

Awesome @jeremyabel, thank you!


Add support for complex symmetric and unsymetric matrices

Adding support for complex symmetric mtype = 3 and complex unsymmetric mtype=13 matrices.

There's an error in some of the results that I'm seeing so let me investigate before reopening this.

Thanks a lot for this PR. Hope you find the reason for the errors, would be happy to integrate this! Never had a usecase for complex matrices, that's why I never took the time to include it.


Feature/project ownership

Description

Folders and templates were originally owned by users, now they will be owned by projects that users belong to, allowing multiple users to work on the same folders and templates at the same time.
Related issues
Fixes #282
Frontend issue https://github.com/CSCfi/metadata-submitter-frontend/issues/625
Type of change

[x] New feature (non-breaking change which adds functionality)
[x] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[x] This change requires a documentation update (API changes)

Changes Made

deprecated folders and templates keys from user
added new key projects to user
added new key projectId to folder and template collections
added new project collection
new mandatory userinfo value from AAI sdSubmitProjects
if user has no sdSubmitProjects they are redirected to /noproject
new mandatory query parameter projectId for GET /folders
new endpoint GET /templates?projectId=... that replaces GET /users/current "templates":[...]
projectId is not mandatory when requesting specific resource, e.g. GET /folders/{folderId}, the permissions are read from session
new mandatory key projectId for POST /folders and /templates

Other Points

user project affiliations are re-validated on every login, if project affiliations have changed, user data is updated
breaking change that requires fresh database, because "project" is new information that did not exist before, and it can't be migrated to existing user-owned hierarchy due to missing relationship information
I don't know if this PATCH /user is needed anymore, at least it is no longer possible to update user's folders and templates, since they don't exist. I added deprecation comments to those parts, but left the function intact https://github.com/CSCfi/metadata-submitter/blob/feature/project-ownership/metadata_backend/api/handlers/user.py#L19-L70

Testing

[x] Unit Tests
[x] Integration Tests
[x] Manual Testing

The previous commit 00cec8f updated integration tests, they also showed some bugs in the code which were fixed in the same commit. Also removed some deprecated features, and will continue to prune through deprecated features in next commit.

Added integration test case for GET /templates. I tried to copy folder unit tests (query_folders) for templates (query_templates_by_project) (because they are almost identical), but because the query loops over collections, it raises a StopAsyncIteration, and I'm not sure how to deal with that.

StopAsyncIteration
Would this be usful:

try:
    ....
except StopIteration:
    pass

Added integration test case for GET /templates. I tried to copy folder unit tests (query_folders) for templates (query_templates_by_project) (because they are almost identical), but because the query loops over collections, it raises a StopAsyncIteration, and I'm not sure how to deal with that.

Would this be usful:
try:
    ....
except StopIteration:
    pass

https://github.com/CSCfi/metadata-submitter/blob/135af96bc22ce651019955356c97122a9746369f/tests/integration/run_tests.py#L549

I thought about this also initially and tested it, but it still errored, and I think it's also not the correct way to approach this, because it would stop the test
This is the function
https://github.com/CSCfi/metadata-submitter/blob/feature/project-ownership/metadata_backend/api/operators.py#L308-L365
It queries multiple collections
It's based on this
https://github.com/CSCfi/metadata-submitter/blob/feature/project-ownership/metadata_backend/api/operators.py#L812-L854

It's based on this https://github.com/CSCfi/metadata-submitter/blob/feature/project-ownership/metadata_backend/api/operators.py#L812-L854

I think this: https://github.com/CSCfi/metadata-submitter/blob/feature/project-ownership/metadata_backend/api/operators.py#L395 might be more appropriate as you want object query not folder query and maybe pay attention to: https://github.com/CSCfi/metadata-submitter/blob/feature/project-ownership/metadata_backend/conf/conf.py#L113

wait you want query_templates_by_project not to filter the templates content ... :thinking: maybe it will  be easier if you put them all in one collection templates and add the type for each document in the collect e.g. study, sample etc. I am not sure we should invest the effort in making cross collection queries at this point for templates

however if you really want JOIN like from SQL i think https://docs.mongodb.com/manual/reference/operator/aggregation/lookup/ is what you need.

however if you really want JOIN like from SQL i think https://docs.mongodb.com/manual/reference/operator/aggregation/lookup/ is what you need.

I will look into this if it makes it possible to request the data in one query

Looks like lookup is used for joining data together into one object from separate collections, but we want to get a list of all objects from multiple collections that have the same project id

Returned old object assigning code into use with templates-in-project (as opposed to templates-in-user).
Missing feature: PATCH /template requires feature for updating templates-key in project (similarly to PATCH user templates)


Add nix flake

This pull request will add support for reproducibly building and running symbolator using the nix package manger. This way you can run symbolator without the need to install any dependencies like PyGObject.
nix run github:zebreus/symbolator

See https://github.com/hdl/symbolator as a fork.

Thanks for the info. I also created the same PR on the fork as hdl/symbolator#10


Syntax highlighter breaks on << ... >> in some cases

Issue Description
Syntax highlighter seems to break when it encounters this line in my code:
int startDX = (((current.MinX << SubpixelBits) + SubpixelHalf) - current.X) >> (SubpixelBits - circleShift);
Steps to Reproduce
Paste this line into any C# method.
Expected Behavior
Highlighting works normally.
Actual Behavior
Highlighting on this line (and any after it) will be wrong.
Interestingly, if I add a line break so that left-shift and right-shift operators are on different lines, the issue does not appear.
Logs
OmniSharp log
OmniSharp log didn't appear to contain any information related to syntax highlighting.
C# log
Output-->C# is empty.
Environment information
VSCode version: 1.59.0
C# Extension: 1.23.14
Dotnet Information
.NET SDK (reflecting any global.json):
 Version:   5.0.302
 Commit:    c005824e35
Runtime Environment:
OS Name:     Windows
OS Version:  10.0.19042
OS Platform: Windows
RID:         win10-x64
Base Path:   C:\Program Files\dotnet\sdk\5.0.302\
Host (useful for support):
Version: 5.0.8
Commit:  35964c9215
.NET SDKs installed:
3.1.301 [C:\Program Files\dotnet\sdk]
5.0.301 [C:\Program Files\dotnet\sdk]
5.0.302 [C:\Program Files\dotnet\sdk]
.NET runtimes installed:
Microsoft.AspNetCore.All 2.1.28 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.28 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.28 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.1.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 3.1.16 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.7 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.8 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
To install additional .NET runtimes or SDKs:
https://aka.ms/dotnet-download

Visual Studio Code Extensions

Extension
Author
Version

csharp
ms-dotnettools
1.23.14

nugetpackagemanagergui
aliasadidev
1.1.6

python
ms-python
2020.9.114305

shader
slevesque
1.1.5

unity-debug
Unity
2.7.5

Please try enabling the semantic highlight service and see if you still get bad highlighting.

Thanks! That fixed the problem :-)


bug(build): fails on requirement of very old LLVM version

Error: RuntimeError: Building llvmlite requires LLVM 10.0.x or 9.0.x, got '18.1.8'. Be sure to set LLVM_CONFIG to the right executable path.
AFIK LLVM 10 is really outdated?
pip install -e .                                                                                (main)
using uv for pip...
   Built file:///Users/samm/git/nexa-sdk                                                                                Built 1 editable in 19.62s
Resolved 90 packages in 485ms
error: Failed to download distributions
  Caused by: Failed to fetch wheel: llvmlite==0.34.0
  Caused by: Failed to build: llvmlite==0.34.0
  Caused by: Build backend failed to build wheel through `build_wheel()` with exit status: 1
--- stdout:
running bdist_wheel
/Users/samm/Library/Caches/uv/.tmp6mwwHF/.venv/bin/python /Users/samm/Library/Caches/uv/built-wheels-v2/pypi/llvmlite/0.34.0/fvZMsz9m1zjUw501fEtSU/llvmlite-0.34.0.tar.gz/ffi/build.py
LLVM version... 18.1.8
--- stderr:
Traceback (most recent call last):
  File "/Users/samm/Library/Caches/uv/built-wheels-v2/pypi/llvmlite/0.34.0/fvZMsz9m1zjUw501fEtSU/llvmlite-0.34.0.tar.gz/ffi/build.py", line 191, in <module>
    main()
  File "/Users/samm/Library/Caches/uv/built-wheels-v2/pypi/llvmlite/0.34.0/fvZMsz9m1zjUw501fEtSU/llvmlite-0.34.0.tar.gz/ffi/build.py", line 185, in main
    main_posix('osx', '.dylib')
  File "/Users/samm/Library/Caches/uv/built-wheels-v2/pypi/llvmlite/0.34.0/fvZMsz9m1zjUw501fEtSU/llvmlite-0.34.0.tar.gz/ffi/build.py", line 143, in main_posix
    raise RuntimeError(msg)
RuntimeError: Building llvmlite requires LLVM 10.0.x or 9.0.x, got '18.1.8'. Be sure to set LLVM_CONFIG to the right executable path.
Read the documentation at http://llvmlite.pydata.org/ for more information about building llvmlite.

error: command '/Users/samm/Library/Caches/uv/.tmp6mwwHF/.venv/bin/python' failed with exit code 1

The error occurs because your LLVM version (18.1.8) is much newer than what llvmlite 0.34.0 supports (10.0.x or 9.0.x). Maybe you could install an older version of LLVM that's compatible with llvmlite 0.34.0.

@sammcj Please try
https://github.com/NexaAI/nexa-sdk/releases/tag/v<IP_ADDRESS>
and let us know

Fixed :) Thanks!


Integration to existing Apps: require_relative not working as expected. Only works if the App is inside the React Native root ios directory

Description
I am trying to use require_relative, but it seems like it only works when the folder structure is like this:
- src
---- ios
-------- podFile
---- node_modules

and try to do pod install with podFile as:
# Dev Changes
require_relative '../node_modules/react-native/scripts/react_native_pods'
require_relative '../node_modules/@react-native-community/cli-platform-ios/native_modules'

this succeeds installation, however, if I update the structure like this:
- src
---- ios
-------- node_modules
-------- podFile

and try to do pod install with podFile as:
require_relative './node_modules/react-native/scripts/react_native_pods'
require_relative './node_modules/@react-native-community/cli-platform-ios/native_modules'

this fails with the following error:
Analyzing dependencies
Fetching podspec for DoubleConversion from ../../reactNativeApp/content//node_modules/react-native/third-party-podspecs/DoubleConversion.podspec
Fetching podspec for RCT-Folly from ../../reactNativeApp/content//node_modules/react-native/third-party-podspecs/Folly.podspec
/usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/user_interface/error_report.rb:34:in force_encoding': can't modify frozen String (FrozenError) from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/user_interface/error_report.rb:34:in report'
from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/command.rb:66:in report_error' from /Library/Ruby/Gems/2.6.0/gems/claide-1.1.0/lib/claide/command.rb:396:in handle_exception'
from /Library/Ruby/Gems/2.6.0/gems/claide-1.1.0/lib/claide/command.rb:337:in rescue in run' from /Library/Ruby/Gems/2.6.0/gems/claide-1.1.0/lib/claide/command.rb:324:in run'
from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/command.rb:52:in run' from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/bin/pod:55:in <top (required)>'
from /usr/local/Cellar/cocoapods/1.11.3/libexec/bin/pod:23:in load' from /usr/local/Cellar/cocoapods/1.11.3/libexec/bin/pod:23:in

'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/uri/generic.rb:1490:in find_proxy': relative URI: /reactNativeApp/content/node_modules/react-native/third-party-podspecs/Folly.podspec (URI::BadURIError) from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/open-uri.rb:209:in block in open_loop'
from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/open-uri.rb:226:in block in open_loop' from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/open-uri.rb:224:in catch'
from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/open-uri.rb:224:in open_loop' from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/open-uri.rb:165:in open_uri'
from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/external_sources/podspec_source.rb:19:in blo ssahil1@LAMU02D86CPMD6R native_ios_app % pod install Analyzing dependencies Fetching podspec for DoubleConversionfrom../node_modules/react-native/third-party-podspecs/DoubleConversion.podspec/usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/user_interface/error_report.rb:34:inforce_encoding': can't modify frozen String (FrozenError)
from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/user_interface/error_report.rb:34:in report' from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/command.rb:66:in report_error'
from /Library/Ruby/Gems/2.6.0/gems/claide-1.1.0/lib/claide/command.rb:396:in handle_exception' from /Library/Ruby/Gems/2.6.0/gems/claide-1.1.0/lib/claide/command.rb:337:in rescue in run'
from /Library/Ruby/Gems/2.6.0/gems/claide-1.1.0/lib/claide/command.rb:324:in run' from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/command.rb:52:in run'
from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/bin/pod:55:in <top (required)>' from /usr/local/Cellar/cocoapods/1.11.3/libexec/bin/pod:23:in load'
from /usr/local/Cellar/cocoapods/1.11.3/libexec/bin/pod:23:in <main>' /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/uri/generic.rb:1490:in find_proxy': relative URI: /reactNativeApp/content/node_modules/react-native/third-party-podspecs/DoubleConversion.podspec (URI::BadURIError)
from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/open-uri.rb:209:in block in open_loop' from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/open-uri.rb:226:in block in open_loop'
from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/open-uri.rb:224:in catch' from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/open-uri.rb:224:in open_loop'
from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/open-uri.rb:165:in open_uri' from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/external_sources/podspec_source.rb:19:in blo
ssahil1@LAMU02D86CPMD6R native_ios_app % pod install
Analyzing dependencies
Fetching podspec for DoubleConversion from ../../reactNativeApp/content/node_modules/react-native/third-party-podspecs/DoubleConversion.podspec
Fetching podspec for Folly from ../../reactNativeApp/content/node_modules/react-native/third-party-podspecs/Folly.podspec
/usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/user_interface/error_report.rb:34:in force_encoding': can't modify frozen String (FrozenError) from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/user_interface/error_report.rb:34:in report'
from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/command.rb:66:in report_error' from /Library/Ruby/Gems/2.6.0/gems/claide-1.1.0/lib/claide/command.rb:396:in handle_exception'
from /Library/Ruby/Gems/2.6.0/gems/claide-1.1.0/lib/claide/command.rb:337:in rescue in run' from /Library/Ruby/Gems/2.6.0/gems/claide-1.1.0/lib/claide/command.rb:324:in run'
from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/command.rb:52:in run' from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/bin/pod:55:in <top (required)>'
from /usr/local/Cellar/cocoapods/1.11.3/libexec/bin/pod:23:in load' from /usr/local/Cellar/cocoapods/1.11.3/libexec/bin/pod:23:in '
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/uri/generic.rb:1490:in find_proxy': relative URI: /reactNativeApp/content/node_modules/react-native/third-party-podspecs/Folly.podspec (URI::BadURIError) from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/open-uri.rb:209:in block in open_loop'
from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/open-uri.rb:226:in block in open_loop' from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/open-uri.rb:224:in catch'
from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/open-uri.rb:224:in open_loop' from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/open-uri.rb:165:in open_uri'
from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/external_sources/podspec_source.rb:19:in block in fetch' from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/user_interface.rb:86:in titled_section'
from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/external_sources/podspec_source.rb:11:in fetch' from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/installer/analyzer.rb:993:in fetch_external_source'
from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/installer/analyzer.rb:972:in block (2 levels) in fetch_external_sources' from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/installer/analyzer.rb:971:in each'
from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/installer/analyzer.rb:971:in block in fetch_external_sources' from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/user_interface.rb:64:in section'
from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/installer/analyzer.rb:970:in fetch_external_sources' from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/installer/analyzer.rb:117:in analyze'
from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/installer.rb:416:in analyze' from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/installer.rb:241:in block in resolve_dependencies'
from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/user_interface.rb:64:in section' from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/installer.rb:240:in resolve_dependencies'
from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/installer.rb:161:in install!' from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/command/install.rb:52:in run'
from /Library/Ruby/Gems/2.6.0/gems/claide-1.1.0/lib/claide/command.rb:334:in run' from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/lib/cocoapods/command.rb:52:in run'
from /usr/local/Cellar/cocoapods/1.11.3/libexec/gems/cocoapods-1.11.3/bin/pod:55:in <top (required)>' from /usr/local/Cellar/cocoapods/1.11.3/libexec/bin/pod:23:in load'
from /usr/local/Cellar/cocoapods/1.11.3/libexec/bin/pod:23:in `'

I have two things to question:

What's causing this unexpected behaviour? Because when I keep the same node_modules folder next to ios it works fine which is kind of wierd.
Let's say my two folders ios and the node_modules sit into two different directories: https://github.com/sedhha/<IOS> and https://github.com/sedhha/<node_modules>. In that case, I can't use require_relative is there a better way to fetch them globally, something like require_absolute: git: https://<address_to_git_repo>.

Version
0.68.2
Output of npx react-native info
System:
OS: macOS 12.3.1
CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
Memory: 6.64 GB / 32.00 GB
Shell: 5.8 - /bin/zsh
Binaries:
Node: 16.2.0 - /usr/local/bin/node
Yarn: 1.22.19 - ~/.yarn/bin/yarn
npm: 8.6.0 - /usr/local/bin/npm
Watchman: 2<IP_ADDRESS> - /usr/local/bin/watchman
Managers:
CocoaPods: 1.11.3 - /usr/local/bin/pod
SDKs:
iOS SDK:
Platforms: DriverKit 21.4, iOS 15.4, macOS 12.3, tvOS 15.4, watchOS 8.5
Android SDK: Not Found
IDEs:
Android Studio: 3.5 AI-191.8<IP_ADDRESS>10548
Xcode: 13.3.1/13E500a - /usr/bin/xcodebuild
Languages:
Java: 1.8.0_312 - /usr/bin/javac
npmPackages:
@react-native-community/cli: Not Found
react: 17.0.2 => 17.0.2
react-native: 0.68.2 => 0.68.2
react-native-macos: Not Found
npmGlobalPackages:
react-native: Not Found
Steps to reproduce

Create an ios project
Place a node_modules folder inside ios project
try to do require_relative and you will see the above error when doing pod install.

Snack, code example, screenshot, or link to a repository
CASE 1: podFile when installation succeeds:
# Dev Changes
require_relative '../node_modules/react-native/scripts/react_native_pods'
require_relative '../node_modules/@react-native-community/cli-platform-ios/native_modules'

CASE 2: podFiles when installation fails:
# Dev Changes
require_relative '../project/node_modules/react-native/scripts/react_native_pods'
require_relative '../project/node_modules/@react-native-community/cli-platform-ios/native_modules'

CASE 3: podFiles whne installation fails:
# Dev Changes
require_relative './node_modules/react-native/scripts/react_native_pods'
require_relative './node_modules/@react-native-community/cli-platform-ios/native_modules'

After diving into the source code, I realized that if you don't supply the path in use_react_native by default it goes to ../nod_modules/react-native, so I updated my use_react_native to use_react_native!(:path => './Content/node_modules/react-native') which got me installation successful, however the other question still remains that I want to install this from remote git repo and not by adding the files in a folder within my repo. I have created a SO post around the same, can someone help me to achieve that? Here's the link: https://stackoverflow.com/questions/72456941/remotely-install-react-native-pods-ios-podfile-instead-of-require-relative
I am keeping the thread open for knowing the second thing I asked in my question, however the issue has been resolved, if moderators feel I should open a new thread, feel free to close this, otherwise I would close this once I get answer to my second question.

Hello @sedhha, please refer to RN-Tester: https://github.com/facebook/react-native/tree/main/packages/rn-tester
It has flat directory structure.
As for your 2nd question: node_modules must be physically present locally on your machine, so require_relative must work if the relative path is correct.

can you update https://reactnative.dev/docs/integration-with-existing-apps @dmitryrykun
RNTester is really much different with the doc


Bias step rebiasing tagged incorrectly

Looking at observations starting to come out of satp3, looks like rebiasing with bias steps isn't tagging things the way we'd like. They're all tagged as oper,bias_steps,oper,biasstep_rebias. We'll want to fix this because it's going to create a whole lot of unnecessary books

Let us confirm to make sure. Is it correct that the tagging needs to be corrected from  oper,bias_steps,oper,biasstep_rebias to oper,bias_steps ?

The PR above fixes the issue, the tag should be oper,biasstep_rebias,<user-specified tags>


Clean Up Temp Directory After "Close"

@keller-mark Correct me if I'm wrong, but I think the temporary directory persists which can get a bit hairy with system memory since it is hard to find.  We should add a function or something to clean up/delete the directory once you're "done" with it.  Not sure what "done" looks like in code though...what actions cause it to fire.

We can add a __del__ method for AbstractWrapper in which the directory is cleaned up if it was a temporary directory


[SPARK-13435] [MLlib] Add Weighted Cohen's kappa to MulticlassMetrics

JIRA: https://issues.apache.org/jira/browse/SPARK-13435
What changes were proposed in this pull request?
Add the missing Weighted Cohen's kappa to MulticlassMetrics.
How was the this patch tested?
unit tests and manual tests were done.
The calculation correctness was tested on several small data, and compared to two python's package: sklearn and ml_metrics.

Can one of the admins verify this patch?

Jenkins, test this please

Any thoughts @mengxr ?
As for code formatting, I don't think IJ will change lines you don't touch. However you can edit the style settings in the editor preferences, and just make it write the scaladoc without the extra level of indent.
I wouldn't worry too much about the style -- just follow what you see around your change and undo changes that aren't related

Test build #51766 has started for PR 11303 at commit 4272aca.

Test build #51766 has finished for PR 11303 at commit 4272aca.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
   * (equals to precision for multiclass classifier

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51766/
Test PASSed.

@srowen yes, the style problem was caused by that the IJ's default 'reformating' style for scala dont match spark's.

cc @jkbradley @yanboliang


network: allow non-Android platforms to force IPv6 sockets

We suspect that under some conditions, non-Android platforms (i.e. iOS) also would benefit from forcibly using IPv6 sockets, so to validate this this change adds a new runtime feature generalizing the one added for Android.
Risk Level: None, runtime guard defaults to false.
Testing: No possible tests here. EM will provide testing.
Docs Changes: N/A
Release Notes: N/A
Platform Specific Features: Not platform specific, but intended to be used on Apple platforms.
Runtime guard: envoy.reloadable_features.always_use_v6 (default false)

Marked as draft as I test in Envoy Mobile.


permission update for pre-hook-update-db

Codecov Report
Attention: Patch coverage is 0% with 1 line in your changes missing coverage. Please review.

Please upload report for BASE (develop@2317d17). Learn more about missing BASE report.
Report is 2 commits behind head on develop.

Files
Patch %
Lines

...liance-web/src/components/App/Staff/StaffModal.tsx
0.00%
1 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             develop      #27   +/-   ##
==========================================
  Coverage           ?   22.48%
==========================================
  Files              ?       59
  Lines              ?      876
  Branches           ?      203
==========================================
  Hits               ?      197
  Misses             ?      679
  Partials           ?        0

Flag
Coverage Δ

compliance-web
22.48% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Booyer Moore Algorithm [SWOC]

I am a participant of SWOC 2020,
I would like to contribute by adding the code for Boyer Moore under the string algorithms.
Kindly assign this issue to me.
Thank you.

@HardevKhandhar please get started.

@HardevKhandhar please get started.


correspnding designation does not work

Hi,
First, thank you so much for the nice and helpful extension. It's really cool! I am just wondering why the corresponding option does not work for me with docx files. The equal contribution works and everything is good, but although I put corresponding in the right order
---
title: "Cannot share"
subtitle: "Manuscript draft."
authors:
  - name: Abdullah Abdelaziz
    affiliations:
      - ref: c1
    corresponding: true
  - name: cannot share
    affiliations:
      - ref: u1
      - ref: u2
affiliations:
  - id: c1
    name: xxxx
  - id: u1
    name: xxxxxx
  - id: u2
    name: xxxxxxxxx

filters:
  - authors-block
  - docx-landscape.lua

format:
  docx:
      output-file: "my manuscript draft"
  pdf: default
editor: visual
csl: jama.csl
reference-doc: custom-reference-doc.docx
bibliography: references.bib
---

This is how my header looks like.
Thanks

Hi @abduazizR , thanks for reporting this. Could you please provide me with some additional information so that I can try to reproduce the issue? On which operating system are you working? Which quarto-version are you using?

Hi @kapsner
Thank you for the quick response. My operating system is Windows 11, and my Quarto version is 1.3.330

@abduazizR moving to quarto 1.3.* also introduced some issues with the PDF version, which I've fixed on the development branch. Could you please try out, if this suggestion eventually helps to solve the problem?

I've just merged the quarto 1.3.* enhancements to the main branch so you can just reinstall the extension with the usual command to get the latest version for testing.
Unfortunately, I was not able to reproduce your error on my system with quarto 1.3.336, however, I am also using Linux and have no access to a Windows 11 workstation to check, if the code behaves differently there.
@abduazizR could you please check if the updated version also solves the issue with the missing correspondence for you?

Thank you for the replies. I tried it now. I run the following the terminal
A new release of Deno is available: 1.28.2 → 1.34.0 Run deno upgrade` to install it.
Quarto extensions may execute code when documents are rendered. If you do not
trust the authors of the extension, we recommend that you do not install or
use the extension.
? Do you trust the authors of this extension (Y/n) » Yes
[>] Downloading
[>] Unzipping
Found 1 extension.
The following changes will be made:
Authors-block   [Update] 0.1.1 -> 0.2.1 (filter)
? Would you like to continue (Y/n) » Yes
[>] Extension installation complete.
Learn more about this extension at https://www.github.com/kapsner/authors-block`
But the problem is not solved

Hmm, strange. I am sorry for that. Unfortunately, I am unable to reproduce the issue. Could you please retry with a minimal header to rule out that is has something to do with the other filter or the reference doc:
---
title: "Cannot share"
subtitle: "Manuscript draft."
authors:
  - name: Abdullah Abdelaziz
    affiliations:
      - ref: c1
    corresponding: true
  - name: cannot share
    affiliations:
      - ref: u1
      - ref: u2
affiliations:
  - id: c1
    name: xxxx
  - id: u1
    name: xxxxxx
  - id: u2
    name: xxxxxxxxx

filters:
  - authors-block

format:
  docx:
      output-file: "my manuscript draft"
  native: default
---

Could you also please post the content of the "native" file (also in the modified header as additional format)?

I tried it but the problem persists

Please add an email address to the entry of the corresponding author and retry.
---
title: "Cannot share"
subtitle: "Manuscript draft."
authors:
  - name: Abdullah Abdelaziz
    affiliations:
      - ref: c1
    corresponding: true
    email<EMAIL_ADDRESS>  - name: cannot share
    affiliations:
      - ref: u1
      - ref: u2
affiliations:
  - id: c1
    name: xxxx
  - id: u1
    name: xxxxxx
  - id: u2
    name: xxxxxxxxx

filters:
  - authors-block

format:
  docx:
      output-file: "my manuscript draft"
  native: default
---

Thank you very much for your efforts. Perhaps I've found the issue. In your example is the email address missing, which, however, is required for the corresponding part. Could you please add an email address to the entry of the corresponding author and retry.
---
title: "Cannot share"
subtitle: "Manuscript draft."
authors:
  - name: Abdullah Abdelaziz
    affiliations:
      - ref: c1
    corresponding: true
    email<EMAIL_ADDRESS>  - name: cannot share
    affiliations:
      - ref: u1
      - ref: u2
affiliations:
  - id: c1
    name: xxxx
  - id: u1
    name: xxxxxx
  - id: u2
    name: xxxxxxxxx

filters:
  - authors-block

format:
  docx:
      output-file: "my manuscript draft"
  native: default
---

It worked!!
Thank you so much for working with me on this.


Guidance needed: Kubernetes API servers in multiple regions

We are running a Kubernetes cluster with nodes in multiple "regions" (it doesn't really matter what that means).
When a pod wants to reach the API server the random selection that is set up in iptables by calico choses any of them in any region. Is there a way to limit these requests to the same region only?

@jlpnetworx Calico isn't actually responsible for making that decision - the kube-proxy programs those rules.
You might want to look into Service Topology in Kubernetes - that might be able to accomplish what you're looking for!
https://kubernetes.io/docs/concepts/services-networking/service-topology/


Fixed build errors, updated gradle plugin version

Verified that @brianytang has signed the CLA. Thanks for the pull request!


feat: create docker container with cron-job to backup database periodically and save on S3 bucket

[x] Create shell-script fetch all databases and backup in s3
[x] Create dockerfile with cron job
[x] Create taskdefinition script
[x] Create service with taskdefinition

Implementation is done


Python in Excel

About
Combining the power of Python and the flexibility of Excel.

Python in Excel makes it possible to natively combine Python and Excel analytics within the same workbook - with no setup required. With Python in Excel, you can type Python directly into a cell, the Python calculations run in the Microsoft Cloud, and your results are returned to the worksheet, including plots and visualizations.

Synopsis
df = xl("A1:B10", headers=True)
df.groupby("Category").agg("mean")

Details
Python in Excel exposes the best of Python analytics via Anaconda.

Anaconda is a leading enterprise Python repository used by tens of millions of data practitioners worldwide.  Python in Excel leverages Anaconda Distribution for Python running in Azure, which includes the most popular Python libraries (e.g. pandas, Matplotlib, scikit-learn, etc.), and is securely built, tested, and supported by Anaconda. Python provided by Anaconda supports a wide array of analytics with Python in Excel.

Python in Excel runs securely on the Microsoft Cloud, with no setup required.

Python code used by Excel runs on the Microsoft Cloud with enterprise-level security as a compliant Microsoft 365 connected experience. The Python code runs in its own hypervisor isolated container using Azure Container Instances and secure, source-built packages from Anaconda through a secure software supply chain.

Status

[!NOTE]
(6/10/24) UPDATE: Python in Excel is gradually rolling out to Enterprise, Business, Education, and Family and Personal users running Current Channel (Preview) builds on Windows. This feature is rolling out to Excel for Windows, starting with Version 2406 (Build 17726.20016).

Backlog

[ ] Level up in packaging. Here: Anaconda.

https://github.com/crate/crash/issues/165
https://github.com/crate/roadmap/issues/31

[ ] Evaluate with pandas' read_sql() method and sqlalchemy-cratedb.


Foiling of cards

Is there any plans to add more information on whether a card has different foiling?
For example an array listing {"nonfoil", "coldfoil"} to indicate the card is available both as non foil and as a cold foil.
This would allow for expansion for future foiling types and would let users describe the cards with more granularity (esp in case there are cards which are only nonfoil or only cold foil.

I didn't have any direct plans to add a field like that, but it doesn't sound like too much work to add, since in the input data some cards (like promos) have a 'foil_type' field, with values like 'Lava' or 'Silver', which I could use for this field. For cards without that field, I'll just have to assume they can have either no foiling or cold folding, like you suggested.
I'll add this new 'foilTypes' field in the next release.
Thanks for the suggestion!

I've just released a new version of the dataset. It includes some big changes, but also the new 'foilTypes' field you requested.


DeprecationWarning: Buffer()

When running the API example linked in the docs, there is a deprecation warning being logged.
[DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
Just thought I would point this out, perhaps the API auth example should be updated!
Thanks!

Looks like this is addressed in PR #30 .  I merged the author's work and the warnings are removed.  Consider merging their PR to update the example fully! :smile:


Feature: Word count and split string into words

Just like str_word_count in php.

New api: WordCount and WordSplit.


Update Jackson to 2.14.1 and fix dependency resolution issues

Signed-off-by: Andriy Redko<EMAIL_ADDRESS>Description
Port https://github.com/opensearch-project/sql/pull/1150 to main
Issues Resolved
N/A
Check List

[ ] New functionality includes testing.

[ ] All tests pass, including unit test, integration test and doctest

[ ] New functionality has been documented.

[ ] New functionality has javadoc added
[ ] New functionality has user manual doc added

[X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

The test failures are not related and caused by https://github.com/opensearch-project/OpenSearch/issues/5515

@reta, thanks for PR. SQL plugin does not working on main currently. Our plan is

Merge 2.x to main. https://github.com/opensearch-project/sql/pull/1063. Pending on https://github.com/opensearch-project/sql/pull/1160
Switch to main.

With merge 2.x to main, We will merge everything from 2.x to main, and will pickup #1150 automatically.

@reta, thanks for PR. SQL plugin does not working on main currently. Our plan is
1. Merge 2.x to main. [Merge 2.x to main after 2.4 release #1063](https://github.com/opensearch-project/sql/pull/1063). Pending on [change case-insensitive to false #1160](https://github.com/opensearch-project/sql/pull/1160)

2. Switch to main.

With merge 2.x to main, We will merge everything from 2.x to main, and will pickup #1150 automatically.

Thanks @penghuo , I will create a pull request for https://github.com/opensearch-project/OpenSearch/commit/c6663fd7f4e74b3ed1b6ad70e758b3e6bd21f816 only


Add Drips?

🔎 Search Terms
drips, funding, open source
The problem
I'm not sure the extent to which other maintainers are in the web3/crypto ecosystem, but I'm in a position where funds are especially tight and something like https://www.drips.network/ seems useful. We'd have to decide on splits among those accepting cryptocurrencies; if @DABH is in that set I recommend he propose first.

@wbt Yeah let's go ahead and try to monetize.  I guess we can do both Drips and GitHub Sponsors, so we aren't locked in too much if people don't use Drips.
How about 50/50 and we just split the pie further if more people want to be maintainers?  We could also do this across the whole ecosystem, e.g. winston-mongodb is maintained by me and someone else.
One additional thing we could do is sell an "enterprise" or "pro" version that has things like guarantees around security patches, support + response times for support requests, prioritize feature requests, etc.  I know some big companies use winston.  It would not be hard to set up the legal side of things to do that (LLC etc.).  I've thought about this for a while, and seems like an easy win (what's $1k/yr to a Big Co), but have never had time to make a little marketing website or a write up about the "pro" version.
Lmk your thoughts!

@maverick1872 as a possible maintainer, and also thinking we give some allocation to the original authors?

I'm on-board with the idea. I don't have much in the way of thoughts about how to go about it personally but I think it's worthwhile to pursue. There are definitely large companies that do use Winston and could likely see that as a potential source of funding. That'd be second on the list IMO over just enabling crowd-sourced funding though.


windows agent and ctl update to 6.1.0

Pull Request Checklist
General

[X] Update Changelog following the conventions laid out here

[X] Update README with any necessary configuration snippets

[X] Cookstyle (rubocop) passes

[X] Foodcritic passes

[X] Rspec (unit tests) passes

[X] Inspec (integration tests) passes

New Features
N/A
Purpose
Updated windows agent and ctl version to 6.1.0 (October 5, 2020)
Corrects undefined_method error found while validating.
Known Compatibility Issues
Chef 15+ replaces seven_zip resource with archive_file which is detected by cookstyle and failes rspec tests related to this resource.

released: https://supermarket.chef.io/cookbooks/sensu-go/versions/1.2.0


Paper reproduction

Hello, thank you for your excellent work. I'm reproducing the results. First, I ran get_ mulway_ base_ data. Py file, the corresponding train and val data are obtained: 0, 1, 2, 3.
I finished running train_ base. SH file to get Figure 1

Then run train SH, but 75 epoch is still 0, as shown in Figure 2

What is the reason

Hi,
Please make sure that the trained base model is saved under the right directory, as shown below:
https://github.com/chunbolang/BAM/blob/main/model/BAM.py#L62

Thank you for your help. I have reproduced the results. Now I want to use my own data set, but I encounter these problems.

I change yaml's classes to 2 and start train SH successfully reproduced the result. Why change it to 2.
My data classification is: 1, 2, 3. sub_ list=[1, 2],  sub_ val_ List = [3], which parameter values of yaml file need to be modified, such as classes
Sorry to disturb your life. Thank you for your help

args.classes=2 indicates the fg-bg (binary) segmentation task.
In your case, modifying the config file is not mandatory; you need to update the list and relevant paths in the dataset.py file .  Before this, please make sure that you have established a PASCAL-like dataset containing img and corresponding mask.
Furthermore, you need to refer to this code to generate the .txt files specific to your dataset.  An alternative is to comment out the L172-198 of code and keep only the L164-170.

Thank you for your reminder. My data set has been started. At present, the training effect is good. Thank you again for your help. If I encounter new problems, I'll contact you again. I wish you a happy life

Hello, I updated the list, train_ base. SH finished running, but I ran train SH has encountered a new problem. My dataset mask category: [1], [1, 2], [2], [2, 3], [1, 2, 3], [3].
This is my self sub_ val_ list=[3]，self. sub_ List = [1, 2], the code will report an error when it runs to the location of the screenshot. According to this logic, only the mask of 3 cannot be added to the list, so there is an error in displaying assert len (label_class) > 0. How can I solve this problem，Trouble you again

Under the current setting (i.e., self.sub_ val_ list=[3], self. sub_ List = [1, 2]),
I think the make_dataset function will filter the images that contain only category 3 (see here), so please check that the derived self.data_list is correct.


Improve documentation

Currently, the walker project has basically no documentation other than what the readme shows. The example config file doesn't contain everything, and doesn't explain what the individual options do. It is necessary to explore the code itself to even just see the which options are available, and even there the project doesn't have any comments or anything to potentially make it simpler to understand these options. Even though for some, it may be relatively clear, that's not necessarily the case for each of these values.
The project could also use some docs to describe what each of the CSS classes do, considering the styling is done with a custom CSS file.
I'd especially love to see documentation on the external modules, as it's fairly confusing to use in my opinion. I know that you have this example in your readme, which I did get to work:
    {
      "name": "filesystem",
      "prefix": "~",
      "src": "fd --base-directory /home/itsdrike/ %TERM%",
      "cmd": "xdg-open file://%RESULT%",
      "transform": true
    }

But there's no explanation on what's the point of that transform key here, how does src and cmd work? I assume src command gets ran on any user input, and cmd gets ran on the picked choice from that list? With this assumption then, I tried implementing a small custom external module, like this:
    {
      "name": "test",
      "prefix": "!",
      "src": "echo '%TERM%'",
      "cmd": "notify-send 'test' '%RESULT%'",
      "transform": true
    }

However, no items appeared in the menu, I don't understand why, there weren't any messages about this in the logs either. I then tried this:
    {
      "name": "test",
      "prefix": "!",
      "src": "ls %TERM%",
      "cmd": "notify-send 'test' '%RESULT%'",
      "transform": true
    }

For which, when there wasn't any input, it did show the ls items of the directory I ran walker from, however, the moment I started typing anything, the log started showing: exit status 2. But why is that? Exit status 2 from ls, according to the man page, means: serious trouble (e.g., cannot access command-line argument). Not exactly helpful in knowing what exactly going on though.
I tried some other things, but ultimately, I just wasn't able to figure out how it's supposed to work. The project really needs to do a better job at describing what it can actually do, and how it should be configured.
Even just a few github wiki pages would be a massive improvement here

I've added a wiki.
For styling: you'll find a list in the wiki... although I'm not sure where further explanation is needed. If something is unclear to anyone, they'd need to open an issue and ask. I can only guess so much. Something that's are clear to me, might not be to others. Those are blind-spots that'll stay blind-spots until someone sheds a light on them.
For config: you'll find a complete json in the wiki. Where needed, i added comments. Some things are self-explanatory in my eyes, but what i said about the styling applies here as well obviously.
For the plugins/external modules:
The transform-option has just been killed by me, it didn't serve any purpose other than telling Walker to transform the output of a cmd into entries for Walker, which is always the case as soon as there's a cmd present. So transform is redundant anyways.
I've tested your echo/notify-send example and it worked just fine. I've fixed a little bad code that might have caused a bug... maybe it was related.

Thanks for adding the documentation, also, the echo custom module does work for me too now, so the bug seems to have been fixed.


unable to install neon-js via yarn and commit hash

When trying to install neon-js with yarn and specifying a branch, tag, commit etc the installation fails with error Can't add "neon-js": invalid package version undefined. because package.json is missing the version key.
If we add a version at ./package.json users will not have this issue

are you trying to install with hash? Can you provide an example because I can install with version tag.
Not sure if adding the version tag at the root package.json will help due to the mono repo style.

Sorry for not being more specific an example would be "neon-js": "git+https://github.com/cityofzion/neon-js.git#3.11.0" or  "neon-js": "git+https://github.com/cityofzion/neon-js.git#129b883fa6aff2df9b965b689886fe754d5c4800"

That will not work with the mono repo setup, the url points to the root of the mono repo and it is unable to find its way into the packages. You have to depend on npm instead of direct urls.