Fix NPE during empty table stats collection

Summary
This PR brings in the following fixed for a couple of issues observed.

Fix an incorrect check for right timestamps of table creation and last modified.
Handle a case where a table does not have any data files.

Also added unit tests to address the above cases.
Changes

[ ] Client-facing API Changes
[ ] Internal API Changes
[X] Bug Fixes
[ ] New Features
[ ] Performance Improvements
[ ] Code Style
[ ] Refactoring
[ ] Documentation
[ ] Tests

Testing Done

[X] Manually Tested on local docker setup. Please include commands ran, and their output.
[X] Added new tests for the changes made.
[ ] Updated existing tests to reflect the changes made.
[ ] No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
[ ] Some other form of testing like staging or soak time in production. Please explain.

Docker testing:
Created an empty table. emptytbl
openhouse@6c67d19838cc:/opt/spark$ hdfs dfs -ls /data/openhouse/db/emptytbl-e0ff80fe-6eeb-4df9-9d04-9437444b0761/ Found 1 items -rw-r--r--   3 openhouse supergroup       2002 2024-05-08 06:46 /data/openhouse/db/emptytbl-e0ff80fe-6eeb-4df9-9d04-9437444b0761/00000-69745f53-4772-44db-a180-919652f4f6da.metadata.json
scala> spark.sql("show tblproperties openhouse.db.emptytbl").show(false)
+------------------------------------------+-------------------------------------------------------------------------------------------------------------------------+
|key                                       |value                                                                                |
+------------------------------------------+-------------------------------------------------------------------------------------------------------------------------+
                                                                                |
|openhouse.creationTime                    |1715150780639                                                                                |
                                                                                |
|openhouse.lastModifiedTime                |1715150780639                                                                                |
                                                                                |
|openhouse.tableId                         |emptytbl                                                                                |
|openhouse.tableVersion                    |INITIAL_VERSION                                                                                |
                                                                                |
+-------------------------

Right tableCreationTimestamp populated and also no NPE due to absence of datafiles:
INFO spark.TableStatsCollectionSparkApp: Publishing stats for table: db.emptytbl","2024-05-08 07:56:04,464 INFO spark.TableStatsCollectionSparkApp: {\"totalReferencedDataFilesSizeInBytes\":0,\"numReferencedDataFiles\":0,\"totalDirectorySizeInBytes\":2002,\"numObjectsInDirectory\":1,\"numCurrentSnapshotReferencedDataFiles\":0,\"totalCurrentSnapshotReferencedDataFilesSizeInBytes\":0,\"numExistingMetadataJsonFiles\":1,\"numReferencedManifestFiles\":0,\"numReferencedManifestLists\":0,\"recordTimestamp\":1715154955889,\"clusterName\":\"LocalHadoopCluster\",\"databaseName\":\"db\",\"tableName\":\"emptytbl\",\"tableUUID\":\"e0ff80fe-6eeb-4df9-9d04-9437444b0761\",\"tableLocation\":\"/data/openhouse/db/emptytbl-e0ff80fe-6eeb-4df9-9d04-9437444b0761\",\"tableCreator\":\"openhouse\",\"tableCreationTimestamp\":1715150780639,\"tableLastUpdatedTimestamp\":1715150780639,\"tableType\":\"PRIMARY_TABLE\"}","2024-05-08 07:56:04,465
For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.
Additional Information

[ ] Breaking Changes
[ ] Deprecations
[ ] Large PR broken into smaller PRs, and PR plan linked in the description.

For all the boxes checked, include additional details of the changes made in this pull request.

non blocking questions: are we currently skipping failing tables ? If so, this fix is for those that are skipped ?

For runtime exceptions yes these would have been skipped. this table failed before the stats collection is complete and so no data was available for this. But there are handling of exceptions to publish data for a table without some stats if we run into exceptions while collecting a subset of stats.


[3.x] OSX: Add signing of directory structure when exporting for OSX

Port of #56047

Please amend the commit to use the original commit message, at least as the first line.

Please amend the commit to use the original commit message, at least as the first line.

This is done :)

Why some checks have been cancel ?

Why some checks have been cancel ?

I re-ran them manually; let's see if they complete this time.

Why some checks have been cancel ?

I don't know. I re-ran them manually; let's see if they complete this time.

Still fails with cancel :s

There's a GitHub Actions bug on this PR, when builds are queued for too long (which happens with macOS jobs as they're more limited so we run out of allocated resources fast) it cancels them.
I think it's because this was a first time contributor PR with manual approval, there must be a buggy codepath there.
I've tried to restart them 4 times but there's always an hour or more of waiting time for a macOS build so they get cancelled.
Maybe try to rebase and force push. Now that you have a merged PR it should no longer require approval and might side step that bug.
Though there seems to be another CI bug on the 3.x branch right now so maybe wait a bit.

Thanks!


raiplay

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I remove or skip any mandatory* field

Checklist

[X] I'm reporting a broken site
[X] I've verified that I'm running yt-dlp version 2022.08.14 (update instructions) or later (specify commit)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[X] I've read about sharing account credentials and I'm willing to share it if required

Region
Italy
Provide a description that is worded well enough to be understood
It seems that some URLs from raiplay do not work.
I haven't seen a pattern, just some do and some other do not.
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
$ yt-dlp -vU https://www.raiplay.it/programmi/ficodindia
[debug] Command-line config: ['-vU', 'https://www.raiplay.it/programmi/ficodindia']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.08.14 [55937202b] (debian*)
[debug] Python 3.10.6 (CPython 64bit) - Linux-5.16.0-1-amd64-x86_64-with-glibc2.34 (glibc 2.34)
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg 5.1-2 (setts), ffprobe 5.1-2, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.11.0, brotli-1.0.9, certifi-2020.06.20, mutagen-1.45.1, pyxattr-0.7.2, secretstorage-3.3.3, sqlite3-2.6.0, websockets-10.2
[debug] Proxy map: {}
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.08.14, Current version: 2022.08.14
yt-dlp is up to date (2022.08.14)
[debug] [RaiPlayPlaylist] Extracting URL: https://www.raiplay.it/programmi/ficodindia
[RaiPlayPlaylist] ficodindia: Downloading program JSON
[RaiPlayPlaylist] ContentSet-5c93f8ad-d049-4f8a-b365-5c8a7a6422f4: Downloading content set JSON
[download] Downloading playlist: Fico d'India
[RaiPlayPlaylist] Playlist Fico d'India: Downloading 1 videos of 1
[download] Downloading video 1 of 1
[debug] [RaiPlay] Extracting URL: https://www.raiplay.it/video/2016/07/Fico-d-India-ea6a12e9-69d1-4948-abed-65f315c6fc2f.json
[RaiPlay] ea6a12e9-69d1-4948-abed-65f315c6fc2f: Downloading video JSON
[RaiPlay] ea6a12e9-69d1-4948-abed-65f315c6fc2f: Downloading XML metadata for platform mon
[RaiPlay] ea6a12e9-69d1-4948-abed-65f315c6fc2f: Downloading m3u8 information
[RaiPlay] ea6a12e9-69d1-4948-abed-65f315c6fc2f: Downloading XML metadata for platform flash
[RaiPlay] ea6a12e9-69d1-4948-abed-65f315c6fc2f: Downloading XML metadata for platform native
ERROR: ea6a12e9-69d1-4948-abed-65f315c6fc2f: An extractor error has occurred. (caused by KeyError('200')); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "/usr/lib/python3/dist-packages/yt_dlp/extractor/common.py", line 666, in extract
    ie_result = self._real_extract(url)
  File "/usr/lib/python3/dist-packages/yt_dlp/extractor/rai.py", line 315, in _real_extract
    relinker_info = self._extract_relinker_info(video['content_url'], video_id)
  File "/usr/lib/python3/dist-packages/yt_dlp/extractor/rai.py", line 113, in _extract_relinker_info
    formats.extend(self._create_http_urls(relinker_url, formats))
  File "/usr/lib/python3/dist-packages/yt_dlp/extractor/rai.py", line 205, in _create_http_urls
    **get_format_info(q)
  File "/usr/lib/python3/dist-packages/yt_dlp/extractor/rai.py", line 180, in get_format_info
    'width': _QUALITY[tbr][0],
KeyError: '200'

[download] Finished downloading playlist: Fico d'India

@nixxo

sorry, a mistake introduced in one of my latest PR. I'll fix it asap.


Receive: Possible memory leak with TSDB growth

Thanos, Prometheus and Golang version used:
I have tested with both 0.8.1 and 0.9.0. Most recent version output is as follows:
  build user:       circleci@8e3da52515a2
  build date:       20191203-17:03:13
  go version:       go1.13.1

Object Storage Provider: N/A
What happened:
When receiver runs for a long period of time it's memory grows until reaching OOM kills
What you expected to happen:
A steady memory footprint
How to reproduce it (as minimally and precisely as possible):
Run thanos receive with a retention time of 5x the block size (default 2h). When the first WAL flush occurs you will see the memory usage start to grow. With each WAL flush there is exponentially more memory usage.
Full logs to relevant components:

Anything else we need to know:

I initially found this problem when our servers started an OOM loop. I tried to clear the WAL and restart to no avail. The only solution was to delete some local TSDB blocks. With each block I deleted it lowered the memory footprint. I finally restarted fresh and found the memory usage to be level and consistent until the first WAL flush at which point there was a small spike (probably from the flush) and then a slope.

You can see the memory growth here. I have the blocks set to 1h which explains the spikes.
UPDATE: it appears that restarting exacerbates the problem. I didn't change the hashring.json or do anything but restart the thanos service.

I pushed master-2019-12-10-352cc305 (which includes the fix from #1856) to one machine in our cluster and it seems to be growing much slower than the rest.

The top line is the machine with master-2019-12-10-352cc305. The yellow line is the average across the cluster and the bottom line is the min().

@bwplotka the issue seems to be resolved with #1856 . The issue I was having with rebooting nodes and doubling memory usage was caused by thrashing. When we rebooted machines the hashring was updated by our watcher app, which monitors availability and updates the hashring. I believe this issue can be closed now with 352cc305


Update building.md

Added alternative method to build project on Macs

This change is

docs/building.md, line 35 at r1 (raw file):
Previously, SanjayVas (Sanjay Vasandani) wrote…

If the above script does not work

I'm still not certain why this is, as the instructions specified here don't seem meaningfully different than what the script is doing. The script adds the following:

Mounting the Bazel and Bazelisk caches.
Detecting if Docker is being run in rootless mode, and specifying the user and group IDs if needed so that mounted files get the right permissions.

Perhaps the cache has been poisoned and running tools/bazel-container clean --expunge would fix things?
Unfortunately I've only been able to test with my local configuration of rootless Docker on Linux-amd64, with a past additional test of rootless Podman.

I'm getting a "The 'clean' command is only supported from within a workspace (below a directory having a WORKSPACE file)." error. Is it supposed to be run in a specific directory?

docs/building.md, line 35 at r1 (raw file):
Previously, SanjayVas (Sanjay Vasandani) wrote…
The tools/bazel-container script is always assumed to be run within the directory where you have the cross-media-measurement source. Hence it should always be run as tools/bazel-container rather than some other path. The script mounts the local directory in the container and runs bazel there.

I realized I had slightly modified the tools/bazel-container file, and when I reverted it it ran. Unfortunately it didn't seem to work well (Mac had to kill the stale process) and using tools/bazel-container to build afterwards still failed (ERROR: /Users/cpy/git/cross-media-measurement/src/main/swig/protocol/liquidlegionsv2/BUILD.bazel:8:13: TreeArtifact src/main/swig/protocol/liquidlegionsv2/java_files was not created ERROR: /Users/cpy/git/cross-media-measurement/src/main/swig/protocol/liquidlegionsv2/BUILD.bazel:8:13: SwigCompile src/main/swig/protocol/liquidlegionsv2/liquid_legions_v2_encryption_utility.cc failed: not all outputs were created or valid).

docs/building.md, line 35 at r1 (raw file):
Previously, SanjayVas (Sanjay Vasandani) wrote…
Hmm, so weird that happens. I wonder if there's some oddness with file system operations within the mounted Bazel cache dir. In this particular example, the rule just shells out to mkdir to make the java_files directory.

Discovery update: I printed the command being run, and after removing "--env BAZELISK_HOME=/Users/cpy/.cache/bazelisk" from the command it actually succeeded. Seems to possibly have something to do with the cache specification.

docs/building.md, line 35 at r1 (raw file):
Previously, chipingyeh wrote…
Discovery update: I printed the command being run, and after removing "--env BAZELISK_HOME=/Users/cpy/.cache/bazelisk" from the command it actually succeeded. Seems to possibly have something to do with the cache specification.

Actually I mistyped - I removed the 2 other --mount flags that were in the statement (--mount type=bind,source=/var/tmp/_bazel_cpy,target=/Users/cpy/.cache/bazel/_bazel_cpy --mount type=bind,source=/Users/cpy/Library/Caches/bazelisk,target=/Users/cpy/.cache/bazelisk) after the first --mount flag, not the cache statement. Do we need all 3 --mount flags?


where is the storage folder for the image? the web won't load the image

where can i find the image assets? so the web can load existing images gambar

same

same

@mujibluth I uploaded storage.zip in the root level. Extract that folder and replace the root level storage folder.


JavaScript now has 64-bit integer support

With the addition of BigInt, JavaScript now has a type that can reliably store snowflakes as Integers. This type has been in node.js for almost a year now, and uses an average of 40% less memory to store snowflakes compared to Strings. I haven't tested it, but I also suspect this will be easier and/or faster to convert to than Strings are as well. For full disclosure, tests done by the Dyno team showed that the 40% decrease in memory for snowflakes made no noticeable difference in the total memory usage. I'm not stating that erlpack absolutely must switch over, as this would also be a major breaking change for the JavaScript DAPI libraries that utilize erlpack, but I thought I would at least bring up the idea and invite discussion.

we experimented with this in discord.js a bit (https://github.com/discordjs/erlpack/tree/feature/bigint)


chore: bump narwhals to v1.14.2

This PR updates pyproject.toml to update narwhals to v1.14.2.
Note: v1.14.2 is not yet on conda-forge (https://anaconda.org/conda-forge/narwhals/files)

hey - just a quick note to clarify that I don't think the bump is strictly required - the code in Altair works perfectly fine as-is, this bump would only be necessary if you wanted to do
from narwhals.stable.v1.dependencies import is_pandas_dataframe
if is_pandas_dataframe(data):
    ...

instead of
import narwhals.stable.v1 as nw
if nw.dependencies.is_pandas_dataframe(data):
    ...

@MarcoGorelli that's what I wanted to do in my PR but @mattijn beat me to it 😅

Thanks @MarcoGorelli, good to know that current situation is fine as is. Will leave this PR open until @dangotbanned finds time to implements these new feature possibilities.

Thanks @MarcoGorelli, good to know that current situation is fine as is. Will leave this PR open until @dangotbanned finds time to implements these new feature possibilities.

Doing this now @mattijn thank you


Cannot import pyfar when working with write protected images (Singularity)

General

pyfar version: 0.5.3
Python version: 3.9
Operating System: Ubuntu (Singularity Container)
Did you install pyfar via pip: yes

Description
I tried to import pyfar from a Python instance running inside a singularity container. However, since I do not have any permission to create a directory inside a write protected image, an error was raised related to the following line:
https://github.com/pyfar/pyfar/blob/a556fb93a6a3e0ea1d196726a89dad30095a422b/pyfar/signals/files.py#L27-L31
There should be some way to manually specify the file path. Alternatively, I believe it would be better to only create the directory if needed.

I'm not sure if we want to handle user specific paths, since it would require some sort of preference file, which might cause follow up issues. I will leave this for discussion in the weekly. Here is a quick fix for now: https://github.com/pyfar/pyfar/tree/fix/run_in_read_only_containers


How to disable this popup?

How to disable popups like this that appears after click "Enter" button?

You can use filters to prevent certain descriptions from showing.

@gsrafael01 Filters don't works anymore. I don't get why my issue was closed as invalid, while the bug exists.

@gsrafael01 If I put ".*" as filter, the popup still pops out. It's a bug? How to completely disable this popup?

@Ack0 it's a bug for sure, I had it working with .* for months. Then suddenly stopped working even if set correctly. But my ticket was closed as invalid too. I don't get it.

It's not a bug. Please read the changelog and search for similar issues before creating a new one, I've already answered this multiple times. You can no longer completely disable the popup, you have to target specific descriptions with the filters (for this, you need to see the descriptions, of course).

Thansk for the answer @gsrafael01
Just to confirm: we can't block all popups anymore with ".*"? If yes, may I ask why you? Honest question.

It was a decision I made as a developer, in the interest of the community, to make sure that giveaway descriptions are shown, and hopefully read. You can still use the filters to prevent specific descriptions from showing (using the "Add Description To Filters" button to add its format to the list).
If blocking all popups is something that you, for some reason, cannot live without, then you're free to fork the project and make your own modifications, or use JustArchi's fork, because his fork doesn't have this restriction.

Good to know, I'll fork it then

@gsrafael01 Thanks for the explanation. I understand and respect your point.
I'll stick with JustArchi's fork, because I help the community in other ways and I think that this popup kills the usability for me.
Anyway, keep up the good work.


ViaVersion v1.6.0 Task Exception Spamming Console

Describe the bug, provide any errors
This task exception is being spammed continuously in my console https://hastebin.com/vudaxosape.pl
How can we reproduce it?
Steps to reproduce the behavior:

Use Spigot 1.12.2
Using latest version of ViaBackwards and ViaRewind

Expected behaviour
N/A
Screenshots
N/A
ViaVersion Dump:
https://dump.viaversion.com/8ac8d8c645161aa164b8bc3ef0aba46271f50533de793d1cbf04c79a026ef974
Additional server info
I use BungeeCord and Spigot

Update ViaRewind


Keep a folder closed option

It would be nice to be able to keep a folder always closed, for example the node modules folder that is very big and open itself for unknown reasons.
Another right click on the folder would give the opportunity to remove the "keep closed" option.

This has been solved in another way with the "close all folder" button introduced in the 1.42.
Great work ! This 1.43 is really good !!


deb.sury.org has updated their repo key, new release required

Describe the bug
According to deb.sury.org patreon post the deb.sury.org key has been updated.
This means that webimage_extra_packages and related things are likely to break as the old one expires.
We need to do a v1.16.7 to beat this back I guess.

This was fixed in #2834 - thanks to @unn for pointing out the upcoming trouble.
I intend to add a test that will detect expiring keys in the future, to let us be more automatically proactive about this kind of problem.


Error converting multilingual universal sentence encoder to TFLite. Input 1 of node StatefulPartitionedCall was passed float from statefulpartitionedcall_args_1:0 incompatible with expected resource.

System information

OS Platform and Distribution: Ubuntu 19.10
TensorFlow installed from (source or binary): pip install tensorflow==2.3.0

Command used to run the converter or code if you’re using the Python API
If possible, please share a link to Colab/Jupyter/any notebook.
# I've downloaded model and unarchived it to save_path
converter = tf.lite.TFLiteConverter.from_saved_model(save_path)
tflite_model = converter.convert()

InvalidArgumentError                      Traceback (most recent call last)
~/.local/lib/python3.7/site-packages/tensorflow/python/framework/importer.py in _import_graph_def_internal(graph_def, input_map, return_elements, validate_colocation_constraints, name, producer_op_list)
    496         results = c_api.TF_GraphImportGraphDefWithResults(
--> 497             graph._c_graph, serialized, options)  # pylint: disable=protected-access
    498         results = c_api_util.ScopedTFImportGraphDefResults(results)

InvalidArgumentError: Input 1 of node StatefulPartitionedCall/sequential/keras_layer/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall was passed float from Func/StatefulPartitionedCall/sequential/keras_layer/StatefulPartitionedCall/StatefulPartitionedCall/input/_1007:0 incompatible with expected resource.

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-10-55fd8585264a> in <module>
      1 #convert model to tensorflow lite
      2 converter = tf.lite.TFLiteConverter.from_saved_model(save_path)
----> 3 tflite_model = converter.convert()
      4 # open("converted_model.tflite", "wb").write(tflite_model)

~/.local/lib/python3.7/site-packages/tensorflow/lite/python/lite.py in convert(self)
   1074         Invalid quantization parameters.
   1075     """
-> 1076     return super(TFLiteConverterV2, self).convert()
   1077
   1078

~/.local/lib/python3.7/site-packages/tensorflow/lite/python/lite.py in convert(self)
    876     frozen_func, graph_def = (
    877         _convert_to_constants.convert_variables_to_constants_v2_as_graph(
--> 878             self._funcs[0], lower_control_flow=False))
    879
    880     input_tensors = [

~/.local/lib/python3.7/site-packages/tensorflow/python/framework/convert_to_constants.py in convert_variables_to_constants_v2_as_graph(func, lower_control_flow, aggressive_inlining)
   1107
   1108   frozen_func = _construct_concrete_function(func, output_graph_def,
-> 1109                                              converted_input_indices)
   1110   return frozen_func, output_graph_def
   1111

~/.local/lib/python3.7/site-packages/tensorflow/python/framework/convert_to_constants.py in _construct_concrete_function(func, output_graph_def, converted_input_indices)
    999   new_func = wrap_function.function_from_graph_def(output_graph_def,
   1000                                                    new_input_names,
-> 1001                                                    new_output_names)
   1002
   1003   # Manually propagate shape for input tensors where the shape is not correctly

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/wrap_function.py in function_from_graph_def(graph_def, inputs, outputs)
    648     importer.import_graph_def(graph_def, name="")
    649
--> 650   wrapped_import = wrap_function(_imports_graph_def, [])
    651   import_graph = wrapped_import.graph
    652   return wrapped_import.prune(

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/wrap_function.py in wrap_function(fn, signature, name)
    626           signature=signature,
    627           add_control_dependencies=False,
--> 628           collections={}),
    629       variable_holder=holder,
    630       signature=signature)

~/.local/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
    984         _, original_func = tf_decorator.unwrap(python_func)
    985
--> 986       func_outputs = python_func(*func_args, **func_kwargs)
    987
    988       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/wrap_function.py in __call__(self, *args, **kwargs)
     85
     86   def __call__(self, *args, **kwargs):
---> 87     return self.call_with_variable_creator_scope(self._fn)(*args, **kwargs)
     88
     89   def call_with_variable_creator_scope(self, fn):

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/wrap_function.py in wrapped(*args, **kwargs)
     91     def wrapped(*args, **kwargs):
     92       with variable_scope.variable_creator_scope(self.variable_creator_scope):
---> 93         return fn(*args, **kwargs)
     94
     95     return wrapped

~/.local/lib/python3.7/site-packages/tensorflow/python/eager/wrap_function.py in _imports_graph_def()
    646
    647   def _imports_graph_def():
--> 648     importer.import_graph_def(graph_def, name="")
    649
    650   wrapped_import = wrap_function(_imports_graph_def, [])

~/.local/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py in new_func(*args, **kwargs)
    505                 'in a future version' if date is None else ('after %s' % date),
    506                 instructions)
--> 507       return func(*args, **kwargs)
    508
    509     doc = _add_deprecated_arg_notice_to_docstring(

~/.local/lib/python3.7/site-packages/tensorflow/python/framework/importer.py in import_graph_def(***failed resolving arguments***)
    403       return_elements=return_elements,
    404       name=name,
--> 405       producer_op_list=producer_op_list)
    406
    407

~/.local/lib/python3.7/site-packages/tensorflow/python/framework/importer.py in _import_graph_def_internal(graph_def, input_map, return_elements, validate_colocation_constraints, name, producer_op_list)
    499       except errors.InvalidArgumentError as e:
    500         # Convert to ValueError for backwards compatibility.
--> 501         raise ValueError(str(e))
    502
    503     # Create _DefinedFunctions for any imported functions.

ValueError: Input 1 of node StatefulPartitionedCall/sequential/keras_layer/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall was passed float from Func/StatefulPartitionedCall/sequential/keras_layer/StatefulPartitionedCall/StatefulPartitionedCall/input/_1007:0 incompatible with expected resource.

https://tfhub.dev/google/universal-sentence-encoder-multilingual/3
I've tried the large model also and got the same error. Can someone help me?

This model can be converted using TF_SELECT option, since it has some unsupported operations by TF Lite.
See https://www.tensorflow.org/lite/guide/ops_select
Also, added @abattery who converted this model before and can guide you if needed.

@abattery Hi! Could you help me?
This approach doesn't work either
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir) converter.target_spec.supported_ops = [tf.lite.OpsSet.SELECT_TF_OPS] tflite_model = converter.convert()

Sorry @Extremesarova
Actually, this model requires e2e hash table support. We are working on delivering the e2e hash table feature. I will update this thread when the feature is landed.

Got it.
Could you, please, suggest some pretrained multilingual embeddings from tf.hub, that can be converted to Tensorflow Lite?

I had experiments on converting models in TF hub, especially for the models which need a hash table support. Except few exceptions, most of models will be convertible to TFLite after e2e hash table proposal including https://tfhub.dev/google/universal-sentence-encoder-multilingual/3.

@abattery Hi! Could you help me with some workaround before delivering this feature? May you provide e2e hash table or something so that I can convert multilingual USE?

I had experiments on converting models in TF hub, especially for the models which need a hash table support. Except few exceptions, most of models will be convertible to TFLite after e2e hash table proposal including https://tfhub.dev/google/universal-sentence-encoder-multilingual/3.

I'm also curious if there are any updates? Or a time-line

Hi, guys! Any updates on the problem?

@abattery Is there any update on the e2e hash table? I'm also running into this issue. Thanks!

@abattery Is there any update on the e2e hash table? I'm also running into this issue. Thanks!

Hey guys there's a new version (4) of the sentence encoder available now.
https://tfhub.dev/google/universal-sentence-encoder/4
I tried converting but failed... maybe someone else wants to try and see if they can get it to work?

Any update would be great!

There is a better support on the recent TF version through the tf.lite.TFLiteConverter.from_saved_model.

was able to do the conversion using the latest tf version, however was running into problems using it on mobile.
this conversion worked for me
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.target_spec.supported_ops = [
  tf.lite.OpsSet.TFLITE_BUILTINS, # enable TensorFlow Lite ops.
  tf.lite.OpsSet.SELECT_TF_OPS # enable TensorFlow ops.
]
tflite_model = converter.convert()

But on mobile i get the error
Regular TensorFlow ops are not supported by this interpreter. Make sure you apply/link the Flex delegate before inference
I think https://www.tensorflow.org/lite/guide/ops_select#android_aar may be related, but wasnt able to get it to work as of now.
see https://github.com/am15h/tflite_flutter_plugin/issues/101 for more details

@Extremesarova,
Thanks for the update and this guide can be a good reference for you to test inference of tf-lite model.

@sanatmpa1, hi!
I've tried everything and the inference on Android's backend doesn't work - error concerns the lack of SentencepieceOp.
But this is another issue, because this issue was about error in convertion to tf-lite - and it seems to work now.

@Extremesarova,
Thanks for the confirmation. In this case where it seems to work now, Can you close this issue and open a new one for Android backend issue, if it still persists in recent stable version?


Can't see CPU Frequency in Perfetto UI (chrome works fine)

Hi!
I've recorded a trace file having checked the CPU Frequency checkbox (for that purpose, I use old monitor tool UI which still works on Windows).
Opened the trace file in Perfetto UI. Works well, except for I cannot see the CPU frequency (should be the height of the bar in the CPU section?).
Checked in chrome://tracing - the data is there.
I'm happy to share the trace file but for some reason github doesn't want it here.
Also tried to report the bug from Perfetto UI itself - redirects to moma internal Google system which I don't have access too.
Thanks!

I think this might be an old outstanding issue where cpu frequency is displayed only if cpuidle information is also present (because they are rendered in the same track).
Can you try also enabling cpuidle recording?

Thank you. Just tried - no luck. Maybe the device doesn't allow for cpu_idle collection (got an EU S8 SM-G950F)?

There's one way to tell:  open the "query" tab and paste the current query:
select name, count(1) from counters where name like 'cpu%' group by name
if it shows only cpufreq, that's the issue.
On a good trace it shows something like this:

ha, it only shows cpuidle and no cpufreq. (but the frequency is still there in chrome tracing!)

That's really weird. Which format is the trace? how did you capture it?
Is there any chance you can attach it somewhere?

The trace has entries like
cfinteractive-953   (  953) [000] ...1  1461.607586: cpu_frequency: state=1456000 cpu_id=0
cfinteractive-953   (  953) [000] ...1  1461.607594: cpu_frequency: state=1456000 cpu_id=1
cfinteractive-953   (  953) [000] ...1  1461.607598: cpu_frequency: state=1456000 cpu_id=2
cfinteractive-953   (  953) [000] ...1  1461.607601: cpu_frequency: state=1456000 cpu_id=3
Try here: https://drive.google.com/file/d/1zYVbRciZvNvNiLgSpG5SDCbL45SG37cD/view?usp=sharing

Ahh, ahahah, yes. The problem is here:
https://github.com/google/perfetto/blob/8d2a9ab275b5761104956281bf07a41f4ecb143a/src/trace_processor/importers/systrace/systrace_line_parser.cc#L115
The text importer does not handle cpufreq.
The text importer is maintained best-effort and has some missing features. Most traces are captured directly through adb shell perfetto in the new proto format.
That seems not too hard to fix. If you were willing to contribute we would happily accept a fix for that.
Essentially we need some code in systrace_line_parser.cc that does something similar to FtraceParser::ParseCpuFreq (The latter is the proto-based parser).
See also https://perfetto.dev/docs/contributing/getting-started for contrib instructions.

Hello,
I am Beginner and I want to contribute. can you please guide me how i can reproduce this?

Here's a link to a systrace which contains cpu frequency events. To fix this issue, you basically need to follow what Primiano said in the comment above yours.
You can test if your change works by building trace_processor_shelland running:
<out directory>/trace_processor_shell <path to trace>
If you then write the query:
select * from counter_track
you should see some cpu_frequency tracks if it worked correctly.
Feel free to hop into our Discord if you want more pointers without the latency of commenting back and forth on GitHub :)

ok thanks. I will try it now.

This has now been implemented


[modify] kana convertor kana_html skip_attributes

Coverage increased (+0.06%) to 84.342% when pulling 7a3b3492da514ab2266e54e8a3be517d2dfe75a2 on pr-kana-convertor-skip-attributes into f4bd6e5b9ffc9cc1f72f1f2569cf8cf0df77df44 on master.


Error (freezes nvim) when opening markdown with code block name in different case (e.g. "Python", "JSON", "R"): no such language: R

Describe the bug
This is a macOS only bug, I cannot reproduce it in linux.
When opening a markdown file that contains code block with same name with the treesitter parser but have different case (e.g. python vs Python, r vs R, Json vs json), you will get the following error:
stack traceback:
        [C]: in function '_ts_parse_query'
        ...m/0.10.0/share/nvim/runtime/lua/vim/treesitter/query.lua:252: in function 'fn'
        ...ovim/0.10.0/share/nvim/runtime/lua/vim/func/_memoize.lua:58: in function 'fn'
        ...ovim/0.10.0/share/nvim/runtime/lua/vim/func/_memoize.lua:58: in function 'get'
        ...0/share/nvim/runtime/lua/vim/treesitter/languagetree.lua:128: in function 'new'
        ...0/share/nvim/runtime/lua/vim/treesitter/languagetree.lua:511: in function 'add_child'
        ...0/share/nvim/runtime/lua/vim/treesitter/languagetree.lua:395: in function '_add_injections'
        ...0/share/nvim/runtime/lua/vim/treesitter/languagetree.lua:447: in function 'parse'
        ....0/share/nvim/runtime/lua/vim/treesitter/highlighter.lua:397: in function <....0/share/nvim/runtime/lua/vim/treesitter/highlighter.lua:392>
Error in decoration provider treesitter/highlighter.win:
Error executing lua: ...m/0.10.0/share/nvim/runtime/lua/vim/treesitter/query.lua:252: no such language: R

This error is actually the same error as in nvim-treesitter: https://github.com/nvim-treesitter/nvim-treesitter/issues/6642
Steps to reproduce

open the following example org mode file, say : hello.org

Open it in neovim, get billions of errors so that you can't do anything with neovim.

https://pastebin.com/sSLFxXBZ
Expected behavior

nvim should open this file normally.

Emacs functionality
No response
Minimal init.lua

local plugins = {
	orgmode = "https://github.com/nvim-orgmode/orgmode",
	ts = "https://github.com/nvim-treesitter/nvim-treesitter",
}

for name, url in pairs(plugins) do
	local install_path = "/tmp/nvim/site/" .. name
	if vim.fn.isdirectory(install_path) == 0 then
		vim.fn.system({ "git", "clone", "--depth=1", url, install_path })
	end
	vim.o.runtimepath = install_path .. "," .. vim.o.runtimepath
end

require("nvim-treesitter.configs").setup({
	-- One of "all", "maintained" (parsers with maintainers), or a list of languages
	ensure_installed = {
		"r",
	},
	sync_install = true,
	highlight = {
		enable = true,
	},
})
require("orgmode").setup()

Screenshots and recordings

OS / Distro
macos 13.4
Neovim version/commit
0.10
Additional context
R and r are both considered as "Canonical" name for R language, so you would expect to see different authors use R and r interchangably from different sources. While treesitter uses r as its name, the orgmode should handles language name R robustly.

Apply the following patch solves the problem for me.
According to this commit, https://github.com/neovim/neovim/pull/28546, treesitter requires the parser name should be always lower case, so we can safely convert the text to lower case.

diff --git a/lua/orgmode/config/init.lua b/lua/orgmode/config/init.lua
index a9db1bf..75e7ca7 100644
--- a/lua/orgmode/config/init.lua
+++ b/lua/orgmode/config/init.lua
@@ -397,7 +397,7 @@ function Config:setup_ts_predicates()
     if not text or vim.trim(text) == '' then
       return
     end
-    metadata['injection.language'] = utils.detect_filetype(text) or text
+    metadata['injection.language'] = utils.detect_filetype(text) or text:lower()
   end, { force = true })
 end


Update to 1.5.2 and port of demo project

I ported the ExoPlayer demo application to C#

Looks good to me! Huge thanks for helping on this! Is it ready to merge?

I'm adding some stuff. Pushing some more in some minutes.

Now it's ready to go :)

Thanks again! I'll try to push an update soon.


Module not found: Can't resolve 'path' on NextJS edge runtime.

Summary
When running langchainjs on the edge runtime in NextJS I see the following exception:
Error: Module not found: Can't resolve 'path'

https://nextjs.org/docs/messages/module-not-found

Import trace for requested module:
./node_modules/langchain/dist/base_language/count_tokens.js
./node_modules/langchain/dist/llms/openai.js
./node_modules/langchain/llms/openai.js

Currently running version 0.0.63.

See https://github.com/hwchase17/langchainjs/tree/main/test-exports-vercel for a working example

yep, I use the Next13 appDir route handlers. Perhaps that env differs somehow.
https://beta.nextjs.org/docs/routing/route-handlers

Me too.

I believe it's because of this line:
https://github.com/hwchase17/langchainjs/blob/main/langchain/src/base_language/count_tokens.ts#L62
Not sure why - maybe Next.js 13 app dir route handlers with runtime: 'edge' import files differently?
Commenting this line fixes the error.
@nfcampos I understand it is intended that tiktoken will not be used in edge/browser (and be allowed to error here)?

Ah interesting, thanks for investigating!
@dqbd any thoughts on this one? Would it be fixed/fixable with https://github.com/hwchase17/langchainjs/pull/847

Hello! That definitely sounds like an issue with @dqbd/tiktoken, will address it as soon as possible

Cool, thank you!


[FEATURE] Support updating model/connector on they fly

When update model, ml-commons will check if model deployed or not. User need to undeploy model first before updating.  That will cause service interruption.
We should support updating model/connector on the fly without undeplying it.
For example user need to rotate their credential in connector, but they don't want to undeploy their model.

https://github.com/opensearch-project/ml-commons/pull/1673


Discuss limits on wallet values

The notion of limits on wallet values has been raised in order to provide some protections to users in cases of security compromise.
Please use this ticket to discuss the issues surrounding that proposal.
Questions to answer:

Is it a good idea generally?
Does it belong on the core wallet?
Is it something we'd be looking to implement?

@litecoincash-project based on the fact that I can't think of any good reason to have millions of dollars in a software wallet then for me it is imperative that devs implement this sort of protection for their users. Without it, presumably at some point legal action will be taken against wallet providers that are hacked.

Without meaning to sound facetious, just because you can't think of a good reason, doesn't mean that no-one will be able to!
There are a few obvious issues at play here, and one is scale. What is a reasonable amount to lock a wallet down to for one person, might be drastically too much, or too little to be useable to some one else. A functioning currency infrastructure has to scale to support all of its users, so once you start getting to the place where a user needs to set their own limit, then its a little arbitrary and somewhat negates the argument to do it in software at all.
Another potential issue with this is the implementation. If this is something you expect 3rd party wallet providers to implement, well their hands are a bit tied by the state of the blockchain coming in. Just because a client has decided that some particular wallet has become "full" - what is it supposed to do with any further transactions to those associated addresses that are coming in and completely valid on the blockchain?
To actually implement such a limit, you would have to do it in the in the coin itself, on the network using the consensus rules, which is obviously down to specific coin developers rather than wallet providers. Incidentally, without wanting to give too much away, some of these safety and protection oriented features are precisely the kinds of things we are looking at including in future version of Litecoin Cash.
As for your presumption that, at some point, developers will be held legally liable for losses resulting from use of their software, well there is an industry standard piece of language boiler plate that is almost universally found in every software license that you agree to by using the software, and it goes like this:

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Software licenses are legally binding documents, though I personally am not aware how far that particular clause has been tested in court.

I suspected you would come back with the "AS IS" defence and not something I would rely on.
The blockchain is not a problem. Just warn users that their wallet is close to the limit and that you will donate any surplus coins that arrive at the wallet to WikiLeaks or UNICEF.
I don't buy the scale issue as big money crypto should be handled with offline private keys from a hardware wallet.

Published an article warning of the dangers of using digital wallets https://cryptodisrupt.com/software-wallets-for-cryptocurrencies-need-a-radical-redesign-before-widespread-adoption/


Ensure all pages have correct titles

Trello card
Trello-3248
Trello-3249
Context
Make sure all pages have an appropriate page title in the format:
Main heading - Service name - GOV.UK
If there is an error on the page then the title should be prefixed with Error:.
Changes proposed in this pull request

Ensure all pages have a correct page title
Correct typo in make command

Guidance to review
Changes inline with GDS guidance

@peteryates that's already in the PR, albeit in a slightly different format?
https://github.com/DFE-Digital/get-teacher-training-adviser-service/pull/1021/files#diff-5f3fc5e3977d242572aa1d08551f5eb557de0ccaff30370838ee9df5386ea0daR13
def format_page_title(title, status)
  prefix = status == 422 ? "Error: " : ""
  title = [title, "Get an adviser", "GOV.UK"].compact.join(" - ")

  "#{prefix}#{title}"
end


[ci] Fix false-positive Bazel airgapped build test

Usually bash sub-scripts inherit these flags from their parent shell, but that doesn't seem to be happening when run via ip and sudo.
We're seeing cases where the cquery fails due to OOM and generates no tests, but then the build succeeds because it only has to build the hard-coced mask_rom target. This success was used as the exit code for the whole script.
Adding -e and -x flags ensures the sub-script uses the same flags as the parent script.

FYI, I just checked the seven most recent passing CI jobs and every single one of them presented this false positive due to bazel cquery getting OOM killed.
I suspect every airgapped build has been failing like this.
When working on Bazel changes I've noticed that in my CI runs the airgapped build times out at two hours. I thought this was something I'd broken, but I think cquery is somehow being "fixed" and the job just takes too long when run properly.
If that's the case, we may need to limit the airgapped test to make it pass.

I've updated this PR to make the test only build mask_rom since the cquery is not working. This is no change in coverage to our current state, but I don't know how useful it is.
If we want more coverage I think building something like a sival test would be useful since it also has to build opentitanlib with all its Rust dependencies and also splice a bitstream.

I'm going to merge this to unblock other work since it's strictly not worse than our current situation. We can add more to this test in the future.


[Suggestion] Possible 'Gold Dragon-ify' function?

Since Gold Dragon (seemingly) has pre-ML8 progression, would it be possible to add Gold Dragon swap support, maintaining the base path's spellbook if feasible?

I toyed with the idea of making GD-ify option that stretches GD progression from MR8..10 to MR3..10 (with abilities scaling form MR, similar to Legendarify)
But don't know if I ever get to it.


./bin/satis keeps asking for credentials

Hello,
i've just installed newest Satisfy version and changed the app/config/parameters.yml to include the Token of my satis user from Gitlab. But as soon as I run ./bin/satis build the CLI is asking me to provide credentials for this user.
Maybe I'm on a wrong path but if an access token is not working this way, how can I provide it? The tool is asking for credentials for every repository I've added to the satis.json file.
So I would prefer providing the credentials through ENV or even better providing a username and SSH key to access Gitlab. Maybe you can point me to the right direction here.

You must configure composer auth not satisfy.
Not enough info/how to reproduce.

@ramunasd thank you for the hint. Placing an auth.json with the user credentials of the private repository (Gitlab) and using the command like this works:
GIT_ASKPASS=echo ./bin/satis build


Connecting to existing proxy host

I already got an Nginx proxy setup running with letsencrypt and everything.
I thought this would be a neat way to administrate those domains and such.
But there is no way I can reach the current setup through this.
Maybe I am doing it wrong and I should kill the current setup and do everything through this instead?

Can you provide more details about what is not working?

Closing this issue.  Please re-open if needed.


Importing timestamps causes unexpected results with local time zone.

For bugs, please provide the following:
Expected behavior
Importing the date 2018-01-01T00:00 with timeQuantum = "YMD" will result in data in views:

2018
201801
20180101

Actual behavior
Because of my local time zone setting (-0600 CST), importing the date 2018-01-01T00:00 with timeQuantum = "YMD" results in data in views:

2017
201712
20171231

Steps to reproduce the behavior
curl -X POST http://localhost:10101/index/i -d '{"options": {}}'
curl -X POST http://localhost:10101/index/i/field/f -d '{"options": {"type": "time", "timeQuantum": "YMD"}}'

pilosa import -i i -f f ~data.csv

where data.csv contains:
1,101,2018-01-01T00:00

Possible Solution
I've determined that the timezone offset is applied in api.Import with this line:
t := time.Unix(0, ts)

In that case, ts is the number of nanoseconds past epoch.
It seems like we should modify this so that the timezone of the local machine is not considered in the conversion of nanoseconds to a time.Time object. Any objections to my changing this to ignore timezone (i.e. use UTC)?
Information about your environment (OS/architecture, CPU, RAM, cluster/solo, configuration, etc.)
MacOSX, Time Zone (-0600 CST)

My read on this is (and I don't think this disagrees with yours, just writing it out):
The local time zone setting doesn't affect the actual instant in time which is returned by the time.Unix call, it changes the Location which is included in the returned time.Time object. Pilosa then presumably uses year, month, day, hour, etc. which are offset by the included Location. So, if we make sure that any timestamp which comes into Pilosa is represented as UTC internally, we won't run into issues with this.
I guess we need to make sure that any other place that timestamps can come in (such as PQL), they are also treated the same way. It looks like we don't support syntax for specifying time zone or location info in PQL or in the csv importer, so treating all times as UTC would seem to be the sane thing to do.

Yes, I think that's correct. Pilosa does not consider time zones at all, so using a function like time.Unix() which takes location into consideration is problematic. I pushed #1651 to address this instance.


What columns does group pass through?

What's up?
Part of the reason I was confused in #2947 is that it appears that the group transform doesn't retain its key_columns (or at least, I don't know how to retain them).
The example below joins a bunch of tables, with the aim of summarizing the names of the tracks for each city (and country).
Here's the output of that query (as-is) from the Playground:

name
ct1

For Those About To Rock (We Salute You)
1

Balls to the Wall
2

Fast As a Shark
1

Restless and Wild
1

Note that billing_city and billing_country do not appear to be in the table passed to the second group transform
And un-commenting either sort or take 1 leaves only the ct1 column.
Is this intended behavior? Thanks.
from inv=invoices
join item=invoice_items (==invoice_id)
join t=tracks (item.track_id==t.track_id)

group {inv.billing_city, inv.billing_country} (

  group {t.name} (
    aggregate {
    ct1 = count t.name,
    }
  #sort {-ct1}
  #take 1
  )
)

Without the nested group, it does retain the columns:
from inv=invoices
join item=invoice_items (==invoice_id)
join t=tracks (item.track_id==t.track_id)

group {inv.billing_city, inv.billing_country, t.name} (
  aggregate (c=count this)
)
sort (-c)
take 10

SELECT
  inv.billing_city,
  inv.billing_country,
  t.name,
  COUNT(*) AS c
FROM
  invoices AS inv
  JOIN invoice_items AS item ON inv.invoice_id = item.invoice_id
  JOIN tracks AS t ON item.track_id = t.track_id
GROUP BY
  inv.billing_city,
  inv.billing_country,
  t.name
ORDER BY
  c DESC
LIMIT
  10

-- Generated by PRQL compiler version:0.9.4 (https://prql-lang.org)

[
  {"billing_city": "London", "billing_country": "United Kingdom", "name": "Firmamento", "c": 2},
  {"billing_city": "Winnipeg", "billing_country": "Canada", "name": "Onde Você Mora?", "c": 2},
  {"billing_city": "Berlin", "billing_country": "Germany", "name": "The Number Of The Beast", "c": 2},
  {"billing_city": "Stockholm", "billing_country": "Sweden", "name": "Polly", "c": 2},
  {"billing_city": "São Paulo", "billing_country": "Brazil", "name": "Untitled", "c": 2},
  {"billing_city": "Helsinki", "billing_country": "Finland", "name": "Eruption", "c": 2},
  {"billing_city": "Berlin", "billing_country": "Germany", "name": "Eruption", "c": 2},
  {"billing_city": "Rome", "billing_country": "Italy", "name": "For Those About To Rock (We Salute You)", "c": 1},
  {"billing_city": "Yellowknife", "billing_country": "Canada", "name": "Balls to the Wall", "c": 1},
  {"billing_city": "Brasília", "billing_country": "Brazil", "name": "Fast As a Shark", "c": 1}
]

What is the role of the nested group?
I'd need to think more about how we should be handling nested groups — I'm not claiming it's handled well — but is it needed for this specific query?

The query above (without the nested group) doesn't really answer my question, which is: What's the most popular song for each city in the database? The result of that query does list "Firmamento" twice for London (which is the expected answer). But further down, it shows rows for London with all the other titles.
My desired result would have a single row for each of the cities, showing its most "popular" (frequently-purchased) song title.

What is the role of the nested group?

Maybe I don't understand the semantics of group... Coming to SQL from a procedural programming perspective, I had expected that group { billing_city, billing_country } would "collect up" all the rows that had the same city/country and then "apply" the inner pipeline - whatever is between the ( ... ) - to those rows.
That's why I expected the inner pipeline to be along the lines of:
# from all the rows that have the same city/country
group {t.name} (                  # group the track names together
  aggregate { c = count t.name }  # count 'em
  sort {-c}                       # find the largest count
  take 1                          # and pass along the first row
)

My questions:

Do I have the correct mental model for group?
Should I expect that the table passed into the nested group (from the outer group { city, country })would contain those columns?
Should I expect that the rows passed out of the nested group would contain [ city, country, name, c ] as well as any other columns passed in from the outer group?

Thanks

That makes sense, thanks for clarifying.
I think this should do what you need (simplified for country only), but unfortunately falls prey to https://github.com/PRQL/prql/issues/3111.
from inv=invoices
join item=invoice_items (==invoice_id)
join t=tracks (item.track_id==t.track_id)

group {inv.billing_country, t.name} (
  aggregate (c=count t.name)
)

group {billing_country} (
  aggregate (
    sort (-c)
    take 1
  )
)

So in this case, I don't think we need a nested group, I think we can just have them be serial. Does that make sense to you?
(I realize it's more difficult to discuss when there's a bug that prevents the query from running! Hopefully we'll solve that one soon...)

Thanks, that helps. It sounds as if my intuition is correct: group is designed to collect lines with identical key_columns, and run them through the inner pipeline. And the inner pipeline I proposed above would do what I want, except that a bug (#3111) causes this not to work presently.
I also agree that the two serial groups will do what I want. Is this a general rule - that you can reformulate a query different ways (as with Perl, "There’s more than one way to do it") or is this example just a convenient special case? (I don't understand enough about relational algebra to say...)
New question: I don't see anyplace where it's documented what columns are passed to the inner pipeline, or the columns that will be passed out. More questions, and maybe answering myself...

If the inner pipeline of a group does not contain aggregate, all columns are passed through. The inner pipeline could conceivably add or remove columns (with derive or select) but those statements could be placed outside the group with substantially the same effect.

If the inner pipeline does contain aggregate, there's an "implicit select" of the key_columns and any columns created by aggregate - no other columns are passed through. That's because the aggregate changes the "shape" of the table, by reducing multiple rows to a single row.  It's not meaningful to talk about values from an individual row anymore - the data from multiple rows has been summarized - so it cannot be passed through un-summarized.

Finally, one last note: this query gives the results (one song per city) I expected. (The query above doesn't need the second aggregate) Thanks as always.
# What's the most popular song in each CITY
from inv=invoices
join item=invoice_items (==invoice_id)
join t=tracks (item.track_id==t.track_id)

group {inv.billing_city, t.name} (
  aggregate (c=count t.name)
)

group {inv.billing_city} (
    sort (-c)
    take 1
)
sort inv.billing_city

As I see it, behavior of group is:
• split the relation into partition using the by argument,
• apply the pipeline function argument to each of the partitions,
• prefix the partitions with the by argument,
• concatenate partitions back together into a single relation.

@aljazerzen I like this description, but I think the third step is more complex: As I wrote above:

If the inner pipeline of a group does not contain aggregate, all columns are passed through. The inner pipeline could conceivably add or remove columns (with derive or select) but those statements could be placed outside the group with substantially the same effect.

If the inner pipeline does contain aggregate, there's an "implicit select" of the key_columns and any columns created by aggregate - no other columns are passed through. That's because the aggregate changes the "shape" of the table, by reducing multiple rows to a single row.  It's not meaningful to talk about values from an individual row anymore - the data from multiple rows has been summarized - so it cannot be passed through un-summarized.

Other observations:

if the group's pipeline merely selects a couple columns, the query does not actually split the relation into partitions - it simply passes the rows through, possibly in the original order. In addition, the by arguments aren't always passed through. For example:

from invoices
group { billing_city, billing_country } (
  select {
    billing_city,
    billing_address, # NOT billing_country
    invoice_id,
    total
  }
)

Simply sorting the inner pipeline does not seem to change the output

from invoices
group { billing_city, billing_country } (
  select {
    billing_city,
    billing_address, # NOT billing_country
    invoice_id,
    total
  }
  sort billing_city
)

However, adding take 2 gives the expected partition and sorting

from invoices
group { billing_city, billing_country } (
  select {
    billing_city,
    billing_address, # NOT billing_country
    invoice_id,
    total
  }
  sort billing_city
  take 2
)

Is this intended? Or is it still related to #3111? Thanks

Summary:

the unexpected behavior can be discussed in the linked issue,
we the following open question:

Should we change the behavior of group to "always prefix the result with the by columns"?
It would mean that in some cases, columns would get duplicated:
from x
select {a, b, c}
group a (take 2)

[
  {a = 4, a = 4, b = ..., c = ...},
  {a = 4, a = 4, b = ..., c = ...},
  {a = 9, a = 9, b = ..., c = ...},
  {a = 1, a = 1, b = ..., c = ...},
  {a = 1, a = 1, b = ..., c = ...},
  {a = 1, a = 1, b = ..., c = ...},
]

A fix for this problem would be to remove the columns from the partitions, before the function is applied to partitions.
from x
select {a, b, c}
group a (
  # here, only columns {b, c} are available

  take 2 # this will produce partitions of form [{b = ..., c = ...}]
)
# concatenated back together, result will have form
# [
#   {a = 1, b = ..., c = ...},
#   {a = 1, b = ..., c = ...},
#   {a = 2, b = ..., c = ...},
# ]

As I see it, behavior of group is:
split the relation into partition using the by argument,
apply the pipeline function argument to each of the partitions,
prefix the partitions with the by argument,
concatenate partitions back together into a single relation.

I think this description is spot on and this behaviour should be retained/strived for in all cases where possible to retain orthogonality.

Should we change the behavior of group to "always prefix the result with the by columns" (not just when the pipeline ends with aggregate)?

Yes, but only logically. The implementation to achieve may need to vary to accommodate this (IDK)?

It would mean that in some cases, columns would get duplicated:
from x
select {a, b, c}
group a (take 2)

No, the current behaviour (at least in the cases I've seen) does have this property.
Take this example:
# 3 albums with the longest track
from tracks
group {album_id} (
  sort {-milliseconds}
  take 1
)
sort {-milliseconds}
take 3

By virtue of keeping all columns in the resultset you are including the by columns and you have still partitioned by them.
I see no good reason to change that at this point.

You are right - currently group changes behavior depending on if it contains aggregate or not. Which is against our orthogonality principle.

My first intuition here is that the rule should probably not be around whether there is an aggregate in the group pipeline but whether it contains a transform with a column restriction like aggregate or select.

if there is a column restriction anywhere in the pipeline then prepend the by columns. Repetition of any by columns would then be intentional.
if there is no column restriction then don't prepend the by columns.

Haven't thought this through thoroughly but it seems to work on the examples I can think of off the top of my head.

@snth Note to self: I want to add something on nested groups but out of time rn.

if there is a column restriction anywhere in the pipeline then ...

This also makes the behavior of group depend on the function that is applied to the partitions.

if there is a column restriction anywhere in the pipeline then ...

This also makes the behavior of group depend on the function that is applied to the partitions.

I'm suggesting this as an implementation detail in the generated SQL in order to keep the logical behaviour of PRQL consistent.
I agree that the PRQL behaviour should be independent of the internals of the group pipeline.

So in this case, I don't think we need a nested group, I think we can just have them be serial. Does that make sense to you?

My gut feeling around this is that this should hold in general and could be an implementation strategy for the compiler. In particular I believe that:
group {a_1, ..., a_i} (
  transform_1
  group {b_1, ..., b_j} (
    transform_2
  )
  transform_3
)

should generally be equivalent to:
group {a_1, ..., a_i} (
  transform_1
)
group {a_1, ..., a_i, b_1, ..., b_j} (
    transform_2
)
group {a_1, ..., a_i} (
  transform_3
)

The only complication I can see is around preservation of ordering if there are sort transforms in transform_1 and transform_2.
Can we prove this or can someone think of a counterexample?
My feeling is that since SQL doesn't have nested GROUP BYs AFAIK, this approach of flattening nested groups would allow the compiler to transform such PRQL code into forms that can be easily compiled to SQL.

Centering on the question of orthogonality above (v good discussion): to summarize — the issue is that "group should add back its columns" makes a lot of sense in:
from artists
select {artist, city, visit_date}
group {city, artist} (
  aggregate (visit_count = count visit_date)
}

...because we want {city, artist, visit_count}. But does not seem correct in:
from artists
select {artist, city, visit_date}
group {city, artist} (
  sort visit_date
  aggregate (take 1)
)

...because we want {city, artist, visit_date} again, not ``{city, artist, city, artist, visit_date}`.

I really like @aljazerzen 's suggestion of removing the columns before passing in the group — simple semantics that works for both cases.

Although this is problematic from another standpoint: group's by argument might be passed columns that are not in the original relation:
group {a + 1} (take 2)[^1] [^2]

And sometimes you might want to reference the column:
from artists
select {artist, city, visit_date}
group {city, artist} (
  sort visit_date
  aggregate (visit_count = count visit_date)
  select message = f"{artist} visited {city} {visit_count} times"
)

So even "remove any columns that are not in the relation" might not be sufficient.
At risk of complicating this — one rule that satisfies all of these examples is:
Remove by columns from the relation within the group from * if they are in the input relation, but let them be accessed by name
I'm not sure how easy it to be completely consistent in SQL though, given we don't have access to the column names. We can make it work for 90% of the cases, but sometimes are we just not going to know whether they're there?
[^1]: One way of thinking of this is that group (foo + 1) desugars to derive _unnamed = foo + 1 | group (_unnamed). Or disallowing expressions there.
[^2]: Separately, we could allow assigns as arguments to functions, so someone can pass group (foo = bar + 1)

And sometimes you might want to reference the column:

That's true, but I think all of the cases where you'd want to reference the by columns from within the group partition function, could be transformed by moving the reference out the function:
from artists
select {artist, city, visit_date}
group {city, artist} (
  sort visit_date
  aggregate (visit_count = count visit_date)
)
select message = f"{artist} visited {city} {visit_count} times"

Remove by columns from the relation within the group from * if they are in the input relation, but let them be accessed by name

This would solve the problem, but we don't do that anywhere else - it would be quite unprecedented. It would make group partition function dependent on the outer context.

That's true, but I think all of the cases where you'd want to reference the by columns from within the group partition function, could be transformed by moving the reference out the function:

Yes true, good point.
So does that leave:

Remove by columns from the relation within the group, if they are in the input relation (i.e. not if there's an expression such as group (foo + 1))
Remove by columns from the relation within the group + disallow expressions in by

Remove by columns from the relation within the group from * if they are in the input relation, but let them be accessed by name

This would solve the problem, but we don't do that anywhere else - it would be quite unprecedented. It would make group partition function dependent on the outer context.

Yes, it would be unprecedented. To confirm the original proposal, it doesn't rely on any context from outside the group transform — "if they are in the input relation" is true when there's a simple ident in the by, and false when there's an expression group (foo + 1).

Remove by columns from the relation within the group from * if they are in the input relation, but let them be accessed by name

Now it's my turn to confess that I haven't completely understood the foregoing discussion. Let me express my (simple-minded) mental model:

The group transform partitions the table according to its by columns (I think that's the name we're using...)

group passes each partition (with all columns, including the by columns) to the inner pipeline

The inner pipeline does its business: changing/selecting rows (select, derive, take, filter, etc) OR summarizing the rows (with aggregate). In either case, the inner pipeline produces a result table.

group ensures that the by columns are present in the result table, adding them back if they weren't present in the result table, which gets passed to the next transform.

Is this where the design is heading? If so, it will be easy to explain to newcomers. Many thanks.

@richb-hanover This is a good way to describe how group works at the moment. The only difference between current behavior and what you wrote is the last point: "group ensures that the by columns are present in the result table". This would be "if the last function in the pipeline is aggregate, by columns are added to result table[^1]".
We are now in the process of discussing a change to this behavior, with the primary goal of making it more orthogonal/independent of the inner function.
[^1]: what you call "table" we call "relation"

@aljazerzen , @max-sixty , @snth

We are now in the process of discussing a change to this behavior, with the primary goal of making it more orthogonal/independent of the inner function.

I had trouble following all the nuances of the discussion above. Have we settled on a design decision? Would you summarize how it will work going forward? Many thanks.

In the last comment, max outlined two options that we have.
I don't have a strong opinion on any of the two, so we could leave it up to what's easier to implement.

While working on our type system, I stumbled into this.
The only solution to express the behavior of group was to update the behavior of group to:

remove the by columns before applying the pipeline to the partition,
always re-add by columns back into the partition.

To do this, I'll have to pick @max-sixty option 2:

Remove by columns from the relation within the group + disallow expressions in by

In practice these are the changes:

by columns are not available within the pipeline function:

    group {a} (derive c = a + 1)
       ╭─[:4:27]
       │     from x
       |     select {a, b}
     4 │     group {a} (derive c = a + 1)
       │                           ┬
       │                           ╰── Unknown name `a`
    ───╯

by columns cannot be expressions

    from x
    select {a, b}
    group {a + 1} (aggregate {sum b})
           ^^^^^
               \__ function `group`, param `by` expected plain identifiers, found `a + 1`

by columns are moved in front of the output relation

from x
select {a, b}
group {b} (take 1)

# will have the type of
[{b, a}]

Thanks @aljazerzen
And to confirm for future travelers — this doesn't represent any loss functionality — it's always possible to add a column before the group. So while this will raise an error:
from x
select {a, b}
group {a + 1} (aggregate {sum b})

...we can just break that out:
from x
select {a, b}
derive foo = a + 1
group {foo} (aggregate {sum b})

In the future possibly we could allow an assignment in there:
from x
select {a, b}
group {foo = a + 1} (aggregate {sum b})

@aljazerzen @max-sixty I still have questions about the query from the original item of this issue.
Where is the best place to continue the discussion? (At this point, it's not a bug report - perhaps in the Discussions?) Many thanks.

Discussions is fine or adding here is OK!

Since this is closed, moved discussion to #3832


Allow Qt to wrap long tooltips

Allow Qt to word-wrap tooltips. Qt can only word-wrap rich text messages.
Implemented without having to touch any translation: by listening for QEvent::ToolTipChange events, then rewriting the tooltips to prefix  and convert to rich text if it is not yet rich text.
No wrap: https://dl.dropboxusercontent.com/u/37902134/Paycoin Core/No wrap.png
With wrap: https://dl.dropboxusercontent.com/u/37902134/Paycoin Core/With wrap.png

Looks good to me (tested on my linux box).


Register in General

@JuliaRegistrator register

[3b96b61c] + ACEpotentials v0.8.0-dev ~/work/ACEpotentials.jl/ACEpotentials.jl

@JuliaRegistrator register

@JuliaRegistrator register

Registration pull request created: JuliaRegistries/General/113750
Tip: Release Notes
Did you know you can add release notes too? Just add markdown formatted text underneath the comment after the text
"Release notes:" and it will be added to the registry PR, and if TagBot is installed it will also be added to the
release that TagBot creates. i.e.
@JuliaRegistrator register

Release notes:

## Breaking changes

- blah

To add them here just re-invoke and the PR will be updated.
Tagging
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.0 -m "<description of version>" a3b29608aedf84ceeb26d2480aa19332d5f644c1
git push origin v0.1.0

@JuliaRegistrator register

Registration pull request created: JuliaRegistries/General/113983
Tip: Release Notes
Did you know you can add release notes too? Just add markdown formatted text underneath the comment after the text
"Release notes:" and it will be added to the registry PR, and if TagBot is installed it will also be added to the
release that TagBot creates. i.e.
@JuliaRegistrator register

Release notes:

## Breaking changes

- blah

To add them here just re-invoke and the PR will be updated.
Tagging
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.1 -m "<description of version>" f00fbec4af449f981b8b1eeba8ae089c78b5ab5f
git push origin v0.1.1

Registration pull request created: JuliaRegistries/General/114451
Tip: Release Notes
Did you know you can add release notes too? Just add markdown formatted text underneath the comment after the text
"Release notes:" and it will be added to the registry PR, and if TagBot is installed it will also be added to the
release that TagBot creates. i.e.
@JuliaRegistrator register

Release notes:

## Breaking changes

- blah

To add them here just re-invoke and the PR will be updated.
Tagging
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.2 -m "<description of version>" dd390cff135ee000fc3be6ec6131af8664e44bc2
git push origin v0.1.2

@JuliaRegistrator register

Registration pull request created: JuliaRegistries/General/115166
Tip: Release Notes
Did you know you can add release notes too? Just add markdown formatted text underneath the comment after the text
"Release notes:" and it will be added to the registry PR, and if TagBot is installed it will also be added to the
release that TagBot creates. i.e.
@JuliaRegistrator register

Release notes:

## Breaking changes

- blah

To add them here just re-invoke and the PR will be updated.
Tagging
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.3 -m "<description of version>" d56e6b1e5327dc2cba62bf9a4194ef84f8357e78
git push origin v0.1.3

@JuliaRegistrator register

Registration pull request created: JuliaRegistries/General/115465
Tip: Release Notes
Did you know you can add release notes too? Just add markdown formatted text underneath the comment after the text
"Release notes:" and it will be added to the registry PR, and if TagBot is installed it will also be added to the
release that TagBot creates. i.e.
@JuliaRegistrator register

Release notes:

## Breaking changes

- blah

To add them here just re-invoke and the PR will be updated.
Tagging
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.4 -m "<description of version>" 4d7fdc4fc0603ee4a9d838a7a6a27f59bb9f9c0f
git push origin v0.1.4

@JuliaRegistrator register

@JuliaRegistrator register

Registration pull request created: JuliaRegistries/General/121737
Tip: Release Notes
Did you know you can add release notes too? Just add markdown formatted text underneath the comment after the text
"Release notes:" and it will be added to the registry PR, and if TagBot is installed it will also be added to the
release that TagBot creates. i.e.
@JuliaRegistrator register

Release notes:

## Breaking changes

- blah

To add them here just re-invoke and the PR will be updated.
Tagging
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.6 -m "<description of version>" ce4a8d2e1e33f39086c9e972ed14a840698ecd9b
git push origin v0.1.6

@JuliaRegistrator register

Registration pull request created: JuliaRegistries/General/121740
Tip: Release Notes
Did you know you can add release notes too? Just add markdown formatted text underneath the comment after the text
"Release notes:" and it will be added to the registry PR, and if TagBot is installed it will also be added to the
release that TagBot creates. i.e.
@JuliaRegistrator register

Release notes:

## Breaking changes

- blah

To add them here just re-invoke and the PR will be updated.
Tagging
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.7 -m "<description of version>" 2d0a3bb04394ae668ad1df58d1b2b0162ddb145c
git push origin v0.1.7


decomp: Re-attempting previously blocked state decomp files

Finished:

[x] snow-ram
[x] snow-bumper
[x] spider-egg
[x] puffer
[x] driller-lurker
[x] dark-crystal
[x] gnawer
[x] mother-spider-egg
[x] swamp-blimp
[x] citb-plat
[x] darkvine
[x] launcherdoor
[x] quicksandlurker
[x] voicebox
[x] bully
[x] seagull
[x] jungle-mirrors

Found better reasons for why other files aren't able to be completed yet (they are 99% decompiled).

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.004%) to 54.644%

Totals

Change from base Build<PHONE_NUMBER>:
0.004%

Covered Lines:
40510

Relevant Lines:
74135

💛  - Coveralls

ok, just one last thing left - the stack structure thing in jungle-mirrors. I think if you remove the stack structure vector at 16 it will owrk


flutter.dev/desktop updates for Linux alpha

Update flutter.dev/desktop to update the docs to be the same for both Linux + macOS, i.e. that they're both in alpha, leaving Windows in tech preview.

This should include any distribution tech we'd like to link to, e.g. how to build a Snap (if we know how to do that), etc.

This is done.


Resize image?

Would a PR to resize images be accepted here?  Or is that beyond the scope of this module?
Edit: I should add that I have no idea how to do it.  But I may be motivated enough to figure it out. :)

https://github.com/SolitudeSF/imageman/blob/master/src/imageman/resize.nim I think this file can be ported to nimPNG

no need to port them, you can use nimPNG and imageman together. You only need to convert PNGResult to imageman Image with RGBAU colorspace. And with some pointer tricks, you actually can have almost free conversion.
example:
type
   MyImage = ref Image[ColorRGBAU]

let png = loadPNG32("pngfile.png")
let img = cast[MyImage](png)

let res = resizeBilinear(img[], target_w, target_h)

that's cool , there's problem imageman relys on libpng which is not pure, would it acceptable to make a PR port of resize.nim?

you can safely import imageman's images and colors module and use it with nimPNG, if LTO(link time optimization) works properly. no need to worry libpng will be included in your final binary.
no need to port it.

Thanks! helpfull , that type casting is magic. solve my problem.

I have another problem , how to construct PNG(or encode png bytes) from MyImage I need pass it to my optimizePNG proc or save it somewhere directly, use imageman when writePng it will need libpng.

I assume you get the image after resizing then it would be something like this:
type
   MyImage = ref Image[ColorRGBAU]
   TImage = ptr object
      width: int
      height: int
      pixels: seq[uint8]

let png = loadPNG32("pngfile.png")
let img = cast[MyImage](png)

var res = resizeBilinear(img[], target_w, target_h)
let timg = cast[TImage](unsafeAddr(res))
var r = savePNG32("filename.png", timg.pixels, timg.width, timg.height)
if r.isOK:
   # do something
else:
  # do something

I got unhandled exception: not enough input to encode [PNGError] whether encodePNG  or savePNG32, I suppose to store it in memory and pass to next proc,
yeah, that tips helpful, seq[ColorRGBAU] cast to string surprised me , since ColorRGBAU is 4 components, I check the source it's array so it's just bytes

the pixels element length should be w x h x 4 for RGBAU and w x h x 3 for RGBU.

yeah, casting can't adjust that seq size , that cause this , I'll retry later . use copymem or something else.


[FYI] Vim Credits -From jun 13 12:29:53 2004 -Till jan 4 21:57:11 2021

Till v8.2.2301
// just a rough list, since no fixed/fomral fmt can be sure the names..
// for detail pls check git commits.

Bram started to set the commit author from the end of April.
So, now we can see the contributors on the Contributors page from that point on.
To see the old contributors, we still need to check the help pages, but that should be acceptable.
I close this.

in case github would be broken someday (Just a supposed) i guess Bram would/should continue to mark contributors names in git commits msg as well (or for those not from github pr but from maillist). :-)


Add GuestNetwork Connected numbers sensor entity #790

Add guest network connected devices number

I think it's ready. Thanks! :+1:


Update http4s-blaze-client, ... to 0.21.18

Updates

org.http4s:http4s-blaze-client
org.http4s:http4s-blaze-server
org.http4s:http4s-circe
org.http4s:http4s-dsl

from 0.21.3 to 0.21.18.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Applied Migrations

dependency:v0_21@org.http4s:http4s-scalafix:0.21.5

Documentation:

https://github.com/http4s/http4s/releases/tag/v0.21.5

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.http4s" } ]

labels: library-update, semver-patch, scalafix-migrations

Superseded by #253.


invalid stub handshake response received

Hello
i got the following issue:
[2024-03-25T15:55:55Z INFO ] Serial port: 'COM5'
[2024-03-25T15:55:55Z INFO ] Connecting...
[2024-03-25T15:55:57Z INFO ] Using flash stub
Error:   x Error while connecting to device
  `-> Invalid stub handshake response received

i can connect with py -m esptool -p .... to my sparkfun esp32 thing board.
I use this instruction guide to flash the device: https://github.com/beeb/coffee-scale-app/blob/main/rs/README.md
Also i set the baudrate to a lower one (115200) in the espflash.toml without success.
I tried to find with google the issue with the handshake and thats why i set the baudrate to a lower part.
What did i wrong or miss?

Hey @SergioGasquez sorry for the late reply.
Still no luck with espflash 3.1.0. Getting a Error: espflash::timeout when I don't pass the --before no-reset option, and I get a Error: espflash::connection_failed when I pass the --before no-reset option.

Ok I updated cargo-espflash to 3.1.0 and the error is slightly different:
without option:
Error:   × Error while connecting to device
  ╰─▶ Invalid stub handshake response received

with --before no-reset:
Error: espflash::connection_failed

  × Error while connecting to device
  ╰─▶ Failed to connect to the device


Improve DMA ergonomics when working with generics

I found our current DMA API to be quite difficult to work with in external crates because of the A, B thingy. This replaces the buffer types with the ones used in the embeddonomicon https://docs.rust-embedded.org/embedonomicon/dma.html
As  far as I can tell, this is safe, but I may very well be missing something.
Because of the way that Transfer is implemented, I found no way to specify a generic TxDma which has a wait method. To fix that, I added the Transferable trait.
Example usage of all of this can be found at https://gitlab.com/TheZoq2/ws2812-spi-dma/-/blob/master/src/lib.rs
DNM because the same treatment has to be given to RxDma, but I just want to make sure I don't do anything stupid before going through with that.

You might want to wait on the resolution of https://github.com/rust-embedded/embedonomicon/issues/64

Oh, hadn't seen that. I'll have a look at that tomorrow, thanks for the heads up!

I added StableDeref now, hopefully I did that correctly :sweat_smile:

Closing this in favor of #274


parentheses and self-assembly options

This happens in Qgis 3.14.0, Windows 10 version
1 - Parentheses
New layer of shapefile polygons.
Adding a new field in the text attributes table (40 long) automatically adds parentheses in this new column.
2- self-assembly
Shapefile polygon cape
A main polygon (AOI) was drawn at the beginning.
Within this polygon other polygons are digitized. With the self-assembly and topology editing options, the goal is for polygons to properly fit their neighboring polygons and share nodes. There is no problem with snapping, the problem is that when the new polygon is closed, it does not take the shape of neighboring polygons.

Hi @Edwrom
Issues with the software should be reported at https://github.com/qgis/QGIS/issues.
This repository is for issues with QGIS web site. I'm then closing; please report there, with screenshots if possible and as separate issues.
Thanks

thank's


HttpEventCollectorSender should more easily support overriding

We'd like to enable our application to send custom messages to splunk. The code-security model around HttpEventCollectorSender makes this difficult.

Can HttpEventCollectorSender be made public and non-final?
Could HttpEventCollectorSender.serializeEventInfo be made protected?
Can HttpEventCollectorLogbackAppender support injection of a sender?

Alternatively, please allow for injecting a function for serializeEventInfo into the configuration of the appender

Hi @frkline,
Thanks the suggestions, these all seem like reasonable changes.
We're happy to bring in a pull request to the develop branch

Above PR#134 resolves this issue.
Hence closing this issue.


Python encountered SyntaxError('invalid syntax', ('/opt/Autodesk/shared/python/FlamePy/sg_batchgroup_from_plates.py', 19, 8, 'print ""\n')) in unknown code

Hi there !
Getting some errors after 472 frames calculated :
flame202231_flame6_app.log
flame202231_flame6_shell.log
Can you point us in the right direction ?
Many thanks for your help :)

the error seems related to the hook mentioned (/opt/Autodesk/shared/python/FlamePy/sg_batchgroup_from_plates.py)
the print statement is for python2, you probably need to add () ===>  print(""\n)


Update Windows CI

Fixes #15

@kyouko-taiga

This PR now have long code in #os (Windows), but there seems to be no other solution
At most, just delete some lib that doesn't work for us

Oh my, this is nuts. But I guess it can't be avoided.


Update docker-hub-image-readme.md

Remove reference to Ruby 2.6.3, resync markdown link anchors with upstream artichoke/artichoke README.

this update has been pushed to Docker Hub


Update downloader to use v5 root layout

What does this PR do?

Motivation

Additional Notes

Review checklist (to be filled by reviewers)

[ ] Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
[ ] PR title must be written as a CHANGELOG entry (see why)
[ ] Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
[ ] PR must have changelog/ and integration/ labels attached
[ ] If the PR doesn't need to be tested during QA, please add a qa/skip-qa label.

The failing jobs are skipped in this PR: https://github.com/DataDog/integrations-core/pull/13195 and a Jira ticket has been made to investigate them.


⚠ Bofrost.at has degraded performance

In e852394, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 3115 ms

Resolved: Bofrost.at performance has improved in df83932.


Strip subresource hints pagespeed automatic system test configuration

Corresponding changes for:

pull https://github.com/pagespeed/mod_pagespeed/pull/1204
issue Strip subresource hints #973 https://github.com/pagespeed/mod_pagespeed/issues/973

no comments, lgtm, @jeffkaufman ?

Maybe hold this one off until we have finished 973, I think we need some additional tests if we add the detection of url changing filters.

Merged as 56cd5e3b6a696ba19d6ddf2cebf7acaffd72d0ad


链码安装报错

感谢大神的文档和代码贡献！
链码安装的时候报如下错误：
Sending proposal to peer0 failed because of: gRPC failure=Status{code=UNKNOWN, description=chaincode error (status: 500, message: Authorization for INSTALL has been denied (error-Failed verifying that proposal's creator satisfies local MSP principal during channelless check policy with policy [Admins]: [This identity is not an admin])), cause=null}
请问这个要怎么解决？

这个问题解决了，在链码通过你写的工具进行安装也成功了，但是在示例化的时候报如下错误：
Error: Error endorsing chaincode: rpc error: code = Unknown desc = error starting container: Failed to generate platform-specific docker build: Error returned from build: 1 "can't load package: package sharedLedger: cannot find package "sharedLedger（这个是我上传的时候zip文件名）" in any of:
/opt/go/src/sharedLedger (from $GOROOT)
/chaincode/input/src/sharedLedger (from $GOPATH)
/opt/gopath/src/sharedLedger
大神能否指导一下

感谢大神的文档和代码贡献！
链码安装的时候报如下错误：
Sending proposal to peer0 failed because of: gRPC failure=Status{code=UNKNOWN, description=chaincode error (status: 500, message: Authorization for INSTALL has been denied (error-Failed verifying that proposal's creator satisfies local MSP principal during channelless check policy with policy [Admins]: [This identity is not an admin])), cause=null}
请问这个要怎么解决？

你好，请问这个怎么解决的？

@tanqiyou1990 请问 This identity is not an admin 这个问题您怎么解决的？感谢


ModuleNotFoundError: No module named 'diffusers.loaders'

(animatediff) u27863@AI-02-09:/ai/u27863/AnimateDiff$ python -m scripts.animate --config configs/prompts/v2/5-RealisticVision.yaml
Traceback (most recent call last):
File "/home/u27863/.conda/envs/animatediff/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/u27863/.conda/envs/animatediff/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/ai/u27863/AnimateDiff/scripts/animate.py", line 15, in
from animatediff.models.unet import UNet3DConditionModel
File "/ai/u27863/AnimateDiff/animatediff/models/unet.py", line 26, in
from diffusers.loaders import UNet2DConditionLoadersMixin, AttnProcsLayers
ModuleNotFoundError: No module named 'diffusers.loaders'
diffusers had been installed
(animatediff) u27863@AI-02-09:/ai/u27863/AnimateDiff$ pip list |grep diffuser
diffusers                 0.11.1

I have this issue too, any luck?


Apps disappear

Steps to reproduce
Install two apps in WSA.
✔ Expected Behavior
It should not disappear.
❌ Actual Behavior
It disappeared.
Other Software
No response
Please specify the version of Windows Subsystem for Android
2311.40000.5.0
A picture

Fixed, after restarting Windows 11.


[network] add connection bench

Motivation
Add a bench that concurrently dial the socket-bench-server and reports the connection throughput.
I got a bunch of ERROR 2020-05-09 02:15:13 network/benches/socket_bench.rs:335 Failed to upgrade Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" } during the bench locally
Have you read the Contributing Guidelines on pull requests?
Y
Test Plan
connection_throughput/noise_connections/8
time:   [7.3759 ms 21.138 ms 31.734 ms]
thrpt:  [252.10  elem/s 378.46  elem/s 1.0846 Kelem/s]
change:
time:   [-99.660% -98.887% -92.724%] (p = 0.13 > 0.05)
thrpt:  [+1274.5% +8882.4% +29290%]
No change in performance detected.
connection_throughput/noise_connections/16
time:   [22.851 ms 35.741 ms 53.560 ms]
thrpt:  [298.73  elem/s 447.67  elem/s 700.19  elem/s]
change:
time:   [-79.029% -36.994% +285.42%] (p = 0.97 > 0.05)
thrpt:  [-74.055% +58.716% +376.85%]
No change in performance detected.
connection_throughput/noise_connections/32
time:   [43.750 ms 93.395 ms 204.73 ms]
thrpt:  [156.30  elem/s 342.63  elem/s 731.42  elem/s]
change:
time:   [-71.189% -22.611% +144.78%] (p = 0.87 > 0.05)
thrpt:  [-59.147% +29.218% +247.09%]
No change in performance detected.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high severe
connection_throughput/noise_connections/64
time:   [96.859 ms 167.15 ms 251.88 ms]
thrpt:  [254.09  elem/s 382.88  elem/s 660.76  elem/s]
change:
time:   [-62.148% -31.225% +32.208%] (p = 0.38 > 0.05)
thrpt:  [-24.362% +45.401% +164.19%]
No change in performance detected.
Related PRs
(If this PR adds or changes functionality, please take some time to update the docs at https://github.com/libra/website, and link to your PR here.)

@bors-libra r=davidiw

:pushpin: Commit 9951d77 has been approved by davidiw

:hourglass: Testing commit 9951d7716526b0268de3f13da8d4aa9dd1256715 with merge 9e80470cdac7c14a20ca28145788a709cbcb8d62...

:sunny: Test successful - checks-actions_land_blocking_test, checks-circle_commit_workflow
Approved by: davidiw
Pushing 9e80470cdac7c14a20ca28145788a709cbcb8d62 to master...


Tooltip with link inside treats whole tooltip as external link.

Describe the bug
In Ch.1 Challenge 1, pt1, there are two tool tips. When you hover over one, it shows a hand. When you over over another one, it shows the mouse cursor. Is this because there is a url embedded in the first tooltip?
Checking figma I don't see a recommended option between mouse cursor vs hand when hovering over a tool tip.
https://user-images.githubusercontent.com/32662508/222543536-e9947565-4dc7-45bd-8922-abda2e136943.mov

I think I know what's going on. If there's an external link inside the tooltip, I think the code is treating the tooltip as an external link as a whole. That's what's causing the mouse to turn into a hand.
This is what's happening when I tap on the tooltip for mobile. It immediately jumps out to a link instead of letting me view the tooltip. I guess really this is the bug.
https://user-images.githubusercontent.com/32662508/222545359-e0dee8b2-fc2a-4f6f-8308-933627075d3e.MP4

@ecurrencyhodler I have a quick fix adding a pointer class to tooltips.
However, with the mobile functionality should I go ahead and make tooltips open on mobile too?

Thanks ben!
What do you mean "make tooltips open on mobile too?"

AFAIK generally tooltips are disabled on mobile

Okay. Right now we have the functionality in mobile. So let's go with just the quick fix.

okay, so the tooltip 'block explorer' IS a link so hovering it on desktop opens the tooltip but clicking it also just goes straight to a link. Do you think we should move the link to the url INSIDE the tooltip?

Yes I think so. Otherwise they'll never see the text inside the tooltip in mobile.

So slight problem, as the tooltip goes away on moving the curson adding a link inside of a tooltip is quite difficult to click on. I think that maybe just having the block explorer tooltip go to the link on mobile is ok with the amount of technical debt it would create

Okay. Let's also raise this on our call tuesday so we're all on the same page. Thanks for investigating it.

In my PR #233 tooltips will no longer become invisible when you move your mouse cursor over them.


HTTPS everywhere

This switches all URLs to HTTPS - All links were verified and archive.org copies were used if the original copy was missing.
PS: I'm not sure about the URLs in the licence text itself, since Apache still uses the HTTP URLs on their homepage, but IANAL...

PS: I'm not sure about the URLs in the licence text itself, since Apache still uses the HTTP URLs on their homepage, but IANAL...

I am also now a lawyer, so changing urls to https in the licence headers just seems unnecessary. I would revert any changes to do with licences or this PR will not get merged.
Changing URLs in documentation is one thing, in licences is another.

Changed to the licence seemed unnecessary. Feedback given on other changed but no reply in well over a month.
Changes have been applied where appropriate outside the PR, closing this PR now.