feat(streaming): Support in memory backfill executor for mv on mv

I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.
What's changed and what's your intention?
PLEASE DO NOT LEAVE THIS EMPTY !!!
Please explain IN DETAIL what the changes are in this PR and why they are needed:

In this PR, we introduce BackfillExecutor based on the RFC: https://github.com/risingwavelabs/rfcs/pull/13
Currently, it is a pure in memory executor without any state persistence logic, but we can add it later for recovery.
Also we only read committed data from the upstream mv in this PR, but we can support uncommitted read later.
In order to keep the code changed as small as possible, we slightly extend the ChainNode and add a chain_type  field to decide which executor will be used to create mv on mv. So at this moment, BackfillExecutor is an implementation of the ChainNode.
BTW,BackfillExecutor has a special read pattern which is it will read the storage through a stream, but likely stop the reading and drop the stream without consuming all the data of the stream. This behavior has never met by storage before, so it is also some kind of testing for the storage.
For more details, please read the comment of the BackfillExecutor.

Checklist

[x] I have written necessary rustdoc comments
[x] I have added necessary unit tests and integration tests
[x] All checks passed in ./risedev check (or alias, ./risedev c)

Documentation
If your pull request contains user-facing changes, please specify the types of the changes, and create a release note. Otherwise, please feel free to remove this section.
Types of user-facing changes
Please keep the types that apply to your changes, and remove those that do not apply.

Installation and deployment
Connector (sources & sinks)
SQL commands, functions, and operators
RisingWave cluster configuration changes
Other (please specify in the release note below)

Release note
Please create a release note for your changes. In the release note, focus on the impact on users, and mention the environment or conditions where the impact may occur.
Refer to a related PR or issue link (optional)
https://github.com/risingwavelabs/risingwave/issues/6275

Will take a look later. 🥵


Add make_confidence_report_spsa.py

@juesato @nottombrown @carlini
I'm trying to clean up SPSA and integrate it into our newest multi-GPU, multi-attack, confidence aware eval system.  I made a few basic changes like having it use the same argument names as ProjectedGradientDescent, support clip_min and clip_max, etc.
Unfortunately, the core SPSA algorithm itself doesn't really seem to work. It results in NaNs.
Steps to reproduce:

Check out this branch
cd cleverhans_tutorials
python mnist_tutorial_picklable.py
cd ../scripts
python make_confidence_report_spsa.py ../cleverhans_tutorials/clean_model.joblib

This results in a NaN. Any suggestions?

I think the NaN may have been because the VM I was using had a GPU hardware or driver problem. After rebooting it, I no longer get the NaN. I do get a different error message though:
File "/home/goodfellow/cleverhans/cleverhans/attacks.py", line 1991, in generate
clip_max=clip_max
File "/home/goodfellow/cleverhans/cleverhans/attacks_tf.py", line 1842, in pgd_attack
back_prop=False)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 3232, in while_loop
return_same_structure)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 2952, in BuildLoop
pred, body, original_loop_vars, loop_vars, shape_invariants)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 2887, in _BuildLoop
body_result = body(*packed_vars_for_body)
File "/home/goodfellow/cleverhans/cleverhans/attacks_tf.py", line 1816, in loop_body
wrapped_loss_fn, [perturbation], optim_state)
File "/home/goodfellow/cleverhans/cleverhans/attacks_tf.py", line 1583, in minimize
grads = self._compute_gradients(loss_fn, x, optim_state)
File "/home/goodfellow/cleverhans/cleverhans/attacks_tf.py", line 1693, in _compute_gradients
x[0] = tf.reshape(x[0], [1] + static_x_shape[1:])
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 6199, in reshape
"Reshape", tensor=tensor, shape=shape, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 454, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3155, in create_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1717, in init
self._traceback = tf_stack.extract_stack()
InvalidArgumentError (see above for traceback): Input to reshape is a tensor with 1003520 values, but the requested shape has 784
[[Node: while_1/Reshape = Reshape[T=DT_FLOAT, Tshape=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:1"](while_1/Identity_1, while_1/Reshape/shape)]]
[[Node: while_3/assert_greater_equal/Assert/AssertGuard/Assert/Switch_1/_349 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:3", send_device_incarnation=1, tensor_name="edge_1182_while_3/assert_greater_equal/Assert/AssertGuard/Assert/Switch_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Is this currently using a batch size of 1280 by any chance?
SPSA assumes a batch size of 1, because internally, the implementation uses batching by evaluating the model on many noised versions of the current input to do the finite difference gradient estimates. Since this already maxes out GPU utilization, I just didn't both implementing the reshaping necessary to support attacking a batch of multiple inputs. This used to error out, but I changed it in https://github.com/tensorflow/cleverhans/pull/509 to assume the Tensor being passed in has batch size 1 and then reshape appropriately.
Harini pointed out to me today that SPSA only supporting batch size 1 isn't actually in the documentation. That's totally my bad, I'll send in a PR tomorrow.

OK, thanks. Do you think it would be possible to support a batched interface by just wrapping what's currently there in a tf.while_loop that makes one adversarial example at a time?

My original reasoning was (copying from https://github.com/tensorflow/cleverhans/pull/509):
"""
The TF implementation still only supports batch size 1. It's implemented this way just for simplicity - when I used it I didn't need more than batch size 1. There's wouldn't really be any gain in efficiency with batching, since SPSA is internally batching the function evaluations of f(x + delta) for all the different delta values at each iteration.
The TF implementation could of course support a tf.while_loop on the outside to loop over the batch dimension, but that seems overly complex to me, since the computation is really handling each image separately, and it's easy to handle outside TF. I think it makes sense for the numpy interface to support batches, so I changed that.
"""
I'm happy to implement this though, if you feel strongly about it.

This is no longer WIP / this is done once the tests pass.

Looks like I'll need some more help with this.
The generate_np caching system is a big mess. While trying to get the tests to pass, I've cleaned up some problems with it, but I'm still left with two failures.
The SPSA attack strength test for the generate method is passing but both tests for the generate_np method are failing:
FAIL: test_attack_strength_np (test_attacks.TestSPSA)
Traceback (most recent call last):
File "/home/goodfellow/cleverhans/tests_tf/test_attacks.py", line 304, in test_attack_strength_np
self.assertLess(np.mean(feed_labs == new_labs), 0.1)
AssertionError: 0.4 not less than 0.1
======================================================================
FAIL: test_attack_strength_np_batched (test_attacks.TestSPSA)
Traceback (most recent call last):
File "/home/goodfellow/cleverhans/tests_tf/test_attacks.py", line 318, in test_attack_strength_np_batched
self.assertLess(np.mean(feed_labs == new_labs), 0.1)
AssertionError: 0.5 not less than 0.1
That being said, the right arguments seem to be coming through to generate, it's not getting a false cache hit, etc. Any ideas what's wrong with it?

Nevermind, I found why the tests were failing. I was detecting the format of the labels wrong. I was testing their shape and assuming int labels would have only one dimension, but they were actually getting passed in with an extra axis of dim 1. I changed it to assume that if the data is ints we need to call one_hot and otherwise we don't.

Just looked at the change, that seems good to me - thanks!

@juesato : do you mean go ahead and merge the branch? I wasn't sure if you meant for your comment to be a whole review, or just commenting on one thing.

I'm happy with the changes related to SPSA (in attacks.py attacks_tf.py test_attacks_tf.py utils_tf.py). I haven't looked at the other additions. Do you want me to review them?


Failed to build the master branch

System information

OS Platform and Distribution (e.g., Linux Ubuntu 18.04 tensorflow/tensorflow:latest-devel-gpu-py3):
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): source
TensorFlow version: master branch
Python version:3.6.9
Installed using virtualenv? pip? conda?:
Bazel version (if compiling from source):Build label: 2.0.0
GCC/Compiler version (if compiling from source):gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)
CUDA/cuDNN version: 10.1
GPU model and memory:

Describe the problem
Master branch build failed
Provide the exact sequence of commands / steps that you executed before running into the problem
bazel build --jobs=4 --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package

Any other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
NFO: Call stack for the definition of repository 'llvm-project' which is a tf_http_archive (rule definition at /home/tx/Downloads/tensorflow/third_party/repo.bzl:134:19):
 - /home/tx/Downloads/tensorflow/tensorflow/workspace.bzl:598:5
 - /home/tx/Downloads/tensorflow/WORKSPACE:19:1
ERROR: /home/tx/Downloads/tensorflow/tensorflow/lite/toco/BUILD:439:1: Target '//tensorflow/lite/toco:toco' depends on toolchain '@local_config_cc//:cc-compiler-k8', which cannot be found: error loading package '@local_config_cc//': Unable to load file '@local_config_cc//:armeabi_cc_toolchain_config.bzl': file doesn't exist'
ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' failed; build aborted: error loading package '@local_config_cc//': Unable to load file '@local_config_cc//:armeabi_cc_toolchain_config.bzl': file doesn't exist
INFO: Elapsed time: 10.477s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (226 packages loaded, 3671 targets configured)
    currently loading: @com_github_grpc_grpc//src/compiler ... (2 packages)
    Fetching @com_google_absl; fetching 4s

Provide the exact sequence of commands / steps that you executed before running into the problem.Please provide details about what platform you are using (operating system, architecture).Thanks!

@ravikyram . Steps

./configure (all default)
bazel build --jobs=4 --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
Then build fails. Operation system is the same as tensorflow/tensorflow:latest-devel-gpu-py3 which is Ubuntu 18.04 64bit.

Can you sync back to master and try again? Also, please post the commit hash at which you are trying to build.
Is this a regression? Was there a moment where you were able to build using these steps?
PS: You should not need --config=cuda if you set CUDA related things in ./configure

I have tried many times on latest master up to 566c03a749. I built successfully once a long time ago. But I did bazel clean after that. Also tried removing --config=cuda, still failed with the same reason.

@372046933,
Is this still an issue?
Could you please update TensorFlow to v2.4.1 and let us know if you are facing the same error. Thanks!

Sorry, I cannot verify v2.4.1 due to insufficient disk space.

@372046933,
Thank you for the update.
Can we close this issue for now? Please feel free to re-open if you face the same error with TF v2.4.1 as well.

Thanks


Unexpected behaviour for budgets in Complete Mode

Bicep version
0.6.18
Describe the bug
I tried to change a budget on a Resource Group by using Bicep. Ideally I wanted to move the start date up on the budget as well.
I started by using Bicep in incremental mode but got the error:
Start date of budgets cannot be updated. Please delete and create a new budget
This is fine, expected.
I then tried instead to use Complete mode, thinking that would delete the budget and then remake it with the correct value.
But got the same error:
Start date of budgets cannot be updated. Please delete and create a new budget
This was less expected, I feel like it should work but understand if it doesn't work that way.
I then tried a Budget with a different name.
This completed with no errors but I now have two budgets on the resource group. This is super unexpected as

In complete mode, Resource Manager deletes resources that exist in the resource group but aren't specified in the template.

I would expect me to be left with only the new budget and the old one deleted.
To Reproduce
Steps to reproduce the behaviour:
First running the below Bicep file:
param startDate string = '2022-06-01'

resource budget 'Microsoft.Consumption/budgets@2021-10-01' = {
  name: 'testBudget'
  properties: {
    amount: 100
    category: 'Cost'
    notifications: {
      actual_GreaterThan_100_Percent: {
          enabled: true
          operator: 'GreaterThan'
          threshold: 100
          contactEmails: [
<EMAIL_ADDRESS>          ]
          thresholdType: 'Actual'
      }
  }
    timeGrain: 'Monthly'
    timePeriod: {
      startDate: startDate
    }
  }
}

with az deployment group create --template-file .\budgetTest.bicep --mode complete --resource-group SSKTestRG
Then run it again with the month changed.
Just:
param startDate string = '2022-07-01'
And then the error occurs:
Start date of budgets cannot be updated. Please delete and create a new budget
I then change the name of the budget:
resource budget 'Microsoft.Consumption/budgets@2021-10-01' = {
  name: 'testBudget2'
...

and run the same az command and I now have two budgets set on the resource group.
Let me know if you need any more information 😃

The first result is as expected. If the code is not changing between deployments and these are the only resources in the RG, then there will be no difference between running in Incremental vs Complete, as it only affects whether we ignore or delete resources that are not in the RG.
In the second case, I would also expect the budget to be deleted, but there may be some quirks of the budget resource.
Complete mode in general has a lot of edge-cases and limitations. We are currently in the process of building a replacement for complete mode called Deployment stacks. We have a private preview now if you'd like to try it out. This resource should be deleted for the same case with deployment stacks.
cc @apclouds / @bmoore-msft

budgets are a funky resource - they aren't returned in scope lists so we don't see that they should be deleted...  deploymentStacks won't be susceptible to that funkiness so should help there... So if you increment your budget name every time (in the template) it should work ok.  Not the ideal declarative model though that has to do more with the budget than the model.

We are currently in the process of building a replacement for complete mode called Deployment stacks. We have a private preview now if you'd like to try it out.

I didn't know this was a thing! I really like the sound of it, it would be neat to try it out if you can let me know how to do so?
Thanks for the response :)


Write and Read

If i want to read and write a string than how can i do it like nrfConnect ?

did you find any solution??

@Hashim-svg pls share your code ,i also using kotlin for development.

To write a string you need to first convert it to a ByteArray. For most strings they'll be sent over in UTF-8 encoding, and the following will work:
val yourString = "something"
val yourByteArray = yourString.toByteArray(StandardCharsets.UTF_8)

// Perform write using `yourByteArray` as the payload


defaultCountry props is not reactive

Example :
<vue-tel-input :defaultCountry='countryCode' v-model="value"></vue-tel-input>
data(){
return{
countryCode : 'US'
}
}

i have the same problem. Without that its not possible using this on Edit Pages, because its always using the Components default country

Edit
It works for me if i use remove the national from mode

Could you explain me in brief?

@darthsoup @Richard-Richie
if you're planing to use defaultCountry prop to re-populate your form for editing in a CRUD operations well...the library uses a method called "choose" to set the active country , it takes a country is "iso" name as its only parameter , so now u need a way to access child component is data there's 3 ways i can think of , in my project i use vue is "ref" feature :
<template>
    <vue-tel-input ref="phoneNumber" v-model="value"></vue-tel-input>
  </template>

<!-- in one of your methods -->
  this.$refs.phoneNumber.choose("USA"); // set country to usa

'toLowerCase' of undefined"

My solution was to force re-render the component on country code change. To do that, I added a key same as country code, on code change, the key should be updated and vue force re-renders your component.
<vue-tel-input :defaultCountry='countryCode' v-model="value" :key='countryCode'></vue-tel-input>

'toLowerCase' of undefined"

this.$refs.phoneNumber.choose({iso2:"USA"});
it works

'toLowerCase' of undefined"

this.$refs.phoneNumber.choose({iso2:"USA"});
it works

From v5 you can do this.$refs.phoneNumber.choose('AU')

From v5 you can do this.$refs.phoneNumber.choose('AU')


Elite Requester Recipe seems wrong

not sure if i'm missing something, but the recipe for the Elite Requester only shows as an un-crafting recipe.

Adding image

You have to install rsrequestify


feat(app): support app level effect scope

close #8787

This would potentially also enable an app.onUnmount() with just one line
app.onUnmount = (fn) => app.runWithContext(() => onScopeDispose(fn))

https://github.com/vuejs/core/pull/4619
https://github.com/vuejs/core/issues/4516

Thank you for your feedback. I have a couple of questions I would like to confirm:

In #4619, the implementation of adding app.onUnmount functionality is located in runtime-core/src/apiCreateApp.ts instead of runtime-dom/src/index.ts. I'm not sure if this PR should follow this approach.

Currently, Vue internally utilizes the new EffectScope(true) approach. Would it be advisable for this PR to adopt a class-based approach instead of using effectScope(true)?

Thank you! Your invaluable guidance means a lot to me.

I have moved the implementation to runtime-core and added a test case. Could you please review?

I'm wondering about whether this could cause memory leaks, and whether that matters.
I've put together an example using Vue Router:

Playground - main
Playground - this PR

Steps to reproduce:

Navigate between the two routes a few times by clicking the links.
Take a Heap Snapshot (make sure you select the main VM, not the workers).
Filter the snapshot by the class VTrackLeak.
On main you'll just find a single instance, but with this PR you'll find an extra instance per navigation.

The key parts of the code are:
router.beforeEach(() => {
  const { doubled } = useBlah()

  if (doubled.value) {
    // ...
  }
})

with:
function useBlah() {
  const raw = ref(1)
  const doubled = computed(() => raw.value * 2)

  raw.marker = new VTrackLeak()

  return { raw, doubled }
}

The callback for beforeEach is wrapped in app.runWithContext by Vue Router. That will capture the effect for the computed in useBlah, preventing it from being GCed until the application is unmounted, which may never happen.

@skirtles-code I'm surprised they don't both leak. I would consider that usage wrong, but it does show a distinction between being able to inject and being able to use reactive effects is important. I think it makes more sense to keep things separate and adding an effectScope to the app would allow to have app.scope.run() to track reactive effects, keeping both APIs separate and forcing libraries like the router to use any they feel are relevant.


Push directory scan to worker and add multiprocessing

Add directory scan to worker queue
Remove some unnecessary update on photo

rewrite _generate_thumbnail
reorder albums creation after photo insert
Use multiprocessing with nb process defined in HEAVYWEIGHT_PROCESS variable environment

I think add environment variable bellow will be added on  librephotos-docker/backend/entrypoint.sh  for docker
export DJANGO_SETTINGS_MODULE=ownphotos.settings

I suspect these changes will undo #131, particularly the memory usage reduction in this commit ; https://github.com/LibrePhotos/librephotos/pull/131/commits/4c41638d3bcbb885453e71fea390a8f6166ab5f6
It could probably be done by adding jobs to the 'pool' as they are found during directory-walking instead of in one big assignment here ;
https://github.com/LibrePhotos/librephotos/blob/76b219d9c74f5d6de21db51aa1ba82f92dcdfab2/api/directory_watcher.py#L206
Otherwise it looks like a great feature 👍 but memory-usage should be considered

agree, i have create iterable class based on walking directory in my first version.
but map() and startmap() get all value to split them into each process (see https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.map)
I propose that : create db table with temporary data and each process can get unprocess path into the table.
sample table :
api_directory_scan_pool (  job_id, path , processed )
Are you agree?

Hmm I don't quite understand that proposal I'm afraid.
An alternative way could be to build a file-list when walking directory up to a max-size (a few hundred file-paths?) .. and then hand them over to the 'pool' , clear the list, and restart adding to the intermediate list.
But I dont know much about the mechanisms here so I think @derneuere must advice what is best

Sorry for my English!!
Create fork is a big cost because of the memory required, restart starmap is not a good idea i think.
My proposition is create list of image path to process in database and processes get image path to process from database. There stop when no path to process. In this way there is no memory use to the path list, whatever the number of files

On my last test with 16840 photos with 6 heavy processes 1Go on Ram only

Thanks for your pull request!
Sorry for my late reply, didn't have a lot of time in the last couple of weeks and had to look over your older pull requests! :D
Do we know how starmap behaves when it is used in a big library? Like a libaray of one TB of photos?
Also starmap seems to have a parameter "chunk size", which should improve memory usage, because it allows us to iterate in chunks. I think the chunk size should be the number of the number of heavyweight processes, or?
I am not a fan of a temporary table because I want to keep the data structure lean.

I have commit my old version with iterator.
But this version need 2 folders scan.
On all my test with chunk size all files are scan before launch processes (add log in walk dir) . whatever the value of chunk all a files a send into the log.

@rafntor @derneuere
I hope that new commit will be ok for you.
I become a multiprocesses expert :-)

after some thought i don't know if it's good because a directory is only handled by one process

I will rewrite with "push photo to queue"

Thank for your comment. This version work good, without list of path

After check List with 180000 time the string "/var/lib/librephotos/data/photos/2020/02-février/Escapade à Cabourg 17 au 20 février/20200217_165525.jpg" take 19Mo in a file.

I'm come back to the first version
After check List with 180 000 time the string "/var/lib/librephotos/data/photos/2020/02-février/Escapade à Cabourg 17 au 20 février/20200217_165525.jpg" take 19Mo in a file.
>>> import sys
>>> a=[]
>>> for tmp in range(180000):
...  a.append("/var/lib/librephotos/data/photos/2020/02-février/Escapade à Cabourg 17 au 20 février/20200217_165525.jpg")
...
>>> sys.getsizeof(a)
1443576

This means that the list take 1.443576 Mo...
Lot of time lost to win nothing

Hmm, maybe the problem wasn't the list but something else.
I merged dev into your branch. Could you check if the multiprocessing still works with wand/imagemagick or if I broke it?

Except issue #211, all working good after the both "Merge branch 'dev' into multiproc"
By default, there is one process

Perhaps the list-problem was something else yes ... but I am very uncertain - without the fix my setup was unable to do any scanning and the time spent counting files is negligible compared to time spent analyzing photos.
And I am not sure the quick test above is conclusive ;
>>>
>>> import sys
>>> a=[]
>>> for tmp in range(180000):
...   a.append("X")
...
>>> sys.getsizeof(a)
1443576
>>>

There is other way to check

But you are right test is not conclusive
for tmp in range(180000):
...   a.append("/var/lib/librephotos/data/photos/2020/02-février/Escapade à Cabourg 17 au 20 février/" + str(a) + ".jpg")

Maybe you are right about the conclusive test.
Bellow test with not the same string (using the tmp value in the string)
180 000 string take less than 8MB in this case

@rafntor Could you test this PR with your large library to check if it leads to a memory leak?

Sure, but I dont have environment to build it ; could you provide a test-image ?

It seems to run just fine 👍
But, surprisingly I see no performance impact, scanning-time and memory-usage are very similar - a 15 minute test-run imports ~110 pictures on my system both with and without these changes.
Running docker desktop on windows in wsl2 mode, on AMD Ryzen 7 1700 (8 cores)

Yes by défaut thé nimber of processus is 1
Set HEAVYWEIGHT_PROCESS=3


Strange issue with replies not being pulled unless shared article link ends with /

Greets,
I recently implemented webmentions on a Blot blog I migrated to from Wordpress.
For reference, URLs to articles on Blot end without a trailing slash, i.e. https://blog.com/post (when I was on Wordpress, they all ended with trailing slashes).
I'm experiencing an odd issue when I crosspost the article on Mastodon Bridgy will find and pull all links, likes and reposts BUT it will not pull replies.
After some experimenting with posting a new article, I found out if I added a trailing slash to the end of the article when crossposting it, replies are found and pulled in.
Unfortunately, Blot defaults to showing articles w/o trailing slashes so they don't show up unless you explicitly add / to the end of the URL when visiting it.
I can confirm in the Bird.gy logs that it's finding the article without the trailing slash as it should.
(I asked the developer for Blot if there is a way to simply set up a redirect so all articles end with / but he said the server doesn't support that (Express) but he could add code to make it so, however he didn't recommend it because of other possible unforeseen consequences when messing with that sort of thing.)
I'm wondering if my Bridgy account is somehow "stuck" in the old Wordpress trailing slash mode and needs to be wiped clean or something?
So strange. Is there something further I can do to help fix this without needing to have the developer hard code a fix?
Appreciate any thoughts or suggestions.  I can provide logs from Brigdy, just let me know what you need in specific.
Ray
P.S. The blog URL in question is https://alongtheray.com and the Mastodon account is https://alongtheray.social/@ray

Hmm! Strange indeed.
I'm not quite seeing the same behavior. It looks like Bridgy's successfully sending webmentions to both URL variants, and your site is separating them by URL. For example, https://alongtheray.com/short-musings-on-losing-100lbs/ currently has two Bridgy webmentions, and https://alongtheray.com/short-musings-on-losing-100lbs has six different ones.

Ah, that was me -- I manually submitted a link with trailing slash for testing purposes so that's why you're seeing it. Normally there will only be the link without the trailing slash.
The issue persists though -- even at the URLs above you'll see no replies were pulled in although they are there in the social media thread.
So in my testing I found if I shared a link on social media with the trailing slash likes, reposts, etc. replies will be seen and pulled in by Bridgy. However, if I share the normal URL without the trailing slash, only likes, reposts etc show up -- no replies. Bridgy doesn't even see them. (The social media post is here: https://alongtheray.social/notice/ALLzfYOZXcsSz9CfFg)
Strange indeed.
If you like I can make another test post (I deleted the prior one, should have left it up for you) and show you the issue.
Thanks for chiming in and all you do!
Ray

Hmm! Thanks for the details. I get "Not found; This resource could not be found" from https://alongtheray.social/notice/ALLzfYOZXcsSz9CfFg . Is it not public?

Apologies if you meant you deleted it!

@ScootRay could you show me an existing public reply to a post with a non-slash URL that should have been backfed but wasn't? The only examples I'm seeing so far all seem to be deleted.
More data points:

I do see webmentions for Twitter replies to non-slash URLs, eg one on https://alongtheray.com/short-musings-on-losing-100lbs#mentions-list
I also see webmentions for Mastodon/Soapbox posts that link to non-slash URLs, eg your own https://alongtheray.social/notice/ALLzfYOZXcsSz9CfFg, which is now deleted

I'm using Pleroma with the Soapbox front end.
To help you out, I put up two test posts and shared them - one with and one without trailing slash and I replied to them on my Fediverse account.
https://alongtheray.com/without-trailing-slash (I shared this without trailing slash)
https://alongtheray.com/with-trailing-slash/ (shared with trailing slash)
Unexpectedly, the one without trailing slash pulled in the reply just fine this time so I'm not sure why previous posts without trailing slash (which is the standard format on Blot.im) didn't pull in replies (which are here: https://alongtheray.social/@ray/posts/ALLzfYOZXcsSz9CfFg - it's not deleted, just give it a few to load).
Color me befuddled.
As for the 100lbs URL above, it is weird it pulled in Twitter replies but not Fedi replies.
As always, thanks for your help.
Ray
UPDATE! I think I have it resolved -- my Pleroma instance was set to not allow public browsing of timelines, etc. Even though Bridgy has access to my account via authentication, it apparently still needs to be able to read public timelines without access. As soon as I opened it up, replies worked.
I say I think that's the issue because for some reason a reply went through anyway on the without-trailing-slash post above before I opened the public timeline so it's still a little confusing but I hope its fixed now....

Nice sleuthing! Glad replies are coming through onw, fingers crossed the public browsing setting fixes or at least helps it. Feel free to close whenever you're comfortable!

Thanks! I'll close it after a couple posts go through and make sure it's working fine. Will report back soon. Thanks again.
Ray

Unfortunately the issue persists... Here's the post in question: https://alongtheray.com/moments-of-bliss-are-allowed
And the thread on Pleroma: https://alongtheray.social/@ray/posts/ALZPdwlR5iyXqcGrZI
It does pick up the likes, just not the replies. It seems pretty random about it -- as I've seen on prior threads it works and on some others it doesn't. I haven't been able to find the common denominator though.
Hopefully a fresh set of eyes can figure it out...?
Thanks!
Ray

Hrm, sorry! I haven't dug into this much yet, but I suspect a Pleroma/Mastodon incompatibility, specifically what they include in API responses. It's great that Pleroma, PixelFed, etc support the Mastodon API so that Bridgy can try to work with them! But I don't officially support anything other than Mastodon yet due to (possible) incompatibilities like this. I'll still take a look eventually though.

Totally understood and no rush! I’m even amazed it works considering all the moving parts involved in different apps, etc. Props to that : )
Ray
--==> Follow me Along the Ray @ https://alongtheray.com <==--
...a wandering alien from a different plane roaming the universe in a tiny camper...
On Jul 18, 2022, 6:08 PM -0400, Ryan Barrett @.***>, wrote:

Hrm, sorry! I haven't dug into this much yet, but I suspect a Pleroma/Mastodon incompatibility, specifically what they include in API responses. It's great that Pleroma, PixelFed, etc support the Mastodon API so that Bridgy can try to work with them! But I don't officially support anything other than Mastodon yet due to (possible) incompatibilities like this. I'll still take a look eventually though.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>

Confirmed, the Pleroma API on your instance is oddly returning"replies_count": 0 for this post. Here's the full API object for the post, fetched from https://alongtheray.social/api/v1/statuses/ALZPdwlR5iyXqcGrZI, with a few unrelated fields elided:
{
   "account": {...},
   "application": {
      "name": "SoapboxFE_2022-06-17T02:46:26.244Z",
      "website": null
   },
   "bookmarked": false,
   "card": {...},
   "content": "I recently added a "Vault of Memories" to the blog because memorable moments kept coming up when writing. Now I'll have a way to store and cross-reference 'em in Obsidian. Wrote a bit about it here:  https://alongtheray.com/a-couple-new-things",
   "created_at": "2022-07-17T03:39:08.000Z",
   "emojis": [],
   "favourited": false,
   "favourites_count": 2,
   "id": "ALZPdwlR5iyXqcGrZI",
   "in_reply_to_account_id": null,
   "in_reply_to_id": null,
   "language": null,
   "media_attachments": [],
   "mentions": [],
   "muted": false,
   "pinned": false,
   "pleroma": {...},
   "poll": null,
   "reblog": null,
   "reblogged": false,
   "reblogs_count": 0,
   "replies_count": 0,
   "sensitive": false,
   "spoiler_text": "",
   "tags": [],
   "text": null,
   "uri": "https://alongtheray.social/objects/fe90c443-5cd1-4017-b32c-e573d3a1cd0f",
   "url": "https://alongtheray.social/notice/ALZPdwlR5iyXqcGrZI",
   "visibility": "public"
}

If I fetch the replies API endpoint https://alongtheray.social/api/v1/statuses/ALZPdwlR5iyXqcGrZI/context , it correctly serves the replies we see on the post. Bridgy uses the post's replies_count field to determine whether to fetch that endpoint, though, so it's currently skipping it.
I don't see anything about this on https://docs-develop.pleroma.social/backend/development/API/differences_in_mastoapi_responses/ . Maybe file a bug with them?

Wow, great find. Thanks for digging deep into it. I’ll find out where to pass this on.
Thanks again!
Ray
--==> Follow me Along the Ray @ https://alongtheray.com <==--
...a wandering alien from a different plane roaming the universe in a tiny camper...
On Jul 18, 2022, 11:15 PM -0400, Ryan Barrett @.***>, wrote:

Confirmed, the Pleroma API on your instance is oddly returning"replies_count": 0 for this post. Here's the full API object for the post, fetched from https://alongtheray.social/api/v1/statuses/ALZPdwlR5iyXqcGrZI, with a few unrelated fields elided:
{
"account": {...},
"application": {
"name": "SoapboxFE_2022-06-17T02:46:26.244Z",
"website": null
},
"bookmarked": false,
"card": {...},
"content": "I recently added a "Vault of Memories" to the blog because memorable moments kept coming up when writing. Now I'll have a way to store and cross-reference 'em in Obsidian. Wrote a bit about it here:  https://alongtheray.com/a-couple-new-things",
"created_at": "2022-07-17T03:39:08.000Z",
"emojis": [],
"favourited": false,
"favourites_count": 2,
"id": "ALZPdwlR5iyXqcGrZI",
"in_reply_to_account_id": null,
"in_reply_to_id": null,
"language": null,
"media_attachments": [],
"mentions": [],
"muted": false,
"pinned": false,
"pleroma": {...},
"poll": null,
"reblog": null,
"reblogged": false,
"reblogs_count": 0,
"replies_count": 0,
"sensitive": false,
"spoiler_text": "",
"tags": [],
"text": null,
"uri": "https://alongtheray.social/objects/fe90c443-5cd1-4017-b32c-e573d3a1cd0f",
"url": "https://alongtheray.social/notice/ALZPdwlR5iyXqcGrZI",
"visibility": "public"
}
If I fetch the replies API endpoint https://alongtheray.social/api/v1/statuses/ALZPdwlR5iyXqcGrZI/context , it correctly serves the replies we see on the post. Bridgy uses the post's replies_count field to determine whether to fetch that endpoint, though, so it's currently skipping it.
I don't see anything about this on https://docs-develop.pleroma.social/backend/development/API/differences_in_mastoapi_responses/ . Maybe file a bug with them?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>


API for change namespace or remove single namespace prefix

Describe the bug
I faced with need to change namespace but with preserving prefix in a document
To Reproduce
I have tried:
#! /usr/bin/env ruby

require 'nokogiri'
require 'minitest/autorun'

class Test < MiniTest::Spec
  describe "Node namespaces tests" do
    it 'test namespace update #1' do
      doc = Nokogiri::XML.parse('<ns1:test xmlns:ns1="test:namespace"/>')
      doc.root.namespaces['ns1'] = 'test:namespace:upd1'

      expect(doc.to_xml).to eq('<?xml version="1.0"?>
<ns1:test xmlns:ns1="test:namespace:upd1"/>')
    end

    it 'test namespace update #2' do
      doc = Nokogiri::XML.parse('<ns1:test xmlns:ns1="test:namespace"/>')
      doc.root.add_namespace 'ns1', 'test:namespace:upd1'

      expect(doc.to_xml).to eq('<?xml version="1.0"?>
<ns1:test xmlns:ns1="test:namespace:upd1"/>')
    end

    it 'test namespace update #3' do
      doc = Nokogiri::XML.parse('<ns1:test xmlns:ns1="test:namespace"/>')
      doc.root.namespace = doc.root.add_namespace 'ns1', 'test:namespace:upd1'

      expect(doc.to_xml).to eq('<?xml version="1.0"?>
<ns1:test xmlns:ns1="test:namespace:upd1"/>')
    end
  end
end

Also I tried:
it 'test namespace update #3' do
    doc = Nokogiri::XML.parse('<ns1:test xmlns:ns1="test:namespace"/>')
    doc.root.namespace = doc.root.add_namespace 'ns2', 'test:namespace:upd1'

    expect(doc.to_xml).to eq('<?xml version="1.0"?>
<ns2:test xmlns:ns1="test:namespace" xmlns:ns2="test:namespace:upd1"/>')
end

But it not allowed to preserve namespace prefix
Expected behavior
I'm not sure, I just need an API to be able modify/remove specific namespaces for specific node
Environment
# Nokogiri (1.10.1)
    ---
    warnings: []
    nokogiri: 1.10.1
    ruby:
      version: 2.5.4
      platform: x86_64-darwin18
      description: ruby 2.5.4p155 (2019-03-13 revision 67245) [x86_64-darwin18]
      engine: ruby
    libxml:
      binding: extension
      source: packaged
      libxml2_path: "/Users/camobap/.rbenv/versions/2.5.4/lib/ruby/gems/2.5.0/gems/nokogiri-1.10.1/ports/x86_64-apple-darwin18.2.0/libxml2/2.9.9"
      libxslt_path: "/Users/camobap/.rbenv/versions/2.5.4/lib/ruby/gems/2.5.0/gems/nokogiri-1.10.1/ports/x86_64-apple-darwin18.2.0/libxslt/1.1.33"
      libxml2_patches:
      - 0001-Revert-Do-not-URI-escape-in-server-side-includes.patch
      libxslt_patches: []
      compiled: 2.9.9
      loaded: 2.9.9

Hi! Thanks for opening this issue.
I'm curious why you need this functionality? It's perhaps meaningful that in 10+ years of existence, nobody has asked for this functionality. Can you help me understand your use case a bit deeper? I followed the issues in which you referenced this issue but it didn't help me understand. Generally speaking, if you change the href associated with a namespace name, you're changing the semantics of the document in a very invasive way. Why is this desirable?
I would think that if you're changing the namespace identifier, you'd want to do a more thorough transformation of the document structure. If that's not the case, I would like to learn more!

I'm not inclined to implement the naive solution, which is for #add_namespace (aliased as #add_namespace_definition) to overwrite the href property in the Namespace object, because the default behavior of libxml2, and I think the expected behavior, is not to create that namespace if it already exists, and there's likely existing code that depends on this behavior.
We could add a Namespace#href= method to allow you to set the associated href for an existing namespace; but again I'm hesitant to introduce new API calls for behaviors that don't make intuitive sense, and what you're asking for does not make intuitive sense to me.
Again, please, help me understand what the desired API will be used for?

@flavorjones thanks for explanation I got your point and agree it smells like 'hack' )

because the default behavior of libxml2, and I think the expected behavior, is not to create that namespace if it already exists, and there's likely existing code that depends on this behavior.

Especially thank for this note
I think we can close this one

@flavorjones just to share more visibility there is exists two namespaces which have very similar or event the same (unfortunately I can prove it 100% because cannot find xsd):

http://schemas.openxmlformats.org/officeDocument/2006/math
http://schemas.microsoft.com/office/2004/12/omml

Probably better way will be just traverse doc


[FEAT] Redirect the user instead of showing 404 page

Is your feature request related to a problem? Please describe.

Currently if accessControlProvider blocks access to the resource for a user, a 404 page is shown. Instead I would like to redirect the user to the first available resource.
Take a look at this example:
{
  name: "orders",
  list: MuiInferencer,
  edit: MuiInferencer,
  show: MuiInferencer,
  create: MuiInferencer,
  canDelete: true,
},
{
  name: "users",
  list: MuiInferencer,
  create: MuiInferencer,
  edit: MuiInferencer,
  canDelete: true,
},

accessControlProvider={{
    async can({ resource }) {
      const userResponse = await supabaseClient.auth.getUser();
      const user = userResponse.data.role;
      if (user.role === "user-manager") {
        return Promise.resolve({
          can: resource === "users",
        });
      }
    },
  }}

If the user has user-manager role, then after sign in, he will see a 404 page. Instead I would like him to redirect to users resource.
Describe alternatives you've considered
No response
Additional context
No response
Describe the thing to improve
Provide a mechanism to redirect a user to the first available resource when access control blocks access.

Hello @Mikilll94, I guess that wouldn't be good UX. If a user doesn't have access to certain page, it should render error page, so user who doesn't have access may ask for access if they know which page they are on. Redirecting to another resource may look like a bug and user wouldn't know which page they don't have access to.

Hello @BatuhanW ,
Current behavior is blocking me because I have a user who has access only to users resource so after logging in he should land on the users page. Instead he lands on the first defined resource which results in 404. How this can be fixed?
IMO would be nice to have something like this:
return Promise.resolve({
          can: resource === "users",
redirectIfAccessDenied: "users"
        });

or add a possibility to define dynamic start page

Hello @Mikilll94,
I think it can be fixed from auth provider, WDYT? In the login method, you can define redirectTo path according the logged in user's permissions.

@BatuhanW
I was trying to do this but unfortunately I have found an issue. Resolving a Promise with a url is not redirecting a user anywhere. I am still using Refine v3 and I don't want to migrate now because my app is almost completed.

Hey @Mikilll94 ,
Thank you for contacting us. Can you provide us with a CodeSandbox or StackBlitz so that we can reproduce the issue?


QM/MM

Setting up a QM/MM interface using the dev version of NWChem. Will support mechanical, electrostatic, and covalent embedding

(this PR ended up being a huge amount of infrastructure ... which will be getting merged into master separately)


chore(dw): 5292 - improve chart look and feel

closes: https://issues.redhat.com/browse/RHOAIENG-5292
Description
improve look and feel of charts. includes:

keeping them from growing huge when on ultra wide screens or getting too small on narrow screens
usage chart legend text will grow as width allows
requested usage charts will grow horizontally without also growing vertically
status overview chart centered as it will always be top and single, donut size larger to maintain the text within better
fix tooltips not to show on top of the usage donut items

Screens (you'll have to expand images to really see the difference in how the charts sizing compares to the size of the text outside of the charts)
Status Overview before:

Status Overview after:

Project metrics before:

Project metrics after:

Usage tooltips:

How Has This Been Tested?
tested locally, existing tests still pass
Test Impact
none
Request review criteria:
make sure charts look good and handle resize well
Self checklist (all need to be checked):

[x] The developer has manually tested the changes and verified that the changes work
[x] Commits have been squashed into descriptive, self-contained units of work (e.g. 'WIP' and 'Implements feedback' style messages have been removed)
[x] Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
[x] The developer has added tests or explained why testing cannot be added (unit or cypress tests for related changes)

If you have UI changes:

[x] Included any necessary screenshots or gifs if it was a UI change.
[ ] Included tags to the UX team if it was a UI/UX change (find relevant UX in the SMEs section).

After the PR is posted & before it merges:

[ ] The developer has tested their solution on a cluster by using the image produced by the PR to main

PTAL @xianli123 @cfullam1 @yannnz

tested this on mocked data and the graphs look good. not testing on live data as this only is changing how things are rendered
/approve

/lgtm


Add "dyon_core"

This PR adds a core abstraction which is used by the “dyon” library.
The motivation is to make it easier to create and experiment with
Dyon-like programming languages without needing to implement the whole
runtime from scratch.
This will also hopefully reduce compile times when working on the
“dyon” library.
Fixes https://github.com/PistonDevelopers/dyon/issues/627

Move Vec4 to core
Move Mat4 to core
Added RuntimeEval trait
Added RuntimeExt (auto impl)
Added RuntimeErrorHandling (auto impl)
Added RuntimeResolveReference (auto impl)
Move Side to core
Move Flow to core
Move Call to core
Added RuntimeCore
Moved resolve to core
Added trait ErrorCore
Added trait VaribleCore
Added macro various
Moved PopVariable to core
Moved PushVariable to core
Added trait VariableType
Moved TINVOTS to core
implemented ErrorCore for Error

Benchmark:
test tests::bench_add           ... bench:  21,424,038 ns/iter (+/- 2,702,393)
test tests::bench_add_n         ... bench:   9,186,424 ns/iter (+/- 895,454)
test tests::bench_array         ... bench:  11,586,593 ns/iter (+/- 1,797,620)
test tests::bench_call          ... bench:  13,937,677 ns/iter (+/- 2,292,874)
test tests::bench_len           ... bench:   1,975,205 ns/iter (+/- 107,235)
test tests::bench_main          ... bench:     370,742 ns/iter (+/- 16,617)
test tests::bench_min           ... bench:  14,569,294 ns/iter (+/- 1,625,534)
test tests::bench_min_fn        ... bench:  42,384,177 ns/iter (+/- 5,553,994)
test tests::bench_n_body        ... bench:  62,099,301 ns/iter (+/- 7,412,392)
test tests::bench_object        ... bench:  17,519,676 ns/iter (+/- 3,331,253)
test tests::bench_primes        ... bench:  29,744,010 ns/iter (+/- 5,670,820)
test tests::bench_primes_trad   ... bench:  23,913,029 ns/iter (+/- 3,280,451)
test tests::bench_push_array    ... bench:  15,793,676 ns/iter (+/- 5,833,354)
test tests::bench_push_in       ... bench: 104,024,110 ns/iter (+/- 31,531,824)
test tests::bench_push_link     ... bench:  12,599,282 ns/iter (+/- 3,533,493)
test tests::bench_push_link_for ... bench:   6,180,317 ns/iter (+/- 1,831,915)
test tests::bench_push_link_go  ... bench:  14,638,699 ns/iter (+/- 10,464,024)
test tests::bench_push_str      ... bench:  70,738,547 ns/iter (+/- 9,654,822)
test tests::bench_sum           ... bench:   6,269,270 ns/iter (+/- 2,552,322)
test tests::bench_threads_go    ... bench:  38,165,237 ns/iter (+/- 27,005,427)
test tests::bench_threads_no_go ... bench:  23,649,905 ns/iter (+/- 11,120,078)

Closing for now, because outdated. Might return to this issue later.


Make ssl_ca_cert_file_path support an array of files

A proposed change to facilitate fixing https://github.com/fluent/fluent-plugin-kafka/issues/287. It would help to avoid parsing certs in the upper level code that should not need to know how to do that.

Thanks!

 I didn't mean to just drop a PR on your lap without an explanation but got pulled into something else. This change really helps us avoid parsing certs in the Fluentd Kafka plugin.


multilabel_plot.py  Error

lb = lbr.label(X)
It's work for running multi-class because we have only one label we will get.
but if we run multi-label, it seems not work due to we will get more than one label.
How to modify it?
thanks

I see, I think it can be change to the following (on line 34):
return yy[np.random.choice(np.arange(len(yy)), 1)[0]]

I will fix this bug in later version.
Thanks for reporting the bug.


case-insensitive is not currently supported

I found that KeywordProcessor(case_sensitive=False) is not supported yet.
Any ETA to support it?
Thank you.

Hey Luca, I added case-insensitive support to the underlying rust crate recently.
So now it is possible to add it here, just have to update src/lib.rs.

Any ETA to support it?

Probably within 2-3 weeks

Great news! If you don't mind, just link the release here to close the issue when it will be ready. Thank you!

You got it man

@lucazav Just released version 1.1.0 on PyPI


Prevent using emotes when in car

Prevent using emotes when in car thanks to SnaiCrys.

Should be as an option in cfg/client.lua, I like to use emotes in vehicles.

Ok will check if I can do that :)

This is now possible in the vrpex branch: https://github.com/ImagicTheCat/vRP/tree/vrpex#special-function-permission


[STORM-2189] RAS_Node::freeCPU outputs incorrect info

CPU trying to free should be _availCPU + amount rather than _availMemory + amount.

Great finding. +1

+1


how to change thaw component text size?

like above picture,
how can I  change calendar font size just like 2024-01-17(smaller) which I have modified using class?
thanks.

If you want to change the popup style of all date picker components, you can add a class to override it.
If you want to change only the popup style of the specified date picker component, I can add a popup_class attribute to the date picker component.

the quick solution is to" change the popup style of all date picker components", could you give me a class override example? I can try it.
thanks in advance.

Add the following classes.
.thaw-date-picker-panel {
    font-size: 12px;
}

it really can work, but the panel position is strange just live above picture.
and date panel can work too, the panel can appear but disappear quickly and date can not be selected?

Try it like this.
<Style>
".font-normal { font-weight: 400; font-size: 0.75rem; }"
".thaw-time-picker-panel, .thaw-date-picker-panel { font-size: 0.75rem; }"
</Style>
"date"
<DatePicker class="font-normal" value=date_value/>
"time"
<TimePicker class="font-normal" value=time_value />

Try it like this.
<Style>
".font-normal { font-weight: 400; font-size: 0.75rem; }"
".thaw-time-picker-panel, .thaw-date-picker-panel { font-size: 0.75rem; }"
</Style>
"date"
<DatePicker class="font-normal" value=date_value/>
"time"
<TimePicker class="font-normal" value=time_value />

thanks，it really work, I don't know why it's class="font-normal", but it really work,
sorry again,
I add button color class as follwing picture, but why the button color keep as default(not changed)?
```
".thaw-time-picker-panel, .thaw-date-picker-panel { font-size: 0.75rem; }"
".thaw-button {background-color:#04AA6D;}"
"next"

There should be a style overlay because of css priority.

There should be style overwriting due to css priority issues.

is there temporary solution?

I add button color class as follwing picture, but why the button color keep as default(not changed)?

Try this.
<Style>
".my-button.thaw-button { background-color: #04AA6D; }"
</Style>
<Button class="my-button">"next"</Button>

splendid! all you solution can work
thanks!


Use MSBuild Solution-parsing API

Currently, there are some files copied over from Roslyn to handle solution parsing. Eventually, the new MSBuild solution parsing API should be used, which is coming in MSBuild v12.
SolutionFile
SolutionConfigurationInSolution

I created an issue requesting a nuget package https://github.com/Microsoft/msbuild/issues/45

This was fixed with #832.


[BUG] Replace the Icons for Settings and such with the one from snappercord's repo.

Describe the bug
Basically when using the default settings icons from downloading Zelk, the majority of icons are bugged, like some of them showing the default powercord logo for Spotify, Theme Toggler, etc.
Because of this, the icons should just be switched to the one from the snappercord repo. Plus it's a very simple fix to implement, just go to zelk.css, and replace the settings icon link with the one from snappercord's repo.
To Reproduce
Steps to reproduce the behavior:

Download Powercord and install it.
Install Theme Toggler.
Install Zelk.
Look at the Settings Icons.

Screenshots
https://imgur.com/8y4wuqL
Setup

OS: [e.g. Windows]
Client Mod: [e.g. BetterDiscord]
Release Channel: [e.g. canary, stable]
Enabled Themes: Zelk - Amazing Activity Cards - Better Badges - Discolored
Enabled Plugins: Default Powercord Plugins - BetterDiscord Plugins | These plugins do not affect the UI.

thought it was abandoned, but after the recent commits i'll revert it back to before https://github.com/schnensch0/zelk/commit/82d4e3a4c33a2d1b7981fe3de180b0fecc64ca53

done


[Fix](inverted index) fix bkd index file is empty error status

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


k8s-infra: Update affiliation

Follow-up of:

https://github.com/kubernetes/website/pull/41508

Update affiliation of one of the authors. "Fast Forward" is a program operated by Fastly

/assign @sftim
cc @fsmunoz

/close


Issue when building angular with buildOptimizer on

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[x ] Bug report?
[ ] Performance issue
[ ] Feature request
[ ] Documentation issue or request
[ ] Support request
[ ] Other... Please describe:

Current behavior
After building an Angular application with buildOptimizer=true the running application throws the following error:
ERROR Error: Uncaught (in promise): ReferenceError: Must call super constructor in derived class before accessing 'this' or returning from derived constructor
ReferenceError: Must call super constructor in derived class before accessing 'this' or returning from derived constructor
    at new AppStore (cs-performance-manager.js.pre-build-optimizer.js:2809)
    at factory (cs-performance-manager.js.pre-build-optimizer.js:2818)
    at core.js.pre-build-optimizer.js:27000
    at tn (core.js.pre-build-optimizer.js:26943)
    at en (core.js.pre-build-optimizer.js:26902)
    at fn.get (core.js.pre-build-optimizer.js:27995)
    at core.js.pre-build-optimizer.js:2169
    at G (core.js.pre-build-optimizer.js:2169)
    at factory (cs-performance-manager.js.pre-build-optimizer.js:4735)
    at core.js.pre-build-optimizer.js:27000
    at tn (core.js.pre-build-optimizer.js:26943)
    at en (core.js.pre-build-optimizer.js:26902)
    at fn.get (core.js.pre-build-optimizer.js:27995)
    at en (core.js.pre-build-optimizer.js:26907)
    at core.js.pre-build-optimizer.js:26977
    at new v (zone-error.js.pre-build-optimizer.js:100)
    at M (zone.js.pre-build-optimizer.js:831)
    at M (zone.js.pre-build-optimizer.js:788)
    at zone.js.pre-build-optimizer.js:892
    at t.invokeTask (zone.js.pre-build-optimizer.js:423)
    at Object.onInvokeTask (core.js.pre-build-optimizer.js:21825)
    at t.invokeTask (zone.js.pre-build-optimizer.js:422)
    at e.runTask (zone.js.pre-build-optimizer.js:195)
    at g (zone.js.pre-build-optimizer.js:601)

The error in the application occurs here:
/**
 * @fileoverview added by tsickle
 * @suppress {checkTypes,extraRequire,uselessCode} checked by tsc
 */
/**
 * @return {?}
 */
function createInitialState() {
    return {
        isImpersonated: false,
        canAccessAdmin: false,
        isDebugUser: false,
        activeModule: null,
        activeSection: null,
        navbarResultParams: null,
        allowNegativeDataEntry: false
    };
}
let AppStore = class AppStore extends Store {
    constructor() {   // <--- error line number: 2809
        super(createInitialState());
    }
};

It appears that the this used in the constructor in the Store class is the cause of the error when buildOptimizer is turned on
export class Store<S> {
  private store: BehaviorSubject<Readonly<S>>;
  private storeValue: S;
  private inTransaction = false;
  private _initialState: S;
  protected cache: StoreCache = {
    active: new BehaviorSubject<boolean>(false),
    ttl: null
  };

  constructor(initialState: Partial<S>, protected options: Partial<StoreConfigOptions> = {}) {
    this.onInit(initialState as S); // <---- cause of the issue??
  }

When buildOptimizer is turned off the application start en runs as expected.
Expected behavior
Compile with following settings:
 "mock-dev-packages-prod": {
              "fileReplacements": [
                {
                  "replace": "src/environments/environment.ts",
                  "with": "src/environments/environment.mock.ts"
                }
              ],
              "assets": [
                "src/favicon.ico",
                "src/assets"
              ],
              "optimization": true,
              "outputHashing": "all",
              "sourceMap": false,
              "extractCss": false,
              "namedChunks": false,
              "aot": true,
              "extractLicenses": true,
              "vendorChunk": false,
              "buildOptimizer": true,
              "preserveSymlinks": true,
              "budgets": [
                {
                  "type": "initial",
                  "maximumWarning": "2mb",
                  "maximumError": "5mb"
                }
              ]
            }

Minimal reproduction of the problem with instructions
create a store in a Angular application with dependencies:
"@datorama/akita": "~3.0.0"
"@angular/animations": "~7.2.8",
"@angular/common": "~7.2.8",
"@angular/compiler": "~7.2.8",
"@angular/core": "~7.2.8",
"@angular/forms": "~7.2.8",
"@angular/http": "~7.2.8",
"@angular/platform-browser": "~7.2.8",
"@angular/platform-browser-dynamic": "~7.2.8",
"@angular/router": "~7.2.8",
devDependencies:
"@angular-devkit/build-angular": "~0.13.5",
"@angular-devkit/build-ng-packagr": "~0.13.5",
"@angular/cli": "~7.3.5",
"@angular/compiler-cli": "~7.2.8",
Environment

Angular version: 7.2.8

Browser:
- [ x] Chrome (desktop) version XX
- [x ] Chrome (Android) version XX
- [x] Chrome (iOS) version XX
- [ x] Firefox version XX
- [x ] Safari (desktop) version XX
- [x ] Safari (iOS) version XX
- [x ] IE version XX
- [x ] Edge version XX

For Tooling issues:
- Node version: v8.11.3
- Platform:   Windows

Thanks for the report. Can you create a simple repo that reproduces the error? We are not working with this option.

@NetanelBasal thanks for your quick response. While creating the repro, i've found the cause:
The target was es2015 and not es5 while building the application. The internal published lib that was used by the application was using the fesm2015 build.
FESM2015 usedLib.js
function createInitialState() {
    return {
        isImpersonated: false,
        canAccessAdmin: false,
        isDebugUser: false,
        activeModule: null,
        activeSection: null,
        navbarResultParams: null,
        allowNegativeDataEntry: false
    };
}
let AppStore = class AppStore extends Store {
    constructor() {
        super(createInitialState());
    }
};

FESM5 usedLib.js
function createInitialState() {
    return {
        isImpersonated: false,
        canAccessAdmin: false,
        isDebugUser: false,
        activeModule: null,
        activeSection: null,
        navbarResultParams: null,
        allowNegativeDataEntry: false
    };
}
var AppStore = /** @class */ (function (_super) {
    __extends(AppStore, _super);
    function AppStore() {
        return _super.call(this, createInitialState()) || this;
    }
    AppStore.decorators = [
        { type: Injectable, args: [{ providedIn: 'root' },] }
    ];
    /** @nocollapse */
    AppStore.ctorParameters = function () { return []; };
    /** @nocollapse */ AppStore.ngInjectableDef = defineInjectable({ factory: function AppStore_Factory() { return new AppStore(); }, token: AppStore, providedIn: "root" });
    AppStore = __decorate([
        StoreConfig({ name: 'app' }),
        __metadata("design:paramtypes", [])
    ], AppStore);
    return AppStore;
}(Store));

I've added the repro to this comment.
akita-repro.zip

Why is the source code compiled to FESM2015?

Changing the target value in tsconfig to "es5" solves the issue.

It's not an Akita issue. See https://github.com/angular/angular-cli/issues/10658

thanks for following up. You are correct by stating that the target build should be es5. That's what i also found out when i mentioned that i found the cause. I only posted the repro to make you aware that stuff is breaking when using es2015 and akita. But as you stated it's not a akita problem but a buildOptimizer problem so no issue here ;)
keep up the good work, 🧡 akita


Duckdb's TypeMismatchException raised in CytoTable's convert() workflow due to nan values being stored as strings instead of expected types

CytoTable's convert() function seems to capture nan's as string types within the cell-helath dataset causing duckdb to raise duckdb.TypeMismatchException error.
Below is the code to replicate the problem:
import pathlib
import cytotable

# sqlite file path
sqlite_file = str(pathlib.Path("./SQ00014613.sqlite").resolve(strict=True))

# execute covert workflow
data = cytotable.convert(sqlite_file, dest_path="./parquet_data/test.parquet", dest_datatype="parquet", preset="cellprofiler_sqlite", source_datatype="sqlite")

link to download data
Traceback
From Prefect:
13:38:45.442 | ERROR   | Task run '_read_data-1' - Encountered exception during execution:
Traceback (most recent call last):
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/site-packages/prefect/engine.py", line 1533, in orchestrate_task_run
    result = await run_sync(task.fn, *args, **kwargs)
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 156, in run_sync_in_interruptible_worker_thread
    tg.start_soon(
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
    raise exceptions[0]
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 135, in capture_worker_thread_and_result
    result = __fn(*args, **kwargs)
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/site-packages/cytotable/convert.py", line 59, in _read_data
    _duckdb_with_sqlite()
duckdb.TypeMismatchException: Mismatch Type Error: Invalid type in column "Nuclei_Correlation_Costes_AGP_DNA": expected float or integer, found "nan" of type "text" instead.

13:38:45.680 | ERROR   | Flow run 'wonderful-cuckoo' - Encountered exception during execution:
Traceback (most recent call last):
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/site-packages/prefect/engine.py", line 665, in orchestrate_flow_run
    result = await run_sync(flow_call)
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 156, in run_sync_in_interruptible_worker_thread
    tg.start_soon(
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
    raise exceptions[0]
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 135, in capture_worker_thread_and_result
    result = __fn(*args, **kwargs)
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/site-packages/cytotable/convert.py", line 730, in _to_parquet
    common_schema = _infer_source_group_common_schema(
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/site-packages/prefect/tasks.py", line 469, in __call__
    return enter_task_run_engine(
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/site-packages/prefect/engine.py", line 965, in enter_task_run_engine
    return run_async_from_worker_thread(begin_run)
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 177, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/site-packages/anyio/from_thread.py", line 49, in run
    return asynclib.run_async_from_thread(func, *args)
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
    return f.result()
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/site-packages/prefect/engine.py", line 1114, in get_task_call_return_value
    return await future._result()
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/site-packages/prefect/futures.py", line 237, in _result
    return await final_state.result(raise_on_failure=raise_on_failure, fetch=True)
  File "/home/erikserrano/Programs/miniconda3/envs/pycytominer/lib/python3.9/site-packages/prefect/states.py", line 103, in _get_state_result
    raise MissingResult(
prefect.exceptions.MissingResult: State data is missing. Typically, this occurs when result persistence is disabled and the state has been retrieved from the API.

It seems that the second exception being raised by Prefect is caused by the previous exception thrown by duckdb, which prevents it to change the state of the data.

related to cytomining/pycytominer#79

Thank you for raising this @axiomcura ! This is worth considering for CytoTable. Contextually it's related to conversations in https://github.com/cytomining/pycytominer/issues/198#issuecomment-1140022182 which resulted in work found within sqlite-clean, specifically clean_like_nulls.
"Like nulls" is a reference to data values which look like null-types but are actually strings which have found their way into numeric type columns (SQLite allows this flexibility). The sqlite-clean package was created in part to address some of the data typing challenges within SQLite and also to help with performance (among other things).
CytoTable might need to use sqlite-clean as a dependency or as a code reference to detect or fix datasets where this occurs. Fixes for larger datasets, which requires rewriting modified SQLite data, are not performance optimized. One minimal way we could start with work here would be to raise a specific exception on detection (for example using contains_str_like_null) and warn the user about the potential for data type mismatches.

Hi @d33bs,
@MattsonCam and I are getting this same error when downloading SQLite files from AWS and converting to parquet files.
Code
We are using this file here, but below are the exact convert parameters we are using is: https://github.com/WayScience/JUMP-single-cell/blob/main/0.convert/1.convert.ipynb
%%time
what = cytotable.convert(
    source_path="/".join(manifest_df.sqlite_file[2].split("/")[0:-1]),
    dest_path="test2.parquet",
    dest_datatype="parquet",
    chunk_size=150000,
    parsl_config=parsl_config,
# changed preset to this since compartments don't use prefix, but the CP version is not the same
    preset="cellprofiler_sqlite"
)

Output
The error we are receiving is as follows:
TypeMismatchException: Mismatch Type Error: Invalid type in column "Cells_Correlation_K_DNA_Mito": expected float or integer, found "nan" of type "text" instead.

Solution
There is no solution we can come up with at this time since even if we were to download all SQLite files from AWS onto our local machine, we would still have this error. We will likely have to use pycytominer SingleCells class instead which I notice takes longer to merge single cells than CytoTable.
We hope to see a solution to this and are happy to explain more of the issue!

Hi @jenna-tomkinson , thank you for adding to this issue, and sorry to hear this is giving you and @MattsonCam trouble. This hasn't been yet resolved with code additions. There are some open code changes related to this which seek to resolve the issue in #50. This work hasn't been yet merged into main and is blocked by review needs at the moment, but please feel free to reference in case it helps in the meantime. In addition to Pycytominer, other alternatives might include using sqlite-clean, referred to earlier in this issue.

We just merged #50! 🎉
@MattsonCam - if you're able, please test the newest version and report back if it solves this issue. We can then close it :)

@MattsonCam - were you able to test the newest version? Can we close this issue?

Hi @MattsonCam, @jenna-tomkinson, and @axiomcura - I just wanted to double check on this. Do you know if this issue may be closed (or does the challenge still occur)? I'm also working on validating this as well but it's taking some time to process due to the large data size (will follow up).

I was able to confirm this is now addressed with a completed CytoTable run on SQ00014613.sqlite using modified code based on @axiomcura's original description. See this Google Colab notebook or the related backup Gist. In order to achieve this I had to move to the ThreadPoolExecutor instead of the HighThroughputExecutor (default). I've created #169 to address the HTE issues separately.
Thanks again @axiomcura, @jenna-tomkinson, and @MattsonCam for your help with addressing this issue! Closing it for now. Please don't hesitate to reopen or reach out if you have any questions.


Add Prepare Request support to RequestTracker and OpenTelemetry extension

cc @simaoribeiro
It's a draft PR because I'll have to rebase this when #613 is merged, then I'll remove the draft status

I decided to rename RequestTrackingInfo to SessionRequestInfo and HostTrackingInfo to NodeRequestInfo, I think these names are more appropriate, wdyt @simaoribeiro

I think it makes sense @joao-r-reis . It makes more clear what those objects are

I'm happy that you found the time to support Prepared Statements. This is looking really good

Sneak peek:

These images are looking sweet.
Btw, I just realized that we might have a problem regarding the Keyspace. The keyspace information only appears if its set in the Statement object which means that the db.namespace attribute may not appear in all use cases

Btw, I just realized that we might have a problem regarding the Keyspace. The keyspace information only appears if its set in the Statement object which means that the db.namespace attribute may not appear in all use cases

I already knew this but I thought this is ok since the attribute is only required if it's available and you even added a test that verifies that this attribute is not there when the keyspace is not set in the session.
I didn't test this but it is possible that for Prepared Statements the Keyspace will be there even if it is not set at the session level (but then again it's not that relevant since all we can do is set the attribute if the keyspace is there and not set it if it isn't)

Oh, yeah, I forgot that I included that test. Yes, it's fine if the attribute is available when its set.

Hi @joao-r-reis, we have plan to use this feature on our codebase, do you have a ETA for releasing the 3.22.0?

3.22.0 is going out today


Code clean up to to fix issue #234 - ConnectionsState exposes a setter into internal state

@swift-server-bot test this please

@swift-server-bot test this please

@swift-server-bot add to whitelist


[Feature] Add useTheme hook. Update Link and Alert to use themes

Description of Change

Imports new themes from tokens package
Update all instances of Colors to renamed colors
Adds useTheme() hook which checks for light/dark mode from useColorScheme() and returns the appropriate theme
Updates Link and Alert components to use themes
Ticket to update remainder of the components: https://github.com/department-of-veterans-affairs/va-mobile-library/issues/371

Testing Packages

N/A
Screenshots/Video

Link
Before/After

Before/After (Dark Mode)

Alert
Info Before/After

Info Before/After (Dark Mode)

Warning Before/After

Warning Before/After (Dark Mode)

Error Before/After

Error Before/After (Dark Mode)

Testing

[ ] Tested on iOS
[ ] Tested on Android
[ ] Tested on Web

PR Checklist
Code reviewer validation:

General

[x] PR is linked to ticket(s)
[x] PR has changelog label applied if it's to be included in the changelog
[x] Acceptance criteria:

All satisfied or
Documented reason for not being performed or
Split to separate ticket and ticket is linked by relevant AC(s)

[x] Above PR sections adequately filled out
[x] If any breaking changes, in accordance with the pre-1.0.0 versioning guidelines: a CU ticket has been created for the VA Mobile App detailing necessary adjustments with the package version that will be published by this ticket

Code

[x] Tests are included if appropriate (or split to separate ticket)
[x] New functions have proper TSDoc annotations

Publish

If changes warrant a new version per the versioning guidelines and the PR is approved and ready to merge:

[ ] Merge main into branch
[ ] Merge branch to main
[ ] Verify that Check Component Integrations workflow ran successfully
[ ] Publish new version after merging

Removed redundant logic for button variants in Alert, updated tokens README to include tokens. Ready for review.


Performance enhancements

Following from the issues discussed in #7 it would be worth looking into ways to improve the performance of the parser.
Reducing allocations
In particular with parsing the structured data it was noted:

That sub-parser allocs some little Vec instances, which is where the hunch about allocations generally comes from. An alternate approach to remove any overhead associated with allocation would be to wire in a bump allocator as GlobalAlloc, drive down that cost as low as possible.

I don't know much about allocators so can't really comment, but would be very interested in exploring it. Is that something that would need to be done on Vectors end, or could syslog_loose setup it's own allocator?
There isn't really a strict reason beyond convenience why the structured data has to be stored in a Vec. We could provide an iterator that allows you to iterate over items. There would still need to be an allocation for the iterator and we would need to parse the structured data twice, the first time just to move past it to get to the message and the second time for the iterator to extract each item. So this might not provide a win.
Unless,
Message could implement Iterator itself. The initial parse would just parse up to the start of the structured data and store a pointer to that point. Iterating would return the next structured data pair, and advance the pointer. Once all the structured data is extracted, a final call could be made to return the rest of the message.
It does not make for a pleasant API, but it could be an acceptable trade off if it the performance gains were significant. There are also some caveats around error handling and determining if the message is an RFC5424 or a RFC3164 message.
Parsing multiple lines at once

Another interesting avenue for exploration would be to see if we can't expose something to parse multiple lines at once. We already have a hint from this PR that the larger the line the cheaper the per-byte cost. Does the same hold true if we're parsing multiple lines in one pass?

I can't think of a reason why this would make much of a difference. nom is fairly linear as it parses the text. There may be some caching things going on that could be worth exploring. Performance issues are never predictable!
Optimizing the parser
The way syslog_loose currently works is it attempts to parse the message as a 5424 message. If at any point it can't do this, it will restart the whole parse and attempt to parse it as a 3164. So theoretically, if there was an error at the very end of the structured data, the whole message would be parsed almost twice. This is probably unnecessary as it is generally known in advance what the type of the message is. Simply allowing the caller to indicate the type of the message would potentially almost double the throughput for 3164 messages.
It would be worth creating some benchmarks for 3164 messages to track this..

I don't know much about allocators so can't really comment, but would be very interested in exploring it. Is that something that would need to be done on Vectors end, or could syslog_loose setup it's own allocator?

Today this would be Vector's job. Until allocator-wg wraps up the only transparent mechanism for boxed types -- explicit calls to Box, anything doing heap allocation implicitly like Vec -- will interact with whatever is linked into GlobalAlloc. The exact nature of the allocator is less interesting to me, for this discussion, than exploring whether by removing allocations the library would be faster. That is, per byte of line how many cycles are we spending in our library versus in the allocator? Open question today.

There isn't really a strict reason beyond convenience why the structured data has to be stored in a Vec. We could provide an iterator that allows you to iterate over items. There would still need to be an allocation for the iterator and we would need to parse the structured data twice, the first time just to move past it to get to the message and the second time for the iterator to extract each item. So this might not provide a win.

This general strategy is one that crossed my mind, but with regard to the timestamps. How often is the caller actually using the timestamp? It's parse time shows up in the flamegraph pretty clearly. If we were lazy in parsing fields only when the caller requests them that could be a win. Downside of this approach is if the caller always uses all fields we don't really gain anything, but it does mean something for callers that use a subset.

It does not make for a pleasant API, but it could be an acceptable trade off if it the performance gains were significant.

Agreed. My general notion around questions like this is to err on the side of having straightforward code and only go wacky once there's a clear need to do so. Fast but wrong is useless and the more complicated the program the likelier it is to go wrong, is my thought. I do, however, see how such an approach would allow us to implement a "lazy" API that we could then use to back the current "strict" API.

I can't think of a reason why this would make much of a difference. nom is fairly linear as it parses the text. There may be some caching things going on that could be worth exploring. Performance issues are never predictable!

Cache coherency is exactly what I'm thinking about here. I didn't measure it -- though it'd be straightforward to do with perf -- but I bet the more we buffer and parse in a batch the better we exploit CPU cache. Above all the other things we've discussed so far this would be my next area of investigation. Strikes me that there'd be very little code change needed to expose a batch interface.

It would be worth creating some benchmarks for 3164 messages to track this..

Excellent call out.

How often is the caller actually using the timestamp?

It isn't really Vector's call on which fields get used. It will pull out all the fields, a transform down the road could choose to drop fields if configured to do so. So there isn't really much scope for laziness. We could allow some form of configuration to specify which fields are wanted, but I strongly suspect there would be little advantage gained from it. It is highly likely that the timestamp field would always be wanted. It looks like most of the timestamp parsing time is spent in chrono, which has likely had many eyes on it.
I'll have a bash at the other suggestions above shortly..

It isn't really Vector's call on which fields get used. It will pull out all the fields, a transform down the road could choose to drop fields if configured to do so. So there isn't really much scope for laziness. We could allow some form of configuration to specify which fields are wanted, but I strongly suspect there would be little advantage gained from it.

I buy it.

It is highly likely that the timestamp field would always be wanted. It looks like most of the timestamp parsing time is spent in chrono, which has likely had many eyes on it.

Likely. If timestamp parsing were dominate it'd be worth confirming, but it's not so that seems like a lower priority path to explore.

I'll have a bash at the other suggestions above shortly..

Cool. I'll be curious to see what turns up.


Support returning MultiPart as a server reply

I have a server that needs to return JSON metadata and several related binary blobs (Float32Arrays that are directly passed on to the GPU) in one reply. So far I return the blobs as base64 encoded strings in the JSON metadata, but decoding base64 into an array in the browser is slow.
I am thinking that through the fetch API I can interpret a response as formData() which could allow me to get the binary data directly as arrays. For this I need to return multipart/* in my server.
I tried doing this by using multipart::client::Multipart, but I cannot construct it without a request - which I do not have, since this is the reply in the server. Why can I not create a Multipart, add my binary data to it and ask it for the encoded body as a string?

How about an adapter that implements Read? With multipart::client::lazy::Multipart you can build your multipart body then call .prepare() which returns a type that implements Read, that you should then be able to return as your response body, or copy out to your response depending on what HTTP framework you're using.

That might work, but it still sounds complicated. Could you explain why tying the construction of the body of the HTML request is so strongly tied to a Request or a Client? Is that because of streaming?

It's primarily to ensure that the right header is supplied with the request. The multipart boundary that separates each field in the body has to be supplied in the Content-Type header of the request, and each HTTP framework does headers slightly differently. This approach has better ergonomics for the common case where you're constructing the multipart body in the process of submitting a request.
I've done a different approach in src\local_test.rs where I have a mock implementation of HttpRequest that simply writes the request to a Vec and stores the boundary. I could extract this to the library portion but I didn't see the utility in it originally.

Thanks for explaining everything!
I took a different approach for my problem now, so this is not required for me anymore and I think my use case was rather esoteric. I would suggest you do not pull the code out until the next one with a similar problem comes around. Closing the issue for now.


Support mocking constructor in mockImplementationOnce

Summary
Mocking constructor wasn't supported inside mockImplementationOnce.
Fixes #4595.
Test plan
Added a test with mocking constructor.

Codecov Report

Merging #4599 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #4599      +/-   ##
==========================================
+ Coverage   56.16%   56.17%   +<.01%
==========================================
  Files         194      194
  Lines        6543     6544       +1
  Branches        3        3
==========================================
+ Hits         3675     3676       +1
  Misses       2867     2867
  Partials        1        1

Impacted Files
Coverage Δ

packages/jest-mock/src/index.js
84.96% <100%> (+0.05%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d175f39...af518bb. Read the comment docs.

Good fix! Thanks.


Unable to add Google drive sync plugin to iPhone due to missing file?

hi - NON-coder here trying to add the Google drive sync plugin to my iPhone so I can stop spending $5/month to sync across devices.
I’m getting an error when trying to upload the sync plugin through BRAT, but the plugin works fine on my desktop. Checking in with you to see if this error is something on the BRAT side or if I need to ask the Google sync plugin owner for the file type specified in the error…?
BRAT is looking for a main.js file and the sync plugin has a main.ts file?
Thanks for any help you can provide on this.

I just tested and this is working. Likely the developer fixed the release.


More examples

The more examples we can create, the better, both in terms of helping people understand the capabilities of SvelteKit (and having useful references) and in terms of our own dogfooding.
Application Holotypes, while written as a guide for when not to use an SSR framework, contains some good inspiration for the sorts of things it would be interesting to build.
If it doesn't naturally fall out of those examples, then it would also be good to focus on page transition examples.
Also: an example of a component library

If it doesn't naturally fall out of those examples, then it would also be good to focus on page transition examples.

That looks awesome and would definitely make me go "wow!"

Thinking about good examples, something that every developer might find useful and take immediately as starter point to develop something and evetually put in production, I think the following would be great additions:

blog
portfolio page
presentation

I think it would be pretty useful to have a wel organized, with sane defaults, great looking blog/presentation/portfolio developed with SvelteKit that you could just fork and deploy to vercel with one command! It would be a great way to help developers give their first steps with SvelteKit with something real.
A big plus would be to support https://mdsvex.com/ out of the box.

Agree with OOTB mdsvex support (though I think we should use .svelte.md as the extension), and I love the idea of a SvelteKit presentation framework — it would be a really good opportunity to think deeply about page transitions

I think the main issue with transitions is https://github.com/sveltejs/kit/issues/628

A SvelteKit based presentation framework would really boost Svelte difussion and adoption. If you could prepare one of your breath-taking presentations for the launch of SvelteKit, and include the presentation project in the SvelteKit example folder, it would let us all shamelssly steal it away from y... I mean get insparion from your hard work. It would be a great starter for everyone willing to showcase Svelte

I've put together several examples of using SvelteKit with different GraphQL clients, so the exact project and endpoint but utilising different GQL clients for each one, is that something that would be of interest?

i am committing to maintaining https://github.com/sw-yx/swyxkit/ for the time being - happy to donate it eventually as i probably wont maintain it long term - but it should serve the blog usecase, and i'd like it to serve the dashboard usecase after 1.0 is baked.

I nominate https://skeleton.dev for a component example.
:)

Should we organize those examples or demos on some page? ie: https://kit.svelte.dev/showcase

SvelteKit + Supabase + Stripe example :)
https://github.com/supabase-community/sveltekit-subscription-payments

@Rich-Harris I know we commissioned some examples; does this issue still need to be open?


Store hashing algorithm in the metadata table

mbtiles and martin-cp standardize how a tile data is hashed in the mbtiles. tilelive-copy has been generating normalized mbtiles schema with MD5 hashes of the tile data as the table foreign key.  So it was possible to validate the content of the tile by re-computing MD5 hash.
tippecanoe (cc @bdon) has used a faster fnv1a checksum algorithm, so clearly there is need for more than one algo.
Proposal

Add hash_algorithm metadata value with the possible values md5, fnv1a, ...
If the value is not there, check if agg_tiles_hash metadata value is present, and use its length to decide algorithm
If not, and mbtiles uses normalized or flat-with-hash schema, use the size of the hashing keys to decide the algorithm
If the hashing algorithm is detected, add it to the metadata, possibly after validating that all hashes match their data

Multiple Algorithms Consideration
MBTiles use hashes for the tile_data column and for the agg_tiles_hash metadata field.  In theory these could use different algorithms for performance and security(?) reasons.  If so, we could either have two metadata fields (hash_algorithm and agg_hash_algorithm), or better yet, rename agg_tiles_hash into agg_tiles_hash_md5 (or other algo).  This keeps the algorithm and its value together, and is fairly easy to detect at runtime.

faster fnv1a checksum algorithm

In tippecanoe we needed a content hash that had the smallest implementation to inline, unsure how the speed compares to md5, but if you need speed you'll probably want something like xxhash anyways.

@bdon this proposal is to support and standardize all such algorithms, so that if tippecanoe created mbtiles file, other tools can validate it and possibly add more content, while using the same algo as the original


cargo-bloat: 0.6.3 -> 0.6.4

Motivation for this change
Update cargo-bloat to the latest release.
https://github.com/RazrFalcon/cargo-bloat/releases/tag/v0.6.4
I want to add another useful cargo subcommand cargo-make into nixpkgs.
Can anyone review #61265, if possible?
Thank you!
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Assured whether relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

@GrahamcOfBorg build cargo-bloat

it was a bump release: https://github.com/RazrFalcon/cargo-bloat/compare/v0.6.3...v0.6.4, tested on darwin and NixOS nonetheless
Thanks!


Brown noise stops after ~3h

Description: Brown noise stops playing after having it on non-stop for about 3h continuously. Switching to white or pink noise restarts that type of noise, and I can then switch back again to brown noise. I don't know if this is specific to brown noise, as I haven't tested the other noise types.
Steps to reproduce:

Disable "wave" and "volume" icons.
Unset timer.
Choose Brown noise.
Click play.

Expected behavior: Brown noise continues playing non-stop even after 3h.
Actual behavior: Brown noise stops playing after ~3h.
App version: 1.14
Device: Surface Duo, Android 10

Thanks for the detailed bug report! This should be fixed in the newest version (2.0). Let me know if you get a chance to try it.
I'm not sure what exactly what was causing the issue but I was able reproduce in 1.14 as well. 2.0 has some audio library updates and playback in 2.0 seems much more reliable now.


[map] add more color contrast

https://autopush.datacommons.org/tools/map#%26sv%3DCount_Person_PerArea%26pc%3D0%26pd%3DEarth%26pn%3DEarth%26pt%3DPlanet%26ept%3DCountry

In this chart, low density, no data and the tool background colors are all too similar

In the tool, we could have a uniform, diverging, color scheme. TBD on what to do for heavily skewed data.

https://datacommons.org/tools/map#%26sv%3DCount_Person_Employed%26pc%3D1%26denom%3DCount_Person%26pd%3Dcountry%2FUSA%26ept%3DCounty%26domain%3D8106479329266892%3A4053239664633446%3A0

Please also verify fix with the screenshots in the duplicate bugs