feat: add labelConsistency validator to validation-gen

What type of PR is this?
/kind feature

What this PR does / why we need it:
This PR adds a labelConsistency validator to validation-gen.  This validator is to address the ~30 validations in the currentl kubernetes validation codebase which attempt to validate a label (or annoatation) with a field value.  The syntax is still WIP, currently it is used like the following:
type JobExample struct {
	TypeMeta int `json:"typeMeta"`

	// +labelConsistency={"label": "job-name", "required": true}
	Name string `json:"name,omitempty"`

	// +labelConsistency={"label": "controller-uid", "required": true}
	UID string `json:"uid,omitempty"`

	Template *PodTemplate `json:"template,omitempty"`
	Metadata ObjectMeta   `json:"metadata,omitempty"`
}

Which issue(s) this PR fixes:

This is part of the conditional cross field validation TODO.
Special notes for your reviewer:
This required field is added currently as there are some annotation checks that are conditional on whether the label exists from my analysis (it isn't needed for the label consistency per-se).  Still trying to understand the best way to go about this aspect of the validations.
Currently this PR is still WIP with the following changes desirable:

extend the syntax to support annotations
potentially use function chaining and or an extended syntax s.t. the current "required" field is potentially not needed and so that a more robust when X > Y, etc. syntax might be possible (need to see real world cases to identify the exact # of things this would be needed by)

Does this PR introduce a user-facing change?
N/A

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

/assign @yongruilin
Let me know when it looks good to you and I'll make a pass as well.

@aaron-prindle  LGTM in general, left some comments so that I can better understand the code.

@yongruilin you mentioned:

left some comments so that I can better understand the code

I don't currently see any comments in the PR currently, feel free to ask anything in the PR here.  Wanted to make sure they might not be queued as an unsubmitted the code review to make sure didn't miss anything, thanks

It feels like there is an unsolved problem brought up by this PR:  How, in general, do we get access to values at different fields paths in the object?
My intuition is that the validator should be generated at the root of the object for a case like this, even if the field being validated is deeply nested. That way the validator has access to dereference both the labels and the field.  But that diverges significantly from how we've been doing things up to this point, where we put tags directly on the field being validated and then generate a validator at the the corresponding location.  Should we instead be either (a) defining the tag to be at the root and then referring to the field with a path, or (b) modify the labelConsistency validator plugin to search more deeply for the +labelConsistency tag and then generate a validator in the root of the generated code for the a +labelConsistency tag found at some field path?

We will need some way to reference fields.  We can use field path syntax, but we will need a parser for it, decomposing into something like field.Path.

It feels like there is an unsolved problem brought up by this PR: How, in general, do we get access to values at different fields paths in the object?
+1, when trying to reference two fields in a clause for comparison (as is done for label consistency and would be needed for the other identified patterns of conditional cross field validations - eg: field presence, value dependant constraints, etc.) currently there is no built-in way to do this.

We will need some way to reference fields. We can use field path syntax, but we will need a parser for it, decomposing into something like field.Path. then, given a field.path, use reflect to walk the object? It will be slow, so it shouldn't be the most common thing we do.

+1 to having some parser  -> field.Path -> reflect walk to resolve the field path - sounds like a good solution here.   Would it make sense to re-use any CEL libs for this or is that likely not necessary?  Not sure if we might have more enhancements to do for the parser over time.

My intuition is that the validator should be generated at the root of the object for a case like this, even if the field being validated is deeply nested.
+1, this makes sense to me.  Would this mean that the validation tag would be root level as well (or ?  My initial implementation had an approach using a root struct tag which I changed to a field level tag to be more inline with the other tags and the UX seemed a bit better.  This looked like:

// +labelConsistency={"label": "app", "field": "SelectorMatchLabel", "required": true}
type ReplicaSetExample struct {
	TypeMeta int `json:"typeMeta"`
	// SelectorMatchLabel represents the expected value for template labels
	SelectorMatchLabel string       `json:"selectorMatchLabel,omitempty"`
	Template           *PodTemplate `json:"template,omitempty"`
	Metadata           ObjectMeta   `json:"metadata,omitempty"`
}

Should we instead be either (a) defining the tag to be at the root and then referring to the field with a path, or (b) modify the labelConsistency validator plugin to search more deeply for the +labelConsistency tag and then generate a validator in the root of the generated code for the a +labelConsistency tag found at some field path?

a) Discussion
I initially implemented the solution here using a) although it didn't support field paths (only direct child fields).  I have an example above showing the tag format for the root level // +labelConsistency tag.   Going down this path then I thought the UX might be an issue as for each conditional cross field validation on a struct you might have N root level validations that might be hard to discern so I pivoted to this field level tag (additionally the other tags were field level).
b) Discussion
I think this might be the better approach from a UX standpoint (can look at a field validation tag directly and understand how it is validated vs. potentially looking at N root level validation tags).

I'm still not clear which cases this one is handling.
Is it checking that a deployment selector actually selects pods that its own template produces? Or is it checking that one field matches a known label (e g used to be a label, became a real field)? Or something else?

The "labelConsistency" validator plugin is attempting to handle validations where objects needed to specify that a selector matches a label, like the suggested example here- deployment selector actually selects pods that its own template produces.
Below are some example of k8s code from the validation logic survey:

For ReplicaSet - validate the pod template spec of a ReplicaSet and ensure that the the ReplicaSet selector matches the labels in template
For batch Jobs, if selector generation was requested, then expected labels must be present on pod template, and must match job's uid and name.

From k8s.io/kubernetes/pkg/apis/apps/validation/validation.go
// if template == nil {
//     allErrs = append(allErrs, field.Required(fldPath, ""))
// } else {
//     if !selector.Empty() {
//         labels := labels.Set(template.Labels)
//         if !selector.Matches(labels) {
//             allErrs = append(allErrs, field.Invalid(fldPath.Child("metadata", "labels"), template.Labels, "`selector` does not match template `labels`"))
//         }
//     }
// }

From k8s.io/kubernetes/pkg/apis/batch/validation/validation.go
// allErrs = append(allErrs, apivalidation.ValidateHasLabel(obj.Spec.Template.ObjectMeta,
//     field.NewPath("spec").Child("template").Child("metadata"),
//     batch.ControllerUidLabel, string(obj.UID))...)
// allErrs = append(allErrs, apivalidation.ValidateHasLabel(obj.Spec.Template.ObjectMeta,
//     field.NewPath("spec").Child("template").Child("metadata"),
//     batch.JobNameLabel, string(obj.Name))...)

@aaron-prindle Let me know if you'd like another review pass on this.  I remember you were looking into how to handle references to nested fields. If you decide you'd like to handle that in a separate PR let me know, otherwise I'll set this one aside until you ping me.  Thanks!


Training error

Hello, Thank you for the code!
I have tried training with the ped2 dataset,  but I got this error with the default pred2_wresnet.yaml.
python train.py \
    --cfg config/ped2_wresmet.yaml

RuntimeError: Given groups=1, weight of size [2048, 16384, 1, 1], expected input[2, 8192, 28, 36] to have 16384 channels, but got 8192 channels instead
Could you provide me with ped2_wresnet.yaml that works for the training of UCSped2? (http://www.svcl.ucsd.edu/projects/anomaly/dataset.html)

Hi,
Thanks for your interest!
Please, check this line


The epinio install command should quickly fail if the environment doesn't fulfill the requirements

When I was installing Epinio on RKE2 and RKE, I got the error message:
error installing Epinio: timed out waiting for LoadBalancer IP on traefik service
Ensure your kubernetes platform has the ability to provision a LoadBalancer IP address.

The error came at the end of the installation process. I think the user experience would be better if Epinio would quickly fail if the installation requirements aren't fulfilled because it would save some time.

The ability to assign an IP to load balanced services is most of the time supplied by non-kuberentes components which Epinio can't inspect. In other words, Epinio can't know if a LoadBalanced service will get an IP address unless it creates one and waits to see what happens. You are right, the UX is not good here but that's merely because the cluster doesn't support LoadBalanced services properly. I can't think of a way to solve this.

The new way to deploy Epinio is using the helm charts here: https://github.com/epinio/epinio-helm-chart
There no waiting happening anymore for the service to get an IP address. This issue can be closed.


what does -1 mean in strides and groups variable

Hi, thanks for your nice work. I'm very confused with the negative values (-1) in strides and groups. For example, strides=[-1,-1,1,1] or groups=[-1.-1.3,6]. This also triggers errors when creating weight tensors due to that negative dimensions are not allowed in tensors:
RuntimeError: Trying to create tensor with negative dimension -96: [-96, 1, 9, 9]
It would be appreciated if you can explain what does -1 mean here.
Thanks in advance!

This stride controls the stride of the offset generation module and only takes effects at the last two stages. I set it to a negative number as default for debugging since an error will be triggered if it is not set properly.


The macOS instructions are empty

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 29dd905c-43f7-9045-4eee-c087e27f48e0
Version Independent ID: 0e52d919-2745-c8c5-eb12-77b564f5e3d1
Content: Connect your SQL Server instance using Windows authentication (Kerberos) - Azure Data Studio
Content Source: docs/azure-data-studio/enable-kerberos.md
Product: azure-data-studio
Technology: azure-data-studio
GitHub Login: @markingmyname
Microsoft Alias: maghan

@istandleet -- Robert, thank you for bringing this to our attention.  Please consider these resources:

SQL Server on Microsoft Q&A
DBA Stack Exchange
Stack Overflow

@markingmyname -- Mark, please look into this issue.

@istandleet
Apologizes for the delayed response.
We've seen multiple pieces of feedback related to this section.
We apologize for leaving it blank and not being clear on the section.
We have made changes to the article.
Changes are scheduled to appear live by tomorrow.
Thank you.

We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion, and we will reopen the issue. To see what the community is wanting, please visit Azure Data Studio repo(https://github.com/microsoft/azuredatastudio)


Composer: architecture refactor

This PR:

Splits Searcher into three components:

Collector (collects transactions from multiple rollups and funnels them into one stream)
Bundler (bundles multiple transactions into a bundle that can be turned into one sequencer tx using MEV strategies)
Executor (executes bundles - passing them into block builders or to the sequencer directly)

Also addresses #246

Tiny bit of work left in this refactor, but the core logic is ready to review

UPDATE: synced with @joroshiba and splitting bundler + strategy into a different PR


Fixed file permission issues for camel-amq quickstart

Due to file permission issues on some docker images, I removed the routes-to-files from the camel-amq quickstart. I also fixed sample files to comply with xpath expressions defined in the routes.

ok to test

[merge]


YAML Configuration Support (GSoC 2020)

YAML is quite popular in the configuration management space nowadays (especially in the Kubernetes domain), and it would be great to have support for YAML configuration files in WinSW.
Jenkins project has published a GSoC 2020 project idea for this project: https://jenkins.io/projects/gsoc/2020/project-ideas/winsw-yaml-config-support/ (and also for better config file validation in #196 ). If anyone is interested to work on this project as a student or as a mentor, please let us know!

For whoever is interested in why there is no YAML support in .NET core libraries: dotnet/runtime# 28414

@oleg-nenashev I am interested to work on this project as a student in GSoC2020. I am a final year undergraduate of University of Moratuwa, Sri Lanka and I did my internship as a trainee .Net engineer.

Some links from the meeting:

Recording: https://youtu.be/on4erbebd0E
Meeting notes: https://docs.google.com/document/d/1YomPI1J4ZnpdmdEyYDj6t2wXY5mVISwZG05uxSajquM/edit?usp=sharing

Just an update here, the project has been accepted to GSoC. Congrats @buddhikac96!
@NextTurn, @mikecirioli and me will be the mentors in this project. If anyone else is interested to join as a co-mentor or a stakeholder, please let us know! We also plan to have a project kick-off meeting soon, here is a Doodle link for voting if you are interested to join the meeting.

I have started testing a few services with Yaml today, and even ported an extension to support the new format. Both without much effort. Nice job, thanks! 👍

Thanks for the feedback!
On Fri, Aug 28, 2020, 21:36 offa<EMAIL_ADDRESS>wrote:

I have started testing a few services with Yaml today, and even ported an
extension to support the new format. Both without much effort. Nice job,
thanks! 👍
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/winsw/winsw/issues/436#issuecomment-683109399, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAW4RIB6ZGHZEFLZGIOLMOTSDABL5ANCNFSM4LFSHM7A
.


Add missing dotnet8 bootstrap support copy command

Fixes local issue when running on dotnet8.
This change should have been included in #3734.

I see, yeah I'll tell the team.

Hey @jayair, just nudge on this 🙏

Hmm might've slipped through the cracks. Will ping again.


helperClass doesnt work. it doesnt show any class on conatiner in the browser

What browser? The helper class is applied to the clone that is being dragged, not the container.

I'm going to change the stylesheets of element dragging. But I added helperClass, but I can't see the changes.


docs of dsl_ode_trigger_custom_new is missing parametes

I quote this constructor in my code to get the meta in buffer ,but the IDE reminds me that the constructor is missing two parametes:"source" and "client_post_processor".
retval = dsl_ode_trigger_custom_new('my-custom-trigger',         DSL_ODE_ANY_CLASS, DSL_ODE_TRIGGER_LIMIT_NONE, my_check_for_occurrence_cb, my_client_data)
so I add my own source into it just like other constructors ,but I did't find any methods to create a "client_post_processor" callback.
finally I find a declaration of it in the source code of "dsl.py".
# dsl_ode_check_for_occurrence_cb DSL_ODE_CHECK_FOR_OCCURRENCE = \     CFUNCTYPE(c_bool, c_void_p, c_void_p, c_void_p, c_void_p)
but I still don't know how to use it ,because the meaning of parametes in this function is unknown to me,maybe you guys should update the docs since you have already update the code.

@master-LIN777 thanks for bringing this to my attention... and it looks we need an example for using the custom-trigger... give me a couple of days I'll try and take care of this.

My apologies for the very late response.  After reviewing my notes, I was able to recall what happened with the Trigger.  Unfortunately, I was only able to support this with C++ (not python).  One of only two API that have this issue.  It has to do with the callback parameters  frame_meta and object_meta which can not by mapped using the NVIDIA python bindings.
typedef boolean (*dsl_ode_check_for_occurrence_cb)(void* buffer,
    void* frame_meta, void* object_meta, void* client_data);

New DSL mappings will need to be created for these types.  This will need to be addressed in a future release.  The documentation has been updated to reflect this.


np.prod does not work with jacfwd

This example illustrate this issue.
np.prod works with jacrev but not with jacfwd. Both work and give the same result when explicitly code the multiplication.
def func(x):
  x = x.reshape((2,3))
  a = np.array([[2.,2.,2.],[3.,3.,3.]])
  o = np.exp(-(a-x)**2)
#   o = o[:,0] * o[:,1] * o[:,2]
  o = np.prod(o,axis=1).flatten()
  return o
jrfunc = jacrev(func)
jffunc = jacfwd(func)

I'm testing this in colab. The built-in version appears to be 0.1.44

This seems to work with jax version 0.1.46, so I'm guessing whatever the  issue was has already been fixed.
You can upgrade your jax release in colab by  running  pip install --upgrade jax  in a cell.
Hope  that helps!


TOML incorrect duplicate key error for tables nested within an array of tables

What version of CUE are you using (cue version)?

$ cue version
cue version v0.11.0

go version go1.23.3
      -buildmode exe
       -compiler gc
       -trimpath true
     CGO_ENABLED 0
          GOARCH amd64
            GOOS linux
         GOAMD64 v1
cue.lang.version v0.11.0

Does this issue reproduce with the latest stable release?
Yes.
What did you do?
# eval cue example
exec cue eval example.cue

# eval equivalent toml example
exec cue eval example.toml
-- example.cue --
blocks: [
	{
		x: 42
		properties: {
			style: "plain"
		}
	},
	{
		x: 100
		properties: {
			style: "special"
		}
	},
]
-- example.toml --
[[blocks]]
x = 42

[blocks.properties]
style = 'plain'

[[blocks]]
x = 100

[blocks.properties]
style = 'special'

What did you expect to see?
Both cue eval commands run successfully with no errors.
What did you see instead?
$ cue eval example.toml
duplicate key: blocks.properties:
    ./example.toml:10:2

This is incorrect according to the TOML spec (v1.0.0), which states:

Any reference to an array of tables points to the most recently defined table element of the array. This allows you to define sub-tables, and even sub-arrays of tables, inside the most recent table.

This issue has the undesirable effect of breaking, for example, the cue eval and cue vet commands for TOML files containing such data.

Thanks for reporting. It's entirely possible that I got the logic for arrays of tables slightly wrong. I'll take a look when I can; you're more than welcome to try debugging our code in the meantime.


Code improvements

Subject: Component documentation, 2.3 Example
${config_string}=    catenate
...                  {
...                  "address": "<IP_ADDRESS>",
...                  "port": 8022,
...                  "username": "root",
...                  "password": "",
...                  "authentication": "password",
...                  "key_filename": null
...                  }

${config}=    evaluate    json.loads('''${config_string}''')    json

connect    conn_name=test_ssh
...        conn_type=SSHClient
...        conn_conf=${config}

This code is very long winded and does not work. The 'evaluate' line causes a JSONDecodeError. Seems that the dollar operator is not resolved when used within a Python expression.
A version without dollar operator expression like
${config}=    evaluate    json.loads("""{"address": "<IP_ADDRESS>", "port": 8022, "key_filename": null}""")    json
works like expected.
But nevertheless, for me it completely makes no sense to define values in JSON format and convert them to a dictionary using the 'evaluate' keyword, because of a keyword requires a dictionary. You better define the values in dictionary format immediately. Or in other words: You should prefer using Robot Framework core functions. The keyword 'evaluate' is more for complex things that cannot be realized by existing Robot Framework keywords.
The connection parameters depend on the connection type. Therefore I do not understand why this is separated:
...        conn_type=SSHClient
...        conn_conf=${config}

In my opinion this belongs together and should not be splitted in 2 separate parameters.
Therefore I would prefer this solution (that is also adapted to our style):
Library    QConnectBase.ConnectionManager    WITH NAME    conn_manager

${HOST}=    localhost
${PORT}=    1234

&{TCPIPClientParam}    conn_type=TCPIPClient
...                    address=${HOST}
...                    port=${PORT}

conn_manager.connect    conn_name=Test-Connection
...                     conn_conf=${TCPIPClientParam}

Addendum:
Instead of defining connection parameters hard coded within tests, it would also be a nice solution to read parameters from JSON files. How this can be done, depends on the availability of the Testsuites Management. Maybe this is a good occasion to make a bit advertising. You should mention in the component documentation that another component called Testsuites Management, is able to provide parameters read from external JSON files.
But nevertheless, QConnectBase should provide such a functionality also.
That should be realized in this way:
${conn_conf_file}=    ./TCPIPClientParam.json

conn_manager.connect    conn_name=Test-Connection
...                     conn_conf_file=${conn_conf_file}

With content of TCPIPClientParam.json:
{
    "conn_type" : "TCPIPClient",
    "address"   : "localhost",
    "port"      : 1234
}

As part of the documentation it's O.K. if the values are hard coded.
How to use json files for configuration is another topic.

Hi Thomas,
OK, but why not mention that a so called "Testsuites Management" is available, that can do the job?
And it has to be decided if QConnectBase shall provide a conn_conf_file parameter or not.

B.t.w.:
evaluate    json.loads returns a dict only, and not a dotdict.
But
&{TCPIPClientParam}    conn_type=TCPIPClient
...                    address=${HOST}
...                    port=${PORT}

returns a dotdict.
Therefore we should not recommend using evaluate    json.loads


Mock Service default works only with VSS 3.1.1

Mock Service default mock.py works only with the VSS 3.1.1 VSS 4.0 creates the following error:
ERROR:base_service:Failed to register datapoints
Traceback (most recent call last):
  File "/home/imu6fe/kuksa.val.services/.venv/lib/python3.8/site-packages/lib/baseservice.py", line 106, in _on_broker_connectivity_change
    self.on_databroker_connected()
  File "mockservice.py", line 74, in on_databroker_connected
    loader_result = PythonDslLoader().load(self._vdb_metadata)
  File "/home/imu6fe/kuksa.val.services/.venv/lib/python3.8/site-packages/lib/loader.py", line 140, in load
    mocked_datapoints = self._load_mocked_datapoints(vdb_metadata)
  File "/home/imu6fe/kuksa.val.services/.venv/lib/python3.8/site-packages/lib/loader.py", line 65, in _load_mocked_datapoints
    metadata = vdb_metadata[datapoint["path"]]
KeyError: 'Vehicle.Cabin.Seat.Row1.Pos1.Position'

After further investigation the only thing thats not working with VSS 4.0 are the examples in mock.py. iw ould suggest we write this in the README and solve this issue through this. PR will follow shortly.

will be fixed by #95

@lukasmittag  is this issue already fixed? As merge of #95 is already merged

It is


When disk is full, after expanded disk ood-daemon run panic

Describe the bug
When disk is full, after expanded disk ood-daemon run panic,  And the ood-daemon monitor process will return to pull up and repeatedly throw this exception
[2023-04-15 08:51:31.967738 +00:00] INFO [ThreadId(1)] [component/cyfs-debug/src/panic/panic.rs:114] stack_hash=
c04118763b3f5f6d1046c0ef5df13379
[2023-04-15 08:51:31.967811 +00:00] INFO [ThreadId(1)] [component/cyfs-debug/src/panic/panic.rs:31] thread 'main' panicked at 'assertion failed: `(left != right)`
  left: `Some(ThreadId(1))`,
 right: `Some(ThreadId(1))`': /mnt/e/BuildAgentWork/4a71081484236186/src/component/cyfs-debug/src/check/reentry_checked_mutex.rs:29
0: 0x0000562f87422470
1: 0x0000562f87417400
2: 0x0000562f873afe50
3: 0x0000562f87986990
4: 0x0000562f879868b0
5: 0x0000562f87985000
6: 0x0000562f87986620
7: 0x0000562f86b5e3f0
8: 0x0000562f879abf80
9: 0x0000562f86b2d690
10: 0x0000562f86b93bc0
11: 0x0000562f86c5e590
12: 0x0000562f86c8c250
13: 0x0000562f86c5b8d0
14: 0x0000562f86c8f600
15: 0x0000562f86c97450
16: 0x0000562f86ca4880
17: 0x0000562f86cab0a0
18: 0x0000562f86c6ac20
19: 0x0000562f86bc3890
20: 0x0000562f86cab8c0
21: 0x0000562f86cabf20
22: 0x0000562f86cab780
23: 0x0000562f86ba20f0
24: 0x0000562f86cf8ce0
25: 0x0000562f86caaed0
26: 0x0000562f86b80210
27: 0x0000562f879744e0
28: 0x0000562f86cf8f50
29: 0x00007ffbe7522d10
30: 0x00007ffbe7522dc0
31: 0x0000562f86b5ed50
32: 0x0000000000000000
[2023-04-15 08:51:31.967867 +00:00] INFO [ThreadId(1)] [component/cyfs-debug/src/panic/panic.rs:32] thread 'main' panicked at 'assertion failed: `(left != right)`
  left: `Some(ThreadId(1))`,
 right: `Some(ThreadId(1))`': /mnt/e/BuildAgentWork/4a71081484236186/src/component/cyfs-debug/src/check/reentry_checked_mutex.rs:29
   0: cyfs_debug::panic::manager::PanicManager::start::{{closure}}
   1: std::panicking::rust_panic_with_hook
   2: std::panicking::begin_panic_handler::{{closure}}
   3: std::sys_common::backtrace::__rust_end_short_backtrace
   4: rust_begin_unwind
   5: core::panicking::panic_fmt
   6: core::panicking::assert_failed_inner
   7: core::panicking::assert_failed
   8: cyfs_debug::check::reentry_checked_mutex::ReenterCheckedMutex<T>::lock
   9: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  11: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  12: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  13: ood_daemon::daemon::daemon::Daemon::run_check_update::{{closure}}
  14: ood_daemon::main_run::{{closure}}
  15: std::thread::local::LocalKey<T>::with
  16: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  17: async_io::driver::block_on
  18: std::thread::local::LocalKey<T>::with
  19: std::thread::local::LocalKey<T>::with
  20: std::thread::local::LocalKey<T>::with
  21: async_std::task::builder::Builder::blocking
  22: ood_daemon::main
  23: std::sys_common::backtrace::__rust_begin_short_backtrace
  24: std::rt::lang_start::{{closure}}
  25: std::rt::lang_start_internal
  26: main
  27: <unknown>
  28: __libc_start_main
  29: _start

ood-daemon_3872228_rCURRENT.log

This panic is caused by the reentry detection of Mutex in cyfs-debug. In the code, after entering the error branch, it will try to reenter this Mutex. Therefore, even if ReenterCheckedMutex is not used here, it would still lead to a deadlock.
https://github.com/buckyos/CYFS/blob/314d358d1bfbd52cdcd66c7775fe2d44b2903577/src/service/ood-daemon/src/service/service.rs#L572-L575
Besides this bug, it may also be necessary to review other similar logic in ood-daemon service to avoid similar problems.

This issue has been fixed in commit 1f7689843cfe86f9c6da9a3e8c97bed6e8024c2f
Another thing to note is that under extreme conditions where the disk containing the {cyfs} directory is full, there may be many problems. These situations may require constructing similar extreme environments for testing to see how the services perform under extreme conditions.

This issuse has been tested and verified in version <IP_ADDRESS>1.


Fix another ALPC parameter type error and add support for vs2019.

I missed another ALPC parameter type error, and hope that phnt can continue to support vs2019.

Would it be possible to separate the changes into two commits - one for the fix and another one for the vs2019 support?

Ok, i will break up the commit


Implement fast min/max accumulator for binary /  strings (now it uses the slower path)

Is your feature request related to a problem or challenge?
https://github.com/apache/arrow-datafusion/pull/6904 introduces some fancy new hashing and ways to implement aggregates
See related blog post: https://datafusion.apache.org/blog/2023/08/05/datafusion_fast_grouping/
min/max for strings (StringArray / LargeStringArray, etc) now uses the slower Accumulator implementation which could be made much faster
Describe the solution you'd like
I would like to implement a fast GroupsAccumulator for Min/Max
Describe alternatives you've considered
here is one potential way to implement it:
We could store the current minimum for all groups in the same Rows 🤔 and track an index into that Rows for the current minimum for each group.
This would require an extra copy of the input values, but it could probably be vectorized pretty well, as shown in the following diagram.
Sorry what I meant was something like the following where the accumulator only stored the current minimum values.
This approach would potentially end up with min_storage being full of "garbage" if many batches had new minumums, but I think we could heuristically "compact" min_storage (if it had 2*num_groups, for example) if it got too large
                                    ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐

                                    │           Accumulator             │
                                                state
┌─────────┐       ┌─────────┐       │ ┌─────────┐          ┌─────────┐  │
│ ┌─────┐ │       │ ┌─────┐ │         │ ┌─────┐ │          │ ┌─────┐ │
│ │  A  │ │       │ │  A  │─┼────┐  │ │ │  D  │ │    ┌─────┼─│  1  │ │  │
│ ├─────┤ │       │ ├─────┤ │    │    │ ├─────┤ │    │     │ ├─────┤ │
│ │  B  │ │       │ │  B  │ │    └──┼─┼▶│  A  │◀┼────┘     │ │  0  │ │  │
│ ├─────┤ │       │ ├─────┤ │         │ └─────┘ │          │ └─────┘ │
│ │  A  │ │       │ │  A  │ │       │ │         │          │         │  │
│ ├─────┤ │       │ ├─────┤ │         │         │          │         │
│ │  A  │ │       │ │  A  │ │       │ │         │          │         │  │
│ ├─────┤ │       │ ├─────┤ │         │         │          │         │
│ │  C  │ │       │ │  C  │ │       │ │         │          │         │  │
│ └─────┘ │       │ └─────┘ │         │         │          │         │
└─────────┘       └─────────┘       │ └─────────┘          └─────────┘  │

  input              input          │ min_storage:         min_values   │
  values             values           Rows
  (Array)            (Rows)         └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘
          step 1:           step 2: for
          convert           any value                   step 3: min value
          arguments to      that is a new               (per group) is
          Row format        group                       tracked as an
                            minimum, copy               index into
                            it to a                     min_storage `Rows`
                            second `Rows`

See https://github.com/apache/arrow-datafusion/pull/6800#issuecomment-1622290981 for more details
Additional context
No response

One observation here is that min and max on strings is not that common of an operation from what it seems -- grouping on strings is more common.
Maybe there is some binary usecase where it is important (e.g. embeddings 🤔 )

@alamb is there anyone working on this + is this issue still relevant? I would love to tackle it as it seems like an interesting feature/optimization.

@alamb is there anyone working on this + is this issue still relevant? I would love to tackle it as it seems like an interesting feature/optimization.

I dont know of anyone working on this @devanbenz  -- but I also don't know of any benchmarks or actual queries that use min / max on string columns. The place it shows up is when computing statistics when writing parquet, but I think parquet is already pretty good at this (and has its own code to compute min/max)

It actually turns out that Min / Max on string/binary columns are in several ClickBench queries:
https://github.com/apache/datafusion/blob/a08f923c2acb1a46614970231d9a672c36ce3ad2/benchmarks/queries/clickbench/queries.sql#L22-L23
https://github.com/apache/datafusion/blob/a08f923c2acb1a46614970231d9a672c36ce3ad2/benchmarks/queries/clickbench/queries.sql#L29
I don't think min/max have appeared as priorities in benchmarking before because the queries in question are doing other string heavy operations that tend to dominate. Thus the use of GroupsAccumulatorAdaptor for Min/Max on strings, while bad, is overshadowed by other things
However, while working on https://github.com/apache/datafusion/pull/12092 it turns out that Min / Max on BinaryView and StringView are suuuuper slow. We can likely restore their speed to something similar to https://github.com/apache/arrow-rs/issues/6408 but I also think this is a good time to actually make Min / Max on strings faster. I will write up some ideas on how to do this

Background
(I will make a PR shortly to add this to the actual datafusion docs)
GroupsAccumulator logically does this:
      ┌─────┐
      │  0  │───────────▶   "A"
      ├─────┤
      │  1  │───────────▶   "Z"
      └─────┘
        ...                 ...
      ┌─────┐
      │ N-2 │               "A"
      ├─────┤
      │ N-1 │───────────▶   "Q"
      └─────┘

    Logical group      Current Min/Max
       number          value for that
                       group

GroupsAccumulator to store N aggregate
values: logically keepa a mapping from
each group index to the current value

Today, String / Binary min/max values are implemented using GroupsAccumulatorAdapter which results in
                                                              Individual String
                                                              (separate
                                                              allocation)

   ┌─────┐            ┌──────────────────────────┐
   │  0  │───────────▶│  ScalarValue::Utf8("A")  ├──────────▶   "A"
   ├─────┤            ├──────────────────────────┤
   │  1  │───────────▶│  ScalarValue::Utf8("Z")  │──────────▶   "Z"
   └─────┘            └──────────────────────────┘
     ...                 ...                                    ...
   ┌─────┐            ┌──────────────────────────┐
   │ N-2 │            │  ScalarValue::Utf8("A")  │──────────▶   "A"
   ├─────┤            ├──────────────────────────┤
   │ N-1 │───────────▶│  ScalarValue::Utf8("Q")  │──────────▶   "Q"
   └─────┘            └──────────────────────────┘

 Logical group         Current Min/Max value for that group stored
    number             as a ScalarValue which points to an
                       indivdually allocated String

   How GroupsAccumulatorAdaptor works today:
   stores each current min/max as a
   ScalarValue

Potential Design
One high level idea is to build a data structure that uses the same internal format (views/buffers) as StringViewArray in Arrow:
                  ┌───────────────────────────────────────────┐
                  │                        Stored in Vec<u8>  │
                  │   Stored in a         ┌─────────────────┐ │
                  │     Vec<u128>         │ ┌───────────┐   │ │
                  │ ┌─────────────┐    ─ ─│▶│some value │   │ │
   ┌─────┐        │ │ ┌─────────┐ │   │   │ └───────────┘   │ │
   │  0  │────────┼─┼▶│  View   │ │       │                 │ │
   ├─────┤        │ │ ├─────────┤ │   │   │                 │ │
   │  1  │────────┼─┼▶│  View   │─│─ ─ ┐  │      ...        │ │
   └─────┘        │ │ └─────────┘ │   │   │                 │ │
     ...          │ │    ...      │    │  │                 │ │
   ┌─────┐        │ │ ┌─────────┐ │   │   │                 │ │
   │ N-2 │        │ │ │  View   │─│─ ─ │  │   ┌────────────┐│ │
   ├─────┤        │ │ ├─────────┤ │     ─ ┼ ─▶│other value ││ │
   │ N-1 │────────┼─┼▶│  View   │ │       │   └────────────┘│ │
   └─────┘        │ │ └─────────┘ │       └─────────────────┘ │
                  │ └─────────────┘  String values are stored │
                  │                  inline or in extra byte  │
 Logical group    │                  buffer                   │
    number        └───────────────────────────────────────────┘
                   New structure: MutableStringViewBuilder

                   Current Min/Max value for that group stored in
                   same format as StringViewArray

In this design, the current value for  each group is stored in two parts (as described on arrow docs)

a fixed size u128 view
a variable length part with the string data

As new batches are updated, each View is updated if necessary
Benefits of design:

Hopefully use the same code as in arrow-rs
Allows Zero copy conversion to StringView / BinaryView at output
Use inlined values for quick min/max comparison

I believe (though we will have to verify it) that the conversion from MutableStringViewBuilder to just StringArray (not StringViewArray) should also be better than the current GroupsAccumulatorAdapter. Both conversions need to copy the string bytes again into the packed StringArray format, but the GroupsAccumulatorAdapter also has to allocate/free owned Strings as well
Potential challenges
I think the trickiest part of this code, other the low level code optimizations is that as min/max values are replaced, data in the variable length buffer will become "garbage" (not reachable) thus consuming more memory than necessary:
                        Stored in Vec<u8>
                     ┌────────────────────┐
                     │ ┌────────────────┐ │
┌─────────────┐      │ │prev max value 1│ │
│ ┌─────────┐ │      │ └────────────────┘ │
│ │  View   │─│─ ┐   │        ...         │
│ └─────────┘ │      │ ┌────────────────┐ │
│     ...     │  │   │ │prev max value m│ │
│             │      │ └────────────────┘ │
│             │  │   │ ┌────────────────┐ │
│             │   ─ ─│▶│prev max value m│ │
│             │      │ └────────────────┘ │
│             │      │        ...         │
│             │      └────────────────────┘
│             │
└─────────────┘      Previous min/max values
                     are not pointed to
                     anymore and need to be
                     cleaned up

I think this means the code will need something GenericByteViewArray::gc run occasionally
Random Thoughts
Thoughts: maybe this structure (MutableStringViewBuilder??) could be upstreamed eventually

@alamb is there anyone working on this + is this issue still relevant? I would love to tackle it as it seems like an interesting feature/optimization.

@devanbenz I think this would be a fun and interesting project as well as valuable to DataFusion. However, I also think it is pretty advanced -- I would enjoy helping with it, but also maybe @Rachelint or @jayzhan211  are interested in helping out 🤔

@alamb is there anyone working on this + is this issue still relevant? I would love to tackle it as it seems like an interesting feature/optimization.

@devanbenz I think this would be a fun and interesting project as well as valuable to DataFusion. However, I also think it is pretty advanced -- I would enjoy helping with it, but also maybe @Rachelint or @jayzhan211 are interested in helping out 🤔

Great! I'll get started on this later in the week/over the weekend :) Will likely bug folks as I require help haha 😆

In terms of implementation, what I suggest is:

Do a POC implementaiton: wire up just enough `StringView, don't worry about GC, basic unit tests
Verify it makes the clickbench query faster
Flesh out testing, documentation, add support for StringArrary, etc
Merge and profit (bonus points for blogging about it)

For the POC here is the reproducer I recommend:
Step 1. Get hits_partitioned using bench.sh:
cd benchmarks
./bench.sh data clickbench_partitioned

Step 2: Prepare a script with reproducer query:
set datafusion.execution.parquet.schema_force_view_types = true;

SELECT REGEXP_REPLACE("Referer", '^https?://(?:www\\.)?([^/]+)/.*$', '\\1') AS k, AVG(length("Referer")) AS l, COUNT(*) AS c, MIN("Referer")
FROM hits_partitioned
WHERE "Referer" <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;

andrewlamb@Andrews-MacBook-Pro-2:~/Software/datafusion2/benchmarks/data$ cat q28.sql
set datafusion.execution.parquet.schema_force_view_types = true;

SELECT REGEXP_REPLACE("Referer", '^https?://(?:www\\.)?([^/]+)/.*$', '\\1') AS k, AVG(length("Referer")) AS l, COUNT(*) AS c, MIN("Referer")
FROM hits_partitioned
WHERE "Referer" <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;

Step 3: Run script (with release build of datafusion-cli):
datafusion-cli -f q28.sql

set datafusion.execution.parquet.schema_force_view_types = true; -->  Elapsed 18.431 seconds.
set datafusion.execution.parquet.schema_force_view_types = false; -->  Elapsed 6.427 seconds.

The goal is to get set datafusion.execution.parquet.schema_force_view_types = true;  to be the same (or better) than when it is false
If you look at the flamegraph-string-view.svg, you can see most of the time is spent doing GroupsAccumulator

@alamb is there anyone working on this + is this issue still relevant? I would love to tackle it as it seems like an interesting feature/optimization.

@devanbenz I think this would be a fun and interesting project as well as valuable to DataFusion. However, I also think it is pretty advanced -- I would enjoy helping with it, but also maybe @Rachelint or @jayzhan211 are interested in helping out 🤔

Seems really interesting, I am reading the related disscusions.

BTW in case anyone is interested, I recorded a short video on how to make these flamegraphs: https://youtu.be/2z11xtYw_xs
I will add a link to that in the docs later

The challenge of String seems that?

If we just simply use a Vec<String> like primitives to keep the min/max values, it is too expensive to convert them to StringArray/StringViewArray(many many copy)
But if we use StringArray like approach to keep the values, we can't update the min/max values.
So Finally we need to use a StringViewArray like approach to make it, but still have the new challenge about gc?

The challenge of String seems that?

If we just simply use a Vec<String> like primitives to keep the min/max values, it is too expensive to convert them to StringArray/StringViewArray(many many copy)

I think the overhead is actually mostly that there is an additional (small) allocation for each String. For queries with a small numer of groups (like 100) an extra 100 allocations isn't all that bad.  For queries with millions of groups the overhad is substantial

But if we use StringArray like approach to keep the values, we can't update the min/max values.

I suppose we could potentially update the values as long as the new strings were shorter :thinking:

So Finally we need to use a StringViewArray like approach to make it, but still have the new challenge about gc?

@Rachelint  your implicit idea of using Vec<String> to store the state I think is actually quite interesting and maybe we should try that one first:
It would at least avoid calling Array::slice and likely be better than using GroupsAccumulatorAdapter, even if we could improve it later with more explciit memory management 🤔

Maybe it is not a good way, but can we

@Rachelint your implicit idea of using Vec<String> to store the state I think is actually quite interesting and maybe we should try that one first:
It would at least avoid calling Array::slice and likely be better than using GroupsAccumulatorAdapter, even if we could improve it later with more explciit memory management 🤔

🤔 Can we still use the string view like approach to store states https://github.com/apache/datafusion/issues/6906#issuecomment-2355604402, but for the uninlined state, we use a single String to store it?
And when we output them to StringViewArray, we make convert this single String to a tiny buffer.
For example, if all states are uninlined(len > 12), the output StringViewArray may be like:
row1: view1 buffer1(with only one string)
row2: view2 buffer2(with only one string)
...
rown: viewn buffern

I am not  familiar enough with StringViewArray, is it ok to do that? And will it lead to a extremely bad performance?

@Rachelint your implicit idea of using Vec<String> to store the state I think is actually quite interesting and maybe we should try that one first:
It would at least avoid calling Array::slice and likely be better than using GroupsAccumulatorAdapter, even if we could improve it later with more explciit memory management 🤔

Yes... At least it will be better than now, even we just use Vec<String> to impl a specific GroupsAccumulator for String type...

I am not familiar enough with StringViewArray, is it ok to do that? And will it lead to a extremely bad performance?

I think using a single Buffer for each string will be bad for performance (likely worse than storing as String and copying them at the end. StringViewArray is really optimized for a small number of buffers (even though in theory it could have 2B of them as it is indexed on i32)

I think
https://github.com/apache/datafusion/pull/6800#issuecomment-1622236683

I am not familiar enough with StringViewArray, is it ok to do that? And will it lead to a extremely bad performance?

I think using a single Buffer for each string will be bad for performance (likely worse than storing as String and copying them at the end. StringViewArray is really optimized for a small number of buffers (even though in theory it could have 2B of them as it is indexed on i32)

Ok, for StringView min/max, seems we can just start with using Vec<u128>(views) to store the inlined state(<= 12), use Vec<String> to store the unlined.
And when conerting it to StringViewArray, we just copy the Vec<String> to create the buffer (GroupsAccumulatorAdapter copy the states too).
For the short strings(<=12), it can avoid allocating String, and for the long ones, it just do the same thing as GroupsAccumulatorAdapter. Seems it can have a better performance(due optimization for shorts)?

I am not familiar enough with StringViewArray, is it ok to do that? And will it lead to a extremely bad performance?

I think using a single Buffer for each string will be bad for performance (likely worse than storing as String and copying them at the end. StringViewArray is really optimized for a small number of buffers (even though in theory it could have 2B of them as it is indexed on i32)

Ok, for StringView min/max, seems we can just start with using Vec<u128>(views) to store the inlined state(<= 12), use Vec<String> to store the unlined.
And when converting it to StringViewArray, we just copy the Vec<String> to create the buffer (GroupsAccumulatorAdapter copy the states too).
For the short strings(<=12), it can avoid allocating String, and for the long ones, it just do the same thing as GroupsAccumulatorAdapter. Seems it can have a better performance(due optimization for shorts)?

Seems like a reasonable place to start in my opinion. If we want to get more sophisticated at a later time we can try something more exotic. I suspect there will be times when individual allocations will be faster and times when a buffer will be faster and there will be memory consumption tradeoffs as well.
TLDR we should implement something and as long as it is better than what is currently going on that will be good.

while working on https://github.com/apache/datafusion/pull/12092 it turns out that Min / Max on BinaryView and StringView are suuuuper slow.

Fixed in #12575, now min/max on StringViewArray should be slightly faster than StringArray

while working on #12092 it turns out that Min / Max on BinaryView and StringView are suuuuper slow.

Fixed in #12575, now min/max on StringViewArray should be slightly faster than StringArray

I mean, fixed for that particular clickbench query which does not leverage stats to prune parquet access.

while working on #12092 it turns out that Min / Max on BinaryView and StringView are suuuuper slow.

Fixed in #12575, now min/max on StringViewArray should be slightly faster than StringArray

I mean, fixed for that particular clickbench query which does not leverage stats to prune parquet access.

Cool, I'm assuming this change didn't impact the GroupsAccumulatorAdapter call in anyway since I'm still seeing a similar performance + flamegraph with the following query:
set datafusion.execution.parquet.schema_force_view_types = true;

SELECT REGEXP_REPLACE("Referer", '^https?://(?:www\\.)?([^/]+)/.*$', '\\1') AS k, AVG(length("Referer")) AS l, COUNT(*) AS c, MIN("Referer")
FROM hits_partitioned
WHERE "Referer" <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;

BTW I was thinking more about this issue -- while a native Min/Max for strings / stringview will help, I have an idea that might make it simply it an added bonus: pushing the cast down into the parquet reader https://github.com/apache/datafusion/issues/12509#issuecomment-2366775581

BTW I was thinking more about this issue -- while a native Min/Max for strings / stringview will help, I have an idea that might make it simply it an added bonus: pushing the cast down into the parquet reader https://github.com/apache/datafusion/issues/12509#issuecomment-2366775581

This would only work for parquet correct? It would effectively be apart of the TableScan within the logical plan?

@Rachelint  your implicit idea of using Vec<String> to store the state I think is actually quite interesting and maybe we should try that one first:
It would at least avoid calling Array::slice and likely be better than using GroupsAccumulatorAdapter, even if we could improve it later with more explciit memory management 🤔

So this would effectively be using a Vec for state instead of your proposal of using something akin to StringView above @alamb?

I have had more time to take a look at this and sort of just wrap my head around how GroupsAccumulatorAdapter works a bit. I'm seeing that the performance impact is happening here https://github.com/apache/datafusion/blob/a35d0075744a058f81bd9ebed747e2e597434019/datafusion/functions-aggregate-common/src/aggregate/groups_accumulator.rs#L475 in slice_and_maybe_filter I'm under the assumption that this is happening due to the difference between the BinaryView and the Binary Scalar values. Taking a look at https://www.influxdata.com/blog/faster-queries-with-stringview-part-one-influxdb/ I understand that BinaryView is effectively a non-contiguous structure where as Binary is contiguous so it is easily sliced. So the idea here is to effectively change the underlying state in which the accumulator structure receives that data thus making it either easier to call slice or remove the calling of slice entirely?

@Rachelint  your implicit idea of using Vec<String> to store the state I think is actually quite interesting and maybe we should try that one first:
It would at least avoid calling Array::slice and likely be better than using GroupsAccumulatorAdapter, even if we could improve it later with more explciit memory management 🤔

So this would effectively be using a Vec<String> for state instead of your proposal of using something akin to StringView notated here #6906 (comment) @alamb?

I think what @alamb means is that just simply using Vec<String> to store the store will be at least not worse than StringArray + GroupsAccumulatorAdapter, and it is easy to start from.
And after improve it to be not worse than StringArray + GroupsAccumulatorAdapter, we can continue to push forward enabling string view by default.

@Rachelint  your implicit idea of using Vec<String> to store the state I think is actually quite interesting and maybe we should try that one first:
It would at least avoid calling Array::slice and likely be better than using GroupsAccumulatorAdapter, even if we could improve it later with more explciit memory management 🤔

So this would effectively be using a Vec<String> for state instead of your proposal of using something akin to StringView notated here #6906 (comment) @alamb?

I think what @alamb means is that just simply using Vec<String> to store the store will be at least not worse than StringArray + GroupsAccumulatorAdapter, and it is easy to start from.
And after improve it to be not worse than StringArray + GroupsAccumulatorAdapter, we can continue to push forward enabling string view by default.

I have had more time to take a look at this and sort of just wrap my head around how GroupsAccumulatorAdapter works a bit. I'm seeing that the performance impact is happening here
https://github.com/apache/datafusion/blob/a35d0075744a058f81bd9ebed747e2e597434019/datafusion/functions-aggregate-common/src/aggregate/groups_accumulator.rs#L475
in slice_and_maybe_filter I'm under the assumption that this is happening due to the difference between the BinaryView and the Binary Scalar values. Taking a look at https://www.influxdata.com/blog/faster-queries-with-stringview-part-one-influxdb/ I understand that BinaryView is effectively a non-contiguous structure where as Binary is contiguous so it is easily sliced. So the idea here is to effectively change the underlying state in which the accumulator structure receives that data thus making it either easier to call slice or remove the calling of slice entirely?

I guess the current goal is to remove the calling of slice, and get an at least not worse performance than StringArray + GroupsAccumulatorAdapter as mentioned above.

I think what @alamb means is that just simply using Vec to store the states will be at least not worse than StringArray + GroupsAccumulatorAdapter, and it is easy to start from.

Yes indeed. Thank you

I guess the current goal is to remove the calling of slice, and get an at least not worse performance than StringArray + GroupsAccumulatorAdapter as mentioned above.

💯

Finally got around to trying out a little POC locally. I'm getting much better results by just storing the state as Vec<String> it's very much not polished at this moment in time but I will have some more time on Sunday + early next week to continue working on this.

Nice! If you want to put up a POC / draft PR I would be more than happy to help work on it with you

Nice! If you want to put up a POC / draft PR I would be more than happy to help work on it with you

Sounds good I pushed up what I have so far as a draft. There are a few things I would like to do including making the accumulator more generic so I use a single method for min or max and pass an aggregate fn similar to how other UDFs do it.

I will check out https://github.com/apache/datafusion/pull/12677 -- I hoped to do it today but I appear to be almost out of time

I think we can store (len, prefix, index) for string/binary view cases. We then compare the value based on len + inline or prefix first, if they are the same we look up the actual value with index. In this way, we don't have the garbage collect issue, since we store the index instead of the whole string.
@alamb @Rachelint @devanbenz

I am a bit confused about where we store actual string values?
We still store it using Vec<String>, and through copying way when constructing the StringView array?

I think we can store (len, prefix, index) for string/binary view cases. We then compare the value based on len + inline or prefix first, if they are the same we look up the actual value with index. In this way, we don't have the garbage collect issue, since we store the index instead of the whole string.

I think the garbage collection issue happens when the current "min" for a group is updated to a new value. We have to copy the new min value into the buffer and adjust the offset, but the old min value is still int he pages

er

My approach only works within batch. The garbage collection issue here is that whenever we got a better "min", we need to append the bytes and the old bytes is now "garbage". But given a batch, we can get the string value by the row index of the StringArray, therefore we don't need to update the possible long bytes but update the (len, prefix, row - index) . We only store this "min" (which create garbage) after computing the whole batch and update it to the accumulator. The more the string value is in the same group within the batch, the less "garbage" we created compare to the original proposed design. The best case is that the whole batch has the same group value so we just do 1 update to the accumulator, the worst case is that every group value is distinct so we need n (n is batch size) updates to the accumulator.

The more values are in the same group within the batch, the less "garbage" we created compare to the original proposed design.

this makes a lot of sense (and is similar to what the current non groups accumulator does -- finds the min per batch and then updates the current overall min from that, rather than updating the min for each input row).

I started hacking on this today: https://github.com/apache/datafusion/pull/12792. I am fighting with the generics, but I think I am pretty close.

Ok, I think https://github.com/apache/datafusion/pull/12792 is ready to review. It has a nice 10% improvement on Q28 and avoids the slowdown when string view is enabled


init() missing 2 required positional arguments: 'env' and 'cell_factory'

Hi, I have run this in google colab, just running through your ReadMe and got this error.
from go_explore import GoExplore

go_explore = GoExplore("MontezumaRevenge-v0")
go_explore.explore(total_timesteps=100000)
go_explore.robustify(total_timesteps=1000)

TypeError: __init__() missing 2 required positional arguments: 'env' and 'cell_factory'

Thank you for reporting.
I am currently working on this repository. It is not fully functional yet. Contributions are welcome.
For the moments, the exploration seems to work:
from go_explore import GoExplore
from go_explore.cells import ImageGrayscaleDownscale
from stable_baselines3 import DQN

cell_factory = ImageGrayscaleDownscale(height=10, width=15, nb_shades=30)
go_explore = GoExplore(DQN, "MontezumaRevenge-v0", cell_factory, model_kwargs=dict(learning_starts=2000))
go_explore.explore(50000)

But you can expect major changes in the coming days on this repo.


Contribution: Discord server links expired

All the Discord Server Links are expired, I would like to be able to join the community.
Ocurrencies:
https://github.com/LadybirdBrowser/ladybird/blob/master/CONTRIBUTING.md#communication
https://github.com/LadybirdBrowser/ladybird/blob/master/README.md#get-in-touch-and-participate
https://ladybird.org/#gi (Img: Screenshot_1)

Cannot reproduce

Yeah the link isn't expired for me either. Maybe contact Discord about it.

Cannot reproduce, links are working as expected on my side Have you tried copying them into the Discord (+) -> Join a server menu? Perhaps you have been blocked in some way?

This worked


Add new Web features module with networking feature installation

Codecov Report
Attention: Patch coverage is 35.48387% with 80 lines in your changes are missing coverage. Please review.

Project coverage is 90.81%. Comparing base (eabe5ca) to head (60dbcba).
Report is 32 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #566      +/-   ##
==========================================
- Coverage   91.03%   90.81%   -0.22%
==========================================
  Files         116      118       +2
  Lines       29562    29683     +121
  Branches     2177     2187      +10
==========================================
+ Hits        26912    26958      +46
- Misses       2615     2690      +75
  Partials       35       35

Components
Coverage Δ

agent
80.70% <100.00%> (+0.07%)
:arrow_up:

api
94.15% <14.89%> (-3.62%)
:arrow_down:

common
98.68% <ø> (ø)

credentials
95.26% <ø> (ø)

crypto
93.81% <ø> (ø)

dids
97.69% <ø> (ø)

identity-agent
96.70% <ø> (ø)

crypto-aws-kms
100.00% <ø> (ø)

proxy-agent
96.70% <ø> (ø)

user-agent
96.70% <ø> (ø)

I would double check this, seems like all the dependabot stuff would be merged in too

Features already implemented in a different branch that was merged.


["ID"] correct format

Error: categoryID and closedCategoryID of each configured ticket category must be an array, such as ["ID"]
    at findAvailableCategory (/home/pi/MusicBot/LucasB25/PanaisSupport/index.js:216:11)
    at Object.execute (/home/pi/MusicBot/LucasB25/PanaisSupport/events/interactionCreate.js:1654:40)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Hey, please make sure to check each category in case you missed one where it's not an Array. Also please don't rename configuration options, the option is called closedCategoryID and not closedcategoryID, changing the name of the options might also be causing the issue.
The example in the latest config.yml.example looks like this:
    categoryID: ["111111111111111111"] # The IDs of the categories where those tickets will be created, for example ["000000000000000000", "111111111111111111"]
    closedCategoryID: ["111111111111111111"] # The IDs of the categories where the closed tickets will go, for example ["000000000000000000", "111111111111111111"]


feat: add bech32 support

Closses #376
On the SDK side we want to use addresses in the bech32 format (see BIP 0173). The core client works with plain addresses so we need to convert bech32 addresses behind the scenes when querying a provider.
This PR adds Bech32Address to fuels-types. Bech32Address relies on the bech32 crate to encode/decode.
Internally, Bech32Address  has a plain Address as a field to easily convert when needed.  Two small tests are included just as sanity check since the bech32 crate already covers encoding/decoding tests.
User-facing, wallet and provider methods that previously used Address now accept Bech32Address.

I'm not sure "bech32address" is the best term. This encoding could be used for both addresses and contract IDs.

I didn't consider this. Do we want to use it for both?
If so, here is a few ways we could implement it:
1.
Just rename it to Bech32 and have the following methods  to_address(), to_contract_id().

PRO: straightforward UI
CON: no way to know what the actual underlying data is

2.
Rename it, have an enum field with variants ContentType::Address and ContentType::ContractId.

PRO: the user can check what data is encoded
-CON: one more parameter when initializing from string e.g. Bech32::from_string("...", ContentType::Address)

3
Make it hold a generic Bech32<T>  and implement it for the fuel_type Address and ContractId

PRO: more rust like I guess
CON: extra info on the UI side needed, e.g. Bech32<Address>::from_str(...), and we still expose the fuel_types

cc @digorithm

I'm not sure "bech32address" is the best term. This encoding could be used for both addresses and contract IDs.

I didn't consider this. Do we want to use it for both?
If so, here is a few ways we could implement it:
1. Just rename it to Bech32 and have the following methods  to_address(), to_contract_id().

PRO: straightforward UI
CON: no way to know what the actual underlying data is

2. Rename it, have an enum field with variants ContentType::Address and ContentType::ContractId.

PRO: the user can check what data is encoded
-CON: one more parameter when initializing from string e.g. Bech32::from_string("...", ContentType::Address)

3 Make it hold a generic Bech32<T> and implement it for the fuel_type Address and ContractId

PRO: more rust like I guess
CON: extra info on the UI side needed, e.g. Bech32<Address>::from_str(...), and we still expose the fuel_types

cc @digorithm

I like number 2. Instead of passing ContentType::<type> to from_string could we have two separate methods new_address_from_string and new_contract_id_from_string? IMO it would be easier to remember or look up the method names than thinking about passing enums as an argument.

I like number 2. Instead of passing ContentType::<type> to from_string could we have two separate methods new_address_from_string and new_contract_id_from_string? IMO it would be easier to remember or look up the method names than thinking about passing enums as an argument.

Number 2. is implemented. I changed the PR description btw.


Make it clear that this driver is for the framebuf interface

This helps users find all available methods on the driver object.

Thanks for the quick review, @ladyada. :)
On Tue, Oct 8, 2019 at 1:10 PM ladyada<EMAIL_ADDRESS>wrote:

Merged #33
https://github.com/adafruit/Adafruit_CircuitPython_SSD1306/pull/33 into
master.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/adafruit/Adafruit_CircuitPython_SSD1306/pull/33?email_source=notifications&email_token=AAB5I43Z7L6R54C7KHOZLFLQNTSLDA5CNFSM4I6WPLLKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOUC6LK3I#event-2696721773,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB5I45UONDP44GZ53TR76LQNTSLDANCNFSM4I6WPLLA
.

:)


Zepto.js support

Hi! I'm working on a mobile product now, so i'm using Zepto instead of jQuery there.
If you review the commit - there are really small changes that allow to use Rivets with Zepto out of the box.

@mikeric how come this changed was reverted? This still breaks with latest version of Zepto.


Vite dev server holds a lock on directories that have nested directories

Describe the bug
While debugging a firebase issue in a sveltekit project, I discovered that many folks are having directory renaming permission errors. I tracked this down to a minimal use case of running just a Vite dev server.
While Vite dev server is running, renaming a directory in the project root fails, if and only if it contains a subdirectory.
Note that this does not occur on stackblitz (https://stackblitz.com/edit/vitejs-vite-cbseww?file=package.json), so it's probably OS-dependent. I'm using git bash on Windows 11. However, folks in the linked bug see this on Windows 10 as well.
Reproduction
https://github.com/bmcbarron/vite-bug-repro
Steps to reproduce
# On Windows 11, git bash version 5.1.16(1)-release (x86_64-pc-msys)
npm create vite@latest # Choose all defaults
cd vite-project
npm i
npm run dev # Leave running

# In new shell
cd vite-project
mkdir a
mkdir a/b
mv a c # Fails with permission denied
rmdir a/b # OR stop vite instead of removing nested directory
mv a c # Succeeds

System Info
System:
    OS: Windows 10 10.0.22621
    CPU: (12) x64 Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz
    Memory: 2.41 GB / 15.94 GB
  Binaries:
    Node: 18.12.1 - C:\Program Files\nodejs\node.EXE
    npm: 9.4.0 - C:\Program Files\nodejs\npm.CMD
  Browsers:
    Chrome: 106.0.5249.103
    Edge: Spartan (44.22621.1105.0), Chromium (109.0.1518.70)
    Internet Explorer: 11.0.22621.1
  npmPackages:
    vite: ^4.1.0 => 4.1.1

Used Package Manager
npm
Logs

Click to expand!
```shell
brian@yellow MINGW64 /c/git/vite.test/vite-project
$ mkdir a
brian@yellow MINGW64 /c/git/vite.test/vite-project
$ mkdir a/b
brian@yellow MINGW64 /c/git/vite.test/vite-project
$ mv a foo
mv: cannot move 'a' to 'foo': Permission denied
brian@yellow MINGW64 /c/git/vite.test/vite-project
$ rmdir a/b
brian@yellow MINGW64 /c/git/vite.test/vite-project
$ mv a foo
brian@yellow MINGW64 /c/git/vite.test/vite-project
</details>

### Validations

- [X] Follow our [Code of Conduct](https://github.com/vitejs/vite/blob/main/CODE_OF_CONDUCT.md)
- [X] Read the [Contributing Guidelines](https://github.com/vitejs/vite/blob/main/CONTRIBUTING.md).
- [X] Read the [docs](https://vitejs.dev/guide).
- [X] Check that there isn't [already an issue](https://github.com/vitejs/vite/issues) that reports the same bug to avoid creating a duplicate.
- [X] Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to [vuejs/core](https://github.com/vuejs/core) instead.
- [X] Check that this is a concrete bug. For Q&A open a [GitHub Discussion](https://github.com/vitejs/vite/discussions) or join our [Discord Chat Server](https://chat.vitejs.dev/).
- [X] The provided reproduction is a [minimal reproducible example](https://stackoverflow.com/help/minimal-reproducible-example) of the bug.

This is a great finding. Thanks. I was able to reproduce.
Instead of running Vite, just running this script seems to reproduce.
import chokidar from 'chokidar'

chokidar.watch('.')

Looks like there's an upstream issue https://github.com/paulmillr/chokidar/issues/320 from a long time ago. Checking the links from there on, looks like it's an issue from fs.watch. I'm not sure if there's much we can do though.

Closing as this as it's an upstream issue and there's not much we can do. Perhaps upgrading node might fix the issue.


community: Add relyt vector database

@baskaryan hello, can you help to review this pr? thx

@baskaryan thx for above review, can you asign a reviewer to this pr and help to complete this pr


lychee.ui.Icon: Initial Implementation

Figure out how to use the Font generator to render icons from a webfont-converted icon font.
The lychee.ui.Icon Implementation is a dependency for all lychee.ui.Button instances, so the button can have a 'label' (String) property or 'icon' (String) property.
The lychee.ui.Icon interface shall have an internal ENUM that maps all strings accordingly to the matching icon. That means you can create something alike:
new lychee.ui.Icon({
    icon: 'browser'
});

As of @3d31955c209e81b0ec496b964f474bdc61d8a165 the Design Rule is that all external buttons always have the same icon, dependent on their real triggered action. That means all file-manager starting Buttons have a File icon, all Browser-starting Buttons have a web icon etc. pp.
Therefore this issue is resolved as per se the design guidelines only require Helpers to have icons for now.


fix demo urls

github.com is no longer accepted for github pages sites, it needs to be github.io instead

Hi @dsingleton86! Thanks for the contribution and good spot! :)


IntelliJ dialog box too big for window - Not allowing to set Foreign Assets for AppCall

I can't set Foreign Assets for an Application Call transaction.

Possible workaround?
Thanks!
PS: This tool is awesome. I love using AlgoDEA!

@GuidoDipietro Did you try to change the screen resolution ?
There is a min size set for this panel.
I will try to add a scrollbar to this screen in the next iteration.
thanks.

I wanted to see if there was any possible workaround without changing the resolution, but I think I'll go with that for now.
Thanks 👍


Swap error: signing: rpc error: code = Unknown desc = permission denied

lnd 0.13.1-beta.rc2
lit v0.5.0
windows
Loop out always fails with RPC error. I have plenty of outgoing capacity/channels and even with the minimum of 250k sats it doesnt work. In the past i have used a past version of lit to successfully make loopouts but now it doesnt work at all since upgrading lnd and lit.
Any ideas what i need to check and/or change?

2021-07-25 11:36:06.784 [INF] RPCS: Received new block notification: height=692576
2021-07-25 11:36:06.784 [INF] LOOP: Received block 692576
2021-07-25 11:36:06.784 [INF] LOOP: Received block 692576
2021-07-25 11:36:06.787 [INF] LOOP: 86848c Checking preimage reveal height 692655 exceeded (height 692576)
2021-07-25 11:36:06.787 [INF] LOOP: 5aad29 Checking preimage reveal height 692655 exceeded (height 692576)
2021-07-25 11:36:06.788 [INF] LOOP: 86848c Swap script confirmed on chain
2021-07-25 11:36:06.789 [INF] LOOP: 86848c Htlc tx<PHONE_NUMBER>9ee2555517e684ff0dedc2234b02ed80e0c4c107fb4ec674ebf18d at height 692576
2021-07-25 11:36:06.789 [INF] LOOP: 86848c Htlc value: 0.0025 BTC
2021-07-25 11:36:06.963 [INF] LOOP: 86848c Server update: SERVER_HTLC_CONFIRMED received, timestamp: 2021-07-25 11:36:08.598107516 +0200 CEST
2021-07-25 11:36:07.564 [INF] LOOP: 86848c Payment 210290ea38c6937ae0175685d985b1acd959e4279a23285f0395ee9c27401816: state=SUCCEEDED
2021-07-25 11:36:08.540 [INF] LITD: Handling gRPC web request: /poolrpc.Trader/ListAccounts
2021-07-25 11:36:08.757 [INF] LITD: Handling gRPC web request: /lnrpc.Lightning/GetInfo
2021-07-25 11:36:16.797 [ERR] LOOP: 86848c Swap error: signing: rpc error: code = Unknown desc = permission denied
2021-07-25 11:36:16.798 [INF] LOOP: 86848c Loop out swap state: FailTemporary
2021-07-25 11:36:16.800 [ERR] LOOP: Execute error: signing: rpc error: code = Unknown desc = permission denied

You need to re-generate lnd's macaroons if you upgraded from an older version.

You need to re-generate lnd's macaroons if you upgraded from an older version.

On startup, lnd checks to see if the admin.macaroon, readonly.macaroon and invoice.macaroon files exist. If they don't exist, lnd updates its database with a new macaroon ID, generates the three files admin.macaroon, readonly.macaroon and invoice.macaroon, all with the same ID.

I stopped lnd. renamed and oved all the old macaroons in C:\Users\XXXX\AppData\Local\Lnd... but no new macaroons are created. am i doing this wrong?

You need to re-generate lnd's macaroons if you upgraded from an older version.

After regenerating the macaroons all the loops with temporary failure switched to preimage revealed... seems to be fixed. closing


Validate skiplist nodes in seek operations

What problem does this PR solve?
Problem Summary:
pmem record of skiplist maybe updated during seek, so we need to validate them
What is changed and how it works?
What's Changed:

catch exceptions for key comparation
check validation of skiplist dram/pmem node after comparing keys

Check List
Tests

Unit test
stress test

Side effects

Performance regression
Not tested yet

ABA problem will be resolved in another patch


Bug 1300076 - SIGPIPE events kill go process [rhel7-1.10.3]

same as https://github.com/projectatomic/docker/pull/148 for rhel7-1.10.3

ping @jwhonce @rhatdan

LGTM


:construction_worker: Create lint and test workflow

:construction_worker: Creates a lint-and-test.yaml github workflow, to automate testing and enforce some code style checks

:construction_worker: Add Dependabot pip dependency management config

:heavy_plus_sign: add black and isort dev dependencies

:pushpin: pin dependency versions for best practice

:art: apply black and isort formatting to python files

:arrow_up: Upgraded test dockerfile base image to use ubuntu:20, so that python can be upgraded to 3.9. Previous base image used 3.6, and ubuntu base image was chosen because it's easier to install the libindy dependency on ubuntu than debian, which official python base images run on.

(Side note: no offense, but it's kinda bad to be running Python 3.6 in 2024 -- it's been end-of-life since Dec 2021! And it's not a heavy lift to update to the new python versions - nevertheless it will be a process to make that migration across the board)

@esune, Assigning you and as reviewers since I think we should both take a look.  Assigned to me to get it on my backlog.


Add a EVAL_STANDALONE_START and EVAL_STANDALONE_END events and change RUD to not wait_for_workers every eval

What does this PR do?
Previously, RemoteUploaderDownloader.wait_for_workers was called on the eval_end event. This meant that wait_for_workers would happen after every individual evaluator loop. This results in unnecessary hangs waiting for checkpoint upload in the middle of a run. This PR adds new events to signify that the standalone trainer.eval is being called, as opposed to eval from within the training loop, and changes the RemoteUploaderDownloader to use that event instead of the eval_end event. It also spawns a new issue CO-2268.
What issue(s) does this change relate to?
Closes CO-2267
Manual test
Manual test is by running
composer train.py yamls/finetune/mpt-7b_dolly_sft.yaml max_duration=5ba global_train_batch_size=4 device_train_microbatch_size=1 save_folder=oci://mosaicml-internal-checkpoints/daniel/checkpoints/{run_name} eval_subset_num_batches=2 eval_first=False optimizer.name=decoupled_lionw model.pretrained=False dist_timeout=20 loggers.wandb={} model.config_overrides.d_model=256 model.config_overrides.n_layers=12 run_name=test-run eval_interval=2ba autoresume=False save_overwrite=True save_interval=1ba

Before this PR it would hang after each eval, after this pr it just waits at the end of the run for the uploads to complete.
Before submitting

[x] Have you read the contributor guidelines?
[ ] Is this change a documentation change or typo fix? If so, skip the rest of this checklist.
[x] Was this change discussed/approved in a GitHub issue first? It is much more likely to be merged if so.
[x] Did you update any related docs and document your change?
[x] Did you update any related tests and add any new tests related to your change? (see testing)
[x] Did you run the tests locally to make sure they pass?
[x] Did you run pre-commit on your change? (see the pre-commit section of prerequisites)

@mvpatel2000 In looking at that, I've discovered we don't document the "predict" events anywhere, which would be analogous to this. Any suggestions where to put these new event docs?

@mvpatel2000 In looking at that, I've discovered we don't document the "predict" events anywhere, which would be analogous to this. Any suggestions where to put these new event docs?

hm... in this case I think it's fine to leave off for now? I would rather keep the events section focused on training and simple vs. detailing the more "advanced" events. How about we open a jira and include this in the docs refactor (I'll follow up on this actually happening)

CO-2270


Shreyasi17

Hii MLCC guys ! I've solves Assignment 2 problems .Please check it out.Thank you

Don't send pull request to master branch.


Auto-GPT not installing and not running when "installed"

I am having an issue with Auto-GPT not installing or not showing that it has been installed in the UI, because Install button doesn't change to Uninstall and there is no Open In Terminal button.
When I did have it installed, it would not open and run. Getting a message in CLI showing package inventory and a Tea prompt.
See attached screen recording.
https://github.com/teaxyz/cli/assets/963889/899f7ce5-22d7-4317-9c6b-96f40dde107e

This may have something to do with having had it installed previously, but uninstalling it so I could demo install for the Twitter spaces I did with @mxcl

It is though not running properly.
https://github.com/teaxyz/cli/assets/963889/c1d34c05-367c-460d-b9c7-5cd910ad0be1

@ABevier @getneil


The Footer Text is Not readable

What Happened
The kubernetes.io/partners footer is having text which is not readable and instead of logos it is having a white box which is making the page inconsistent.
How to resolve it:
Improve the look and feel of partners css file in which footer design should be improved. So that the element written in footer can be easily visible.
Page to resolve :
https://kubernetes.io/partners/
Image The issue is talking about( To understand it easily)

/sig docs
/kind bug
/language en

Page related Issue: https://kubernetes.io/partners/

I would like to work on this issue
/assign

/area web-development
Raised PR #48226 for this issue

This issue is not only in English it also in Other Localization .

/retitle Some pages have unreadable footer text
I think the community page is affected too BTW

/triage accepted


Fix french-canadian translation in bitcoin_fr_CA.ts

This fixes some very blatant grammar mistakes in the french canadian translation file.

Approved!

Thanks @gabgosrob for the submission and @AbcSxyZ for the review.


Issue with installing Vue with Laravel Vite

Laravel Vite Plugin Version: 2.9.14
Laravel Version: 9.20.0
Node Version: 18.5.0
NPM Version: 8.14.0
Host operating system: macOS 12.4
Web Browser & Version: Edge 103.0.1264.51
Running in Sail / Docker: No (Just Valet)

Description:
I tried to install @vitejs/plugin-vue in a fresh Laravel application with version 9.20.0 and I faced the issue that the current version of @vitejs/plugin-vue is 3.0.0 and it requires Laravel Vite with version 3.0.0
And when I tried to upgrade Laravel Vite to version 3.0.0 I faced another issue. It is not compatible with ("laravel-vite-plugin": "^0.4.0")
Notes:

I got the configurations from https://laravel.com/docs/9.x/vite#configuring-vite
I submitted an issue related to Inertia and someone answered me that the support of Laravel Vite 3.0.0 will be in the next version of Laravel as I understood. But what should I do to fix this issue in the current version?

Steps To Reproduce:

run: laravel new vite
run: cd vite
run: npm install
run: npm install vue@next vue-loader@next (Optional)
run: npm i @vitejs/plugin-vue

Also see https://github.com/laravel/vite-plugin/pull/76


implicit_hyphen_keys error fix

In the plover_dutch.py file, the field implicit_hyphen_keys should not contain '0-' and '5-'. This causes an error! I do not know how to change this myself

Thanks for reporting!


Any pointers to help me set my picture?

First, I would like to thank you for publishing this project! I found your project at jekyllthemes.org and have already started using it for what will be my new blog and homepage at www.brodybits.com.
Unfortunately Jekyll and Material Design Lite are both very new for me even though I did look at them in the past. The tutorials that I have found so far cover creating a blog from scratch (which is not needed for this theme) or creating a blog from another template (such as Poole).
I am having trouble figuring out how to change my photo, any pointers would really help.
I think this project definitely needs a tutorial. I am happy to contribute as I figure out how to do the customizations and start writing blog posts.
I will also try to import some of my old Blogger postings, will post my results.

Hi @brodybits. Thanks for pointing out the tutorial need. I guess you figured out avatar config (by md5 hash & gravatar), right?

Sorry, I was wrong. You should not change the src attribute of img tag. Just update the email_md5 property in _config.yml with md5 hash of your email address (which is used to fetch matching gravatar image).
See gravatar.com if you're not familiar.

Thanks. I am actually not using gravator.com, just using a static image instead.


Unexpected EOF running unused-ports

2023-04-06 08:23:51 [INFO] - querying port 4 of 51 for workload 3 of 8 - port 59131 UDP on /orgs/65886/workloads/fd359f86-3eda-4537-aebd-cbd8a393fce4
2023-04-06 08:24:08 [INFO] - querying port 5 of 51 for workload 3 of 8 - port 58588 UDP on /orgs/65886/workloads/fd359f86-3eda-4537-aebd-cbd8a393fce4
2023-04-06 08:24:23 [ERROR] - get traffic_flows/async_queries - unexpected EOF

@vvaj  - does this happen every time you run the command or is it transient?
Also, can you run on latest version to verify.

I received the same error. Tried 2 times with v.9.8.4. It errors out after a few hours, with no results saved.


VMPodScrape support for authentication

Hi,
first of all, I'm testing this project now since 3 1/2 hours and its amazing... I was able to replace prometheus near to 90% for now :-)
While checking the VMPodScrape I detected that it seems not to be possible to define basicAuth for the requests, is this right?
Or do you maybe see a way to add such a parameter to the CRD?
Thank you in advance,
Best regards,
Björn

Hello, thanks for reporting.
Its a bug, need to add missing auth params (basic and bearer). Will fix it soon.

Hello, thanks for reporting.
Its a bug, need to add missing auth params (basic and bearer). Will fix it soon.

Was added to v0.8.0 release.


NPM deprecated packages

Problem
I run slate migrate and I get the following warnings:
npm WARN deprecated<EMAIL_ADDRESS>Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN deprecated<EMAIL_ADDRESS>Use uuid module instead
npm WARN deprecated<EMAIL_ADDRESS>ReDoS vulnerability parsing Set-Cookie https://nodesecurity.io/advisories/130
npm WARN deprecated<EMAIL_ADDRESS>Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN deprecated<EMAIL_ADDRESS>Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN deprecated<EMAIL_ADDRESS>

Same!

After looking into this, I have come to the conclusion that this is intended behaviour. These warnings are the result of our dependency's dependencies (aka gulp's dependencies) being out of date. They are not errors.
I played with the idea of hiding npm warnings and only showing errors using the loglevel flag, but I feel that there could potentially be useful information in npm warnings so I'm leaving it.


[BUG]: LoRA does not support the training of reward models

🐛 Describe the bug

Environment
env
OS：ubuntu 20.04
GPU：4 x A10
python==3.9.0
torch==1.13.1-cu116
colossalai==0.2.5
command
python train_reward_model.py --pretrain "bigscience/bloom-560m" --lora_rank 16

I am very confused why the training encounters errors after setting lora_rank, whether it is the wrong way of my use。

I read your modifications, I don't think these modifications can fix the problem, and I noticed that the latest main version of ColossalAI removed the use of LoRA during rm model training, and now I think ColossalAI does not support LoRA at all, right?


fyne, overview should show only a few transactions

5 transactions should be fine

This has already been fixed


install kconfiglib using apt repo

Note: Please adhere to Contributing Guidelines.
Summary
Closes #14918
Impact
No functional impact

.github/workflow/build.yml [Line 363], not sure if i should update over there as well? Because it seems to build in a venv which is a recommended approach as well.


Use testing-library in test-utils/render

Fixes

This is an alternative PR to #2041
Description

Testing Instructions

Checklist

[x] My pull request has a descriptive title (not a vague title likeUpdate index.md).
[x] My pull request targets the default branch of the repository (main) or a parent feature branch.
[x] My commit messages follow best practices.
[x] My code follows the established code style of the repository.
[x] I added or updated tests for the changes I made (if applicable).
[ ] I added or updated documentation (if applicable).
[x] I tried running the project locally and verified that there are no visible errors.
[ ] I ran the DAG documentation generator (if applicable).

Developer Certificate of Origin

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

@obulat I'm unfamiliar with many aspects of this part of the project, do you mind providing a description of these changes which might help guide my review and give me context for what's being changed?

@obulat I'm unfamiliar with many aspects of this part of the project, do you mind providing a description of these changes which might help guide my review and give me context for what's being changed?

Sure, I'll add more context by Monday. I opened the PR as soon as it passed the CI locally to show @sarayourfriend what I meant in the comment to #2041, and was planning to add context for reviews later :) Got too much into fixing the tests that didn't get time to add it yesterday.

Thank you! I saw the critical label and wanted to review ASAP but I wasn't sure how to interpret the change set 😅

Looks good to me, but I would still like the ESLint rule to be added to prevent unintentional usages of @vue/test-utils render. Here it is from the original PR.
https://github.com/WordPress/openverse/blob/e72972cd9186371ed76cc031f0c2da0d1fb3a8b8/.eslintrc.js#L101-L106
If it can't be included in this PR for some reason then a separate issue to add it would be good.

I forgot to add it, thank you for the link :)


Perform key check when wallet is encrypted

Coverage decreased (-0.2%) to 94.529% when pulling 21ec247b3ff3687d933d745e3f906c7afd84d6bc on isocolsky:fix/key-validation into e41d6ef459fa25bd403d0c1a12baa53d13cf5348 on bitpay:master.


Clickable Founding Members

In https://fairdom.github.io/partners.html the founding members, and the people, are not clickable, They probably should be

Yes, we can do that, though not sure it adds much value. Thanks Alan.

Dear all,
I think Alan has a point. With the association there were members and founding members.
Best,
Wolfgang
--
Priv. Doz. Dr. Wolfgang Müller
Group leader, SDBV group
HITS gGmbH
Schloß-Wolfsbrunnenweg 35
69118 Heidelberg
Germany
phone: +49-6221-533 231
fax: +49 6221 533298
email: @.***
http://www.h-its.org

Amtsgericht Mannheim / HRB 337446
Managing Director:
Dr. Gesa Schönberger
Scientific Director:
Prof. Dr. Frauke Gräter

Von: Munazah @.***>
Gesendet: Dienstag, 21. September 2021 22:31
An: FAIRdom/fairdom.github.io
Cc: Subscribed
Betreff: Re: [FAIRdom/fairdom.github.io] Clickable Founding Members (#29)
Yes, we can do that, though not sure it adds much value. Thanks Alan.

You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com/FAIRdom/fairdom.github.io/issues/29#issuecomment-924361196, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEIF4WX3DNJU3YEPM2BGKD3UDDTSNANCNFSM5EPCJ5EA.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Resolved, founding members are now clickable.


Write a C++ program to convert hours to seconds

Description
Write a C++ program to convert hours to seconds
1 hour = 60 minutes = 3600 seconds
Input  : 5
Output : 18000

How to contribute

Save the solution in ConvertHoursToSeconds.cpp file
Add ConvertHoursToSeconds.cpp file in convert-hours-to-seconds folder

Can you assign it to me?

!assign


What's with the gradle build errors?

I looked at the commit and clicked on the red X and looked at the log. Something seems to be missing.

Don't know, what does it say is missing?

Could not find com.oroarmor:oro-config-forge:3.1.0.

Ah, you might want to fetch the latest commit. That was a custom version of one of my libraries that i was testing with. They should all be reverted to 3.0.0 now

I just looked at it. And there's no latest commit thing.


Remove redundant check in PrefabDiff::generateDiff

Since output is uninitialized, the comparison if (output == nullptr) will always be true

I believe I kept it on purpose. If someone modifies that code by adding a new field to compare, he also needs to remember to re-add that check (if his code comes earlier), which I think is an easy thing to forget.

That's a fair point. So how about this approach ?
We create PrefabObjectDiff output on stack, and mark it as not valid.
Every check afterwards accesses already valid stack allocated diff, and marks the diff as valid
At the end of that function we move the stack allocated object into an heap allocated instance, thus hopefully re-using already allocated Vector memory ?

Hmm. We are now creating a PrefabObjectDiff every time that method is called, whether it's needed or not. Sure it's on the stack, which saves a dynamic memory allocation, but all the constructors for all the types it contains still need to be called, and their data initialized. And stuff like std::vector or std::string don't guarantee it won't dynamically allocate something in its constructor either (although most implementations wont).
And then at the end we also need to pay the cost of copying everything back. Again we save on copying dynamic memory since we're using std::move, but there's a lot of non-movable contents that still need to be copied.
Branches should be considerably cheaper than copying any kind of memory around, so I feel it's a hefty cost to pay to eliminate a few branches.

I'll try to verify this on godbolt, see overhead this vs multiple sharedptr constructor calls .

https://godbolt.org/g/XwSzCN
Instruction counts  with -O3 -fno-rtti  -fno-exceptions
       'original' -   'new'
CLANG 6    574    -    437
GCC 8.1    623    -    778
VS17      2543    -   2900

You're right, the original code is also faster according to http://quick-bench.com/sr4MqfrAnyQLRh85zk0g037WZag

Thanks for the detailed analysis. Interesting result with Clang - I'm guessing it sees more optimization opportunities with the stack allocated version?

Yup, the ast is a bit simpler with stack allocated var.
It would be interesting to see how the assembly looks like with EASTL, since their containers are guaranteed not to allocate heap until first push/insert/emplace is done.


NAMCr Sprint

Task 1
Fixing specific issues with the NAMCr package.

#29
#21
#31
#28
#30

Task 2
More general testing of each API endpoint via NAMCr.
Iterate over all API endpoints (use NAMCr::cli() to see a list of the API endpoints) and call them repeatedly over as many permutations of data as possible. Philip will provide a table of API endpoints ranked by priority. The goal is to ensure that all endpoints operate without throwing an error; the most common of which is a bad formatting caused by an erroneous curly brace "{". It is recognized that Frank does not possess the context necessary to interpret the data returned. Therefore the goal is just to check for errors.
I see a few options for how best to provide the arguments to the endpoints:

Philip provides a CSV file for each endpoint containing arguments. For example, for the siteInfo endpoint Philip will provide a CSV of ~ 32,000 sites and Frank will load this CSV and iterate over each row, calling the siteInfo with the values from the CSV.
Some endpoints can be used to retrieve the list of inputs for other endpoints. For example, the sites endpoint can be used to get a list of all sites that are then fed into the siteInfo endpoint.
Philip could code a quick Python script(s) that spits out a giant R script with all the API calls. Frank would then just run this monster script in R Studio and investigate any errors. This avoids the need for Frank to learn the API endpoints and arguments.

A Little Context
The NAMCr package was written before the API was finished. By the time the API was complete we had introduced several endpoints that return data in new structures that cause NAMCr problems. The two suspected data structures that cause NAMCr problems are lists and JSON.
While we explicitly avoided using JSON data types in the API, new data that has been inserted since the system launch has included JSON in some fields. The API should be formatting this in a consistent way, but NAMCr appears to struggle in some cases.

Option 3 above does suggest it would save time for Frank.

@Feldspar5 I just talked to Frank. We agreed to go with option 3 because:

Avoids Frank needing to learn the system. He just runs the giant script of R commands.
The giant script of R commands can be reused as a regression test in the future. We can quickly run the R commands after any change to the system to reveal if the change breaks an R command.


doesnt work

can someone give me a cheat engine like bepinex because it doesnt work

doorstop is patched, we are working on a solution


Support special x-lucust-X fields

for example:
...
    "paths": {
        "/availability": {
            "get": {
                "x-lucust-min" : 50,
...

Support x-locust-default-value for fields.
Support x-locust-min and x-locust-max on endpoints (or on root level?)

Fixed by #5


Swolmanac instead of words.

Swog = swolmanac the blog
Swolmanac =

http://slog.stronglifters.com/ it is.


Add minimal fix for getStyles and css entry point #3

Here is a more minimal fix to make the getStyles method return css entry points.
I can confirm that in my setup, using this fix with the other one here fixes the issue.

Thanks for your patience and the PR of course!


attribute 'exclude_srcs_packages' is missed on npm_packages

Traceback (most recent call last):
File "/private/....c3f5902eaf83e2fcc574a67b7a2c04ab/external/aspect_rules_js/npm/private/npm_package.bzl", line 159, column 41, in _impl
exclude_srcs_packages = ctx.attr.exclude_srcs_packages,
Error: No attribute 'exclude_srcs_packages' in attr. Make sure you declared a rule attribute with this name.
Available attributes: _action_listener, _config_dependencies, _windows_constraint, allow_overwrites, applicable_licenses, compatible_with, deprecation, exclude_srcs_patterns, exec_compatible_with, exec_properties, expect_failure, features, generator_function, generator_location, generator_name, include_external_repositories, include_srcs_packages, name, out, package, replace_prefixes, restricted_to, root_paths, srcs, tags, target_compatible_with, testonly, toolchains, transitive_configs, version, visibility

Starting with RC3 the above attribute is missing causing npm_package fails.
Confirmed that this attribute is missed, but also it is used internally in the _impl function

Your WORKSPACE dependency on aspect_bazel_lib is likely behind what rules_js requires: https://github.com/aspect-build/rules_js/blob/59e56921652a1942520d3f16283acf4b46853c6c/js/repositories.bzl#L39.

Thanks a lot I will try it.

Just to confirm that upgrading the dependency fix the problem. Thanks a lot for your reply. A bigger thanks for this awesome rules.