Simple Functions

Is your feature request related to a problem or challenge?
Verbosity
Currently implementing a scalar function is a pretty involved process. For example a simple function calculating greatest common divisor looks like this (a lot of detail omitted for brevity):
impl ScalarUDFImpl for GcdFunc {
    fn name(&self) -> &str {
        "gcd"
    }

    fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
        Ok(Int64)
    }

    fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
        make_scalar_function(gcd, vec![])(args)
    }
}

/// Gcd SQL function
fn gcd(args: &[ArrayRef]) -> Result<ArrayRef> {
    match args[0].data_type() {
        Int64 => {
            let arg1 = downcast_arg!(&args[0], "x", Int64Array);
            let arg2 = downcast_arg!(&args[1], "y", Int64Array);

            Ok(arg1
                .iter()
                .zip(arg2.iter())
                .map(|(a1, a2)| match (a1, a2) {
                    (Some(a1), Some(a2)) => Ok(Some(compute_gcd(a1, a2)?)),
                    _ => Ok(None),
                })
                .collect::<Result<Int64Array>>()
                .map(Arc::new)? as ArrayRef)
        }
        other => exec_err!("Unsupported data type {other:?} for function gcd"),
    }
}

/// Computes greatest common divisor using Binary GCD algorithm.
pub fn compute_gcd(x: i64, y: i64) -> Result<i64> {
   ...
}

This function is still "relatively simple" because:

it handles only int64 type (has no overloads)

a more complicated function would calculate return type based in input types in return_type and again in invoke

doesn't not support constant x or constant y parameters

the make_scalar_function  helper  expands scalar constant argument into an array, thus gcd(x, 5) is less efficient than gcd(x, y) because it first needs to allocate a temporary buffer for 5s repeated times the batch length

it doesn't handle dictionaries or run-end (REE / RLE) encoded values

i doubt people frequently calculate gcd on partitioning keys, but this is just an example

it doesn't handle all-non-null case, so probably misses optimization/vectorization opportunity

It should be possible to express functions using simple row-level operations, because in query engines most function logic is structured like that anyway (compute_gcd in the example).
Local data structures
Currently DataFusion functions are singletons plugged into the execution engine. They have no way to store and reuse buffers or compiled regular expressions, etc.
Thread local storage is not an option because DataFusion, when embedded, does not control creation of application threads.
Describe the solution you'd like
Simple
It should be possible to separate function logic from the mechanics.
Exemplary GCD function needs to provide fn compute_gcd(x: i64, y: i64) -> Result<i64> and the rest of boilerplate should not be hand-written for every function separately.
It should be possible to implement a function that accepts string values, without having to deal with the 5 different runtime representations that can carry string values: StringArray, LargeStringArray, StringViewArray, DictionaryArray<Key>, RunArray<Key> (maybe more than 5 because they are recursive in theory: can DictionaryArray contain a DictionaryArray? can it contain RunArray?)
Focused
Because SQL is statically typed, it is necessary to select function overload during planning time, so that return type is also known (the return_type function). The invoke function needs to implement same logic again. This process should be less error-prone: once the function is bound to the query call site, its signature is known and the implementation should not need to do type checks.
Performant / Efficient
It should be the compiler's / framework's job to provide vectorization, without hand-writing same logic in every function separately.
It should be the compiler's / framework's to do null checks, providing efficient tight loops for the all-non-null case without having to write such logic in every function separately.
It should be possible to write efficient functions that need thread local data structures, for example for regular expressions, without having to use thread locals and/or shared pools which introduce contention.
Describe alternatives you've considered

https://github.com/apache/datafusion/issues/11413

However, direct use of the library is not possible because

it operates on Arrow arrays directly, doesn't fit DataFusion execution model
it operates on Arrow data types, but DataFusion needs its own type system: https://github.com/apache/datafusion/issues/11513
The problem space for a query engine is much more complicated

The library could still be part of the solution, but doesn't have to be and it's a non-goal to use a particular library.
Additional context

https://github.com/apache/datafusion/issues/11513
https://github.com/apache/datafusion/discussions/7421
How function definitions look like in Trino, for example a function operating on Int64-equivalent https://github.com/trinodb/trino/blob/e6aabfec55fcf5e4dd3c283b9a29da97c1301f6c/core/trino-main/src/main/java/io/trino/operator/scalar/MathFunctions.java#L128-L135
Velox "Simple Functions" https://vldb.org/pvldb/vol15/p3372-pedreira.pdf

cc @alamb, @andygrove, @jayzhan211, @ozankabak, @notfilippo, @comphead

I think the idea of making it easier to write functions that include specialized implementations for different types is a great idea. This would likely both make our code faster (more specializations, more uniformly handle constants) as well as reduce the replication (there are several different patterns for dispatch / specialization now -- consolidating would be really nice)

Thanks @findepi I think this process go through iterations, and easier than was before but still far from perfect.
The ScalarUDFImpl common trait is already a huge help, but for example dynamic typing, like MAX/MIN/GREATEST when return type depends on input is gonna be challenging, in Trino it seems also implemented in runtime rather than static.
Rust can benefit from macros and generate functions with all possible output types combinations, but this is just 1 problem.
I'm totally down if we can make it easier

Just to be clear, what I am imagining comes out of this work is:

No changes to ScalarUDFImpl
Some new way (generic functions, macros, etc) that would make creating the specialized function implementations (perhaps following the pattern of declarative macros on https://github.com/apache/datafusion/issues/11413)

FYI i touched upon the topic of types on DataFusion meetup in Belgrade yesterday.
The slides are here if anyone is interested: https://docs.google.com/presentation/d/1VW_JCGbN22lrGUOMRvUXGpAmlJopbG02hn_SDYJouiY . It was an attempt to summarize why we need both: simpler types (https://github.com/apache/datafusion/issues/11513), more types (https://github.com/apache/datafusion/issues/12644), and simple function "SDK" (https://github.com/apache/datafusion/issues/12635).
The document has comments disabled to avoid diverging the discussion from the github issue.

. It was an attempt to summarize why we need both: simpler types (https://github.com/apache/datafusion/issues/11513), more types (https://github.com/apache/datafusion/issues/12644), and simple function "SDK" (https://github.com/apache/datafusion/issues/12635).

It was also a very nice and understandable talk

I experimented with this on the way from DataFusion meetup in Belgrade.
i came up with something like this
function author would write this
row-oriented function logic
// hand-written
pub struct MyGcd {}
impl MyGcd {
    pub fn call(x: i64, y: i64) -> Result<i64> {
        datafusion::functions::math::gcd::compute_gcd(x, y)
    }
}

a macro would generate
generated (syntactically derived)
impl datafusion::logical_expr::ScalarUDFImpl for MyGcd {
    fn as_any(&self) -> &dyn Any {
        self
    }

    fn name(&self) -> &str {
        "MyGcd"
    }

    fn signature(&self) -> &Signature {
        todo!() // TODO here we should use logical typyes
    }

    fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
        todo!() // TODO here we should use logical typyes
    }

    fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
        lifted_invoke::<MyGcd>(self, args)
    }
}

impl Invocable for MyGcd {
    const ARGUMENT_COUNT: u8 = 2;
    type ArgumentTypeList = (i64, (i64, ()));
    type OutArgType = ();
    type FormalReturnType = Result<i64>;

    fn invoke(&self, regular_args: Self::ArgumentTypeList, _out_arg: &mut Self::OutArgType) -> Self::FormalReturnType {
        let (a, (b, ())) = regular_args;
        MyGcd::call(a, b)
    }
}

All the heavy lifting happens inside lifted_invoke (source https://gist.github.com/findepi/f1b0b84fc5f2002cb3d31b19a5bb983e#file-library-rs)
Notes on API design:

type ArgumentTypeList = (i64, (i64, ())); -- this  form of list presentation yields to template programming
OutArgType  + FormalReturnType -- we need both to support plain returning functions (numerics) and out args (for returning allocatable stuff like strings or structs)
FormalReturnType = Result<i64>; -- lifted_invoke supports functions fallible and infallible functions, as well as nullable return values (via Option<T>)

Note:

for continuing work on this would be best to have logical types defined (https://github.com/apache/datafusion/issues/12622). @notfilippo could we perhaps have a logical types stub in main at some point?

@notfilippo could we perhaps have a logical types stub in main at some point?

Planning on opening the PR on main soon. Then once it gets merged I can merge main in logical-types and continue my work.

I experimented with this on the way from DataFusion meetup in Belgrade.
i came up with something like this
function author would write this
row-oriented function logic
// hand-written
pub struct MyGcd {}
impl MyGcd {
    pub fn call(x: i64, y: i64) -> Result<i64> {
        datafusion::functions::math::gcd::compute_gcd(x, y)
    }
}

a macro would generate
generated (syntactically derived)
impl datafusion::logical_expr::ScalarUDFImpl for MyGcd {
    fn as_any(&self) -> &dyn Any {
        self
    }

    fn name(&self) -> &str {
        "MyGcd"
    }

    fn signature(&self) -> &Signature {
        todo!() // TODO here we should use logical typyes
    }

    fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
        todo!() // TODO here we should use logical typyes
    }

    fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
        lifted_invoke::<MyGcd>(self, args)
    }
}

impl Invocable for MyGcd {
    const ARGUMENT_COUNT: u8 = 2;
    type ArgumentTypeList = (i64, (i64, ()));
    type OutArgType = ();
    type FormalReturnType = Result<i64>;

    fn invoke(&self, regular_args: Self::ArgumentTypeList, _out_arg: &mut Self::OutArgType) -> Self::FormalReturnType {
        let (a, (b, ())) = regular_args;
        MyGcd::call(a, b)
    }
}

All the heavy lifting happens inside lifted_invoke (source https://gist.github.com/findepi/f1b0b84fc5f2002cb3d31b19a5bb983e#file-library-rs)
Notes on API design:

type ArgumentTypeList = (i64, (i64, ())); -- this  form of list presentation yields to template programming
OutArgType  + FormalReturnType -- we need both to support plain returning functions (numerics) and out args (for returning allocatable stuff like strings or structs)
FormalReturnType = Result<i64>; -- lifted_invoke supports functions fallible and infallible functions, as well as nullable return values (via Option<T>)

Note:

for continuing work on this would be best to have logical types defined ([EPIC] Decouple logical from physical types #12622). @notfilippo could we perhaps have a logical types stub in main at some point?

Thanks @findepi I was thinking about proc macros as well, having the the original function I hope we can extract necessary information and generate the DF code and the documentation, I do some testing in https://github.com/apache/datafusion/pull/12822.
If we think about simplication on this phase as to generate boilerplate code leaving the developer to implement the logic it sounds achievable

    pub fn call(x: i64, y: i64) -> Result<i64> {
        datafusion::functions::math::gcd::compute_gcd(x, y)
    }

Wouldn't this incur a significant amount of overhead calling a function for every single row?

@Omega359 this is how the function logic is structured anyway -- https://github.com/apache/datafusion/blob/a08f923c2acb1a46614970231d9a672c36ce3ad2/datafusion/functions/src/math/gcd.rs#L85

@Omega359 this is how the function logic is structured anyway

🤔


Use six to fix python 2.7 compatibility error

As "urllib.request" is not a valid import in Python 2.7, the use of six compatibility package is required to fix backwards compatibility

Did you test it both with Python 2.7 and 3.x?
Anaconda env https://conda.io/docs/user-guide/tasks/manage-environments.html
can help.

I disabled temporally unit tests in continuous integration (CI) because they were broken (see https://github.com/ig-python/ig-markets-api-python-library/issues/35 )
but I enabled PEP8 checks in CI https://github.com/ig-python/ig-markets-api-python-library/issues/37
So merging anything I'd prefer that PEP8 checks pass successfully.
Your contributions are welcome.

HiHope that with my last commit it works also following PEP8 checks.
El Lunes 28 de agosto de 2017 18:45, femtotrader<EMAIL_ADDRESS>escribió:

I disabled temporally unit tests in continuous integration (CI) because they were broken (see #35 )but I enabled PEP8 checks in CI #37So merging anything I'd prefer that PEP8 checks pass successfully.Your contributions are welcome.—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

https://travis-ci.org/ig-python/ig-markets-api-python-library/jobs/269389560 shows PEP8 errors.
I think you should create first an other PR to solve PEP8 issue.
You can run locally PEP8 checks using
$ flake8 trading_ig

You need to have flake8 installed. Otherwise, just type:
$ pip install flake8

HiIndeed there are a lot of errors, but most of them are not caused by my changes but come from the current master branch...
El Martes 29 de agosto de 2017 9:12, femtotrader<EMAIL_ADDRESS>escribió:

https://travis-ci.org/ig-python/ig-markets-api-python-library/jobs/269389560 shows PEP8 errors.I think you should create first an other PR to solve PEP8 issue.You can run locally PEP8 checks using$ flake8 trading_ig
You need to have flake8 installed. Otherwise, just type:$ pip install flake8
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

@Oneirag I'm aware of this
see https://github.com/ig-python/ig-markets-api-python-library/issues/37#issuecomment-325411119

Ok, no worries. With this last commit I get no PEP8 errors, neither in Pyhton 3.5.3 nor in 2.7.13.
Hope this helps

What you did in this PR looks very strange
See your diff https://github.com/Oneirag/ig-markets-api-python-library/commit/31ef419a399a988eaab9e634ec642822bfde089a
We should whole files which is very odd.
Moreover see this screenshot

you should first sync your branch with master before creating a branch
See this message "This branch is 6 commits ahead, 3 commits behind ig-python:master."

I also forget to tell you that CI is broken
https://travis-ci.org/ig-python/ig-markets-api-python-library/builds/269651880?utm_source=github_status&utm_medium=notification

Sorry for the confusion. To prevent further messes, I close this PR and open a new one, with all the changes altogether and after forking the last version of the master branch.
Hope this helps


Allow overlapping ranges in case of single stack IPv6.

How to categorize this PR?

/area networking
/kind enhancement
What this PR does / why we need it:
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Release note:

  Allow overlapping network ranges in case of single stack IPv6.

/cc @ScheererJ @DockToFuture

/hold cancel


Vlan trunking

I'm starting to work on the vlan trunking. It's not ready yet, but the docs are there and I'd like commentary on the design.
//cc @gsilvis

@gsilvis, I speced out the changes for node_detach_network. I think this will make it possible to find the right arguments to apply_networking by diffing this with the networks attached, but perhaps we should just change apply_networking to take the diffs directly, with None being delete all. What do you think?

What would you say are some valid options for the channel argument? With VLANs at least, HaaS will be the one who picks the VLAN IDs for networks.

@henn, the idea was, once the rest of this was implemented, to have the show_network call contain a list of the valid channels. for a VLAN network this would just be something like ["native", "253"] if, for example, the driver ended up choosing vlan 253 for the network.
We talked a bit about this this afternoon, and it needs work. In particular, for some drivers (e.g. vxlans), the list of valid channels may be effectively infinite, so listing them all is out of the question. Iirc, @patcable mentioned that he has a juniper switch that let's you use different vlan numbers on different ports -- my understanding is in that case each network would end up basically having the entire vlan space in the list.

Since all we support right now is VLANs, would it make sense to leaves channels till later and go forward with the rest of your proposed changes?
node_connect_network, instead of taking a "channels" argument could have a "native" flag. Users could figure out the VLAN id from show_network (or show_project :)

I wrote some docs to reflect the channels proposal as discussed today.
For posterity, @henn suggested an alternative API, which was very similar, but dealt with things at a less abstract level.
Differences:

subtitute channel with VLAN id and don't have the separate discussion of driver-specific semantics.
possibly disallow selection of a specific VLAN id, and just say "native" or "tagged."

@henn also suggested (in the context of the channel based driver), having a legal channels field on the nic, with similar semantics. He argued that we may run into situations where a node is on a switch which does not support tagging, in which case regardless of what the network thinks is okay, attaching to e.g. vlan/260 isn't going to work.
I don't think the nic channel list is something that deeply affects the implementation now, or disturbs the rest of the API proposal significantly. I have no objection to adding it -- worst case, we never get any hardware with this restriction, and the driver can just always return vlan/*.
Did I miss anyting?

Oh, worth noting: I also started expanding the documentation for other stuff in rest_api.md. I'd like to move in a direction where this file is a full reference, whereas now it documents arguments and urls and such, but you have to know the semantics already.

This addresses #47.
@zenhack: Before merging the code changes, it might be good to verify the security of our implementation against the known VLAN hopping attacks documented on this wikipedia page.
Perhaps that would make a good project for one of the students?

@henn, probably a good intern task, if they've got the bandwidth. The double tagging exploit seems like something that depends on a bug in the switch though, so it's going to be hardware dependant, and I don't know how we go about working around it if it exists -- not having native VLANs isn't really an option.

Update: the double header attack in the wiki article makes more sense to me now; it looks like it can happen if one switch is tagged on the shared port and the other isn't. In that case, the mitigation measures for both of these are things the haas doesn't actually touch itself; the sysadmin is currently responsible for making that change. We should verify that we aren't perturbing the negotiation bits with what we're doing, and perhaps document this as a concern when configuring the network, but this is out of the scope of what HaaS actually does itself.
At some point we may want the complex vlan driver to know about the connections between switches, beyond just trusting that they exist, but for now, that's something that's done manually.

What do people think about the contents of show_network's output? Do we want to be including ownership/access information?

This is officially ready for review. I'll go over the structure with y'all on Tuesday, as planned.

Since we are rewriting the async network code, please address #318

Also, on the network action queue, we still have the problem that a large number of requests on a busy server could perpetually not complete because of DB transactions on related items failing to commit.
Imagine there being a busy server, with a request being added every second. Every time the server tries to process the queue, it takes 3 seconds. When the server is done, it will try to execute db.commit(), which will fail because at the top it used .all(), which is now stale. Each successive time it runs will take even longer and be more likely to fail.
This could be solved if we handled one transaction at a time instead of batching (.first() instead of .all() combined with an ordering). Handling one at a time would also have the advantage of keeping an error on one of the operations from blocking all of the others.

FYI - we decided that we need to have an architectural discussion over what we want the serialization guarantees of HaaS to be

@gsilvis, @lokI8 and I talked about the serialization issues a bit on IRC today. 70c2aad2 was pretty clearly indented to cause strict ordering (though see #420). I don't have strong opinions about the way we should do this, but I think changing it is orthoginal to this pull request, and given that @henn was clearly correct about the intent of the original implementation, I think we should not disturb that. I think we should also fix #420 while we're at it.

It's worth noting that restoring the old semantics is easy --- we just add a modification date to the attachments table and order by that.

I think #420 should be fixed independently.  Because of that, I would argue that this PR should add the modification date, so it doesn't change the process of fixing #420---it keeps the same "semantics" of ordering the database read, and then throwing the order out.

If others agree, I'll submit a separate pr that fixes #420, then pull it back into this branch once it's merged.

While I'm thinking of it --- I seem to recall having a conversation with @gsilvis about relaxing the one-action-per-nic to one-action-per-channel or somesuch, and deciding that there was enough subtlty to defer it. I think the original reason for restricting to one action per nic was that otherwise we'd either have to look at the whole queue to decide if the action was compatible with all of the pending actions (which seemed a bit overly tricky) or do the saftey checks at application time, rather than request time.

@henn, @gsilvis and I had a discussion about the serialization issue. We agreed to re-separate the newtork operations and attachments, and keep serializing things, with the same basic strategy.

Just a note, I pulled the networking actions back out into their own table like we talked about. I still need to run the deployment tests and fix any bugs they uncover, which I'll do tomorrow.

re: #420, I'd like to get this merged first, since it's likely going to have merge conflicts with the fix for that.

Regarding what to do if the user doesn't indicate a specific channel (ie - connect_network/node_connect_network), would it make more sense to explicitly say that we will only connect something as native if they don't specify a channel?
I think that would be closer to expected behavior for a user not trying to user channels.

@henn, per the changes to rest_api.md, we are defaulting to native/untagged in node_connect_network, and this is documented.

Will the null_vlan.py driver still work with get_default_channel()/other channel stuff?

Quoting henn (2015-05-28 19:12:20)

Will the null_vlan.py driver still work with
get_default_channel()/other channel stuff?

It will; the interface hasn't changed enough to disturb what it actually
does (which isn't too surprising, since that's not much).

Re: null_vlan.py, I was wondering if this code (from node_connect_network()) would throw an exception if a channel weren't passed to the null_vlan driver:
driver = _get_driver()
if channel is None:
    channel = driver.get_default_channel(db)

This might be a case where expressing driver capabilities (ala #410) could come in handy.

That's why the if statement is there --- the driver needs a channel passed, but it's optional in the api call, so if it's not provided, we fill it in with the driver default.
re capabilities: A driver which wants to support only VLANs with no trunking support could always just accept only one possible channel (denoting untagged). We do need some way to deal with different underlying network types, but that's not dealt with with this patch. Have a look at the driver refactoring proposal; it touches on this.

@zenhack: Are you saying that the code above wouldn't try to call the null_vlan driver's get_default_channel() if the user didn't pass in a channel?

@henn the opposite; if they supply a channel it will just use that instead of calling the method.

@zenhack: For this PR, I think we either need to implement driver capabilities ala:
driver = _get_driver()
if 'CHANNELS' in driver.capabilities and channel is None:
    channel = driver.get_default_channel(db)

or implement all of the necessary channel interfaces in every driver.
Otherwise, the server will crash when doing operations like trying to connect a network and not passing an explicit channel when using the null_vlan.py driver.
I'd be happy to take a first pass at the capabilities mechanism since it seems to be popping up a lot.

Quoting henn (2015-05-29 11:23:48)

or implement all of the necessary channel interfaces in every
driver.

I did this (with the exception of the nexus, which is done on a
different branch, and is its own can of worms).  Please actually read
the code carefully.  The null driver doesn't need to change because even
after the interface changes, apply_networking still takes a dict (the
values are different), and that's as closely as the null driver looks at
it (it doesn't actually do anything after all).

superseded by #455, closing.


[SPARK-16094][SQL] Support HashAggregateExec for non-partial aggregates

What changes were proposed in this pull request?
The current spark cannot use HashAggregateExec for non-partial aggregates because Collect (CollectSet/CollectList) uses a single shared buffer inside. Since SortAggregateExec is expensive in some cases, we'd better off fixing this.
This pr is to change plans from
SortAggregate(key=[key#3077], functions=[collect_set(value#3078, 0, 0)], output=[key#3077,collect_set(value)#3088])
+- *Sort [key#3077 ASC], false, 0
   +- Exchange hashpartitioning(key#3077, 5)
      +- Scan ExistingRDD[key#3077,value#3078]

into
HashAggregate(keys=[key#3077], functions=[collect_set(value#3078, 0, 0)], output=[key#3077, collect_set(value)#3088])
+- Exchange hashpartitioning(key#3077, 5)
   +- Scan ExistingRDD[key#3077,value#3078]

How was this patch tested?
Checked non-partial aggregates (collect_set and collect_list) worked well for HashAggregateExec in DataFrameSuite.

Test build #60919 has started for PR 13802 at commit 517d7ea.

Test build #60919 has finished for PR 13802 at commit 517d7ea.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60919/
Test FAILed.

Merged build finished. Test FAILed.

Test build #60928 has started for PR 13802 at commit 0506453.

Test build #60933 has started for PR 13802 at commit 88ba697.

Test build #60928 has finished for PR 13802 at commit 0506453.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60928/
Test PASSed.

Test build #60933 has finished for PR 13802 at commit 88ba697.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60933/
Test PASSed.

@maropu this won't work for other hive_udfs since these also maintain internal state, and currently require per group processing. This also has a greater potential of creating out-of-memory errors than the sort based approach.
I do think there is merit in the general ideal; but I think we should be focussing on creating a growable bytes-to-bytes map and creating byte backed mutable ArrayData and MapData implementations.

@hvanhovell oh, I see. okay, I'll check we can implement mutable ArrayData and MapData.
btw, I have some question;

Any reason to use SortAggregateExec for all the non-partial aggregates? It seems it is okay to use HashAggregateExec for non-partial ones except for collect_xxx and hive_udaf.
Why do we have no hashCode and equals in UnsafeMapData? ArrayBasedMapData already has these override functions.

@maropu all aggregates that current set supportsPartial = false cannot be partially aggregated and require that the entire group is processed in one step. So the name is a bit misleading. I suppose we could rename it.
UnsafeMapData is typically part of an UnsafeRow which already implements equals() and hashCode() without requiring its elements to implement these methods (it uses the backing byte array). I suppose we can add these methods.

@hvanhovell As for UnsafeMapData, could you check  #13847?

As for supportPartial, I could understand that collect and hive_udaf has such a limitation though,
how about AggregateWindowFunction? It seems these functions RowNumber and Rank work well even for HashAggregateExec and they dont have the limitation. BTW, we at least need to fix comments for supportPartial, "Currently Hive UDAF is the only one that doesn't support partial aggregation." is incorrect now; https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala#L178

I think we should rename and document supportsPartial to reflect what it actually does.
Rank and RowNumer are window functions. They both rely on ordered evaluation, and they should never be evaluated in any *AggregateExec operator. We have a rule to prevent this in the analyzer.

Thanks for your explanation!

Is it okay to make a new pr to fix these?

What do you want to fix? WindowAggregateFunctions?

No, I'd like to fix incorrect comments.

Could you have a go? Would be great!

okay

@maropu could you close this one? It is not that relevant anymore. Thanks for working on it though!

yea, thanks!


Added right-click context menu

I've theoretically been able to add a context menu.
You can now highlight a word and search with spot.

Current issues:

It simply opens the popup, doesn't really search Search functionality added in 6fd092f
Cannot open the actual popup.  Prohibited by chrome
Please refer
https://stackoverflow.com/questions/13783500/context-menus-in-chrome-extensions

Ping.

Ping.

Apologies haha, busy week at work!


add docs on http constants

@adamw I didn't update the existing docs to use these constants. Let me know if you want me to do that, and if you prefer objects or traits. I wasn't sure the best place to place this new page, so I just made it top-level for now.

Thanks!


[SAC-150][SQL] Drop attribute 'lastAccessTime' in table

What changes were proposed in this pull request?
This patch proposes dropping attribute 'lastAccessTime' in table entity due to difficulty of updating such attribute.
Please refer #150 for more details on the decision.
How was this patch tested?
Manually tested with Atlas 1.1 (embedded HBase/Solr/Kafka) & Spark 2.4.

This closes #150.

I guess this is approved by @bikassaha .
For me, +1.

Thanks @dongjoon-hyun  and @HyukjinKwon! Merging.

Merged into master, atlas-1.1, branch-2.4 respectively, and also internal branches.


Revert "Update dependencies"

This reverts commit 3f7929c29931b1fdc395ca555cfcad875d58b2f1.
Master is currently red for the JDK10 build and it seems to have started failing after this was commited. Checking if this makes the build go green again.

Travis build is green. Merging.


Client hangs on connect

I'm trying to port parts of a javascript project to .NET Core, but when running it, it immediately disconnects, then hangs the thread. It is entirely likely that I am doing something wrong, but I can't seem pinpoint the issue. I never reach any of the event handlers.
    private async Task ConnectToQueue()
    {
        var factory = new MqttFactory();

        var mqClient = factory.CreateMqttClient(new Logger());
        var options = new MqttClientOptionsBuilder()
            .WithClientId(_authInfo.MqttClientId)
            .WithTcpServer(_authInfo.MqttEndpoint)
            .WithTls(certificates: new[] { _rawCertificate })
            .WithCommunicationTimeout(TimeSpan.FromHours(24))
            .Build();

        mqClient.ApplicationMessageReceived += Client_ApplicationMessageReceived;
        mqClient.Connected += MqClient_Connected;
        var connection = await mqClient.ConnectAsync(options);

    }

    private void MqClient_Connected(object sender, MqttClientConnectedEventArgs e)
    {
        var client = sender as IMqttClient;
        client.SubscribeAsync($"DB510/{_productInfo.MacAddress}/commandOut");
        client.SubscribeAsync($"DB510/{_productInfo.MacAddress}/commandIn");
    }

    private void Client_ApplicationMessageReceived(object sender, MqttApplicationMessageReceivedEventArgs e)
    {
        Console.WriteLine(Encoding.UTF8.GetString(e.ApplicationMessage.Payload));
    }

Above yields the following log messages and hangs at ConnectAsync.
VER: Trying to connect with server.
VER: Connecting [Timeout=1.00:00:00]
VER: Connection with server established.
VER: Start receiving packets.
VER: TX >>> Connect: [ClientId=<ClientId>] [Username=] [Password=] [KeepAlivePeriod=15] [CleanSession=True] [Timeout=1.00:00:00]
VER: Disconnecting [Timeout=1.00:00:00]
VER: Disconnected from adapter.
INF: Disconnected.
VER: Stopped receiving packets.

I'll give it a try later today, but I don't think it'll make a difference as the thread hangs on the await ConnectAsync line, thus never reaching the following lines where the while loop is.

Indeed, with the sample code I am stuck in an infinite loop as it fails to connect. How can I find out what the cause of the immediate disconnect is?
`VER: Trying to connect with server.
VER: Connecting [Timeout=1.00:00:00]
VER: Connection with server established.
VER: Start receiving packets.
VER: TX >>> Connect: [ClientId=] [Username=] [Password=] [KeepAlivePeriod=15] [CleanSession=True] [Timeout=1.00:00:00]
VER: Disconnecting [Timeout=1.00:00:00]
VER: Disconnected from adapter.
INF: Disconnected.
DISCONNECTED FROM SERVER
VER: Stopped receiving packets.
VER: Trying to connect with server.
VER: Connecting [Timeout=1.00:00:00]
VER: Connection with server established.
VER: Start receiving packets.
VER: TX >>> Connect: [ClientId=] [Username=] [Password=] [KeepAlivePeriod=15] [CleanSession=True] [Timeout=1.00:00:00]
VER: Disconnecting [Timeout=1.00:00:00]
VER: Disconnected from adapter.
INF: Disconnected.
DISCONNECTED FROM SERVER
VER: Stopped receiving packets.
VER: Trying to connect with server.
VER: Connecting [Timeout=1.00:00:00]
VER: Connection with server established.
VER: Start receiving packets.
VER: TX >>> Connect: [ClientId=] [Username=] [Password=] [KeepAlivePeriod=15] [CleanSession=True] [Timeout=1.00:00:00]
VER: Disconnecting [Timeout=1.00:00:00]
VER: Disconnected from adapter.
INF: Disconnected.
DISCONNECTED FROM SERVER
VER: Stopped receiving packets.
VER: Trying to connect with server.
VER: Connecting [Timeout=1.00:00:00]
VER: Connection with server established.
VER: Start receiving packets.
VER: TX >>> Connect: [ClientId=] [Username=] [Password=] [KeepAlivePeriod=15] [CleanSession=True] [Timeout=1.00:00:00]
VER: Disconnecting [Timeout=1.00:00:00]
VER: Disconnected from adapter.
INF: Disconnected.
DISCONNECTED FROM SERVER
VER: Stopped receiving packets.`

I have downloaded the source, and it hangs on the 24h timeout while trying to authenticate. I suspect a client certificate issue as there is no other auth mechanism other than client id. Is this possible to somehow get more information on from the client? I know the server works as it is possible to connect from node.js.

Which broker do you use? I also got such issues when using TLS with an unencrypted broker or disabled TLS with an encrypted broker.

@chkr1011 I'm not sure what's on the other end. I'm trying to integrate with my robot lawnmower. All I have to go by is some working javascript code from here:
https://github.com/MeisterTR/ioBroker.landroid-s/blob/master/lib/landroid-cloud-2.js
Right now, I'm trying to get a dump from the android app and compare it to what my C# implementation does.

@biapar: I do now own or operate the server. I am accessing a 3rd party system.

I finally managed to track the issue down, and it was a problem with the client certificate. While actually valid, it was not accepted by the server.
I don't know how the protocol responds to this, but if it were possible to add an error somehow, that would've saved me a lot of time :)

Based on what you said, I did some further investigations, and I've come to conclusion that I don't think the client is behaving in any way it shouldn't. It is the result of requesting a client certificate with a wrong authentication token and thus getting a valid certificate, but not for anything meaningful.
Basically, I'm subscribing to a queue that in my book should not exist, and instead of returning the equivalent of a 404 or some other error, it simply subscribes me to an unused queue.
Sorry for the confusion, this one was really difficult to track down. I had to install both the MQTTnet source and the source javascript library with all its dependencies.


HDDS-10713. Run JUnit tests with Java 17

What changes were proposed in this pull request?
Bump Java version to 17 in integration check.
https://issues.apache.org/jira/browse/HDDS-10713
How was this patch tested?
CI:
https://github.com/adoroszlai/ozone/actions/runs/8788103037

Thanks @smengcl for the review.


Voice Route has no associated PSTN Usage

In step 3 we are creating a voice route but the cmdlet is incomplete.
PS C:\> New-CsOnlineVoiceRoute -Identity analog-interop -NumberPattern "^\+1(425)(\d{7}])$" -OnlinePstnGatewayList sbc.contoso.com -Priority 1 -OnlinePstnUsages "
We are missing a OnlinePstnUsages as show above.  I'm not even sure step 3 is required as Step 7 defines the specific analog range and assigns it to the PSTN usage Interop
PS C:\> New-CsOnlineVoiceRoute -Identity analog-interop -NumberPattern "^\+1(4254)(\d{6}])$"  -OnlinePstnGatewayList sbc.contoso.com -Priority 1 -OnlinePstnUsages "Interop"

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ea0bc170-d9d9-c17a-adc8-d37cb5fe3df5
Version Independent ID: 3c761043-bea5-0cfb-d85e-13671ed0fe9a
Content: Direct Routing - Connecting analog devices - Microsoft Teams
Content Source: Teams/direct-routing-analog-devices.md
Service: msteams
GitHub Login: @CarolynRowe
Microsoft Alias: crowe

@dinocaputo Thank you for submitting feedback and contributing to the docs. We are currently investigating this.


Collect timing information for commands running on travis

Adds timestamps to most of our subtasks for test when running on travis (so you know if we spend more time building or testing or linting or what have you).

:+1:


Update exercise 3.7.1 to work with {ggplot2} 3.3.0

The geom_pointrange solution no longer works with ggplot 3.3.0, as fun.ymin, fun.ymax, and fun.y have been deprecated. Replaced that code with new functions fun.min, fun.max and fun.

Thanks!

Forgot to say thank you!!! for this study guide - I'm going through R4DS with a group of coworkers and your solutions manual has been invaluable.


Exchange Onprem's OWA

Hi Team,
Can you please clarify if this feature is support with Exchange Onprem's OWA?
Thanks in advanced.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 37e10f52-d726-d29a-e0a8-93729a32dfae
Version Independent ID: 83555f80-a8ca-ee85-c1fc-e39891029758
Content: Use the Microsoft Teams Meeting add-in in Outlook - Microsoft Teams
Content Source: Teams/Teams-add-in-for-Outlook.md
Service: msteams
GitHub Login: @LanaChin
Microsoft Alias: v-lanac

@scanum  Thank you for submitting feedback and contributing to the docs. We are currently investigating this.

Hi @JoseLampreia  Thanks for raising this issue! I will be looking into it!

@JoseLampreia Yes, this is supported in On-Premises OWA

@JoseLampreia Thank you for submitting feedback. We understand that this issue has been resolved.
Please feel free to re-open this issue if there is a specific area of the docs that we can improve or make better. Thank you.


State between requests?

I understand how to set up a supervisor, set defaults and get the state within an endpoint.  It took a bit of fiddling, maybe it's obvious to experienced Elixir devs.  I'm trying to understand what state is doing and if I should be messing with it.  I can definitely set up defaults in a mix new --sup type app:
# lib/application.ex
...
def start(_type, _args) do
    children = [
      {MyApp, []},   # <-- here is the default state

But it seems like state is used internally?  Ok, so if I want things to survive between requests (imagine a counter), should I just spin up another GenServer/Agent?  Does Ace not speak to this problem (not in criticism but in design)?
Is state just for app defaults maybe, like the README says for greeting: "Hello".  So it can't be mutated (or returned really).

I got something working using a simple Agent.  I keep "app state" out of Ace's boundaries and it seems to work just fine.  It's just vanilla Elixir really.  Sorry for using issues for help but I thought it might be useful for future searches.

Sure no problem. Probably a better forum for questions is  the slack channel, more people are likely to try and offer an answer there. https://elixir-lang.slack.com/messages/C56H3TBH8/
You more or less have the answer, state only lasts the duration of a request/response, it can change but only in the case of streaming, in which case the request response lives longer
If you think it warrants it could you add this particular question to the Raxx FAQ. (we should probably add a link from the Ace project to Raxx FAQ)


python311Packages.amazon-ion: 0.10.0 -> 0.11.2

Diff: https://github.com/amzn/ion-python/compare/refs/tags/v0.10.0...v0.11.2
Changelog: https://github.com/amazon-ion/ion-python/releases/tag/v0.11.2
Description of changes
Fix build (https://hydra.nixos.org/build/241207251)

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Priorities
Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 270380 run on x86_64-linux 1

8 packages failed to build:

python310Packages.ionhash
python310Packages.ionhash.dist
python310Packages.pyqldb
python310Packages.pyqldb.dist
python311Packages.ionhash
python311Packages.ionhash.dist
python311Packages.pyqldb
python311Packages.pyqldb.dist

4 packages built:

python310Packages.amazon-ion
python310Packages.amazon-ion.dist
python311Packages.amazon-ion
python311Packages.amazon-ion.dist

Result of nixpkgs-review pr 270380 run on x86_64-linux 1

8 packages failed to build:

python310Packages.ionhash
python310Packages.ionhash.dist
python310Packages.pyqldb
python310Packages.pyqldb.dist
python311Packages.ionhash
python311Packages.ionhash.dist
python311Packages.pyqldb
python311Packages.pyqldb.dist

4 packages built:

python310Packages.amazon-ion
python310Packages.amazon-ion.dist
python311Packages.amazon-ion
python311Packages.amazon-ion.dist


Dynamic List

Hi, I dont know the types prior to loading, can you help me, I need do something like...
How to get the TypedList from the Excel Or CSV file? I dont know what the col types & headers are ahead of time.

How do I get that DynamicListFromExcel?

var dynamicListFromExcel= JsonConvert.DeserializeObject<List<DynamicListFromExcel>>(testData);

Hi,
Are you asking how to get the column types and headers out of a spreadsheet ahead of time? Something similar to issue #367?

Hi,
Note that creating DB tables and/or JSON is somewhat outside the scope of this project, as ExcelDataReader's primary concern is parsing Excel files and making the data accessible to developers. How to process the parsed data depends on the specific use case, requirements and data quality.
There are some suggestions how to create SQL tables from a DataTable here:
https://stackoverflow.com/questions/1348712/creating-a-sql-server-table-from-a-c-sharp-datatable
Check issue #348 for two examples of generating JSON from Excel without knowing the column names.
Using a search engine you can find more examples how to convert from IDataReader or DataSet instances into JSON or DB tables.


SS4 caching: update Zend_Cache or PSR-6?

Original RFC for upgrading Zend Framework: https://github.com/silverstripe/silverstripe-framework/issues/5359.
ZF1 is EOL, so we need to either upgrade or replace. Since it was created, there's efforts from the PSR standards group around caching, with two competing standards: http://www.php-fig.org/psr/psr-6/ and https://github.com/php-fig/fig-standards/blob/master/proposed/simplecache.md.
Our SS_Cache implementation directly exposes Zend_Cache APIs, so will be hard to replace.
By removing a dependency on Zend_Locale and Zend_Translate, we have also removed indirect dependencies on Zend_Cache.
@kinglozzer has created a PoC for upgrading Zend_Cache at https://github.com/kinglozzer/sapphire/commits/zf2-example
Related

Zend_Translate
Zend_Locale

I think we should switch to using PSR-6 for this, and then providing a default PSR-6 library as a framework dependency. This is what we did with PSR-3 / monolog, too.
I think that we should refactor SS_Cache to accept any PSR-6 implementation and ditch the Zend_Cache support.

This library seems to be one of the more popular PSR-6 implementations:
http://www.stashphp.com/

IIRC, one of the goals of simplecache was that it adds a thin API layer on top of PSR-6 to simplify it.  Whether that’s something that the library would provide, I’m not sure - but I don’t think the two standards are mutually exclusive/competing.
+1 for PSR-6.

Another implementation is symfony/cache, but stash seems to be more popular (https://packagist.org/providers/psr/cache-implementation)

👍  for simplecache over psr-6

SimpleCache is a layer on top of PSR-6, that doesn't current have very widespread implementations. So "SimpleCache over PSR-6" doesn't really make sense. Using SimpleCache means using PSR-6.
That said, I think that SS classes directly accessing PSR-6 is a bit overblown. There needs to be some layer between the two, and we have a few choices for that:

Continue to use SS_Cache in more or less its current form, with SS_Cache connecting to a PSR-6 backend
Remove SS_Cache and replace it with a mechanism for fetching a SimpleCache class. The uncertainty here is that there aren't many popular SimpleCache implementation yet.
Refactor SS_Cache to become a SimpleCache implementation. Remember that SimpleCache doesn't actually do any caching itself, it's syntatic sugar for PSR-6. This might be appropriate if the existing SimpleCache implementations aren't sufficiently lightweight.

This is the only SimpleCache implementation, apparently: http://www.scrapbook.cash/. It provides it as part of a caching library.
At 20,000 Packagist downloads it's less popular than Stash and Symfony cache.

Surely the point of using a PSR-6 caching lib is that we can use SS_Cache and have it powered by any PSR-6 compliant library? So why not just do that and we pick a default backend (symfony seems fine to me)...

If there are no SimpleCache implementations yet, my vote goes to refactoring SS_Cache to be one (assuming it’s not going to be a massive undertaking - which reading between the lines it shouldn’t be). If SimpleCache implementations are then released by other maintainers later, it’ll be a piece of cake to drop SS_Cache and switch to one of them.

So, what it sounds like the implication of what Loz is suggesting is:

Cache::factory() returns a Psr\SimpleCache\CacheInterface object (https://github.com/php-fig/simplecache/blob/master/src/CacheInterface.php)
There's a way of configuring different cache backends by name, proviidng PSR-6 cache pools (https://github.com/php-fig/cache/blob/master/src/CacheItemPoolInterface.php)
The configuration arguments (Cache::set_cache_lifetime() and the $frontend and $frontendOptions arguments of Cache::factory()) no longer work.

To be honest I would probably split out our simplecache implementation as silverstripe/simplecache – it may be of use to other developers, given that a standalone PSR-16 implementation doesn't currently exist (it only comes bundled with another caching library). It would be a PSR-16 compatible wrapper around a PSR-6 caching back-end.
What's not decided yet is how we define the caches. We could provide injected services such as SilverStripe\Core\Cache.<backend-name>. Or we could provide a range of injected services that live within the namespaces of the subsystems using the caches. So, for example, i18n::get_cache() could use the service SilverStripe\i18n\i18n\Cache.
In either case, it also begs the question of whether we need Cache::factory() or we should just get the services from the Injector directly. It seems like an unnecessary extra layer.

For reference, these are the caches that we use in core:

LeftAndMain_CMSVersion: Cache the CMS version derived from composer.lock
CMSMain_SiteTreeHints: Cache the calculation of CMSMain::SiteTreeHints
GDBackend_Manipulations: Retain the pass/fail state of manipulation calls between executions
SS_Configuration: Cache the ConfigManifest
i18n: Passed to Zend_Translate
cacheblock: Cache partial caches

Just to confirm: the new Cache and associated configuration API would be specifically designed for whichever PSR-6 library we choose to bundle with framework, right? Rather than attempting to be generic enough to be able to configure other PSR-6 libraries.

What's not decided yet is how we define the caches. We could provide injected services such as SilverStripe\Core\Cache.<backend-name>. Or we could provide a range of injected services that live within the namespaces of the subsystems using the caches. So, for example, i18n::get_cache() could use the service SilverStripe\i18n\i18n\Cache.

My preference is the latter - my understanding of the first option is that we couldn’t then (easily) set different configuration options for different uses of the same backend. Let’s say I want to override core caches to cache everything with Apc, and have different ttl values for i18n vs the config manifest - could I achieve that with the first option?

In either case, it also begs the question of whether we need Cache::factory() or we should just get the services from the Injector directly. It seems like an unnecessary extra layer.

I think it could still be useful as a convenience layer. We could use Injector to map short names for drivers to full classes:
$cache = SilverStripe\Cache\Cache::factory('Apc', ['ttl' => 600]);

// vs...

$cache = SilverStripe\Core\Injector\Injector::inst()->get('SilverStripe\Core\Cache.Apc');
$cache->getDriver()->setOptions(['ttl' => 600]);

There’s not a huge difference there, but it should be a pretty thin API layer to maintain for the added convenience.

Just to confirm: the new Cache and associated configuration API would be specifically designed for whichever PSR-6 library we choose to bundle with framework, right? Rather than attempting to be generic enough to be able to configure other PSR-6 libraries.

If we provided a configuration API then I would say so, but my preference would be that we just pass responsibility for configuration over to the 3rd part API and don't attempt to build any kind of configuration adapter at all — I don't think it would be a value-adding layer.

This is the only SimpleCache implementation, apparently: http://www.scrapbook.cash/. It provides it as part of a caching library.

Looks like it’s a SimpleCache implementation specifically for scrapbook, but not a PSR-6 “wrapper” as such.
PSR-16/SimpleCache is under review at the moment, and these messages ([1] and [2]) seem to suggest that a PSR-6 to PSR-16 bridge will live in a simple-cache-utils package. If that’s the case, then we likely wouldn’t need to maintain our own implementation once (and assuming) PSR-16 is approved.

This might be a good time to come up with a solution for the cache cross-pollination issue raised by Nicolaas - we basically need a way for the system to influence the cache key, e.g. by adding a domain.

Having spent a bit of time working with caches in the last few days, but NOT having read all the stuff above or understanding it all ... Here are a few things I have thought of:

It would be great if you could change out SS_Cache class for something else.  If SS_Cache were to  extend Object then that is easy AFAIK (using a custom class directive).

In my work, I have not been too worried about cache expiry times.  Basically, I would not let caches expire by themselves, but a flush would remove ALL of them.  The idea that you would say: this cache expires in an hour and this one in two hours just seems rather complicated to fully understand in its implications and management.  Rather than cache key expiry, you could perhaps simply overwrite cache values, clear it or point to new ones.

In my frustration working with memcache (https://github.com/silverstripe/silverstripe-framework/issues/6383), I started to write my own cache and came across the idea of using a Mysql table with set length fields (CHAR) stored using Engine=MEMORY (a MYSQL table that is stored in RAM in its entirety) together with an InnoDB table for longer strings.  I  like this idea as the default cache for Silverstripe because:
-- (a) Mysql Memory table is exactly made for such a thing.
-- (b) MYSQL is available and there is no need to set up anything additional (such as REDIS / Memcache)
-- (c) it is super easy to inspect the values being stored (where reading values in, for example, memcache takes a lot more effort).
-- (d) I imagine it to be faster and easier to work with than the current default of temp files.
-- (e) as a database is always linked to one site, there is no question about cross-pollination that you may have with temp files and/or memcache
This is only an idea right now, but . I would be super happy to research it if any of you thought that this idea might be a good one.

A caching system should be a storage place for keys with associated values (like a one dimensional  associative array) that you are happy to lose at any time (flush, server reset) ...  nothing more.  Keeping it really simple will make it faster and easier to understand.

Sorry for the long rant.  I hope it helps in getting a better caching system soon ;-)

@sunnysideup I appreciate you’ve had a few frustrating issues with caching recently, thanks for your work in tracking them and their causes down. We might not be able to fix them all, or fix them cleanly, in the 3.x releases but it’s helpful to know what problems we need to avoid in future.
I won’t touch on all of your points above, but we’ll definitely avoid writing our own caching system from scratch. Caching is deceptively hard to get right, and there are many mature, well-tested libraries out there that handle it very well (doctrine/cache, symfony/cache, tedivm/stash to name a few). If you’re thinking of writing your own for use in a live project, I would recommend instead investing the time in using a library like one of those listed above (of course, it could still be a fun side project to play around with!).

@kinglozzer  - thank you for your reply. Totally agree with the maxim of using something existing and that being better than homebaked.
Having said that, you should consider this: because the current caching system is hard to understand and tries to allow for so many options, with different settings, etc... it basically meant that SS 3 did not have a proper caching system beyond file caching.
In other words, you may move the grunt work of the caching to a third-party system, but if that third-party system is complex and has many options then you may end up writing a larger amount of code to deal with the options then the amount of code required to write our own solution.
Thus, I would advocate for something really fast and simple with the ability to switch it out for something more complex.

One problem you may (probably will) encounter is that there's no way (or its hard/hacky) to change to a different cache backend.
Currently you can do:
define('SS_MANIFEST_CACHE', 'MyCustomClass');

That works perfectly fine for Filesystem cache or APC, but as soon as you want to use something like redis which requires configuration, this falls over.
eg.
$adapter = new FilesystemAdapter;

$redisClient = new Predis\Client([ // .. some config ... ]);
$adapter = new RedisAdapter($redisClient);

The user doesn't really have a way to configure redis and then pass that redis client into the adaptor.
This is one example but there's lots of other cache backends which have different dependencies and require different configuration. https://github.com/symfony/cache/tree/master/Adapter
Using config won't be possible here as config will also rely on the cache.
There's probably a wider discussion on bootstrapping needed but I thought I'd just leave this brain dump here for now.

I think that we should assume that for things other than the cache of config, config will be available to define the cache set-up.
Config can have a more hard-coded cache set-up.
As a first cut, we can just dish out PSR-6 CachePool objects from the Injector, and have a hardcoded PSR-6 CachePool object in Core.php for the config cache.

I've updated the description of this ticket to reflect some recommendations, and sprinkle in my own research (and opinions). Personally I think exposing SilverStripe devs to PSR-6 vs. the simpler PSR-16 isn't a big deal, it's three instead of two lines for an average cache key lookup/write. That being said, Symfony cache supports both. I'm tending towards StashPHP (PSR-6 only), because of nice features like cache stampede protection.

I used stash briefly in my config work but found it to be quite slow and didn't have as many cache backend options as symfony.
so +1 for symfony cache.

@micmania1 Did you compare this with Symfony Cache on the same workload afterwards? Both libs have roughly the same amount of code, so intuitively I wouldn't expect there to be a lot of performance difference.

@silverstripe/core-team We're planning to implement this in the next days. Can you please state your preference for one of the outlined options in the updated ticket descriptions, and whether you have a library preference? Also looking for feedback on the configuration and pool management approach.
So far, we have the following opinions (please let me know if I'm misrepresenting).
Standards:

PSR-6: @dhensby (?), @sminnee (?)
PSR-16/simplecache: @tractorcow @kinglozzer (?), @chillu

Libraries:

Symfony Cache: @micmania1, @dhensby, @chillu

Options:

Option 1a: Replace SS_Cache with a PSR-6 library: @sminnee (?)
Option 1b: Adapt SS_Cache to use a PSR-6 library internally: @dhensby
Option 2a: Replace SS_Cache with a PSR-16 library ("simplecache"): @chillu, @kinglozzer (changed from "Option 2c" because we now have simplecache libraries available)

Note that I've changed my opinion to use PSR-16 and Symfony Cache, since I've realised that Symfony Cache implements both standards. In reality, the lack of Stampede Protection in Symfony Cache isn't a big deal, since sites needing it can simply extend the built in cache pool interface.

I'd be okay with something like:

Different caches are identified as named services
Cache services must comply with PSR-16 (or PSR-6, happy to go with Ingo's preference)
We bundle symfony/cache as a framework dependency, as we do with monolog
Our default yaml defines symonfy/cache services for our default caches

Basically, give it the same treatment that we gave logging.

I’m still voting for option 2a. I think whichever library we choose, it’s going to offer enough flexibility for 99.9%. For the 0.01% that do want a different library, they can either roll their own code if it’s “userland” or write a PSR-16 adapter if they really need it for SS_Cache. That may be more difficult for some libraries than others, but at least it’s possible - so it’s a significant improvement on the status quo. I’m happy with Symfony too

I'm changing my vote to PSR-6 and symfony/cache.
Also vote for 1b: Option 1b: Adapt SS_Cache to use a PSR-6 library internally:, where we use SS_Cache as a factory for generating a cache "using whatever we've declared as the default". E.g. a Filesystem cache with a standard cache path set by silverstripe.

OK so based on chats I've updated standards preferences to be:

PSR-6: @dhensby, @tractorcow
PSR-16/simplecache: @kinglozzer, @chillu, @sminnee

So it sounds like the main point of contention is retaining SS_Cache as an API (Option 1b and 2b, two votes), or replacing it with PSR-6/PSR-16 (Option 1a and 2a, three votes).
Looking at the SS_Cache API, we have the following methods:

add_backend($name, $type, $options = array()
pick_backend($name, $for, $priority = 1)
get_cache_lifetime($for)
set_cache_lifetime($for, $lifetime = 600, $priority = 1)
factory($for, $frontend = 'Output', $frontendOptions = null) (returns a non-PSR Zend_Cache object)

Just to clarify, is anybody suggesting to create a "PSR-6/PSR-16 to Zend_Cache" adapter, to avoid any existing cache users changing their code? Or is this just about retaining SS_Cache::factory() as an entry point and wrapper around a DI service creation based on the $for argument, while still returning a PSR-6/PSR-16 cache interface?
Sorry, the wording of the options ("internally") implied that we weren't requiring devs to change their own code use, and retain the Zend_Cache_Core signature: load($id, $doNotTestCacheValidity = false, $doNotUnserialize = false) and save($data, $id = null, $tags = array(), $specificLifetime = false, $priority = 8). That doesn't sound feasible, so even Option 1b and 2b would expose PSR-6/PSR-16 cache interfaces to SilverStripe devs.

Ingo I think that level of detail is best addressed at the PR stage. Might be best to start coding?

I'm becoming less attached to the idea of keeping SS_Cache. What I was concerned with was making sure we have a unified API where backends are easily swapped out. It sounds like that can be solved without keeping SS_Cache, right?
If so, I'm happy to go with the crowd on this one

The reason I'd be against keeping SS_Cache is that PSR's are there to agree upon an API and if we add our own layer, we're not really buying into PSR's. Also, it means devs coming to SS would need to learn our API instead of using what they already know.

I'm becoming less attached to the idea of keeping SS_Cache. What I was concerned with was making sure we have a unified API where backends are easily swapped out. It sounds like that can be solved without keeping SS_Cache, right?

I was imagining something like:
$partialCache = Injector::inst()->get('Psr\Cache\CacheItemPoolInterface.Partials');
$item = $partialCache->getItem('foo');
if($item->isHit()) {
  return $item->get();
} else {
  $value = doSomething();
  $item->set($value);
  return $value;
}

Instead of the base service name Psr\Cache\CacheItemPoolInterface we could just use Cache.

Status update: I've started implementing this and hitting some conceptual road blocks. I've identified a few implied requirements from the 3.x-style API:

Globally exchange cache driver for all cache implementations without knowledge of which named caches exist
Set different cache driver for one particular named cache
Set default cache lifetime without duplicating all configuration for the cache driver(s)
Allow creating a new named cache without writing a bunch of boilerplate YAML config

In Symfony's PSR-16 cache drivers, this is achieved by $namespace and $defaultLifetime constructor args. These differ based on implementation (argument position).
FilesystemCache::__construct($namespace = '', $defaultLifetime = 0, $directory = null)
MemcachedCache::__construct(\Memcached $client, $namespace = '', $defaultLifetime = 0)

Every cache has it's own namespace in order to be clearable without affecting other caches (cacheblock, GDManipulation, LeftAndMain_Version, etc). In the filesystem driver, this translates to subfolders in TEMP_FOLDER. Because of the constructor arg use in Symfony (rather than setters), we need to define an entire YAML injector configuration for each of these variations. Here's a mock up of how this would look - using different drivers to illustrate the issue, but it would be the same amount of config when using the same driver for both (less with some YAML referencing magic):
---
Name: corecache
---
SilverStripe\Core\Injector\Injector:
  Cache.default:
    class: 'Symfony\Component\Cache\Simple\FilesystemCache'
    constructor:
      namespace: 'default'
      defaultLifetime: 300
      directory: `TEMP_FOLDER`
  Cache.GDBackend_Manipulations:
    class: 'Symfony\Component\Cache\Simple\MemcachedCache'
    constructor:
      client: '%$MemcachedCacheService
      namespace: 'GDBackend_Manipulations'
      defaultLifetime: 100

Namespace option 1: Create lightweight driver-specific factories return PSR-16 objects, e.g. SilverStripe\Core\Cache\FilesystemCacheFactory and SilverStripe\Core\Cache\MemcacheCacheFactory. These would pass through constructor args, but also take a $namespace argument separately in order to insert it in the correct place depending on the constructor method signature.
Namespace option 2: Use StashPHP + PSR-6, which means we can use Pool->setNamespace() and consistently rely on the SS Injector with a minimal Cache::factory() wrapper to create these services. Since StashPHP doesn't support PSR-16, this automatically means exposing PSR-6 to SilverStripe devs (unless we want to write our own adapters).
Namespace option 3: Drop the "Allow creating a new named cache without writing a bunch of boilerplate YAML config" requirement. This would mean any project would need to know about all cache implementations used throughout core, modules and project code, and duplicate the same YAML config block for each of them to consistently set a different driver (e.g. exchanging filesystem for memcache).

I think that option 1 is the best road to go down. I'd define a CacheFactory interface for these factory objects, and have all the default cache services use something like this in their service definition
Cache.partials:
  factory: %$CacheFactory
  properties:
    Namespace: "partials"

Encourage module developers to do the same.This will mean that the CacheFactory service can be redefined and all default caches will be replaced.

It's your call as to whether the cache factory interface needs a setLifetime as sell

@chillu I amended my example above after looking at how Injector actually works. ;-) Principle is the same

Oh the other thing you could do is have a generic cache factory that lets you define the argument order:
SilverStripe\Core\Cache\CacheFactory:
  class: 'SilverStripe\Core\Cache\DefaultCacheFactory'
  properties:
    Class: 'Symfony\Component\Cache\Simple\FilesystemCache'
    Constructor:
     - '$namespace'
     - '$lifetime'
     - `TEMP_FOLDER`

Or
SilverStripe\Core\Cache\CacheFactory:
  class: 'SilverStripe\Core\Cache\DefaultCacheFactory'
  properties:
    Class: 'Symfony\Component\Cache\Simple\ MemcachedCache'
    Constructor:
     - '%$MemcachedCacheService
     - '$namespace'
     - '$lifetime'

With code something like this:
class DefaultCacheFactory implements CacheFactory {
  public $Class, $Constructor;

  function create($namespace, $lifetime) {
    $mappings = [ '$namespace' => $namespace, '$lifetime' => $lifetime ];
    foreach($this->Constructor as $arg) {
      $mappedArgs[] = isset($mappings[$arg]) ? $mappings[$arg] : $ar;g
    }
    return Injector::inst()->create($this->Class, $mappedArgs);
  }

Pull requests: https://github.com/silverstripe/silverstripe-framework/pull/6648 and https://github.com/silverstripe/silverstripe-cms/pull/1741


add generic botorch base optimizer

This PR brings a fairly generic BOTorch based optimizer to the table, which we can flexibly use with various models and acquisition functions. For a simple example, see the reference tests implementation.
Signed-off-by: Grossberger Lukas (CR/AIR2.2<EMAIL_ADDRESS>

I think this PR is ready to be merged. I have no further comments. Thank you @LGro!


shadcn-ui@latest init doesnt work on node v20.6.0

> node -v
v20.6.0
> uname -a
Linux fedora 6.2.9-300.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 30 22:32:58 UTC 2023 x86_64 GNU/Linux

Tried the same project with shadcn-ui@latest init with node -v lts (v18.17.1) and it worked correctly.
To reproduce
nvm use node # get the latet sone using nvm install node (in this case v20.6.0)
pnpm create next-app@latest my-app --typescript --tailwind --eslint
pnpm dlx shadcn-ui@latest init

I'm facing the same issue with Node 20.6.0 but it works properly with Node 20.5.1

It's possibly outside the repo for now since babel (and even node) itself is issuing in the same way
https://github.com/babel/babel/issues/15927

Hello @Bonchouuuur! Since the problem is solved, and it was on the Node.js side, maybe the issue could be closed?

Yes ! @arnvgh can you close it if update Node with 20.6.1 solved your issue ?

Yep sure, closing it as we got the issue solved.

Oops, my bad, I thought you were the author of the issue 😅


Find/Search window broken on wayland

Description
When opening the find window the windows can either float around, disappear when typing, or do not show at all.
Screenshot example:

Log files
dbeaver-debug.log
hidden_log.log
DBeaver Version
Community Edition <IP_ADDRESS>411031831
Operating System
Ubuntu 22.04.5 LTS
Database and driver
I do not think it matters
1.MS SQL Server / SQL Server 12.8.0.x64
Steps to reproduce

Connect to database
Open/Created SQL script.
Open the find window (Ctrl + f)

Additional context
Using Wayland Session
Java Version: openjdk 11.0.17 2022-10-18
No external GPU using  i7-1165G7

Thanks for the bug report.
This new Find/Search dialog comes to DBeaver from Eclipse. However, we need to reproduce and check it, before claiming that it's not a DBeaver issue.
Here is a way how to return back old dialog https://stackoverflow.com/questions/79012836/eclipse-2024-09-is-there-a-way-to-get-the-old-find-replace-window-back/79013331#79013331 Please let me know, if it helps.

Indeed using the above fix (disabling the overlay) makes the search functionality usable again.

Here is an eclipse bug https://github.com/eclipse-platform/eclipse.platform.swt/issues/1447
It's already fixed. So, probably the problem will be solved in the next Eclipse version.


List successful cases as PASS! for 7.27

Fixes #373

Ok, I left it as INFO because I wasn't 100% sure of the accuracy of awk '/Principal/ || /Condition/ && !skip { print } { skip = /Deny/} ') but, if in your opinion and after using it for long, you think that is all right, I can change it.


Problem on android 11

It doesn't work on android 11 it makes internet connection so slow when I activate exception and look in android studio logs it always says read socket timeout exception!!

Do you mind expanding a little bit. This is android to socket you define where the socket leads.

https://github.com/DrBrad/Android-VPN-to-Socket/blob/master/app/src/main/java/vpntosocket/shadowrouter/org/vpntosocket/VPN/Proxy.java

Problem is on Proxy.java class where you  read data from the server! It waits to read data from server for long time but it doesn't success and it throws SocketReadTimeoutException.

Your code is great and understandable 👏👏👏 .Is it possible that you have a problem with ip headers you didnt set cheksum and change the length of the data in ipheaders Or you think these fields are not important? I have no so much info about ip packets!!

It would surprise me if the issue pertained to any of the headers, however its possible. I would guess still that its somewhere in the proxy.java.
Would you be willing to post the error log?

When you route only one ip address .addRoute("<IP_ADDRESS>",32) and remove .addRoute("<IP_ADDRESS>",0)  it works perfectly. So I think  problem is  service.protect(server) in first thread it protect the server but in second and in the other thread server connection it cannot protect new server connections so problem begins here it waits for long time to connect server and then it throws SocketReadTimeOutException !!!

Maybe we need to protect after the connect on Proxy.java?

That or it may be an issue with bad passing of the intent?

Try to protect serverSocket in the below part not the server in multiple thread!!! I will try it tomorrow!
Socket socket;
ServerSocket serverSocket = new ServerSocket(port);
port = (short) (serverSocket.getLocalPort() & 0xFFFF);
Log.e("info", "VPNtoSocket VPN started on port: "+serverSocket.getLocalPort());

İ made some modification on Proxy.java , https://github.com/ubeyid/Android-VpnSocket-VpnProxy/blob/master/app/app/src/main/java/vpntosocket/shadowrouter/org/vpntosocket/VPN/VPNService.java NatSession.java and a little bit more on the remaining code and now it works perfectly !!! Project links is https://github.com/ubeyid/Android-VpnSocket-VpnProxy


SVN support for dfetch import

Currently import only supports Git.
Add support for SVN.

It appears that SVN is now required to be installed on the system to be able to run the import command.

It appears that SVN is now required to be installed on the system to be able to run the import command.

Ah, off course :see_no_evil:

Ah, off course :see_no_evil:

Created #73

Created #73

mmm, could you paste dfetch -v import output? "It works on my machine"

mmm, could you paste dfetch -v import output? "It works on my machine"


No directory added to packages, and thus no CSS added per package file

Answered it w/ help of MeteorClub Slack group ;-)
Thanks for packaging up framework7!

For those newer to Meteor, when you run your app the first time - packages will compile and run their package manifest file... then you can find the package contents and assets etc. following this path typically:


[lis] Investigate user report of kernel does not exist

Context
Reported via https://2i2c.freshdesk.com/a/tickets/547
User server reporting kernel does not exist
I logged into server and worked for me.

99639058 (<IP_ADDRESS>): Kernel does not exist: 62e648e2-f524-408c-b53e-1af21e3fbcd3
[W 2023-03-15 17:00:39.132 SingleUserLabApp handlers:738] wrote error: 'Kernel does not exist: 62e648e2-f524-408c-b53e-1af21e3fbcd3'
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.10/site-packages/tornado/web.py", line 1713, in _execute
        result = await result
      File "/opt/conda/lib/python3.10/site-packages/jupyter_server/services/kernels/handlers.py", line 72, in get
        model = await ensure_async(km.kernel_model(kernel_id))
      File "/opt/conda/lib/python3.10/site-packages/jupyter_server/services/kernels/kernelmanager.py", line 496, in kernel_model
        self._check_kernel_id(kernel_id)
      File "/opt/conda/lib/python3.10/site-packages/jupyter_server/services/kernels/kernelmanager.py", line 527, in _check_kernel_id
        raise web.HTTPError(404, "Kernel does not exist: %s" % kernel_id)

Proposal
No response
Updates and actions
No response

Google Logs Explorer query
resource.type="k8s_container"
resource.labels.namespace_name="lis"
resource.labels.pod_name="jupyter-matthew-2dbrett"
textPayload=~"Kernel does not exist"

First failure
{
  "textPayload": "[W 2023-03-15 16:24:50.388 SingleUserLabApp web:1796] 404 GET /user/matthew-brett/api/kernels/b560b61e-57c4-4308-a3d8-9f3c159fea27?1678897490300 (<IP_ADDRESS>): Kernel does not exist: b560b61e-57c4-4308-a3d8-9f3c159fea27",
  "insertId": "ep7g279ua0u83cjz",
  "resource": {
    "type": "k8s_container",
    "labels": {
      "project_id": "two-eye-two-see-uk",
      "namespace_name": "lis",
      "cluster_name": "two-eye-two-see-uk-cluster",
      "pod_name": "jupyter-matthew-2dbrett",
      "container_name": "notebook",
      "location": "europe-west2"
    }
  },
  "timestamp": "2023-03-15T16:24:50.388977704Z",
  "severity": "ERROR",
  "labels": {
    "k8s-pod/release": "lis",
    "k8s-pod/hub_jupyter_org/username": "matthew-2dbrett",
    "k8s-pod/heritage": "jupyterhub",
    "k8s-pod/chart": "jupyterhub-2.0.1-0.dev.git.5987.h46a632f5",
    "compute.googleapis.com/resource_name": "gke-two-eye-two-see-uk-cluste-nb-user-893a49dd-wxhq",
    "k8s-pod/app": "jupyterhub",
    "k8s-pod/hub_jupyter_org/network-access-hub": "true",
    "k8s-pod/component": "singleuser-server",
    "k8s-pod/hub_jupyter_org/servername": ""
  },
  "logName": "projects/two-eye-two-see-uk/logs/stderr",
  "receiveTimestamp": "2023-03-15T16:24:52.321358327Z"
}

OOMs
https://grafana.uk.2i2c.cloud/d/gVxNqwHVk/cluster-information?orgId=1&viewPanel=13&from=1678879701140&to=1678901301140

OOM logs

OOM Logs:
[
  {
    "textPayload": "[195998.443208] node invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=991\\r\\n",
    "insertId": "7710",
    "resource": {
      "type": "gce_instance",
      "labels": {
        "zone": "europe-west2-c",
        "instance_id": "4616395539098064229",
        "project_id": "two-eye-two-see-uk"
      }
    },
    "timestamp": "2023-03-15T15:50:35.927515678Z",
    "severity": "INFO",
    "labels": {
      "compute.googleapis.com/resource_name": "gke-two-eye-two-see-uk-cluste-nb-user-893a49dd-wxhq",
      "is_text_escaped": "true"
    },
    "logName": "projects/two-eye-two-see-uk/logs/serialconsole.googleapis.com%2Fserial_port_1_output",
    "receiveTimestamp": "2023-03-15T15:50:37.933070466Z"
  },
  {
    "textPayload": "[196120.737205] ZMQbg/IO/0 invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=991\\r\\n",
    "insertId": "7810",
    "resource": {
      "type": "gce_instance",
      "labels": {
        "instance_id": "4616395539098064229",
        "project_id": "two-eye-two-see-uk",
        "zone": "europe-west2-c"
      }
    },
    "timestamp": "2023-03-15T15:52:38.229654772Z",
    "severity": "INFO",
    "labels": {
      "compute.googleapis.com/resource_name": "gke-two-eye-two-see-uk-cluste-nb-user-893a49dd-wxhq",
      "is_text_escaped": "true"
    },
    "logName": "projects/two-eye-two-see-uk/logs/serialconsole.googleapis.com%2Fserial_port_1_output",
    "receiveTimestamp": "2023-03-15T15:52:40.235787886Z"
  },
  {
    "textPayload": "[196230.671388] python invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=991\\r\\n",
    "insertId": "7905",
    "resource": {
      "type": "gce_instance",
      "labels": {
        "zone": "europe-west2-c",
        "instance_id": "4616395539098064229",
        "project_id": "two-eye-two-see-uk"
      }
    },
    "timestamp": "2023-03-15T15:54:28.171153047Z",
    "severity": "INFO",
    "labels": {
      "is_text_escaped": "true",
      "compute.googleapis.com/resource_name": "gke-two-eye-two-see-uk-cluste-nb-user-893a49dd-wxhq"
    },
    "logName": "projects/two-eye-two-see-uk/logs/serialconsole.googleapis.com%2Fserial_port_1_output",
    "receiveTimestamp": "2023-03-15T15:54:30.180922898Z"
  },
  {
    "textPayload": "[196421.940495] python invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=991\\r\\n",
    "insertId": "7977",
    "resource": {
      "type": "gce_instance",
      "labels": {
        "project_id": "two-eye-two-see-uk",
        "zone": "europe-west2-c",
        "instance_id": "4616395539098064229"
      }
    },
    "timestamp": "2023-03-15T15:57:39.453041790Z",
    "severity": "INFO",
    "labels": {
      "is_text_escaped": "true",
      "compute.googleapis.com/resource_name": "gke-two-eye-two-see-uk-cluste-nb-user-893a49dd-wxhq"
    },
    "logName": "projects/two-eye-two-see-uk/logs/serialconsole.googleapis.com%2Fserial_port_1_output",
    "receiveTimestamp": "2023-03-15T15:57:41.464249729Z"
  }
]


[BugFix] Bind storage volume to db when it has been created successfully

Fixes #issue
MySQL [(none)]> create database test properties("storage_volume"="storage_volume4");
ERROR 1064 (HY000): Unexpected exception: Unknown storage volume "storage_volume4"
MySQL [(none)]> create database test properties("storage_volume"="storage_volume4");
ERROR 1064 (HY000): Unexpected exception: Can't create database 'test'; database exists
MySQL [(none)]>show databases;
+--------------------+
| Database           |
+--------------------+
| _statistics_       |
| information_schema |
| starrocks          |
| test              |
+--------------------+
5 rows in set (0.01 sec)

database will be created if the storage volume not exists.
database should be created after checking the existence of storage volume and the edit log is written successfully.
What type of PR is this:

[x] BugFix
[ ] Feature
[ ] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Checklist:

[x] I have added test cases for my bug fix or my new feature
[ ] This pr will affect users' behaviors
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[x] I have checked the version labels which the pr will be auto-backported to the target branch

[x] 3.1
[ ] 3.0
[ ] 2.5
[ ] 2.4

[FE PR Coverage Check]
:heart_eyes: pass : 7 / 7 (100.00%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/server/LocalMetastore.java
7
7
100.00%
[]

[FE PR Coverage Check]
:heart_eyes: pass : 8 / 8 (100.00%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/server/LocalMetastore.java
8
8
100.00%
[]

[FE PR Coverage Check]
:disappointed: fail : 15 / 19 (78.95%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/server/LocalMetastore.java
9
13
69.23%
[397, 401, 402, 403]

:large_blue_circle:
com/starrocks/server/StorageVolumeMgr.java
6
6
100.00%
[]

[FE PR Coverage Check]
:heart_eyes: pass : 11 / 12 (91.67%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/server/LocalMetastore.java
5
6
83.33%
[395]

:large_blue_circle:
com/starrocks/server/StorageVolumeMgr.java
6
6
100.00%
[]

@Mergifyio backport branch-3.1


fix up owners files

/assign @adambkaplan

@bparees shouldn't someone from @openshift/sig-master own this?

/lgtm
disregard my prior comment

@adambkaplan no, devex owns the image config observation/handling part.  (and other teams own the observation of their config objects)


Create LICENSE

Why is build job not running?

Why is build job not running?

I'm not sure. But since the change is a text file, I think I'll try to forcefully merge...


Depth to Disparity Transformation and Training Process

Hello,
Thank you for your valuable contribution.
I am currently reproducing the training code and performing fine-tuning on pseudo-labeled data as described in the article. I have two questions:

In section 3.1, you mention that "the depth value is first transformed into the disparity space by d = 1/t." Is this formula applied to all public datasets you use, or does it depend on the dataset (for example: for the TartanAir dataset, the disparity formula is disp = 80/depth)?
In section 4.1, you mention: "In the second stage of joint training, we train a student model to sweep across all unlabeled images for one time." Does this mean that we pass through all the pseudo-labeled data only once per epoch, as in standard training, or that the student model is trained for only one epoch in total?

Thank you in advance for your response.

Hi Reffad,

By default, we transform depth to disparity by d = 1/t. Although TartanAir provides the formula d = 80 / t, it indeed does not matter in our framework, because we will further normalize the disparity map by min-max value to 0-1, which will diminish the effect of the scalar 80.
The latter case. The student model is only trained on all the 62M unlabeled images for one epoch.


Add on_success and on_failure callbacks to Result

Ruby has the Object#tap to perform some "side actions" outside the "main path" of the code. This introduces something similar, but for Result. It adds two callbacks, on_success and on_failure. This helps keep the method chains since it avoids needing to add if expressions and explicitly returning the original monad again.
Ex:
def fetch_repositories
  with_loading_spinner("Fetching repositories...") do |spinner|
    Repository
      .fetch(language: "ruby") # Returns a Result
      .on_success { spinner.success }
      .on_failure { spinner.error }
  end
end

# vs

def fetch_repositories
  with_loading_spinner("Fetching repositories...") do |spinner|
    result = Repository.fetch(language: "ruby")
    spinner.success if result.success?
    spinner.error if result.failure?

    result
  end
end

I'd love to make this more generic and add it to RightBiased so we could use it on Maybe and Try as well, but I wasn't sure about the naming.
To me, this looks a bit like Object#tap, so tap_left and tap_right would be options, but it feels a bit weird (too low level, maybe?). Another option would be to reference Result#either, but I don't like requiring both left and callbacks simultaneously.
def fetch_repositories
  with_loading_spinner("Fetching repositories...") do |spinner|
    Repository
      .fetch(language: "ruby") # Returns a Result
      .tap_either(
        ->(_success_result) { spinner.success },
        ->(_failure_result) { spinner.failure },
      ) # what if I just wanted one of these?
  end
end

I'm happy to change the API with any new idea. Also, LMK if there's a better way to handle this with the current API.

Hey, thanks for submitting the PR. I gave it a thought and I don't agree with this proposal. Case analysis is meant to be done at the very end, preferably via pattern matching:
include Dry::Monads[:result]

def fetch_repositories
  with_loading_spinner("Fetching repositories...") do |spinner|
    case Repository.fetch(language: "ruby")
    in Success then spinner.success
    in Failure then spinner.failure
    end
  end
end

This way it is clear where you compose monadic results and where you "handle" them. I know there are similar APIs in other languages/libraries but it's not how dry-monads should be used IMO.

I like the separation between composing and handling monads! I think it's just a bit cumbersome having to return the result at the end of the method (otherwise it would return whatever spinner.{success.error} returns. It's not possible to omit a branch (we could ignore success cases, for example).
def fetch_repositories
  with_loading_spinner("Fetching repositories...") do |spinner|
    case result = Repository.fetch(language: "ruby")
    in Success then spinner.success
    in Failure then spinner.failure
    end

    result
  end
end

Also, it's not always that we wanna/can handle the results in the same place. In this example above, the spinner.error expression is a secondary side-effect outside the main path (where fetch_repositories is called).
I hope those points may help you to reconsider this, but I understand if that's not how y'all envision the best usage of this gem.

If you don't need both branches, there's a bunch of methods that work on success or failure separately. I consider "folding" a value as a termination step hence exhaustive checks by pattern matching are preferable. If you really need these methods, you can patch dry-monads locally or via gem, it'd be fine.


Feature Request: Support Recurring Events

It would be great if recurring events were supported. A good example of this is in Outlook, where you can set events to recur daily/weekly/monthly. Other similar components support this as well, including FullCalendar and jqxScheduler

@tgBryanBailes
Hi, supporting recurring events is a good feature. I think we have two ways to implement this:
the first one is that we save the recurring events(or the rules to generate recurring events) in database, and return the event list (including the recurring events) in the time window when a browser requests, then browser sets the event list to the SchedulerData object to display. If this, there will be no change in the Scheduler component, what we need to do is in the back-end.
the other one is that the Scheduler component should support the rules which can generate recurring events. Before calling setEvents to the SchedulerData object, Scheduler component should generate recurring events according the rules, add them into non-recurring event list, and then call the setEvents method.
Which way should we choose, or any better idea?:-)

@StephenChou1017
The issue with doing this in a DB is that recurring events often do not have an End Date. They just recur until they are cancelled. A DB would then have to regenerate events as the calendar is traversed, causing lots of unnecessary data transfers.
The Scheduler component supporting rules for recurring events would be preferable in this scenario, and can be set up with fairly limited data transfer. A great way to do this would be to accept an  rrule.js object to define recurrence.

OK, I got you and agree with you, I'll try to add this feature in Scheduler, thank you:-)

@tgBryanBailes
I've added the supporting recurring events feature to Scheduler, you may download the latest repo and have a try:-)

This is fantastic! I will begin testing immediately. Thank you so much!

It doesn't seem to obey the UNTIL rrule command.
FREQ=WEEKLY;DTSTART=20180712T173000Z;UNTIL=20180812T233000Z;BYDAY=TU,FR
Still continues showing events after 2018-08-12

Oh yes, that's a bug, and I think I've fixed it:-), thanks for the feedback!

This is wonderful and working great! Thanks for this! Closing the issue.

Is there any way to added some exceptional dates in recurrence rule to avoid displaying event on some days ?

Is there any way to added some exceptional dates in recurrence rule to avoid displaying event on some days ?

I believe you can use rrule exdate to do that, the string would like this (untested):
const rruleStr = `
    DTSTART:20220501T090000
    RRULE:FREQ=DAILY;COUNT=10
    EXDATE:20220503T090000,20220505T090000
`;

Is there any way to added some exceptional dates in recurrence rule to avoid displaying event on some days ?

I believe you can use rrule exdate to do that, the string would like this (untested):
const rruleStr = `
    DTSTART:20220501T090000
    RRULE:FREQ=DAILY;COUNT=10
    EXDATE:20220503T090000,20220505T090000
`;

Also you can pass add exdates to the event item, which takes a list of dates. Example (again, untested):
 events: [
      {
          id: 10,
          start: '2017-12-19 17:30:00',
          end: '2017-12-19 23:30:00',
          resourceId: 'r1',
          title: 'R1 has recurring tasks every week on Tuesday, Friday',
          rrule: 'FREQ=WEEKLY;DTSTART=20171219T013000Z;BYDAY=TU,FR',
          exdates: ['2023-05-05','2023-05-12','2023-05-07'],
          bgColor: '#f759ab'
      }
]

I tried this both but didn't got any success.. :-(

I tried this both but didn't got any success.. :-(

Could you share some code?


[Snyk] Security upgrade golang from 1.22.3-bullseye to 1.22.5-bullseye

This PR was automatically created by Snyk using the credentials of a real user.![snyk-top-banner](https://github.com/andygongea/OWASP-Benchmark/assets/818805/c518c423-16fe-447e-b67f-ad5a49b5d123)
Snyk has created this PR to fix 4 vulnerabilities in the dockerfile dependencies of this project.
Keeping your Docker base image up-to-date means you’ll benefit from security fixes in the latest version of your chosen image.
Snyk changed the following file(s):

Dockerfile

We recommend upgrading to golang:1.22.5-bullseye, as this image has only 177 known vulnerabilities. To do this, merge this pull request, then verify your application still works as expected.
Vulnerabilities that will be fixed with an upgrade:

Issue
Score

Improper Verification of Cryptographic Signature SNYK-DEBIAN11-GNUTLS28-6159417
  614

Information Exposure SNYK-DEBIAN11-GNUTLS28-6159419
  614

Out-of-bounds Write SNYK-DEBIAN11-LIBSSH2-5861756
  614

Cleartext Transmission of Sensitive Information SNYK-DEBIAN11-CURL-3320493
  500

Cleartext Transmission of Sensitive Information SNYK-DEBIAN11-CURL-3320493
  500

[!IMPORTANT]

Check the changes in this PR to ensure they won't cause issues with your project.
Max score is 1000. Note that the real score may have changed since the PR was raised.
This PR was automatically created by Snyk using the credentials of a real user.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information:
🧐 View latest project report
📜 Customise PR templates
🛠 Adjust project settings
📚 Read about Snyk's upgrade logic

Learn how to fix vulnerabilities with free interactive lessons:
🦉 Learn about vulnerability in an interactive lesson of Snyk Learn.

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 88.83%. Comparing base (27cbb93) to head (8589138).

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff           @@
##           master      #47   +/-   ##
=======================================
  Coverage   88.83%   88.83%
=======================================
  Files          15       15
  Lines        1164     1164
=======================================
  Hits         1034     1034
  Misses        115      115
  Partials       15       15

Flag
Coverage Δ

unittests
88.83% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Policy Sync Policy doesn't upload new versions

Policy Template Name
--Policy Template Synchronization Policy Template--
policy_bootstrapper script
Issue Type
The policy bootstrap script should not use force-upgrade option.
This should get things working reasonably well. There is still a window where the incident can get stuck but this will be addressed presently when actions are supported.
Use Case (Feature Request -or- New Policy Template Request Only)
Policy Template version (Bug Only)
Code Snippet (Bug Only)
[Please provide any relevant code snippets (ie. datasource,escalation,policy, RCL, etc.) as it relates to the bug]
Error Messages (Bug Only)
[Provide any error messages returned (if applicable)]

Fixed by merged PR:
https://github.com/rightscale/policy_templates/pull/68