[SPARK-20396][SQL][PySpark] groupby().apply() with pandas udf

What changes were proposed in this pull request?
This PR adds an apply() function on df.groupby(). apply() takes a pandas udf that is a transformation on pandas.DataFrame -> pandas.DataFrame.
Static schema
schema = df.schema

@pandas_udf(schema)
def normalize(df):
    df = df.assign(v1 = (df.v1 - df.v1.mean()) / df.v1.std()
    return df

df.groupBy('id').apply(normalize)

Dynamic schema
This use case is removed from the PR and we will discuss this as a follow up. See discussion https://github.com/apache/spark/pull/18732#pullrequestreview-66583248
Another example to use pd.DataFrame dtypes as output schema of the udf:
sample_df = df.filter(df.id == 1).toPandas()

def foo(df):
      ret = # Some transformation on the input pd.DataFrame
      return ret

foo_udf = pandas_udf(foo, foo(sample_df).dtypes)

df.groupBy('id').apply(foo_udf)

In interactive use case, user usually have a sample pd.DataFrame to test function foo in their notebook. Having been able to use foo(sample_df).dtypes frees user from specifying the output schema of foo.
Design doc: https://github.com/icexelloss/spark/blob/pandas-udf-doc/docs/pyspark-pandas-udf.md
How was this patch tested?

Added GroupbyApplyTest

Can one of the admins verify this patch?

cc @HyukjinKwon @BryanCutler

cool - this is a bit understated but potentially huge (to me anyway)

Should we maybe consider SPIP?

Thanks all for comment.
@HyukjinKwon
This is part of SPIP https://issues.apache.org/jira/browse/SPARK-21190
I have been engaging in discussion of SPARK-21190 and make sure whatever I am doing here doesn't diverge from the general design the vectorized UDF.
I don't expect we start to merge this until we have a solid design in SPARK-21190. It would be great if Spark commiters can help move the discussion on SPARK-21190 forward.

there's actually a number of key people participating in the discussion in JIRA/SPIP, so I think we are good
I think perhaps SPARK-20396 should be a subtask instead for tracking the overall discussion/design.

Can one of the admins verify this patch?

Hi,
Thanks to the vectorized udf change, this PR is much more smaller than the original. I think this is a useful feature and would love to get some feedback on this.
Thoughts? cc @BryanCutler @HyukjinKwon @ueshin

ok to test

Test build #82288 has started for PR 18732 at commit 07bccca.

Test build #82288 has finished for PR 18732 at commit 07bccca.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82288/
Test FAILed.

I believe I should cc @cloud-fan too BTW. Will take a closer look too soon.

Based on the doc of pandas_udf, we've already define pandas udf is an user defined function (UDF) that accepts Pandas.Series as input arguments and outputs a Pandas.Series of the same length. Is it a good idea to break this definition and let pandas udf can accept pandas.DataFrame?

Thanks all for the initial review! I will address some comments and upload a new version today.

Test build #82335 has started for PR 18732 at commit 8d98b3e.

Test build #82335 has finished for PR 18732 at commit 8d98b3e.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
 class InnerIterator[T](iter: Iterator[T], batchSize: Int) extends Iterator[T]

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82335/
Test FAILed.

Test build #82336 has started for PR 18732 at commit 96ce587.

Test build #82336 has finished for PR 18732 at commit 96ce587.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82336/
Test FAILed.

Merged build finished. Test FAILed.

Is this just a mapGroups function?

@rxin This is similar to flatMapGroups since the return value of the user function is a list of rows (pd.DataFrame) rather than a single row.

What's the difference between this one and the transform function you also proposed? I'm trying to see if all the naming makes sense when considered together.

@rxin, transform takes a function: pd.Series -> pd.Series and apply the function on all columns:
df.show()

 id   v1   v2  v3
 a  1.0  4.0  0.0
 a  2.0  5.0  1.0
 a  3.0  6.0  1.0

df.groupby('id').transform(pandas_udf(lambda v: v - v.mean(), DoubleType())).show()

 id   v1   v2        v3
 a  -1.0 -1.0    -0.666667
 a   0.0  0.0     0.333333
 a   1.0  1.0     0.333333

This is mimicking pd.DataFrame.groupby().transform
apply takes a function: pd.DataFrame -> pd.DataFrame and is similar to flatMapGroups
The name apply is originated from the R paper "The Split-Apply-Combine Strategy for Data Analysis" and is used in both pandas and R to describe this function, so the name apply should be pretty straight forward to pandas/python user.

@HyukjinKwon Thanks for the feedback. I will address those and update tomorrow.

Test build #82430 has started for PR 18732 at commit d37a9e6.

Test build #82430 has finished for PR 18732 at commit d37a9e6.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82430/
Test FAILed.

Test build #82431 has started for PR 18732 at commit 1ea2b71.

Test build #82431 has finished for PR 18732 at commit 1ea2b71.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82431/
Test FAILed.

Merged build finished. Test FAILed.

Test build #82438 has started for PR 18732 at commit 4943ceb.

Test build #82440 has started for PR 18732 at commit 21fed0d.

Test build #82438 has finished for PR 18732 at commit 4943ceb.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82438/
Test FAILed.

Test build #82440 has finished for PR 18732 at commit 21fed0d.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82440/
Test FAILed.

Test build #82457 has started for PR 18732 at commit d9a3e8d.

Test build #82457 has finished for PR 18732 at commit d9a3e8d.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82457/
Test FAILed.

I pushed a new commit addressing the comments. Let me scan through the comments again. I think there are some comments around worker.py not being addressed yet.

Test build #82466 has started for PR 18732 at commit fa88c88.

Test build #82469 has started for PR 18732 at commit e4efb32.

Test build #82463 has finished for PR 18732 at commit 657942b.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82463/
Test FAILed.

Test build #82466 has finished for PR 18732 at commit fa88c88.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82466/
Test FAILed.

Test build #82469 has finished for PR 18732 at commit e4efb32.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82469/
Test FAILed.

Test build #82477 has started for PR 18732 at commit f572385.

Ongoing discussions that (I think) might block this PR:

https://github.com/apache/spark/pull/18732#discussion_r142735696 by @BryanCutler: ArrowPandasSerializer able to serialize pandas.DataFrames

https://github.com/apache/spark/pull/18732#issuecomment-333065737 by @viirya: breaking this definition into two (groupping and normal udfs).

https://github.com/apache/spark/pull/18732#issuecomment-333326073 by @rxin and answer https://github.com/apache/spark/pull/18732#issuecomment-333432266 by @icexelloss: naming suggestion

@HyukjinKwon Thanks for the summarry!

https://github.com/apache/spark/pull/18732#discussion_r142735696 ArrowPandasSerialzerI will spend some time address this today.
https://github.com/apache/spark/pull/18732#issuecomment-333065737 (Breaking into two pandas udf API) I think is addressed here https://github.com/apache/spark/pull/18732#discussion_r141830344. But I am happy to discuss more.
https://github.com/apache/spark/pull/18732#issuecomment-333326073 (API Naming) I will wait on feedback here.

Test build #82477 has finished for PR 18732 at commit f572385.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82477/
Test PASSed.

Test build #82489 has started for PR 18732 at commit 5162ed1.

Test build #82489 has finished for PR 18732 at commit 5162ed1.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82489/
Test FAILed.

Test build #82490 has started for PR 18732 at commit d628f4e.

Test build #82490 has finished for PR 18732 at commit d628f4e.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82490/
Test FAILed.

Test build #82493 has started for PR 18732 at commit 20fb1fe.

Test build #82493 has finished for PR 18732 at commit 20fb1fe.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82493/
Test FAILed.

Merged build finished. Test FAILed.

retest this please

Test build #82501 has started for PR 18732 at commit 20fb1fe.

Test build #82501 has finished for PR 18732 at commit 20fb1fe.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82501/
Test PASSed.

Hi All, I think all comments should be addressed at this point, except for the naming comment from @rxin.
If I missed something or if there is anything else you want me to address, please let me know.
Otherwise, I will just wait for Reynold's feedback regarding naming.

Test build #82515 has started for PR 18732 at commit 284ba00.

Test build #82515 has finished for PR 18732 at commit 284ba00.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82515/
Test PASSed.

Test build #82553 has started for PR 18732 at commit 876b118.

Test build #82553 has finished for PR 18732 at commit 876b118.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class FlatMapGroupsInPandas(

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82553/
Test PASSed.

I'm OK with the naming. We can change them later if needed before the release.

Test build #82576 has started for PR 18732 at commit b0410a2.

Merged some last minute changes from @BryanCutler to make the wrapping a bit cleaner. Thanks @BryanCutler!

Test build #82577 has started for PR 18732 at commit a064b21.

Test build #82576 has finished for PR 18732 at commit b0410a2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82576/
Test PASSed.

Test build #82577 has finished for PR 18732 at commit a064b21.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82577/
Test PASSed.

LGTM

LGTM

add to whitelist

Test build #82584 has started for PR 18732 at commit a036f70.

Test build #82585 has started for PR 18732 at commit b88a4d8.

Test build #82587 has started for PR 18732 at commit 9c2b10e.

Test build #82584 has finished for PR 18732 at commit a036f70.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82584/
Test PASSed.

Test build #82585 has finished for PR 18732 at commit b88a4d8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82585/
Test PASSed.

I had some minor comments on the docs, otherwise LGTM!

Test build #82587 has finished for PR 18732 at commit 9c2b10e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82587/
Test PASSed.

Test build #82599 has started for PR 18732 at commit dc1d406.

Test build #82599 has finished for PR 18732 at commit dc1d406.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82599/
Test PASSed.

Merged to master.

Nice work 👍

@HyukjinKwon Thanks!
Thanks for everyone for reviewing this tirelessly.

A late question: shall we create another API for it instead of reusing pandas_udf? cc @ueshin

I think @viirya raised this question too - https://github.com/apache/spark/pull/18732#issuecomment-333065737 and I think I also left few worries about thus here and there.  To me, +0.

@cloud-fan, it's a good question, I thought quite a bit about it and discussed with @viirya -https://github.com/apache/spark/pull/18732#pullrequestreview-66106082
Just to recap, I think from a API perspective, having just one decorator pandas_udf making it easier for user to use - they don't need to think about which decorator to use where.  It does make it a little bit complicated for implementation because some code have to interpret the context in which a pandas_udf is used, i.e., pandas_udf in groupby()apply() is a pandas.DataFrame -> pandas.DataFrame, and in withColumn, select it's pandas.Series -> pandas.Series.
Another thought is even if we were to introduce something like pandas_df_udf, for instance, we might still run into issues in the future, where, say, we want a aggregate pandas udf that defines mapping pandas.Series -> scalar, so I don't think we can define a decorator for every input/output shape because there can potentially be many.

@ueshin is working on pandas UDAF, let's wait for his feedback.

I'm +0 for now.
I'm just wondering whether we can support struct types in vectorized UDF when needed in the future.
As for adding pandas UDAF, I think we need another decorator or something to specify it supports partial aggregation or not and the related parameters if needed.

Grouped UDFs, or Grouped Vectorized UDFs.

I submitted a pr #19505 to introduce @pandas_grouped_udf instead of reusing @pandas_udf.

I am still not crazy about introducing a pandas_grouped_udf unless there is strong reason to.  @ueshin do you think this is just an issue of returnType or is there some other reason?

@icexelloss I think as an API, it's a little confusing that @pandas_udf can define both Series* -> Series function and DataFrame -> DataFrame function. Besides, to support StructType as the return type of Series* -> Series function, I think we have to add an extra flag to @pandas_udf. For the coming DataFrame -> Scalar pandas UDAF, we also need extra flags to represent partial aggregate ability.
From my experience of Java/Scala API design, I think it's a bad idea to have a method with many parameters as flags. We'd better have more methods. For this case, @pandas_udf, @pandas_grouped_udf and @pandas_udaf looks better to me.

@cloud-fan Thanks for your feedback.
I think it makes sense to define pandas_udaf as it's own function because it is a multi-step udf and is very different from the existing pandas_udf
I also agree we shouldn't add many parameters as flag. However, here are something I am not sure about:

Use different function name (i.e, pandas_udf and pandas_grouped_udf) for different input/output type (pd.Series, pd.DataFrame, or Scalar):
There could be potentially many combination of input/output types that doesn't fit into pd.Series -> pd.Series and pd.DataFrame -> pd.DataFrame. For instance, for a vectorized window function and aggregation, it would probably be a pd.Series -> scalar function, which is different from either pandas_udf or pandas_grouped_udf, I am not sure if we want to introduce another decorator for those cases.

Distinguish between a Struct column and a DataFrame:
I think we can accomplish this without introducing a new function decorator or a parameter, but by the context how pandas_udf is used. For instance, withColumn add a column, so the return type specifies the column type, where groupby().apply() maps a DataFrame, so the return type specifies dataframe schema.

Use different function name for different input/output type

Yea it's a bad idea as there are many combinations, and I just wanna use different APIs for different scenarios, e,g, @pandas_udf for select/withColumn(Series* -> Series), @pandas_grouped_udf for groupBy(apply: DataFrame -> DataFrame, reduce: DataFrame -> Scalar and more) and @pandas_udaf for aggregate.
Different scenarios usually have different requirements, having different APIs can help us satisfy these requirements individually.

Let's discuss more on the new PR. At least we should create different UDF types in the implementation, the user-facing API can remain @pandas_udf.

@cloud-fan Sounds good. Thanks!


⚠ Quirk Club Argentina (API endpoint) has degraded performance

In 997ea31, Quirk Club Argentina (API endpoint) (https://us-central1-quirkclub-dev.cloudfunctions.net/api/check/api) experienced degraded performance:

HTTP code: 200
Response time: 6279 ms

Resolved: Quirk Club Argentina (API endpoint) performance has improved in e118dd2.


Add net/http integration to trace the outgoing requests

/cc @jba
We will also use this plugin in the tracing of the google-api-go-client clients.

LGTM.


Website: Config selection dropdown values are invisible in dark mode

Describe the bug
In dark mode of the website - Configuration selection dropdown options are almost invisible.
To Reproduce

Navigate to https://www.graphql-code-generator.com
Select night mode
Click on the dropdown to see the options

Expected behavior

Dropdown values should be visible in dark mode as well.

Environment:

Production

@entbit ^

@B2o5T fixed most of these


fixed a path in instructions at readme, also fixed executable in .pyr…

absolute paths in config file made example notebooks fail.

pyron executable should be in path inside the proper environment

Thanks @mdforti !


Restrict liquid to continue support Ruby 2.4

affect https://github.com/metanorma/mn-templates-itu/pull/27

I have a feeling that this isn't the right approach to freeze dependency. If we install gems with bundle install there are no issues, but for gem install metanorma-cli it happens https://github.com/metanorma/mn-templates-nist/runs/1740861407?check_suite_focus=true
I'm wondering why gem install don't handle this situation
@ronaldtse @opoudjis maybe you have an understanding of why gem install cannot handle this?

I have a feeling that this isn't the right approach to freeze dependency. If we install gems with bundle install there are no issues, but for gem install metanorma-cli it happens https://github.com/metanorma/mn-templates-nist/runs/1740861407?check_suite_focus=true
I'm wondering why gem install don't handle this situation
@ronaldtse @opoudjis maybe you have an understanding of why gem install cannot handle this?

I'm afraid I don't know why "bundle install" would work and "gem install" would not — although it is very clear by now that whatever is forcing us to depend on Ruby 2.4 needs to be addressed as the highest priority. Sorry. @ronaldtse?

I'm afraid I don't know why "bundle install" would work and "gem install" would not — although it is very clear by now that whatever is forcing us to depend on Ruby 2.4 needs to be addressed as the highest priority. Sorry. @ronaldtse?

Like it or not, we are stuck with ruby-packer which only has 2.4 right now. We’re not going to deal with this until later, there’s too much on our plate right now.

Like it or not, we are stuck with ruby-packer which only has 2.4 right now. We’re not going to deal with this until later, there’s too much on our plate right now.


Different text sizes makes them vertically unaligned

Hello! I found this project rather interesting and decided to try it and play around with it.
I noticed something weird when having text on the same "line" but with different sizes. I tried it on version 0.3.0 with the code:
use iced::{Color, Element, Length, Row, Sandbox, Text, VerticalAlignment};

fn main() -> iced::Result {
    Hello::run(iced::Settings::default())
}

struct Hello;

impl Sandbox for Hello {
    type Message = ();

    fn new() -> Self {  Hello  }
    fn title(&self) -> String {  "Hello World".to_string()  }
    fn update(&mut self, _message: Self::Message) { }

    fn view(&mut self) -> Element<'_, Self::Message> {
        let content: Element<()> = Row::new()
            .height(Length::Units(60))    // Different height values do not seem to affect it
            .push(
                Text::new("Hello")
                    .size(20)
                    .height(Length::Fill)
                    .vertical_alignment(VerticalAlignment::Bottom),
            )
            .push(
                Text::new("World")
                    .size(50)
                    .height(Length::Fill)
                    .vertical_alignment(VerticalAlignment::Bottom),
            )
            .into();

        content.explain(Color::BLACK)
    }
}

Whether I vertically align the text to the bottom or to the top, the base line of the text is different. In the code above, you can see the "Hello" part is lower than the "World". I was expecting that both texts would have the same bottom/top alignment even with different sizes but that was not the case.
Is there some other way to have this differently sized text in the same line that I missed or is this unintended behavior?

This might answer your question, its a talk from a different library in a different language, but the point still stands. The video later talks about his solution, but the problem he presents exists in both CSS and Iced. Text can be very weird. Unfortunately you are just going to have to manually align the text.


Subject Identifier of the first factor(basic) is not used in multi-factor authentication when conditional authentication is enabled

Steps to reproduce

Create a user.
Configure a service provider.
Configure multi-factor authentication with Twitter as the second factor.
Use the subject identifier from the first step.
Tick "use this script" and add a simple script under the Javascript based conditional step.
Update the application.

When you log into the sample application with the user successfully you'll see that user's Twitter ID will be returned as the subject ID, not the username.

Fixed with https://github.com/wso2/carbon-identity-framework/pull/1554


I cannot seem to update from JSON to SAV

I follow all the updates (even the one 10 minutes ago) but i keep getting this error, not quite sure what to do with it. I am missing something?

looks like when i  copy the key/value i didnt copy enough (i still dont know how much i need to copy, the og level file have to many stuff) wasnt able to fix it at the end because it keep trying to make the player make a new character

now i got this problem so I am not quite sure what to do next


$and not returning what I would expect

I'm sure I'm doing something simple wrong here. Can you please help me?
      const options = { keys: ['content'], includeScore: true, includeMatches: true, shouldSort: true }
      const notes = [
        {
          content: 'the lazy frog jumped over the log',
        },
        {
          content: 'the lazy dog jumped over the banana',
        },
        {
          content: 'the crazy man ate a frog',
        },
      ]
      const fuse = new Fuse(notes, options)
      const searchExp = { $and: [{ content: `=frog` }, { content: `=lazy` }] }
      const result = fuse.search(searchExp)
      expect(result.length).toEqual(1)

I expected that searchExp to return only one result (the first one, because it's the only one that contains bothfrog and lazy. But instead, it returns two (the first and third).
I have tried this expression in other forms (without the equals):
const searchExp = { $and: [{ content: 'frog }, { content: 'lazy }] } //apostrophe, returns the same 2 results
const searchExp = { $and: [{ content: frog }, { content: lazy }] } //no extended, returns all three (3) results
I'm definitely missing something here. Can you please help me?
Thank you!
David

Closing for now. I think I found a workaround. Thanks for this library!


#184 Unit test

@juslee ,@championswimmer , travis.yml is not configured with karma . Should it be done something like this http://stackoverflow.com/questions/19255976/how-to-make-travis-execute-angular-tests-on-chrome-please-set-env-variable-chr


Disable the security survey for now

We are not making good use of the results we have and we can go through that data at any time.
(cc @mozfreddyb )

I opened #1386


plot net aggregate alignment by financial exposure

To provide an additional overview chart for an ongoing project, we want to display the net aggregate alignment metric by financial exposure at the sector level and by categories, such as the name of the loan book or the type of bank.
Thus, workflow.aggregate.loanbooks...

gains functions prep_scatter_alignment_exposure and plot_scatter_alignment_exposure, both stored in R/plots.R
calls new functions in the main plot generation script plot_aggregate_loanbooks.R

NOTE

once we have a clearer view of the P4S product, this function may need to be moved to a package and exported
in that case it would likely undergo further visual improvements

Note: I am not testing these PRs locally.
Maybe we can go over together at some point if/ how I can test these locally?


Nested eager load

Is it possible, for example, to get all the chain: Restaurant - > Reviews - > User? I cant find a way to preload user for each review in this case. (Model scheme: restaurant -> one to many -> reviews; reviews -> one to one (inverse) -> user)

Nope, right now there is no way to do nested eager loading. Your case should be doable with a through, when support for that is added. It's being worked on, but it might still be a while until it's released.

curious about this issue as I'm running into a similar need. any updates here from the kallax maintainers?


Permission denied issue (learning linux still)

error.log
If anyone is able to help with this ?

user error


Hotfix: allow development hostnames only

Hotfix to unblock Pedro and Mateo regarding deploys and auth.

LGTM
Thanks @CarlosQ96


[X] The brew doctor above contains no "Warning" lines.

What were you trying to do (and why)?
Checking outdated formulae and casks by brew upgrade --greedy --dry-run.
What happened (include all command output)?
Homebrew starts to upgrade packages.
What did you expect to happen?
Homebrew should not do upgrades with --dry-run.
Step-by-step reproduction instructions (by running brew commands)
1. Have at least one outdated formula
2. Run `brew upgrade --dry-run`

This is introduced in #10880:

(EDIT last screenshot is wrong)

#10907 might help, but I suspect @Bo98 should have a fix for this shortly.


Paimon in php

Please make sure you have read the Wechaty Puppet Service: Paimon from Wechaty Official Website before you continue writting this issue.
Question
Hi everyone  i dont know how to set up the end point for php and i already got token

, can anyone help me please

I believe this issue is related to PHP Wechaty itself.

composer install


Refactor AbstractShuffleBuffer to remove the capacity and related code

What changes were proposed in this pull request?
Remove capacity and related code within AbstractShuffleBuffer
Why are the changes needed?
Fix: #1847
Does this PR introduce any user-facing change?
(Please list the user-facing changes introduced by your change, including

Change in user-facing APIs.
Addition or removal of property keys.)

No
How was this patch tested?
No need.

Please use mvn spotless:apply -Pspark3 -Pspark2 -Ptez -Pmr -Phadoop2.8 to format the code.
And use mvn checkstyle:check -Pspark3 -Pspark2 -Ptez -Pmr -Phadoop2.8 to find checkstyle problems in advance.

Codecov Report
Attention: Patch coverage is 80.00000% with 1 line in your changes missing coverage. Please review.

Project coverage is 53.53%. Comparing base (dddcced) to head (9ecfdc9).
Report is 30 commits behind head on master.

Files
Patch %
Lines

...he/uniffle/server/buffer/ShuffleBufferManager.java
50.00%
1 Missing :warning:

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #1853      +/-   ##
============================================
- Coverage     53.53%   53.53%   -0.01%
- Complexity     2356     2392      +36
============================================
  Files           368      371       +3
  Lines         16852    17124     +272
  Branches       1540     1570      +30
============================================
+ Hits           9022     9167     +145
- Misses         7303     7418     +115
- Partials        527      539      +12

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

@rickyma Thanks for your tips, it works for me, I correct the style issue and push a new commit, PTAL.

Thanks for your contribution @maobaolong. Thanks for your review @xianjingfeng.

@rickyma @xianjingfeng Thanks for your review! As this PR merged, I close the PR #1848


Receive links to preview URLs of the changed documents

Feature Story
As a contributor, I should receive links to preview changed documents so I can ensure they look as expected.
Notes
This should only work on the staging environment.
The intended implementation is for a bot to post these links in comments on the working PR.
Product Requirements Document
Technical Tasks

[x] Add github "check" webhook endpoint to the developer-portal application
[x] Set up bot to post link in PR

preview links are available from vercel


Make possible to disable gravatar

We should have an option that makes possible to disable gravatar usage.

Does this mean than we should also add an option to upload your own photo then?

This is a long term goal. Right now we just want to remove the gravatar image and maybe show a generic user icon (like the http://fortawesome.github.io/Font-Awesome/icon/user/)


iOS renders a darker control with rounded corners inside the contentbutton

Hi,
First of all, thanks for this neat control. Trying to get something done with it, but am getting some inconsistency.
On iOS, I seem to be getting an additional grey button (perhaps?) inside the contentbutton control.
So, for example, when I set the contentbutton to have cornerRadius=0 and a color, like Red (#FF0000) it renders that just fine, but it also renders a grey control with rounded corners inside it, which darkens the overall contentbutton (from FF0000 to E91314) and you can see the rounded corners because they have a different colour.
here's an example

The text is also supposed to be pure red, and shouldn't be visible against a pure red background, but it is, due to that additional button shaped thing in there.
this is the same image as above, but desaturated and rebalanced to highlight the differences

here is another image, showing the iOS contentbutton with a white background on the bottom, vs a white control above it.

this stops you from using this in your UI if you expect to have white backgrounds, as this will be the lightest it'll ever be.
Plus, the additional rounded corners, which make it a bit untidy.
any ideas if we can fix this from our end?

ah I think I got a workaround for this
added a new ContentButtonHandler
    public class CustomContentButtonHandler : ContentButtonHandler
    {
        protected override UIButton CreatePlatformView()
        {
            var button = base.CreatePlatformView();
            button.Configuration = UIButtonConfiguration.BorderlessButtonConfiguration;
            return button;
        }
    }

and in mauiProgram.cs added a new handler entry
#if IOS
                    handlers.AddHandler<ContentButton, CustomContentButtonHandler>();
#endif

Switching the button configuration from UIButtonConfiguration.BorderedButtonConfiguration (which is what this package is doing) to UIButtonConfiguration.BorderlessButtonConfiguration achieves the seamless configuration I'm after.


[Cosmetic Bug] All Single Bloc Machine with power dont connect to Energy Cable (1.11.2-1.2.2)

All is in the title ;)

List of the machine not connecting :

Fueling Station
Rocket Monitoring Station
Rocket Assembling Machine
Satellite Terminal
Space Station Assembler
Unmanned Vehicle Assembler
Oxygen Vent

The power input plug also dont connect to the cable

Is this still a issue? Please comment if it is and i will reopen if if you are unable? We are closing old issues to see what is still relevant and needs to be fixed.


The problem of reconnection of web end network

How do you do? I'd like to ask you a question.
When using aws-mqtt, the browser closes the network. When we open the network again, we can see that the information displayed on the console has been linked, closed, offline, linked, closed and offline, and it has been recycled. I want to ask how to solve such a problem.
I look forward to your answer and thank you first.

@kmamykin

I am not sure I follow. Can you post a minimum example that exhibits the problem somewhere?

This must be fixed on master, will release a new version soon


Lift 4: update dependencies

https://groups.google.com/g/liftweb/c/sG2HWm4I3g0/m/lYvrejBkCAAJ?utm_medium=email&utm_source=footer
Vitalli: Another thing I see here is the end of community support for Jetty 9.x. All the newer versions of Jetty use Servlet versions higher than 3.1. It would be nice to bring support for the newer Jetty versions.
Andreas: Yea, and change all references from javax => jakarta

I think this is not a lift 4 only thing but should even be done in a lift 3.n+1 one.
This is becoming a blocking issue for maintenance of old lift applications as soon as they are integrated with Java framework.
The last example in date is spring / spring security. We use it for authentication, and spring is dropping the 5.x branch in August 2024, which means that from that point, we won't even have security update.
For that reason, the port to servlet 4+ should be made in a no other breaking change.
The actual change was already done in the Olon fork of lift :
Carlos Saltos wrote on the mailing list https://groups.google.com/g/liftweb/c/oApVVmAUop0/m/whz2hX98BAAJ :

Here a reference commit for an update to Jetty 11 and Servlet API 5 with a Lift fork -> https://github.com/talenteca/olon-web-framework/commit/31cc0dcf91eb7fa4693c2d3166d7a7c39b31bd79
The main changes are at JettyTestServer and HTTPRequestServlet with the new usage of multiparts upload included at Jakarta new Servlets 5.
These changes are still under testing but I hope once stable soon these changes can be included at official Lift new versions.
The new Jakarta Servlets 5 are not working for filters so an Apache commons fileupload2 for Jakarta is using now very similar to the original javax old API usage, more info at -> https://github.com/talenteca/olon-web-framework/commit/4e7a7de8059cf8250024957f7f42c7f619a69dfe

on lift versionining scheme
Actually, I think lift should adopt the following versionining scheme to let more people contribute to their use case and make Olon commit easier to integrate :

3.x branch is kept for possible security issues on currently supported scala branches. It targets lift applications in maintenance / phase out mode that are expected to be upgraded ever, but might need a security fix. It's expected to have almost no commits, but who knows

4.x branch is used for old lift application that are still actively maintained. Changes try to keep existing api with minimal changes and the immediate focus is on updating libraries and getting a better compatibility to Scala 2.13 (typically on scala XML). If stars align scala 3 compat might be imagined but it's not the primary goal, and it shouldn't be done at the price of Scala 2.13 compat. It can also be used in the longer term to bridge the path with lift 5. Some of the changes from Olon could be directly used for that branch.

5.x branch is the fresh start that focus on scala 3, cleaning up things, and all the nice ideas that are tagged lift4 for now. Since there is a 4.x branch for maintenance, that 5.x branch could be quite free to move fast and break things, which I fear would be a concern in the current existing versionining scheme.


[mdc-101] 3. Add text fields >> Install the MDC Text Field and other copyedits

Add --save option to npm commands on this an the next page; e.g.:
npm install<EMAIL_ADDRESS>--save

In "Subelements mdc-text-field__input, mdc-text-field__label, and mdc-line-ripple" maybe mdc-text-field__label should be mdc-floating-label (which is what we see in the code excerpt just above the line).

cc @kwalrath

Thanks for reporting this issue!
RE #1, --save no longer exists in recent versions of npm (definitely not in 5 and up). Recent versions now have --save-prod as an equivalent, but this is already the default behavior.
RE #2, good catch! I've updated the prose and am pushing an update now.

Hi Kenneth. Oh right, it is supposed to be the default behavior! But I'm using npm 6.x and it isn't saving by default. Strange. Also --save works just fine:
> npm -v
6.2.0
> npm install<EMAIL_ADDRESS>--save
+<EMAIL_ADDRESS>added 3 packages and audited 8969 packages in 5.095s
found 0 vulnerabilities

Wow that made me realize that my config setting was set to false for save-prod. (I wonder how that came to be. Anyhow, it's fixed.)

I've updated the prose and am pushing an update now.

Great, thanks!


Actual:
$ az vm list -ojson
$

Expected:
$ az vm list -ojson
[]
$

See https://github.com/Azure/azure-cli/issues/575#issuecomment-243504152

It looks like jq works just fine with this.
➜  ~ echo '' | jq '.[].name'
➜  ~ echo '[{"name":"x"}]' | jq '.[].name'
"x"

however this is still concerning as empty string is NOT valid json contents. So outputting [] from empty results would still be preferable.

Here's how jpterm handles empty string.
$ echo '' | jpterm
Unable to load the input JSON: No JSON object could be decoded

$ echo $?
0

Whilst jpterm does not open, jpterm returns exit code 0.
$ echo 'null' | jpterm

With this, jpterm opens as expected.


Jsrt: Use ArrayBuffer instead of BYTE for *Serialize

/cc @curtisman @digitalinfinity @jianchun @liminzhu

@liminzhu I read this PR code and it solves the API inconsistency issue by use "JsValueRef" (ArrayBuffer) consistently in both byte code buffer generation and consumption. Looks like a nice approach to me.
@obastemur There might be a Var lifetime issue. I didn't see clearly how the JsParse/RunSerialized input bufferVal being referenced in the runtime. This is similar to the scriptVal needing to be stored (in Utf8SourceInfo) to keep alive. If bufferVal not stored in runtime, its finalizer could be called early and underlying byte buffer be destroyed early.

@obastemur What's the benefit of replacing byte buffer with var?
A very common scenario of using serialization is that someone implements a service that creates a new context and serves the same script for every incoming tenant. The host needs to serialize the common script, store the buffer, and share it with every context. By changing byte buffer to var, the host would need extra getArrayBufferStorage call (one-time) and createArrayBuffer calls (per-context) for the case I mentioned.

@obastemur I'm still not getting why not just an unload callback.

@liminzhu Actually I was also pro unloadCallback but after discussing with @curtisman I'm convinced with this one. This one is compatible with others and so simple to use.
Coming back to your scenario; You can create multiple ExternalArrayBuffer from a single source and performance wise, this is not much a killer IIRC.

    (const byte*)((Js::JavascriptString*)(scriptVal))->GetSz();

Use Js::JavascriptString::FromVar instead, same below

Refers to: lib/Jsrt/Jsrt.cpp:4447 in d33b7df. [](commit_id = d33b7df98c11b9880772ea10c9a43842f9b1f24c, deletion_comment = False)

@obastemur I do like the consistency as it is similar with using arraybuffer for the script input.
How expensive is JsCreateExternalArrayBuffer? If that basically just wraps a byte buffer into an arraybuffer (and is not trying to do anything crazy like copying the buffer), then I guess it may be cheap enough, though it's still unfortunate because serialization is a perf feature and this action has to be done for every context in the scenario I mentioned.
Also trying to understand the lifetime of the arraybuffer created by JsSerialize. Does the call require an active script context and is the result buffer affiliated to the context? I'd imagine so, then the host has to either keep that context alive, or copy the arraybuffer storage into a different buffer (adding a chunk to peak memory) in order for other context to use it. In short, it does add quite a bit of hassle if someone wants to use serialized bytecode in multiple contexts.

How expensive is JsCreateExternalArrayBuffer?

IMO, the answer is not expensive + insignificant. Especially comparing to creating a whole new sub runtime / context / run stuff there...

Also trying to understand the lifetime of the arraybuffer created by JsSerialize...

This is one case vs others. IMO, once again this is not as expensive as it sounds'. What we are talking on this edge case is that an extraalloc and copy/move.

Reviewed again and left 3 nit comments, otherwise LGTM. Thanks!

Should this go to 1.4 by this time?

@jianchun you mean redirect this to 1.4 and merge?

// We currently don't free SRCINFO object so we don't want to add extra variables here.

Is this comment still accurate? If SRCINFO doesn't ever get freed, is there a possibility of a leak here with the new serializedScriptBuffer field?

Refers to: lib/Runtime/Base/ScriptContext.h:41 in ce4f197. [](commit_id = ce4f197e24a664b065d0cbb025c2e12b6d85df8d, deletion_comment = False)

Is this comment still accurate? If SRCINFO doesn't ever get freed, is there a possibility of a leak here with the new serializedScriptBuffer field?

@digitalinfinity It doesn't use SRCINFO now. Most probably that was one of the reasons @jianchun was asking me to change from SRCINFO to JsrtSourceHolder.


fix: add listener to repaint on visibility change for canvas

SUMMARY

We're seeing with the latest release of Chrome, some canvas elements on the dashboard will disappear, and only reappear on mouseover. I believe that there was an experimental feature in Chrome that saved memory by removing canvas elements when the browser tab wasn't active, and it just went GA.
Echart's library zrender is responsible for animating the canvas on mouseover which is making the canvas visible again, but we need another solution to repaint the canvas when the document is visible again.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

/testenv up

This appears to fix the behavior in the Dashboard app, but not in explore (?) should we do this lower level in the Chart component that's ?

@mistercrunch it doesn't seem to affect explore or individual charts, but only dashboards. One issue with adding it to explore is that we need to assign an event listener to the window, and only one event listener is allowed at a time, or rather each time we create a new event listener, it will overwrite the last one. This change uses the already existing visibilitychange event listener used for logging.

Oh interesting, I'm positive I've seen the behavior in explore too. I'm on Mobile now so I can't tell reproduce... I think it was big number, blank after switching tab, reappears on hover.

Ok, I was just able to repro on explore. I can add another event listener there, but outside of the slice so that we don't wind up with multiple event listeners.

@eschutho It looks like they disabled the feature that was causing the problem in 125.0.6422.76 which is already available. Can we revert this PR to avoid introducing logic that fixes a dependency problem?


Use up-to-date Rubocop

Summary
This PR relaxes the dependency on rubocop / rubocop-rails, in line with our policy of sticking with the wider Ruby community's shared consensus by default.
Additionally, we add rubocop-rake, which runs some basic sanity checks that are helpful given Rake's DSL doesn't always behave as might be expected. (These checks would've saved my bacon in the past!)

Comparing before-vs-after outputs against a large project:
- 1029 files inspected, 4300 offenses detected
+ 1029 files inspected, 5491 offenses detected, 3058 offenses auto-correctable

Top 20
Before:
$ ag -o ': \w:( \[Correctable\])? \w+/\w+:' old.output | sed 's/\[Correctable\] //' | sort | uniq -c | sort -nr | head -n 20
    905 : C: Style/StringLiterals:
    450 : C: Style/Documentation:
    286 : C: Layout/LineLength:
    165 : C: Layout/EmptyLineAfterGuardClause:
    162 : C: Metrics/MethodLength:
    142 : C: Layout/TrailingWhitespace:
    123 : C: Style/PercentLiteralDelimiters:
    123 : C: Metrics/AbcSize:
    113 : C: Rails/Output:
     73 : C: Rails/BulkChangeTable:
     69 : C: Layout/HashAlignment:
     67 : C: Layout/ExtraSpacing:
     58 : C: Rails/ReversibleMigration:
     53 : C: Layout/SpaceAfterComma:
     52 : C: Rails/FilePath:
     49 : C: Layout/SpaceAfterColon:
     48 : C: Layout/ArrayAlignment:
     43 : E: Lint/Syntax:
     43 : C: Layout/IndentationConsistency:
     41 : C: Rails/HasManyOrHasOneDependent:

After:
$ ag -o ': \w:( \[Correctable\])? \w+/\w+:' new.output | sed 's/\[Correctable\] //' | sort | uniq -c | sort -nr | head -n 20
    905 : C: Style/StringLiterals:
    497 : C: Naming/VariableNumber:
    409 : C: Style/Documentation:
    286 : C: Layout/LineLength:
    165 : C: Layout/EmptyLineAfterGuardClause:
    162 : C: Metrics/MethodLength:
    150 : C: Layout/SpaceAroundMethodCallOperator:
    142 : C: Layout/TrailingWhitespace:
    139 : C: Metrics/AbcSize:
    123 : C: Style/PercentLiteralDelimiters:
    113 : C: Rails/Output:
    106 : W: Lint/DuplicateBranch:
     73 : C: Rails/BulkChangeTable:
     72 : C: Style/StringConcatenation:
     70 : C: Metrics/CyclomaticComplexity:
     69 : C: Layout/HashAlignment:
     61 : C: Style/RedundantRegexpCharacterClass:
     59 : C: Layout/ExtraSpacing:
     58 : C: Rails/ReversibleMigration:
     53 : C: Layout/SpaceAfterComma:

Looking at the significant new entrants:
TOP_K=20
comm -13 \
  <(ag -o ': \w:( \[Correctable\])? \w+/\w+:' old.output | sed 's/\[Correctable\] //' | sort | uniq -c | sort -nr | head -n $TOP_K | cut -d: -f 3 | sort) \
  <(ag -o ': \w:( \[Correctable\])? \w+/\w+:' new.output | sed 's/\[Correctable\] //' | sort | uniq -c | sort -nr | head -n $TOP_K | cut -d: -f 3 | sort)
 Layout/SpaceAroundMethodCallOperator
 Lint/DuplicateBranch
 Metrics/CyclomaticComplexity
 Naming/VariableNumber
 Style/RedundantRegexpCharacterClass
 Style/StringConcatenation

Of these, Naming/VariableNumber is the biggest swing (being a new introduction).
I tried also with the configuration:
Naming/VariableNumber:
  EnforcedStyle: snake_case

but got remarkably similar results (both styles are used regularly):
- 1029 files inspected, 5491 offenses detected, 3058 offenses auto-correctable
+ 1029 files inspected, 5484 offenses detected, 3058 offenses auto-correctable

So on balance it's probably better to stick with the Ruby community's chosen default.

Comparing before-vs-after outputs against a large project:
- 1029 files inspected, 4300 offenses detected
+ 1029 files inspected, 5491 offenses detected, 3058 offenses auto-correctable

Top 20
Before:
$ ag -o ': \w:( \[Correctable\])? \w+/\w+:' old.output | sed 's/\[Correctable\] //' | sort | uniq -c | sort -nr | head -n 20
    905 : C: Style/StringLiterals:
    450 : C: Style/Documentation:
    286 : C: Layout/LineLength:
    165 : C: Layout/EmptyLineAfterGuardClause:
    162 : C: Metrics/MethodLength:
    142 : C: Layout/TrailingWhitespace:
    123 : C: Style/PercentLiteralDelimiters:
    123 : C: Metrics/AbcSize:
    113 : C: Rails/Output:
     73 : C: Rails/BulkChangeTable:
     69 : C: Layout/HashAlignment:
     67 : C: Layout/ExtraSpacing:
     58 : C: Rails/ReversibleMigration:
     53 : C: Layout/SpaceAfterComma:
     52 : C: Rails/FilePath:
     49 : C: Layout/SpaceAfterColon:
     48 : C: Layout/ArrayAlignment:
     43 : E: Lint/Syntax:
     43 : C: Layout/IndentationConsistency:
     41 : C: Rails/HasManyOrHasOneDependent:

After:
$ ag -o ': \w:( \[Correctable\])? \w+/\w+:' new.output | sed 's/\[Correctable\] //' | sort | uniq -c | sort -nr | head -n 20
    905 : C: Style/StringLiterals:
    497 : C: Naming/VariableNumber:
    409 : C: Style/Documentation:
    286 : C: Layout/LineLength:
    165 : C: Layout/EmptyLineAfterGuardClause:
    162 : C: Metrics/MethodLength:
    150 : C: Layout/SpaceAroundMethodCallOperator:
    142 : C: Layout/TrailingWhitespace:
    139 : C: Metrics/AbcSize:
    123 : C: Style/PercentLiteralDelimiters:
    113 : C: Rails/Output:
    106 : W: Lint/DuplicateBranch:
     73 : C: Rails/BulkChangeTable:
     72 : C: Style/StringConcatenation:
     70 : C: Metrics/CyclomaticComplexity:
     69 : C: Layout/HashAlignment:
     61 : C: Style/RedundantRegexpCharacterClass:
     59 : C: Layout/ExtraSpacing:
     58 : C: Rails/ReversibleMigration:
     53 : C: Layout/SpaceAfterComma:

Looking at the significant new entrants:
TOP_K=20
comm -13 \
  <(ag -o ': \w:( \[Correctable\])? \w+/\w+:' old.output | sed 's/\[Correctable\] //' | sort | uniq -c | sort -nr | head -n $TOP_K | cut -d: -f 3 | sort) \
  <(ag -o ': \w:( \[Correctable\])? \w+/\w+:' new.output | sed 's/\[Correctable\] //' | sort | uniq -c | sort -nr | head -n $TOP_K | cut -d: -f 3 | sort)
 Layout/SpaceAroundMethodCallOperator
 Lint/DuplicateBranch
 Metrics/CyclomaticComplexity
 Naming/VariableNumber
 Style/RedundantRegexpCharacterClass
 Style/StringConcatenation

Of these, Naming/VariableNumber is the biggest swing (being a new introduction).
I tried also with the configuration:
Naming/VariableNumber:
  EnforcedStyle: snake_case

but got remarkably similar results (both styles are used regularly):
- 1029 files inspected, 5491 offenses detected, 3058 offenses auto-correctable
+ 1029 files inspected, 5484 offenses detected, 3058 offenses auto-correctable

So on balance it's probably better to stick with the Ruby community's chosen default.


[BUG] Unable to load lap data for Saudi Arabia race

Describe the issue:
Lap data for the 2022 Saudi Arabia GP fails to load. I did some debugging and it appears to be related to the section of code that creates the total laps counter in _load_laps_data. That section executes fine for all drivers except Tsunoda (driver 22). I have not been able to figure out a workaround.
Reproduce the code example:
import fastf1 as ff1

session = ff1.get_session(2022, 2, 'R')
session.load()
laps = session.laps

Error message:
core           INFO     Loading data for Saudi Arabian Grand Prix - Race [v2.2.1]
api            INFO     Fetching driver list...
api            INFO     Fetching timing data...
api            INFO     Parsing timing data...
api            INFO     Fetching timing app data...
core           INFO     Processing timing data...
api            INFO     Fetching session status data...
core        WARNING     Failed to load lap data!
api            INFO     Fetching car data...
api            INFO     Parsing car data...
api            INFO     Fetching position data...
api            INFO     Parsing position data...
api         WARNING     Driver 241: Position data is incomplete!
api         WARNING     Driver 242: Position data is incomplete!
core        WARNING     Failed to load telemetry data!
api            INFO     Fetching weather data...
core           INFO     Finished loading data for 19 drivers: ['1', '16', '55', '11', '63', '31', '4', '10', '20', '44', '24', '27', '18', '23', '77', '14', '3', '6', '22']
Traceback (most recent call last):
 [REMOVED PRECEEDING TRACE]
  File "[REMOVED]\saudigpissue.py", line 5, in <module>
    laps = session.laps
  File "[REMOVED]\fastf1\core.py", line 945, in laps
    return self._get_property_warn_not_loaded('_laps')
  File "[REMOVED]\fastf1\core.py", line 910, in _get_property_warn_not_loaded
    raise DataNotLoadedError("The data you are trying to access has not "
fastf1.core.DataNotLoadedError: The data you are trying to access has not been loaded yet. See `Session.load`

That whole section of code is a bit of trickery, and a fake lap for Yuki is added even though it should not be added. Commenting it out is fine for this race.
@ezkomoly can you clarify, what do you mean, you can't use the 'LapNumber' attribute at all? I can't reproduce this right now.

That whole section of code is a bit of trickery, and a fake lap for Yuki is added even though it should not be added. Commenting it out is fine for this race.
@ezkomoly can you clarify, what do you mean, you can't use the 'LapNumber' attribute at all? I can't reproduce this right now.

I checked again, and it was actually my bad. I didn't look at the error message.
I had plt.savefig trying to save to my external SSD while it was not connected.
So not adding the "fake" laps to Yuki does work and
.loc[laps['LapNumber'] == 49.iloc[0]
does plot the data from lap 49.

Fixed now in new version v2.2.2


Error: [mobx-state-tree] No matching type for union

Hi
I am trying to get pre-annotations up and running for the Automatic Speech Recognition using Segments recipe.
This is my labeling interface is defined like this:
<View>
  <Labels name="labels" toName="audio">
    <Label value="Background" background="#99c1f1"/>
    <Label value="Music" background="#8ff0a4"/>
    <Label value="Silence" background="#f9f06b"/>
    <Label value="Speech" background="#f66151"/>
  </Labels>

  <AudioPlus name="audio" value="$audio"/>

  <TextArea name="transcription" toName="audio" rows="2" editable="true" perRegion="true" required="true"/>
</View>

My model is only able to predict the labels, the text area will have to be filled by annotators.
My model does predict an output, as I can see it in the task source:
{
  "id": 56,
  "data": {
    "audio": "/data/upload/2/9f0ec1c0-RADIO_YYY_1642722540.wav"
  },
  "annotations": [],
  "predictions": [
    {
      "id": 6,
      "model_version": "1655800486",
      "created_ago": "34 minutes",
      "result": [
        {
          "type": "labels",
          "value": {
            "end": 41,
            "start": 0,
            "labels": [
              "Speech"
            ]
          },
          "to_name": [
            "audio"
          ],
          "from_name": "labels",
          "original_length": 59
        },
        {
          "type": "labels",
          "value": {
            "end": 42,
            "start": 41,
            "labels": [
              "Music"
            ]
          },
          "to_name": [
            "audio"
          ],
          "from_name": "labels",
          "original_length": 59
        },
        {
          "type": "labels",
          "value": {
            "end": 59,
            "start": 42,
            "labels": [
              "Speech"
            ]
          },
          "to_name": [
            "audio"
          ],
          "from_name": "labels",
          "original_length": 59
        }
      ],
      "score": 1,
      "cluster": null,
      "neighbors": null,
      "mislabeling": 0,
      "created_at": "2022-06-21T08:34:54.745103Z",
      "updated_at": "2022-06-21T08:34:54.745110Z",
      "task": 56
    }
  ]
}

However, nothing is shown in the AudioPlus interface, and when I exit and reenter a task I get the following error:
Error: [mobx-state-tree] No matching type for union ({ pid: string?; score: (number | null?); readonly: boolean?; hidden: boolean?; parentID: (string | null?); fromSuggestion: boolean?; dynamic: boolean?; locked: boolean?; origin: ("prediction" | "prediction-changed" | "manual")?; id: identifier?; ouid: number?; results: { id: identifier?; score: (number | null?); from_name: reference((AudioModel | AudioModel | PolygonRegionModel | BrushRegionModel | KeyPointRegionModel | RectRegionModel | EllipseRegionModel | ImageModel | LabelModel | LabelsModel | ParagraphLabelsModel | TextAreaRegionModel | TextAreaModel | ChoiceModel | ChoicesModel | RatingModel | ParagraphsModel | RichTextModel | RichTextModel | TableModel | ChannelModel | TimeSeriesModel | VideoModel | BrushModel | BrushLabelsModel | HyperTextLabelsModel | KeyPointModel | KeyPointLabelsModel | ListModel | NumberModel | PairwiseModel | PolygonModel | PolygonLabelsModel | RankerModel | RectangleModel | RectangleLabelsModel | Shortc

What am I doing wrong?
Any help would be greatly appreciated.

Getting same error after adding a random UUID to the prediction:
{
  "id": 450,
  "data": {
    "audio": "/data/upload/38/62bd9e8f-RADIO_YYY_1642203600.wav"
  },
  "annotations": [],
  "predictions": [
    {
      "id": 168,
      "model_version": "1655812104",
      "created_ago": "0 minutes",
      "result": [
        {
          "id": "de37ff44-c8cc-4f1a-a372-a35137332ad4",
          "type": "labels",
          "value": {
            "end": 59,
            "start": 0,
            "labels": [
              "Danish Speech"
            ]
          },
          "to_name": [
            "audio"
          ],
          "from_name": "labels",
          "original_length": 59
        }
      ],
      "score": 1,
      "cluster": null,
      "neighbors": null,
      "mislabeling": 0,
      "created_at": "2022-06-22T06:48:18.265554Z",
      "updated_at": "2022-06-22T06:48:18.265561Z",
      "task": 450
    }
  ]
}

      "to_name": [
        "audio"
      ],

=>
"to_name": "audio",

Closed as outdated, solved or no more actual issue.


Bug in voted?

voted? can be returned as true when the playlist empties and another track is queued, if the user voted off the one and only song playing. It fixes itself after another call to ls.

Can you paste a log of messages sent? I don't really understand what your describing.

I have no log at the moment but steps to reproduce are:

Run netd and playd.
Queue one song
Vote it off
Run ls. voted? will be "1"
Let song finish, or skip it.
Queue another song.
Run ls. voted? will be "1".  ==[BUG]
Run ls. voted? will be "0".

Ah this is referencing your pullrequest?
On Mon, Oct 10, 2011 at 1:53 PM, Tristan Linnell
<EMAIL_ADDRESS>wrote:

I have no log at the moment but steps to reproduce are:

Run netd and playd.
Queue one song
Vote it off
Run ls. voted? will be "1"
Let song finish, or skip it.
Queue another song.
Run ls. voted? will be "1".  ==[BUG]
Run ls. voted? will be "0".

--
Reply to this email directly or view it on GitHub:
https://github.com/vext01/hgd/issues/119#issuecomment-2347039

Yes. Sorry. The bug exists now though.

fixed


Grab radio/checkbox icons for use in new palettes

At the moment we just have Black, Blue and Silver images. These won't fit the palette colours that I'm planning to introduce, i.e. red, green, orange etc. Wonder if there is a way to extract these images, or create a generic .psd file?

Look in the resources directory
e.g. CB2010Blue.bmp

Are there different colours i.e in PowerPoint or Excel? Or are they all the same?

Never use PowerPoint, And hardly ever use Excel, So would not know about the differences in the themes.


Update index.html

Added name and profile link to the html file.

@Subham-Pratap  contribute here https://github.com/Ishaan28malik/Get-Dev-Resources


Summary of DQM Harvesting - workloadSummary

Dear developers,
the DQM harvesting via WMAgent is already being used for a while, however we are still missing a summary/report about it.
Currently we only can see - in the summary - the 4 default graphs for the "DQMHarvestMerged" task and log about failures (not possible failures while uploading).
Is it possible to add more information to the summary pages, for example:

the server/url in which the harvesting root files were uploaded
a list with the root file names properly produced and uploaded to the DQM Gui

Moreover, it seems we are not handling the exit code/status from the Gui server, which means we have no control on what was successful uploaded or not.
Please let me know about anything that must be clarified here.
Best, Alan.

Closing this one in favor of #4483


normalization utils + protobuf

Hi George can you resolve the merge conflicts? Thanks!

just resolved and re-added folder structure - should be good for review


Publish a new version for jxl-oxide-cli

It's still stuck at 0.2.3 👀 https://crates.io/crates/jxl-oxide-cli/versions

Uh, actually never mind, it's using jxl-oxide 0.4.0.


iOS & Android: Update Activity Number on Phones

We trigger process_activity_notice_seeds_triggered_by_batch_process (WeVoteServer/activity/controllers.py) every minute. I would recommend adding a new block (at the end of the function, so it is run after all of the other processes finish):
    update_interval = 20
    time_now = now()
    if time_now.minute % update_interval == 0:
        continue_retrieving_notices_to_be_updated = True
        activity_notice_seed_id_already_reviewed_list = []
        safety_valve_count = 0
        while continue_retrieving_notices_to_be_updated and \
                safety_valve_count < 1000 and \
                when_process_must_stop > now():
            safety_valve_count += 1

I think having it run every 20 minutes would be sufficient.
Within this block we can call a function that reaches out to all registered phones so they can update their activity number.
We currently don't have a way to calculate the number of ActivityNotice entries which have not been seen by the voter (activity_notice_seen == False), but that function can be added to ActivityManager (I would recommend starting with retrieve_activity_notice_list_for_recipient).

I worry about sending 72 mostly unnecessary messages to the phone of every voter every day.
We can just keep track of the previous number we sent, and only send updates if it is different.

@SailingSteve I totally agree -- we should only reach out to the phone if there is a change.

@DaleMcGrew
process_activity_notice_seeds_triggered_by_batch_process seems fragile, and like it is going to have some big problems in the future.  I don't see how it could scale to 10X or 100X the volume.
As a first step, what would you think about me not using the "activity app" for the "What's on your mind?" comment (plus any others that would update the Cordova icon notification badge count) and simply doing the notifications in-line (which would provide almost instant notifications to Cordova apps)?
Someday we might have voters with thousands of followers, and at which point we might want to consider using an AWS Lambda function handler in Python. This would remove all the processing from our API server, and move it into an infinitely scalable cloud process.
Actually, if we replaced the batch oriented "activity app" with some real-time AWS Lambda functions it would become completely scaleable -- all "slow batch processing" issues would go away, as well as all "process collision" issues.  The voter_daily_summary processing, could also be in AWS Lambdas that were started at some specific time of day.
Alternatively we could replace the "activity" app's homegrown queuing with an AWS Simple Message Queue, which would also solve the scaling problems, and reduce the complexity.

This has been resolved, with a different solution than what was proposed in this issue.

This has been resolved, with a different solution than what was proposed in this issue.


update example

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 85.82%. Comparing base (8bc68fb) to head (723bf2f).
Report is 4 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #70   +/-   ##
=======================================
  Coverage   85.82%   85.82%
=======================================
  Files          16       16
  Lines         381      381
=======================================
  Hits          327      327
  Misses         54       54

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Load remote TemplateUrl

How do I load remote template Url's in the component directive.
const merchantid = "xfhdfggjdf67688c";

templateUrl: 'http://example.com/templates/' +merchantid + '.html',

@maytis I am not sure if this is something currently supported. it will almost certainly break with AOT. Try asking in the gitter channel and see if there are other people trying to solve a similar problem.

We don't support this currently, no.


CupertinoNavigationBar rendering issue in combination with NestedScrollView and a floating SliverAppBar

Is there an existing issue for this?

[X] I have searched the existing issues
[X] I have read the guide to filing a bug

Steps to reproduce

Create a CupertinoPageScaffold
Add a navigationBar (CupertinoNavigationBar)
Set the child to NestedScrollView
Inside the headerSliverBuilder add a single SliverAppBar with pinned: false, floating: true, properties
Inside the body of the NestedScrollView add a CustomScrollView
Inside CustomScrollView's slivers, add a SliverToBoxAdapter containing a CupertinoTextField

Expected results

Navigation bar is rendered at the top, the floating sliver app bar is rendered underneath in the main axis direction
Whenever scrolling, the floating sliver app bar is gradually hidden/shown.
When tapping on the text field, the keyboard appears.
When typing, the text field is filled and the floating sliver app bar is not being moved around.
Text field stays static on the page and does not move behind the navigation bar.

Actual results

Navigation bar is rendered on top of the floating sliver app bar (Z-axis)
When typing inside the text field, the floating sliver app bar disappears behind the navigation bar.
Based on the color of the navigation bar, it is rendered on top of the text field during typing (Z-axis)
Text field does not stay static on the page and moves behind the navigation bar

Code sample
Code sample
class MyNestedScrollView extends StatefulWidget {
  const MyNestedScrollView({super.key});

  @override
  State<StatefulWidget> createState() => _MyNestedScrollViewState();
}

class _MyNestedScrollViewState extends State<MyNestedScrollView> {
  int segmentedControlValue = 0;

  Widget _buildSegmentedControl() {
    return CupertinoSlidingSegmentedControl<int>(
      children: const {
        0: Text('First'),
        1: Text('Second'),
      },
      onValueChanged: (int? newValue) {
        setState(() {
          segmentedControlValue = newValue ?? 0;
        });
      },
      groupValue: segmentedControlValue,
    );
  }

  @override
  Widget build(BuildContext context) {
    return CupertinoPageScaffold(
      navigationBar: const CupertinoNavigationBar(
        // --- Remove this to break the navigation behavior
        backgroundColor: Colors.teal,
        // ---
        middle: Text('NestedScrollView'),
      ),
      child: NestedScrollView(
        floatHeaderSlivers: true,
        headerSliverBuilder: (BuildContext context, bool innerBoxIsScrolled) {
          return <Widget>[
            SliverAppBar(
              pinned: false,
              floating: true,
              backgroundColor: Colors.yellow,
              flexibleSpace: FlexibleSpaceBar(
                background: _buildSegmentedControl(),
              ),
            ),
          ];
        },
        body: IndexedStack(
          index: segmentedControlValue,
          children: [
            _buildPage(0, 'First Page'),
            _buildPage(1, 'Second Page'),
          ],
        ),
      ),
    );
  }

  Widget _buildPage(int pageIndex, String title) {
    return Builder(builder: (context) {
      return CustomScrollView(
        key: PageStorageKey<String>('page$pageIndex'),
        slivers: [
          SliverToBoxAdapter(
            child: Container(
              color: Colors.green,
              child: const CupertinoTextField(
                placeholder: 'Placeholder',
                padding: EdgeInsets.all(10.0),
              ),
            ),
          ),
          SliverList.builder(
            itemBuilder: (context, index) => CupertinoListTile(
              title: Text('$title $index'),
            ),
          ),
        ],
      );
    });
  }
}

Screenshots or Video

Screenshots / Video demonstration

Logs
Logs
[Paste your logs here]

Flutter Doctor output
Doctor output
[Paste your output here]

Hi @petarhristov94. Thanks for filing this. I can reproduce this on the latest versions of Flutter but this might be the expected behaviour. Are there statements in the docs saying otherwise (i.e. quoting the expected behaviour you shared) or could you help with more information on where these work as you described in the Expected?

Thanks for your detailed response.
From my understanding of this as well, I think this seems to be the expected behaviour of the interactions between the combination of widgets being used here. Not too certain about your use case for combining these widgets but you can see https://flutter.dev/community for resources and ask more questions on this, you may also get some information if you post it on Stack Overflow.
Closing as this issue doesn't seem to describe a bug or a feature request, if you disagree please comment and I'll reopen this.
Thank you!


fix worker

Description

Motivation and Context

How Has This Been Tested?

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

[ ] My code follows the code style and structure of this project.
[ ] I have updated the documentation accordingly.

:tada: This PR is included in version 0.9.8 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Prevent ability to enter an email address as a handle

At this time it’s possible to enter an email address in the “change my handle” flow but it shouldn’t be possible because, if I’m not mistaken, “@“ is not a valid character in the subdomain field.

@pfrazee doing some issue triage in the atproto, moved this one over here since it's more relevant to the app 👍

Cool, assigning to @ansh


Use a stored token in a temporary location and try first before re-login

Sessions (auth tokens) usually have a fairly long lifetime (default ~30 days). Re-using the token can provide a slight speed up and reduce the number of sessions created on the server (particularly for repeated auth API calls).
Perhaps a helper method for userLogin() that can use a temporary file storage, auth cache, and then fallback to logging in.

This is a really nice idea, I'll have a look into it.

Feature implemented
Availability

latest 2.2. branch
latest 2.4 branch
2.2.1 tag / release
2.4.1 tag / release

What changed
I've updated the userLogin() method to store / cache the auth tokens, so no action is required.
By default caching of auth tokens is enabled and they will be stored in the /tmp directory.
Here's an example:
/tmp/.zabbixapi-token-640c8a5376aa12fa15cf02130ce239a6

Security
All cached tokens will be stored in the filesystem, so there is a bit of a security concern.
To make it a bit more secure, usernames in the filename will be MD5 hashed, and the file mode is set to 0600 (read-only by user).
Change cache directory
If you want to change the path of the cache dir, set $tokenCacheDir to a different directory path.
Example:
$params = array('user' => 'foo', 'password' => 'bar');
$api->userLogin($params, '', '/some/different/directory');

Disable caching of auth tokens
If you want to disable the caching, set $tokenCacheDir to an empty value ''.
Example:
$params = array('user' => 'foo', 'password' => 'bar');
$api->userLogin($params, '', '');


Issues with installing pysam (and velocyto) in windows

Hi velocyto team,
I'm trying to install velocyto but am running into a commonly described issue when attempting to first install pysam.
Currently running python 3.6.7 on conda 4.5.11 on windows 10.
I've followed the installation instructions and run all the required dependencies through conda (conda install numpy scipy cython numba matplotlib scikit-learn h5py click) but when trying to run 'pip install pysam' I run into the following error:
"Collecting pysam
Using cached https://files.pythonhosted.org/packages/73/59/c319f1bde3019bbce4583cecb12b9e3e52ffcfbe2c96d8b1fb131c0d4fb7/pysam-0.15.1.tar.gz
Complete output from command python setup.py egg_info:
'.' is not recognized as an internal or external command,
operable program or batch file.
'.' is not recognized as an internal or external command,
operable program or batch file.
# pysam: cython is available - using cythonize if necessary
# pysam: htslib mode is shared
# pysam: HTSLIB_CONFIGURE_OPTIONS=None
# pysam: htslib configure options: None
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\Patt\AppData\Local\Temp\pip-install-ynuifqik\pysam\setup.py", line 223, in
htslib_make_options = run_make_print_config()
File "C:\Users\Patt\AppData\Local\Temp\pip-install-ynuifqik\pysam\setup.py", line 69, in run_make_print_config
stdout = subprocess.check_output(["make", "-s", "print-config"])
File "C:\Users\Patt\Anaconda3\lib\subprocess.py", line 336, in check_output
**kwargs).stdout
File "C:\Users\Patt\Anaconda3\lib\subprocess.py", line 403, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\Patt\Anaconda3\lib\subprocess.py", line 709, in init
restore_signals, start_new_session)
File "C:\Users\Patt\Anaconda3\lib\subprocess.py", line 997, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in C:\Users\Patt\AppData\Local\Temp\pip-install-ynuifqik\pysam"
I've attempted to update gcc, setuptools and everything else I could find related to the issue but always run into the same pysam installation issue as shown above.
Would appreciate any help with getting this up and running.
Thanks!

I am afraid we don't support Windows, it might accidentally work but velocyto was never tested by us on a Windows OS. However,  this particular error might not be platform specific, it started to be reported recently even on macOs. It is probably related to an update in pysam and I am worried the only solution is downgrading pysam to 1.4 version but I don't have a recipee to fix this yet and I am trying to define one. Sent from my mobile - excuse the typos

Thanks for replying about the issue.
Bugger, that's a pity. If all else fails, would it possibly work if I installed Ubuntu virtual system in windows and installed from there?
Good luck with the fix!

Just as another comment, pysam is not tested on windows either, best option would be to use linux.


Snap package

Hi,
Do we have any plan to add snap helmfile package ?

No. Would you mind contributing it? 😃

Hello. Just FYI, I've created https://snapcraft.io/helmfile-snap
Unfortunately, someone already took helmfile so I needed to request an alias to helmfile instead of helmfile-snap.helmfile


NetCommand: UnicodeDecodeError on French Windows 10 while reporting socket error

Environment data

VS Code version: 1.36.1
Extension version (available under the Extensions sidebar): 2019.6.24221
OS and version: Windows 10 Enterprise 1703
Python version (& distribution if applicable, e.g. Anaconda): Python 2.7.x
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): virtualenv
Relevant/affected Python packages and their versions: -
Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info #3977): -

Expected behaviour
Socket server error should not make whole application crash in debug
Actual behaviour
Application crash because of the decoding error
Steps to reproduce:

Have a django project (for example)
Use the django runserver command in debug mode
Fire a request
Before completion, cancel the request
Error is handled, and the debugger crashes

Logs
Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)
[...]
  File "D:\Code\myproject\pyvirtualenv\lib\site-packages\django\core\management\commands\runserver.py", line 61, in execute
    super(Command, self).execute(*args, **options)
  File "D:\Code\myproject\pyvirtualenv\lib\site-packages\django\core\management\base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "D:\Code\myproject\pyvirtualenv\lib\site-packages\django\core\management\commands\runserver.py", line 100, in handle
    self.run(**options)
  File "D:\Code\myproject\pyvirtualenv\lib\site-packages\django\core\management\commands\runserver.py", line 111, in run
    self.inner_run(None, **options)
  File "D:\Code\myproject\pyvirtualenv\lib\site-packages\django\core\management\commands\runserver.py", line 148, in inner_run
    ipv6=self.use_ipv6, threading=threading, server_cls=self.server_cls)
  File "D:\Code\myproject\pyvirtualenv\lib\site-packages\django\core\servers\basehttp.py", line 174, in run
    httpd.serve_forever()
  File "c:\python27\Lib\SocketServer.py", line 236, in serve_forever
    self._handle_request_noblock()
  File "c:\python27\Lib\SocketServer.py", line 295, in _handle_request_noblock
    self.handle_error(request, client_address)
  File "D:\Code\myproject\pyvirtualenv\lib\site-packages\django\core\servers\basehttp.py", line 80, in handle_error
    super(WSGIServer, self).handle_error(request, client_address)
  File "c:\python27\Lib\SocketServer.py", line 354, in handle_error
    traceback.print_exc() # XXX But this goes to stderr!
  File "c:\python27\Lib\traceback.py", line 233, in print_exc
    print_exception(etype, value, tb, limit, file)
  File "c:\python27\Lib\traceback.py", line 128, in print_exception
    _print(file, line, '')
  File "c:\python27\Lib\traceback.py", line 13, in _print
    file.write(str+terminator)
  File "c:\Users\foobar\.vscode\extensions\ms-python.python-2019.6.24221\pythonFiles\lib\python\ptvsd\_vendored\pydevd\_pydevd_bundle\pydevd_io.py", line 40, in write
    r.write(s)
  File "c:\Users\foobar\.vscode\extensions\ms-python.python-2019.6.24221\pythonFiles\lib\python\ptvsd\_vendored\pydevd\pydevd.py", line 1902, in write
    cmd = py_db.cmd_factory.make_io_message(s, self._out_ctx)
  File "c:\Users\foobar\.vscode\extensions\ms-python.python-2019.6.24221\pythonFiles\lib\python\ptvsd\_vendored\pydevd\_pydevd_bundle\pydevd_net_command_factory_json.py", line 249, in make_io_message
    return NetCommand(CMD_WRITE_TO_CONSOLE, 0, event, is_json=True)
  File "c:\Users\foobar\.vscode\extensions\ms-python.python-2019.6.24221\pythonFiles\lib\python\ptvsd\_vendored\pydevd\_pydevd_bundle\pydevd_net_command.py", line 58, in __init__
    text = json.dumps(as_dict)
  File "c:\python27\Lib\json\__init__.py", line 244, in dumps
    return _default_encoder.encode(obj)
  File "c:\python27\Lib\json\encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "c:\python27\Lib\json\encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 35: invalid continuation byte

I've found that there's a dict containing a str (not unicode) delivered by the OS network layer that has the value: 'error: [Errno 10053] Une connexion \xe9tablie a \xe9t\xe9 abandonn\xe9e par un logiciel de votre ordinateur h\xf4te\n', which is encoded either in latin1, cp1252 or cp850.

Actually this is a bug with ptvsd.

Bug submitted to ptvsd and I made a pull request.

Bug submitted to ptvsd and I made a pull request.

This should be open until the fix is in, but we can transfer it to the right repo. I don't have write permissions here tho, so I can't do a move. @DonJayamanne, can you do the transfer so that it moves it?


"mutable" monkey still loses attached functions

I'd like to derive an interval tree based on the state within my baobab tree using a monkey. That interval tree comes from a library that includes helpful features like searching the interval tree and others.
Unfortunately, it seems like the returned value from the monkey is stripped of its attached functions.
Here's an example:
https://tonicdev.com/tnrich/mutable-monkey-loses-attached-functions
Thanks for all the help with this monkey business :)

Your example does not show a mutable monkey. You do not seem to have disabled immutability through the monkey's option, no?
I was sure that this naming would bring this kind of puns. Keep them going :).

Right, sorry about that, I'm not able to get the<EMAIL_ADDRESS>on tonicDev. Here's the actual code I'm looking at, but it behaves the same:

var some = require('lodash').some
var IntervalTree = require('interval-tree2')
var Baobab = require('baobab');
var monkey = Baobab.monkey;
var tree = new Baobab({
    iTree: monkey([
        function function_name (argument) {
            var iTree = new IntervalTree(400)
            iTree.add(22,54,'bar');
            return iTree;
        },{
            immutable: false
        }
        ])
}, {lazyMonkeys: false});

console.log('tree.get(): ' + JSON.stringify(tree.get(['test']),null,4));
var monkeyITree = tree.get(['iTree'])

var normalITree = new IntervalTree(400)
normalITree.add(22,54,'bar');
debugger;

Here's a repo with the two errors in it if you'd like to inspect closer: https://github.com/tnrich/baobab-nested-monkey-error-1.0.0
#418
#417

The problem seems linked to the monkeys' laziness somehow.

Ok found it.


Make Realm object sendable

How frequently does the bug occur?
All the time
Description
Xcode13.3 beta need realm object conform Sendable.
Stacktrace & log output
No response
Can you reproduce the bug?
Yes, always
Reproduction Steps
No response
Version
10.20.2
What SDK flavour are you using?
Local Database only
Are you using encryption?
No, not using encryption
Platform OS and version(s)
iOS 15.4
Build environment
Xcode version: 13.3
Dependency manager and version: 10.20.2

Hi @underthestars-zhy you might be interested in checking our new feature (from v10.17.0) [@ThreadSafe](https://docs.mongodb.com/realm/sdk/swift/advanced-guides/threading/#use-the--threadsafe-wrapper) wrapper which allow you to pass thread-confined instances to another thread and it is Sendable.

The reason being, our Objects cannot cross thread boundaries, so by definition they aren't sendable. Marking an object @ThreadSafe like Diana mentioned will make them so, however.

However, In the xcode13.3 I need to add a extension like this extension RealmSwift.Realm: @unchecked Sendable {} Otherwise, try await Realm() warns.

I cannot use @ThreadSafe  on try await Realm()

Hi you cannot use @ThreadSafe with Realm you can use it on an instance of an Object, for example
let realm = try await Realm()
let results = realm.objects(Card.self)
@ThreadSafe var threadSafeObject = results.first

This threadSafeObject instance can be passed and used on a different thread.

How ever, if you run this code (let realm = try await Realm()) in Xcode 13.3 beta, it will warn you, like: Realm is not Sendable, Realm.config... is not Sendable, etc

HI @underthestars-zhy you get this warning because we use @MainActor annotation in our Realm async await constructor.
You add this annotation to your function to force running it on the main thread and remove the warning-
@MainActor
func test() async throws {
        let realm = try await Realm()
        let results = realm.objects(Card.self)
        @ThreadSafe var swiftStringObject = results.first
}

If you are using Realm only as a local database I would recommend to use let realm = try Realm(), which will execute synchronously same as await Realm() for local databases, and will not force you to run this on the main thread.

Yes, frozen objects are conceptually Sendable but we can't mark them as such because they're not a distinct type, so that's an appropriate use of @unchecked Sendable.