TrailingCommaMultiLineRule not included by default

Rule(s) related to or rule(s) missing
TrailingCommaMultiLineRule
Expected behavior
The documentation states that TrailingCommaMultiLineRule is included by default, but it doesn't seem to be the case. If you look at \TwigCsFixer\Standard\TwigCsFixer it includes TrailingCommaSingleLineRule but not TrailingCommaMultiLineRule.
Actual behavior
I had to add the .twig-cs-fixer.php file to get this working properly, otherwise this rule was not being applied.
<?php

$ruleset = new TwigCsFixer\Ruleset\Ruleset();
$ruleset->addStandard(new TwigCsFixer\Standard\TwigCsFixer());

// And then add/remove/override some rules
$ruleset->addRule(new TwigCsFixer\Rules\Punctuation\TrailingCommaMultiLineRule());

$config = new TwigCsFixer\Config\Config();
$config->setRuleset($ruleset);

return $config;

Hmm I guess this is where it was removed from the standard rules:
https://github.com/VincentLanglet/Twig-CS-Fixer/pull/259
Any plans for re-adding it?

The rule wasn't working properly on 2.x version and I had to make some changes in 3.x https://github.com/VincentLanglet/Twig-CS-Fixer/pull/265 to fix the bugs.

https://github.com/VincentLanglet/Twig-CS-Fixer/issues/257
https://github.com/VincentLanglet/Twig-CS-Fixer/issues/256

But sure, I'm opened to adding back this rule.
I didn't have time yet to try it on my personal work codebase. Does it work correctly for you (on a huge codebase ?)
Can you do the PR to add it back ? :)

I wouldn't say my codebase is huge but I tested it against 175+ files and didn't experience any issues with it, and it fixed all my trailing comma issues in multi-line arrays. But I'm not sure I have any examples of where the bug was produced previously...


The text font does not look right

Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, v1.9.1+hotfix.4, on Mac OS X 10.15
    19A582a, locale zh-Hans-CN)

[✓] Android toolchain - develop for Android devices (Android SDK
    version 28.0.3)
[✓] Xcode - develop for iOS and macOS (Xcode 11.0)
[!] Android Studio (not installed)
[✓] IntelliJ IDEA Ultimate Edition (version 2019.1.3)
[✓] VS Code (version 1.39.1)
[✓] Connected device (1 available)

! Doctor found issues in 1 category.

Without additional information, we are unfortunately not sure how to resolve this issue. We are therefore reluctantly going to close this bug for now. Please don't hesitate to comment on the bug if you have any more information for us; we will reopen it right away!
Thanks for your contribution.
Could everyone who still has this problem please file a new issue with the exact descriptions what happens, logs and the output of ' flutter doctor -v'  please.
All system setups can be slightly different so its always better to open new issues and reference related issues.


[BUG] FirestoreQuery.select with array as parameter results in 400.

Describe the bug
Passing an array to FirestoreQuery.select doesn't format the query request to Firestore properly and results in a 400
To Reproduce
Assuming a Firestore collection stage with fields name and rating:
Running:
var query: FirestoreQuery = FirestoreQuery.new().select(["name", "rating"]).from("stage", false)
var task: FirestoreTask = Firebase.Firestore.query(query)
var result: Array = yield(task, "result_query")

Results in a 400 response by Firestore
Expected behavior
var query: FirestoreQuery = FirestoreQuery.new().select(["name", "rating"]).from("stage", false)
var task: FirestoreTask = Firebase.Firestore.query(query)
var result: Array = yield(task, "result_query")

should request Firestore with   {fields:[{fieldPath:name}, {fieldPath:rating}]} instead of  {fields:[name, rating]}
Screenshots
Not applicable
Environment:

OS: Windows 11
Godot 3.5.2

Additional context
I will be submitting a PR with a fix

That is super weird. I will have to look into this closer. Very interesting. Thanks for submitting it! Will probably merge it soon, just haven't had time lately to validate it or anything.


RFC: Automated churn investigation

I'll keep this short and sweet. By chatting with @daibhin I realised that nobody is currently owning any sort of consistent investigation when a user churns. Currently it's spread out between many individuals, who share feedback ad hoc, and there isn't a consistent method for gathering it that I'm aware of.
This is valuable feedback to have and could be done relatively discretely and painlessly.
There are two potential ways we can solve this with some automation, so I'll expand on what they are and we can decide how/if to implement.
CC relevant parties @raquelmsmith @simfish85 @camerondeleone
Option 1: Email automation
We could relatively easily use Customer.io to send an automated email to users who perform the event billing subscription cancelled, asking them to share feedback on why they cancelled. We'd probably direct them to a short form, or use landing page reporting to get pre-made responses.
To my knowledge we don't have per-product cancellation events, so if we wanted to know why someone cancelled a particular product then this may fall through the gaps. That's a disadvantage. The advantage is that we could follow up with users after the event more easily.
Option 2: User survey
We could use the PostHog user surveys system to target users in greater detail, and even run experiments over the process in a better way (ask for interview, etc).
The advantage is that we could use this to hit users with a feedback request at the point of cancellation. Disadvantage is that it's more intrusive to the process and we cannot target them with surveys after the cancellation.
Feedback wanted
Any info on best practices or strongly held opinions is appreciated. My feeling is that Option 2 would be a better process, and that I can (with probably a bit of assistance around the billing system from @raquelmsmith) set this up relatively quickly.

That would feel excessive to me personally

For Option 1 it could be useful to use this suggestion from @jamesefhawkins in Slack a while back https://posthog.slack.com/archives/CT2BU33N1/p1687815678991669
There is probably also a third option...
Option 3: During cancellation flow
Certainly not the most flexible and requires the most be built but having an optional textarea / multiple choice list within the product immediately after someone cancels could be enough.
It maybe feels like Option 2 with the benefit that it more nicely fits in the flow rather than being perceived as a distraction when the user is trying to cancel, and hence be less likely to fill it out.

I love this idea. Perhaps the move is to do something more automated from customers that have spent, say, less than $100, and then offer a call for customers over $100 in exchange for the refund. I'm happy to do these calls...

Oh, I like that approach. We can then also test the different response rates between methods.
It sounds like the right way to do this then would be:

Utilize user surveys to target users with a feedback prompt at the moment then churn
Users who spend less than $100 get a text prompt asking them why they're cancelling
Users who spend more than $100 get an interview prompt (directed to Cam) offering them a $100 refund in exchange for a call

The challenge for this would be that we don't seem able to currently target surveys based on events. 🤔

Quick chat with Neil regarding ☝ suggests event-based targeting won't be done for a long while yet, for infra issues.
@daibhin How big a lift would it be to build this directly in to the product as per Option 3? 🤞

@neilkakkar You mentioned we could actually do this with surveys by setting a property. Can you let me know what's involved in that so I know whether it's a me-thing or a ask-for-help-thing? 🙏

Whenever a user churns from a product, send an event like posthog.setPersonProperties({churned_from_flags: True}) - this sets the churned_from_flags property on the current person to True.
On surveys, create a new survey that targets churned_from_flags OR churned_from_recordings, etc. etc. - and a specific URL if you'd like. Whenever this person visits the url then, they'll be shown the survey.
It's also fine if you want to update this property on the backend instead, it just means the survey wouldn't be shown immediately, but only after the property is ingested.

.. might be a good idea actually to create different surveys based on the churned product?

Took a brief look but I haven't worked in the billing area at all so I could very well be misinformed. It looks like we capture an event when someone cancels their subscription that I'm sure could be hooked into & user properties added to.
Unfortunately it's on the backend which would mean that it's not going to be instantaneous as Neil explained. I also think that even if we set the properties that there is no way to target the survey on properties that have recently been set (correct me if I am wrong @neilkakkar) meaning the survey could be sent to all historical cancellations too.
Also having looked at the cancellation flow, it all happens outside of the product in Stripe. From my recollection redirecting to conditional pages (e.g. go here if a subscription is cancelled) isn't super easy in Stripe Billing so it might not be possible to redirect them to anything we custom build. There's also no guarantee that users return to PostHog at all after they cancel.
If Customer.io support event based email triggers that might actually be the best first version of this

This feels kind of... stilted?

I triggered unsubscribe events for the other 3 products so you should be able to make all the surveys now @joethreepwood

How about "Why did you unsubscribe?"
cc @liyiy for design feedback here.

Close enough!

That looks good, you can also edit these surveys even while they're in progress if you want to re-word anything in the question or description text (just try to avoid renaming the survey name completely)

this is now running and feeding in to the feedback channel

Just adding a note here to log that, because of targeting issues with surveys, we were showing this to users on all domains. We really only want to show this within the app, so I've changed the targeting -- but surveys can't currently target multiple domains. As a result, I've created separate surveys for EU and US users.
I've also added a thank you message and 5 day wait period, as these features are now available.


Store the context of each request in the utterance log

So we don't have to drop all contextual commands when we analyze the log.

Pull Request Test Coverage Report for Build 3127

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.005%) to 65.959%

Totals

Change from base Build 3126:
-0.005%

Covered Lines:
6892

Relevant Lines:
9803

💛  - Coveralls


Cannot read property 'id' of null

Maybe you can give me a hint whats the problem here?
Email receiving is enabled
/home/user/meemo/src/mail.js:191
                        logic.add(result.id, body, [], function (error) {
                                        ^

TypeError: Cannot read property 'id' of null
    at /home/user/meemo/src/mail.js:191:41
    at Object.profile (/home/user/meemo/src/users.js:97:40)
    at /home/user/meemo/src/mail.js:184:27
    at EventEmitter.<anonymous> (/home/user/meemo/src/mail.js:133:9)
    at EventEmitter.g (events.js:291:16)
    at emitNone (events.js:86:13)
    at EventEmitter.emit (events.js:185:7)
    at /home/user/meemo/node_modules/imap/lib/Connection.js:1524:19
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickCallback (internal/process/next_tick.js:98:9)

The issue is that whatever was extracted as the username from the mail is not be found in the user listing. Basically the email is<EMAIL_ADDRESS>where username is the key in the object in the .users.json file.
Does that help?


Export Network to Image file

It would be very useful to be able to export an image of the network, for example as a PDF or JPEG.

In addition, the ability to export the computed edge list is critical.

Agree, @ells: going to spawn an new issue for that, since it's a technically distinct problem.

See #21

Resolved by ee860e5578e81d9a773f5f02bdc3fc07070184d6

Improved by 1e2afb7ddfeed837c94ae515005d86e827b9e01b, added JPEG output (plus non-functional commented code stub for BMP)


force numerical sorting

Hello! I am sorry if this was covered somewhere but I can't find an answer.
I am using columns to build some simple data tables, and it seems like the default sort treats everything as a string. so I sort by 'count' DESC (arrow down)

and I get 98, 950, 900, and 90 in that order. I need that to look like 950, 900, 98 and then 90.
Am I right to think that the only way to do this is is to override the default sorting by writing a custom plugin? Or is there a way to specify type in the schema somewhere? Thanks in advance.

Bump! Has anyone else had this issue?

OK! ALERT FELLOW JAVASCRIPT GRAPH MAKERS!
This product is obsolete. It is not supported and you should probably move to bootgrid. I have removed all of my columns class calls, implemented the bootstrap framework, and now use bootgrid as a very simple js call on ACTUAL HTML tables, as opposed to using this product with csv files.
I could not be happier, and you should all realize that this WAS a great product, but whose time is now done. Good luck to you all.
http://www.jquery-bootgrid.com/examples


Type false is not assignable to type this[keyof this] (in 2.1.4)

TypeScript Version:  2.1.4
Code
// A *self-contained* demonstration of the problem follows...
export class ClassA {
  public clean() {
    for (let prop in this) {
      if (typeof this[prop] === "boolean")  this[prop] = false;
    }
  }
}

Expected behavior:
Compiled without error
Actual behavior:
Error Type false is not assignable to type this[keyof this]

Try adding interface BoolMap { [key: string]: boolean; } and change the initial class declaration to class ClassA implements BoolMap. TypeScript doesn't know that ClassA, or subclasses thereof, could even have any dynamic boolean properties, because it can only infer so much. Alternatively, you could use a temporary reference and cast the stored value to {[key: string]: boolean} & this.
(Oh, and you're iterating the prototype chain with that; might be a good idea to guard that with hasOwnProperty.)

Should be narrowing this[prop] in the body of the if

this is a duplicate of https://github.com/Microsoft/TypeScript/issues/10530. element access expressions are not narrowed.

Thank you for the recommendation, but I'm such solutions are not suitable.
I used  (<any>this)[prop] = false;


help how to just use the transformer encoder layer?

I just want to use the transformer encoder.
While the tensor2tensor framework is too complex. I spend almost two days from beginner to give up.
And I tried the third implement like https://www.github.com/kyubyong/transformer while it may have bug my model cannot converge and even worse than simple average embedding model.
Will you please provide the encoder in Tensorflow just as a function call? No need tpu/ctpu support. Thanks.

I think T2T is modular enough and you can use just the Transformer encoder and use in a different TF project.
If you want a more easy-to-understand (but still functional) Transformer implementation, see e.g. NeuralMonkey or OpenNMT-py.

Thanks @martinpopel
I understand the implementation. While I want a pure transformer.py just rely tensorflow. Like OpenNMT-py just use pytorch.
The NeuralMonkey imported too much other modules and the OpenNMT-py is not a tensorflow lib.
So it's still not easy to use. :(

Hi @doubler, we support library usage:
from tensor2tensor.models import transformer
import tensorflow as tf

hparams = transformer.transformer_base()
encoder = transformer.TransformerEncoder(hparams, mode=tf.estimator.ModeKeys.TRAIN)
x = <your inputs, which should be of shape [batch_size, timesteps, 1, hparams.hidden_dim]>
y = encoder({"inputs": x})

@rsepassi your code seems like what I need. while miss the "targets", and throw an error
  File "train.py", line 48, in transformerEncoder     enc = encoder({'inputs': enc})   File "/usr/lib/python2.7/site-packages/tensorflow/python/layers/base.py", line 717, in __call__     outputs = self.call(inputs, *args, **kwargs)   File "/usr/lib/python2.7/site-packages/tensor2tensor/utils/t2t_model.py", line 154, in call     sharded_logits, losses = self.model_fn_sharded(sharded_features)   File "/usr/lib/python2.7/site-packages/tensor2tensor/utils/t2t_model.py", line 209, in model_fn_sharded     sharded_logits, sharded_losses = dp(self.model_fn, datashard_to_features)   File "/usr/lib/python2.7/site-packages/tensor2tensor/utils/expert_utils.py", line 230, in __call__     outputs.append(fns[i](*my_args[i], **my_kwargs[i]))   File "/usr/lib/python2.7/site-packages/tensor2tensor/utils/t2t_model.py", line 253, in model_fn     losses["training"] = self.loss(logits, features)   File "/usr/lib/python2.7/site-packages/tensor2tensor/utils/t2t_model.py", line 412, in loss     return self._loss_single(logits, target_modality, features["targets"]) KeyError: 'targets'

I have the same problem as doubler, why does the encoder need to know targets?

Hi @doubler and @marhlder , do you have any solution for the problem of missing the "targets"?

I've just set it like this:
encoder({"inputs": x, "targets": 0, "target_space_id": 0})
Works for me

Thank you very much @marhlder

Hi @doubler and @marhlder , do you have any solution for the problem of missing the "targets"?

I didn't use the tensor2tensor. The gpt model code https://github.com/openai/finetune-transformer-lm is very clear and easy to use.

So when I run the code in Keras as below, the model works:

def transformer_code(inputLayer):
hparams = transformer.transformer_base()
#hparams.hidden_size=inputLayer.shape[-1].value
#hparams.num_heads=7
encoder = transformer.TransformerEncoder(hparams, mode=tf.estimator.ModeKeys.TRAIN)
x = keras.backend.expand_dims(inputLayer, axis=2)
y = encoder({"inputs": x, "targets": 0, "target_space_id": 0})
y = keras.backend.squeeze(y[0], 2)
return y
def trainModel(args, trainInput, trianOutput, testInput, testOutput, taskName, tags):
inputLayer = keras.layers.Input(shape=(len(trainInput[0]),
len(trainInput[0][0])), dtype='float32')
inputAfterDense = keras.layers.Dense(512, activation='relu')(inputLayer)

crfLayer = CRF(len(tags), sparse_target=True, name='result')

y = keras.layers.Lambda(transformer_code)(inputAfterDense )
modelPred = crfLayer(y)

model = keras.Model(inputs=inputLayer, outputs=modelPred)
model.compile(
	optimizer='adam',
	loss = {'result': crfLayer.loss_function},
	metrics={'result': crfLayer.accuracy}
	)
print 'finish model setting'
print model.summary()

but if I remove the Dense layer 'inputAfterDense = keras.layers.Dense(512, activation='relu')(inputLayer)' the training breaks and prediction accuracy is nearly zero all the time.
Why is that?

I am a newbie into the transformer and I wanted to replace the transformer with LSTM which I used it recently, any insight would be appreciated.
The following is my network and the input shape is (batch_size,time_step,no_feature): but I have the target with shape of (batch_size,1) meaning 1 label for each 100 time_step,
adam = optimizers.Adam(lr=0.00075,decay=1e-6)
input1 = keras.layers.Input(shape=(X_tr.shape[1], 25),name='inp1')
input2 = keras.layers.Input(shape=(X_tr.shape[1], 11),name='inp2')
x2 = Embedding(40, 5,name='emb',mask_zero=True)(input2)
x2 = Lambda(lambda x2: x2, output_shape=lambda s:s,name='lambdax2')(x2)
x2 = Reshape((int(x2.shape[1]),int(x2.shape[2]*x2.shape[3])),name='reshape1')(x2)
merge = keras.layers.Concatenate(axis=-1,name='concat')([input1, x2])
mask  = Masking(mask_value=0.,name="maski")(merge)
inputAfterDense = keras.layers.Dense(512, activation='relu',name='densetransformer')(mask)
transformer = keras.layers.Lambda(transformer_code,name='lambdatransformer')(inputAfterDense)
out = Dense(1,activation="sigmoid")(transformer)
model = keras.models.Model(inputs=[input1, input2], outputs=out)
model.compile(loss='binary_crossentropy', optimizer=adam, metrics=[f1,sensitivity, specificity,'accuracy'])
model.summary()

history = model.fit_generator(train_gen,validation_data=val_gen,#class_weight=class_weight,
                              validation_steps=val_steps,steps_per_epoch=train_steps
                              ,epochs=1,verbose=1,shuffle=True,callbacks = callbacks_list)

but when I want to train the model It returns the following error:
ValueError: Error when checking target: expected dense_1 to have 3 dimensions, but got array with shape (127, 1)


Rule list context menu operations should act on the full selection

Right now they only act on the item under the context menu, even though the default selection UI implies otherwise.


stdout always detected as a character device

After 78ac85cce1d stdout is always detected as a character device. Is this expected?
This breaks my usage of wazero in RethinkRAW.
To Reproduce
git clone --branch atty https://github.com/ncruces/wazero-bugs
cd wazero-bugs
go run main.go

This is the offending code in dcraw:
https://github.com/ncruces/dcraw/blob/8dcf1ff24ad0b5a09c8b0c1030d983c09d159284/dcraw.c#L10314-L10317
Expected behavior
IMO, isatty should not blindly return true in all cases. I'm capturing the output of a tool to a byte.Buffer. It's expected that it is binary. I can patch dcraw, but other tools (e.g. gzip) will similarly complain.
Environment (please complete the relevant information):

Go version: go1.19.4
wazero Version: 78ac85cce1d674c1c47c7ff2dacbfbc2e3a3e2b4
Host architecture: amd64
Runtime mode: compiler

Thanks for the report and agree this is a bug and will sort it out

@robbertvanginkel so I need to be more precise about character devices. I'll send a PR in a bit and likely we need to special case os.Stdout, etc. and maybe also Stat.Sys() if we can get the real char device info (not always possible, but anyway I'll play with it). This is simpler now that we know the file system context is fixed for the life of the module.

Can confirm, everything working. Thanks!


[DNM] Cross-repo testing

Replace this paragraph with a description of your changes and rationale. Provide links to external references/discussions if appropriate.

Resolves SR-NNNN.

https://github.com/apple/swift-driver/pull/721
@swift-ci please smoke test


signature problem on testing models

Hi,
since to build dev apk we need to give Oculus signature file in assets folder but it's becoming very difficult to generate apk based on each device id.
Is there any way to generate single apk which wil fit for all our testing models.

Hi @spakanati123. sideloadvr.com might be of use to you.

Thanks @liaxim

@spakanati123 Can we close the issue?

Closing for now. Let us know if you have other questions.


Bulk export/query XACML policies (or Entitlements)

How can I bulk export/query XACML policies from product-is? The aim is to query all or a subset of policies  and keep in-sync (periodically) with the services (at the PEP side) to avoid network calls to the IS server.

To refine/simplify my question further: is it possible to have remote access to policy store or PAP via REST or SOAP based API?

You can use the EntitlementPolicyAdminService SOAP service for this. There are methods to get a single policy or all the available policies in PAP. For more info on calling admin services, refer the section Web Service API in [1]
[1] - http://xacmlinfo.org/2013/09/27/xacml-policy-administration/


Wave model time series

-New UI interface with time series data
-Significant_wave_height, energy_period arrays used to calculate wave energy converter performance
-API calls hidden, still in development
-New exclusive_var used to specify whether probability distribution or time series data are used in calculations
-New 3-hour time series outputs
-New wave resource time series data library with defaults matching locations of JPD file locations
-New function in library.cpp for handling changes to time series library

Need to delete file_name from defaults for ME Wave / None configuration.
Do we need both wave_data_file_ts and wave_resource_filename_ts for the time series option? Same question for wave_data_file_freq and wave_resource_filename for the frequency option.

I suppose not. the vars with filename are what go into the compute modules. So I think I will keep those to be consistent.


Fixed #28233 Improve the "price_per_page" example in the aggregation documentation

The meaning of the new wording isn't clear to me. Maybe including the formula of the calculation would help.

Added a simplified SQL query that is executed. Please, let me know if it requires any more change.

The added explanation is good. But the example itself doesn't demostrate a very useful use case. Maybe change the Sum to Avg? @Rajesh-Veeranki

We can make it Avg which makes it evaluate the average of the price per page of all books. (In short Avg of Avg)
I also suggest we also add an additional example like this
Book.objects.all().aggregate(price_per_page=Sum(F('price'), output_field=FloatField())/Sum(F('pages'), output_field=FloatField())) which makes it evaluate the more meaningful average in my opinion and translates to
SELECT (SUM(price) / SUM(pages)) AS avg_price_per_page FROM app_book

That's a good idea! I think both use cases will be useful.

I have updated the examples. Wondering, if there is a standard test database against which these queries are executed? Please let me know if there is one. As of now, I'm using dummy values, which may be fine?. Also please suggest any more changes required.

@jarshwah, I see you added the original "Cost per page" example in f59fd15c4928caf3dfcbd50f6ab47be409a43b01. Any input here? Personally, I think I would prefer to see simple explanations of queries in English rather than than diving into how they translate to SQL.

Considering it's a cheat sheet, the aim is to show how different aggregates can be constructed. I readily admit that cost_per_page is a trivial and silly example, and am open to the changing - provided multiple expressions are combined in some way.
I think if you want to have examples that link simplified SQL and queries together, then another section would probably be more appropriate.
That said, I don't feel strongly about this. I'd usually defer to @timgraham regarding docs

I haven't used aggregation too much so I'm not sure if I can give a good suggestion. I tired Book.objects.aggregate(alias=Max('price') - Avg('price'), output_field=FloatField())) as a naive try to compute the price difference between the most expensive book and the average price, but it gives 'FloatField' object has no attribute 'resolve_expression'.

output_field is an argument of expressions, not aggregate/annotate. Either of these will (should) work:
Book.objects.aggregate(alias=ExpressionWrapper(
    Max('price') - Avg('price'), output_field=FloatField()
))

#or

Book.objects.aggregate(alias=Max('price', output_field=FloatField()) - Avg('price'))

Oops, do you think it's feasible/worth trying to improve that error message? Is it correct to say that all kwarg values should be expressions, i.e. have resolve_expression?

Actually yes, improving that error would be really helpful. I'm fairly sure all args and kwargs to aggregate and annotate should be expressions

Proposed patch:
diff --git a/docs/topics/db/aggregation.txt b/docs/topics/db/aggregation.txt
index 2bc30f8..129df68 100644
--- a/docs/topics/db/aggregation.txt
+++ b/docs/topics/db/aggregation.txt
@@ -67,11 +67,11 @@ In a hurry? Here's how to do common aggregate queries, assuming the models above
     >>> Book.objects.all().aggregate(Max('price'))
     {'price__max': Decimal('81.20')}

-    # Cost per page
+    # Difference between highest price book and average price
     >>> from django.db.models import F, FloatField, Sum
-    >>> Book.objects.all().aggregate(
-    ...    price_per_page=Sum(F('price')/F('pages'), output_field=FloatField()))
-    {'price_per_page': 0.4470664529184653}
+    >>> Book.objects.aggregate(
+    ...     price_diff=Max('price', output_field=FloatField()) - Avg('price')))
+    {'price_diff': 46.85}

     # All the following queries involve traversing the Book<->Publisher
     # foreign key relationship backwards.

and created https://code.djangoproject.com/ticket/28277#ticket for adding the discussed validation.

Eliminating the confusing example and adding this makes sense to me. BTW you missed the Max import
from django.db.models import FloatField, Max

Thanks for pointing out the imports that need updating. I think we don't need to import Avg/Max again since they are used previously in the shell session.


fix ice connectivity race condition

Fixes the race condition around ice connection by delaying the send_done in the ice_gathering loop until we either finish processing all ice candidates, or until we hit a timeout.

hey if it works then it works. is the go side correct?

afaict the Go side remains correct. I can go into more detail if you'd like, but the short version is we hit another expression of the "async is weird and deals with infinite loops weirdly" problem that we encountered a bit back, and this fixes it. So it turns out this wasn't really a webrtc issue at all.

dope.


add missing end parens

Hi there,
Thanks for creating this pull request.
Please ensure you follow the PR checklist and not just tick things off; otherwise, we won't take you seriously. Respect the requirements to earn the respect of your fellow maintainers. You will also save everyone time :)
Since you are new here, please slow down and read our contributing guidelines, as we see that you may still need to catch up on a few things.

Here are some examples (expand)

Do not edit files directly on GitHub – while you can, it's not a good idea.
Typos and formatting errors can break the tests.

Use the correct way to link issues.
Add the issue number only in the description of the PR by updating the XXXXXX. Do not just add issue numbers everywhere and anywhere you feel like.

Keep mentions and review requests to a minimum.
We understand you are excited about contributing, and our maintainers will get back to you when they can.

Do not work directly off your main branch.
You can always create a new branch for the changes you are working on. That way, you can sync change to your PR branch as the main repository moves ahead while your PR is waiting in the merge queue.

Don't worry. You don't have to close this PR. Feel free to ask specific queries on improving your PR here; someone will guide you.
We are happy you are excited to contribute and appreciate you taking the time to help us. Looking forward to more contributions!
Happy Contributing.

Hey @nedredmond
Thank you for opening this pull request.
This is a standard message notifying you that we've reviewed your pull request and have decided not to merge it. We would welcome future pull requests from you.
Thank you and happy coding.

Hey @nedredmond
Thank you for opening this pull request.
This is a standard message notifying you that we've reviewed your pull request and have decided not to merge it. We would welcome future pull requests from you.
Thank you and happy coding.

Whoops, didn't realize I made the PR. I saw another more junior user had made a PR for the same issue. Thought I shut this down before submitting. Didn't mean to waste your time.


Fix duplicates includes in assembly jar

After shimlayer was merged if you didn't clean you saw an error:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project rapids-4-spark_2.12: Error creating shaded jar: duplicate entry: META-INF/services/com.nvidia.shaded.spark.orc.DataMask$Provider -> [Help 1]
The problem was actually there already that we are including the already built assembly jar, the new service loader transformer that was added just caused it to fail.
This modifies it to only include the META-INF pom file from that package when assembling:
0 Thu Jul 23 21:51:06 CDT 2020 META-INF/maven/com.nvidia/rapids-4-spark_2.12/
5987 Thu Jul 23 21:47:32 CDT 2020 META-INF/maven/com.nvidia/rapids-4-spark_2.12/pom.xml
123 Thu Jul 23 21:50:08 CDT 2020 META-INF/maven/com.nvidia/rapids-4-spark_2.12/pom.properties
This is what we were including previously.

build


PyPi Release Version

Hi Jason,
I'm thinking that this library would be a lot more useful if people could find it on PyPi as this would make it easier to install but also mean it could be used directly in Home assistant and probably other system without the need for the user to install/setup.
I'm just glancing at the process and I've already noticed that it would require removing all "print" statements and removing the dockerfile. So, I'm just wondering if you have any thoughts on the best way to proceed. The last thing I want to do is just fork this and create a separate PyPi version without seeing if you have any preferences/ideas on the best way to go.

That's a great idea!  Take a look at this and let me know if the interface would be appropriate for Home Assistant and other systems: https://github.com/jasonacox/tuyapower/tree/master/tuyapower
I'll submit it if this looks like it would work.  Something like this:
import tuyapower

# Poll Smart Swich for Power Data
(on, w, mA, V, err) = tuyapower.deviceInfo(PLUGID,PLUGIP,PLUGKEY,PLUGVERS)

# Check for error
if err != 'OK':
        print(' ERROR: %s\n' % err)

# Print Output
print('    Switch On: %r' % on)
print('    Power (W): %f' % w)
print('    Current (mA): %f' % mA)
print('    Voltage (V): %f' % V)

Give this a try:
# Install required libraries
 sudo apt-get install python-crypto python-pip
 pip install pycrypto
 pip install pytuya
 pip install Crypto		# some systems will need this
 pip install pyaes		# some systems will need this
 pip install tuyapower

# Run a test
 $ python
 Python 2.7.13 (default, Sep 26 2018, 18:42:22)
[GCC 6.3.0 20170516] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import tuyapower
>>> PLUGID = '01234567891234567890'
>>> PLUGIP = '<IP_ADDRESS>'
>>> PLUGKEY = '0123456789abcdef'
>>> tuyapower.deviceInfo(PLUGID,PLUGIP,PLUGKEY,'3.1')
(True, 1.2, 70.0, 121.1, 'OK')
>>> tuyapower.deviceInfo(PLUGID,PLUGIP,PLUGKEY,'3.3')
(False, -99.0, -99.0, -99.0, 'Timeout polling device')
>>>

Unfortunately I don't think that would work for inclusion in PyPi which is arguably the best way to get it into Home Assistant (and possibly others). However, it might be ok for inclusion in HACS, but I've not looked into that.
I did however, start working on something myself for PyPi, that reports the stats via MQTT which -speaking from my own perspective- is a more complete way of dealing with things. The less the end user needs to do to get this up and pulling stats into HA the better it will be for them and for general uptake.

Understood.  I uploaded it to PyPi as others may want to use it (https://pypi.org/project/tuyapower/).  Good luck with your project!  Let me know how it goes.


Loosen the win32-api dep to allow for the latest releases

We're using a version from 2015.
https://github.com/cosmo0920/win32-api/blob/master/CHANGES
Signed-off-by: Tim Smith<EMAIL_ADDRESS>

Is this failure relevant to this PR?
https://buildkite.com/chef-oss/chef-chef-main-verify/builds/12035#274048ce-7a33-4a14-8078-eed863d65049

Hit retry and it works


Log the version at startup

Asked by the sys admin to be able to track in Kibana the version
of the print.

LGTM


Remove support for GD images in Stream implementation

Q
A

Documentation
no

Bugfix
yes

BC Break
yes

New Feature
no

RFC
no

QA
no

Description
GdImage-backed Stream instances have evidently never worked, but feature removal needs to happen in a new major.
See #57, #134, and #147 for full details behind removal.

Should we re-introduce the stream compat check in StreamFactory as it was removed in https://github.com/laminas/laminas-diactoros/pull/45/files#diff-565876e19198a94175bb659207c750dcb80cbbfa104490c6fad5322dd05d5febL49-L53 or do we verify that by waiting for the check in __construct => setStream of Stream#__construct?

It's stil there, just done in a different way. In isValidStreamResourceType, we check to see if (a) we have a resource, and (b) if it's in the list defined by the class constant ALLOWED_STREAM_RESOURCE_TYPES... which now only defines stream as a valid resource type.
Keeping it this way allows us to more easily expand it in the future, and keeps the same check that we had prior to adding GD support anyways.

It's stil there, just done in a different way.

StreamFactory validates valid resource type by counting on Stream#__construct to validate resource.
So yes, it is there, but its more implicit than explicit.
That was what I wanted to clarify. If we are happy with implicit validation, thats okay for me as well.

I would like mentions of GD removed from documentation in v2 folder too so it won't be on live site anymore when this is released since it never worked.

Done.

Thank you!


perf: use dryAeSdk to collect multisig info

fixing one part of the issue raised here
https://github.com/superhero-com/superhero-wallet/blob/d233c480fcbc0a0ab1ddf79deda42b30d96495c5/src/protocols/aeternity/libs/AeSdkSuperhero.ts#L35
by simply using dryRunSdk that doesn't have accounts and will not call our overridden _resolveAccount method, every time we are updating multisig vaults info.

In general LGTM. The issue with v3 contracts has been fixed.
I got an issue with pending multisig Tx displaying, but only once. I tried to reproduce it, but my attempts were not successful.
Probably it worth to add it to our backlog...
steps to reproduce it:

create multisig
propose Tx from new multisig with 0 balance
observe an issue:

https://github.com/superhero-com/superhero-wallet/assets/69896204/49c01b6f-4422-42a4-b343-626f736c137b


FIXES: #2597

Historical Transaction retrieval no longer needs to pass current validity check;
Introduced additional RPC error code for database deserialization error.

Codecov Report

Merging #3084 into develop will decrease coverage by <.01%.
The diff coverage is 31.57%.

@@             Coverage Diff             @@
##           develop    #3084      +/-   ##
===========================================
- Coverage    68.74%   68.73%   -0.01%
===========================================
  Files          686      686
  Lines        55820    55821       +1
===========================================
- Hits         38371    38370       -1
- Misses       17449    17451       +2

Impacted Files
Coverage Δ

src/ripple/protocol/ErrorCodes.h
100% <ø> (ø)
:arrow_up:

src/ripple/app/misc/Transaction.h
100% <ø> (ø)
:arrow_up:

src/ripple/protocol/impl/ErrorCodes.cpp
75% <ø> (ø)
:arrow_up:

src/ripple/app/misc/impl/Transaction.cpp
54.09% <0%> (+0.87%)
:arrow_up:

src/ripple/app/ledger/ConsensusTransSetSF.cpp
0% <0%> (ø)
:arrow_up:

src/ripple/rpc/handlers/Tx.cpp
62.5% <50%> (ø)
:arrow_up:

src/ripple/app/ledger/impl/TransactionMaster.cpp
53.19% <71.42%> (+2.08%)
:arrow_up:

src/ripple/ledger/impl/ApplyView.cpp
80% <0%> (-2.61%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 38f8211...4956612. Read the comment docs.

the xcode11 failure in travis is because it's picking up boost 1.71 from homebrew instead of the 1.70 that we install. Please ignore this failure for now -- I'll submit a separate CI fix for that.

I'm very excited to see this bug fixed. Thank you for tackling it. :tada:

@seelabs sorry took forever to see and resolve this. made the changes accordingly!

Looking now. I think the commit message still needs to be addressed tho, right?
Ref: https://github.com/ripple/rippled/pull/3084#pullrequestreview-299518291

@ShangyanLi One minor thing, I find it easier to review things if they aren't squashed down until the review ends. With a patch like this it isn't too bad, but with larger patch I find it much easier if I can easily see what's changed since the last time I looked. Thanks!

@seelabs updated message - will take note re squashing in the future

@ShangyanLi Thanks for fixing the commit message, but it's still not quite right. To see what I'm seeing, run the following command: git log --oneline Putting a blank line between the subject and the body should fix this.

@seelabs 3rd time's the charm 😂

@ShangyanLi Commit looks good here. Thanks for fixing that. :+1:


[enhancement] Add enque/deque counters to PerValidatorQueue

Summary
We only had the drop counter, but it would be good to have counters for queued, dequeued messages in libra_channel. This PR adds those counters.
Test Plan
Compiles successfully
Related to #1509

@bora-libra r=dmitri-perelman

@bors-libra r=dmitri-perelman

:pushpin: Commit 5849fdc has been approved by dmitri-perelman

@bors-libra r=dmitri-perelman

:bulb: This pull request was already approved, no need to approve it again.

There's another pull request that is currently being tested, blocking this pull request: #1524

:pushpin: Commit 5849fdc has been approved by dmitri-perelman

:hourglass: Testing commit 5849fdc3e36e60293d7ac8c9f7bcecf44a1b0cb6 with merge 847b9ee885b6a4911c64a846a5cc937a9174f4c2...

:sunny: Test successful - checks-circle_commit_workflow
Approved by: dmitri-perelman
Pushing 847b9ee885b6a4911c64a846a5cc937a9174f4c2 to master...


Add note about region discovery.

If you omit bucket region from your config and it is not residing in
us-east-1, the minio client fails with not-very-helpful errors: 400
error unless the aws policy: s3:GetBucketLocation is assigned to your
aws user.
This patch makes a note of this in the docs.

this is related to this issue discussed in the minio-go repo: https://github.com/minio/minio-go/issues/923

LGTM. Thanks!


Fix type members in REXML

Motivation
None of the classes from REXML are generic, we have to fix type members.
For example with Document:
# typed: true

class Foo
  extend T::Sig

  sig { returns(REXML::Document) } # Error:  Malformed type declaration. Generic class without type arguments REXML::Document
  def foo
    REXML::Document.new
  end
end

With this PR, the previous snippet returns No errors! Great job..
Test plan
See included automated tests.

We have a policy of testing changes to Sorbet against Stripe's codebase before
merging them. I've kicked off a test run for the current PR.  When the build
finishes, I'll share with you whether or how it failed. Thanks!
Stripe employees can see the build result here:
→ https://go/builds/bui_Hjp2NZhc9Qn2yh


Build system

Hi @johandc,
This is a PR to fix a few build system related failures.  Hope you like it.
For csp_if_i2.c, I just removed the file from build because meson wasn't building it. Do you have any opinion how to proceed with it?
For static rtable, I haven't remove it just yet.  If you are OK to remove it, I'll add the file to the commit.
Other changes should be trivial.  Let me know if you have any question.

Thanks a lot for this.
Good suali stuff all around. Even caught the frame buf alignment issue.
Yes for now, if I2C needs to be deactivated. Gomspace has promised to port and test that during the fall.


(PUP-9943) Remove user type's custom retrieve()

The user type has a long-standing custom retrieve() method which
returns values not in keeping with the standard retrieve() from
Puppet::Type.retrieve(). This patch removes the custom method
and adjusts the tests dependent on the particular behavior from
it.

travis is failing to install jruby 9.2, seems to be a travis issue.

CLA signed by all contributors.

@seanmil, can you rebase this PR?  Thanks

The override may be causing us to skip the suitable provider check (in the base implementation) and may be the cause of https://tickets.puppetlabs.com/browse/PUP-1162, https://tickets.puppetlabs.com/browse/PUP-1131, https://tickets.puppetlabs.com/browse/PUP-3148

Just noting some differences:

User#retrieve doesn't check for suitable provider
User#retrieve returns a Hash, not a Puppet::Resource, though the Type#retrieve_resource will create a Puppet::Resource if given a Hash. However, in the former case, the resource is created by passing in the the hash as the parameters, but in the latter case, an empty resource is created and []= is called for each managed property. It's not clear if those result in the same resource object once the override is removed?
User#retrieve only retrieves the current state for properties that are specified in the manifest, unlike the base implementation which always retrieves the ensure state (provided the type is ensurable and doesn't override needs_ensure_retrieved). This may explain some variations in ensure behavior as described in https://tickets.puppetlabs.com/browse/PUP-9033

@seanmil, can you rebase this PR? Thanks

Done. I'm not sure why the Appveyor tests are failing, but it doesn't seem related to this change.

jenkins please test this on
debian9-64a,debian9-32a,fedora29-64a,osx1013-64a,osx1014-64a,redhat5-64a,centos6-32a,redhat7-64a,redhat8-64a,redhatfips7-64a,sles12-64a,sles12-POWERa,sles15-64a,solaris10-64a,solaris114-64a,ubuntu1804-64a,windows2016-64a,windows2019-64a


Adding the ability to get filename from header field

Adding the ability to get filename from header field for ios (android already has it)

Thanks!


docs: Fix createdAt date formatting in Movie Quotes App tutorial

Change all occurrences of new Date(quote.createdAt) to new Date(Number(quote.createdAt)) otherwise the date renders as Invalid Date.
Related commit in the companion application repo: https://github.com/platformatic/tutorial-movie-quotes-app/commit/219d86d992a4135ac4bde204461f0b128495dde3

There's plenty I don't understand here but it seems that for quotes.created_at in the tuorial sqlType ends up as datetime in the entity object and that ends up falling through to the default graphql.GraphQLString in sqlTypeToGraphQL.  So the conversion to number seems to be required.


Popovers get blurry when zooming out with browser

I guess this is just an inevitable fact of using transform-3d, but I found it curious that unstable_portal actually produces blurrier results, and I wondered if we know why that might be. Other than that one issue I've personally had better luck with unstable_portal than without.
I assume there's no good alternative to transform-3d here?
Zoom 100%:

Zoom 90%
Left side: normal popover --- Right side: unstable_portal
(https://stackoverflow.com/questions/ask)

It seems to be an issue on popper.js (or on the browser itself).
Related:

https://github.com/FezVrasta/popper.js/issues/764
https://github.com/FezVrasta/popper.js/issues/682

Thanks for taking the time to document the suspected underlying cause @diegohaz!
Do you have any idea why unsafe_portal makes the issue worse? Is it simply a matter of requiring a "larger" transform-3d value to position properly, increasing blurriness? Is it simply a mystery?
The reason I ask is that the default entry motion without unsafe_portal is triggering a Tippy element within my popover as it slides under the mouse and into view. Definitely not the end of the world in any case.

It’s a mystery for me. I can’t really reproduce it. Not sure if it’s the code (tried with the popover examples on the site), the browser or the MacBook screen, which has high PPI. I can check later with my wife’s PC, but its screen has also high resolution, so I’m afraid I’ll not be able to see this issue.


Can't read json from /data/addons/git/*/repository.json while adding this repository

Home Assistant release with the issue:
0.97.1
Hass.io supervisor: 172
Last working Home Assistant release (if known):
N/A, fresh install
Operating environment (Hass.io/Docker/Windows/etc.):
Hostname: hassio
System: HassOS 2.12
Hardware: Raspberry pi 3B+
Description of problem:
Clean install, while trying to add this repo I got the following error
19-08-10 17:38:55 INFO (MainThread) [hassio.store.git] Clone add-on https://github.com/home-assistant/home-assistant repository
19-08-10 17:39:08 ERROR (MainThread) [hassio.utils.json] Can't read json from /data/addons/git/9e7b8c66/repository.json: [Errno 2] No such file or directory: '/data/addons/git/9e7b8c66/repository.json'
19-08-10 17:39:08 WARNING (MainThread) [hassio.store.data] Can't read repository information from /data/addons/git/9e7b8c66/repository.json

Only other repo installed is https://github.com/hassio-addons/repository which works fine.
https://github.com/home-assistant/home-assistant appears as 'unknown' in GUI Addon Store-Repositories.

It's not an add-on repo. So don't add it?

It's not an add-on repo. So don't add it?

It was my understanding I had to first add a repository in order to use components such as  xiaomi aqara?

That is built in.

That is built in.

Thanks for clarifying.


mounts: implement new org.osbuild.bind mount

[draft: to get early feedback, I think adding a few more test cases would make sense here too but it should be okay]
[edit: the remaining issue is that org.osbuild.user may need to be able to create /var/home as useradd will not follow symlinks when creating dirs]
This adds a new org.osbuild.bind mount feature to the osbuild
mount modules. This allows to (r)bind mount parts of another mount
into the tree (or replace the default tree for a stage entirely).
The use case is the bootc install to-filesystem where we get
a populated disk and need to do customizations directly there
without going through an intermediate tree.
Note that right now only "--rbind" is supported and used but
we could trivially change that to become an option in either
direction. Given that the main use-case right now is to be
paried with org.osbuild.ostree.deployment and here the
rbind is crucial I would leave that the default.
Here is an example what this looks like:
        {
          "type": "org.osbuild.users",
          "options": {
            "users": {
              "alice": {
                "home": "/home/alice",
                "groups": [
                  "wheel"
                ],
                "password": "$6$NV3P7UzUqP3xb1ML$3qnHpWs037VRTaOc.kirQ4.RwNz4gu9dkhAhpBYVCkHw8CMhpBKnegyyqw0QfURowarZnRnQi.jo4JEzIOvPO/",
                "key": "ssh-rsa AAA<EMAIL_ADDRESS>              }
            }
          },
          "devices": {
            "disk": {
              "type": "org.osbuild.loopback",
              "options": {
                "filename": "disk.raw",
                "partscan": true
              }
            }
          },
          "mounts": [
            {
              "name": "part4",
              "type": "org.osbuild.ext4",
              "source": "disk",
              "target": "/",
              "partition": 4
            },
            ...
            {
              "name": "ostree.deployment",
              "type": "org.osbuild.ostree.deployment",
              "options": {
                "source": "mount",
                "deployment": {
                  "default": true
                }
              }
            },
            {
              "name": "bind",
              "type": "org.osbuild.bind",
              "target": "tree://",
              "options": {
                "source": "mount://"
              }
            }
          ]
        },

It would be cool to be able to do something like this, but rather define it at the pipeline level such that the tree:// for all stages in a pipeline is actually a mount://.

It would be cool to be able to do something like this, but rather define it at the pipeline level such that the tree:// for all stages in a pipeline is actually a mount://.

This idea did come up while we were discussing approaches for this problem.  What I like about the solution in the PR though is that it requires very minimal changes (none of which are internal to osbuild's core) and fits a lot cleaner into the current flow.
There's definitely something a bit more elegant to your proposed solution though.  The general principle that pipelines operate on a single tree is maintained and it makes it easier to reason about when that holds.


VSCode plugin on save garbles up the file.

Environment information
CLI:
  Version:                      1.6.3
  Color support:                true

Platform:
  CPU Architecture:             aarch64
  OS:                           macos

Environment:
  BIOME_LOG_DIR:                unset
  NO_COLOR:                     unset
  TERM:                         "xterm-256color"
  JS_RUNTIME_VERSION:           "v20.11.1"
  JS_RUNTIME_NAME:              "node"
  NODE_PACKAGE_MANAGER:         "npm/10.2.4"

Biome Configuration:
  Status:                       Loaded successfully
  Formatter disabled:           true
  Linter disabled:              false
  Organize imports disabled:    false
  VCS disabled:                 false

Workspace:
  Open Documents:               0

What happened?
We use prettier formatter because it supports plugins that biome doesn't.
We perform formatting on save.
We also run quickFix by biome on save as well.
VSCOde settings:
 "editor.defaultFormatter": "esbenp.prettier-vscode",
 "editor.formatOnSave": true,
 "editor.codeActionsOnSave": {
    "quickfix.biome": "explicit",
    "source.organizeImports.biome": "explicit",

But recently we had to turn both of the biome features off because when any biome quickfix seems to randomly remove LARGE CHUNKS OF THE FILE. Sometimes the corruption is just a line or two. Sometimes it removes the last 70% of the file.
Expected result
What I would expect is for biome to not garble up the file!
Code of Conduct

[X] I agree to follow Biome's Code of Conduct

Are you able to provide a reproduction?

VSCode unfortunately has a bad history with Prettier formatting and code actions after save (https://github.com/prettier/prettier-vscode/issues/716) It looks a little bit as the prettier format is too slow and the codeActionsOnSave are triggered before prettier is finished. I don't know if there is a "codeActionsAfterFormat" or something (e.g. Zed editor is doing it like that)

@zauni I see. So it's an interaction where prettier writes later, and biome does a write before that and that's whey when I added biome I started seeing issues?
Well, for me personally, I'll keep prettier on save, and will turn off biome on save. The reason being formatting pretty much always a safe operation. Where as some of the biome rules I have enabled make development annoying, i.e. if I copy an import from another file and then start writing code to use said import, if I pause (which triggers auto save), the import I just copied gets removed.
Thanks for the explanation!
P.S. When biome formatting has more features, will gladly switch to it from prettier.

As a workaround, is possible to introduce a delay in biome action on autosave?

Going to close then, it seems it's not a Biome issue per se


When "cache" option is on, some assets are not emitted (related to extract text plugin?)

To keep track on https://github.com/mzgoddard/hard-source-webpack-plugin/issues/44 which is not resolved.

No more webpack cache in v1 as webpack is (sort of) optional.


refactor: Replace tracing with make_constraints_unique

Apologies for another large PR.
I've significantly reorganised and cleaned up the way the builder works:

remove "graph tracing" which was hard to reason about. Instead, duplicate states before modifying them, then try_merge_new_nodes to reduce the number of states where possible.
fix: previously the tracer could introduce cycles. Now checking before merging nodes that no cycles are created
fix: handle matches in non-leaf positions of the constraint tree. It was possible before to create mutually exclusive trees that had constraint IDs in non-leaf nodes, but such cases were not handled properly. Now this is supported.
Allow multiple transitions with same constraint during building, removing complicated recursive logic when adding new edges. The uniqueness of the transitions is restored one state at the time in the builder's finish step.
Cleaner separation between automaton traversal and mutating API. The state automaton can no longer be modified outside of the builder.

Thank you so much, very useful review!


Update "readonly account" to "read-only account"

Description the task
Update "readonly account" to "read-only account"

Version test: https://file.cdmteck.com/s/4oimk9zDJ8R5Aai/download?path=%2F&files=./955-merge-build-0.7.6-1671703068109.zip
Validation passed


Allow optimized queries for indexing

Implemented in https://github.com/Sparrho/bungiesearch/pull/66 .


[AutoRelease] t2-monitor-2022-06-23-02388(Do not merge)

https://github.com/Azure/sdk-release-request/issues/2904
Live test success
https://dev.azure.com/azure-sdk/internal/_build?definitionId=1014
BuildTargetingString
azure-mgmt-monitor
Skip.CreateApiReview
true

issue link:https://github.com/Azure/sdk-release-request/issues/2904


macos 14.5 performance issue

Hi! I was testing the polars plugin for nushell, and found that it became super slow after upgrading to macOS 14.5. I checked the same benchmarks with dply and found that it also runs way too slow!
Also, some other person reproduced slowliness on macos 14.5, and, I tested the polars nushell plugin under macos 14.2 and it runs quickly there.
Dear @vincev, I'm wondering if you have a macOS to test nearby and if you have any thoughts on the problem below.

I downloaded and unpacked this dataset.
I executed this command in the terminal:
time dply -c 'csv("./Data7602DescendingYearOrder.csv") | group_by(year) | summarize(count = n(), sum = sum(geo_count)) | show()'
shape: (21, 3)
┌──────┬────────┬──────────┐
│ year ┆ count  ┆ sum      │
│ ---  ┆ ---    ┆ ---      │
│ i64  ┆ u32    ┆ i64      │
╞══════╪════════╪══════════╡
│ 2000 ┆ 240730 ┆ 9107793  │
...
│ 2020 ┆ 273265 ┆ 13517976 │
└──────┴────────┴──────────┘
dplyr -c   0.50s user 14.57s system 297% cpu 5.073 total

So we can see that it took 5 seconds, while it should run in 0.5s (based on my intuitions).
Plus, what I found is that timings are really unstable - sometimes much lower, sometimes much higher.
Though without sum = sum(geo_count) timings are like normal:
> time dply -c 'csv("./Data7602DescendingYearOrder.csv") | group_by(year) | summarize(count = n()) | show()'

│ 2020 ┆ 273265 │
└──────┴────────┘
dply -c   0.43s user 0.08s system 467% cpu 0.110 total

P.S. I also ran the python script, and it goes with no problem whatsoever - very fast as usual:
import polars as pl

df = pl.read_csv("Data7602DescendingYearOrder.csv")
res = df.groupby("year").agg(pl.col("geo_count").sum())
print(res)

Before asking here, I checked the issues in the Polars repository, and it seems like nobody is aware of such problems even after 5 days since the release of the update.

Hi @maxim-uvarov I have released a new version the build on my box using your dataset takes 150ms to run your query:
time dply -c 'csv("./Data7602DescendingYearOrder.csv") | group_by(year) | summarize(count = n(), sum = sum(geo_count)) | show()'
shape: (21, 3)
┌──────┬────────┬──────────┐
│ year ┆ count  ┆ sum      │
│ ---  ┆ ---    ┆ ---      │
│ i64  ┆ u32    ┆ i64      │
╞══════╪════════╪══════════╡
│ 2000 ┆ 240730 ┆ 9107793  │
│ 2001 ┆ 239301 ┆ 9034416  │
│ 2002 ┆ 240243 ┆ 9128109  │
│ 2003 ┆ 243470 ┆ 9458073  │
│ 2004 ┆ 252073 ┆ 10273596 │
│ …    ┆ …      ┆ …        │
│ 2016 ┆ 266958 ┆ 12551199 │
│ 2017 ┆ 268989 ┆ 12894723 │
│ 2018 ┆ 270072 ┆ 13031973 │
│ 2019 ┆ 272229 ┆ 13297923 │
│ 2020 ┆ 273265 ┆ 13517976 │
└──────┴────────┴──────────┘
dply -c   0.62s user 0.09s system 471% cpu 0.151 total

Give it a try let me know how it goes.

@vincev Thank you for such a swift release!
I installed the new version, but I haven't noticed any changes.
I'm wondering what your performance was like before the new release. Did you experience the same slowness?
time dply -c 'csv("./Data7602DescendingYearOrder.csv") | group_by(year) | summarize(count = n(), sum = sum(geo_count)) | show()'
shape: (21, 3)
┌──────┬────────┬──────────┐
│ year ┆ count  ┆ sum      │
...
│ 2020 ┆ 273265 ┆ 13517976 │
└──────┴────────┴──────────┘
> dply -c   0.48s user 25.10s system 365% cpu 7.001 total

> dply -V
dply 0.3.1

@maxim-uvarov I have tried version 0.3.0 that used polars 0.38.3 and the performance is about the same:
$ sw_vers
ProductName:            macOS
ProductVersion:         14.5
BuildVersion:           23F79
$ dply --version
dply 0.3.0
$ time  dply -c 'csv("./Data7602DescendingYearOrder.csv") | group_by(year) | summarize(count = n(), sum = sum(geo_count)) | show()'
shape: (21, 3)
┌──────┬────────┬──────────┐
│ year ┆ count  ┆ sum      │
│ ---  ┆ ---    ┆ ---      │
│ i64  ┆ u32    ┆ i64      │
╞══════╪════════╪══════════╡
│ 2000 ┆ 240730 ┆ 9107793  │
│ 2001 ┆ 239301 ┆ 9034416  │
│ 2002 ┆ 240243 ┆ 9128109  │
│ 2003 ┆ 243470 ┆ 9458073  │
│ 2004 ┆ 252073 ┆ 10273596 │
│ …    ┆ …      ┆ …        │
│ 2016 ┆ 266958 ┆ 12551199 │
│ 2017 ┆ 268989 ┆ 12894723 │
│ 2018 ┆ 270072 ┆ 13031973 │
│ 2019 ┆ 272229 ┆ 13297923 │
│ 2020 ┆ 273265 ┆ 13517976 │
└──────┴────────┴──────────┘
dply -c   0.65s user 0.09s system 511% cpu 0.144 total

so it looks the drop in performance you have observed is not related to Polars.
One thing I have noticed from your timings is that the system time is very high could it be that you are low on memory?
Maybe you can try using /usr/bin/time -l that gives more info, this is what I get:
/usr/bin/time -l dply -c 'csv("./Data7602DescendingYearOrder.csv") | group_by(year) | summarize(count = n(), sum = sum(geo_count)) | show()'
shape: (21, 3)
┌──────┬────────┬──────────┐
│ year ┆ count  ┆ sum      │
│ ---  ┆ ---    ┆ ---      │
│ i64  ┆ u32    ┆ i64      │
╞══════╪════════╪══════════╡
│ 2000 ┆ 240730 ┆ 9107793  │
│ 2001 ┆ 239301 ┆ 9034416  │
│ 2002 ┆ 240243 ┆ 9128109  │
│ 2003 ┆ 243470 ┆ 9458073  │
│ 2004 ┆ 252073 ┆ 10273596 │
│ …    ┆ …      ┆ …        │
│ 2016 ┆ 266958 ┆ 12551199 │
│ 2017 ┆ 268989 ┆ 12894723 │
│ 2018 ┆ 270072 ┆ 13031973 │
│ 2019 ┆ 272229 ┆ 13297923 │
│ 2020 ┆ 273265 ┆ 13517976 │
└──────┴────────┴──────────┘
        0.14 real         0.62 user         0.08 sys
           473284608  maximum resident set size
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
               39063  page reclaims
                  29  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
                   0  messages sent
                   0  messages received
                   0  signals received
                   0  voluntary context switches
                 750  involuntary context switches
<PHONE_NUMBER>  instructions retired
<PHONE_NUMBER>  cycles elapsed
           384780992  peak memory footprint

@vincev, yep, seems like something is messing with dply on my system.
I have 18888  involuntary context switches versus yours 750  involuntary context switches
I'll try to debug it today. I will keep you informed about my findings.
@vincev Thank you so much for your attention!
/usr/bin/time -l dply -c 'csv("./Data7602DescendingYearOrder.csv") | group_by(year) | summarize(count = n(), sum = sum(geo_count)) | show()'
shape: (21, 3)
┌──────┬────────┬──────────┐
│ year ┆ count  ┆ sum      │
│ ---  ┆ ---    ┆ ---      │
│ i64  ┆ u32    ┆ i64      │
╞══════╪════════╪══════════╡
│ 2000 ┆ 240730 ┆ 9107793  │
│ 2001 ┆ 239301 ┆ 9034416  │
│ 2002 ┆ 240243 ┆ 9128109  │
│ 2003 ┆ 243470 ┆ 9458073  │
│ 2004 ┆ 252073 ┆ 10273596 │
│ …    ┆ …      ┆ …        │
│ 2016 ┆ 266958 ┆ 12551199 │
│ 2017 ┆ 268989 ┆ 12894723 │
│ 2018 ┆ 270072 ┆ 13031973 │
│ 2019 ┆ 272229 ┆ 13297923 │
│ 2020 ┆ 273265 ┆ 13517976 │
└──────┴────────┴──────────┘
       27.91 real         0.43 user       168.87 sys
           523173888  maximum resident set size
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
               39232  page reclaims
                 515  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
                   0  messages sent
                   0  messages received
                   0  signals received
                   0  voluntary context switches
               18888  involuntary context switches
<PHONE_NUMBER>31  instructions retired
<PHONE_NUMBER>69  cycles elapsed
           370100992  peak memory footprint

I installed a VM with macOS 14.5 on a Mac with macOS 14.5. Inside the virtual machine, dply runs very fast. So I can conclude that the problem is not with 14.5, but with my host operating system.

This is very strange, one thing you could try to see if it is disk related is to convert the data to parquet and see if there is any improvement:
dply -c 'csv("./Data7602DescendingYearOrder.csv") | parquet("data.parquet")'

or you could run a profiler like instruments to see what is taking all the time but that is a lot of work.

@vincev 👏👏👏
indeed!
converting to parquet helped! How did you know that??
> 1..100 | each {timeit {dply -c 'parquet("./data.parquet") | group_by(year) | summarize(count = n(), sum = sum(geo_count)) | show()' | complete | null}} | {max: ($in | math max), min: ($in | math min), median: ($in | math median)}
╭────────┬──────────────────╮
│ max    │ 73ms 401µs 917ns │
│ min    │ 61ms 454µs 209ns │
│ median │ 65ms 562µs 166ns │
╰────────┴──────────────────╯

And to you happen to have any ideas of the ways of fixing that?

These fluctuations in the csv file are a bit weird. Here's two tests with parquet and two with csv on my MBP M1Max 32gb ram 1TB ssd.
Parquet
❯ 1..100 | each {timeit {polars open data.parquet | polars group-by year | polars agg (polars col geo_count | polars sum) | polars collect| null}} |   | {max: ($in | math max), min: ($in | math min), median: ($in | math median)}
╭────────┬──────────────────╮
│ max    │ 43ms 129µs 709ns │
│ min    │ 20ms 727µs 166ns │
│ median │ 21ms 661µs 562ns │
╰────────┴──────────────────╯
❯ 1..100 | each {timeit {polars open data.parquet | polars group-by year | polars agg (polars col geo_count | polars sum) | polars collect| null}} |   | {max: ($in | math max), min: ($in | math min), median: ($in | math median)}
╭────────┬──────────────────╮
│ max    │ 37ms 550µs 542ns │
│ min    │ 20ms 334µs 292ns │
│ median │ 21ms 293µs 896ns │
╰────────┴──────────────────╯

CSV
❯ 1..10 | each {timeit {dply -c 'csv("./Data7602DescendingYearOrder.csv") | group_by(year) | summarize(count = n(), sum = sum(geo_count)) | show()' | complete | null}} | {max: ($in | math max), min: ($in | math min), median: ($in | math median)}
╭────────┬────────────────────────╮
│ max    │ 21sec 75ms 635µs 958ns │
│ min    │ 538ms 485µs 291ns      │
│ median │ 5sec 547ms 601µs 520ns │
╰────────┴────────────────────────╯
❯ 1..10 | each {timeit {dply -c 'csv("./Data7602DescendingYearOrder.csv") | group_by(year) | summarize(count = n(), sum = sum(geo_count)) | show()' | complete | null}} | {max: ($in | math max), min: ($in | math min), median: ($in | math median)}
╭────────┬──────────────────────────────╮
│ max    │ 1min 16sec 684ms 528µs 250ns │
│ min    │ 553ms 545µs 500ns            │
│ median │ 1sec 784ms 562µs 375ns       │
╰────────┴──────────────────────────────╯

I have a 4tb hard drive, and lot's of small text files on it. Maybe some indexing process, that was initiated after upgrade and haven't finished yet is misbehaving.
@fdncred do you have any deviations from average users in terms of hard drive size or the number of files and text files?

not really. i don't keep a lot of small text files around on purpose, other than rust code in repos.

@fdncred the polars plugin looks very nice.

Thanks. We're working on it but we're seeing some inconsistencies, that's when we decided to try dply since it's somewhat related. We wanted to see if we saw the same similarities, specifically with csv files. I'm still not sure what's going on specifically with macos.
These timings are with an old desktop system in Windows and seem somewhat reasonable with parquet and csv.
dply with csv
❯ 1..100 | each {timeit {dply -c 'csv("./Data7602DescendingYearOrder.csv") | group_by(year) | summarize(count = n(), sum = sum(geo_count)) | show()' | complete | null}} | {max: ($in | math max), min: ($in | math min), median: ($in | math median)}
╭────────┬───────────────────╮
│ max    │ 388ms 375µs 700ns │
│ min    │ 264ms 181µs 400ns │
│ median │ 286ms 849µs 200ns │
╰────────┴───────────────────╯

dply with parquet
❯ 1..100 | each {timeit {dply -c 'parquet("./data.parquet") | group_by(year) | summarize(count = n(), sum = sum(geo_count)) | show()' | complete | null}} | {max: ($in | math max), min: ($in | math min), median: ($in | math median)}
╭────────┬───────────────────╮
│ max    │ 315ms 194µs 500ns │
│ min    │ 175ms 592µs 100ns │
│ median │ 193ms 747µs 300ns │
╰────────┴───────────────────╯

nushell with csv
❯ 1..100 | each {timeit {polars open Data7602DescendingYearOrder.csv | polars group-by year | polars agg (polars col geo_count | polars sum) | polars collect| null}} |   | {max: ($in | math max), min: ($in | math min), median: ($in | math median)}
╭────────┬───────────────────╮
│ max    │ 221ms 192µs 800ns │
│ min    │ 147ms 270µs 200ns │
│ median │ 163ms 65µs 750ns  │
╰────────┴───────────────────╯

nushell with parquet (the max and min are pretty far apart here for some reason)
❯ 1..100 | each {timeit {polars open data.parquet | polars group-by year | polars agg (polars col geo_count | polars sum) | polars collect| null}} |   | {max: ($in | math max), min: ($in | math min), median: ($in | math median)}
╭────────┬───────────────────╮
│ max    │ 491ms 423µs 200ns │
│ min    │ 75ms 194µs        │
│ median │ 91ms 373µs 200ns  │
╰────────┴───────────────────╯

@vincev Also, I've noticed this for sure that my battery is draining while the mac is idle. So perhaps there is lots IO happening

@maxim-uvarov another test you can do is try with duckdb that uses a completely different code base and language and see if you notice the same variance:
1..100 | each {timeit {duckdb -c "select year, count(*) n, sum(geo_count) s from './Data7602DescendingYearOrder.csv' group by year order by 1" | complete | null}} | {max: ($in | math max), min: ($in | math min), median: ($in | math median)}
╭────────┬───────────────────╮
│ max    │ 266ms 566µs 625ns │
│ min    │ 132ms 364µs 958ns │
│ median │ 138ms 10µs 667ns  │
╰────────┴───────────────────╯

Maybe you can use Activity Monitor to see if it shows processes doing lots of IO following the upgrade.

@vincev Many thanks for keeping us in mind!
No problem with duckdb, as no problem with Python Polars. Everything runs very smoothly.
> 1..100 | each {timeit {duckdb -c "select year, count(*) n, sum(geo_count) s from './Data7602DescendingYearOrder.csv' group by year order by 1" | complete | null}} | {max: ($in | math max), min: ($in | math min), median: ($in | math median)}
∙
╭────────┬───────────────────╮
│ max    │ 119ms 656µs 708ns │
│ min    │ 108ms 837µs 41ns  │
│ median │ 113ms 817µs 229ns │
╰────────┴───────────────────╯

I'm going to learn how to use the Instruments Mac profiler for my case. But before that, I want to complete some other computational tasks that are occupying my mind.

polars-cli is also slow with csv.
P.S. I renamed polars to polars-cli as this name is already used in nushell for the plugin
> timeit {polars-cli -c "select year, count(*) n, sum(geo_count) s from read_csv('Data7602DescendingYearOrder.csv') group by year;"}
┌──────┬────────┬──────────┐
│ year ┆ n      ┆ s        │
│ ---  ┆ ---    ┆ ---      │
│ i64  ┆ u32    ┆ i64      │
╞══════╪════════╪══════════╡
│ 2007 ┆ 259065 ┆ 11348772 │
│ 2005 ┆ 255917 ┆ 10724928 │
│ 2010 ┆ 259891 ┆ 11513931 │
│ 2011 ┆ 260366 ┆ 11523552 │
│ 2015 ┆ 265129 ┆ 12305010 │
│ …    ┆ …      ┆ …        │
│ 2014 ┆ 262630 ┆ 12004401 │
│ 2012 ┆ 260584 ┆ 11510928 │
│ 2003 ┆ 243470 ┆ 9458073  │
│ 2013 ┆ 260208 ┆ 11588178 │
│ 2001 ┆ 239301 ┆ 9034416  │
└──────┴────────┴──────────┘
13sec 69ms 280µs 83ns

@maxim-uvarov that's interesting, the Polars team just released version 0.40, I have update dply dependencies on main, I am not sure if it solves your problem maybe you can give it a try if it doesn't probably the best thing is to open an issue with the Polars team that shows the difference between duckdb and polars-cli.

@vincev I updated to dply 0.3.2, yet it didn't help :( Thanks for the suggestion. I will create an issue in the polars-cli repository. But before that, I want to try the instruments to gather more details. It seems like a handy skill and an interesting activity.

I ran this
sudo flamegraph ...[-o my_flamegraph2.svg] -- dply -c 'csv("./Data7602DescendingYearOrder.csv") | group_by(year) | summarize(count = n(), sum = sum(geo_count)) | show()'

and got this:

I run it on my box, I rebuilt dply using:
[profile.release]
debug = true

I got the following:

It looks that in your case is spending much more time doing memory stuff but I am not sure what is causing that.

I also built it with
[profile.release]
debug = true

here is the picture.

It looks like is spending lots of time doing memory stuff, I don't really know what is causing that, maybe ask on the Polars discussions someone may have had experienced the same problem, it may be some MacOS configuration setting that has changed after the upgrade on your box.

Yesterday I collected 'sys' template information and loaded it into instruments. The file is 120 MB and full of data. Though it's really difficult to understand where to look for insights, I don't give up: I chat with GPT and compare profilings from loading Parquet.
Also, I contacted Apple support but was rather disappointed by their answers. I sent the issue via Feedback Assistant, yet I don't hope that they will investigate them properly.
Today im going to ask at polars git or in their chat

I just got my hands on a used min-spec Mac Mini M1 (8 GB RAM, 256 GB SSD). It's still running Sonoma 14.4 and I haven't upgraded it.
I will hold off on upgrading in case there is any more repro to be done. But so far it seems like 14.4 probably doesn't have this problem:
dply with CSV:
~/polars-slow-repro> 1..10 | each {timeit {dply -c 'csv("./Data7602DescendingYearOrder.csv") | group_by(year) | summarize(count = n(), sum = sum(geo_count)) | show()' | complete | null}} | {max: ($in | math max), min: ($in | math min), median: ($in | math median)}
╭────────┬───────────────────╮
│ max    │ 141ms 195µs 208ns │
│ min    │ 110ms 584µs 334ns │
│ median │ 113ms 445µs 791ns │
╰────────┴───────────────────╯

dply with parquet:
~/polars-slow-repro> 1..100 | each {timeit {dply -c 'parquet("./data.parquet") | group_by(year) | summarize(count = n(), sum = sum(geo_count)) | show()' | complete | null}} | {max: ($in | math max), min: ($in | math min), median: ($in | math median)}
╭────────┬───────────────────╮
│ max    │ 113ms 848µs 542ns │
│ min    │ 67ms 554µs 584ns  │
│ median │ 71ms 265µs 938ns  │
╰────────┴───────────────────╯

Nushell with CSV:
~/polars-slow-repro> 1..100 | each {timeit {polars open Data7602DescendingYearOrder.csv | polars group-by year | polars agg (polars col geo_count | polars sum) | polars collect| null}} |   | {max: ($in | math max), min: ($in | math min), median: ($in | math median)}
╭────────┬──────────────────╮
│ max    │ 80ms 650µs 500ns │
│ min    │ 58ms 476µs 458ns │
│ median │ 61ms 510µs 500ns │
╰────────┴──────────────────╯

Nushell with Parquet:
~/polars-slow-repro> 1..100 | each {timeit {polars open data.parquet | polars group-by year | polars agg (polars col geo_count | polars sum) | polars collect| null}} |   | {max: ($in | math max), min: ($in | math min), median: ($in | math median)}
╭────────┬──────────────────╮
│ max    │ 48ms 886µs 584ns │
│ min    │ 24ms 792µs 459ns │
│ median │ 26ms 442µs 521ns │
╰────────┴──────────────────╯

@vincev Question for you: are you on Apple Silicon or Intel?
My pet hypothesis at the moment is that this has something to do with GoFetch vulnerability mitigations, but I'm not sure. The 'involuntary context switches' kind of lines up with that idea too.

We discussed with Devyn that before upgrading, it might be useful to use Xcode instruments to obtain profiles of a system before it gets upgraded, which might cause the problem.
Here is a quick instruction to get started:
To get started, you need to install Xcode (it can be done via the App Store, but the download was very slow for me):
https://download.developer.apple.com/Developer_Tools/Xcode_15.4/Xcode_15.4.xip
In Xcode, choose:

After installation, if it refuses to work, you can try these commands (that found in my history, while was troubleshooting):
/usr/bin/xcodebuild -version
xcrun xctrace

Then install Cargo Instruments:
https://github.com/cmyr/cargo-instruments
In your clone directory of dply, add this to Cargo.toml:
[profile.release]
debug = true

After that, from that folder, you can execute commands like:
cargo instruments -t 'Activity Monitor' -- dply -c 'csv("/Users/user/git/nushell.github.io/book/test/Data7602DescendingYearOrder.csv") | group_by(year) | summarize(count = n(), sum = sum(geo_count)) | show()'

> cargo instruments -l
Xcode Instruments templates:

built-in            abbrev
--------------------------
Activity Monitor
Allocations         (alloc)
Animation Hitches
App Launch
Audio System Trace
CPU Counters
CPU Profiler
Core ML
Data Persistence
File Activity       (io)
Game Memory
Game Performance
Leaks
Logging
Metal System Trace
Network
RealityKit Trace
SceneKit
Swift Concurrency
SwiftUI
System Trace        (sys)
Tailspin
Time Profiler       (time)
Zombies

I found those templates interesting:
Activity Monitor
io
sys

I can share my profiles. The largest one is over 100 MB. Apple says there might be some sensitive information to share it with the public, but I have nothing to hide. Therefore, I can easily share them with the participants of this conversation if they are willing.

I converted all the data to JSON. Reading from a JSON file five times bigger takes 50 times less time.
> open Data7602DescendingYearOrder.csv | each {to json -r} | str join (char nl) | save data.json -f

> 1..50 | each {timeit {dply -c 'json("./data.json") | group_by(year) | summarize(count = n(), sum = sum(geo_count)) | show()' | complete | null}} | {median: ($in | math median), min: ($in | math min), max: ($in | math max), stddev: ($in | into int | into float | math stddev | into int | $'($in).ns' | into duration --unit sec)}
╭────────┬───────────────────╮
│ median │ 662ms 454µs 521ns │
│ min    │ 540ms 567µs 916ns │
│ max    │ 841ms 515µs 291ns │
│ stddev │ 69ms 853µs 152ns  │
╰────────┴───────────────────╯

though! If I remove the calculation of sum everything flyes!
> 1..10 | each {timeit {dply -c 'csv("./data.csv") | group_by(year) | summarize(count = n()) | show()' | complete | null}} | {median: ($in | math median), min: ($in | math min), max: ($in | math max), stddev: ($in | into int | into float | math stddev | into int | $'($in).ns' | into duration --unit sec)}
╭────────┬──────────────────╮
│ median │ 91ms 958µs 833ns │
│ min    │ 89ms 849µs 917ns │
│ max    │ 97ms 538µs 292ns │
│ stddev │ 2ms 284µs 986ns  │
╰────────┴──────────────────╯

With sum
❯ 1..50 | each {timeit {dply -c 'csv("./data.csv") | group_by(year) | summarize(count = n(), sum = sum(geo_count)) | show()' | complete | null}} | {median: ($in | math median), min: ($in | math min), max: ($in | math max), stddev: ($in | into int | into float | math stddev | into int | $'($in).ns' | into duration --unit sec)}
╭────────┬────────────────────────╮
│ median │ 1sec 337ms 902µs 979ns │
│ min    │ 220ms 778µs 209ns      │
│ max    │ 17sec 860ms 1µs 584ns  │
│ stddev │ 2sec 662ms 457µs 129ns │
╰────────┴────────────────────────╯

Without sum
❯ 1..50 | each {timeit {dply -c 'csv("./data.csv") | group_by(year) | summarize(count = n()) | show()' | complete | null}} | {median: ($in | math median), min: ($in | math min), max: ($in | math max), stddev: ($in | into int | into float | math stddev | into int | $'($in).ns' | into duration --unit sec)}
╭────────┬───────────────────────╮
│ median │ 94ms 813µs 416ns      │
│ min    │ 89ms 422µs 42ns       │
│ max    │ 1sec 12ms 978µs 375ns │
│ stddev │ 174ms 418µs 404ns     │
╰────────┴───────────────────────╯

polars cli doesn't like sum either:
> 1..10 | each {timeit {polars-cli -c "select year, count(*) n from read_csv('Data7602DescendingYearOrder.csv') group by year;" | complete | null}} | {median: ($in | math median), min: ($in | math min), max: ($in | math max), stddev: ($in | into int | into float | math stddev | into int | $'($in).ns' | into duration --unit sec)}
╭────────┬──────────────────╮
│ median │ 45ms 243µs 354ns │
│ min    │ 43ms 239µs 875ns │
│ max    │ 49ms 414µs 167ns │
│ stddev │ 2ms 12µs 745ns   │
╰────────┴──────────────────╯

> 1..10 | each {timeit {polars-cli -c "select year, count(*) n, sum(geo_count) s from read_csv('Data7602DescendingYearOrder.csv') group by year;" | complete | null}} | {median: ($in | math median), min: ($in | math min), max: ($in | math max), stddev: ($in | into int | into float | math stddev | into int | $'($in).ns' | into duration --unit sec)}
╭────────┬─────────────────────────╮
│ median │ 8sec 258ms 594µs 312ns  │
│ min    │ 3sec 202ms 705µs 958ns  │
│ max    │ 20sec 641ms 701µs 583ns │
│ stddev │ 5sec 21ms 84µs 752ns    │

I inspected the geo_count column visually.
I converted the file to CSV once again in case it had artifacts.
open data.csv | save data1.csv

Still no luck.

for a sake of synchronization: these are screenshots from nuhslell discord with @devyn

@vincev Question for you: are you on Apple Silicon or Intel?
@devyn I am running on a 16GB Apple M1.

Thanks, I guess that potentially eliminates that. Hmm...

@vincev (or any other rust-knowing participants) great Ritchie from Polars advised to try some things to improve performance here
Namely:

I see they run on the default allocator. Setting jemalloc on macos might resolve it. Further if performance is important they should consider activating simd, performant and maybe compiling binaries on a nightly compiler with avx features activated.

If you ever have time for this issue again - would you please help me to incorporate proposed settings into dply to test?
p.s. I'm sorry for occupying your attention.
Our polars master is on vacation, so I can't reach him with this request.

Link for reference: https://github.com/pola-rs/polars/blob/019dfe87b28afa51a74fc87ebbccb354fd81b53a/.github/workflows/release-python.yml

Well, I think i can do that myself. At least im trying now

@maxim-uvarov okay let me know how it goes, in the meantime I can add jemalloc and activate some of the other features but you'll have to test it as I cannot reproduce it on my box.

Just pushed a branch with jemalloc in case it helps https://github.com/vincev/dply-rs/tree/perf

@vincev thank you so much! Helped a lot!
And yes, it solved the problem!!!
> 1..100 | each {timeit {dply -c 'csv("./data.csv") | summarize(count = n(), sum = sum(geo_count)) | show()' | complete | null}} | {median: ($in | math median), min: ($in | math min), max: ($in | math max), stddev: ($in | into int | into float | math stddev | into int | $'($in).ns' | into duration --unit sec)}
╭────────┬──────────────────╮
│ median │ 75ms 471µs 521ns │
│ min    │ 71ms 384µs 250ns │
│ max    │ 85ms 949µs 625ns │
│ stddev │ 2ms 635µs 56ns   │
╰────────┴──────────────────╯

@vincev @devyn @fdncred, especially @vincev Thank you so much for your time, help and participation! I'm super grateful!!!

@maxim-uvarov you are welcome I'll do a release with jemalloc then.


delivery challan

[x] status change response message for
[x] revert to open
[x] mark as delivered
[x] mark as returned
[x] Home icon not needed
[x] sales person field not needed

issue closed


Added Redis for Dummies from Redis site

What does this PR do?
Add resource
For resources
Description

Added Redis for Dummies book from Redis official site

Why is this valuable (or not)?

Valuable for beginners

How do we know it's really free?
It's available from Redis site leaving a mail address, Then there will be a download link
For book lists, is it a book? For course lists, is it a course? etc.

book

Checklist:

[x] Read our contributing guidelines
[x] Search for duplicates.
[x] Include author(s) and platform where appropriate.
[x] Put lists in alphabetical order, correct spacing.
[x] Add needed indications (PDF, access notes, under construction)

Followup

Check the output of Travis-CI for linter errors!

Recover

I opened by mistake before hacktoberfest. I will open another tomorrow.

I opened by mistake before hacktoberfest. I will open another tomorrow.

If is your preference, close it again.  But I do not see much sense.
Anyways, you can integrate the review suggestion meanwhile :wink:

Resubmitted at #5625


Other language supporting plan?

Hi there, fist thank you for your great work. It looks like this project only supports English now, is there any plan for supporting other languages such as Chinese or French?

Hi @ywwhxxtw. Definitely! The idea is supporting all the available models for HuggingFace, including those shared by the community (https://huggingface.co/models). I'm currently quite busy but I'll try to solve this and the other issues in the following days.

Wonderful! looking forward to your good news :)


Terraform replicated dataset as binary

binary copy enabled

https://github.com/lui91/airflow_ingestion/commit/267480853c279e0d5a59431189a7e9154ec01385


Find appropriate Data Augmentation Script and Apply it to the dataset (subset of dataset)

Find some scripts for dataAug here: https://github.com/AgaMiko/data-augmentation-review/blob/master/README.md#Computer-vision

The interest of performing data augmentation for this project is double:

Improving the performances of the model.
Creating model abstraction of the colors and the textile density.

One way of measuring the second point is to apply anomaly detection on another type of textile and see how it can perform.
It is important to choose the right augmentation operations to avoid creating anomalies by augmenting the data.
Some ideas to implement

[ ] Blurring: to simulate different densities of textile.
[ ] Colors
[ ] others...

Using this library : albumentations

Default
Code:
        self.data_transforms = transforms.Compose([
                        transforms.Resize((args.load_size, args.load_size), transforms.InterpolationMode.LANCZOS),
                        transforms.ToTensor(),
                        transforms.CenterCrop(args.input_size),
                        transforms.Normalize(mean=mean_train,
                                            std=std_train)])

Image:

Default mode but without normalization and centercrop.
Code:
        self.data_transforms = transforms.Compose([
                        transforms.Resize((args.load_size, args.load_size), transforms.InterpolationMode.LANCZOS),
                        transforms.ToTensor()    ])

Image:

If you want to add some augmentations.

Import it by adding the class in the import line
from kornia.augmentation import RandomBoxBlur, ColorJitter, Normalize [OTHER.....]

Add in the sequential composition of augmentations:

class DataAugmentation(nn.Module):
    """Module to perform data augmentation using Kornia on torch tensors."""

    def __init__(self, apply_color_jitter: bool = False) -> None:
        super().__init__()
        self._apply_color_jitter = apply_color_jitter

        self.transforms = nn.Sequential(
            RandomBoxBlur(kernel_size=(3, 3), border_type='reflect', p=0.75),
            Normalize(mean=mean_train, std=std_train)
            [ADD HERE]
        )

Run to observe output files:
!python train_resnet.py --phase augment --category carpet --num_epochs 3

open the 2 generate files.

"img_augment.png"
"img_before_augment.png"

I tried to add the RandomRotation, but the problem is that we lose information on the image and a part of the image is set to black pixels. Which could be detected as an anomaly. By using the RandomAffine(degrees=45.0, scale=(1,2), padding_mode=2, p=.75), we can set the scale value and the padding_mode which allows us to keep the texture of the textile everywhere on the image.
Se the difference of RandomRotation and RandomAffine on the newt two images.

Now let's compare when playing with colors, contrast, brightness, ...
Using the kornia.augmentation.ColorJiggle class.
To better understand the difference, I did not normalize the data before saving the image.
The two next images are

with brightness=0.1, contrast=0.1, saturation=0.1, hue=0.1

with brightness=0.2, contrast=0.2, saturation=0.2, hue=0.2

And if we apply the normalization for the 0.2 case, we obtain the following image

https://www.mydrive.ch/shares/38536/3830184030e49fe74747669442f0f282/download/420937607-1629951964/leather.tar.xz

Adding some blur to obtain something similar to leather.
        self.transforms = nn.Sequential(
            RandomBoxBlur(kernel_size=(2,2), border_type='reflect', p=0.2),
            RandomBoxBlur(kernel_size=(3,3), border_type='reflect', p=0.2),
            RandomBoxBlur(kernel_size=(5,5), border_type='reflect', p=0.2),
            RandomBoxBlur(kernel_size=(7,7), border_type='reflect', p=0.2),
            RandomBoxBlur(kernel_size=(5,5), border_type='reflect', p=0.2),
            RandomBoxBlur(kernel_size=(15,15), border_type='reflect', p=0.1),
            RandomAffine(degrees=45.0, scale=(1,2), padding_mode=2, p=.75),
            ColorJiggle(0.1, 0.1, 0.1, 0.1, p=1.),
            #Normalize(mean=mean_train, std=std_train)
        )


python39Packages.qcs-api-client: 0.20.10 -> 0.20.12

Description of changes
https://github.com/rigetti/qcs-api-client-python/releases/tag/v0.20.12

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 170457 run on x86_64-linux 1

2 packages failed to build:

python310Packages.cirq
python39Packages.cirq

18 packages built:

python310Packages.cirq-aqt
python310Packages.cirq-core
python310Packages.cirq-google
python310Packages.cirq-ionq
python310Packages.cirq-pasqal
python310Packages.cirq-rigetti
python310Packages.cirq-web
python310Packages.pyquil
python310Packages.qcs-api-client
python39Packages.cirq-aqt
python39Packages.cirq-core
python39Packages.cirq-google
python39Packages.cirq-ionq
python39Packages.cirq-pasqal
python39Packages.cirq-rigetti
python39Packages.cirq-web
python39Packages.pyquil
python39Packages.qcs-api-client

Result of nixpkgs-review pr 170457 run on x86_64-linux 1

8 packages built:

python310Packages.cirq
python310Packages.cirq-rigetti
python310Packages.pyquil
python310Packages.qcs-api-client
python39Packages.cirq
python39Packages.cirq-rigetti
python39Packages.pyquil
python39Packages.qcs-api-client


style: no text wrap for method badges, fix #871

This PR improves the styling of the method badge in the sidebar.

Text doesn’t wrap anymore.
The size of the badge (width, height, line-height) is configured in em now, so it’s relative to the font size.
The margin is removed, it’s using gap now.
The badge is set to overflow: hidden.

One thing changed:
The link and the badge were set to flex-start, so they were attached to the top. For long operation names the badge was still on top, I think that was expected.
Unfortunately, this required also to fix the position of the badge for one-line operation names with a top: 3.5px; to make it look centered. Which feels very hacky to me, and doesn’t work well with a dynamic (em) size.
That’s why I actually centered it now:

See #871

I'll come up with a better solution to keep it top aligned soon. I need to spend a few hours on the sidebar in the next week or two :)


Interested in Hiring

Hey, saw your work on GitHub, really impressive stuff. My company is interested in hiring a dev and would love to chat with you.
Thanks,
Chan

What is your company ?

Hi John,
Please refer to the below. For more information, reach out to me via email. :)
https://twitter.com/lava_xyz
https://www.linkedin.com/company/lavaxyz/?trk=similar-pages


issue 13

feat(formatter): logger output option exchange from entry to formatter.
feat(format): output format adjust

#13


add an experimental 'flutter build web' call to the backend

Add an experimental 'flutter build web' call to the backend:

add an experimental api/v2/_flutterBuild call to the backend
this is given a source file and invokes flutter build web; this is currently slower than we'd want but I believe there's some room for improvement in the execution time of flutter build web
the flutter web frontend has been updated to use this compile call for both dart and flutter snippets

Many of the changes here are multiple copies of the proto related files. With this change, the new front-end can now run flutter snippets.

[x] I’ve reviewed the contributor guide and applied the relevant portions to this PR.

Contribution guidelines:

See our contributor guide for general expectations for PRs.
Larger or significant changes should be discussed in an issue before creating a PR.
Contributions to our repos should follow the Dart style guide and use dart format.
Most changes should add an entry to the changelog and may need to rev the pubspec package version.
Changes to packages require corresponding tests.

Note that many Dart repos have a weekly cadence for reviewing PRs - please allow for some latency before initial review feedback.


page.click(selector[, options]),clickCount  doesn't work

Steps to reproduce
Tell us about your environment:

Puppeteer version:9.1.1
Platform / OS version:mac 10.15.7
URLs (if applicable):
Node.js version:10.16.3

What steps will reproduce the problem?
Please include code that reproduces the issue.
1."compos.lsspacing_add",for page elements
2.const lsspacing_btn = await this.page.waitForSelector(compos.lsspacing_add); await lsspacing_btn.click({clickCount: 4});
3.set clickCount  doesn't work
What is the expected result?
Able to click the number of times set
What happens instead?
Actually setting the number of clicks does not work

@xuxk1  Did you get the solution of it? because i have the same issue.

@xuxk1 Did you get the solution of it? because i have the same issue.

Use "puppeteer": "^9.1.1" to resolve the problem

@xuxk1 Did you get the solution of it? because i have the same issue.

await this.page.click(Po.tool_zoomInput, { clickCount: 2 });
await this.page.click(Po.tool_zoomInput, { clickCount: 2 });
Like executing twice can be solved

what is Po.tool_zoomInput? You selector? so basically click count is not working. You are using the click statement twice


re-enable gts check and linting

The work here is to get npx gts fix passing.  Then the lint command should do a gts check, and the fix command should do a gts fix.  This will be a very large change, so it's ok to do it in small chunks.

@JustinBeckwith type "any" is in use at multiple places, and as per "/tslint.json" which extends "gts/tslint.json", follows "no-any" rule.
At some places it's hard to decide if type "any" should be removed,
like: src/subscriber.ts ln 129 col 67
Shall I set rule { "no-any": false } in "tslint.json"?

Heh, nah.  This is going to hurt.  What you do is set noImplicitAny to true in the tsconfig.  That's going to break a lot of stuff.  You start with one file, and add types to fix pretty much all of the ambiguous types.  Then, you turn noImplicitAny back off, and submit a PR for just that file.
Here are a few examples of repos that have been transitioned in this way:

https://github.com/googleapis/nodejs-translate
https://github.com/googleapis/nodejs-storage
https://github.com/googleapis/nodejs-bigquery

It's admittedly a tad tedious.  Generally the only places you should use any are in tests.

@JustinBeckwith Thanks!
But raising PR for individual file-change my break the stuff, for changes like:
 export class IAM {    Promise?: PromiseConstructor;
to
 export class IAM {    promise?: PromiseConstructor;
that is changing the property name.
Any concerns with raising PR for all changes together?

So we absolutely do not want to introduce breaking changes here.  In this example, we would us a tslint ignore.  Place this on the pine above the property.
// tslint:disable-next-line variable-name
If we don't limit this to one file at a time, the PR will quickly get out of control.  I'd the changes to other files are small and required to get basic compilation going that's fine.


fix social media icons when opening the page in dark mode

 this makes it transparent not add a background to it by default.

got it let me work on this need some time

this makes it transparent not add a background to it by default.

got it let me work on this need some time

Sure take your time


Add Snap CI build status support

I would like to suggest that Snap CI can also be integrated into this badges/shields project.

Thanks for the suggestion!

:+1:

:+1:

Thanks @pine613! Any way we could get support for private repos as well? This would effectively turn this line
var url = 'https://snap-ci.com/' + userRepo + '/branch/' + branch + '/build_image.svg';

into this:
var url = 'https://snap-ci.com/' + repoToken + '/build_image.svg';

I'm not 100% certain that I know what needs to change though...

@kentcdodds Thanks for indicating this! I assume repoToken is meant to be secret, which makes testing awkward. Copying the same Snap CI code in shields under a different URL (say, /snap-ci/private/TOKEN.svg) ought to work?
I made a bit of a mistake during the review by not requiring /snap-ci/ci/user/repo.svg, but I hope it won't hurt too much in the future.

Unfortunately snap-ci/private/TOKEN.svg doesn't currently work (were you saying that it would?)

It doesn't, but it is a plausible Shields URL if it were to be implemented.


base image clean up and discussion

Originally posted by @rootfs in https://github.com/sustainable-computing-io/kepler/pull/924#discussion_r1326183257

Agree to also have a CI for testing new base image proposal.
We might have a separate Dockerfile that use the dev base image.


Remove dependency on cats

Cats is a heavy dependency, which is used for minor issues in phobos. It should be removed to prevent conflicts in projects, which depend on phobos

Closed in #110