FIX doc generating for custom array type body param

Describe the PR
FIX doc generating for custom array type body param
Relation issue
https://github.com/swaggo/swag/issues/550
Additional context

Codecov Report

Merging #587 into master will increase coverage by 0.03%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #587      +/-   ##
==========================================
+ Coverage   86.24%   86.27%   +0.03%
==========================================
  Files           7        7
  Lines        1599     1603       +4
==========================================
+ Hits         1379     1383       +4
  Misses        136      136
  Partials       84       84

Impacted Files
Coverage Δ

operation.go
92.69% <100%> (+0.06%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eb4805c...da9c7f6. Read the comment docs.

#579 had fix this issue

@wuhuizuo thanks for your PR. Have fixed by #579. Please raise a new PR for any improvement or issues. 👍

@wuhuizuo thanks for your PR. Have fixed by #579. Please raise a new PR for any improvement or issues. 👍

#579 is better 👍


About train resent-18 on the dataset CIFAR-100, the accuracy is  just 55.

I have ran 'main_supcon.py'. Parameter: CIFAR-100  resnet-18， other parameters have no change. Then I ran 'main_linear' to test resnet-18's accuracy(Load the encoder weights previously trained in  'main_supcon.py' ), just train the last layer(fc layer). But the accuracy is about 55(Baseline is resent-18 train alone, the accuracy is 70).  I think it is strange, so want to consult you about it.

Hi, have you found the reason?


Error when customize build.yaml

Hi, when I customize build.yaml file to specify on which folder I would like reflectable work, I get some errors at build time:
targets:
  $default:
    sources:
      include: ["lib/**", "web/**", "test/**"]
      exclude:
        - web/nodes_modules/**
    builders:
      build_web_compilers|entrypoint:
        release_options:
          dart2js_args:
            - -DwcfHost=xxxxxxx
      sass_builder:
        release_options:
          outputStyle: compressed
      reflectable:
        generate_for:
          - lib/src/**.dart
          - web/main.dart
#          - test/**_test.dart
        options:
          formatted: true

Error reported:
[SEVERE] reflectable on lib/src/etablissement/etablissement_search.g.dart:

Asset [agde_mobilite_web|lib/src/etablissement/etablissement_search.g.dart] is not a Dart library. It may be a part file or a file without Dart source code.
[SEVERE] reflectable on lib/src/etablissement/etablissement.g.dart:

Asset [agde_mobilite_web|lib/src/etablissement/etablissement.g.dart] is not a Dart library. It may be a part file or a file without Dart source code.
[SEVERE] reflectable on lib/src/mixins/paginer.g.dart:

Asset [agde_mobilite_web|lib/src/mixins/paginer.g.dart] is not a Dart library. It may be a part file or a file without Dart source code.
[SEVERE] reflectable on lib/src/wcf/wcf_paging.g.dart:

I'm using json_serializable and it seems to try reflect files before the *.g.dart files generation. Is the a way to resolve this?

I'd try deleting .dart-tool/build and run again

Still get errors

The same errors?

Yes, exactly the same

Maybe you don't actually want to specify lib/src/**.dart in generate_for. The reflectable code generation approach is inherently working on a whole program, so you should specify exactly the libraries which are entry points, that is, roughly, libraries that contain a main function and that you're thinking about as "programs".
So you'd presumably include lib/main.dart in the context of a Flutter package, and nothing else.
All libraries which are directly or indirectly imported from your entry points will be taken into account because they'll be inspected during code generation. All results from this process will be expressed as a single generated file (here: lib/main.reflectable.dart, which would be imported by lib/main.dart), and that file contains elements which are used in the support of reflection on instances and types from all the reachable libraries.

Oh ok, I got it! I did not understand the concept of the entry point when I've read the doc.
So in my case including lib/src/** is useless. I removed that line and everything working fine now
Thanks a lot @eernstg !

Maybe you don't actually want to specify lib/src/**.dart in generate_for. The reflectable code generation approach is inherently working on a whole program, so you should specify exactly the libraries which are entry points, that is, roughly, libraries that contain a main function and that you're thinking about as "programs".
So you'd presumably include lib/main.dart in the context of a Flutter package, and nothing else. Maybe there's another entry point in bin and some in test, and then you can include them as well, but it should still only be entry points.
All libraries which are directly or indirectly imported from your entry points will be taken into account because they'll be inspected during the analysis and code generation steps of reflectable. All results from this process will be expressed as a single generated file (here: lib/main.reflectable.dart, which would be imported by lib/main.dart), and that file contains elements which are used in the support of reflection on instances and types from all the reachable libraries.
If you do this, you'd presumably stop seeing messages about libraries in lib/src/** (but if they are part files then you'll get access to the declarations in there because they will be loaded as such during the analysis phase).

@eernstg How do I handle this, in my test project? I have hundreds of test files, who are all having a main() function. Do I need to generate a reflectable for every of those hundred test files?
To have multiple main() functions in multiple test files, seems to be the go-to, since the flutter example projects are written like this: https://github.com/flutter/flutter/tree/master/examples/catalog/test

It should not be a problem to generate a '*.reflectable.dart' file for each of those hundreds of test files, and it might be possible for you to select the entry points using a wildcard or two:
// build.yaml
targets:
  $default:
    builders:
      reflectable:
        generate_for:
          - test/**_test.dart

This will work if all your entry points have a name ending in _test.dart and none of the other files in that directory has a name of that form. Maybe you could use that directly, or adjust some file names such that it will work?

Ok, i already did what you described. I just thought that this is a little bit hacky to generate this file for every entry point. But now I know that how I did it is the go-to way. So, thanks @eernstg!


Fix per-transaction receipts in admin

Fixes #761.
This pull request makes SendReceiptView send an email for transactions instead of orders, which is how OrderReceiptMailer works now.
The button to send the receipt shows up once per transaction on the order details page, like this:

There may be a better design for this, but it seems okay to me.

@harrislapiroff @melinath This is ready for review.

I haven't tested yet, but this all looks good. Just one thing: We should restrict display of this button & access to the view to payment transactions – i.e. this email is not intended for refunds.

@melinath Should be good to test, we're now only showing the button for payments.

I made a couple minor tweaks. I'm down with merging this PR; if my changes look good to you, go ahead!


Update SBT/scala frameworks

Several things to do in the project regarding SBT and Scala frameworks.

Right now we are using scalastyle to cover the style and productivity of the code. Seems like the project has not been updated for long and scalafix could be a better library to use in the project.
Update plugins: delete the ones that are not used and update the versions.
Sbt version: update the version of the SBT building tool to the last one.

I was thinking that the problem is with scalafmt, since the task is to replace scala-style with more modern scalafix, then it looks promising. Apologies.

The update is done for main-1.0.0. I suggest to close the issue.


MissingReference of type CelestialBodyGenerator when changin scene

Hello everyone, I've been struggling for few days with a null ref when I want to reload or change scene.

I feel like the script keeps running when I change scenes. I tried to destroy and re instantiate or disable and enable my script for my planet and my camera but still the same error.
The error seems to come from the PlanetEffects.cs because when I remove it, the error disappears but I have no more ocean or atmosphere.
I also noticed that in runtime, when I refresh PlanetsEffect (line break and save) the error disappears and everything works again.
I'm pretty new to gamedev and I'm sure I'm missing a concept or something.
If anyone can enlighten me that would be great!
I hope I was clear in my explanation.
in advance, thank you

I think you figured out most of it. PlanetEffects.cs, unlike many other scripts, is a scriptable object, which means its instance is always saved. (when in edit mode, it's also saved on the disk) Workaround? Probably would involve making pivate effectHolders list public, clearing it and calling the Init() method when done changing the scene.
Though this is my speculation and I couldn't test it, I believe that's what's going on.

Thanks for your reply :
I tried something like this in my loading system

I also made my effectHolders list public. I did the same for the Init method to be able to call them in my loading system, but still the same error. Did I do something wrong?


[Request-for-external] table 83 "Item Journal Line" - GetUnitAmount

Can you change procedure GetUnitAmount in table 83 "Item Journal Line" to global?

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.


Simple Image (Native Controls)

Hello.
Currently, I'm making a Custom Data Visualizer for Visual Studio and want to use WinApi as a Gui (low memory usage and lightning-fast ). My problem is that I want to show a simple image. And I need some guidance on where to start. (Do windows have an image Control or i just draw to the window itself?) I appreciate any help.

I recommend you use the standard GDI+ in .NET (System.Drawing.Bitmap class etc.). It's not worth it to use the WinAPI directly in this case.


Class loader issue when applying a Gradle plugin

Hi,
I'm have applied a Gradle plugin (com.netflix.nebula:gradle-ospackage-plugin) to my build.gradle.kts and I get the following error now.
Error:Cause: loader constraint violation: when resolving method "org.slf4j.impl.StaticLoggerBinder.getLoggerFactory()Lorg/slf4j/ILoggerFactory;" the class loader (instance of org/gradle/script/lang/kotlin/provider/PostDelegatingClassLoader) of the current class, org/slf4j/LoggerFactory, and the class loader (instance of org/gradle/internal/classloader/VisitableURLClassLoader) for the method's defining class, org/slf4j/impl/StaticLoggerBinder, have different Class objects for the type org/slf4j/ILoggerFactory used in the signature
Not sure if its a problem of gradle-script-kotlin.
I'm using Gradle 3.0 and the IntelliJ Communit Edition with Kotlin Plugin 1.1-M01

same here. effectively stops me from using it in anything other than reviewing the sample code.

Yes


Improved parsing for data URL scheme.

Task/Issue URL: https://app.asana.com/0/1200905986587319/1203594326324933/f
Description
Updated the data URI parsing.
Steps to test this PR

[ ] Install from this branch and download a blob. I.e https://dopiaza.org/tools/datauri/examples/index.php

NO UI changes

Shouldn't we have a test against this specific case?

We have tests. This is just removing the pattern matching on large data.


Whitespace is added at the end of the text in certain elements

Hi,
I noticed that there is a trailing whitespace at the end of the text in  and  elements, and possibly there are more, in your formatter.
Examples:

    test

!

The following

    text

will be an example.

the outputs are:
Test !
instead of Test!
The following text will be an example.
Not formatting these elements on new lines would keep the text as is and it'd display fine. For instance:
The following text will be an example.

Hi @Konstantinos-Sk. Thanks for reporting this bug. I can indeed confirm that this is a lacking feature in the formatter: it should match inline HTML elements and not apply the multiline format-rule on them. The fix is relatively straightforward, but there might be some delay before I find a free block of time.

Hi @artagnon. Thank you for this extension. It's the only one that works – I thought. If only this bug reported here, was solved. I just switched from prettier with the melody plugin to your extension in a production project at work this week. Now I'm working on a personal one and realised this. 🙈 Is there any way to motivate you, to fix this, if it's straight forward? I can also make a PR, if you give me a quick hint what to do. I'd prefer that much more than not having any formatter for twig files! I will also make a small donation, if you fix it right away.


Make it clearer that extensions can only be installed during initial setup

I've only changed the wording of the main section titles to "Starter extensions", because changing the childs titles may feel too repetitive. Also:

Tried to enforce the idea of installation - initial project in the descriptions and subtitle.
Added a visual hint in the main section and the Available Extensions section.

Closes #82

Tysm @Pabl0cks!! Umm not sure if "Starter Extension" seems a perfect word to me now I mean it makes sense just as word but for me it seems a bit weird while reading it in a sentence.

Agree with that! It felt a bit strange when reading it again. Changed back to Extensions and updated Hint => Info 🙌
TYSM for the feeback ♥

Tysm @Pabl0cks!!


Attaching the role policy to lambda function doesn't work

Hi there. I have a lambda function that needs to be inside a VPC, so I attach the managed policy arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole to it right after defining it
const importHandler = new aws.lambda.CallbackFunction(
      'sso-eagle3-import-handler',
      {
        vpcConfig,
        memorySize: 6144,
        callback: async (req: any, _ctx: any, cb: any) => {
          return import('./').then((m) => m.handleStart(req, cb, env));
        },
      }
    );

    const policyArn =
      'arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole';

    const att1 = new aws.iam.RolePolicyAttachment(
      'sso-eagle3-import-handler-execution-role-policy',
      {
        role: importHandler.role,
        policyArn,
      }
    );

However, I still keep getting an error message that indicates the policy is not attached.
The provided execution role does not have permissions to call CreateNetworkInterface on EC2

I think I'm doing everything correctly here. I'm wondering if this is  a bug.

Could you please provide the resulting diffs from your program execution? Does it include the role policy attachments that you expect?

Could you please provide the resulting diffs from your program execution? Does it include the role policy attachments that you expect?

Experiencing the same. Looks like pulumi is not adding the RolePolicyAttachment to the role. It looks all right from the pulumi console resource view, just the resource is not created in the AWS.

yep, we are having the same issue. Apparently the problem is the fact that pulumi thinks that the policy attachement was created (as this is displayed in resources stack), but it wasn't. Hence, the deployment fails.
In our experience this is not happening always. Sometimes it works and it creates the policy attachment properly, but sometimes it doesn't. I'm not sure why

The issue is that the aws.lambda.Function is being created before the aws.iam.RolePolicyAttachment is added, and this triggers the error in question.
If you ensure that the role and its attachments are created before the Function, you will not hit this.
This reply is the easiest way to ensure this: https://github.com/pulumi/pulumi-aws/issues/1308#issuecomment-876530939
For the comment about creating an independent role and attaching policies, that is totally fine as well - but you would need to ensure you use dependsOn to make sure the Function waits to create until after those role policy attachments are applied.  Otherwise, there is no way to know there is a dependency, and the policy attachments and Function will be created in parallel.  This may mean that sometimes the policy is attached before the Function so that things work, and sometime not.


flutter sdk integration issue

Why I'm getting following issue when I integrate Free-RSAP sdk. I have fixed it by implementing the code which was in guide of android setup then integration issue gone fixed , app is runnable but not working fine.

Dear contributors, I had fixed this issue using android configuration too in flutter. If we should configuration in build.gradle file than you should also enlist them in flutter guide.
Anyhow my new questions is that how we have to work this api?
I successfully integrated and callbacks also invoke on special actions i.e. emulator or rooted device.
How I have to secure my app from reverse engineering or what i have to do?

Hello.
Unfortunately, we are not able to reproduce your problem. Anyway, we will add your solution to documentation until further investigation is not done.
Reactions for each detection are fully handled by developer. We'll try to come up with some recommendations on how application should behave when certain threat is detected.


Deprecate ort.pl, change to proxy t3c

Also adds the arguments to t3c that ort had but t3c didn't.
Also fixes t3c to log atstccfg stderr logs instead of dropping them.
Includes changlog.
No docs, no interface change. Includes ort readme and changelog documenting its deprecation.
No new tests, no interface change, changes to existing arguments are covered by traffic_ops_ort.pl and t3c integration tests.

[x] This PR is not related to any other Issue

Which Traffic Control components are affected by this PR?

Traffic Ops ORT

What is the best way to verify this PR?
Run tests. Run ort, verify behavior is unchanged.
If this is a bug fix, what versions of Traffic Control are affected?
Not a bug fix.
The following criteria are ALL met by this PR

[ ] This PR includes tests OR I have explained why tests are unnecessary
[ ] This PR includes documentation OR I have explained why documentation is unnecessary
[ ] This PR includes an update to CHANGELOG.md OR such an update is not necessary
[ ] This PR includes any and all required license headers
[ ] This PR DOES NOT FIX A SERIOUS SECURITY VULNERABILITY (see the Apache Software Foundation's security guidelines for details)

Additional Information

There should be some documentation changes associated with this...

There is also a section that auto-docs the traffic_ops_ort Python package - you can get rid of that if you think it's confusing. I was gonna wait until we could get rid of the actual code, but since there's no plan to use it as a replacement for the old ORT anymore there's no reason users need it to be documented, so it can get stripped out of the docs any time.

Are we able to run t3c in CiaB instead of ORT.py?

Are we able to run t3c in CiaB instead of ORT.py?

t3c itself doesn't use Perl, which I believe was the issue.
If we want to actually install the RPM -- which we really, really, really should to catch production issues -- the RPM still has a dependency of base perl for the traffic_ops_ort.pl shim, but that shouldn't be an issue. I believe the issue was libraries; CentOS should have perl installed by default (and moreover it's part of LSB).

t3c itself doesn't use Perl, which I believe was the issue.

The issue was that ORT needed access to systemd, which I believe is still true under t3c

The issue was that ORT needed access to systemd, which I believe is still true under t3c

Ah, I'm not sure, but if it does, we intend to add flags to allow avoiding that.

There are a lot of warnings being generated about "t3c" not being in the glossary but being presented as a term. "ORT" is in the glossary and it looks like you're mostly replacing usages of "ORT" with "t3c" - if that's globally the case then you can just replace the ORT glossary section entirely.

Done.


farming station and damaged hoes no working

when i start the farming station it will work once with a hoe and then it says no hoe and when i remove it and place a new hoe in it works once again and then says no hoe

i am using version <IP_ADDRESS>6

Later versions fixed this.

Yep


Define multi-line format for Write-HostFeItemDecorator

For any command using Write-HostFeItemDecorator, the current display of just a single line for each item processed can result in too wide an output causing excessive wrapping ruining the resultant output. An  example of this is Rename-ForeachFsItem (Rename-Many).
We need to define appropriate options for Write-HostFeItemDecorator that will enable its output to be properly formatted over multiple lines for each iterated item.

I just noticed some really pointless code, that was probably being refactored at the time, but should be corrected when this issue is addressed.
Write-HostFeItemDecorator:
    [string]$writerFn = 'Write-ThemedPairsInColour';

    if (-not([string]::IsNullOrWhiteSpace($message))) {
      $parameters['Message'] = $message;
    }

    if (-not([string]::IsNullOrWhiteSpace($writerFn))) {
      & $writerFn @parameters;
    }

why is $writerFn being checked for IsNullOrWhiteSpace? Totallu pointless and should eb removed. And there is no need to writerFn , instead we c`an do a direct invoke:

Write-ThemedPairsInColour @parameters;

The first line should be considered the primary line. All subsequent lines are 'child' lines that probably should be indented. The indent could be a fixed value, or it be be aligned to match the the location after the No entry is displayed in the primary line.

Given:

[🏷] Rename Item (WhatIf)  // ["No" => "  1", "File" => "loopz.application.t1.log", "To" => "application.t1.log"]

Currently, Rename-Many adds the 'To' to the Pairs collection:

$properties += , @('To', $newItemName, $affirm);

So the invoke result is the PSCustomObject, which we can add extra items to. We can add a new property, called Lines, meaning any additional lines that need to be rendered. The Pairs collection can be seen as the primary line, which automatically contains the Product and along with 'No' is rendered as the first/primary line. Any additional content will be in the Lines property.
We need additional entry in PassThru: 'LOOPZ.WH-FOREACH-DECORATOR.INDENT'.
It will be up to the user to define this value, but it should take into account:

length of 'LOOPZ.HEADER-BLOCK.MESSAGE'
length of 'LOOPZ.WH-FOREACH-DECORATOR.PRODUCT-LABEL'
extra characters as defined in the Krayola theme (MESSAGE-SUFFIX, SEPARATOR, FORMAT & OPEN)

It will be a matter for the client to determine the correct INDENT to use via trial and error.
Write-HostFeItemDecorator will be modified to invoke Write-ThemedPairsInColour for the promary line and then for each addition line found in the Lines property on the invoke result.

Oops the fix just checked in doesnt take into account when WhatIf is not enabled which means the 'To' line will be misaligned

The issue of the wrong indentation has been narrowed down to when the items are files. When directories are renamed, there is just a single space between the // and the message, which is correct since the message separator includes the single space. So what's not clear now is where is the extra space coming from when items are files?

I discovered why for files the indentation was out by 1, whilst working on issue #44. The icon being used to represent a file was the label emoji:

🏷

For some reason, presumably, it was using one more space than other emojis, since the indent value of 39, used for directories had to be modified to 40 for files. This seemed to make no sense and still doesnt. The only way round this for now is to avoid the label emoji until it's found out what's different about this one (and I suppose potentially there are other emojis which are likewise afflicted).
Actually, we can see the problem as follows:
$sigs = Get-Signals
$sigs['FILE-A'][1]   # => 🏷
($sigs['FILE-A'][1]).Length # => returns 3 !!!!!
($sigs['FILE-B'][1]).Length # => returns 2 as expected

So some emojis are length 3 and not the normal 2.


Adds inline make documentation and 'help' command.

Summary
Adds a make help command that uses inline comments to document the commands.
Ticket Link
n/a
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] Ran make check-style to check for style errors (required for all pull requests)

If people like this pattern I'll add it to the other repos (but don't want to add it to mattermost-server until https://github.com/mattermost/mattermost-server/pull/7892 is merged).

Alternatively we can do something like this...
help:
	@cat $(MAKEFILE_LIST) | awk -F ':' '/^[a-zA-Z-]*:/ {print $$1}' | sort

...which doesn't require the line to be documented and give simple output like this...

Thoughts?

First option is a lot better imo. The comments are useful, and Makefiles often include targets that are used as intermediate build artifacts only and not meant to be seen or invoked by the user. Plus I already get that functionality from my autocompletion (make <tab> gives me that list).


Foundation

14x faster serialization in my perf. tests ... (including on linux)
Foundation:
https://travis-ci.org/vapor/json/jobs/211792443#L259
Jay:
https://travis-ci.org/vapor/json/jobs/211792443#L255

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@430c078). Click here to learn what that means.
The diff coverage is 34.78%.

@@            Coverage Diff            @@
##             master      #33   +/-   ##
=========================================
  Coverage          ?   52.13%
=========================================
  Files             ?       13
  Lines             ?      328
  Branches          ?        0
=========================================
  Hits              ?      171
  Misses            ?      157
  Partials          ?        0

Impacted Files
Coverage Δ

Sources/JSON/Optional+JSONConvertible.swift
0% <0%> (ø)

Sources/JSON/Dictionary+JSONConvertible.swift
0% <0%> (ø)

Sources/JSON/Array+JSONConvertible.swift
0% <0%> (ø)

Sources/JSON/Node+JSONRepresentable.swift
0% <0%> (ø)

Sources/JSON/InvalidContainer.swift
0% <0%> (ø)

Sources/JSON/JSON+Bytes.swift
100% <100%> (ø)

Sources/JSON/JSON+Serialize.swift
100% <100%> (ø)

Sources/JSON/JSON+Parse.swift
64.77% <63.95%> (ø)

Tests/JSONTests/JSONTests.swift
90.47% <66.66%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 430c078...2830ab4. Read the comment docs.


Finalize proxy helper methods

A finalized version of #305
Either cleanup or a rewrite.
This should enable us to easily wrap arbitrary request/response client/server interfaces over messaging channels.

Oudated due to different approach. We now use an RPC client to do the calls instead of a proxy


Installing KB4565503 appears to cause "access is denied" when starting WSL2 (custom VHDX location)

Environment
Microsoft Windows [Version 10.0.19041.388]
Ubuntu 20.04 LTS (GNU/Linux 4.19.104-microsoft-standard x86_64)
WSL2

Steps to reproduce

Move Ubuntu WSL2 VHDX to a custom location – see for example https://github.com/MicrosoftDocs/WSL/issues/412#issuecomment-501913246
Install KB4565503 – reboot
Attempt to start Ubuntu

Expected behavior
Being greeted by a Bash-prompt.
Actual behavior
Access is denied.

[process exited with code<PHONE_NUMBER>]

Additional Info
The relation to KB4565503 is somewhat speculative: I installed the update this morning (CET) before starting my working day, at which point I was unable to start Ubuntu (worked fine yesterday).
The issue appears to be that ownership of the VHDX file got handed over to SYSTEM. Changing ownership back to my current user permanently resolves the issue. Alternatively, starting WSL2 from an elevated prompt resolves the issue without modifying the access rights of the VHDX.
I've only got one copy of Ubuntu installed on my machine (via the Windows Store), so I'm unable to say if other distros are affected too.
I do have Docker Desktop (WSL2) installed (for which I've also moved the VHDX files). Docker Desktop doesn't appear to be affected by this issue; its VHDX files are still owned by the correct user.
See also this post on Microsoft Answers: https://answers.microsoft.com/en-us/windows/forum/all/kb4565503-installed-then-wsl-2-failed-to-start/25794c4f-0b20-465e-bbdb-a8af3d9e0e88

This too is happening to me after that update.  Thank you for the suggestion of changing the ownership. Is there any danger in doing so? I'm assuming that they changed the ownership for some specific reason?

This too is happening to me after that update. Thank you for the suggestion of changing the ownership. Is there any danger in doing so? I'm assuming that they changed the ownership for some specific reason?

The KB-article mentions some security related changes, but doesn't go into enough detail to make any definitive statements.
I'm no expert, but I guess there's a case to be made that someone (already having gained broad access to your system) might modify the VHDX and add some malicious code in there which – when you start WSL – would run with higher privileges than it regularly would...
Given what recently happened at Twitter, that person is probably be better off trying to socially engineer you out of some of your bitcoin though 🙂

Thanks for the response but I highly doubt somebody gained access to my system, I’m using ESET AV & scan my system regularly. I heard about Twitters breach and very impressed with their execution. I just created a btc wallet on cashapp the day before 😭

On Jul 17, 2020, at 12:25 PM, Thijs Putman<EMAIL_ADDRESS>wrote:

This too is happening to me after that update. Thank you for the suggestion of changing the ownership. Is there any danger in doing so? I'm assuming that they changed the ownership for some specific reason?
The KB-article mentions some security related changes, but doesn't go into enough detail to make any definitive statements.
I'm no expert, but I guess there's a case to be made that someone (already having gained broad access to your system) might modify the VHDX and add some malicious code in there which – when you start WSL – would run with higher privileges than it regularly would...
Given what recently happened at Twitter, that person is probably be better off trying to socially engineer you out of some of your bitcoin though 🙂
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

In my case running ubuntu as administrator and then reopening it without admin resolved the issue. I did notice that the folder containing my vhdx had Administrators as owner

Environment
Microsoft Windows [Version 10.0.19041.388]
Ubuntu 20.04 LTS (GNU/Linux 4.19.104-microsoft-standard x86_64)
WSL2

Steps to reproduce

Move Ubuntu WSL2 VHDX to a custom location – see for example MicrosoftDocs/WSL#412 (comment)
Install KB4565503 – reboot
Attempt to start Ubuntu

Expected behavior
Being greeted by a Bash-prompt.
Actual behavior
Access is denied.

[process exited with code<PHONE_NUMBER>]

Additional Info
The relation to KB4565503 is somewhat speculative: I installed the update this morning (CET) before starting my working day, at which point I was unable to start Ubuntu (worked fine yesterday).
The issue appears to be that ownership of the VHDX file got handed over to SYSTEM. Changing ownership back to my current user permanently resolves the issue. Alternatively, starting WSL2 from an elevated prompt resolves the issue without modifying the access rights of the VHDX.
I've only got one copy of Ubuntu installed on my machine (via the Windows Store), so I'm unable to say if other distros are affected too.
I do have Docker Desktop (WSL2) installed (for which I've also moved the VHDX files). Docker Desktop doesn't appear to be affected by this issue; its VHDX files are still owned by the correct user.
See also this post on Microsoft Answers: https://answers.microsoft.com/en-us/windows/forum/all/kb4565503-installed-then-wsl-2-failed-to-start/25794c4f-0b20-465e-bbdb-a8af3d9e0e88

Can you give us the steps to change the permissions to current user? Thank You.

I didn't have to mangle permissions per se, as long as the instance is running I'm fine, so it looks like there's something that doesn't like starting up.
From one admin powershell
PS H:\> wsl --list --running
There are no running distributions.

and then from my regular non admin shell
PS C:\Users\tanant> wsl
Access is denied.

then bouncing back to the admin shell
PS H:\> wsl -u tanant
tanant@cuberdon:/mnt/h$

and in my regular shell
PS C:\Users\tanant> wsl -u tanant
tanant@cuberdon:/mnt/c/Users/tanant$

(I could also solve this by giving Users full control)

Ran into this same issue after the recent Windows update, the solution by @Uldiniad above worked.
Seems like I only needed to start it once as administrator. Afterwards it went back to normal.
One particular difference in my setup is that I have previously moved the WSL installation (its vhds) to a different drive. Not sure if this triggered the bug or not.

I need to run it as administrator after every reboot. After running as admin, WSL works. But after I restart it goes back to the "access is denied" state until I re-run it as admin again.

Happened to me but VHDX was not in a custom location.

Happened for me exactly as described by @thijsputman - on rollback of KB4565503 WSL2 ubuntu worked fine.  Ubuntu is installed at an alternate location.
However, having read @Uldiniad 's comment I reinstalled KB4565503.  I noticed that the owner of the ext4.vhdx (and the containing directory) did not change.  But I couldn't start wsl from a standard (unelevated) prompt.
I am now in @summatix 's situation - after every reboot I run wsl as administrator, then everything works fine.
Windows version currently reporting as 10.0.19041.388.  Ubuntu is 20.04.

I have now used the suggestion from here - https://answers.microsoft.com/en-us/windows/forum/all/kb4565503-installed-then-wsl-2-failed-to-start/25794c4f-0b20-465e-bbdb-a8af3d9e0e88 - and assigned Full Control to Users and i'm good even without the workaround.

I'm having the same issue as described by @summatix and @david-bakin . In addition, to be sure if the problem was with my Ubuntu with alternate locale (or with Ubuntu distro in general) i've installed a Debian from Microsoft Store without any changes. It worked but, when I restarted my system it stoped working and started showing the same error message.

@david-bakin The link in your suggestion worked for me. Thanks for posting that.

Same issue here. Getting Access denied after every reboot when trying to start WSL via either the wsl or bash commands, unless I ran it from an administrator prompt.
Rather than uninstalling KB4565503, simply locating and granting my user full permission to the .vhdx virtual disk file fixed it.

Another thing you can do to resolve this:

(as admin) wsl.exe --set-version distro 1
wsl.exe --set-version distro 2

I have a similar but different variation of this issue.
I had WSL1 installed with Ubuntu 18.04 (manually as my network is air-gapped).
I attempted to update Docker Desktop and it prompted me to switch to WSL2 which I attempted. Docker unfortunately crashed immediately with an access denied error. I then went and tried to run wsl/ubuntu directly and got an Access is Denied error there.
Seeing the messages about admin/non-admin, I had to go disable the GPO someone put in place that disabled UAC prompts to be able to get a non-admin window. Afterwards, I confirmed I can run WSL in a non-admin window but not in admin. I unregistered my Ubuntu distro, re-applied the WSL2 update, converted to WSL2 default wsl --set-default-version 2 and restarted Docker and it works.
I do not currently have any distro installed but still cannot run WSL.exe at all from admin shell (cmd/PoSH/etc). My original installation was not VHDX and was not in the %appdatalocal% folder; it was a folder structure under c:\Ubuntu.
I've tried removing WSL from the computer, restarting, reapplying, restarting, and that did not have any affect on the issue. So there's something outside the VHDX files itself that's causing my problem.
For reference, I had a second internet connected computer I tried the Docker WSL2 update on and it works without issue, but that machine never had WSL1/Ubuntu on it.

I'm seeing this too. I cannot run wsl at all -- I get "access is denied". I tried uninstalling wsl, and noticed that it did not remove c:\windows\system32\wsl.exe! I could not remove that myself in any way, so I eventually resorted to AdvancedRun which allowed me to remove it by running as TrustedInstaller. But now when I reinstall wsl (using powershell or the "windows features" dialog) it does not recreate wsl.exe. Seems like some further permissions issues are happening.
I am running Win11, fresh install as of a few weeks ago. My vhdx is in a custom location (and I have docker, which creates its own vhdx files & distros -- those are in the standard location).
I tried sfc /scannow which replaced wsl.exe but I still get "access is denied".
I am totally stuck at this point; short of a full OS reinstall I'm not sure what to do.

I have now used the suggestion from here - https://answers.microsoft.com/en-us/windows/forum/all/kb4565503-installed-then-wsl-2-failed-to-start/25794c4f-0b20-465e-bbdb-a8af3d9e0e88 - and assigned Full Control to Users and i'm good even without the workaround.

@garyo The suggestion outlined in the link that David provided worked for me. Have you given it a try?

I have now used the suggestion from here - https://answers.microsoft.com/en-us/windows/forum/all/kb4565503-installed-then-wsl-2-failed-to-start/25794c4f-0b20-465e-bbdb-a8af3d9e0e88 - and assigned Full Control to Users and i'm good even without the workaround.

@garyo The suggestion outlined in the link that David provided worked for me. Have you given it a try?

Yes, I tried all the permission-related things, uninstall/reinstall, change ownership of wsl.exe and WindowsApps folders, etc.
I finally (just a couple of minutes ago) got it working again though! I had to not only uninstall wsl from "Turn Windows Features On/Off", but also uninstall all of my distros and all WSL-related items from "Add/Remove Programs" and reset Docker to factory defaults. At that point, weirdly, it started working again! (Yes, even before reinstalling anything. So c:\Program Files\System32\wsl.exe still existed.) After that I reinstalled things as per usual, and now I'm back working again. I cannot explain what was happening but it was extremely scary -- all my work is now in WSL.


Add support for other COS milestones (+109, +105)

In order to add support for further milestones, we can take the patches from master and apply those. This then removes the requirement of specifying the overlays branch name, so that is removed as well.
CCing @a-crate, @dorileo, @zmarano.

/ok-to-test


Functionality to upload zipped experiment

A user may want to deploy a custom experiment that is not part of expfactory-experiments, so we should allow for upload of a zipped file, and then have a "private" experiment only accessible to those with permissions for it.

This could be a security hazard, and we aren't offering this for users as a service, so I'm going to close for now.


Format selection html

Is this possible?

same question

+1

+1

+1

Sorry been incredibly busy, are we talking about formatting just the currently selected / highlighted portion of the file? If this is the case it won't be a problem to add.

Yes, like below:

Got it, good suggestion I'll add it in shouldn't be too bad.

Any update on this?


Shift + Left mouse button does not work in the crafting menu 1.19.2

It is impossible to quickly remove objects through shift + left-click

This is a duplicate of #426 which will be fixed in the next update

This is a duplicate of #426 which will be fixed in the next update

Nice))))


Spurious rebuilds of paths-overrides

I think I've noticed something related to Servo's "spurious rebuilds" issue! They seem to be triggered by Cargo deciding that one of the paths-overrided dependencies needs to be rebuilt. So, we see it a LOT when working on a rustup (or things like my android changes touching a lot of stuff), but still see it a bit on Servo itself because we have a paths override even in our default build (https://github.com/servo/servo/blob/master/components/servo/.cargo/config#L1 ).
Basically, without touching my machine I can just re-reun a build and it will seemingly randomly occasionally have a "good rebuild" log, but much more often have a "bad rebuild" log. The bad ones always start with things that are being overridden - in this case, I'm overriding skia, angle, gleam, rust-offscreen-rendering-context, and glutin.
Fingerprint log of a good rebuild:
https://drive.google.com/file/d/0B9-EBhw4XiWEZlVRZUpGcjFqYVU/view?usp=sharing
Fingerprint log of a bad rebuild:
https://drive.google.com/file/d/0B9-EBhw4XiWEWU1TRGFaZVJfR2s/view?usp=sharing
Hope this is enough info to debug it!

With some output from the newer cargo fingerprinting stuff:
https://pastebin.mozilla.org/8848748

[16:24:44]  <acrichto>	larsberg: hm ok, so this is kinda coming back to me
[16:24:54]  <acrichto>	so we don't record a lockfile for path overrides
[16:25:09]  <acrichto>	larsberg: and as a result when you override, if there are two deps in the graph which satisfy there override's dep it may switch between those two
[16:25:22]  <acrichto>	larsberg: e.g. due to the nondeterminism of HashMap or something along the way
[16:25:37]  <acrichto>	larsberg: e.g. in that one gleam switched khronos_api deps
[16:25:51]  <acrichto>	not entirely sure what's up with the others though
[16:30:28]  <@larsberg>	acrichto: Makes sense, and explains the non-determinism. So it would go away if we really hard-enforce that there are never two copies of a dependency?
[16:30:43]  <@larsberg>	acrichto: We were planning to do that anyway for Servo; this would just move up the priority a lot :-)
[16:31:35]  <acrichto>	larsberg: for the gleam rebuild I think that'd do the trick, but this is definitely a bug in cargo b/c there should be no reason to not have two copies of a dep
[16:31:55]  <acrichto>	larsberg: for the others where the fingerprint file just couldn't be found, I'm not sure what's up there
[16:32:09]  <acrichto>	the only explanation I have is that the package id for things like the layout crate changed entirely
[16:32:13]  <acrichto>	between rebuilds
[16:32:16]  <acrichto>	(which makes no sense...)
[16:32:24]  <@larsberg>	acrichto: I assume for the others it's just because we're hitting the cargo OS path crash right now :-)
[16:32:38]  <acrichto>	larsberg: OS path crash?
[16:32:55]  <@larsberg>	acrichto: The one you said you just fixed today? (OS paths too long error)
[16:33:09]  <acrichto>	larsberg: oh did those crates just not build at all?
[16:33:23]  <acrichto>	if that's the case then that's totally fine
[16:33:24]  <acrichto>	lol
[16:33:33]  <@larsberg>	acrichto: That's my suspicion :-) Thanks for the debugging help!
[16:33:55]  <acrichto>	larsberg: note that if you wanna fix this *now* you could change the dep of gleam -> khronos_api to be more exact
[16:34:01]  <acrichto>	e.g. make it a semver dep instead of a * dep
[16:34:18]  <acrichto>	yu're guaranteed there's only ever one dep in a semver range, so that'd make the problem moot temporarily as well

To recap, my belief is that this is the following bug:

There is a dependency A which shows up twice in the resolution graph (e.g. two semver incompatible versions).
The dependency B which is being overridden depends on A and the version dependency covers both of the versions of A in the graph.
Each time B is overridden with the custom path, it nondeterministically selects either version of A, causing rebuilds whenever it selects a new version.

There's a few possible solutions to this that I can think of:

Emit a lockfile for the resolved graph after overrides (not a great solution, not even sure if this is practical)
Try to "always choose the biggest" version instead of picking either version (I'm not sure if this is 100% reliable either).
Be more picky about path overrides in terms of their upstream dependencies (e.g. must perhaps match exactly what's already in the graph structure itself).

The question of overriding a dependency with a whole new set of transitive dependencies has always been a little sketchy to me, so it's unclear how robust this can be made in the long-term.
@larsbergstrom for now I'd recommend not using path overrides for large scale refactors but instead using normal git and path dependencies which should always be reliably.

I'm having a lot of success right now just removing the * dependencies and removing any duplicatly included dependencies (it's an error for Servo, anyway), which is an easy workaround we should have been doing anyway :-)
I'm not really sure what you mean by normal git and path dependencies here instead of paths. Do you mean editing the Cargo.toml that has a Cargo reference to instead have a pointer to a git repo? If so, I suspect that will have really huge dependency chains - e.g., that I'd have to clone and edit a slew of intermediate repos to get, say, everybody who references gleam pointing at the same thing. Sorry if I'm misunderstanding!

Ah yeah I meant editing Cargo.toml to point to the new dependency (in a first class, not overridden fashion). But if that's too painful then ah well, worth a shot!

Hi !
Is this still an issue, @larsbergstrom ?

@cyplo Yes, but we can now use [replace] to avoid many of the scenarios, so long as the replaced dependency has not moved to a newer Cargo version number.

Another case #3136 has exposed that wasn't handled correctly before:
If you've got a paths override pointing at a crate that itself has path dependencies, those dependencies have historically been followed. For the path dependencies, though, there's no lock file entries to indicate what they should be locked to. This would introduce spurious recompiles as they wouldn't always re-select the right versions.


Clarify ambiguity around use of artifact name

As input, startship release requires the group name, artifact name, and version.
Many projects, including mine, publish multiple artifacts, so the requirement for an artifact name was initially concerning. As it turns out, it's not a problem: the entire multi-artifact repository is closed and then released as expected. But the requirement of an artifact name as input and the messages shown in the terminal along the way give the false impression that only one of the artifacts is being released.
So I'd like to:

Rephrase the command-line messages to avoid suggesting that only one artifact is released.
Document what exactly the artifact name is used for.
If possible, make the artifact name optional: Identify the staging repository only by group and version, failing only if there is more than one match. (If this is possible, maybe the version can also be optional.)

TIL. I have zero experience releasing multi-artifact projects so I did not account for it. I agree with all your suggestions. Matching only group and version sounds good enough to me!

For example, OkHttp has the main OkHttp artifact plus Logging Interceptor, MockWebServer, and a bunch of others.
OK I will dive into the source code and see if I can do a PR for this one too!

So part 3 is difficult because you use the artifact ID to check the maven-metadata for the updated version. Will have to think about it a bit more.

Riiight. I wanted to look if Sonatype has an API for inspecting artifacts inside staged repositories but I couldn't get there. Maybe startship can show a warning message and ask for user's input if the maven coordinates don't match instead of throwing an error? The user can choose to continue anyway. Wdyt?

I wanted to look if Sonatype has an API for inspecting artifacts inside staged repositories

Exactly, this would be ideal. "couldn't get there" meaning you didn't have time to dig through the APIs, or you determined no such API exists?

Maybe startship can show a warning message and ask for user's input if the maven coordinates don't match instead of throwing an error?

Which error is this?

"couldn't get there" meaning you didn't have time to dig through the APIs, or you determined no such API exists?

Didn't have enough time 😄

Which error is this?

Here you go: https://github.com/saket/startship/blob/1874f6437ca54fd5bb7a981efe4d6bb11244899b/src/main/kotlin/nevam/ReleaseCommand.kt#L118

OK I think I figured out how to do this. As far as I can tell it's not documented as part of the API, but your current validation via NexusApi.stagingMavenMetadata was my clue to figure it out.
/repositories/{repositoryIdKey}/content/ is the content directory for the (staging) repository, and from there it's effectively a file system. The response looks like this:
<content>
  <data>
    <content-item>
      <resourceURI>https://oss.sonatype.org/service/local/repositories/{repositoryIdKey}/content/{directoryName}/</resourceURI>
      <relativePath>/{directoryName}/</relativePath>
      <text>{directoryName}</text>
      <leaf>false</leaf>
      <lastModified>2020-11-21 10:27:47.0 UTC</lastModified>
      <sizeOnDisk>-1</sizeOnDisk>
    </content-item>
  </data>
</content>

Where directoryName is the first part of the group being published. directoryName can be appended to the path to read that directory, and so on through each directory. The <data> element will have multiple <content-item> children if it has multiple child directories.
From here it depends on where you want to draw the line between strict validation and too many network calls. If the user of the command provides a group, multiple artifact names, and a version; the tool can either (1) validate that those artifacts exist in the repository, or (2) validate that only those artifacts at that group with that version exist in the repository.
Even with the extra API calls I think (2) is the better option, to ensure you don't accidentally publish something that can never be unpublished.
Finally, there is a question of API design. I envision 3 new command line options: -g for the group, -a for a list of artifacts, and -v for the version. I guess a user would have to either (a) pass all 3 of those, (b) pass the existing -c, or (c) pass none of these to rely on the Gradle properties. I'm not sure if there are any command line conventions around mutually-required and/or mutually-exclusive options, or if Clikt has any conveniences to make this easy. Also not sure what the delimiting character should be for the -a list.
So I guess let me know if you have any opinions or knowledge on anything I summarized in this comment, otherwise I'll work my way through implementing it!

I just got a chance to see this. I love this idea.
I agree that validating all groups (option 2) would be great but may not be worth doing it for the complexity. Option 1 is still useful nonetheless.

I guess a user would have to either (a) pass all 3 of those, (b) pass the existing -c, or (c) pass none of these to rely on the Gradle properties. I'm not sure if there are any command line conventions around mutually-required and/or mutually-exclusive options, or if Clikt has any conveniences to make this easy. Also not sure what the delimiting character should be for the -a list.

Is this still a challenge? I saw your proposed syntax in #9 ($ startship release -c com.example:nicolascage,dianekruger:4.2.0) and that looks like a solved problem now?

Yep, if you like the -c with commas then I'd say it's solved. I like that it doesn't require new options or complex rules about which options take precedence over others.

Yeah sounds good then. Let's go with this!


Thread - Edited link glitches back to original when sending a message

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 9.0.66-0
Reproducible in staging?: Y
Reproducible in production?: Y
If this was caught on HybridApp, is this reproducible on New Expensify Standalone?: N/A
**If this was caught during regression testing, add the test name, ID and link from TestRail:**https://expensify.testrail.io/index.php?/tests/view/5256663
Email or phone of affected tester (no customers<EMAIL_ADDRESS>Issue reported by: Applause - Internal Team
Action Performed:

Go to a chat and send Google.com
Open thread to the above message and edit the link to test.com ( edit both the link adress and the name i.e (test.com)[test.com]
Observe the thread title and parent message are both edited
Send a message in the thread and observe the title and parent message glitch and revert back to Google.com for a few seconds

Expected Result:
Edited link should not glitch and revert to original content
Actual Result:
Edited link glitches and reverts back to original content for a few seconds
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android: Standalone
[ ] Android: HybridApp
[x] Android: mWeb Chrome
[ ] iOS: Standalone
[ ] iOS: HybridApp
[ ] iOS: mWeb Safari
[ ] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos

Add any screenshot/video evidence

https://github.com/user-attachments/assets/1f43e620-0ce1-497b-8198-be827693190b
View all open jobs on GitHub

Wasn't able to repro this, looks to be working now!
https://github.com/user-attachments/assets/131c2e2e-aba6-4d06-b380-fa44b09f2feb


[FX-317] Add ForageLogger

What

Why

Test Plan

✅ | ❌ Not a front-end change
✅ | ❌ iOS QA Tests passed locally
✅ | ❌ Unit Tests passed locally

How

Current dependencies on/for this PR:

main

PR #54   👈

PR #55

This comment was auto-generated by Graphite.


Long latency in multiple threads/task environment.

for (int i = 0; i < 500; i++)
{
	Task.Factory.StartNew(() =>
	{
		bus.Publish(new Message());

		// do other job.
		Thread.Sleep(10000);
	});
}

Like the code shows, We have 500 threads, and each publish a message, then do other job,
However , No message will be publish out,  Because There is no Thread used to publish message.
MassTransit.RabbitMqTransport/Contexts/RabbitMqConnectionContext.cs
public async Task<IModel> CreateModel()
{
//   Task.Factory.StartNew will not work, Because There is no available thread in threadpool to do this .
//    return await Task.Factory.StartNew(() => _connection.CreateModel(),
//        _tokenSource.Token, TaskCreationOptions.HideScheduler, _taskScheduler);

    // should be below.
    return _connection.CreateModel();
}

MassTransit.RabbitMqTransport/Contexts/RabbitMqModelContext.cs
 async Task ModelContext.BasicPublishAsync(string exchange, string routingKey, bool mandatory, bool immediate, IBasicProperties basicProperties,byte[] body)
{
    // No thread to publish.
    // PendingPublish pendingPublish = await Task.Factory.StartNew(() => PublishAsync(exchange, routingKey, mandatory, immediate, basicProperties, body),
    //    _tokenSource.Token, TaskCreationOptions.HideScheduler, _taskScheduler);
    // await pendingPublish.Task;

	PendingPublish pendingPublish = PublishAsync(exchange, routingKey, mandatory, immediate, basicProperties, body);
	await pendingPublish.Task;
}

Remove all Task.Factory.StartNew will solve this issue.
Will you accept my pull request for this issue ?

I'm not sure what's being asked here, so I'm going to request that either more information is added, or that this issue be closed.
Also, bus.Publish is an asynchronous method, so you don't need to use Task.Factory.StartNew, just don't await the call or collect the tasks into a List and then await the entire list with Task.WhenAll.

Calling BasicPublish on a single channel is not thread safe, and should not be called by two threads at the same time, so your changes will actually break and cause thread concurrency issues.

OK, I check the BasicPublishAsync method , and notice It use the QueuedTaskScheduler to do the scheduler task,
It's the Task.Factory.StartNew here really block tasks,  It will be great to support inject a task scheduler, otherwise suggest to not use async here, because there is no way to control this task.
diff --git a/src/MassTransit/Util/QueuedTaskScheduler.cs b/src/MassTransit/Util/QueuedTaskScheduler.cs
index 8380f50..45cdaeb 100644
--- a/src/MassTransit/Util/QueuedTaskScheduler.cs
+++ b/src/MassTransit/Util/QueuedTaskScheduler.cs
@@ -333,8 +333,7 @@ namespace MassTransit.Util
                 // If necessary, start processing asynchronously
                 if (launchTask)
                 {
-                    Task.Factory.StartNew(ProcessPrioritizedAndBatchedTasks,
-                        CancellationToken.None, TaskCreationOptions.None, _targetScheduler);
+                    ProcessPrioritizedAndBatchedTasks();
                 }
             }
         }

In the newest MT, use LimitedConcurrencyLevelTaskScheduler.
this._taskScheduler = new LimitedConcurrencyLevelTaskScheduler(1);

I suggest to check the SynchronizationContext.Current property first to support inject a task scheduler.

I think you're missing the point of using the task scheduler - it's to keep all of the use of the channel serialized.

I agree, However, It will be great if We can control this task scheduler,  I prefer to use a single forground thread instead of use thread pool thread.

I'm not sure there is any action that will be taken with this one, so I'm closing it.


remove 'align' wrapper and external padding from control widgets

Many of the control widgets are wrapped in an Align widget that is intended to draw them centered if they are in containers larger than themselves; I think this is a mixing of responsibilities, and the container should be responsible for centering. (this is addressed in #583)
In addition, I do not think that control widgets should have external padding, that is they should not draw additional whitespace around themselves that is outside of the area that they paint.

fixed in #605


Removed TabletStatistics.getSplitCreationTime SPI method

Fixes #3595

I will create a follow-on task on merge to mark this method as deprecated in main.

I will create a follow-on task on merge to mark this method as deprecated in main.

Was this done?

Was this done?

Done in #3977


Add iPhone X launch screen image size

Although it's not the adviced strategy, the iPhone X still supports static images for the launch screen. This PR adds support for generating that format.

Didn't try but test looks good and I trust you for it :)
Interested in taking over this project?

Mind pushing out a new release to NPM that contains this?

Done! 0.24.0 is on NPM


[PLUGIN-386] Displaying BigQuery MaterializedViews in Wrangler

Wrangler was unable to load BigQuery datasets that contained MaterializedViews. This was caused by a bug in the BigQuery java library which was fixed in March 2020. This PR updates the Google dependencies to resolve this bug and ensure there are no version conflicts.

The source doesn't support views either. Are we planning to add support in the same release? Otherwise it will break later when you go from wrangler to a pipeline.


Change default DMA channel to 10

As shown in this commit, the root library follows this same default for general safety.

Done. Thank you for your contribution.


README includes instructions for using the spec with the most popular API gateways

Add a section to the README for "Using the spec with API gateways" and add some sample instructions for using the spec with at least one of Kong, Apigee, or AWS API gateways.

If it helps, here are some benefits of importing your spec into your API gateway (and the internet may have even more ideas):

Rate limit connections to Conjur
Be a part of enterprise's standardization of all 3rd party APIs in use
Devs can use LDAP to authenticate to all APIs they have access to consume via the API gateway (including Conjur)
Allows for collecting data / metrics on routes & requests
Enables applying access control list (ACL) to routes from all APIs used across the enterprise.
Allows for easy mocking and virtualization to reduce load on microservices.


Risk score optimisation

Feature: risk scores would be more refined: The bot will follow these steps after every new installation
It needs to prepare all the files with associated risk scores beforehand to react to PR events.
To do so, it needs to fetch all repositories on which the bot has been installed.
Then, iterate on each repository and get all files.
For each file, first calculate the risk score and then save it.
Now, the bot can give the files with their respective scores in every PR.

Added this feature in the PR #14


EPIC - Records Mode Redesign

I think it would be useful to create an Epic here (series of user stories) that captures what we want in our Records mode redesign (these ideas can suffixed with linked issues and mailing list threads for further details and discussion):
From Google Research:

The  data  used  in  web  and  scientific  computing  is  often  non-relational.   Hence,  a flexible data model is essential in these domains.

From Mailing List:
The future of records mode

Flexible Data Modeling:

[ ] Support flexible data model representations of Records #2298 , #737
[ ] Nested Hierarchies
[ ] Flat Hierarchies

Flexible Data Representation:
Querying:
JSON

[ ] Allow easier querying of data not in first normal form (multiple nested levels).

[ ] TODO: Research https://rumbledb.org/

[ ] TODO: which relies on JSONiq a JSON query language

[ ] TODO: Research jsonpath-plus https://github.com/OpenRefine/OpenRefine/issues/2515

XML

[ ] TODO: Querying XML might be slightly easier with JSONiq Extension to XQuery?

Views:

[ ] Tree lists are very useful. (and not so useful, the further you go into a tree the more difficult it is to see where you came from as the leaves can obscure the broader context of the roots)

[ ] IDEA: but sometimes NOT showing branches but instead only leaves(that hold values) that are the same "color" across many branches is sometimes useful. Allowing to quickly see where you came from. A complete nesting level picture.  A refreshable Accordion nav might be presented instead of Tree nav.

[ ] is "color" here really an alternative way to say "groups of similar or related interesting levels"? I.E. mailing address, and location of birth both have similar subfields (city, state, zip, country), I.E. A Location.

[ ] So Schema overlays (and metadata) seem to apply here at any particular nesting level?  I.E. it could be said that mailing address and location might share parts of a schema.  Giving users the power to change schema mapping of related fields would be useful and different schema views might help show users those branches with the "same color" or "similar schema".

[ ] Branch only views?
[ ] Zooming in/out of nesting levels to work with all records at a particular level.  An example of this would be an Icicle chart with zoom and selection as demonstrated at https://mbostock.github.io/d3/talk/20111018/partition.html
[ ] Cascaded treemaps can more clearly and quickly show structure (shallow and deep levels) of a hierarchy even with large number of nodes. https://homes.cs.washington.edu/~jfogarty/publications/gi2008.pdf

Transforming (meaning Mass Editing - this is different than Editing):

[ ] Allow remapping a child node to a different parent node. This might be done visually by using click, drag lines between nodes somehow (examples Quickbase etc.), or just textually by filling in a form showing some Remap Node dialog (New Parent:  / Child:)
[ ] Transform within only a zoomed set of nesting level across all records
[ ] Transforming sub data sometimes in arrays into columns/fields of a record #36

Editing (meaning Single Editing - not Transforming):

[ ] Allow optional single record editing mode (work with 1 parent record at a time, no grid needed?, and uses vertical representation, instead of currently multi-column horizontal representation)
[ ] Edit within only a zoomed set of nesting level across all records

Navigation:

[ ] Allow showing a Tree view navigation of hierarchical data next to the data grid (Examples: https://jsoneditoronline.org/  which uses https://ace.c9.io/) with optional filtering, sorting mechanisms in the Tree nav itself.  Another example is using Accordion style nav to always see the root levels (always knowing where you came from) https://examples.sencha.com/extjs/6.5.0/examples/kitchensink/?modern#tree-list

[ ] Furthermore, Accordion nav is still unwieldy when there are many root nodes, so having a search box that allows typed in text within the Accordion nav at any parent level would be better.  This would allow faster filtering of the tree nodes.  The best example I have seen that also incorporates recursive selection is demonstrated here: https://rsuitejs.com/en/components/multi-cascader

[ ] Support tabbed navigation during Record field editing (similar quick edit experience to other spreadsheet software)  #2636

[ ] It would be useful during single record editing mode (vertical representation), to allow some option to click on a subfield's label or a + plus sign next to it to quickly show the grid again of flattened record fields horizontally having that particular subfield (so that tabbed navigation can be done for all records, or mass edits can be performed against all records with a given filter(s)).

[ ] TODO:  This needs a visual drawing for clarity.
[ ] Might correlate with Branch only views?  To quickly reshape the grid to work only within sets of subfields (not showing parent branches?  I.E. a way to quickly zoom in and out of the nesting levels to work and tab navigate edits within only those levels?

Some discussion is here: https://groups.google.com/forum/#!searchin/openrefine/records$20mode|sort:relevance/openrefine/X9O8NBC1UKQ/alxV_f3BBwAJ

Hmm,  this might fit into one of the issues/stories off of this Epic. https://rumbledb.org/
Putting it under Querying for now

@wetneb Hmm, Validation (JSON Schema, etc.) probably needs to fit into the big picture here somewhere.
At import time, I can see someone using OpenRefine to validate that a particular large json file has records that do validate against a particular schema.  Conversely, at export time, they want to conform to maybe another schema.
Perhaps we need an Import/Export section to cover that.

I do not think JSON validation is related to this issue.

I'd suggest that rather than thinking about a "Records-mode redesign", we instead focusing on designing ways to better satisfy the user needs that records mode is used, sometimes with great difficulty, for, as well as other related needs like JSON round-tripping. We don't necessarily need a single "mode" or tool to satisfy all the needs.
Implicit in this is going to be reworking the tree-based importers since that is one of the primary sources of "records" currently -- and the only source of column groups.

@tfmorris Records mode (fields grouped with values) currently in OpenRefine is used myriad of ways; I know this because I myself use it in ways not even Records based, but rather like Set-based or Collections-based at times.  David H. and I talked at length about various presentations of data (besides fields grouped with values) that could be done in OpenRefine.  But when it comes to Records editing...only 2 ways I've seen, Single editing of a record, and Mass editing of records.  Agree?
@tfmorris Do you also agree with the 4 scope areas of Querying, Views, Editing/Transforming, and Navigation from a high level when it comes to the concept of Records (fields grouped with values)? That's what I am traditionally used to in my domain expertise.

As a product manager, I'd collect use cases, extract the commonalities and synthesize requirements from them. I wouldn't start from technology choices or implementation details, although I might later look at how others solved similar user requirements. Users who are dealing with mostly grid data that has one column of multivalued cells might have entirely different needs from users who are trying to do round trip transforms of JSON data.
Do we have the use cases that we want to address enumerated somewhere? I know both Antonin and Martin mentioned some on the email thread, but having them all collected in one place for people review, add to, organize, etc would be a good start.

@tfmorris Do a search on "records" in our issues. https://github.com/OpenRefine/OpenRefine/issues?q=is%3Aissue+is%3Aopen+record
Collecting them and organizing the use cases into this EPIC is what I've been trying to do.  The tech examples provide context not direction or even motivation.  This is a living document that can hold links to other documents as necessary.  We wanted 1 big issue to hold it all together as an Epic (many user stories), instead of a single user story.

Added:  Transforming sub data sometimes in arrays into columns/fields of a record #36

An interesting idea for Records visualization of nested structure might use Bracket guides such as in VSCode's usage of faint thin lines that outer wrap the sub-records (substructures):

This issue is a bit messy and GitHub issues do not feel like the best place to have such high-level design discussions. We now have a category on the forum for such discussions, and I created a thread there: https://forum.openrefine.org/t/representing-hierarchical-data-beyond-the-records-mode/425
So I propose to close this.


Make ksuid hashable so it can be used within HashSet & HashMap

Feature Request
Motivation
I would like to use ksuid within an HashMap<Ksuid/KsuidMs, T> and I would prefer not to convert to a string, i.e. HashMap<String, T>.
Right now I am wrapping it within a struct (I am not aware of a better way, like a macro to implicitly call to_string before hashing):
use std::hash::{Hash, Hasher};
use svix_ksuid::Ksuid;

pub struct Id(pub Ksuid);

impl Hash for Id {
    fn hash<H: Hasher>(&self, state: &mut H) {
        self.0.to_string().hash(state);
    }
}

Proposal
I believe the following would be enough:
use std::hash::{Hash, Hasher};

impl Hash for Ksuid {
    fn hash<H: Hasher>(&self, state: &mut H) {
        self.to_string().hash(state);
    }
}

impl Hash for KsuidMs {
    fn hash<H: Hasher>(&self, state: &mut H) {
        self.to_string().hash(state);
    }
}

Pros:

simple
not prone to bugs or hash collisions (not more than ksuid as far as my understanding)

Alternatives
Having an intermediate String before computing the hash is not the most efficient way to do this but I personally would prefer to have something even if it is not the most performant 😇
Next
If you don't mind my proposal, would you like me to make a PR with it?
If you do mind adding this, fair enough, I will handle this on my end. 😉
If you do mind using the intermediate string, do you already have in mind a way to do it or would you like me to look into it? (my experience is limited)

Either way, thank you for the library 😄

Thanks a lot, and yes, please open a PR!

Sure, I have just checked your source code, the internal representation is only:
pub struct Ksuid([u8; TOTAL_BYTES]);

I thought you had both timestamp and byte array and that it could be cumbersome.
I will probably do this tomorrow. (it is late right now, where I live) 😉

Yup, the internal representation is just the byte array, so it should be easy. :)
Looking forward to seeing that PR!


minor refactors and clean ups

I have performed minor refactors and clean ups so that code became less "noisy" :)

Wow - you really went through it all - thank you so much! That was not a trivial scan through the codebase. I just had a question about that variable change, otherwise everything else looks great to me. It was in need of a cleanup!

I just had a question about that variable change

Hi, thanks for the review. What variable exactly?

The NULL_VAL variable in QueryFilter.cs

The NULL_VAL variable in QueryFilter.cs

According to Microsoft's Framework Design Guidelines const values should be named in PascalCase,
Please refer to the following stackoverflow discussion in order to see other details: https://stackoverflow.com/a/242549


chromium{Beta,Dev}: fix build

Motivation for this change
fix build as the patched for Chromium 69 got rejected by Chromium 70

@GrahamcOfBorg build chromiumDev

@GrahamcOfBorg build chromiumBeta

Unexpected error: command failed with exit code 1 on aarch64-linux (full log)
Attempted: chromiumDev
Partial log (click to expand)
copying path '/nix/store/ny2hnimw473ixlx153a96x2iihdv66ng-openssl-1.0.2p-dev' from 'https://cache.nixos.org'...
cannot build derivation '/nix/store/js7pr7qbm6n03q4jaa0ydgjgsjrcf60v-chromium-71.0.3559.6.tar.xz.drv': 3 dependencies couldn't be built
copying path '/nix/store/1msxcdd19ggxzvh4vjk82glaqgp45wp3-mesa-noglu-18.2.1-drivers' from 'https://cache.nixos.org'...
copying path '/nix/store/wd157di32smpqd3xspnjmafbpkgqzxyp-mesa-noglu-18.2.1-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/68nrjw5j9xwlykcqngzrfnzi2v8knyn0-libGL-1.0.0-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/qd9vmi95r8lzifj4wss267kwnwic67dw-glu-9.0.0-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/ijk5ygmsdp597a4mp1075gjam1g1clkv-libGLU-combined' from 'https://cache.nixos.org'...
cannot build derivation '/nix/store/3h28bny5yd4q92wvbz2n7gjzwmsgk8wh-chromium-71.0.3559.6.drv': 29 dependencies couldn't be built
cannot build derivation '/nix/store/kql3y0svzjjc83p0vyi1rscz6r345if0-chromium-dev-71.0.3559.6.drv': 6 dependencies couldn't be built
error: build of '/nix/store/kql3y0svzjjc83p0vyi1rscz6r345if0-chromium-dev-71.0.3559.6.drv' failed

Unexpected error: command failed with exit code 1 on aarch64-linux (full log)
Attempted: chromiumBeta
Partial log (click to expand)
copying path '/nix/store/ndl5qb0qgybbsaisknnakb369c2675yz-python3.6-setuptools-40.2.0' from 'https://cache.nixos.org'...
copying path '/nix/store/lhijirbmqi4swgvd1jhpfpja9bamfcd8-ffmpeg-3.4.4-bin' from 'https://cache.nixos.org'...
copying path '/nix/store/9xn2nc7k8ydii627gnrb449dpvbydyds-ffmpeg-3.4.4-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/zdm1fzf7llb9cwf5w507mpmrg2iwxk9j-cups-2.2.6-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/gaq77n95ajv9mddprfs56j867g8nq3p5-python3.6-pyxdg-0.25' from 'https://cache.nixos.org'...
copying path '/nix/store/wjz7hqic9vn4j4zjbh2i95r6lh3q4gp2-speech-dispatcher-0.8.8' from 'https://cache.nixos.org'...
100  594M  100  594M    0     0  21.9M      0  0:00:27  0:00:27 --:--:-- 20.3M
cannot build derivation '/nix/store/dwi25bg986b5j4mgrrlpxmgmw17pm8gc-chromium-70.0.3538.35.drv': 4 dependencies couldn't be built
cannot build derivation '/nix/store/7hlpi9d5qlj8cfb1z90f4z89kl0qbv51-chromium-beta-70.0.3538.35.drv': 3 dependencies couldn't be built
error: build of '/nix/store/7hlpi9d5qlj8cfb1z90f4z89kl0qbv51-chromium-beta-70.0.3538.35.drv' failed

Timed out, unknown build status on x86_64-linux (full log)
Attempted: chromiumDev
Partial log (click to expand)
[903/1124] CXX obj/v8/v8_base/js-collator.o
[904/1124] CXX obj/v8/v8_base/js-date-time-format.o
[905/1124] CXX obj/v8/v8_base/js-list-format.o
[906/1124] CXX obj/v8/v8_base/js-locale.o
[907/1124] CXX obj/v8/v8_base/js-number-format.o
[908/1124] CXX obj/v8/v8_base/js-plural-rules.o
[909/1124] CXX obj/v8/v8_base/js-relative-time-format.o
building of '/nix/store/qwfgif4mp267ipc78vigfj2gms03yz0r-chromium-71.0.3559.6.drv' timed out after 1800 seconds
cannot build derivation '/nix/store/d0yc9bm1l1i4z5gbphpsy1lw05aj64jr-chromium-dev-71.0.3559.6.drv': 1 dependencies couldn't be built
error: build of '/nix/store/d0yc9bm1l1i4z5gbphpsy1lw05aj64jr-chromium-dev-71.0.3559.6.drv' failed

Timed out, unknown build status on x86_64-linux (full log)
Attempted: chromiumBeta
Partial log (click to expand)
[1546/32116] ACTION //extensions/common/api:generated_api_types_schema_generator(//build/toolchain/linux:x64)
[1547/32116] ACTION //extensions/common/api:generated_api_types_schema_generator(//build/toolchain/linux:x64)
[1548/32116] ACTION //extensions/common/api:generated_api_types_schema_generator(//build/toolchain/linux:x64)
[1549/32116] ACTION //extensions/common/api:generated_api_types_schema_generator(//build/toolchain/linux:x64)
[1550/32116] ACTION //extensions/common/api:generated_api_types_schema_generator(//build/toolchain/linux:x64)
[1551/32116] CXX obj/gin/gin/per_context_data.o
[1552/32116] CXX obj/gin/gin/per_isolate_data.o
building of '/nix/store/4vwrdk3ahz3nxnpz56dys80r54561a4w-chromium-70.0.3538.35.drv' timed out after 1800 seconds
cannot build derivation '/nix/store/qc3ka43pk7d7r4izh0a33n113s8w9y31-chromium-beta-70.0.3538.35.drv': 1 dependencies couldn't be built
error: build of '/nix/store/qc3ka43pk7d7r4izh0a33n113s8w9y31-chromium-beta-70.0.3538.35.drv' failed

wip as there is another failure at the end


Comment out console.log statement and add flex justify-center class to div also  Disabled the dropdown when you were on that slide.

Before in large screen

After

Before

After

#109 I solve this issue and also add one feature

Hey how did you resolved the dependency issue?

Hey how did you resolved the dependency issue?

If you are facing issues then just remove the package.lock file and then install the dependency

Sorry seems like there are merge conflicts

@hkirat Resolved conflict

sorry seems like there are more conflicts
Should be an easy fix tho


feat: macros

Create a section under "Advanced Features" that explains how to use available macrosconsteval_int!, array!, and how they work under the hood.
references: https://github.com/starkware-libs/cairo/commit/35953ba63e05c42c547960749a78255f4832cffe
https://github.com/starkware-libs/cairo/blob/main/crates/cairo-lang-plugins/src/plugins/inline_macros
https://doc.rust-lang.org/book/ch19-06-macros.html
est. 1 hour

Can I take this one ser ? 👉👈


Parquet support

Codecov Report

Merging #45 into latest will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##            latest       #45   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           22        24    +2
  Lines         1412      1465   +53
=========================================
+ Hits          1412      1465   +53

Impacted Files
Coverage Δ

tests/test_menu.py
100.00% <ø> (ø)

tests/test_other.py
100.00% <ø> (ø)

tests/test_table.py
100.00% <ø> (ø)

rumydata/__init__.py
100.00% <100.00%> (ø)

rumydata/table.py
100.00% <100.00%> (ø)

tests/conftest.py
100.00% <100.00%> (ø)

tests/test_parquet.py
100.00% <100.00%> (ø)

tests/test_row.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8a41f7c...76c2498. Read the comment docs.


[TieredStorage] enum-based AccountsFileProvider

Problem
Currently, all existing code paths for creating a new writable AccountsFile
are hard-coded to create an AppendVec.  As we introduce a new AccountsFile
format, we will need a more programmatic way to create writable AccountsFiles
using a different format.
Summary of Changes
This PR introduces AccountsFileProvider enum, which allows both AppendVec
and TieredStorage to provide their implementations to create a new AccountsFile.
This will later allow tests and the production code-path to switch between different
formats programmatically.
For existing places where AppendVec is created, this PR simply replaces them by
passing an AccountsFileProvider::AppendVec instance.  Will create follow-up PRs
that change them to share the same instance from AccountsDb.

I think this is probably the best way forward. Testing stuff like this is messy.

Codecov Report
Attention: Patch coverage is 96.15385% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 81.8%. Comparing base (b1e1799) to head (97407ec).
Report is 7 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff            @@
##           master     #457     +/-   ##
=========================================
- Coverage    81.8%    81.8%   -0.1%
=========================================
  Files         842      842
  Lines      228462   228497     +35
=========================================
+ Hits       187084   187105     +21
- Misses      41378    41392     +14


Draft of error handling functionality

I'm opening this so that we can discuss it.
The idea is that one can raise a SATOSAProcessingHaltError from any backend/frontend or microservice and that will cause SATOSA to stop executing and either redirect the user to a given url (external error service)
raise SATOSAProcessingHaltError(redirect_uri="https://this.is.url")

or return a 500 with a message
raise SATOSAProcessingHaltError("This is an error to be shown")

We should probably add support for some rudimentary  templating engine ( mako ) if we want to support local errors ( 5xx, 4xx ) instead of only redirects
Ideas and feedback are welcome

Thanks for the comments @c00kiemon5ter ! Much appreciated. I would welcome some more feedback on the architecture/design/logic if anyone has some


Add TransposeSequence exporter to ONNX-Chainer

This PR adds TransposeSequence exporter to ONNX-Chainer
This exporter supports only limited cases where input or output length is 1.

Jenkins, test this please!

Jenkins, test this please!

Jenkins CI test (for commit 27700612db5e7ef5764d87e238b9e52f35000cf6, target branch master) succeeded!

Jenkins CI test (for commit 27700612db5e7ef5764d87e238b9e52f35000cf6, target branch master) succeeded!


the compression format requested when saving a dataset in json format is not respected

Describe the bug
In the documentation of the to_json method, it is stated in the parameters that

**to_json_kwargs – Parameters to pass to pandas’s pandas.DataFrame.to_json.

however when we pass for example compression="gzip", the saved file is not compressed.
Would you also have expected compression to be applied? :relaxed:
Steps to reproduce the bug
my_dict = {"a": [1, 2, 3], "b": [1, 2, 3]}

Result with datasets
from datasets import Dataset

dataset = Dataset.from_dict(my_dict)
dataset.to_json("dic_with_datasets.jsonl.gz", compression="gzip")
!cat dic_with_datasets.jsonl.gz

output
{"a":1,"b":1}
{"a":2,"b":2}
{"a":3,"b":3}

Note: I would expected to see binary data here
Result with pandas
import pandas as pd

df = pd.DataFrame(my_dict)
df.to_json("dic_with_pandas.jsonl.gz", lines=True, orient="records", compression="gzip")
!cat dic_with_pandas.jsonl.gz

output
4��a�dic_with_pandas.jsonl��VJT�2�QJ��\� ��g��yƵ���������)���

Note: It looks like binary data
Expected results
I would have expected that the saved result with datasets would also be a binary file
Environment info

datasets version: 1.16.1
Platform: Linux-4.18.0-193.70.1.el8_2.x86_64-x86_64-with-glibc2.17
Python version: 3.8.11
PyArrow version: 5.0.0

Thanks for reporting @SaulLu.
At first sight I think the problem is caused because pandas only takes into account the compression parameter if called with a file path (not a file buffer). And in our implementation, we transform the file path to file buffer before calling pandas to_json.
We should fix this.

I was thinking if we can handle the compression parameter by ourselves? Compression types will be similar to what pandas offer. Initially, we can try this with 2-3 compression types and see how good/bad it is? Let me know if it sounds good, I can a raise a PR for this next week

Hi ! Thanks for your help @bhavitvyamalik :)
Maybe let's start with gzip ? I think it's the most common use case, then if we're fine with it we can add other compression methods


git merge conflicts due to python 2 vs 3

We use nbstripout to prevent conflicts due to notebook output, and to prevent output to be commited. Sadly, we still have conflicts when a notebook is commited with one Python version but run with another. Example:
$ git diff
--- a/notebooks/MR/interactive/a_fully_sampled.ipynb
+++ b/notebooks/MR/interactive/a_fully_sampled.ipynb
@@ -340,21 +340,21 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 2",
    "language": "python",
-   "name": "python3"
+   "name": "python2"
   },
   "language_info": {
    "codemirror_mode": {
     "name": "ipython",
-    "version": 3
+    "version": 2
    },
    "file_extension": ".py",
    "mimetype": "text/x-python",
    "name": "python",
    "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.7.1"
+   "pygments_lexer": "ipython2",
+   "version": "2.7.15rc1"
   }
  },
  "nbformat": 4,

looks harmless but
$ git pull
Updating 2e88c7a..471f7f4
error: Your local changes to the following files would be overwritten by merge:
	notebooks/MR/interactive/a_fully_sampled.ipynb
Please commit your changes or stash them before you merge.
Aborting

I actually submitted a PR to nbstripout to address this ages ago

https://github.com/kynan/nbstripout/pull/92

ah. too bad they didn't accept it yet.
I guess for now we'll just have to reopen with python2 and recommit.

no, they accepted it fine. Sorry i seem to have implied it was still open

any suggestions on how to do that (possibly have to remove nbstripout first?)

ah. not yet on pip?

nbstripout>=0.3.4

~/.local/bin/nbstripout  --version
0.3.5

you'll also have to do (at least once):
git config --global filter.nbstripout.extrakeys '
  metadata.celltoolbar metadata.kernel_spec.display_name
  metadata.kernel_spec.name metadata.language_info.codemirror_mode.version
  metadata.language_info.pygments_lexer metadata.language_info.version'

That seems to have worked. thanks!
Could you maybe add this to the instructions here, but also in https://github.com/CCPPETMR/CCPPETMR_VM/blob/master/scripts/UPDATE.sh
or ask for help from someone else...

@paskino FYI I just updated the VM repo

~casperdcl, the files now after #24 still contain kernelspec still giving conflicts unfortunately. feel like correcting it? Could you also say how we correct it?
Does a git pull after correcting it resolve everything, or do we need a new clone?

fixed again. I was filtering out kernel_spec rather than kernelspec.

@KrisThielemans in case you want to know:
pip install -U nbstripout
git config --global filter.nbstripout.extrakeys '
  metadata.celltoolbar metadata.kernelspec.display_name
  metadata.kernelspec.name metadata.language_info.codemirror_mode.version
  metadata.language_info.pygments_lexer metadata.language_info.version'
find SIRF-Exercises -name '*.ipynb' -exec nbstripout \{} \;

thanks Casper. So I guess @johannesmayer will have to run this on #25 before merging? (or we could do it for him). (as the git config isn't a hookon github, but one installed locally).
I'm just not 100% sure what this does if you then do a git commit with nbstripout installed. I'd imagine that it doesn't do anything at all then.

On second thought, I agree that this is uninstall/reinstall is not necessary. As far as I can see, here's how this works:

we do the nbstripout to cut out some fields
we do a commit, which will first do the nbstripout with all the fields configured before passing it on to git

As far as I can see, the first step is only there to make sure that git thinks the file is modified.
So now I think that to be 100% safe, you'd need to add extra arguments to the explicit nbstripout line as well (but I don't know how to do that).

@casperdcl, here's an illustration of my struggles, despite all this.
My local SIRF-Exercises is at ba34dd4af4d0f2e0928bb87e14389b5849364d65. Due to whatever happened before, I have 2 "changed" files
sirfuser@vagrant:~/devel/SIRF-Exercises$ git status
On branch master
Your branch is behind 'origin/master' by 31 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   notebooks/MR/interactive/a_fully_sampled.ipynb
	modified:   notebooks/MR/interactive/d_undersampled_reconstructions.ipynb

no changes added to commit (use "git add" and/or "git commit -a")

However, the changes are "irrelevant":
sirfuser@vagrant:~/devel/SIRF-Exercises$ git diff

(i.e. there's no diff after nbstripout). Nevertheless
sirfuser@vagrant:~/devel/SIRF-Exercises$ git pull
Updating ba34dd4..c13ab98
error: Your local changes to the following files would be overwritten by merge:
	notebooks/MR/interactive/a_fully_sampled.ipynb
	notebooks/MR/interactive/d_undersampled_reconstructions.ipynb
Please commit your changes or stash them before you merge.
Aborting

git reset --hard doesn't help, as the files remain as they were. The only way that I get round this is to junk the repo and clone again.
Any suggestions?

Also, I suspect that @DANAJK's observation that github no longer displays our notebooks nicely is probably because we stripped a bit too much. That's not a show-stopper, but a bit sad.

@KrisThielemans

sounds like you need nbstripout --uninstall before git reset --hard. Maybe should open an issue on the nbstripout repo realted to this?
I don't know if github ever would've displayed those images properly regardless of stripping - would be nice to confirm.

I don't know if github ever would've displayed those images properly regardless of stripping - would be nice to confirm.

sorry. ignore my comment. I thought I saw some notebooks displayed in raw format, but it seems fine now.

summary of status:

we need kernelspec.name and .display_name as otherwise jupyter complains about invalid notebook. If a given name doesn't exist on your system, that's ok as jupyter will try to find a kernel based on kernelspec.language (which is python). However, git will then see changes, and if you commit, it'll create trouble.
git refusing to merge seem almost unavoidable. Scenario:

checkout and run a notebook, which generates some output
git will think the file has changed, even though git diff says it hasn't
git merge will refuse to do anything.

we can git round the git reset --hard problem by adding nbstripout as a smudge filter. No idea what that isn't done by default.

We're therefore still in the case that all commits have to be made via a system that has the same kernelspec. sigh.


Rubinius works not with Rails 5

So let’s just exclude that build combination on Travis.
cc/ @osheroff @jacobat @grosser

👍


[v3 ModRemplacer] set' object has no attribute 'keys'

Other bugs
What were you trying to do?
[p]mute server @anUserAccount without specifying the mute time (expecting it to be infinite).
What actually happened?
On discord channel: Error in command 'mute server'. Check your console or logs for details.
Traceback:
Exception in command 'mute server'
Traceback (most recent call last):
  File "D:\Anaconda3\envs\discordbot\lib\site-packages\discord\ext\commands\core.py", line 61, in wrapped
    ret = await coro(*args, **kwargs)
  File "D:\DEV\Data\cogs\CogManager\cogs\modreplacer\mod.py", line 1062, in guild_mute
    guild_wide=True,
  File "D:\DEV\Data\cogs\CogManager\cogs\modreplacer\mod.py", line 1168, in process_mute
    await self.cancel_tempmute_expires(target, *successful.keys())
AttributeError: 'set' object has no attribute 'keys'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "D:\Anaconda3\envs\discordbot\lib\site-packages\discord\ext\commands\bot.py", line 899, in invoke
    await ctx.command.invoke(ctx)
  File "d:\dev\src\redbot\redbot\core\commands\commands.py", line 409, in invoke
    await super().invoke(ctx)
  File "D:\Anaconda3\envs\discordbot\lib\site-packages\discord\ext\commands\core.py", line 1024, in invoke
    await ctx.invoked_subcommand.invoke(ctx)
  File "D:\Anaconda3\envs\discordbot\lib\site-packages\discord\ext\commands\core.py", line 614, in invoke
    await injected(*ctx.args, **ctx.kwargs)
  File "D:\Anaconda3\envs\discordbot\lib\site-packages\discord\ext\commands\core.py", line 70, in wrapped
    raise CommandInvokeError(e) from e
discord.ext.commands.errors.CommandInvokeError: Command raised an exception: AttributeError: 'set' object has no attribute 'keys'

How can we reproduce this issue?
Install ModRemplacer from git repo V3 branch, and execute command.

:wave: hey, thanks for this. Unfortunately, I'm unlikely to have this fixed soon. I have this particular one hidden from V3's downloader for a reason right now.
I should have time to get this fixed up a bit next week, but it is not a high priority. I'd suggest not using this for now, (and I'll be restructuring the branches next week as well so that the beta test stuff is on another branch, or possibly a different repo entirely).