reduce some TypeInfo bloat

This partially addresses https://issues.dlang.org/show_bug.cgi?id=14758 "TypeInfo causes excessive binary bloat".
Partial in that it only does it for TypeInfo_Struct instances. If it is successful, the same technique can be propagated to the other TypeInfo instances.
The TypeInfo_Struct itself is in a COMDAT, meaning it is only pulled into the executable if it is actually referenced. The trouble is the string name; member - the actual string for the name is stored in global data and is always linked in. The solution in this PR is to append the string data to the TypeInfo_Struct instance as part of the same COMDAT section.

Updated to do the same for enum, function, and delegate typeinfo.

Strings are usually placed into the CONST segment so they are not scanned by the GC and cannot create false pointers. I guess we loose that with placing them into the same COMDAT as the typeinfo. Unfortunately the typeinfo itself cannot be placed into the CONST section because it contains the mutex sometimes used by synchronized(T.classinfo).
Is it possible to create a different COMDAT for the strings that is merged with the CONST segment?

As the bug report suggests, I think the bloat problem is much more serious.
I'm not sure why anything would ever synchronize on TypeInfo, as the rest of it is immutable (and shouldn't need to be scanned). Like the .init data should probably go into an unscanned immutable section.

Why is TypeInfo given such special treatment? The compiler already has all the plumbing needed to parse and interpret D code.  Therefore, it should be able to read the definition from imported folders (druntime in this case), emit compiler errors for any definition that's needed but not found, and emit linker errors if any implementation that is needed but not found.  Constant strings will go to the same section that all other constants strings go to (.rodata in GCC), and because it goes through the same process as all other D code, one should be able to compile with -fdata-sections and link with -gc-sections (at least with GCC) to eliminate the dead name strings.
The current TypeInfo implementation seems pretty janky as it tries to enforce correct TypeInfo validity by comparing struct size.  So as long as you put enough nonsense in the implementation to fulfill the size requirement, it builds and risks runtime errors.
Forgive me if I'm ignorant of this problem's complexity, but isn't it more reasonable to just run TypeInfo though the standard plumbing, getting compile-time enforcement and compatibility with existing dead code elimination techniques and tools?

The struct size check is in case (happens frequently) someone uses a dmd with a mismatched Phobos. In any case, that's not the topic of this PR.

The struct size check is in case (happens frequently) someone uses a dmd with a mismatched Phobos. In any case, that's not the topic of this PR.

Indeed, but the point I was trying to make with my comment is that there seems to be a larger problem with the TypeInfo implementation that, if addressed more completely, would also be a solution to this problem without introducing the problems alluded to by @rainer's comment

Added the same change to emission of TypeInfo for classes. Unfortunately, that is only a step towards solving the problem because TypeInfo for classes are not emitted in COMDATs, and also are referred to by the ModuleInfo, meaning they always get inserted into the executable.

I'm not sure why anything would ever synchronize on TypeInfo

An unfortunate anti-pattern to get a lock per type for free.

I agree it is an anti-pattern, but it's used in druntime/phobos in some places:
https://github.com/D-Programming-Language/druntime/blob/master/src/core/sys/windows/stacktrace.d#L111
https://github.com/D-Programming-Language/phobos/blob/master/std/parallelism.d#L1428
The compiler itself even produces it for this code:
class C
{
	synchronized static test()
	{
	}
}

Indeed, but the point I was trying to make with my comment is that there seems to be a larger problem with the TypeInfo implementation

There is: type info generation is often requested from the glue layer, sometimes without having been analyzed semantically before. But at that stage in the compilation process, no further analysis can be made anymore.

Argh! :-)

Strings are usually placed into the CONST segment so they are not scanned by the GC and cannot create false pointers. I guess we loose that with placing them into the same COMDAT as the typeinfo. Unfortunately the typeinfo itself cannot be placed into the CONST section because it contains the mutex sometimes used by synchronized(T.classinfo).
Is it possible to create a different COMDAT for the strings that is merged with the CONST segment?
Unfortunately I have to bring some bad news here. As soon as the Dll support is done, typeinfos can't be placed into CONST segements anyway. They reference data symbols which might reside in different dlls so they need a data reference patchup which requires them to mutable. So at least for windows placing them into a CONST segment is not an option.

Why do we care so much about binary size? For most use-cases it doesn't matter whether a binary is 10MB or 20MB, and compared with a shared phobos library the numbers are a joke anyhow.
That said it still makes sense to optimize so that linkers can create small static binaries, I just don't see why it's such a high priority (maybe b/c it's one of those simple topics where everyone can afford an opinion).

As soon as the Dll support is done, typeinfos can't be placed into CONST segements anyway.

There is a separate segment for read-only data with relocations on linux, maybe something similar exists on Windows? That way you could make the data read-only after the fixup.

Why do we care so much about binary size?

Probably the same reason why we wash our cars and make our code look nice. It's just embarrassing for D to create gigantic binaries for trivial programs, and doesn't reflect well on us. And, as the bugzilla issue submitter wrote in the n.g., it makes D impossible to use for embedded systems.

I just don't see why it's such a high priority

The high priority is switching to ddmd. However, there's nothing visible happening there, so I'm doing some things that keep getting neglected.

Auto-merge toggled on

see #4914 for the review comments

And, as the bugzilla issue submitter wrote in the n.g., it makes D impossible to use for embedded systems.

You'll never want to use dynamic typeinfo on a small SoC (see https://issues.dlang.org/show_bug.cgi?id=14758#c10).

It's just embarrassing for D to create gigantic binaries for trivial programs, and doesn't reflect well on us.

The std.stdio hello world is 31KB when linked with libphobos2.so, might be time to make that the default on supported platforms.

The std.stdio hello world is 31KB when linked with libphobos2.so, might be time to make that the default on supported platforms.

I guess for shared libraries we'll always have to generate all TypeInfo anyway? And for non-embedded shared libraries should at some point be the default for various reasons (any kind of plugin support, mixing D compilers, ...).

The high priority is switching to ddmd. However, there's nothing visible happening there, so I'm doing some things that keep getting neglected.

That's appreciated, but I thought you wanted to avoid most non-DDMD conversion changes for 2.069 so that 2.068 and 2.069 are mostyl equal regarding features and bugs?

According to http://digger.k3.1azy.net/trend/ this PR increased (!) the size of "Hello world" by 2 kB.

There is a separate segment for read-only data with relocations on linux, maybe something similar exists on Windows? That way you could make the data read-only after the fixup.

Windows does not have a build in relocation mechanism. You have to do it yourself. C++ generates initializers which do the fixup, in D there will be some code in druntime which does the relocations.

As soon as the Dll support is done, typeinfos can't be placed into CONST segements anyway.

TypeInfos could also go into a segment that is still writable, but can be excluded by the GC. Not sure if its worth the trouble, though, because they won't contain a lot of false pointers anyway if additional data like strings and init are placed into CONST.

this PR increased (!) the size of "Hello world" by 2 kB.

That's likely because inlining the string contents means that some pad bytes need adding. When strings are emitted to a separate section, they don't need padding.
The savings from putting class typeinfo into comdats are not there because that hasn't been implemented.

That's appreciated, but I thought you wanted to avoid most non-DDMD conversion changes for 2.069 so that 2.068 and 2.069 are mostyl equal regarding features and bugs?

That was the plan, but the ddmd conversion is stalled.


Allow auth.NewAuthorizerFromEnvironment() to pick up Azure CLI credentials

Feature Request
If we want to use to Azure Go SDK for developer tooling, the user has to jump through some hoops to get authenticated:

When using auth.NewAuthorizerFromEnvironment() there are multiple environment variables that have to be set manually.
When using auth.NewAuthorizerFromFile() the user has to first explicitly create a set of credentials and create an environment variable pointing to the file that contains the credentials.

If we compare this with AWS it's much easier: You simply run aws configure which saves your credentials to ~/.aws/credentials. All tooling can look for this file and use the credentials in this file to call the AWS API.
This command looks a lot like az login from the Azure CLI. It would be great if the following could be accomplished:

The developer simply runs az login.
auth.NewAuthorizerFromEnvironment() (or perhaps a new auth.NewAuthorizerFromCli()) now automatically uses the credentials set by Azure CLI as an authorizer.

Take for example a look at this project: https://github.com/mozilla/sops
It's way more complicated for a developer to configure Azure KeyVault than it is to configure AWS KMS.

The Azure SDK for .NET has implemented this in their AppAuthentication package. Specifically the AzureCliAccessTokenProvider.
Although instead of using a config file, it simply calls Azure CLI itself. Which sounds a bit more brittle, but has been working fine for me.

The Python SDK for Azure has also implemented this: https://docs.microsoft.com/en-us/python/azure/python-sdk-azure-authenticate?view=azure-python#mgmt-auth-cli
It even uses the Azure CLI's config file. They cheated a bit though, because they simply called functions of the Azure CLI code directly, since it's also written in Python.

I've created this issue on the Azure CLI side of things: https://github.com/Azure/azure-cli/issues/7217

@vladbarosan Was this closed because it was implemented? I don't see it and I'm in the same position as @nphmuller.

@mattfarina It's implemented since version 11.1.0 of go-autorest :)
A possible reason you're not seeing it is because the azure sdk for go has not updated its go-autorest dependency to that version yet.

The ‘latest’ branch of the sdk has just been updated. So the next release will probably contain the new auth method.

@nphmuller When I look at https://github.com/Azure/azure-sdk-for-go/blob/f27fc0e7e6fb0b34a2a6229fcd46f36ed471f88f/services/keyvault/auth/auth.go I don't see it. Where should I be looking? You can change the commit id for latest. I just grabbed it so we see the same thing.

@mattfarina When I wrote the response yesterday it was only available here:
https://github.com/Azure/go-autorest/blob/ea233b6412b0421a65dc6160e16c893364664a95/autorest/azure/auth/auth.go
https://github.com/Azure/azure-sdk-for-go/pull/2927 just got merged, so now it's also available in the file you mentioned (latest branch only for now).

@nphmuller thanks.
Because there are now two methods... is there a best way to use the two methods together? Wait for one to return an error and then try the other?

The ones in go-autorest are the default one. It requires the azure resource url to be passed as a parameter. Since for many resources the same url is used (https://management.azure.com/), there’s an overload which uses that url by default.
KeyVault has it’s own resource url. So the KeyVault authorizer contains a couple of methods which call the methods in go-autorest with the KeyVault resource URL.
So for KeyVault both could be used. Although you would have to pass the KeyVault resource URL manually each time if you used the methods in go-autorest. So it's probably easier to use the ones in the KeyVault package.


Embedder-net()

Hi,I have some questions  (this is my graduation project and this is extremly important for me )
-the output of embedder-net() function is a [N,256] I need to understand what is N exactly is it the number of sliding windows (240ms)?
-Can we use this output (embedder-net() function output) for speaker diarization (can we apply clustering algorithms to this sequences for speaker diarization)?
-Can I understand how did you build train-sequence and train-cluster-id (the input of uis rnn ) because my dataset is different from TIMIT-corpus (Timit-corpus is a speaker recognition dataset not a speaker diarization dataset )?
this is a link to the corpus I am using : https://github.com/EMRAI/emrai-synthetic-diarization-corpus
Thank you in advance for your help

N is the number of embeddings.
You can. That is what https://github.com/google/uis-rnn does
(Answering #41 ) The align_embeddings averages the "window level embeddings" into the "segment level d-vectors"

Since this is for your graduation project, I recommend reading https://arxiv.org/pdf/1810.04719.pdf, particular section 2. The dvector_create script just follows what is described in that section.


Suggestion

Hey, how are you?
Did you already think to add a game score functionality in this app?

Hi @AldairAvelino
I hadn't actually thought of that, that would be great!
Thanks

Thanks for replying


TEST-327: Update Jenkins branch

In order to have Nighly run pass Cargotracker we need current state on Jenkins branch.

Do Not Merge during code freeze.


Chain Multiplier and Skill Stack fields

When building a unit and optimizing for a stacking skill (Akstar for AMoE or Mediena for Raging Ice +2, for example), entering a value in the Chain Modifier field causes the Skill Stack field to vanish, although the last value entered into the Skill Stack field is utilized for calculations. Setting the Chain Mod to 0 and moving the focus causes the Skill Stack field to reappear.

Confirmed, still an issue

I dug in the code for a bit and found the following.
The unitStack box is being hidden by
https://github.com/lyrgard/ffbeEquip/blob/61b7ba62a8a16f60374459fe10945da76954e9ba/static/builder.js#L387
Earlier, the 'formula' is defined like this
https://github.com/lyrgard/ffbeEquip/blob/61b7ba62a8a16f60374459fe10945da76954e9ba/static/builder.js#L382-L383
In case of chainMultiplier being set, readSimpleConditions() creates a custom formula object with type '*' and two subvalues.
https://github.com/lyrgard/ffbeEquip/blob/61b7ba62a8a16f60374459fe10945da76954e9ba/static/builder.js#L401-L429
However, hasStack() mentioned above can only process basic formula objects and has no clue what formula type '*' is, and returns false, thus collapsing the input box.
https://github.com/lyrgard/ffbeEquip/blob/61b7ba62a8a16f60374459fe10945da76954e9ba/static/formula.js#L937-L955
This explains why visually the builder UI malfunctions when displaying a stacking skill together with a set chaining modifier. There might also be logical issues coupled with this, like how the stacking value seems to be global and carries over when toggling between characters in a team.


Salief/clean up curation logic interface

https://www.loom.com/share/2abc52ff8c974d4c8f162d333a800ec4

I was conscious of that, but not sure if the approach I just used is redundant. Let me know what you think, and also I think adding the prettier config should be a separate PR.


License correct?

While inspecting the dataset, I found this license: https://creativecommons.org/licenses/by-nc/4.0/legalcode.
This conflicts with the license in the metadata, which is CC BY. I would also suggest the last one, not the CC BY-NC.
Why did we choose for the CC BY-NC?
See also https://github.com/riparias/pov-fishes-occurrences/issues/29

I asked via mail to @filipmahieu

@filipmahieu answer:

We nemen de CC-BY-licentie. Deze link meen ik. https://creativecommons.org/licenses/by/4.0/legalcode.nl

New run of the DwC mapping has solved this issue.


Implement UptimeDots

This adds a couple of things, the main thing is the UptimeDots with all
the behavior that we want it to have.
Besides that, it implements the MSW handles for the different monitors.
@matsimitsu I tried the "unit testing" approach that we also spoke about with the candidates in this PR. This way testing becomes actually fun again :-)

Merging without reviews due to lack of reviewers working.


[DSIP-46] Unify the logic task params data structure between front-end and back-end

Purpose of the pull request
close https://github.com/apache/dolphinscheduler/issues/16153
Brief change log

Remove extra parse logic parameter code in task instance, use task channel to parse task params
Remove some duplicate code of task type

Verify this pull request

This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

(or)
Pull Request Notice
Pull Request Notice
If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

Codecov Report
Attention: Patch coverage is 9.73783% with 241 lines in your changes missing coverage. Please review.

Project coverage is 41.21%.

Files
Patch %
Lines

...master/runner/task/switchtask/SwitchLogicTask.java
0.00%
37 Missing :warning:

...ache/dolphinscheduler/service/utils/DagHelper.java
54.54%
8 Missing and 7 partials :warning:

...ster/runner/task/condition/ConditionLogicTask.java
0.00%
13 Missing :warning:

...k/dependent/DependentAsyncTaskExecuteFunction.java
0.00%
8 Missing :warning:

...r/plugin/task/api/parameters/SwitchParameters.java
0.00%
8 Missing :warning:

...inscheduler/plugin/task/api/TaskPluginManager.java
0.00%
7 Missing :warning:

...api/service/impl/ProcessDefinitionServiceImpl.java
14.28%
6 Missing :warning:

.../server/master/runner/WorkflowExecuteRunnable.java
0.00%
6 Missing :warning:

...subworkflow/SubWorkflowLogicTaskPluginFactory.java
0.00%
6 Missing :warning:

.../task/switchtask/SwitchLogicTaskPluginFactory.java
0.00%
6 Missing :warning:

... and 67 more

Additional details and impacted files
@@             Coverage Diff              @@
##                dev   #16193      +/-   ##
============================================
+ Coverage     40.99%   41.21%   +0.21%
+ Complexity     5147     5126      -21
============================================
  Files          1399     1403       +4
  Lines         44877    44455     -422
  Branches       4759     4712      -47
============================================
- Hits          18399    18322      -77
+ Misses        24670    24327     -343
+ Partials       1808     1806       -2

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


setup electron notifications

Title: [Persona name] should (not) be able to [action]
Currently:
[What happens now in the regression]
Expected:
[What the correct behavior should be]
STEPS TO REPRODUCE:
[If needed, add steps to reproduce]
REFERENCE:
[Link or attach screenshot(s) to story if relevant]
Possible fixes:
(If you can, link to the line of code that might be responsible for the problem)
Related issues

Related Merge Requests
https://github.com/particle4dev/dicoapp-e/pull/48
/label ~bug

Moved it here:
https://github.com/KomodoPlatform/dicoapp-e/issues/14


Selecting scheme to edit from browse page is broken if authority name is wrong

Do you mean the "Edit scheme locally" button? Yeah, that's expected, and is why we have https://github.com/acteng/atip/blob/main/src/scripts/assign_authority.ts previously and now the new atip-scheme-data one. What file + wrong authority did you hit?

So this was using the legacy all_schemes_data.geojson, or rather one derived from it, so it's not surprising there was a wrongly named authority. I wonder if we can use the logic we use elsewhere to automatically select an authority if the name is wrong


SemanticClassificationType should not be internal

This is needed downstream for ionide and FSAC.
Already merged to fsharp/FSharp.Compiler.Service (see https://github.com/fsharp/FSharp.Compiler.Service/pull/696)

@forki At the moment all these types are internal (or should be) in FSharp.LanguageService.Compiler (the name given to the internal Microsoft compilation of the compiler service component in the Visual F# Tools distribution of F#).  FSC is not an official API in the Microsoft shipped DLLs, and it's Microsoft policy to make everything internal that's not part of an official API. The internal attributes are one of the remaining sources of divergence between FCS and this repo
There should actually be tests that check that all types are internal this so I don't know why those wren't failing.  Maybe they have been disabled somewhere along the way.
It is possible that we will decide to break Microsoft policy and allow public types in these APIs.  Ot maybe we should use a whole swag of #if in order to dual-compile, which would probably allow us to bring the last remaining diffs across from FCS - @brettfo @KevinRansom @cartermp @Pilchie may have opinions which way we should go.
Basically the tension is between two states of existence:

FCS as it is today:  a fast-and-loose no-guarantees community API/nuget package, with frequent major version revisions and no real binary-compat story.
FCS as a Microsoft component, possibly with a binary compat story.

cheers
don

@dsyme regarding the policy, can this be bypassed by simply stating that FSharp.LanguageService.Compiler assembly as a whole is not by any mean a public / supported API? This can also be stated in the assembly description.
Having the code totally align on this with FCS one will for sure make things easier in the frequent merges you have to do.
Currently, it is difficult to contribute code which:

can be built as part of VisualStudio.sln (depends on FSharp.LanguageService.Compiler)
doesn't depend on VisualStudio assemblies (not put in current FSharp.Editor project)
depends on FCS when built for other tooling

Such code would make a spot for all F# tools to leverage, we still don't have that and the community is copy/pasting features among the various tools.
I see opportunity in having assembly which can be compiled with either FSharp.LanguageService.Compiler or FCS and believe such assembly should be maintained in this repository (so the features can be integrated to VS).


Missed five Math.Round() overloads

http://forums.bridge.net/forum/general/feature-requests/721
FYI: they are already in v2.

The following methods added into Math class:
        public static double JsRound(double d);
        public static double Round(double d, int digits);
        public static double Round(double d, MidpointRounding method);
        public static double Round(double d, int digits, MidpointRounding method);

        public static double IEEERemainder(double x, double y);
        public static double Pow(int x, int y)

        public static decimal Round(decimal d, int digits);
        public static decimal Round(decimal d, int digits, MidpointRounding method);

        public static int DivRem(int a, int b, out int result);
        public static long DivRem(long a, long b, out long result);


Custom tile source to prioritize tiles rendering using rectangles

I have an CustomViewer that has a CustomTileSource which accepts the hight and width as the level 0 dimensions (highest resolution):
export class CustomTileSource extends OpenSeadragon.TileSource {
  /**
   * Create a new CustomTileSource.
   *
   */
  constructor(port: number, width: number, height: number) { // width and height from backend
    super({
      width: width,
      height: height,
      tileSize: 254,
      minLevel: 0,
      maxLevel: 15,
    });
  }
  override getTileUrl(level: number, x: number, y: number) {
    // async call to Rust backend for rendering tile
    return `${this.renderingApiHost}:${this.port}/tile/${level}/${x}/${y}`;
  }
}

Everything works okay, but as I read the logs of my backend I see that openseadragon is attempting to load many background images. In my images after opening I have an array of Rect that points me directly to my area of interest.
Any idea on how I can use this information to prioritize the calls inside my rect bounds?

OSD already prioritizes the most important tiles. First it loads the highest resolution tile that fits the whole image, so the user has an overview of the whole image, even if it's a little blurry. Then it loads the next highest resolution tiles for what is visible on the screen, starting with whichever tile is closest to the center of the viewer. It continues loading higher resolution tiles until it's got full resolution for the current view. If you then move to a different part of the image, it starts the process over again, not loading any tiles that have already been loaded.
If you want to control what gets loaded, make sure that area is the only thing visible in the viewer. Beyond that, there isn't really a way to control what tiles it loads in what order. If you have a rectangle for your area of interest, you can zoom in on it by calling viewer.viewport.fitBounds with it.


Doughnut Chart

As Chart JS Integration’s user,
I need to have Doughnut chart
So that I can have Doughnut chart view

Done in v.0.0.11


[Event Request] CU10861 "Payment-Apply" - local procedure Apply()

Describe the request
Add a event in CU10861 "Payment-Apply" - local procedure Apply()
Additional context
local procedure Apply(var PaymentLine: Record "Payment Line")
.....
case AccType of
AccType::Customer:
begin
end;
AccType::Vendor:
begin
end;
else
----------------------- ADD EVENT ----------------------
IsHandled := false;
OnElseAccTypeOnApply(var PaymentLine, var GenJnlLine, IsHandled)
if not IsHandled then
Error(
Text005,
GenJnlLine.FieldCaption("Account Type"), GenJnlLine.FieldCaption("Bal. Account Type"));
Internal work item: AB#551202

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.
Build ID: 25286.


Fix clearing input filter on editRecipePage

On the editRecipePage, when filtering inputs, one can now clear the filter by clicking the X icon or Clear All Filters link. Also, when navigating away from the editRecipePage, the filter resets. This covers issue #139

The fix is verified and all tests are passed in my local env.
The Travis test will be re-runed when #202 got merged.

@jkozol, This PR has code conflict with master after #197 got merged. Could you please fix the conflict? Thanks.

Codecov Report

Merging #201 into master will decrease coverage by 0.06%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master     #201      +/-   ##
==========================================
- Coverage    43.4%   43.33%   -0.07%
==========================================
  Files          70       70
  Lines        2341     2340       -1
  Branches      319      319
==========================================
- Hits         1016     1014       -2
- Misses       1112     1113       +1
  Partials      213      213

Impacted Files
Coverage Δ

core/reducers/inputs.js
81.81% <ø> (+6.81%)
:arrow_up:

core/actions/inputs.js
100% <ø> (ø)
:arrow_up:

pages/blueprintEdit/index.js
52.86% <0%> (-0.44%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fc7fb25...6e8629c. Read the comment docs.

@jgiardino, I'll merge this PR if you have no comment and concern. Thanks.

@henrywang Thanks for reviewing this.

These updates work as expected. Thanks for fixing this!


Gather a Crowd: The result didn't match with hint and content

Describe the bug
Gather a Crowd: The result didn't match with hint and content
Steps to reproduce the behavior
1.Navigate to  https://arcade.makecode.com/--skillmap#turkey.
2.Gather a Crowd->Start.
3.Execute to step5, compare the result and hint & content.
Actual behavior:
The result shows checkered square, hint and content show grey.
Expected behavior:
The hint and content need to be updated that match with the result.
Screenshot:

Additional context:
1.OS: Windows 11
2.arcade version: 1.13.50
3.Microsoft MakeCode version: 9.3.6

This bug also reproduced on below steps:
Make the Turkey skillmap step9 in https://arcade.makecode.com/--skillmap#turkey.
Gather a Crowd skillmap step3, step7, step8, step10 in https://arcade.makecode.com/--skillmap#turkey.
Win or Lose skillmap step3 in https://arcade.makecode.com/--skillmap#turkey.


Cannot read property 'users' of undefined

In line number 69, we have the following line of code,
msg.reactions.cache.get('<a:tick:791208682607870003>').users.remove(client.user.id)
This is the error message
Cannot read property 'users' of undefined
The emojis are the only change I made in the code. Is there a bug here? How can I solve it?

Yes, the emoji seems to be the problem here, I'll look into it right now.

Okay, here's the mistake: You can't just get emojis like that, if you want to use a custom emoji you have to define it first and then put the definition into the code.
For example:
const tick_emoji = client.emojis.cache.get('emoji-id')
await msg.react(tick_emoji)
msg.reactions.cache.get(tick_emoji)
The emoji-id only needs the number for example: ~<a:tick:~791208682607870003~>~


Added tecmilenio.mx

Added "Universidad Tecmilenio D.R. Enseñanza e Investigación Superior" to the repository...

@jorge-marquez-Tecmilenio I am sorry, the domain tecmilenio.mx is removed from our trusted domains list because it was abused in the past. If you are the student of this school, please enroll to GitHub Student Developer Pack at https://education.github.com/discount_requests/application. Then apply for JetBrains educational subscription with your GitHub account (GitHub tab at https://www.jetbrains.com/shop/eform/students).


Separate constraints file download breaks Docker image builds (IDFGH-7722)

Environment

Development Kit:      all of em
Module or chip used:  all of em
IDF version (run git describe --tags to find it): v5.0-dev-1426-ge899edd793
Build System:        idf.py
Compiler version (run xtensa-esp32-elf-gcc --version to find it): xtensa-esp32-elf-gcc (crosstool-NG esp-2021r2) 8.4.0
Operating System:     Linux
Using an IDE?: no
Power Supply:         Unlimited power

Problem Description
The commit here https://github.com/espressif/esp-idf/commit/b28d7e6850936a9e5bbf542fa1d55adf6f9ae2f3 caused esp-idf to start downloading a constraints file from the internet on every install / export.
This eventually broke my Docker build container which is a custom build of esp-adf that works around some odd issues and contains a specific checkout of esp-idf at the aforementioned e899edd793. (the build fails on recent masters but at the time of e899edd793 a bug was fixed that hadn't been backported to 4.4 at the time I made the container, so it was deemed necessary to venture into unstable territory)
At the time I built the container, everything was working. At some point between then and now, the constraints file changed. (somewhere between June 6-9 by the looks of where our job logs started all failing)
Now, all builds fail because the new constraints file is not satisfied. Rebuilding the Docker container should work, but that means the Dockerfile is not reproducible. It depends on a specific point in time and not just a specific git version.
I see a comment in the commit:
# We need to constrain package dependencies as well. So all installed packages need to be checked.
# For example package A requires package B. We have only A in our requirements. But the newest version of B could
# broke at some time and in that case we add a constraint for B (on the server) but don't have to update the
# requirement file (in the ESP-IDF repo).

I appreciate the desire to not have to do a whole commit-release cycle to fix something like that. Ironically enough, the fact that this file changed on the server is the only thing that actually broke something, at least so far, for me. Granted, this is an unreleased, unstable branch as of yet. But even so, if it is discovered that a newer version of a sub-dependency package does break something, I think it's best for it to be committed in the repo so that the same git checkout will build the same build environment every time. Ideally, dependencies are locked in at some way. I am not sure what sort of blackbox dependency hell auto update torture prompted this change, but the cure feels worse than the disease. Unfortunately the alternative does mean, in the event it happens, new checkout/installs of old versions that lack the necessary upper limits on dependency versions will of course break.
Expected Behavior
Reproducible builds
Actual Behavior
Build tool downloads an external file which is not a part of the repo which can break things when it changes outside of version control
Steps to reproduce

go back in time to before https://dl.espressif.com/dl/esp-idf/espidf.constraints.v5.0.txt required idf-component-manager~=1.1
make a Docker image for an esp-idf build container based on any checkout from b28d7e6850936a9e5bbf542fa1d55adf6f9ae2f3 or newer. i.e. https://github.com/espressif/esp-idf/blob/master/tools/docker/Dockerfile with a specific IDF_CHECKOUT_REF
go back forward in time
try to build using the Docker image produced in step 2

CI Job Log
...
Downloading espidf.constraints.v5.0.txt to /root/.espressif/espidf.constraints.v5.0.txt.tmp
Done
The following Python requirements are not satisfied:
idf-component-manager~=1.1
To install the missing packages, please run "install.sh"
Diagnostic information:
    IDF_PYTHON_ENV_PATH: /opt/esp/python_env/idf5.0_py3.8_env
    Python interpreter used: /opt/esp/python_env/idf5.0_py3.8_env/bin/python
Constraint file: /root/.espressif/espidf.constraints.v5.0.txt
Requirement files:
 - /opt/esp/adf/esp-idf/requirements.core.txt
Python being checked: /opt/esp/python_env/idf5.0_py3.8_env/bin/python
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

Perhaps something like pip-tools can help. It bakes the requirements.txt file similar to how a Gemfile.lock works for ruby gems if I understand correctly
https://github.com/jazzband/pip-tools
Or just an option to opt into the newer constraints file

First of all, we are sorry for the inconvenience. The master branch currently is the work-in-progress v5.0 which is a major release and will contain breaking changes in comparison with v4.4.x. You cannot except to get the same results for all commits between the top of the v4.4 and the master branch. Once v5.0 will be released, the constraint file will freeze and will be modified only to solve potential bugs.
Back to your problem. I can help to work around the issue. I assume that you want to keep using install.sh and export.sh. (At the lower level script there are options to avoid the constraint file.) For the use case with install.sh and export.sh you can use the following.
You can modify your /root/.espressif/espidf.constraints.v5.0.txt file with requirements which suits your needs (compatibility with older master commits).  After that, you can add touch /root/.espressif/espidf.constraints.v5.0.txt to the beginning of your docker file before calling install.sh or export.sh. That will avoid the download and overwite with a newer version from the server.
I hope this helps.

Thanks for your attention to this issue. Sorry to trouble you.
I know that 5.0 is not released yet, but when it is, the same problem will apply. Everybody building with containers will run into it (built themselves or downloaded from the espressif docker hub)--eventually the image they're using will become out of sync with the constraints file on the server and builds will fail.
I should mention I did expect to be traveling some bumpy roads going off of the stable branch, and I usually don't bother anyone about what I run into since it's in development, and that's how it goes. This is a little different than the usual bleeding-edge branch excitement, if it's how it's going to be done once it is stable.

Once v5.0 will be released, the constraint file will freeze and will be modified only to solve potential bugs.

Will it freeze, or will it be modified to solve potential bugs? Those are two different things.

If the constraints file for that version ever does change (even to solve potential bugs), I suspect the build will still fail, the installed versions will be different than the ones stated in the constraints file downloaded at the time of building the image
If the file will not change once released, why not commit to version control along with the rest of it?

The constraints format seems prone to causing confusion. It certainly confused me when I ran into it this month. Maybe that's just the covid that's taking a while to wear off.
I still think if you could try pip-tools it might be a better way. They solve this for you by pinning the versions. You don't automatically get new features or fixes from future dependency updates, but that's the only downside.  (Probably not a common problem, just a bonus you get from the additional check)
It would generate something else in the future when idf component manager 1.1.4 is released, or urllib3 releases 1.26.10, etc.
See also the first paragraph of jwhitlock's answer here https://stackoverflow.com/a/36848206/2619499
Otherwise, I understand and will work around it. (In that case, I think your proposed workaround could be included in the official Espressif Dockerfile to prevent future Docker builds from running into this)
I would be interested to know more about the option to skip the constraints file. Is there any chance the option could be made available to export.sh? Perhaps that is what belongs in the Docker image now.
I assume it would be modifying this line in export.sh...
-python "${IDF_PATH}/tools/idf_tools.py" check-python-dependencies || return 1
+python "${IDF_PATH}/tools/idf_tools.py" check-python-dependencies --no-constraints || return 1

or some way that passes it along as a command line switch (does export.sh have any other switches? could just pass the args straight through if not, then . path/to/export.sh --no-constraints just works)


Cannot be used on version 0.7 of Julia

I first installed version 1.4 of Julia, and everything worked well, and I used version 901 of Gurobi. at the same time.
But for some reason, I have to use the old version of Julia,I usePkg.add ("Gurobi")everything was OK during installation, but the following error occurred when I entered"using Gurobi :
ERROR: LoadError: Gurobi not properly installed. Please run Pkg.build("Gurobi")
Stacktrace:
 [1] error(::String) at .\error.jl:33
 [2] top-level scope at C:\Users\24829\.julia\packages\Gurobi\dlJep\src\Gurobi.jl:8
 [3] include at .\boot.jl:317 [inlined]
 [4] include_relative(::Module, ::String) at .\loading.jl:1038
 [5] include(::Module, ::String) at .\sysimg.jl:29
 [6] top-level scope at none:2
 [7] eval at .\boot.jl:319 [inlined]
 [8] eval(::Expr) at .\client.jl:399
 [9] top-level scope at .\none:3
in expression starting at C:\Users\24829\.julia\packages\Gurobi\dlJep\src\Gurobi.jl:5
ERROR: Failed to precompile Gurobi [2e9cd046-0924-5485-92f1-d5272153d98b] to C:\Users\24829\.julia\compiled\v0.7\Gurobi\do9v6.ji.
Stacktrace:
 [1] error(::String) at .\error.jl:33
 [2] macro expansion at .\logging.jl:313 [inlined]
 [3] compilecache(::Base.PkgId, ::String) at .\loading.jl:1185
 [4] _require(::Base.PkgId) at .\logging.jl:311
 [5] require(::Base.PkgId) at .\loading.jl:852
 [6] macro expansion at .\logging.jl:311 [inlined]
 [7] require(::Module, ::Symbol) at .\loading.jl:834

In addition, when I use Pkg.build("Gurobi"), the error is as follows:
  Building Gurobi → `C:\Users\24829\.julia\packages\Gurobi\dlJep\deps\build.log`
┌ Error: Error building `Gurobi`:
│ ERROR: LoadError: Unable to locate Gurobi installation. Note that this must be downloaded separately from gurobi.com
│ Stacktrace:
│  [1] error(::String) at .\error.jl:33
│  [2] top-level scope at C:\Users\24829\.julia\packages\Gurobi\dlJep\deps\build.jl:62
│  [3] include at .\boot.jl:317 [inlined]
│  [4] include_relative(::Module, ::String) at .\loading.jl:1038
│  [5] include(::Module, ::String) at .\sysimg.jl:29
│  [6] include(::String) at .\client.jl:398
│  [7] top-level scope at none:0
│ in expression starting at C:\Users\24829\.julia\packages\Gurobi\dlJep\deps\build.jl:61
└ @ Pkg.Operations C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v0.7\Pkg\src\Operations.jl:1068

What caused this error, or what should I do to use Gurobi. on version 0.7 of Julia.

You will need to install an older version of Gurobi like 8.0. You cannot use Gurobi 9.0 with Julia 0.7.

您将需要安装旧版本的Gurobi，例如8.0。您不能将Gurobi 9.0与Julia 0.7一起使用。

Thank you. Installing the old version of gurobi solved my problem.


Can I support the code snippet?

@jueyoq, How do you want to support the code snippet?

Provides syntax highlighting for md。

I'm sorry I still don't understand what you mean exactly. Could you explain in more than one sentence please.


Provide functions to parse numbers from string_views

Signed-off-by: Henner Zeller<EMAIL_ADDRESS>

the normal std::strtol(), std::strtod(), etc require nul-terminated strings, but with string-views we don't have that.
These will provide the necessary functions to that we can easily parse these from string-views.

I also noticed that NumUtils was entirely untested. The newly added NumUtils_test now just covers the new parsing functions but I'll add a follup change to test the previously existing functions.

Surelog and UHDM are separate projecs, and we should not use private implementations from each other (in particular, UHDM should not provide a public API for number parsing). So I am fine if we just copy them verbatim into each projects.
If we have a common enough code-base we should probably have a a separate utiltiy library.


CudaTensor.index causing buggy state in CUDA 3.5?

I'm using an older GPU (Tesla K20, cuda compute capability 3.5) and getting a very strange error, demonstrated below. Note that the command t[1] = t[1] will execute initially, but after line 4 the system enters an odd state where this command causes an 'invalid device function' error. The command runs fine again after the first failed attempt.
th> require 'cutorch'
                                                                      [4.1952s]
th> t = torch.CudaTensor(1,2)
                                                                      [0.0001s]
th> t[1] = t[1]
                                                                      [0.0002s]
th> torch.CudaTensor(5):index(1, torch.LongTensor{2,3})
 0
 0
[torch.CudaTensor of size 2]

                                                                      [0.0008s]
th> t[1] = t[1]
THCudaCheck FAIL file=/tmp/luarocks_cutorch-scm-1-926/cutorch/lib/THC/THCTensorCopy.cu line=205 error=8 : invalid device function
cuda runtime error (8) : invalid device function at /tmp/luarocks_cutorch-scm-1-926/cutorch/lib/THC/THCTensorCopy.cu:205
stack traceback:
	[C]: at 0x2b82041886e0
	[C]: in function '__newindex'
	[string "t[1] = t[1]"]:1: in main chunk
	[C]: in function 'xpcall'
	/home/lbh/torch/install/share/lua/5.1/trepl/init.lua:670: in function 'repl'
	.../lbh/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:199: in main chunk
	[C]: at 0x004064f0
                                                                      [0.0004s]
th> t[1] = t[1]
                                                                      [0.0001s]

did you compile cutorch on your own machine?
"invalid device function" points to where you might have built cutorch on a machine with a different GPU capability and copied over

Ah, had compiled on a cluster machine with different specs. Just recompiled and that fixed it. Thank you for being so responsive!

@soumith The field of deep learning should hold an annual celebration in honour of your huge contribution.


Please consider fixing issues detected by repository checker

Notification from ioBroker Check and Service Bot
Dear adapter developer,
I'm the ioBroker Check and Service Bot. I'm an automated tool processing routine tasks for the ioBroker infrastructure. I have recently checked the repository for your adapter hiob for common errors and appropiate suggestions to keep this adapter up to date.
Please see the result of the check below.
ioBroker.hiob
   -

ERRORS:

[ ] :heavy_exclamation_mark: [E204] Version "0.1.0" listed at common.news at io-package.json does not exist at NPM. Please remove from news section.

WARNINGS:

[ ] :eyes: [W034] @iobroker/adapter-core 3.1.4 specified. 3.1.6 is recommended. Please consider updating dependencies at package.json
[ ] :eyes: [W040] "keywords" within package.json should contain "ioBroker"
[ ] :eyes: [W162] js-controller 5.0.0 listed as dependency but 5.0.19 is recommended. Please consider updating dependency at io-package.json.
[ ] :eyes: [W184] "common.main" is deprecated and ignored. Please remove from io-package.json. Use "main" at package.json instead.

Please review issues reported and consider fixing them as soon as appropiate.
Errors reported by repository checker should be fixed as soon as possible. Some of them require a new release to be considered as fixed. Please note that errors reported by checker might be considered as blocking point for future updates at stable repository.
Warnings reported by repository checker should be reviewed. While some warnings can be considered as a suggestion and be ignored due to good reasons or a dedicated decision of the developer, most warnings should be fixed as soon as appropiate.
Feel free to contact me (@iobroker-bot) if you have any questions or feel that an issue is incorrectly flagged.
And THANKS A LOT for maintaining this adapter from me and all users.
Let's work together for the best user experience.
your
ioBroker Check and Service Bot
@mcm1957 for evidence

This issue has been replaced by new isse #86
This issue can be closed.
your
ioBroker Check and Service Bot


[5.7][docs] improvements for assertionFailure and preconditionFailure

This is a cherry-pick of https://github.com/apple/swift/pull/59330

the previous docs for assertionFailure implies that it is a nonreturning function, which has always been wrong.
the example usage previously given for assertionFailure was moved to preconditionFailure, where it is more appropriate.

Resolves rdar://94831099 (originally FB10032237)

@swift-ci please test


Use pkg_resources.parse_version() for handling version numbers

The reason for this PR is to demonstrate a fix for the problem identified in https://github.com/projectatomic/osbs-client/pull/250#discussion_r40073359.
Does it make sense to do it this way?

I like this.

Re-based against master.


Able to create table in Simulator but unable to create table in Actual device( iOS 13.3 )

TestProject.zip

Please ask on StackOverflow for this kind of questions (we are tracking bugs on GitHub)


Upgrade socket2 dependency

I made minimal changes to make it compile with the new socket2 release, plus an extra tweak to replace socket_type with a SOCKET_TYPE constant.

Looks good. Thanks!

Released as 0.5.1. Thanks again!


Prepare 0.1.1 release

A release with the signed integer implementations would be helpful for https://github.com/nrf-rs/nrf-hal/pull/209

Thanks for fast response guys, much appreciated! 🥇


wip! use tap.spawn() for each test

Another suggestion for #80. My "crazier idea" as mentioned in #83:

My even crazier idea was to change the runner a little bit. Still one test file, but it starts each test in a child process and communicates the results. That means any shared setup code is still executed for each test without having to be duplicated.

[x] some test failures, maybe due to #79
[x] master has 21 tests, this has 17, so some are not being executed

@jamestalmage @bcoe this works a bit better now. Tests pass without tap --coverage but ironically they now fail with tap --coverage. Perhaps #89 is useful there.
I had to fix some tests, these fixes seem to be generally applicable if I'm understanding everything correctly.
If you run with --reporter=spec you'll see that some describe() groups are output as tests in master. The nesting is better in this branch. Perhaps a tap bug? Regardless that explains the number of tests being different in master.
Note that I needed #88 for my tests to pass. I was getting an error in lazy-cache. Not sure why that is.

I really think https://github.com/jamestalmage/nyc/commit/ad9a7baa7e5077bfe1070376468c50b956870390 is the best strategy for providing our own coverage. Between require hooks and spawn-wrap I think we will continue to run the risk of false positive tests like #86. I could pull that out of PR #89 if we decide this is the forking strategy to go with.
As for this forking strategy vs what I am doing in #89, there are some tradeoffs:
Things this does better:

Prints prettier
Does not generate a ton of files
Does not require a build script (probably runs quite a bit faster)

Things that are better about #89

If a single test is giving you a problem, you can run just that test by calling the generated file. It's almost like only support in TAP. Which is something I have missed from mocha.

You can look inside each generated file for a clearer understanding of what is happening

tap's mocha globals option doesn't give you beforeEach/afterEach, but with #89 you can haz.

describe('foo', function () {
  rimraf.sync('foo') // this will actually behave like a beforeEach for this describe block

  it('foo-test-1', function() {
    // ...
  })

  it('foo-test-2', function () {
     // ...
  })

  assert(fs.readDirSync('foo').length === x) // an afterEach assertion - applicable only to the `foo` describe block
})

describe('bar', function() {
  // The "beforeEach" / "afterEach" from the `foo` block will not be run in this fork
  // My understanding of this PR is that they would be
})

Where they both suck:

They hack the tap testing framework in ways that may be surprising to new collaborators. That said, due to the nature of NYC, I think we are stuck some variation of one of these.


Show similar collectives

Alice visits MochaJS and scans through the page. At the bottom of the page, she should be able to see 3 other related collectives (other open source collectives) with a link to more.
Jon donates to Yeoman and on the thank you page, he sees three related collectives that he can donate to with a link to more.
It's the same component that can be reused. We can use the Collective Cards that were built for the homepage.

Here: https://github.com/OpenCollective/opencollective-website/pull/357


purity issue in Fs2RabbitConfig

Hi,
the Documentation suggests using PureConfig to load the configuration. However there doesn't seem to be a pure, referentially transparent way to create the SSLContext that is needed for that; all the static factory methods to create an SSLContext throw exceptions or return null. Users are thus likely to end up with code like
  implicit val sslContextReader: ConfigReader[SSLContext] =
    ConfigReader[String].map(SSLContext.getInstance)

which performs side effects in an uncontrolled manner. Would you agree that this is an issue? There's probably a number of ways to tackle this issue, but before discussing them I'd first like to get a consensus that in an environment where side effects are generally visible in the types this is not an ideal behaviour.

Hi @mberndt123 ,
Probably adding a note on the documentation would suffice. pureconfig is still the recommended way for reading configuration values but since the creation of SSLContext is side-effectful is recommended to fill that value once your config is filled with the pure configuration values such as host, port, etc.
This is purposely left on he user's hand because I believe that's the best option. The user is free to choose how that Fs2RabbitConfig is filled with values and passed to the creation of the Fs2Rabbit interpreter.

This is one way to do it:
IO(SSLContext.getInstance("SSL").some).handleError(_ => none[SSLContext]).map { ctx =>
  val config = Fs2RabbitConfig(
    virtualHost = "/",
    host = "<IP_ADDRESS>",
    username = Some("guest"),
    password = Some("guest"),
    port = 5672,
    ssl = false,
    sslContext = ctx,
    connectionTimeout = 3,
    requeueOnNack = false
  )
  Fs2Rabbit[IO](config)
}.flatMap { implicit fs2Rabbit =>
  ResilientStream
    .run(new AckerConsumerDemo[IO]().program)
    .as(ExitCode.Success)
}

Feel free to send in a PR if you would like to document such use case.

pureconfig is still the recommended way for reading configuration values but since the creation of SSLContext is side-effectful is recommended to fill that value once your config is filled with the pure configuration values such as host, port, etc.

How exactly would that work? After all the whole point of pureconfig is that I don't need to worry about parsing the configuration object itself. I only need to provide ConfigReader instances for the individual fields and pureconfig will figure out how to parse the object for me. But in this case that requires a ConfigReader[SSLContext], and there's no way to do that in a pure way. So frankly I can't see how to idiomatically parse an Fs2RabbitConfig using pureconfig. Your example code works around this issue by not using pureconfig at all.

One way would be to sidestep the issue entirely by removing sslContext from Fs2RabbitConfig entirely and pass it as a separate argument to Fs2Rabbit.apply. Of course the downside here is that you can't configure the SSL Context from the configuration at all (or you have to do it manually).
Another solution would be to add an F[_] type parameter to Fs2RabbitConfig and change the type of sslContext to Option[F[SSLContext]]. Then you can do something like
implicit def sslContextReader[F[_]: Sync] =
  ConfigReader[String].map(s => Sync[F].delay(SSLContext.getInstance(s)))

This way everything is pure and very flexible, since the user is free to create the SSL Context in any way they see fit based on the configuration. Of course it's a bit unusual to have F things in the configuration…

Thanks for the input @mberndt123 , I'll cut a release with these changes later today. Just out of curiosity are you using fs2-rabbit in production?

Awesome, make sure to add it to the list of Adopters ;)


dummy thread to hold images

Using technique from here to keep the repo nice and lightweight.

inception avg pool


No transcript_info table with pipeline_genesets

pipeline_annotations used to make a table called transcript_info https://github.com/CGATOxford/CGATPipelines/blob/94b975256fa5142f15e82511f4b62a6c19ae38b9/obsolete/pipeline_annotations.py#L1049-L1054 which is used in multiple pipeline, for example in a tracker for rnaseqdiffexpression https://github.com/CGATOxford/CGATPipelines/blob/94b975256fa5142f15e82511f4b62a6c19ae38b9/CGATPipelines/pipeline_docs/pipeline_rnaseqdiffexpression/trackers/Genelists.py#L5-L30.
The config.ini for pipeline_genesets still contains an option for this table https://github.com/CGATOxford/CGATPipelines/blob/8ebe37408aa512ec910b767e07ade4d4b1733177/CGATPipelines/pipeline_genesets/pipeline.ini#L318 but the pipeline doesn't use this option and doesn't generate the table. This breaks at least one of my pipelines which integrates with pipeline_annotations/genesets. Is there any reason pipeline_genesets doesn't generate this table?

When developing the geneses pipeline the intention was to initially support the bare minimum needed for the CGAT-flow pipelines to work. It was removed because it wasn't used in pipelines and CGATreports isn't supported going forward. If you think you need it then feel free to add it back in. I am using the new cgat-developers code now though.

OK, I'll move this comment to cgat-developers


Add a panel that shows what 3rd party crates are active

Add a panel to the playground that shows what crates are available and what version it is.

This icon looks like it would suit.

Could use smol-toml to parse the manifests on the repo via the GitHub api.


fix(p2p): periodically re-dial to disconnected bootstrap nodes

Summary of changes

This PR allows a forest node to re-dial to disconnected bootstrap nodes when discovery is disabled or not functioning in some rare conditions. (And if a forest node does not publicly expose its p2p port, other peers cannot initialize the connection either)
Changes introduced in this pull request:

periodically re-dial to disconnected bootstrap nodes
do not ban bootstrap nodes

Manually tested on calibnet

run forest on calibnet
run FULLNODE_API_INFO="$(cat /tmp/forest_token):/ip4/<IP_ADDRESS>/tcp/2345/http" forest-cli net disconnec t 12D3KooWEiBN8jBX8EBoM3M47pVRLRWV812gDRUJhMxgyVkUoR48
see log 2024-02-27T13:40:09.741739Z  INFO forest_filecoin::libp2p::service: Re-dialing to bootstrap peer at /dns4/bootstrap-calibnet-2.chainsafe-fil.io/tcp/34000/p2p/12D3KooWEiBN8jBX8EBoM3M47pVRLRWV812gDRUJhMxgyVkUoR48

Reference issue to close (if applicable)

Closes
Other information and links

Change checklist

[x] I have performed a self-review of my own code,
[x] I have made corresponding changes to the documentation,
[x] I have added tests that prove my fix is effective or that my feature works (if possible),
[x] I have made sure the CHANGELOG is up-to-date. All user-facing changes should be reflected in this document.

How about dialing the bootstrap nodes only if the node is not connected to any peers at all?

@LesnyRumcajs There's an edge case that, when all the connected nodes are stateless, a stateful node still cannot sync

How about dialing the bootstrap nodes only if the node is not connected to any peers at all?

@LesnyRumcajs There's an edge case that, when all the connected nodes are stateless(e.g. crawler) or out of sync by thousands of epochs, a stateful node still cannot sync

Aren't stateless nodes able to point to relevant peers? To my understanding, please correct me if I'm wrong, the bootstrap nodes (Forest ones) are stateless anyway.

Aren't stateless nodes able to point to relevant peers?

@LesnyRumcajs They are, but if discovery is disabled and p2p port is not publicly exported, connections to other nodes still cannot be established


[SPARK-34897][SQL][3.0] Support reconcile schemas based on index after nested column pruning

This PR backports https://github.com/apache/spark/pull/31993 to branch-3.0. The origin PR description:
What changes were proposed in this pull request?
It will remove StructField when pruning nested columns. For example:
spark.sql(
  """
    |CREATE TABLE t1 (
    |  _col0 INT,
    |  _col1 STRING,
    |  _col2 STRUCT<c1: STRING, c2: STRING, c3: STRING, c4: BIGINT>)
    |USING ORC
    |""".stripMargin)

spark.sql("INSERT INTO t1 values(1, '2', struct('a', 'b', 'c', 10L))")

spark.sql("SELECT _col0, _col2.c1 FROM t1").show

Before this pr. The returned schema is: `_col0` INT,`_col2` STRUCT<`c1`: STRING> add it will throw exception:
java.lang.AssertionError: assertion failed: The given data schema struct<_col0:int,_col2:struct<c1:string>> has less fields than the actual ORC physical schema, no idea which columns were dropped, fail to read.
	at scala.Predef$.assert(Predef.scala:223)
	at org.apache.spark.sql.execution.datasources.orc.OrcUtils$.requestedColumnIds(OrcUtils.scala:160)

After this pr. The returned schema is: `_col0` INT,`_col1` STRING,`_col2` STRUCT<`c1`: STRING>.
The finally schema is `_col0` INT,`_col2` STRUCT<`c1`: STRING> after the complete column pruning:
https://github.com/apache/spark/blob/7a5647a93aaea9d1d78d9262e24fc8c010db04d0/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala#L208-L213
https://github.com/apache/spark/blob/e64eb75aede71a5403a4d4436e63b1fcfdeca14d/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala#L96-L97
Why are the changes needed?
Fix bug.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Unit test.

Test build #137852 has started for PR 32310 at commit b54bb45.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42382/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42382/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42382/

Test build #137852 has finished for PR 32310 at commit b54bb45.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137852/

Merged to branch-3.0


add balsamiq

name: Balsamiq Wireframes
about: PR is filled with _vendor yaml. I'm happy to adjust this to the $49 tier being the 'base' if we see fit.

What is the vendor's name?
Balsamiq Wireframes
What is the vendor's pricing site?
https://balsamiq.com/buy/#cloud
What is the base pricing? Use the lowest tier that looks sane for a small business customer, not free or personal tiers.
$9 / 2 projects / month
$49 / 20 projects / month
What is the minimum pricing for SSO support?
$199 / 200 projects /month
Does this pricing info come from a quote or other non-public source?
N/A
Are there any caveats we should list in the footnotes?
There are slight discounts for annual pricing. In our personal use of Balsamiq, we do not need more than 1 'project' of wireframes, though I could see a larger operation than mine using the $49 tier.

Pricing is still the same as when this was submitted, though I did bump their base price up to the $49 tier since the cheaper tier looks explicitly like an eval / hobby license. Thanks.


fix: Fixed an error when executing the command "npm run watch"

fix: Fixed an error when executing the command "npm run watch"

@zhaokun1219 thx!


Android version

Hey guys, first of all thanks for your contribution, this is a brilliant peace of code and very usefull for RN comunity.
While iOS version looks complete, the lack of Android is very frustrating.
Do you have any plans to support Android and if so, when do you think it will be available for beta testing?

Hi @veedeo, we're working on the Android version and it will be available soon. stay tuned!

Any updates? 😄

Hi @lidanh thanks for the work on this!
I saw the android branch, anything you need help with?

Do you plan on supporting notification templates in Android?

Hey, sorry for the delayed response, was on vacation.
The plan is to fully support android notifications, same as iOS.  it takes time, but we're on it.
I'll check the status of the android version and keep you updated here.
Thanks,
Lidan

I'm happy to announce that thanks to @amitdwix, we've added Android support!
Currently it's still basic support (receive remove notifications), but we're on it and new features will be released soon.   You're more than welcome to help us and PR!
FYI- @dininski @nicolabortignon @andreasbhansen @veedeo @oriharel


Jobs with same group are being split

I have a long read assembly workflow designed to work with tibanna for AWS execution. As part of this, I'm trying to use groups to intelligently split jobs among compute nodes. This mostly works? But for some reason, I'm getting more groups than I would expect based on the snakemake documentation. For reference, here is a dry run that demonstrates what I'm seeing...
$ snakemake -j 1 -p --tibanna --dryrun \
>     salk-tm-dev/assembly_test/ecoli.flye.fasta \
>     salk-tm-dev/assembly_test/ecoli.miniasm.fasta \
>     --default-remote-prefix salk-tm-dev/assembly_test/ \
>     --config reads=SRR11460449.fastq.gz basename=ecoli size=500m
Building DAG of jobs...
Job counts:
        count   jobs
        1       flye
        1       miniasm
        2
[Mon Aug 31 18:19:12 2020]

group job assembly (jobs in lexicogr. order):

    [Mon Aug 31 18:19:12 2020]
    rule flye:
        input: salk-tm-dev/assembly_test/SRR11460449.fastq.gz
        output: salk-tm-dev/assembly_test/ecoli.flye.fasta, salk-tm-dev/assembly_test/ecoli.flye.gfa
        jobid: 1
        threads: 32
        resources: mem_mb=240000, disk_mb=1000

        flye --asm-coverage 30 --nano-raw salk-tm-dev/assembly_test/SRR11460449.fastq.gz --genome-size 500m --threads 32 --out-dir ecoli.flye
        mv ecoli.flye/assembly.fasta salk-tm-dev/assembly_test/ecoli.flye.fasta
        mv ecoli.flye/assembly_graph.gfa salk-tm-dev/assembly_test/ecoli.flye.gfa

[Mon Aug 31 18:19:12 2020]

group job assembly (jobs in lexicogr. order):

    [Mon Aug 31 18:19:13 2020]
    rule miniasm:
        input: salk-tm-dev/assembly_test/SRR11460449.fastq.gz
        output: salk-tm-dev/assembly_test/ecoli.miniasm.gfa, salk-tm-dev/assembly_test/ecoli.miniasm.fasta
        jobid: 0
        threads: 32
        resources: mem_mb=240000, disk_mb=1000

        minimap2  -x ava-ont -t 32 salk-tm-dev/assembly_test/SRR11460449.fastq.gz salk-tm-dev/assembly_test/SRR11460449.fastq.gz > overlaps.paf
        miniasm  -f salk-tm-dev/assembly_test/SRR11460449.fastq.gz overlaps.paf > salk-tm-dev/assembly_test/ecoli.miniasm.gfa
        awk '/^S/{print ">"$2"\n"$3}' salk-tm-dev/assembly_test/ecoli.miniasm.gfa | fold > salk-tm-dev/assembly_test/ecoli.miniasm.fasta

Job counts:
        count   jobs
        1       flye
        1       miniasm
        2
This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.

Notice that there are two groups both named "assembly". Any ideas what is going on here? Is this somehow expected behavior?
EDIT : Came up with a simpler example. Used to be about duplicate "polish groups"

I think what is happening is that the group logic only works properly when the jobs are connected directly. If the jobs aren't connected in the dag, then the group tag is basically ignored.

The issue with my proposed approach is the following case. Imagine a dag with four groups and two tags...
Group_A_1 -> Group_B_1
Group_B_2 -> Group_A_2
...Where A and B are the tag associated with the group, 1 and 2 are just numbers to note that these are seperate groups, and the arrows denote rule dependencies. Group_A_1 and Group_A_2 are in parallel as are Group_B_1 and Group_B_2, but merging either  of these groups causes the other to no longer be parallel which means we would have to make a decision about which parallel groups to merge. This seems inelegant at the very least.
I suppose current behavior is at least a reasonable place to be.

New alternative approach. We could attempt to merge all jobs with the same group tag and then check for circular dependencies. If any are detected, we fall back to current behavior. So if you had a graph like in my previous post (type 1), we would attempt to collapse it to...
Group_A -> Group_B -> Group_A
... This is obviously circular so we revert to current behavior. But if we had a graph (type 2) that looked like...
Group_A_1 -> Group_B_1
Group_A_2 -> Group_B_2
...We can collapse it to...
Group_A -> Group_B
...which doesn't have any circular dependencies so we get to just run in two groups.
Pros : I think this works more intuitively in typical cases. Graphs of type 2 are probably vastly more common than graphs of type 1. This is a computationally cheap sollution O(E + V)
Cons : This makes the code base more complicated, makes explaining behavior more complicated, and probably makes the CLI more complicated as we would want an option to skip this behavior, which would be desirable in some cases.

I'm going to close this issue as it is incorrectly labelled and open a new enhancement issue in which I propose my second sollution.


Add inclusion of the astro.config.mjs file to Manual Setup

Several errors relating to the renderers appear after following the Manual Setup instructions in the docs: https://docs.astro.build/en/guides/manual-setup/. This happens with the latest astro version v0.23.7.
These errors go away by adding the astro.config.mjs file to the root of the project and simply adding the renderers: [], line into the configuration.  This step is currently missing from the manual setup instructions.

Thanks for reporting, and I believe we've got this covered in docs now! (Of course, it's now going to be integrations: [], but we've got that in documentation ready to go for the next release!)


Sec-Token-Binding-Context header is not defined in draft-ietf-tokbind-ttrp-01

@zandbelt First, thanks for your wonderful work. I'm reaching out to you for a clarification over the Sec-Token-Binding-Context header that is sent to the back-end servers by mod_token_binding.
draft-ietf-tokbind-ttrp-01 only describe Sec-Provided-Token-Binding-ID and Sec-Referred-Token-Binding-ID headers.
I did come across draft-campbell-tokbind-tls-term-00 that describe about a header named Token-Binding-Context. It seemed like this header refers to that one.
May be you kept this header for backward-compatibility reasons? I do not see any issues keeping it, it just confuses us the first time. :) I'm simply looking for a clarification.
Please find a httpbin output retrieved via the HTTPD server run via your docker image.
{
  "headers": {
    "Accept": "text/html",
    "Connection": "close",
    "Host": "httpbin.org",
    "Sec-Provided-Token-Binding-Id": "AgBBQBTIBpQkkzv8aCwcfYNeELX8viGOw0rRlcFqhB_MvI_JP_6JzTdR2fKAb1HfDO3vSpwlMA2rjN8hjhf99IKs0As",
    "Sec-Token-Binding-Context": "AA0CQlNMtfvYwjx1Qv6Hzssmqo61-b6wcRt3ZzRt0IyKsn0",
    "Upgrade-Insecure-Requests": "1",
    "User-Agent": "Chrome/60.0",
    "X-Forwarded-Host": "localhost:4433",
    "X-Forwarded-Server": "www.zmartzone.eu"
  }
}

Thanks,
KasunG

Yes, it is for "backwards compatibility" with a feature that could also be useful in certain environments but has been taken out of the more recent draft.

That is good to know. I see that it has been added to README few days ago. I missed that.
Btw, would it be better to provide a configuration option to decide which headers we need to send to the backend server? I'm asking because Token-Binding-Context is apparently only need to be added to the requests if it was explicitly configured to do so -- as per draft-campbell-tokbind-tls-term-00#section-2.
   Reverse proxies MUST only add the "Token-Binding-Context" header when
   explicitly configured to do so and MUST only dispatch requests
   containing it to trusted backend servers.  ...

Thanks.

Fair point, though that draft is obsolete now and installing/configuring this module could be seen as the "explicit configuration" that is referred to. But it does make sense to send only what is explicitly required. FWIW: I'll take a PR any time ;-)

Cool. I was reading a bit on whether there is a rule for Sec-Provided-Token-Binding-Id header as well. draft-ietf-tokbind-ttrp-01 section-3 says that producing these headers SHOULD be an configuration option. Default should be to not use the headers.
I'm interested in providing a PR. But, I need your input on how the configuration in httpd.conf should look like. Do you have any suggestions?
I fiddled with nginx token binding module a bit. This is their way of configuring it which uses custom variables.
upstream example.com {
   ip_hash;
   server localhost:9443;
   server localhost:9444;
}

server {
   listen 443;
   server_name example;
   token_binding on;

   location / {
      proxy_pass https://example.com;
      proxy_http_version 1.1;

      proxy_set_header Sec-Provided-Token-Binding-Id $provided_token_binding_id;
      proxy_set_header Sec-Referred-Token-Binding-Id $referred_token_binding_id;
   }

   ssl on;
   ssl_certificate /usr/local/nginx/ssl/nginx.crt;
   ssl_certificate_key /usr/local/nginx/ssl/nginx.key;

   access_log /var/log/nginx/access.log;
}

Thanks.

I'd suggest:
TokenBindingPassVariable [provided|referred|context|all]+

It turned out as:
TokenBindingPassVar [provided|referred|context]+

with the default being all of them.


Bug 1920159: kube-apiservers overstate steady-state slightly

The kube-apiserver is one of our most dynamic scaling components,
but the majority of clusters have a base load that is fairly
predictable of around 300 millicore. In order to get more headroom
for smaller clusters, adjust the out of the box request to a more
accurate default. All of the sidecars were measured in heavily
loaded production clusters as taking less than 5 millicore each,
and the core kube-apiserver is tweaked slightly.  CPU is burstable
so the apiserver will grow as necessary into slack capacity, while
still yielding to etcd on saturated masters.
/hold
Testing multiple changes together

/bugzilla refresh

/bugzilla refresh

/retest
Not related to resource reduction.
/hold cancel
Combined tests looked ok.  Team should review and see the comments in the bug.

/retest
Not related to resource reduction.
/hold cancel
Combined tests looked ok.  Team should review and see the comments in the bug.

/retest

/retest

/lgtm
/approve
/retest


Getting cell values is returning different format - OpenXML

Describe the bug
Getting cell value returns many decimal places.
Screenshots
I have this "4.9" value in excel

When retrieving the cell value in OpenXML. 4.9 was stored as 4.90000000000000000000004. It should return value as it was inputted.

To Reproduce
// Please add a self-contained, minimum viable repro of the issue.
// If you require external resources, please provide a gist or GitHub repro
// An Xunit style test is preferred, but a console application would work too.
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Observed behavior
A clear and concise description of what you expected to happen.
Expected behavior
A clear and concise description of what you expected to happen.
Desktop (please complete the following information):

OS: [e.g. Windows]
Office version [e.g. 16.0.15427.20178]
.NET Target: (e.g. .NET Framework, .NET Core, UWP, Xamarin...)
DocumentFormat.OpenXml Version: (e.g. 2.11.0)

Additional context
Add any other context about the problem here.

Can you share how you're doing this? Sounds like this is just an artifact of using a floating point number.

This looks like something excel is adding and the SDK is surfacing that correctly. @tomjebo @mikeebowen please correct me, but I'm assuming this is because you mark it as number and general number must be using a float which are not able to accurately represent numbers. Could the s="2" value be the significant digits?

Yes, the number in my case "4.9" which is a general number resulted to accurately represent numbers. which is unwanted.
Could the s="2" value be the significant digits?
Answer: I tried changing the value of AK3 cell on excel from 4.9 to string like "SAMPLE" and is in General format, the value of s remains "2". Refer to screenshot below:

So it is not a significant basis. t="s" refers to SharedString value I presume.
Please let me know if there will be some workaround on this issue. Thanks

@tomjebo @mikeebowen thoughts as to what excel does here and if we can replicate it in the sdk?

The s attribute is the index of the cell's style.

I think the SDK is doing the correct thing here. Excel stores numbers using double-precision, so the value the SDK retrieves is correct. If the values stored by Excel in the underlying xml need to match what is displayed in the UI, users can set the precision as displayed option in Excel. With that option set Excel will store the exact decimal places showing in the UI. Note, this isn't retroactive, so if you set the "precision as displayed" option on a spreadsheet, new numbers will match the decimal places in the UI, but values entered before setting the option will remain double-precision.

How does excel know when to show  4.9 or 4.90000000000000000000004?

@gartenkralle By the number formatting.
More:
Excel uses the s attribute in the cell element in the sheet.xml part:
        <row r="7" spans="1:7" x14ac:dyDescent="0.25">
            <c r="A7" s="1">
                <v>3.05</v>
            </c>
        </row>

In this case the following s attribute is "1":
 <c r="A7" s="1">

That corresponds to the second (zero based 1)  cellXfs entry in the styles.xml part:
    <cellXfs count="2">
        <xf numFmtId="0" fontId="0" fillId="0" borderId="0" xfId="0"/>
        <xf numFmtId="164" fontId="0" fillId="0" borderId="0" xfId="0" applyNumberFormat="1"/>
    </cellXfs>

which specifies the attribute value numFmtId="164" and  applyNumberFormat="1" (meaning to apply the number format of 164). That identifies the format in numFmts:
    <numFmts count="1">
        <numFmt numFmtId="164" formatCode="0.000"/>
    </numFmts>

Therefore displaying the value:

I also have relevant concern in date formatting
Observed behavior
Custom Date format and other date format returns inconsistent NumberFormatId
Please refer to the attached file 1.xlsx

I have two custom date formats mm/dd/yyyy and mm-dd-yyyy. Checking it on XML it has numFmtId "164" and numFmtID "165" respectively.

I have also this "2.xlsx" attached file which returns a different number format Ids for the same custom date format in "1.xlsx" file

Checking in on XML , it has numFmts "165" and "166" for mm/dd/yyyy & mm-dd-yyyy respectively

I have also this "3.xlsx" attached file which returns a different range of number format ids as well.

Checking it on XML , it has numFmtId = 171 & numFmtId = 172 for mm/dd/yyyy & mm-dd-yyyy respectively

This also happens to other date formats as well not limited to custom date formats. We still don't know what is the basis of what numFmtId is assigned to particular date format.
How can we identify if a cell value in excel has date format (mm-dd-yyyy) or (mm/dd/yyyy) etc. ??
What we are trying to accomplish here is to check whether a cell value has (mm-dd-yyyy) or (mm/dd/yyyy) . Since NumFmtId assignment is not consistent.
Please see all files for your reference. Thank you! Hope to hear from you soon
1.xlsx
2.xlsx
3.xlsx

In addition, Is it possible to convert a date in c# using these formatCodes?

Hi @kev-ops,

"We still don't know what is the basis of what numFmtId is assigned to particular date format. / How can we identify if a cell value in excel has date format (mm-dd-yyyy) or (mm/dd/yyyy) etc. ?"

There is no way to find this with the Open XML SDK without examining the individual numFmt elements. You will have to examine the numFmt elements until you find the one you are searching for and then look at its numFmtId to find the ID value.
More here on numFmt 2.1.712 Part 1 Section 18.8.30, numFmt (Number Format)
And here 2.1.713 Part 1 Section 18.8.31, numFmts (Number Formats)

"Is it possible to convert a date in c# using these formatCodes? We are struggling converting the date values retrieved from excel in c# using the format codes in the screenshot."

Yes, you can use the DateTime Struct
More on date formatting here Standard date and time format strings

I hope that helps.

I encountered the same problem but without style formatting. I just created a new sheet entered 147.61 to the first cell. I changed no style.
This was saved:
<c r="B3"><v>147.61000000000001</v></c>
And this is also what I get from CellValue.Text.
What is the recommended way to original the value?

Can you please confirm that you used Excel when entering 147.61 initially? And what's your code when using CellValue.Text using the SDK?

Can you please confirm that you used Excel when entering 147.61 initially?

yes, this is easy to reproduce:

here the created sheet: Mappe1.xlsx

And what's your code when using CellValue.Text using the SDK?

Here the reduced code:
string cellValue = cell.CellValue?.Text; // is '147.61000000000001'
if (cellValue == null)
    return null;
if (cell.DataType is { Value: var value }) // is null!
{ ... }
if (cell.StyleIndex != null) // is null!
{ ... }
return cellValue

cell.OuterXML is <x:c r="A1" xmlns:x="http://schemas.openxmlformats.org/spreadsheetml/2006/main"><x:v>147.61000000000001</x:v></x:c> and that matches exactly the Mappe1.xlsx.

@greuelpirat, this is the expected behavior, it is a result of how Excel stores numbers not the SDK. See this post above: https://github.com/dotnet/Open-XML-SDK/issues/1413#issuecomment-1520954222

@greuelpirat, this is the expected behavior, it is a result of how Excel stores numbers not the SDK. See this post above: #1413 (comment)

The option Precision as displayed does not work here. The value is still saved with the appendix. For me it looks like the option only applies to calculated cells. but not for inputs by the user since the option is the section When calculating this workbook
Anyway, I personally do not agree that this is expected behaviour (I definitely did not expect this). Anyone that comes across this issue will have to implement their own solution for this.

I agree with @greuelpirat. If it is expected behavior and you set the option precision as displayed. Then it's a bug!

Hi @hakito and @greuelpirat, the extra decimals you see are a result of how Excel stores and calculates floating-point numbers, they can be confusing, but they are by design. More details on that how Excel handles floating-point numbers here: Floating-point arithmetic may give inaccurate results in Excel.
There is no bug with the OpenXML SDK here. The SDK is correctly reading the values that Excel has stored. If you believe this is a bug in Excel, you can submit feedback on the Excel Feedback Portal or submit it to the Microsoft Feedback Hub, instructions on how to submit here.


@threaded macro

Basic attempt at providing a @threaded [kwargs] for ... API. The idea is to just transform the code into tforeach or tmapreduce calls.
Basic examples:
julia> @threaded for i in 1:3
           println(i)
       end
2
1
3

julia> @threaded scheduler=StaticScheduler() for i in 1:3
           println(i)
       end
1
3
2

julia> @threaded reduce=(+) for i in 1:3
           sin(i)
       end
1.8918884196934453

I also took a stab at adding a @tasklocal macro that can be used within the loop-body and essentially expands to the common TaskLocalValue pattern.
Example @tasklocal:
(with 8 threads)
julia> @threaded scheduler=StaticScheduler() for j in 1:16
           @tasklocal C::Matrix{Int64} = fill(Threads.threadid(), 2,2)
           println(taskid(), " -> ", C, " (", pointer_from_objref(C),")")
       end
17449308883177734487 -> [11 11; 11 11] (Ptr{Nothing} @0x00001468adc04730)
7858258358232135215 -> [10 10; 10 10] (Ptr{Nothing} @0x00001468adfba410)
17449308883177734487 -> [11 11; 11 11] (Ptr{Nothing} @0x00001468adc04730)
7858258358232135215 -> [10 10; 10 10] (Ptr{Nothing} @0x00001468adfba410)
14141364989831668232 -> [9 9; 9 9] (Ptr{Nothing} @0x00001468ae528070)
731939243900798756 -> [4 4; 4 4] (Ptr{Nothing} @0x00001468b41d9630)
12158275713140852179 -> [5 5; 5 5] (Ptr{Nothing} @0x00001468ae634070)
731939243900798756 -> [4 4; 4 4] (Ptr{Nothing} @0x00001468b41d9630)
12158275713140852179 -> [5 5; 5 5] (Ptr{Nothing} @0x00001468ae634070)
14141364989831668232 -> [9 9; 9 9] (Ptr{Nothing} @0x00001468ae528070)
9445684018239188219 -> [6 6; 6 6] (Ptr{Nothing} @0x0000146b1052c190)
9445684018239188219 -> [6 6; 6 6] (Ptr{Nothing} @0x0000146b1052c190)
12917150328223514965 -> [7 7; 7 7] (Ptr{Nothing} @0x00001468ae4d5510)
12917150328223514965 -> [7 7; 7 7] (Ptr{Nothing} @0x00001468ae4d5510)
1380909043146770272 -> [8 8; 8 8] (Ptr{Nothing} @0x00001468ae500190)
1380909043146770272 -> [8 8; 8 8] (Ptr{Nothing} @0x00001468ae500190)

This expands to
julia> @macroexpand @threaded scheduler=StaticScheduler() for j in 1:16
           @tasklocal C::Matrix{Int64} = fill(Threads.threadid(), 2,2)
           println(taskid(), " -> ", C, " (", pointer_from_objref(C),")")
       end
quote
    #= /scratch/pc2-mitarbeiter/bauerc/devel/OhMyThreads.jl/src/macro.jl:74 =#
    begin
        #= /scratch/pc2-mitarbeiter/bauerc/devel/OhMyThreads.jl/src/macro.jl:61 =#
        var"##242" = OhMyThreads.TaskLocalValue{Matrix{Int64}}((()->begin
                        #= /scratch/pc2-mitarbeiter/bauerc/devel/OhMyThreads.jl/src/macro.jl:61 =#
                        fill(Threads.threadid(), 2, 2)
                    end))
    end
    #= /scratch/pc2-mitarbeiter/bauerc/devel/OhMyThreads.jl/src/macro.jl:75 =#
    OhMyThreads.tforeach(1:16; scheduler = StaticScheduler()) do j
        #= /scratch/pc2-mitarbeiter/bauerc/devel/OhMyThreads.jl/src/macro.jl:76 =#
        begin
            #= /scratch/pc2-mitarbeiter/bauerc/devel/OhMyThreads.jl/src/macro.jl:64 =#
            C = var"##242"[]
        end
        #= /scratch/pc2-mitarbeiter/bauerc/devel/OhMyThreads.jl/src/macro.jl:77 =#
        begin
            #= REPL[44]:2 =#
            #= REPL[44]:3 =#
            println(taskid(), " -> ", C, " (", pointer_from_objref(C), ")")
            #= REPL[44]:4 =#
        end
    end
end

TODOs

[ ] Do we want to have something like @tasklocal?
[ ] Currently, the task local storage (gensym()'ed variable name) is available after the loop. Could put everything into a let block? But the user could be interested in this after loop. Maybe let them choose a name?
[ ] Support multiple @tasklocal variables, either as multiple @tasklocal statements or via begin end.
[ ] Bikeshedding, see below.

Closes #49

@MasonProtter, would be great if you could take a look. I'm not that well versed in writing macros :)

Bikeshedding:

@threaded indicates multithreading but doesn't indicate Julia's task-based parallelism.
Brainstorming: @taskparallel, @paralleltasks, @threadedtasks, @parallel

One thought I had was trying to copy Chapel's forall syntax. So something like
@forall i in 1:10 begin
    println(i)
end

but it kinda sucks that we have to write the begin. Another thought I had was using a do block, but that's not allowed (ref https://github.com/JuliaLang/JuliaSyntax.jl/issues/414).
It kinda makes me think though we could just do this without a macro at all though:
forall(1:10) do i
    println(i)
end

and
forall(1:10; reducer=(+)) do i
    println(i)
end

but there's various things like the task local storage that might benefit from having a macro available.

It kinda makes me think though we could just do this without a macro at all though:
forall(1:10) do i
    println(i)
end

That's already available, just forall → foreach?
foreach(1:10) do i
	println(i)
end

but there's various things like the task local storage that might benefit from having a macro available.

This. Also, whether we like it or not, people are (much) more used to writing
for i in 1:10
	# do something
end

than
foreach(1:10) do i
	# do something
end

I always have to explain the do syntax thoroughly in my course whereas the for loop is trivial for everyone.

Added support for multiple task local variables
@threaded scheduler=StaticScheduler() for j in 1:16
    @tasklocal begin
        C::Matrix{Int64} = fill(Threads.threadid(), 2,2)
        D::Matrix{Float64} = rand(3,3)
    end
    sleep(0.01*j)
    println(Threads.threadid(), " (", pointer_from_objref(C),") (", pointer_from_objref(D),")")
end

To avoid "leaking" of the task local storages, we now wrap everything in a let block. For example, the above expands to this
let var"##290" = OhMyThreads.TaskLocalValue{Matrix{Int64}}((()->begin
                      fill(Threads.threadid(), 2, 2)
                  end)), var"##291" = OhMyThreads.TaskLocalValue{Matrix{Float64}}((()->begin
                      rand(3, 3)
                  end))
      begin
          OhMyThreads.tforeach(1:16; scheduler = StaticScheduler()) do j
              begin
                  C = var"##290"[]
                  D = var"##291"[]
              end
              begin
                  sleep(0.01j)
                  println(Threads.threadid(), " (", pointer_from_objref(C), ") (", pointer_from_objref(D), ")")
              end
          end
      end
  end

Note to future self: It needs to be clearly documented that assignments within a @tasklocal block need to be self-contained, e.g. they can't reference each other.

Another thought I just had: I kind of dislike that all the keyword arguments (e.g. scheduler or reducer) are all in a single line and push back the actual iteration. Compare
for i in 1:16

and, e.g.,
@threaded scheduler=DynamicScheduler(; nchunks=8, split=:scatter) reducer=(+) for j in 1:16

I was wondering whether we should instead use a separate block for all settings, e.g.,
@threaded for j in 1:16
    @settings begin
        scheduler=StaticScheduler()
        reducer=(+)
    end
    println(j)
end

Another (wild) thought I just had: I kind of dislike that all the keyword arguments (e.g. scheduler or reducer) are all in a single line and push back the actual iteration. Compare
for i in 1:16

and, e.g.,
@threaded scheduler=DynamicScheduler(; nchunks=8, split=:scatter) reducer=(+) for j in 1:16

I was wondering whether we should instead use a separate block for all settings, e.g.,
@threaded for j in 1:16
    @settings begin
        scheduler=DynamicScheduler(; nchunks=8, split=:scatter)
        reducer=(+)
    end
    println(j)
end

or, equivalently,
@threaded for j in 1:16
    println(j)

    @settings begin
        scheduler=DynamicScheduler(; nchunks=8, split=:scatter)
        reducer=(+)
    end
end

I like this idea a lot!

Some bike shedding updates. I think I dislike @threaded because 1) it is very similar to @threads and 2) it is focused on threads not tasks. My current favourites are

@tasks (I like that it's simple and puts a clear emphasis on tasks.)
@parallel
@omt (for OhMyThreads)

I also don't think that @tasklocal is the best choice because regular variables defined in the parallel for loop are also local to the task. The crux really is the "once per task" nature. My current favourites are

@init (It's simple and gets the point across, I think)
@tlv (for TaskLocalValue)

For now, I'll go with / change to @tasks and @init.

These all work now:
@tasks for i in 1:10
    @set scheduler=StaticScheduler()
    println(i)
end

@tasks for i in 1:10
    @set begin
        scheduler=DynamicScheduler(; nchunks=4)
        reducer=(+)
    end
    sin(i)
end

@tasks for i in 1:10
    @set scheduler=DynamicScheduler(; nchunks=4)
    @set reducer=(+)
    sin(i)
end

@tasks for i in 1:10
    @init begin
        C::Matrix{Int64} = fill(Threads.threadid(), 2,2)
        D::Matrix{Float64} = rand(3,3)
    end
    println("asd")
end

@tasks scheduler=DynamicScheduler(; nchunks=4) for i in 1:10
    @set scheduler=StaticScheduler() # takes precedence
    println(i)
end

One thought I had was when we construct the inner function to be passed to tmapreduce or tforeach, it'd be cool if we could make a way so that that function doesn't need to access the task_local_storage during the sequential part of their loop.
I think the way this would have to work though would be to have something like
struct WithTaskLocalValue{F, TLVs <: Tuple{Vararg{TaskLocalValue}}}
    func::F
    tlvs::TLVs
end
initialize(f::WithTaskLocalValue) = f.func(f.tlvs...)
initialize(f::Any) = f

and then
# DynamicScheduler: AbstractArray/Generic
function _tmapreduce(f,
        op,
        Arrs,
        ::Type{OutputType},
        scheduler::DynamicScheduler,
        mapreduce_kwargs)::OutputType where {OutputType}
    (; nchunks, split, threadpool) = scheduler
    check_all_have_same_indices(Arrs)
    if chunking_enabled(scheduler)
        tasks = map(chunks(first(Arrs); n = nchunks, split)) do inds
            args = map(A -> view(A, inds), Arrs)
            @spawn threadpool mapreduce(initialize(f), op, args...; $mapreduce_kwargs...) # <------- Change here
        end
        mapreduce(fetch, op, tasks)
    else
        tasks = map(eachindex(first(Arrs))) do i
            args = map(A -> @inbounds(A[i]), Arrs)
            @spawn threadpool initialize(f)(args...) # <------- Change here
        end
        mapreduce(fetch, op, tasks; mapreduce_kwargs...)
    end
end

This way, someone can do e.g.
function matmulsums_tls(As, Bs)
    N = size(first(As), 1)
    tls = TaskLocalValue{Matrix{Float64}}(() -> Matrix{Float64}(undef, N, N))
    f = WithTaskLocalValue(tls) do C
        function (A, B)
            mul!(C, A, B)
            sum(C)
        end
    end
    tmap(f, As, Bs)
end

This is kinda ugly, but the idea is that the macro could then automatically handle the creation of the WithTaskLocalValue object for the user.
I can try to put this together in a followup PR to this one, or we can try to integrate it here from the get-go.

@tasks is a good name, let's just do that. Maybe this package should have been named "OhMyTasks" :laughing:  oh well

This now also works:
@tasks for i in 1:10
    @set collect=true
    i
end

It translates to tmap and thus result == collect(1:10).
(BTW, I think I love the @set idea.)

I think this is pretty much ready.

TODO: update README

So we'll save the channel support for a future PR?


[Question/Feedback]: Non Resource/Service Health Alert Rules don't have action group assigned

Check for previous/existing GitHub issues

[X] I have checked for previous/existing GitHub issues

Description
I'm deploying the AMBA templates via Powershell and am running into a strange issue; none of the Metrics, Activity Log or Log Search alert rules have an Action Group assigned to them.
The Service Health and Resource Health alerts do though.
This is a brownfield site, and I don't have any failed remediation tasks related to the Notification policy, which is showing as 100% compliant:

Is there something specific I can check as to why these alerts have no action group associated with them?

Hello @anorwood-xten ,
thanks for your feedback. What you reported here is not an issue but it is the normal behavior. We do not assign action group to Metrics, Activity Log or Log Search alert rules; instead we do use the Alert Processing Rule (APR). Service Health alerts cannot use APRs so we had to assign an action group to them.
Hope that clarifies.
Thanks,
Bruno.

Hi Bruno,
That clears it up completely, thank you for that! I'll skulk back into my hole now.
Cheers,
Anthony

Thanks to you Anthony,
based on that I am going to close this one as completed.
Fell free to reopen it or to create a new one should it be the case.
Thanks,
Bruno.


Functional test suite

This is a quick spike into https://github.com/remind101/stacker/issues/437.
This adds some really basic end-to-end sanity tests that exercise a stacker build/destroy using bats.
Assuming you have bats installed, and some AWS access keys set, you can run this with:
$ brew install bats
$ source venv/bin/activate
$ STACKER_NAMESPACE=<my-stacker-tests-namespace> bats tests
 ✓ stacker build - no config
 ✓ stacker build - empty config
 ✓ stacker build - config with no stacks
 ✓ stacker build - config with no namespace
 ✓ stacker build - missing environment key
 ✓ stacker build - missing variable
 ✓ stacker build - simple build destroy
 ✓ stacker build - simple build with environment
 ✓ stacker build - overriden environment key with -e

9 tests, 0 failures

We'd likely want to run this in CI using an isolated AWS account with access keys that are super tight (only allows cloudformation actions, and the ability to create a WaitCondtionHandle).

Alright, I spun up a new isolated AWS account, and created some access keys using the FunctionalTests blueprint in here.
The functional tests will now run automatically on the master branch only. PR's from fork's won't have access to the AWS access keys here, so I think I've covered any potential security/abuse concerns.


fix: use available messages and remove unused messages

Fixes this error: Missing message pkg_utils:errorInvalidPackageId for locale en_US.
When creating a package version when the package isn't installed/created in the devhub.
Also removes some unused messages in the utils file.
@W-11942357@

QA Notes:

Before:
 ➜  sfdx force:package:beta:version:create --package TEST  -x -v na40
ERROR running force:package:beta:version:create:  Missing message pkg_utils:errorInvalidPackageId for locale en_US.
Version create.... done

After:
 ➜  ../../oss/plugin-packaging/bin/dev force:package:beta:version:create --package TEST  -x -v na40
ERROR running force:package:beta:version:create:  The requested resource does not exist
Version create.... done

But, when compared to TB, does the TB error make more sense?
➜  dreamhouse-lwc git:(main) ✗  hub:(GLOBAL - DevHub<EMAIL_ADDRESS> ➜  sfdx force:package:version:create --package TEST  -x -v na40
 ›   Warning: We no longer maintain the current implementation of the "force:package:version:create" command. You should instead use the "force:package:beta:version:create" command,
 ›   which is functionally the same. If you run into issues, file a report against the "force:package:beta:version:create" command on https://github.com/forcedotcom/cli/issues.
ERROR running force:package:version:create:  The provided package ID '0Ho3i000000Gmj6CAC' is invalid.

Update:
ERROR running force:package:beta:version:create:  No package ID was found in Dev Hub for package ID: 0Ho3i000000Gmj6CAC.
Version create.... done

a better error message was spotted in the wild 👀


Build Dev Tools

Build Dev Tools to help manipulate colours also good for client, showcase tweaks

Should be a separate build i.e. ninetails.dev.js


minor: Improve e2e serial test for Leviathan v2

Todo

Add a serial pull or an explicit test for each suite to pull the serial for testbot workers.
Check if this unblocks GitHub actions progress.

Signed-off-by: Vipul Gupta (@vipulgupta2048) <EMAIL_ADDRESS>

@balena-ci retest

@vipulgupta2048 I cherry-picked this commit over here but the builds still fail: https://github.com/balena-os/leviathan/pull/777
leviathan-client-1  | [2022-06-23T22:55:44.417Z][27718d4-e2e]             # Serial file created at /reports/dut-serial.txt
leviathan-client-1  | [2022-06-23T22:55:44.429Z][27718d4-e2e]             not ok 1 - Size of serial output file from DUT shouldn't be 0

Hey man, @klutchell This is taking longer than expected. I haven't pushed the new changes that I am confident will work. For some reason, I am receiving the serial logs as garbled text in a UTF 16LE encoding and I am trying to find a reason why that happens.

@vipulgupta2048 can you squash some of these commits?

@balena-ci rebase