Scala Kinesis Enrich S3 storage via AWS Kinesis Firehose

Kinesis now has a sink built in called Kinesis Firehose.
My question is the events format that is used in Kinesis enriched good stream the same as the one from Kinesis LZO S3 sink ( https://github.com/snowplow/snowplow/wiki/kinesis-lzo-s3-sink-setup ) or https://github.com/snowplow/snowplow/wiki/setting-up-EmrEtlRunner ?
If it is it could eliminate the need for a S3 sink and Redshift sink all together.

Hey @xkidro - using the Kinesis Firehose instead of the kinesis-s3 app is an interesting idea! It's not something we have the bandwidth to investigate at this time.
If you get a chance to experiment with this approach, please update this ticket with your findings!

Any update ? I'm interested too.

Hi @bcharp - using Kinesis Firehose instead of kinesis-s3 is not something we are actively exploring.
I don't believe Kinesis Firehose can work with binary formats like Thrift - if I am wrong please let me know. In the absence of that, we would have to re-engineer the Stream Collector, Hadoop Enrich and Stream Enrich to move to a textual format like JSON for raw events, and this would of course increase the size of our raw events in Kinesis and S3, plus slow down enrichment (because JSON de/serialization is slow).
Related, our kinesis-s3 output format is optimized for fast Hadoop processing - again we would lose this in the switch to Kinesis Firehose.
A Snowplow real-time topology ends up with a lot of different moving parts - porting to Firehose to remove one moving part doesn't feel like a win given the compromises we would have to make. But certainly happy to hear a counter-view...

Well, it made sense in my mind with this specific mindset : As snowplow is always using the most standard way to do things with AWS, Firehose is surely on the way to enable me to have a streaming pipeline directly from my events to Redshift. But, in fact, the more I read on the streaming pipeline, the more I believe that it's actually not made to insert data into Redshift.

As snowplow is always using the most standard way to do things with AWS

One of the nice things about AWS is that there isn't "the most standard way to do things" - instead AWS typically offers a variety of different strategies, which tend to vary based on ease of use vs configurability.
The trend with Snowplow is often to start with the easiest-to-use approach, and then move towards the more configurable approach as we outgrow the easiest-to-use approach. You can see this with collectors:

CloudFront Collector, super easy to use, very inflexible
Clojure Collector, still easy to deploy using Beanstalk, more functionality but no control over logfile rotation and some Beanstalk behaviors are a little mystical
Stream Collector - very flexible, not even AWS specific but can be deployed on AWS using ELB and an ASG

In terms of streaming events from the real-time pipeline into Redshift - this is something we are working on. We are doing a lot of work to extend Iglu and StorageLoader so that they can handle automatic table creation, table migrations etc. These are tough problems but will be very exciting to solve!

It's kind of hacky, but I was able to get streaming to Redshift every minute by doing:
Scala Collector -> Log File Buffer
Log File Buffer -> Scala Enrich
Scala Enrich -> Enriched Log Files
Eriched Log Files -> Firehose via Kinesis Firehose Agent

Thanks for sharing @johnrobertfitz !

@johnrobertfitz Thanks for sharing, I am also intertested in this subject, would you please show me some more details? like how can scala enrich do with log file buffer?

This is also a rough hack, but I thought I'd post it here in case it'd help someone towards their own solution. I was able to modify the kinesis-tee project to push (TSV -> JSON transformed) events to Firehose instead of pushing to another Kinesis stream:
https://github.com/snowplow/kinesis-tee/compare/master...fingerco:firehose
Which makes our current process:
Scala Collector -> Raw Kinesis
Raw Kinesis -> Scala Enrich
Scala Enrich -> Enriched Kinesis
Enriched Kinesis -> Kinesis Tee (JSON Transform)
Kinesis Tee -> Firehose
Firehose -> Redshift

Thanks for sharing @fingerco !

18 months later and Kinesis Firehose is still only available in 3 regions, closing

A year later, Firehose is available in most AWS regions, and there still seems to be some interest on the forums and elsewhere. Is this still a wontfix?

That's right @jdub - Snowplow S3 Loader continues to be our focus.


GOD DAMN PERSISTING COMMANDS

/fly, /god (config option), /gamemode, /spy whatever else is a toggle that can get reset upon relogging or changing worlds

Plugins don't do anything to persist these, so if it doesn't work for us too, it's a Sponge issue.


Fix issues with resumption testing

Description
The test test_save_load_and_resume_checkpoint was erroneously passing because of an incorrect comparion. Fixing this comparison exposed a further error in the test: the train context defined by build_train_ctx defined in the test file didn't conform to the behaviour of _build_train_context and was generating incorrect results which caused the fixed test to fail.
This PR fixes the incorrect comparison, changes the testing train context builder to use the real context builder plus some minor overrides to enable testing, and removes an unused attribute from L1Scheduler that was causing comparisons to fail.
Type of change
Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Checklist:

[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[x] I have not rewritten tests relating to key interfaces which would affect backward compatibility

You have tested formatting, typing and unit tests (acceptance tests not currently in use)

[x] I have run make check-ci to check format and linting. (you can run make format to format code if needed.)

thanks for catching this! really appreciate it!


Add install to script so we can add a command to download phantomjs from a different location

Since we keep hitting the rate limit this update changes the locations of were we get the download from.

@jdforrester please review, also please add the binary here by going to https://github.com/jdforrester/grunt-svg2png/releases and clicking create release and then add the binary we need. I will then update the link to your binary.

Doing in #12 instead.


Reading only specific rows, based on match in one column

Hi,
I am trying to implement the following logic, but I am a bit lost regarding the APIs I should be using to do this efficiently.

I want to read through an ID column, page by page
If I find a specific ID in that column, I want to read the entire row associated with it

It is clear to me how to 1).
When I have a match I am using
        rowReader := parquet2.NewRowGroupRowReader(rg)
	defer func() {
		if err := rowReader.Close(); err != nil {
			logger.App.Errorf("error closing row group reader: %s", err.Error())
		}
	}()
        // Skip to row of interest
	if err := rowReader.SeekToRow(int64(position)); err != nil {
		logger.App.Errorf("error seeking to position %d row group reader: %s", position, err.Error())

	}
        //Read just one row
	rowBuf := make([]parquet2.Row, 1)
	readRows, err := rowReader.ReadRows(rowBuf)

So far so good. But my parquet file has some nesting. And the data in rowBuf is flattened, so I am having trouble to understand how to properly correlate my columns to the values in rowBuf.
Any advice how to efficiently implement my use case using this library? I feel I might not be using the correct approach here.
Thank you!

Hello @berupp
The intent is for the parquet.Row to a be low-level API focused on offering a high performance mechanism for reading and writing rows, which is why the values are flattened.
To decode the content of a parquet.Row into a Go struct value, you may use a parquet.Schema (e.g. from the file that you are reading) and call the Deconstruct method. Alternatively, if you use a parquet.Reader or parquet.GenericReader, this decoding can be done automatically when reading rows off the underlying file.
Let me know if the answer was useful!

I think if I understand correctly that @berupp is trying to determine the most efficient way to read a single row out of a sorted parquet file given a column value.
We have Find to find the page index number given a column index and column value.  With that, we can load the page containing the column value, but not the whole row.
We have SeekToRow to seek to a given row index.
But as far as I can tell, we have no way to determine the row index given a column value.  I think that sounds reasonable, but I'm not sure if it's possible today as I'm not familiar enough with the implementation yet.
Lemme know if this clarified things for anybody.

Thanks @abraithwaite, I had partially understood the question.
Using SeekToRow is the right approach, but maybe the application should use two readers:

the first reader only decodes the column(s) to filter on, keeping track of the current row index
each time there is a match, the second reader is positioned on the current row using SeekToRow and the application reads the entire row

@berupp do you think this approach would work for your use case?

Thank you @achille-roussel
I did in fact use two readers. The first reader is paging through row groups, only looking at a specific column.
One I get a hit in that column, I use a NewRowGroupRowReader to call SeekToRow and ReadRows, as outlined in the snippet above. Now I have the []Value representing the row, but flattened. Meaning that if I have multiple nested columns, or in my case, list columns, with arbitrary entries, it get hard to populate a struct from it.
I ended up using the definition and repetition levels on the Value, to determine the appropriate columns for it. Nesting in my case is limited to list columns, so it wasn't too much trouble.
It would be nice to have a convenience method on the RowGroupRowReader that reads into an annotated struct, similar to the high level API.
But my question is resolved, by using repetition and definition levels available on the Values

It would be nice to have a convenience method on the RowGroupRowReader that reads into an annotated struct, similar to the high level API.

I agree with this, supporting somewhat low-level query mechanisms that would perform filtering/selection internally (predicate push down?) would be a useful addition to the library 👍

It would be nice to have a convenience method on the RowGroupRowReader that reads into an annotated struct, similar to the high level API.

For my own understanding, does this not work?
        // Skip to row of interest
	if err := rowReader.SeekToRow(int64(position)); err != nil {
		logger.App.Errorf("error seeking to position %d row group reader: %s", position, err.Error())

	}
        //Read just one row
        val := MyStruct{}
        err = rowReader.Read(&val)


schema: change rng:empty to tei:empty

Another try to make a small step towards pureODD.
Doesn't affect anything, just a small namespace clean-up in our ODDs.
See #1050 and #1164

@musicEnfanthen?

Thank you @rettinghaus . In general, I am inclined to leave pureODD transformation for a time after a potential 5.1 release.
Since your proposed changes are very focused in scope, they do not seem to do any harm, on the other hand.
Just to double-check: Could this change lead to a conflicting situation for the next release where we end up with some parts of the schema not "cooperating" very well?

@musicEnfanthen No, there shouldn't be any problem at all. What were you thinking of specifically? Could you please elaborate a bit?

Thank you @rettinghaus . In general, I am inclined to leave pureODD transformation for a time after a potential 5.1 release.
Since your proposed changes are very focused in scope, they do not seem to do any harm, on the other hand.
Just to double-check: Could this change lead to a conflicting situation for the next release where we end up with some parts of the schema not "cooperating" very well?

@musicEnfanthen this change is at the very detail level of the ODD and since an empty element is just –empty– it does not make any difference whether emptiness is defined with rng:empty or tei:empty.

Perfect, thank you @rettinghaus and @bwbohl !


Question - How to use these lifestyles with the Classes.FromAssembly registration method?

Hi there, thanks for these lifestyles!
I can easily use them the Component.For registration method like so:
container.Register(Component.For<IUserStore<ApplicationUser, int>>().ImplementedBy<UserStore>().LifeStyle.HybridPerWebRequestPerThread());
But how do I use them with the Classes.FromAssembly registration method?
container.Register(Classes.FromAssembly(Assembly.GetAssembly(typeof(Manager<>))).BasedOn(typeof(Manager<>)).LifestylePerWebRequest());
This returns BasedOnDescriptor instead of ComponentRegistration, so I'm not sure how to use these lifestyles with this registration.
Thanks for any insight,
Pete

Use Configure e.g.:
container.Register(Classes.FromAssembly(Assembly.GetAssembly(typeof(Manager<>)))
                            .BasedOn(typeof(Manager<>))
                            .Configure(x => x.LifeStyle.HybridPerWebRequestPerThread()));

Thanks a bunch @mausch!


C++: Remove infeasible edges to reachable blocks

The existing unreachable IR removal code only retargeted an infeasible edge to an Unreached instruction if the successor of the edge was an unreachable block. This is too conservative, because it doesn't remove an infeasible edge that targets a block that is still reachable via other paths. The trivial example of this is do { } while (false);, where the back edge is infeasible, but the body block is still reachable from the loop entry.
This change retargets all infeasible edges to Unreached instructions, regardless of the reachability of the successor block.

This pull request introduces 126 alerts when merging 56bb9dcde026fe66ecd6c587aab6e7cb604de032 into b8877f1d5f8e59aca3498363a6d66437f0bfc123 - view on LGTM.com
new alerts:

25 for Syntax error
14 for Unused import
12 for Variable defined multiple times
9 for First argument of a method is not named 'self'
8 for Statement has no effect
4 for Empty except
4 for Unused local variable
2 for Use of the return value of a procedure
2 for Unused named argument in formatting call
2 for Illegal raise
2 for Explicit returns mixed with implicit (fall through) returns
2 for Implicit string concatenation in a list
2 for Request without certificate validation
1 for Too few arguments in formatting call
1 for Unmatchable dollar in regular expression
1 for Unmatchable caret in regular expression
1 for Wrong name for an argument in a call
1 for Formatted object is not a mapping
1 for Mismatch in multiple assignment
1 for Commented out code
1 for Unsupported format character
1 for Duplication in regular expression character class
1 for Comparison using is when operands support __eq__
1 for Missing named arguments in formatting call
1 for Non-callable called
1 for Special method has incorrect signature
1 for Maybe missing 'self' in comparison
1 for Iterable can be either a string or a sequence
1 for Except block handles 'BaseException'
1 for Formatting string mixes implicitly and explicitly numbered fields
1 for Missing call to __init__ during object initialization
1 for Unnecessary 'else' clause in loop
1 for First parameter of a class method is not named 'cls'
1 for __init__ method is a generator
1 for Missing part of special group in regular expression
1 for Unused argument in a formatting call
1 for Comparison of identical values
1 for Unreachable code
1 for Explicit export is not defined
1 for Wrong number of arguments for format
1 for Backspace escape in regular expression
1 for __del__ is called explicitly
1 for Module is imported with 'import' and 'import from'
1 for Redundant assignment
1 for Importing value of mutable attribute
1 for __init__ method returns a value
1 for Nested loops with same variable
1 for Hard-coded credentials
1 for Nested loops with same variable reused after inner loop body
1 for Returning tuples with varying lengths
1 for Unnecessary delete statement in function

Comment posted by LGTM.com

The tests failed:
13:40:57 Some tests failed!
13:40:57     ql/cpp/ql/test/library-tests/controlflow/guards-ir/tests.ql: Test failure (Expected output does not match)

The test failures are in the IR Guards tests. Some of it is due to new unreachable edges that were removed, but most of the diffs are due to merging all of the Unreached instructions for a function into one. The problem is that I give the Unreached instruction the AST (and location) of the Function, since there wasn't really a single AST that makes sense. This confuses the mapping between BasicBlock and IRBlock that the IR Guards library does. I considered going back to one Unreached instruction per infeasible edge, but for infeasible edges to reachable blocks, we'd have an Unreached instruction with the AST of the still-reachable block, which causes similar problems. Instead, I've just had the IR Guards library stop reporting an unreached block as being controlled by any guards. It's not that there's much point in knowing what conditions were true in unreachable code anyway.


[FR]: Generate multiple ts_project() targets with Gazelle extension

What is the current behavior?
The Gazelle JS extension currently only generates up to two ts_project() targets, one for production code and another for test code. This is limiting since many projects may want to have multiple ts_project() files based on a naming convention, yet as it is today, Gazelle will always put source files into these two ts_project() targets, which is quite limiting.
Describe the feature
Ideally, we would have a Gazelle directive which maps a filename glob to a target name. This would allow file naming conventions to inform the BUILD structure generated.
In my particular case, I have two sets of TypeScript files with the naming convention *.prerender.ts and *.client.ts. These need to be built in distinct ts_project() targets with distinct dependencies because one targets the Node runtime while the other targets a browser runtime. All the *.prerender.ts files should go under a :prerender target, while the *.client.ts files should go under a :client target.
In a conversation with @jbedard, they suggested an approach like:
# gazelle:js_custom_target_files prerender *.prerender.ts
# gazelle:js_custom_target_files scripts *.client.ts

This should be enough information for Gazelle to generate and multiple ts_project() targets with a reasonable structure.

At Sourcegraph, we're also looking forward to this improvement.


JetBrains: Ghost text (code lenses) will not be available if user do "Edit & Retry" multiple times.

Cody Version
v5.5.10-nightly
IDE Information
IntelliJ IDEA 2023.3.4 (Community Edition)
Build #IC-233.14475.28, built on February 13, 2024
Runtime version: 17.0.10+1-b1087.17 amd64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o.
Windows 11.0
GC: G1 Young Generation, G1 Old Generation
Memory: 2048M
Cores: 12
Registry:
ide.experimental.ui=true
Non-Bundled Plugins:
PythonCore (233.14475.9)
com.soywiz.korge.korge-intellij-plugin (4.0.4)
com.sourcegraph.jetbrains (5.5.10-nightly)
Kotlin: 233.14475.28-IJ
Describe the bug
Platform: Windows
Steps to reproduce:

Have Cody plugin installed in your JB IntelliJ IDEA editor
Login to Cody with Free/Pro user
Open any source code file and select a code snippet
Right click to open context menu and choose "Cody > Document Code"
Click on "Edit & Retry" and type in something i.e. generate doc string and hit enter
Again click on "Edit & Retry" and type in something i.e. create a simple test function and hit enter

Observed behavior:
Ghost text(code lenses) is missing for the added code snippet from step-6 above
Expected behavior
Ghost text should appear correctly so that user can choose Accept, Undo, Edit & Retry, or Show Diff
Additional context
Video link:
https://drive.google.com/file/d/1Mo5sSNxzJlgSvaCwpRKC2i_Tdkgx4Q2j/view?usp=drive_link

@jay-fibi I haven't been able to repro this on 5.5.11-nightly. Are you still seeing this issue?

@taylorsperry Yes, I am still able to repro this issue. Though, it is intermittent and doesn't repro every time. The steps are same as above.

@taylorsperry attaching idea.log file for your reference
idea.log


chore: CalcJoinPoolShares unit tests with single token validated by Wolfram

…olfram

Closes: #XXX
What is the purpose of the change

Add a description of the overall background and high level changes that this PR introduces

(E.g.: This pull request improves documation of area A by adding ....
Brief Changelog
(for example:)

The metadata is stored in the blob store on job creation time as a persistent artifact
Deployments RPC transmits only the blob storage reference
Daemons retrieve the RPC data from the blob cache

Testing and Verifying
(Please pick one of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added unit test that validates ...
Added integration tests for end-to-end deployment with ...
Extended integration test for ...
Manually verified the change by ...

Documentation and Release Note

Does this pull request introduce a new feature or user-facing behavior changes? (yes / no)
Is a relevant changelog entry added to the Unreleased section in CHANGELOG.md? (yes / no)
How is the feature or change documented? (not applicable   /   specification (x/<module>/spec/)  /  Osmosis docs repo   /   not documented)

closing in favor of: https://github.com/osmosis-labs/osmosis/pull/1721


Breaking Modlib mods

Feel free to read my mod's posts.
https://www.nexusmods.com/mountandblade2bannerlord/mods/118/?tab=posts&BH=0

Noted. Will be fixing it

1.0.7 (latest commit) should fix most issues with ModLib

Group Names show
Save/Load/Resetting settings work

One issue left. When resetting the settings, the interface is not updated. Will look into it later.

Alright appreciate that; thank you.


Add new watcher for Dropdown.isActive to emit event

b-dropdown will now emit the event active-change with the payload of the new value of Dropdown.isActive whenever isActive changes.
Edit: change isActiveChange to active-change to match code revision.

@rafaelpimpa Done!

Thanks you very much :)


feat: adds endpoints for fetchGuestToken

Closes https://github.com/openedx/platform-plugin-aspects/issues/23
Testing instructions

Load the Instructor Dashboard
Wait 5 min
Ensure your token continues to work with Superset.
Add the SupersetXBlock to a course
Repeat steps 2-3 with the XBlock in the LMS and Studio.

Merge checklist:
Check off if complete or not applicable:

[ ] Version bumped
[ ] Changelog record added
[ ] Documentation updated (not only docstrings)
[ ] Fixup commits are squashed away
[ ] Unit tests added/updated
[ ] Manual testing instructions provided
[ ] Noted any: Concerns, dependencies, migration issues, deadlines, tickets

Thanks for the pull request, @pomegranited! Please note that it may take us up to several weeks or months to complete a review and merge your PR.
Feel free to add as much of the following information to the ticket as you can:

supporting documentation
Open edX discussion forum threads
timeline information ("this must be merged by XX date", and why that is)
partner information ("this is a course on edx.org")
any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.
This is currently a draft pull request. When it is ready for our review and all tests are green, click "Ready for Review", or remove "WIP" from the title, as appropriate.

Hi @Ian2012 @bmtcril This is a draft PR because I haven't been able to load this branch into tutor dev to test it properly, but I'll rebuild tonight and start that tomorrow.
 Do you have any idea how to fix this issue?

Thank you for your fixes @Ian2012 , I'm getting there.. hope to have this ready for your review tomorrow.

Thank you for your fixes @Ian2012 . I'm still unable to get this running in my dev stack, but I fixed the tests.
If it's working for you, please feel free to merge :) Otherwise I'll keep working on it.

@pomegranited 🎉 Your pull request was merged!  Please take a moment  to answer a two question survey so we can improve your experience in the future.


Missing Images in all Modules

All Modules
Images are missing in all modules.  Please update asap.

Present in https://github.com/MicrosoftLearning/DP-900T00A-Azure-Data-Fundamentals.ja-jp/tree/OLPRODLOC/Instructions/Labs/images


#38 publish helper

mostly gradle scripts to help you publish to maven central.
From here, you'll need to create a gpg key, and a sonatype oss account
then run the commands in the readme file.

Looks like you're publishing your fork (which is totally fine), so I'm going to close this.


CSI support for Kubernetes v1.12.x

The way CSIs are registered/operate has fundamentally changed in Kubernetes v1.12.x. This means that the existing images and manifests no longer work:
Events:
  Type     Reason                  Age               From                                              Message
  ----     ------                  ----              ----                                              -------
  Normal   Scheduled               27s               default-scheduler                                 Successfully assigned default/nginx-csi-hostpath to cs-3f10d2ba-bec9-4f75-92fc-bdd571e26633
  Normal   SuccessfulAttachVolume  27s               attachdetach-controller                           AttachVolume.Attach succeeded for volume "kubernetes-dynamic-pv-34d8871adbc411e8"
  Warning  FailedMount             3s (x6 over 19s)  kubelet, cs-3f10d2ba-bec9-4f75-92fc-bdd571e26633  MountVolume.MountDevice failed for volume "kubernetes-dynamic-pv-34d8871adbc411e8" : driver name csi-hostpath not found in the list of registered CSI drivers

You can easily reproduce this by following the steps listed here.
Other CSIs broke in the same way. For example, you can reference the similar DigitalOcean issue that was recently fixed (though the issue itself is not closed yet).

@mattkelly thanks for pointing out, I will fix it.

Thanks @andyzhangx! Any chance you can provide a rough ETA for a new image being pushed?

@mattkelly It's not related to the csi-hostpath image, I have followed exactly as https://kubernetes-csi.github.io/docs/Example.html, and it works. There is really big change in v1.12.x, by default, CSI is even not enabled. Anyway, the csi-hostpath works now. And thanks for pointing out, my next major work would write azure disk & azure file csi drivers, make it work on v1.12.x

share you the new azure CSI drivers, it's under kubernetes-sig now:
https://github.com/kubernetes-sigs/azuredisk-csi-driver
https://github.com/kubernetes-sigs/azurefile-csi-driver
You could run those drivers on k8s v1.13.x


Fix duplicate attachment/binexplode loops

Found duplicate loops in the "Please Enable Macros" rule that can be merged together.

love me a good red diff 👏


Blocks can be waterlogged in flowing water

Water seems to flow through sign(s).
This is obviously not vanilla parity.
Tested on the dragonfly dev server (<IP_ADDRESS>:19132)
Clips:
https://user-images.githubusercontent.com/20887403/179751694-b52afec9-d4b5-4a40-a364-be209f9b6363.mp4
https://user-images.githubusercontent.com/20887403/179751688-04988e0f-6253-46f9-b1db-46ba1c9f818b.mp4

As discussed on Discord, this seems to be the case with any waterloggable block placed in flowing water.  Dragonfly allows this behaviour, while Vanilla does not. This also explains the "ghost water" that you see in the video, and lava not being affected by this.

Skulls also appear to have this problem


Branch 21.06

need initialize ovs numa before use  ovs_numa_get_n_numas() and ovs_numa_get_n_cores()

Hi @iceCreeam
Thanks for the PR.  You've submitted this PR to branch-21.06.
Normally we apply new patches/PR's to the main branch first and then backport to the stable branches.
Can you please resubmit this PR targetting the main branch ?
You can also send a patch to<EMAIL_ADDRESS>if you prefer.  We generally use ovs-dev ML/patch works.
But PR's are welcome too :)
Thanks

Closing due to inactivity.


Support for private git servers

Does the statement around Azure Databricks not supporting private git servers still apply if the Databricks environment is VNET injected and if there was a VPN/Express Route back to on premise to where the private git server is located?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 8b98be98-e0dc-e08c-8a0d-b87571cc77b9
Version Independent ID: 8b98be98-e0dc-e08c-8a0d-b87571cc77b9
Content: Repos for Git integration - Azure Databricks
Content Source: databricks/repos/index.md
Service: azure-databricks
GitHub Login: @mssaperla
Microsoft Alias: saperla

@pnarsi Thanks for your feedback. We will investigate and update the thread.

@pnarsi Yes Azure Databricks not supports private git servers still apply if the Databricks environment is VNET.
Please Note, GitHub forum is dedicated for docs related issues. For any technical queries or clarifications, we encourage to utilise Microsoft Q & A platform. Kindly raise your query on Microsoft Q&A Platform

Thanks for that. It is probably worth mentioning this fact in the docs as well.


Request to join cdnjs

redux-saga is a great middleware for redux, and I'm using it with public cdn service, and I can not found it in the cdnjs, would you mind to send redux-saga to cdnjs?
here is the cdnjs/cdnjs

You've already filed an issue in their repository - https://github.com/cdnjs/cdnjs/issues/9501 . It got High Priority and an assignee already, so hopefully it will be added there soon. I think it should be handled there from now on, so gonna close the issue here.

Yes, after start this issue, I explore the cdnjs, and start a Add Request issue, it's ok to close this now.


Work around Cypress GPU error

This PR works around the error:
[2134:0211/090347.983213:ERROR:gpu_memory_buffer_support_x11.cc(44)] dri3 extension not supported.

showing in the log files of

actions/workflows/cypress-test-prod.yml and
actions/workflows/build-and-test.yml

The error is caused when the Electron browser, built-in to Cypress, attempts to make use of graphics hardware acceleration, which is not available in the GitHub runner virtual environment.
The PR adds the environment variable to prevent this error message appearing:
env:
  ELECTRON_EXTRA_LAUNCH_ARGS: '--disable-gpu'

(The error has never prevented the Cypress test from completing successfully.)
Verification
Run the workflow .github/workflows/cypress-test-prod.yml and confirm that the workflow continues to succeed and that there is no gpu related error message in the logs.
The workflow .github/workflows/build-and-test.yml is automatically run on pull_request. Similarly check for success and absence of any gpu related error message.
Other references

https://github.com/corona-warn-app/cwa-website/discussions/3305
https://github.com/corona-warn-app/cwa-website/issues/3366

The error did not previously prevent the Cypress test from completing successfully.

Any idea why it does now?

@Ein-Tim

The error did not previously prevent the Cypress test from completing successfully.

Any idea why it does now?

Whenever the ERROR message is displayed the test continues successfully. That has not changed.
The only change is that I now understood better about the reason behind the error message and how to prevent it occurring.
The PR prevents the error occurring when a browser apart from Electron is selected to run the Cypress tests.

@MikeMcC399

Whenever the ERROR message is displayed the test continues successfully. That has not changed.

You're right, sorry for the misleading comment! I was just confused cause the Cypress production test failed in my repo today: https://github.com/Ein-Tim/cwa-website/actions/runs/4155601424/jobs/7188810639

The PR prevents the error occurring when a browser apart from Electron is selected to run the Cypress tests.

Thank you very much for taking care of it!

@Ein-Tim

You're right, sorry for the misleading comment!

No problem!

I was just confused cause the Cypress production test failed in my repo today: https://github.com/Ein-Tim/cwa-website/actions/runs/4155601424/jobs/7188810639

There can be temporary conditions which cause the test to fail. You should be able to select it to re-run manually if you go to https://github.com/Ein-Tim/cwa-website/actions/workflows/cypress-test-prod.yml

Thanks for the PR @MikeMcC399!


openai[patch]: fix #5520

Fixes #5520

Thank you for this! It looks reasonable, CC @sinedied @sarangan12 for a look?
I think we'll also want to apply this to chat models and embeddings.

Thank you, LGTM!

Thank you both!


Permission problems and not complete mongo installation

I ran into several permission issues:
Line 28/29:
sudo echo  "extension = mongodb.so" >> /etc/php/7.0/cli/php.ini;
sudo echo  "extension = mongodb.so" >> /etc/php/7.0/fpm/php.ini;
needed to be adjusted to:
sudo sh -c "echo 'extension = mongodb.so' >> /etc/php/7.0/cli/php.ini";
sudo sh -c "echo 'extension = mongodb.so' >> /etc/php/7.0/fpm/php.ini";
I also needed to manually add the mongodb.service file.
Finally I had to install mongo manually after all of this.
sudo apt-get install -y mongodb-org
And now it is running.

What errors did you get @digixweb ?


structure of the library

As per discussions with @pabloarosado, the library should have individual submodules for each "category". Some categories that we have identified can be:

io: Input/output file management. May include local and remote files (Google Drive, S3, etc.)?
checks: Checks on the data.
clean: Cleaning of the data.

etc.
This is open for discussion.

@Marigold @lucasrodes what about the following structure?

common: Common objects that will be shared by other modules, possibly definitions of shared global constants.
dataframes: Related to operations with dataframes (not necessarily owid-related things). I'm not totally sure about the name, I thought we could call it df, but that could be problematic, since df is often used for the actual dataframe object.
io: Related to reading/writing. For example, read_json, write_json. We could move the S3 object here, but maybe we should avoid making this module too big.
s3: Related to reading/writing to S3.
gdrive: Related to reading/writing to Google Drive (I don't have any functions on this, but Lucas mentioned it).
population: Standardization of countries/regions, adding region aggregations, listing countries in a region, etc. This would be very specific for owid needs (in contrast with the previous modules that are more generic). We have discussed a few times whether this should be part of etl, but etl is going to be a massive repos at some point, so (1) it would be problematic to import it in other repos, and (2) it seems like a good idea to take code out of etl that can be considered independent and generic.
checks: Related to sanity checks. Again, very specific to owid needs. This includes checks on a single dataset (whether countries are standardized, whether values are in reasonable ranges, etc.) and checks on pairs of dataset (e.g. an old and new version of a dataset). I thought that the dataframes.compare could be part of it, but I suppose it's more convenient if the sanity checks import dataframes.compare.

Do you agree on the names and the logic? And what other modules do you think may be missing (for example, Lucas mentioned clean, what would that one have)?

Nice! Thanks for your proposal @pabloarosado. I think these modules would be great.
I probably only have some doubts about the name for population, as - from what I understand - it would have logic that would not necessarily imply population. Maybe using regions instead?
As per the clean module (or similar name), we currently have a cleaning module for the covid project. In it, we have e.g.:

Clean numeric values: From a comma-separated string value ("120,000.00") to an integer (120000). [source code].
Standardise date formats (YYYY-MM-DD), etc. [source code]
Cleaning texts, e.g. working with encodings. [source code].
Cleaning some URLs, e.g. more friendly URLs. [source code]

Perhaps a better naming is format?

Thanks @lucasrodes:

population -> region. I agree that population is not a great name. I'm not sure if regions would be. Maybe geo, given that it's related to geographical entities? This would make sense if we consider that the regions functions can become more complex in the future, if we start considering time-dependent country boundaries and such things.
I prefer format instead of clean.
I agree that io should become a folder containing s3, gdrive, etc.


Not able to download from VRV.co

VRV is not working right.
If somebody can provide a fix, that'd be great.
Thanks in advance!
Error log:
ERROR: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 627, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2227, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/usr/lib/python2.7/urllib2.py", line 435, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 473, in error
return self._call_chain(*args)
File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

Oh sorry.  Didn't know someone already reported it.


[BUG]: Reset Partner Password giving error

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior
The "Reset Partner Password" button in the essential bank's view of the partner is not working.   You get "Failed to invite [partner email] to [partner name]  due to:  "Validation failed:  Resource type is not included in the list"
(Note:  this was found in production.)
Expected Behavior
Partner gets a password reset email, there is a message indicating that they got it.
Steps To Reproduce
In staging,
sign in as org_admin
click on "Partner Agencies"
then... Pawnee Parent Service
Reset Partner Password
fill in email<EMAIL_ADDRESS>Click Reset password.
Environment
No response
Criteria for Completion

[ ] The Reset password function works
[ ] tests to support

Anything else?
Additional note:  if you don't fill in the password, it does accept it --  I would assume that doesn't do anything meaningful, but it shouldn't accept it, really.
Code of Conduct

[X] I've read the Code of Conduct and understand my responsibilities as a member of the Ruby for Good community

FYI We have had an issue reported in production with the same error on partner invite.

Note: this may now be fixed. Will attempt to replicate.

Still shows up in staging. Not seeing the reset password button in our prod test account

We'll retry replicating this today.

Addressed by PR #3331 -- which was just merged into main.


[BUG] - winston failing on web imports without multiple polyfills

Describe the bug
winston is used as the default logging library but is not compatible in browser environments without multiple different polyfills
To Reproduce
Steps to reproduce the behavior:

Setup a react app using create-react-app
Install the ar-io-sdk
Run npm build and you'll see winston causes a lot of build errors related to missing polyfills

Expected behavior
Browser clients need as few polyfills as possible
Environment (please complete the following information):

Environment: browser
SDK Version: v1.0.4
Node Version: v18.17.1
Browser: chrome

Additional context
bunyan seems like a suitable replacement

This will be resolved in v1.0.5


Fix certficate status last day expiration (DEV)

In an edge case where the current date would be the same as the expiration date of the certificate, the status would be set to "Expired" even though the certificate should still be valid on that day as well. This small change sets the status to "ExpiringSoon".

@mlenkeit I understand expiration as the first date of invalidity. How is it meant o be?

If we want to make it more precise, we can change it to check the timestamp, but we should not show it as valid for the whole day.

If we want to make it more precise, we can change it to check the timestamp, but we should not show it as valid for the whole day.

I like this option the most. The expiresAt has a date and time: "2021-06-03T10:12:48.000+02:00". If I'm on 03.06.2021, and it's 9AM, I should still see the certificate as "ExpiringSoon".

I've put it to draft until we get a clear answer about this.

Addressed by https://github.com/corona-warn-app/cwa-app-android/commit/f5e6b716d0273d0f062082c643a80859a7b90e2d


[Import term] vacuum drying

Term name and id
vacuum drying CHMO:0001557
Term link
http://purl.obolibrary.org/obo/CHMO_0001557
Parent term(s)
material processing (OBI:0000094)
Child term(s)
No response
Please state the reason to import this term into dpbo
Commonly done for metabolite extracts before resuspension for GC/LC-MS.
Should be child term of sample drying (CHMO:0001549)
Your ORCID
0000-0003-3393-4224

Thank you for your contribution, the suggested term has been imported into dpbo.


PackageType enum value is formatted badly when creating package version

Describe the bug
While I was trying to add cci to our workflow in a 2GP Managed package, when executing the release_2gp_beta flow cci failed to find the existing package and wanted to create a new one.
This was printed to the screen:
Checking for existing PackageTypeEnum.managed Package named Veezla | Core with namespace vz, notice the enum value string...
And this was the SOQL generated, which also has the enum value incorrectly:
SELECT Id, ContainerOptions FROM Package2 WHERE IsDeprecated = FALSE AND ContainerOptions='PackageTypeEnum.managed' AND IsOrgDependent=False                     AND Name='Veezla | Core' AND NamespacePrefix='vz'
After adding .value to https://github.com/sfdO-Tooling/cumulusci/blob/main/cumulusci/tasks/create_package_version.py#L273 and https://github.com/sfdO-Tooling/cumulusci/blob/main/cumulusci/tasks/create_package_version.py#L276 cci worked as expected.
Reproduction steps

Have a 2GP package
Execute the release_2gp_beta flow

Your CumulusCI and Python versions
CumulusCI version: 3.68.1 (/Users/attila/.local/bin/cci)
Python version: 3.11.0 (/Users/attila/.local/pipx/venvs/cumulusci/bin/python)
You have the latest version of CumulusCI ⛅
See the latest CumulusCI Release Notes: https://github.com/SFDO-Tooling/CumulusCI/releases/tag/v3.68.1
Operating System
macOS
Windows environment
No response
CumulusCI installation method
pipx
Error Gist
No response
Additional information
No response

Thank you, @attilah! That's definitely a bug. We'll get that fixed as soon as we can work it in.

Tracking as W-12045409


Could you please give more datasets? thanks.

Hello, I'm an undergraduate student and I'm very interested in your work. However, it seems quite difficult to find hierarchical datasets related to single-cell. Could you please provide more information about the datasets used in your paper to help me further my experiments in this field?

Hi Xuewen
Sorry for very late reply.
I guess you are looking for single-cell datasets that provide celltype annotation in hierarchical manner.
As I know two datasets provide celltype annotation in hierarchical manner (GSE176078 and SCP259)
They were used in my HiCAT paper. However, as noted, it's quite difficult to use their subset annotation results for performance evaluation.
If you didn't mean it, please let me know.


API service dies and never comes back if you create an invalid JSON config

I made a typo in my /usr/share/osbuild-composer/repositories/rhel-8.json file and this caused it to be invalid JSON. After that, attempting to restart the osbuild-composer.service unit fails with:
Aug 30 15:54:35 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: Stopped OSBuild Composer.
Aug 30 15:54:35 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: osbuild-composer.service: Start request repeated too quickly.
Aug 30 15:54:35 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: osbuild-composer.service: Failed with result 'exit-code'.
Aug 30 15:54:35 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: Failed to start OSBuild Composer.
Aug 30 15:54:37 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: osbuild-composer.service: Start request repeated too quickly.
Aug 30 15:54:37 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: osbuild-composer.service: Failed with result 'exit-code'.
Aug 30 15:54:37 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: Failed to start OSBuild Composer.
Aug 30 15:54:38 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: osbuild-composer.service: Start request repeated too quickly.
Aug 30 15:54:38 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: osbuild-composer.service: Failed with result 'exit-code'.
Aug 30 15:54:38 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: Failed to start OSBuild Composer.
Aug 30 15:54:39 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: osbuild-composer.service: Start request repeated too quickly.
Aug 30 15:54:39 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: osbuild-composer.service: Failed with result 'exit-code'.
Aug 30 15:54:39 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: Failed to start OSBuild Composer.
Aug 30 15:54:40 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: osbuild-composer.service: Start request repeated too quickly.
Aug 30 15:54:40 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: osbuild-composer.service: Failed with result 'exit-code'.
Aug 30 15:54:40 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: Failed to start OSBuild Composer.

From there, fixing the JSON file doesn't allow the service to start. Instead I had to stop both the osbuild-composer.socket and osbuild-composer-api.socket units (the latter of which still won't come back so cockpit integration continues to fail, but composer-cli is functional.
The error reported in this case is:
Aug 30 15:49:39 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: osbuild-composer-api.socket: Socket service osbuild-composer.service already active, refusing.
Aug 30 15:49:39 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: Failed to listen on OSBuild Composer API socket.
Aug 30 15:49:40 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: osbuild-composer-api.socket: Socket service osbuild-composer.service already active, refusing.
Aug 30 15:49:40 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: Failed to listen on OSBuild Composer API socket.
Aug 30 15:49:41 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: osbuild-composer-api.socket: Socket service osbuild-composer.service already active, refusing.
Aug 30 15:49:41 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: Failed to listen on OSBuild Composer API socket.
Aug 30 15:49:41 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: osbuild-composer-api.socket: Socket service osbuild-composer.service already active, refusing.
Aug 30 15:49:41 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: Failed to listen on OSBuild Composer API socket.
Aug 30 15:49:42 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: osbuild-composer-api.socket: Socket service osbuild-composer.service already active, refusing.
Aug 30 15:49:42 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: Failed to listen on OSBuild Composer API socket.
Aug 30 15:55:47 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: osbuild-composer-api.socket: Socket service osbuild-composer.service already active, refusing.
Aug 30 15:55:47 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: Failed to listen on OSBuild Composer API socket.
Aug 30 16:36:27 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: osbuild-composer-api.socket: Socket service osbuild-composer.service already active, refusing.
Aug 30 16:36:27 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: Failed to listen on OSBuild Composer API socket.
Aug 30 16:36:44 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: osbuild-composer-api.socket: Socket service osbuild-composer.service already active, refusing.
Aug 30 16:36:44 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: Failed to listen on OSBuild Composer API socket.
Aug 30 16:36:52 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: osbuild-composer-api.socket: Socket service osbuild-composer.service already active, refusing.
Aug 30 16:36:52 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: Failed to listen on OSBuild Composer API socket.
Aug 30 16:40:14 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: osbuild-composer-api.socket: Socket service osbuild-composer.service already active, refusing.
Aug 30 16:40:14 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: Failed to listen on OSBuild Composer API socket.
Aug 30 17:09:51 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: osbuild-composer-api.socket: Socket service osbuild-composer.service already active, refusing.
Aug 30 17:09:51 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: Failed to listen on OSBuild Composer API socket.
Aug 30 17:10:23 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: osbuild-composer-api.socket: Socket service osbuild-composer.service already active, refusing.
Aug 30 17:10:23 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: Failed to listen on OSBuild Composer API socket.
Aug 30 17:20:20 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: osbuild-composer-api.socket: Socket service osbuild-composer.service already active, refusing.
Aug 30 17:20:20 rhsm-qe-4.rhq.lab.eng.bos.redhat.com systemd[1]: Failed to listen on OSBuild Composer API socket.

Which seems odd since it gives this error even when the osbuild-composer.service unit is down. However, at one point we realized there was a dangling socket file that was never cleaned up and removing that allowed osbuild-composer.socket to start.
I'm not sure what to provide for context beyond that, but am happy to provide any debugging information I can.

Hmm, I wonder if this could be solved by some additional systemd directives making sure sockets are correctly restarted when the unit is.

FWIW If I mangle /etc/osbuild-composer/repositories/rhel90.json I get this in the logs when starting osbuild-composer:
osbuild-composer[1046]: 2021/08/31 09:05:03 Error initializing weldr API: error loading repository definitions: invalid character '{' looking for beginning of object key ...

I haven't tried on RHEL8 to double check but I think the error is easier to see in RHEL 9/upstream.
I fix the error in the JSON and reboot I am able to start osbuild-composer so this may be limited to RHEL8.


[Question]How to validate SecurityStamp on Client

I'm using the CookieAuthenticationMiddleware and now trying to integrate IdentityServer3 in combination with IdentityServer3.AspNetIdentity. To invalidate cookies when user information (password, external accounts, ...) changes you would normally use the SecurityStampValidator in the OnValidateIdentity callback.
With IdentityServer3.AspNetIdentity the security stamp validation is now moved to IsActiveAsync of the AspNetIdentityUserService class.
Please correct me if I'm wrong, but my understanding is that I now have to call one of the IdentityServer3 endpoints within the OnValidateIdentity callback to trigger IsActiveAsync and validate the result?
This would be either the Identity Token Validation endpoint (if I only store the identity token) or the UserInfo / Token Introspection endpoint (where I would also have to store the access token and add a resource scope) - all of them would trigger IsActiveAsync and reflect this information into the response?
Either way, IsActiveAsync would always return true, because the security_stamp is not included in the claims and AspNetIdentityUserService:IsActiveAsync would set ctx.IsActive to true for this cases.

Shouldn't IsActiveAsync set ctx.IsActive to false if EnableSecurityStamp and userManager.SupportsUserSecurityStamp are set to true but no security stamp is available?
As far as I understand to make security stamp validation work on the client side I have to include the security stamp in the identity or access token (depending on which endpoint I use for validation) so IsActiveAsync can validate it?
If this is correct, which concrete combination would you suggest? A custom resource scope with security stamp as claim?

You'd have to build this custom,.

Thanks for your response. Any comments / suggestions for the implementation or some answers to my questions?
I have a more or less working sample, but I'm still open for any feedback.
I'm storing the security_stamp in the identity token:
new Scope()
{
    Name = "security_stamp",
    Type = ScopeType.Identity,
    Claims = new List<ScopeClaim>() { new ScopeClaim("security_stamp") }
}

OnValidateIdentity mimics more or less what SecurityStampValidator.cs does.
OnValidateIdentity = async context =>
{
    // Remote validation of security stamp
    // Adapted version of: http://www.symbolsource.org/MyGet/Metadata/aspnetwebstacknightly/Project/Microsoft.AspNet.Identity.Owin/2.0.0-beta1-140130/Release/Default/Microsoft.AspNet.Identity.Owin/Microsoft.AspNet.Identity.Owin/SecurityStampValidator.cs?ImageName=Microsoft.AspNet.Identity.Owin
    DateTimeOffset currentUtc = DateTimeOffset.UtcNow;

    if (context.Options != null && context.Options.SystemClock != null)
    {
        currentUtc = context.Options.SystemClock.UtcNow;
    }

    DateTimeOffset? issuedUtc = context.Properties.IssuedUtc;

    // Only validate if enough time has elapsed
    bool validate = issuedUtc == null;

    if (issuedUtc != null)
    {
        TimeSpan timeElapsed = currentUtc.Subtract(issuedUtc.Value);
        validate = timeElapsed > TimeSpan.FromMinutes(5);
    }

    if (validate)
    {
        // Check via identity token validation (http://identityserver.github.io/Documentation/docsv2/endpoints/identityTokenValidation.html)
        // Calls IsActiveAsync of IUserService and compares local security_stamp with remote security_stamp
        HttpClient client = new HttpClient();

        string idToken = context.Identity.Claims.Where(x => x.Type == "id_token").Select(x => x.Value).SingleOrDefault();

        if (string.IsNullOrEmpty(idToken))
        {
            context.RejectIdentity();
            context.OwinContext.Authentication.SignOut(DefaultAuthenticationTypes.ApplicationCookie);
            return;
        }

        HttpResponseMessage response =
            await
                client.GetAsync(string.Format("{0}/connect/identitytokenvalidation?token={1}&client_id={2}",
                    Authentication.Configuration.IdentityManagerAuthority, idToken, "webapplication"));

        if (!response.IsSuccessStatusCode)
        {
            // Reject identity, force reauthentication
            context.RejectIdentity();
            context.OwinContext.Authentication.SignOut(DefaultAuthenticationTypes.ApplicationCookie);
            return;
        }

        context.OwinContext.Authentication.SignIn(
            new AuthenticationProperties { IsPersistent = context.Properties.IsPersistent }, context.Identity);
    }
}

I'm still not sure if I'm using the "right" scope / validation endpoint and it only works as long as the identity token lifetime is not expired, which is kind of a drawback.

Putting info in the id_token (I don't think) will help -- it's a one-time thing at a point in time. At some point in the future the password changes, so the security stamp changes, and there's no connection back to the server/db that this has happened. Like I said, it'd have to be custom and an explicit connection/call back to the Id management system.

and there's no connection back to the server/db that this has happened.

That's why I would call one of the IdentityServer endpoints within OnValidateIdentity, which triggers AspNetIdentityUserService:IsActiveAsync, which verifies the security stamp in the token (like in the code posted above)?

That's why I would call one of the IdentityServer endpoints within OnValidateIdentity, which triggers AspNetIdentityUserService:IsActiveAsync, which verifies the security stamp in the token (like in the code posted above)?

Ok, so now I finally got the whole picture of what you're doing. So yes, this sounds like it's fine. I'd just make a custom endpoint for the check (not sure how you're doing that part now).

Why would you make a custom endpoint? At the moment I'm using on of the existing endpoints, because the check IsActiveAsync anyway:
HttpResponseMessage response =
        await
            client.GetAsync(string.Format("{0}/connect/identitytokenvalidation?token={1}&client_id={2}",
                Authentication.Configuration.IdentityManagerAuthority, idToken, "webapplication")

Because identity token validation is validating the id_token (including its expiration), which isn't quite what you needed to check, right? You needed to know if the security stamp has changed. Those are different semantics.

That is true, but it was the best I came up with so far.  If possible, I would like to have an endpoint which checks the IsUserActive function and returns true / false.
How would you suggest to implement this custom endpoint? I scanned over the code and samples and I think it's not possible to integrate it directly into Identity Server? So I guess the best way would be creating a custom middleware, passing the IUserService (or the factory) and instantiate an IsActiveContext and call IsUserActive for the validation?

Sure, a custom middleware after idsvr that uses the OWIN extensions to resolve dependencies from IdSvr's DI system.

@Dresel Did you ever get this working? I am facing a similar challenge right now and would love it if you had a working sample!

Sorry for my late reply, I was partly on vacation and also had other things to do.
I dug a little bit into the IdentityServer code base, this is the solution I came up with:
On the IdentityServer side:

Add security stamp to the Clients and Scope configuration
Override AspNetIdentityUserService to allow retrieval of security stamp as claim
Implement a custom OWIN middleware that expects a clientID and (at least) the sub and security stamp claim (I guess the easiest way to pass this information is to send the identity token?)

This middleware then creates an IsActiveContext instance (Client is accessed via IClientStore and clientID, Subject can be created via Principal.Create and the list of claims)
This context can be passed to the IsActiveAsync function of the AspNetIdentityUserService, which validates security stamp if present as claim
The result of IsActiveAsync is returned by the OWIN middleware

On the client (MVC) side:

Add the security stamp as scope to the OpenIdConnectAuthenticationOptions
Mimic the implementation of the SecurityStampValidator

If revalidation is needed, call the custom middleware on the IdentityServer side to check IsActiveContext

@brockallen Is that what you had in mind? Would you pass the needed information as identity token to the custom middleware or would you use a custom format for clientID and the list of claims?
@mikesigs I don't mind posting a working sample as soon as I have decided on all the details. If there's interest I could also send an PR for the IdentityServer3.Samples repository.

Can I ask why not just have the cookie expiry set by the id_token lifetime instead? OpenIdConnectAuthenticationOptions's UseTokenLifetime value is true by default.  Once the expiry time is reached a subsequent request would trigger the OpenId middleware to authenticate again and thereby do all the IsActive and SecurtyStamp checks.
If the token time is kept short then cookie replay is kept to a minimum.  The down-side is if your app requires consent each login they will be presented a consent screen each time but if that's not your use case then maybe this is an option.

I'm not sure if I understood you correctly.
As far as I understand, as long as the Cookie of the UseCookieAuthentication middleware is valid, there will be not SecurityStamp checks. In our current setup, if the CookieMiddleware cookie is expired, the User is redirected to the Login Page of our ASP.NET project. There he can choose one external provider which redirects to IdentityServer which then redirects back to the ASP.NET project.
That's why we are using the default CookieMiddleware lifetime (14 Days) in combination with custom security stamp validation to react on privilege changes (e.g. roles added / removed).

Sorry, I had assumed your login workflow was hosted by IS3.  In my assumed scenario if the cookie has expired or rejected by the cookie middleware and the user hasn't explicitly logged out of the app/IS3 they will transparently be authenticated again.


QA task: test condition assignment for factorial experiment in staging env

Test the factorial experiment Factorial AA test F1 - Java client library for MATHia. Participants should receive appropriate condition assignments

https://github.dev.carnegielearning.com/CarnegieLearning/ctp/pull/891/files <-- PR for Mathia changes
https://github.dev.carnegielearning.com/CarnegieLearning/ctp/blob/mathia-2023.8.61/runtime/services/services-business/services-section-variant-bl/src/main/java/cl/services/ct/sectionvariant/bl/SectionVariantExperimentServiceImpl.java <-- This is where assign and mark are called

Tested manually marking the Factorial assignment, I get an error.
#1001

When trying this, ask Matt to kick off the branch deployment.


Containers show synced in the synced containers sidebar but not in Multi-account containers tab of firefox

@ramkumar-kr I tried to use the latest version of your add on (version 1.1.2) and I see that the containers of Machine A (OSX 10.15.3 / FF 71.0) are showing up in the synced containers side bar of machine B (Win 10 / FF 71.0 64 bit) but they are not showing up in the Multi-Account containers tab. So I am not sure how do I use the container functionality on machine B

Thanks a lot for pointing it out. I was able to reproduce the issue and am currently working on fixing the bug.

I have released a new version 1.1.14 which fixes the issue. Please check and let me know if you are still facing the same issue. The steps I did are

Click on backup in machine A
Click on sync (firefox sync) in machine A
Click on sync (firefox sync) in machine B
Click on restore in machine B

Hi Ramkumar-Kr
Followed your steps but still see the same issue. Containers seem to be
synced in containers syc tab but not in the multi-account containers tab so
they seem to be cannot used
On Fri, Jan 3, 2020 at 7:35 PM Ram Kumar K R<EMAIL_ADDRESS>wrote:

I have released a new version 1.1.14 which fixes the issue. Please check
and let me know if you are still facing the same issue. The steps I did are

Click on backup in machine A
Click on sync (firefox sync) in machine A
Click on sync (firefox sync) in machine B
Click on restore in machine B

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ramkumar-kr/containers-sync/issues/13?email_source=notifications&email_token=ABGXU5CS5CRN7PLAJDHR2BTQ377W3A5CNFSM4KBRNXV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEICQAKA#issuecomment-570753064,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABGXU5AJIZ24D6QKAWGLJI3Q377W3ANCNFSM4KBRNXVQ
.

Also to confirm the version that I have is 1.1.4 and not 1.1.14 as per your comment.

This is strange. I tried to sync from Ubuntu to Windows with the latest version (1.1.4, sorry), and it created the containers properly. I cannot guarantee but uninstalling and installing the application again may solve the issue. I will update here if I get to reproduce the bug again.

Let me try uninstalling from all my machines and retry
On Thu, Jan 9, 2020 at 2:50 PM Ram Kumar K R<EMAIL_ADDRESS>wrote:

This is strange. I tried to sync from Ubuntu to Windows with the latest
version (1.1.4, sorry), and it created the containers properly. I cannot
guarantee but uninstalling and installing the application again may solve
the issue. I will update here if I get to reproduce the bug again.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ramkumar-kr/containers-sync/issues/13?email_source=notifications&email_token=ABGXU5DIFPNWLOA7CZWOJY3Q46S2HA5CNFSM4KBRNXV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEISCCHI#issuecomment-572793117,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABGXU5DRZBPWFA4KYW7QPB3Q46S2HANCNFSM4KBRNXVQ
.


Synchronization between multiple gpu

在代码中，计算rcnn_iou_new和rcnn_error_new只考虑到了单个GPU的样本，感觉这会造成一定统计的不稳定性，请问作者有考虑在多GPU之间进行一个同步吗？也就是rcnn_iou_new和rcnn_error_new的更新会考虑到整个batch的样本，而不是统计IMS_PER_GPU的样本

Thanks for your question.
Actually, I noticed this issue when I was implementing Dynamic R-CNN. From my viewpoint, adopting the statistics across the whole batch (multiple GPUs) will be more robust. However, I had no quota at that time.
I will add this feature when I am available. Contributions are also welcome!


Flutter 1.24

This PR has some work arounds for the Dart 2.12 beta bundled into Flutter 1.24 beta.

TBR @johnpryan


Video ist norworking

Issue description
Video on page ist not working. page: https://help.sap.com/docs/cloud-integration/sap-cloud-integration/sap-integration-advisor?locale=en-US
Feedback Type (Optional)
other
Page Title on SAP Help Portal (prefilled)
SAP Integration Advisor
Page URL on SAP Help Portal (prefilled)
https://help.sap.com/docs/PRODUCT_ID/368c481cd6954bdfa5d0435479fd4eaf/6b9fe2d753534bebadcfa9080228bd94.html?state=PRODUCTION&version=Cloud&locale=en-US

@ChristianW02  Thank you for your feedback, we will take a look and get back to you.

Hi @ChristianW02
Thank you for your feedback! We’ll look into it and come back to you if we have any questions.

Hello @ChristianW02 , thank you for the feedback. The video has been removed from the chapter.


Hotfix: delete the temporary delta file if there is an error

https://neondb.slack.com/archives/C03F5SM1N02/p1707066787740069

closing in favour of #6612


Refactor CacheAdapter to use filesystem as default

Filesystem is used as default, when cache/filesystem-adapter is installed and APCu is not available


perf: use arrays for low branching runes

This reduces the number of maps in the final trie structure by relying on arrays with a linear search for runes with less than 64 children.
Old:
BenchmarkParserAll/All-16          61072             19564 ns/op             576 B/op         55 allocs/op
BenchmarkParserSingle-16         1996647               518.8 ns/op            31 B/op          2 allocs/op
BenchmarkParserPutAll-16              64          16484833 ns/op        15168293 B/op     248607 allocs/op
BenchmarkParserPutSingle-16       405416              2707 ns/op             448 B/op         12 allocs/op

New:
BenchmarkParserAll/All-16          69675             16349 ns/op             576 B/op         55 allocs/op
BenchmarkParserSingle-16         2621797               440.9 ns/op            26 B/op          2 allocs/op
BenchmarkParserPutAll-16              93          12369175 ns/op         7014044 B/op     176055 allocs/op
BenchmarkParserPutSingle-16       438016              2609 ns/op             448 B/op         12 allocs/op

This also brought down the runtime memory usage from around ~18-19MB to ~7MB.


Option -i does not work with PotreeConverter compiled on Windows (works on linux though)

I could not convert point clouds using PotreeConverter on windows until I removed the "-i" ```
argument prefix before the input file argument in the PotreeCOnverter command line on Windows.
On windows :
PotreeConverter -i input.pts -o outputDir
does not work while
PotreeConverter input.pts -o outputDir
Works.
This behavior is not consistent with the documentation nor with the behavior of PotreeCOnverter under LInux.

Fixed in PR #335. Not sure why that was never merged... issue occurs on Linux when using --source as well when not first argument.

Hi,
Closing this because PotreeConverer 1.7 is now replaced by a complete rewrite, PotreeConverter 2.0, and the old version will no longer be worked on. The requested feature or issue may not apply to PotreeConverter 2.0, either because it is fixed or the feature is not yet ported.


nextjs version with ssg and ssr?

This looks so good, thank you. looking forward to use it as a starter for my new saas project.
How can I convert it to a nextjs project?
I know I can just copy the content in pages file, but there are some dependency like 'react-jss', 'react-feature', 'styles-jsx' etc and nextjs already equipped with most them so I'm little confused how should I configure the pages with minimal dependencies.

Move to #5.

Move to #5.


Add start menu entries for portable applications

Description of the new feature / enhancement
As discussed in this thread, portable applications now have a specification which enables standalone executables to be installed and added to the PATH. However, it would also be nice if there was an option to allow start menu shortcuts to be created for these applications.
Proposed technical implementation details
The implementation would be pretty similar to how portable applications are added to the PATH, except instead of a symlink a shortcut will be created with a title as specified by the manifest, and placed at %ProgramData%\Microsoft\Windows\Start Menu\Programs for machine installations or %AppData%\Microsoft\Windows\Start Menu\Programs for user installations. It might be a good idea in order to avoid conflicts to place shortcuts in a subfolder such as ..\WinGet.

We should look at arguments and settings for this particular scenario as well. Some command-line applications fail to run if they are executed without any arguments. For those applications, it doesn't necessarily make sense to have an icon. As stated in the linked thread, many portable applications with a GUI do make sense.
Users should be able to specify in settings if they want an icon or not. The default behavior should be specified by adding a value in the manifest, but settings or an argument would override.

Also allow those to be configured in the install sub-command for Winget, so that while those could be defaults, e.g., the user can have flags to enable or disable desktop icons, if he doesn't want them, or for example, if there was oversight by the package maintainers that made it so the packages didn't include a desktop icon or start menu entry when it was needed.

We should look at arguments and settings for this particular scenario as well. Some command-line applications fail to run if they are executed without any arguments. For those applications, it doesn't necessarily make sense to have an icon. As stated in the linked thread, many portable applications with a GUI do make sense.
Users should be able to specify in settings if they want an icon or not. The default behavior should be specified by adding a value in the manifest, but settings or an argument would override.

I know, that should be a manifest flag, and a bot can go through every portable WinGet package and open each app, and if it stays for more than 3 seconds, it’s a start menu app, then it closes it, repeat.

The default behavior should be specified by adding a value in the manifest, but settings or an argument would override.

I would propose adding a DesktopIcon and StartMenuEntry InstallerSwitch, as some installers (of all types) allow you to specify whether or not to create the desktop icon. Of course, since this isn't standard between publishers, sometimes the switch enables and sometimes it disables.
Proposal:
InstallerSwitches:
  Silent:
  |
  Upgrade:
  DesktopIcon:
    SwitchValue:  <value>
    SwitchMode:  <enable/disable>
  StartMenuEntry:
    SwitchValue:  <value>
    SwitchMode:  <enable/disable>

This would allow for the arguments --desktop-icon [true/false] and --start-menu-entry [true/false] which could also be controlled through user preference settings
For portable apps, if the SwitchMode is enable, it will add it to the desktop/startMenu, otherwise it will not, and the arguments and user preference would override

I dont the desktop icon should be a seperate option, as desktops usually get cluttered or the user has to manually delete the shortcuts.

@Trenly this looks like a simple issue to do, and the lack of this is annoying many users around the globe.

As @superusercode already stated, this is not all too complex. All you need is to not have manifest maintainers add shortcuts for CLI or non-portable applications and state as much like scoop already does (everything in scoop is portable with default buckets). Or just have the maintainers define each portable application as portable or portable-CLI.
Scoop uses start menu entries pointing to a /current/ symlink of the latest version folder.

I came here after trying several ways to try and make my winget links (e.g. geekuninstaller and syncthingtray) show up in my start menu. It seems the start menu refuses to resolve all manner of symlink and junction, so there's no way I can see to make this work with a simple user workaround.
As a user, I would expect that if winget can create a symlink to an exe in its Links folder after installation, it can just as well pair every one of those symlinks with an lnk shortcut in a start menu folder. Don't really see why it needs to be any more hassle than that.

I came here after trying several ways to try and make my winget links (e.g. geekuninstaller and syncthingtray) show up in my start menu. It seems the start menu refuses to resolve all manner of symlink and junction, so there's no way I can see to make this work with a simple user workaround.

@leaumar, to my knowledge, like how gio doesn't support symbolic links in applications:/ (instead solely supporting .desktop files) the Start Menu doesn't consistently support anything in Environment.SpecialFolder.CommonStartMenu except .lnk files. Why choose symbolic links or junctions?!

@RokeJulianLockhart Because I tried to put 2 and 2 together in the least interfering and lowest-maintenance way possible? Winget makes a folder outside the start menu in which it keeps tidy symlinks to portable apps. I need a folder inside the start menu with shortcuts to those apps. I figured I'd try to just symlink Links into the start menu. When that didn't show up, I thought maybe a junction would: heck if I know for what reason it had no effect when a symlink is supposed to be transparent. Only after testing some more I concluded the start menu doesn't support symlinks in any way, so even if I could merge the Links folder into it, the winget-made symlinks would do nothing either, just like explorer can't even resolve the app icon on a shortcut from them. It's all pretty disappointing really.

a symlink is supposed to be transparent. Only after testing some more I concluded the start menu doesn't support symlinks in any way

@leaumar, they're only transparent to applications which resolve paths above a specific layer, or don't deliberately filter them. You're probably looking for a hardlink, to achieve what you want to. It does seem weird to me that the developers would go to such effort to prevent them being used, though.

explorer can't even resolve the app icon on a shortcut from them

If you mean that explorer.exe doesn't support symbolic links, yes, but that's not a problem if you install an alternative file manager that does. KDE's Dolphin FM is an example (although IDK what its Windows support is like).

"why choose symbolic links" is an odd question when fixing this kind of "some files are over there but I need them here" is exactly what symlinks are for.

It becomes a less weird question to ask when you remember that you added 50% of your comment whilst I wrote mine, so I didn't notice nor read it (because I wasn't notified). Modifications to comments are best done when they're relatively minor, so try to write the entire comment before you post it.


[improve][broker]add ServerCnx state check before server handle request

Motivation
When the server receives a request, it is supposed to check its own state. like handleAck\handleCloseProducer\handleConnect\handleFlow\handleProducer
Modifications
Add ServerCnx state check when server handle request
Verifying this change

[x]  Make sure that the change passes the CI checks.

This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (no)
The public API: (no)
The schema: (no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

Documentation
Check the box below and label this PR (if you have committer privilege).
Need to update docs?

[x] doc-not-needed

/pulsarbot run-failure-checks

The change looks good me. Could you please help add unit test?

Sure.

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

LGTM with minor comment

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks


Further improvements to memory.c.

Compiler TLS is now used only used when the compiler supports it
If compiler TLS is unsupported, we use platform-specific TLS
Only one variable (an index) is now in TLS
We only access TLS once per alloc, and never when freeing
Allocation / release info is now stored within the allocation itself, by
over-allocating; this saves having external structures do the bookkeeping, and
reduces some of the redundant data that was being stored (such as addresses)
We never hit the alloc lock when not using SMP or when using OpenMP (that was
my fault)
Now that there are fewer tracking structures I think this is a bit easier to
read than before

I get core dumps with this patch applied unfortunately
I'm not seeing an actual performance difference

It should be about the same as using TLS from before, but should be as fast for systems that don't support compiler-generated TLS.  The delta between the two was about 15-20% before, now it should be around 1%.  What are you building with (just USE_THREADS=1)?

This patch is not quite right, on ARM64, it leads to improvements for small matrices (even without threading), but large matrix GEMM has gotten worse pretty uniformly (results on pixel2).
Old: 3313e4b
New: This patch.
Since I am testing single threaded performance on ARM64, none of the other changes has any effect.  These numbers are fairly repeatable too.
  I’ll check out the test; I wonder if it’s actualy starting too
many threads.  I did make the assumption that MAX_CPU_NUMBER is the max
number of threads that can be created, is that a false assumption?
At any rate, I put this up so that you all could take a look at it since it
has a fair number of changes.  I’ll dig back into it tomorrow, but thanks
for testing it out!

Just trying to redo the failed CI jobs locally to find the causes that are hidden by the QUIET_MAKE setting. (Removing that option only makes it hit the output limit in travis...)

Seems my problem with the fork utest is directly related to the small NUM_THREADS chosen at build time (4 on the dual-core laptop I used  yesterday, while the CI runs set NUM_THREADS=32). So perhaps it is this default that needs to be changed (although that test used to complete in seconds even with NUM_THREADS=4. Now if I just increase the limits in memory.c to get past the "too many threads" warning, I see one thread each spinning sched_yield with three others waiting on the lock in get_memory_table().)

The issue is actually that at fork() we need to reinitialize the memory table.  Well that and also increase the max thread number.
I think I've also resolved the performance issue for large matrices (needed to pad the alloc_t structure for proper alignment). I'll update the PR soon.

Ok updated. @sandwichmaker and I have both seen some small improvements now with large matrices on x86_64, and he's going to test with ARM hopefully later today.  This also resolves the fork test (that's a very handy test, btw, I never would have realized we needed to reinitialize things without it).

Compile-testing on mips32 now, not sure if I'll bother to run any benchmarks on that poor little router...

Whoops, missed the ALLOC_MMAP ifdef, that should have been below the new code (should fix windows), and also fixed the ifdef.  I should really get set up on more platforms....

Hmm... looks like GitHub's review system doesn't work like gerrit or our internal tools; you can't diff with the previous push, since you have to overwrite the commit.

I believe one can still view changes relative to your original commit (by choosing that option in the "Changes from" dropdown on the diff page), and reviewers have an additional option to "view changes since your last review" there.

Aha.  I think that will work if @sandwichmaker sends his comments as a review, then I can diff against that.  Right now it doesn't have anything in "changes since last review."

Something wrong with the Windows ::TlsGetValue/::TlsSetValue now, perhaps just missing an include ?

When I refactored the HAS_COMPILER_TLS block I left out the MSVC one to use __declspec(thread).  Those functions should be defined in winbase.h, which comes in from windows.h, which I believe is included?

nice cleanup @oon3m0oo.
@martin-frbg  dropping the ifdefs that come from MSVC pre 2005 is a good idea. Does that mean we will be able to get rid of the windows block for compilers without TLS?

If we dropped support for pre-2005 then yes, the windows block would go away, and we'd only have the pthreads part.
I believe the number is discussed in #1616, since this determines the number of threads.

I am always a bit sentimental about removing working code for old systems, there could be someone somewhere still relying on it. Twelve years may be a bit too old to be useful though (but it matches the ancient hardware still supported, Core2 would have been brand new back then).
In any case, removal should be done through a separate PR I think.

ready for merging ?

Hmm... before you merge this, I see some fortran test failures on my side with a couple of tests, let me see if that's this change.

Ok this was only in my internal branch that has lots of other changes to help build things internally, everything passes fine on my github repo.

The issue I see is an illegal instruction in zgemv.c when using STACK_ALLOC, where the allocation size requested is larger than MAX_STACK_ALLOC, which ends up creating an array of size 0, which is indeed illegal.  This doesn't seem to be related to the memory.c changes, though.

Sounds unrelated but decidedly unhealthy, probably best to create a new issue for that

Sent out #1631.


[WW-5401] Improves logging around wrapping request and detecting multipart request

Closes WW-5401

Totally forgot 🤦


Create "complete" user in fake data for development

This would make it quicker to get up to speed during development if make fake is used.
Probably blocked on https://github.com/gratipay/gratipay.com/issues/3837.

Duplicate of #3960.

Oops, submission glitch.


JSX Spread Attributes in IE-11 not working out of the box

Is this a bug report?
Yes I believe this is a bug, but may not be. Based on the documentation I believe so.

Did you try recovering your dependencies?

Yes I did, tested on
Node version 11.2.0
Node version 11.5.0
Which terms did you search for in User Guide?

The issue is with the spread operator for attributes
https://reactjs.org/docs/jsx-in-depth.html#spread-attributes
Environment

Cannot run this, in my env it returns the error ( even run as admin )
npm ERR! code ENOLOCAL
npm ERR! Could not install from "kyleg\AppData\Roaming\npm-cache\_npx\1848" as it does not contain a package.json file.

npm ERR!      C:\Users\kyle\AppData\Roaming\npm-cache\_logs\2018-12-26T05_33_31_814Z-debug.log
Install for create-react-app@latest failed with code 1

Steps to Reproduce

Try to use JSX spread Attributes on an app that will render in IE-11
EXAMPLE: from the website

function App2() {
  const props = {firstName: 'Ben', lastName: 'Hector'};
  return <Greeting {...props} />;
}

Expected Behavior

create-react-app should use babel env 'usage' to polyfill so that you do not get the error.
Object doesn't support property or method 'assign'
In IE-11
Actual Behavior

You get the error:
Object doesn't support property or method 'assign'
In IE-11
Not sure why, but images would not post, Images are hosted on imgur here:
https://imgur.com/a/qFTMRce
UPDATE:
so starting to realize there is very little reason to ever use this anyway... Maybe that is why no one has seemed to mention / notice this before. Basically deciding that there are far to many ways around this + seems like bad practice to use it anyway. You may want to look into this further since it does seem to be not supported by create-react-app out of the box, even though it is still a seemingly useless feature with so many more viable workarounds available.

Please see the documentation about supported browsers: https://facebook.github.io/create-react-app/docs/supported-browsers-features

Hey @Timer good to know, thank you!
Kyle


i con't output somefile.

when i use it in veldrid-raytracer on window server 2019,it con't output *.hlsl.bytes and *.450.glsl.spv file.
but it can output *.glsl *.glsle  *.hlsl *.metal file well . and the demo work on OpenGL and Vulkan as fine.

My guess is that Windows Server 2019 doesn't have the required Direct3D shader compiler, so that step will be skipped. For SPIR-V compilation to work, you need to install glslangvalidator -- the easiest way is to install the Vulkan SDK.

thank you reply me so quickly!
but,,  Windows Server 2019 it work Direct3D shader compiler well .
please see https://github.com/Siyy/BasicComputeShader demo.

YES!

When i installed the Win8 SDK by vs2017, it output the *.hlsl.bytes file. and when i restart the windows ,it can output .450.glsl.spv file now.
Did it out of fxc.exe possible in the near future instead by D3DCompiler_.DLL ?


Flynn CLI instalation will fail on Linux and/or in fish (shell)

Example installation script will fail on Linux if the targeted directory is not writable to user.
(/usr/local/bin/ is owned by root and has drwxr-xr-x permissions)
see: https://flynn.io/docs/cli#installation
$ L=/usr/local/bin/flynn && curl -sSL -A "`uname -sp`" https://dl.flynn.io/cli | zcat >$L && chmod +x $L
bash: /usr/local/bin/flynn: Permission denied
curl: (23) Failed writing body (0 != 16384)

Second issue is with running this script in fish (or any other shell), since this command is using bash syntax.
Therefore I would like to suggest improved version of this "cli instalation command" in following form:
sudo bash -c 'L=/usr/local/bin/flynn && curl -sSL -A "`uname -sp`" https://dl.flynn.io/cli | zcat >$L && chmod +x $L'

This will start new bash as a superuser - which will solve both problems (access rights + bash syntax)
However I'm not sure how this command will work on MAC machine.
Also I wanted to make pull-request for the corresponding DOCs, but I couldn't find it in repo.
https://flynn.io/docs/cli#installation

Hey, just tested this in OSX High Sierra with fish version 2.3.1. Works fine.

It also works fine on ZSH. Can you describe the shell issue you ran into?

@titanous He probably meant that only L=/usr/local/bin/flynn && curl -sSL -A "uname -sp" https://dl.flynn.io/cli | zcat >$L && chmod +x $L won't work in fish or zsh

It definitely works fine with zsh.

@titanous It does not in fish:
Unsupported use of '&&'. In fish, please use 'COMMAND; and COMMAND'.
fish: L=/usr/local/bin/flynn && curl -sSL -A "`uname -sp`" https://dl.flynn.io/cli | zcat >$L && chmod +x $L

@ndyakov @titanous Thanks for the tests :), I've updated the title.
Can you run the cli installation command without root elevation on Mac? If so, who is the owner and what are the rights on /usr/local/bin/ folder on Mac?

@Darkless012 works without sudo, the logged in user is the owner (i guess if it's in the admin group) admin is the group of /usr/local/bin

Flynn is unmaintained and our infrastructure will shut down on June 1, 2021. See the README for details.


Remove default values.

@srujzs ptal

Well for one, we ignore them, and if we didn't we'd have inconsistent semantics between tearoffs and static invocations.
But for another, for the same reason default values don't mean anything on abstract declarations. The 'defaultness' is implemented by the target of these externals, not by our interop layer.

Well for one, we ignore them, and if we didn't we'd have inconsistent semantics between tearoffs and static invocations.
But for another, for the same reason default values don't mean anything on abstract declarations. The 'defaultness' is implemented by the target of these externals, not by our interop layer.

even as documentation they are SUPER useful, I think.
Is there any downside to having them? (Other than potential conflicts with what the runtime system implements and what's in the IDL)

even as documentation they are SUPER useful, I think. Is there any downside to having them? (Other than potential conflicts with what the runtime system implements and what's in the IDL)

So far we've been saying that documentation for the DOM belongs outside of package:web i.e. in the MDN, so while I understand it's kind of nice to have this, removing them is consistent with our policy so far for the rest of package:web.
But really I'm removing this because it blocks rollout of package:web on Dart2Wasm. Specifically, we don't yet have implicit conversions for Dart types to JS types, so package:web currently fails to compile on Dart2Wasm. This CL fixes that issue. We can always revisit this after we have implicit conversions, but TBH I think we should have a general discussion on documenting package:web vs. relying on external documentation sources before we do so.

Okay. Just wanted to check. Thanks for the details!

Well for one, we ignore them, and if we didn't we'd have inconsistent semantics between tearoffs and static invocations.

Note that until we disallow tear-offs, we still have inconsistent semantics. The tear-off will use the default value while a direct call will not.
Another thing to note is that if the default value of a browser API changes, and our package has not yet, this is confusing for users who see a default value in the package. It never gets passed and the browser uses its own value.
In general, default values should be disallowed on external methods if we never use them. If we want documentation, I prefer we use comments to indicate what the default will be.

@srujzs – this should be flagged by the analyzer then! At least a warning, no?

@srujzs – this should be flagged by the analyzer then! At least a warning, no?

The plan is to indeed disallow them through the CFE (js_interop_checks) for newer syntax (inline classes).
Eventually it's worthwhile to port all our interop checks to analyzer so people get errors sooner, but that's part of a larger future effort. :)