Replacement character (\uFFFD) in JavaScript/TypeScript causes to breaks

🔎 Search Terms

TS1490: File appears to be binary
replacement character

🕗 Version & Regression Information
#57008 is incorrect on some real files.
⏯ Playground Link
https://github.com/micromark/micromark/pull/166
💻 Code
Note the � in the first 256 characters.
import assert from 'node:assert/strict'
import test from 'node:test'
import {micromark} from 'micromark'

test('nul', function () {
  assert.equal(
    micromark('asd\0asd'),
    '
asd�asd
',
    'should replace `\\0` w/ a replacement characters (`�`)'
  )
})

🙁 Actual behavior
error TS1490: File appears to be binary
🙂 Expected behavior
0xFFFD should be fine among other regular characters.
Additional information about the issue
Related-to: #21136.
Related-to: #56516.
Related-to: #57008.

Just write '\ufffd' instead? Putting the signifier that says "this file was corrupted" into a file, causing a tool for humans to say "this file looks corrupted", seems like correct behavior.

I always thought the entire point of the 0xFFFD codepoint was that it's only supposed to be generated by a text decoding failure and has no reason to ever appear literally in the source text.  Like Ryan says it's specifically designated as an indicator of corrupted text.

Situation: \ufffd exists because it shouldn't ever appear in a correctly-encoded file on purpose
"We need to put a raw \ufffd in a file so we can test it"
Situation: \ufffd exists in a correctly-encoded file on purpose
https://xkcd.com/927/

Really the thing we need is for Buffer.toString to say when it inserted a replacement character instead of directly reading one. Without that information that's not a lot we can do that isn't expensive.

Since we have a Buffer in sys.readFile (thanks to us dealing with BOMs and so on), we can technically write:
const buf = fs.readFileSync(p);
const asText = buf.toString("utf8");
const badDecode = Buffer.compare(buf, Buffer.from(asText, "utf8")) === 0;

We could do that conditionally based on the presence of U+FFFD, but then we don't have any way to bubble that information up (because the point would be to not error on parse because the lower-level thing said it was okay).

"exists because it shouldn't ever appear in a correctly-encoded file on purpose"
I don't think this is really the case. There are many algorithms in the html spec, and markdown spec, that do things that produce the replacement character.
It's like saying NaN shouldn't exist. Of course you don't want NaN normally but it has to exist as a concept. And that also means that you can check for it in code. And in tests. You have to be able to talk about it. The Unicode spec and Wikipedia and html and markdown need to be able to talk about the character.
Also, no other tool does what TS just started doing.
I'd think it's better to check for control characters instead of a replacement character.

Buffer#toString resulting in � does not mean that Buffer#toString is the only function that is ever allowed to result in � per Unicode.
Markdown for example, which has to deal with potentially malicious authors, has to make documents safe. So, the function markdown(input) also produces �.
Here are some more practical examples:

Here are some cases in WHATWG, https://github.com/search?q=org%3Awhatwg+"�"&type=code
16 hits in my personal code: https://github.com/search?q="�"+user%3Awooorm&type=code

Even if there were never �s in input, I also argue that Buffer#toString (implying Buffer#toString('utf8')) resulting in � does not mean that a file is binary. Whether a buffer is valid UTF-8 is not the same as whether a buffer is binary or not.
Looking on npm for “is binary” and checking out the code, yields:

https://github.com/gjtorikian/isBinaryFile/blob/main/src/index.ts
https://github.com/bevry/istextorbinary/blob/master/source/index.ts
https://github.com/sindresorhus/file-type

In the TS code base, there is access to the buffer. And UTF-8 is enforced already. So the bytes can be checked? Not too complex? https://github.com/wayfind/is-utf8/blob/master/is-utf8.js

(this is probably the rationale behind why direct tests against NaN don't work, come to think of it; it discourages writing them literally for any reason).

Off-topic, but NaN compares false to NaN because there are multiple bit patterns that can represent it and never equal is much better than sometimes equal.

Yes, I'm aware that's the theory - but I tend to suspect there's a more pragmatic rationale behind it too 😅  I don't think many people are pulling NaNs apart to inspect their low-level bit patterns, and if they are that's well outside the purview of the IEEE float spec, so from the perspective of normal FP operations the existence of multiple representations isn't even observable.

¯_(ツ)_/¯ I guess it can be two things

In the TS code base, there is access to the buffer. And UTF-8 is enforced already. So the bytes can be checked? Not too complex?

Everything can be fixed; the question is how much slower everyone's tsc should be in order for a handful of test files to not need to use escape sequences.

Indeed! But then I’d move one more step back: why are people passing 700mb video files through typescript? Apparently them adding an ignore pattern is not good enough.
The patch for that already only looks at the first 256 characters. For every file.
There is apparently code for BOMs too.
I’d wager that looking at 256 first bytes with something like https://github.com/gjtorikian/isBinaryFile/blob/main/src/index.ts, particularly when modified to actually do what the goal is (bail quickly when not UTF-8), will be so slow.

Looking more into the original issue, with the knowledge that many binary file formats have a “BOM-like” signature in the first few bytes: https://github.com/microsoft/TypeScript/issues/21136#issuecomment-359184809

0x47 is "G". I don't know why someone seeing an error when the letter G appears at two random offsets would be less surprised to see "file is binary" than you are when looking at �.

Right. You still can use those uncommon 2 bytes to go into a slightly slower path tho.

We're also not just trying to detect MPEG transport streams. There are other tools, e.g. Expo, which are/were emitting arbitrary binary files into .js file extensions and causing slowdowns because we tried to parse a giant garbage file.

Interesting! I dunno, detecting “arbitrary binary files” made by various tools is just going to be complex I think.
Right now TS throws. I assume that on giant binary files TS was also throwing.
Why not improve that crash with a better message, check whether something appeared binary, and suggest adding an ignore pattern?
Or also when files are like 1mb+, do a small byte check then?
Or when � appears with this recent patch.
And also, as @jakebailey mentions, comparing the buffer: https://github.com/microsoft/TypeScript/issues/57930#issuecomment-2018654475.
I feel like there are a couple cases where a slightly more thorough check can be done. And I’m not 100% that that check is that slow.

PRs accepted


AMQ-5082 unit test and patch

A proposed unit test and patch to address the issues raised in AMQ-5082 (the quorum never recovered after a broker's zookeeper session expires)

I've cleaned up the PR history, applied with thanks here: https://github.com/apache/activemq/commit/a39e51e0519852332f7779443c3db09b6e691d49
Can close this PR.


Difference between '== null' and 'is (not) null' in LINQ

Hi,
Are these expressions equivalent in LINQ, or does it depend on which LINQ provider is used?
// The tried and tested way
var query1 =
    from c in categories
    where c != null;

// C# 7 pattern matching
var query2 =
    from c in categories
    where !(c is null);

// C# 9 pattern matching
var query3 =
    from c in categories
    where c is not null;

Thanks!
/edit: not null was added in C# 9, not C# 8

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 587562bd-a0f5-0fb5-e9c1-8e6e6e142f2a
Version Independent ID: 9ed3467f-5274-5c22-fabb-f0ecc47eb5c2
Content: Handle null values in query expressions (LINQ in C#)
Content Source: docs/csharp/linq/handle-null-values-in-query-expressions.md
Product: dotnet-csharp
Technology: csharp-linq
GitHub Login: @BillWagner
Microsoft Alias: wiwagn

Thanks for writing this @StevenLiekens
This is an area we need to improve. The pattern matching syntax (and some other newer C# features) cannot be part of an expression tree. That means the is pattern matching operator cannot be used with a source that supports IQueryable<T>, only with a source that is IEnumerable<T>.  Here's your sample (slightly modified) and the compiler output in comments:
using System.Linq;

var categories = args.AsQueryable();

// The tried and tested way
var query1 =
    from c in categories
    where c != null
    select c;

// C# 7 pattern matching
var query2 =
    from c in categories
    where !(c is null) // error CS8122: An expression tree may not contain an 'is' pattern-matching operator.
    select c;

// C# 8 pattern matching
var query3 =
    from c in categories
    where c is not null // error CS8122: An expression tree may not contain an 'is' pattern-matching operator.
    select c;

If you remove the AsQueryable() call, all three queries compile and execute as expected.
We should address this behavior by describing what C# syntax isn't allowed in expression trees, and link to that new article from articles like this one pertaining to null checks.

Thanks, that answers my question perfectly. I'll leave this open in case you want to use this issue for tracking documentation updates.

I've suggested this in #11348.


Sporadically hanging mining tasks

It sometimes occurs that mining tasks appear to "hang". This is much more likely when using the update profile, but in general it can also occur for other profiles. Manual tests indicate that it is caused when evaluating >=1000 terms for very slow programs using the functions Evaluator::eval() or Evaluator::check().
Option 1
Measure the runtime for (almost) every term in the eval function. If it exceeds a limit, return with an error. The first problem with this is: the success of evaluating N terms for a program depends on the CPU power of the executing machine. The decision if a program is valid or better than an existing one would become non-deterministic. We don't want that. The second problem is: accessing the clock so often may have performance penality and is not a good design.
Option 2
Use a "watchdog" to monitor the mining process or thread. Kill it if it exceeds a limit. The problem here is that this we won't have a graceful shutdown of the app and that there might be platform-specific problems when killing a process or thread. We must make sure that, from the point of view of the wrapper app, loda exits successfully (exit code 0).

fixed


Domain separate event types by module name Overlaps

Background
While doing mutation tests on tokenfactory, I realized BurnDenom was emitting two events. After looking into this with Roman, we realized that this was due to overlap between the sdk's bank "burn" message and the tokenfactory "burn" message. To fix this, I changed the tokenfactory message name from "burn" to "tf_burn".
Suggested Design
As Roman suggested, it would probably be best for all module messages to have a module name prefix to prevent overlaps like this in the future. This is especially true for messages that might exist between multiple modules (such as burning and minting)
Acceptance Criteria
TBD

Is anyone working on this issue?? I could close it

@hieuvubk no one is working on this! I will assign you now, thanks!

@czarcas7ic  With the tokenfactory module, we use TypeMsg to emit events and msg names both, but unlike other modules which have their own msg name, their own event name. With such modules, should we just change the events name

@hieuvubk sorry I am not following, can you explain further?

@hieuvubk sorry I am not following, can you explain further?

https://github.com/osmosis-labs/osmosis/blob/main/x/tokenfactory/keeper/msg_server.go#L67-L72
Here use msg type to emit events.
And the difference:
https://github.com/osmosis-labs/osmosis/blob/main/x/superfluid/keeper/internal/events/emit.go#L23-L27
https://github.com/osmosis-labs/osmosis/blob/main/x/superfluid/types/events.go#L7
https://github.com/osmosis-labs/osmosis/blob/main/x/superfluid/types/msgs.go#L12
So I mean we just need to change the TypeEvt right?


[OpenGL] Invalid Debug text rendering using DebugText.Print method

Release Type: Official Release/GitHub
Version: At least <IP_ADDRESS>2
Platform(s): Any
Describe the bug
DebugText.Print method renders a black rectangle where text should render.
To Reproduce
Steps to reproduce the behavior:

Set StrideGraphicsApi csproj property to OpenGL
Invoke in any script  DebugText.Print method with the text of your choice
Run

Expected behavior
Correct rendered text
Screenshots

This happens in Windows as well.
The render doc shows the following error on the font's texture:

In comparison, this is what the DirectX version shows

Strange...

I ran this project using Rider and it seems to work....
However running the app with dotnet run does not


Migrate more structs from manually implemented to glib::wrapper! with BoxedInline

CC @bilelmoussaoui
See https://github.com/gtk-rs/gtk3-rs/pull/639
Similar things can also be done to other plain structs. Searching for #[repr(transparent)] and #[repr(C)] gives a few more candidates. Everything except for gdk::Atom should be doable.

gsk::RoundedRect and all these simple structs can also be converted


doc: fix syntax error for Overview code in file.md of rest-api-client

Why

Example code for Overview in file.md occurs syntax error.
What

Added } and format it.
How to test

Copy, paste, and execute it in your environment!!
Checklist

[x] Read CONTRIBUTING.md
[x] Updated documentation if it is required.
[ ] Added tests if it is required.
[ ] Passed yarn lint and yarn test on the root directory.

@shisama
Thank you for fixing!
You're right... and I found another mistake while testing the script.
Please check.


CustomProvider only supports tagged fields

Custom providers are not called for public, untagged fields.
type A {
   B string
}
func main() {
  faker.AddProvider("B", func(v reflect.Value) (interface{}, error) {
    return "XYZ", nil
  }
  var a A
  faker.FakeData(&a)
  if a.B != "XYZ" {
    panic("expected XYZ, got "+a.B)
  }
}

This is expected. The custom providers only support tagged fields. We can't check based on struct's property yet.

Is there a way to provide data through faker for structures imported from an external dependency, or is it restricted to only structures that are local to the package?

What do you mean? is it something like you mention above?
Or do you have any some usecase?

Sure! For example, say I have this code:
import "runtime"
type MyThing struct {
  runtime.BlockProfileRecord
}
func (m MyThing) Agg() int {
  return m.Count * m.Cycles
}
func (m MyThing) HorribleHash() int {
  var hash int
  for _, k := range m.Stack0 {
    hash += k %<PHONE_NUMBER>
  }
  return hash
}

and I'd like to test that Agg() and HorribleHash() work correctly. This is a trivial case; they can be tested entirely outside of faker, by using table driven tests or similar. Or, one could re-implement the functions' logic in the tests and verify randomly-generated faked data. There are cases, however, where we'd like to control the data, and supply a Provider.
However, as you say, Provider will not work here, because the BlockProfileRecord fields are not tagged and can't be tagged, because the source code is external to the project.
This is going to be the situation for any code that uses structures defined by an external library. We may want to test the code, but we can't tag the structures the code uses. When we can test using randomized data, this isn't an issue, but how do we handle cases where we would like to have a Provider for externally-defined structures?

I'll try to look into this in more detail. I can't promise much, it might be possible or it might not.
If you have any documents or references that could help, please help to post them here.  It will be great if we can support this!!!

I haven't looked through the code; are you using introspection to get at record fields? It's straightforward to get the public fields fields through reflect. Supporting providers for exported fields, regardless of marshelling tags, would be the least surprising for users, I think.


Memory Leak or just huge allocation?

I am using mysql as a database, this is headdump using spark.

Let me know if you want the whole 6gb dump.

The user housekeeper should be unloading the users after they quit the server.
Using the debug-logins config option will allow this to be monitored in the console / log files. Try enabling it to see.


Updating worker extensions dependencies

Updating worker extensions dependencies
Pull request checklist

[x] My changes do not require documentation changes

[ ] Otherwise: Documentation issue linked to PR

[ ] My changes should not be added to the release notes for the next release

[x] Otherwise: I've added my notes to release_notes.md

[x] My changes do not need to be backported to a previous version

[ ] Otherwise: Backport tracked by issue/PR #issue_or_pr

[x] I have added all required tests (Unit tests, E2E tests)

Additional information
Additional PR information

this is great - can the underlying WebJobs packages also be updated to fix the CVEs from the transitive dependencies? https://github.com/Azure/azure-webjobs-sdk-extensions/issues/901


Allow requesting additional scopes via the google_sign_in plugin

Use case
The GoogleSignIn APIs for Android, iOS, and Web all provide a mechanism to request additional scopes after sign-in. It would be helpful for the google_sign_in plugin to also provide a mechanism to check granted scopes and grant new scopes (i.e. optional scopes or scopes for new features).
Proposal
Offer two new APIs on GoogleSignIn:
Future<bool> hasGrantedScope(String scope); Which indicates if scope has already been granted by currentUser.
Future<bool> requestScope(String scope); Which requests scope be granted by the currentUser.

I'll try to put together a PR which adds these to the plugin, just wanted to start a conversation about the proposal first.

Feature is available on pub now.

Hey can I check if this is implemented and if there is a documentation for this as I couldn't find it anywhere online.


Update deps, fix FirebaseUI

@kroikie could you fix the instructions and then merge this?
This is in reference to issue #26

Integrating this PR requires some rearranging of the document that accompanies the code. Making those changes now. I will merge this once they are done.


Properly authenticate at proxy server

With this commit Rally will set the proper authentication header for
proxies that require it.
Closes #636

I am documenting how I tested this, in case we want to add an integration test later on for similar stuff:
cat >squid.conf <<EOF
auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/squidpasswords
auth_param basic realm proxy
acl authenticated proxy_auth REQUIRED
http_access allow authenticated
http_port 3128
EOF

Created a squidpasswords file using htpasswd -c squidpasswords testuser username testuser and password testuser with the following content:
cat >squidpasswords <<EOF
cat squidpasswords
testuser:$apr1$WQMVypS2$vUzNTPn6tKhTrRZP16/li.
EOF

and run:
docker run --rm --name squid -d -v $PWD/squidpasswords:/etc/squid/squidpasswords -v $PWD/squid.conf:/etc/squid/squid.conf -p 3128:3128 datadog/squid

and to see access logs:
docker exec -ti squid tail -f /var/log/squid/access.log

On the Rally side I first set:
export http_proxy=http://testuser:testuser@<IP_ADDRESS>:3128

Using current Rally master commit:
esrally --distribution-version=6.6.0 --track="geonames" --track-params="ingest_percentage:15" --challenge="append-no-conflicts-index-only"

shows on the squid log:
1550851149.662      0 <IP_ADDRESS> TCP_DENIED/407 3871 CONNECT github.com:443 - HIER_NONE/- text/html
1550851149.665      0 <IP_ADDRESS> TCP_DENIED/407 3871 CONNECT github.com:443 - HIER_NONE/- text/html
1550851149.667      0 <IP_ADDRESS> TCP_DENIED/407 3871 CONNECT github.com:443 - HIER_NONE/- text/html
1550851149.668      0 <IP_ADDRESS> TCP_DENIED/407 3871 CONNECT github.com:443 - HIER_NONE/- text/html

plus of course Rally itself reports:
[WARNING] No Internet connection detected. Automatic download of track data sets etc. is disabled.

Using this PR with the same Rally command, squid log shows:
1550851566.691  60860 <IP_ADDRESS> TCP_TUNNEL/200 86897 CONNECT github.com:443 testuser HIER_DIRECT/<IP_ADDRESS> -

and the warning is gone from the Rally output.

@dliappis great writeup! I have added integration tests now in https://github.com/elastic/rally/pull/652/commits/421f07cc4babe211ad9e7b4697d5bf1389976a8d based on your comment. Can you please have another look?


fix: otlp json encoding

Which problem is this PR solving?
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes https://github.com/open-telemetry/opentelemetry-js/issues/4216
Short description of the changes
Fixes the breakage introduced in https://github.com/open-telemetry/opentelemetry-js/pull/4062 where OTLP JSON encoding was encoding the intermediate low and high bytes directly to JSON, but in JSON mapping fixed64 can only be either a string or a number.
Removed usage of the parts taken from long.js and used BigInt instead with a fallback to the old worse precision hrTimeToNanoseconds from core if BigInt is not available.
Refactored the createExport{Trace,Metrics,Logs}ServiceRequest calls to unified options and an encoder (getOtlpEncoder), added tests for the encoder.
Type of change

[x] Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?
Tested manually against collector with gRPC, http/proto, http/json exporters.
Checklist:

[x] Followed the style guidelines of this project
[x] Unit tests have been added

but in JSON mapping fixed64 can only be either a string or a number.

That linked table seems to indicate that fixed64 is preferred to be a string, whereas fixed32 is preferred to be a number (although either would be accepted).

Is there any way to expedite getting this patch in?
For some users, the last release (1.17.1 and the experimental packages 0.44.0) is broken. Web-based JSON exporters in this release broke by sending data that does not conform to the JSON serialization of the protobufs expected by the OpenTelemetry Collector.
That release was on October 10, so we are getting close to a month now where we have a broken release as the "latest release".
Could we please get this fix merged and a new release (1.17.2?  0.44.1?) pushed ASAP?


Auto-detect the OS type in Makefile

Hi Kishori,
Here's some logic for the Makefile to auto-detect the OS type.
Cheers,
~T

I added some more fixes, like removing redundancy in the Makefile, and removing the hard-coded S3 path.


Called From Here

Describe your problem here.
C:\HaxeToolkit\haxe\std/sys/Http.hx:534: characters 4-9 : Uncaught exception Http Error #404
C:\HaxeToolkit\haxe\std/sys/Http.hx:67: characters 4-14 : Called from here
C:\HaxeToolkit\haxe\std/sys/Http.hx:253: characters 12-25 : Called from here
C:\HaxeToolkit\haxe\std/sys/Http.hx:70: characters 3-30 : Called from here
C:\HaxeToolkit\haxe\std/sys/Http.hx:536: characters 3-19 : Called from here
C:/HaxeToolkit/haxe/lib/SScript/4,1,0/src/tea/backend/SScriptVer.hx:29: characters 45-142 : Called from here
C:/HaxeToolkit/haxe/lib/SScript/4,1,0/src/macro/Macro.hx:44: characters 3-21 : Called from here
(unknown) : Called from here

Are you modding a build from source or with Lua?
Source
What is your build target?
Windows x64
Did you edit anything in this build? If so, mention or summarize your changes.
Yes, I tried to:
Add an achievement (Achievements.hx, PlayState.hx)
Add a new gameplay changer (GameplayChangersSubstate.hx, ClientPrefs.hx)
etc.

update sscript with haxelib install SScript 6.1.80

Thank you

It's showing me this now!
source/psychlua/HScript.hx:168: characters 89-94 : Type not found : SCall

Nevermind, no hscript


Enable to keep ECS task singleton while deploy/reconcile

What would you like to be added:
Having an option to enable keeping ECS tasks singleton while deploy/reconcile. Just same as Kubernetes recreate option, instead of rolling out new task/taskset before deleting the old one, we want to keep only one task running at a time.
Why is this needed:

@khanhtc1202
Could we close this issue or not (cuz it seems solved)??

#4608 solved this.


Change the running indicator

This is from Milamber work on hdpi (BUG 58426).
This patch does not contains the hdpi part but only the icon change

@milamberspace , I leave it to you ?

@benbenw Thanks for your PR. I've just merge the #159 with include the change for the status icon. You can close your PR
@pmouawad thanks for the ping.


Pin/Limit Namespaces

Description
I don't have access to read the default namespace (or metrics) on my clusters, and would prefer not to get all resources from all namespaces when I connect to one.
Feature Request
Provide option to pre-filter namespaces before initiating connections to list resources on a cluster. I recall Lens had this feature. k9s does with -n flag + --context, similar to kubectl
Would be satisfied if a ServiceAccount / RoleBinding token worked. We use Azure/kubelogin and/or aws-iam-authenticator for Auth, though.
Additional Request
Would like to be able to somehow pin/favorite a namespace, rather than have to search through a long list.

Hello @OneCricketeer
Thanks a lot for giving it a try and for the detailed explanation of the feature request. We’ll definitely look into what we can do in the next release—we’ve actually been thinking about this use case too!


Crash: [GULHeartbeatDateStorage checkAndCreateDirectory:fileCoordinator:]

[REQUIRED] Step 1: Describe your environment

Xcode version:  12.4
Firebase SDK version: 7.6.0
Installation method: CocoaPods
Firebase Component: Crashlytics

[REQUIRED] Step 2: Describe the problem
Getting plenty of crashes on [GULHeartbeatDateStorage checkAndCreateDirectory:fileCoordinator:]
Steps to reproduce:
Random crashes occurs, not able to reproduce it in debug.
Relevant Code:
Fatal Exception: NSInvalidArgumentException
0  CoreFoundation                 0x1834ae9d8 __exceptionPreprocess
1  libobjc.A.dylib                0x197817b54 objc_exception_throw
2  CoreFoundation                 0x1833bebbc -[NSOrderedSet initWithSet:copyItems:]
3  CoreFoundation                 0x1834b101c ___forwarding___
4  CoreFoundation                 0x1834b2f8c _CF_forwarding_prep_0
5  Foundation                     0x184805df4 __85-[NSFileCoordinator(NSPrivate) _coordinateWritingItemAtURL:options:error:byAccessor:]_block_invoke.328
6  Foundation                     0x18469ceb4 -[NSFileCoordinator(NSPrivate) _withAccessArbiter:invokeAccessor:orDont:andRelinquishAccessClaim:]
7  Foundation                     0x18469ef70 -[NSFileCoordinator(NSPrivate) _coordinateWritingItemAtURL:options:error:byAccessor:]
8  Foundation                     0x18469ecb8 -[NSFileCoordinator coordinateWritingItemAtURL:options:error:byAccessor:]
9  *******                        0x10444759c +[GULHeartbeatDateStorage checkAndCreateDirectory:fileCoordinator:] + 81 (GULHeartbeatDateStorage.m:81)
10 *******                        0x1044473e4 -[GULHeartbeatDateStorage fileURL] + 51 (GULHeartbeatDateStorage.m:51)
11 *******                        0x1044478cc -[GULHeartbeatDateStorage heartbeatDateForTag:] + 119 (GULHeartbeatDateStorage.m:119)
12 *******                        0x10433791c +[FIRHeartbeatInfo updateIfNeededHeartbeatDateForTag:] + 31 (FIRHeartbeatInfo.m:31)
13 *******                        0x104337a28 +[FIRHeartbeatInfo heartbeatCodeForTag:] + 45 (FIRHeartbeatInfo.m:45)
14 *******                        0x104362ca0 __96-[FIRInstallationsAPIService requestWithURL:HTTPMethod:bodyDict:refreshToken:additionalHeaders:]_block_invoke + 278 (FIRInstallationsAPIService.m:278)
15 *******                        0x104317828 __38+[FBLPromise(DoAdditions) onQueue:do:]_block_invoke + 33 (FBLPromise+Do.m:33)
16 libdispatch.dylib              0x1830e6298 _dispatch_call_block_and_release
17 libdispatch.dylib              0x1830e7280 _dispatch_client_callout
18 libdispatch.dylib              0x1830991a8 _dispatch_root_queue_drain
19 libdispatch.dylib              0x183099704 _dispatch_worker_thread2
20 libsystem_pthread.dylib        0x1cbbe3568 _pthread_wqthread
21 libsystem_pthread.dylib        0x1cbbe6874 start_wqthread

Crashed: com.google.firebase.crashlytics.ios.exception
0  *******                        0x10434f88c FIRCLSProcessRecordAllThreads + 397 (FIRCLSProcess.c:397)
1  *******                        0x10434fc70 FIRCLSProcessRecordAllThreads + 428 (FIRCLSProcess.c:428)
2  *******                        0x1043499e4 FIRCLSHandler + 34 (FIRCLSHandler.m:34)
3  *******                        0x10434578c __FIRCLSExceptionRecord_block_invoke + 218 (FIRCLSException.mm:218)
4  libdispatch.dylib              0x1830e7280 _dispatch_client_callout + 16
5  libdispatch.dylib              0x183096b8c _dispatch_lane_barrier_sync_invoke_and_complete + 56
6  *******                        0x104344f24 FIRCLSExceptionRecord + 225 (FIRCLSException.mm:225)
7  *******                        0x10434521c FIRCLSExceptionRecordNSException + 111 (FIRCLSException.mm:111)
8  *******                        0x104344b04 FIRCLSTerminateHandler() + 279 (FIRCLSException.mm:279)
9  libc++abi.dylib                0x197913154 std::__terminate(void (*)()) + 16
10 libc++abi.dylib                0x1979130ec std::terminate() + 44
11 libobjc.A.dylib                0x197817d88 _objc_terminate() + 10
12 libdispatch.dylib              0x1830e7294 _dispatch_client_callout + 36
13 libdispatch.dylib              0x1830991a8 _dispatch_root_queue_drain + 996
14 libdispatch.dylib              0x183099704 _dispatch_worker_thread2 + 124
15 libsystem_pthread.dylib        0x1cbbe3568 _pthread_wqthread + 212
16 libsystem_pthread.dylib        0x1cbbe6874 start_wqthread + 8

com.apple.main-thread
0  libsystem_kernel.dylib         0x1af362644 mach_msg_trap + 8
1  libsystem_kernel.dylib         0x1af361a48 mach_msg + 72
2  CoreFoundation                 0x18342e0fc __CFRunLoopServiceMachPort + 376
3  CoreFoundation                 0x183428570 __CFRunLoopRun + 1176
4  CoreFoundation                 0x183427ba0 CFRunLoopRunSpecific + 572
5  GraphicsServices               0x19a165598 GSEventRunModal + 160
6  UIKitCore                      0x185d173d8 -[UIApplication _run] + 1052
7  UIKitCore                      0x185d1c958 UIApplicationMain + 164
8  *******                        0x1020c65d8 main +<PHONE_NUMBER> (<compiler-generated>:4299105752)
9  libdyld.dylib                  0x183106568 start + 4

com.google.FIRCoreDiagnostics
0  libsystem_kernel.dylib         0x1af362680 semaphore_wait_trap + 8
1  libdispatch.dylib              0x18308a18c _dispatch_sema4_wait$VARIANT$mp + 24
2  libdispatch.dylib              0x18308a7e0 _dispatch_semaphore_wait_slow + 136
3  Foundation                     0x1846b31a8 -[NSFileCoordinator(NSPrivate) _blockOnAccessClaim:withAccessArbiter:] + 356
4  Foundation                     0x18469ef18 -[NSFileCoordinator(NSPrivate) _coordinateWritingItemAtURL:options:error:byAccessor:] + 568
5  Foundation                     0x18469ecb8 -[NSFileCoordinator coordinateWritingItemAtURL:options:error:byAccessor:] + 92
6  *******                        0x10444759c +[GULHeartbeatDateStorage checkAndCreateDirectory:fileCoordinator:] + 81 (GULHeartbeatDateStorage.m:81)
7  *******                        0x1044473e4 -[GULHeartbeatDateStorage fileURL] + 51 (GULHeartbeatDateStorage.m:51)
8  *******                        0x1044478cc -[GULHeartbeatDateStorage heartbeatDateForTag:] + 119 (GULHeartbeatDateStorage.m:119)
9  *******                        0x10433b87c -[FIRCoreDiagnostics setHeartbeatFlagIfNeededToConfig:] + 531 (FIRCoreDiagnostics.m:531)
10 *******                        0x10433b6fc __42-[FIRCoreDiagnostics sendDiagnosticsData:]_block_invoke + 516 (FIRCoreDiagnostics.m:516)
11 libdispatch.dylib              0x1830e6298 _dispatch_call_block_and_release + 24
12 libdispatch.dylib              0x1830e7280 _dispatch_client_callout + 16
13 libdispatch.dylib              0x18308fdcc _dispatch_lane_serial_drain$VARIANT$mp + 612
14 libdispatch.dylib              0x1830908a8 _dispatch_lane_invoke$VARIANT$mp + 424
15 libdispatch.dylib              0x18309a338 _dispatch_workloop_worker_thread + 712
16 libsystem_pthread.dylib        0x1cbbe35a4 _pthread_wqthread + 272
17 libsystem_pthread.dylib        0x1cbbe6874 start_wqthread + 8

com.google.firebase.crashlytics.MachExceptionServer
0  libsystem_kernel.dylib         0x1af362644 mach_msg_trap + 8
1  libsystem_kernel.dylib         0x1af361a48 mach_msg + 72
2  *******                        0x10434bfe0 FIRCLSMachExceptionServer + 194 (FIRCLSMachException.c:194)
3  libsystem_pthread.dylib        0x1cbbe1b3c _pthread_start + 288
4  libsystem_pthread.dylib        0x1cbbe6880 thread_start + 8

Hi @cozelight, thanks for the report and sorry for the trouble. We'll look into this and get it addressed!
Seems like it could be related to #5969

@cozelight Could you please share a couple additional details to help us with the investigation:

what platform is it (iOS/tvOS/macOS/etc.)
could you share the exception caused the crash (should be available in the crash report)
does it happen only for the app first launch or for consequent launches as well
could you share a sample project where the crash is reproducible? It will significantly help the investigation.

A possible reason (still to be validated):

we may create several subdirectories when GULHeartbeatDateStorage is used for the first time
we coordinate access to with NSFileCoordinator only to the child dir which may potentially lead to the NSFileCoordinator unexpected behaviour.

Hi @cozelight,
A potential fix for this issue (as part of GoogleUtils 7.4) was planned for release in Firebase 7.11 but is being pulled due to some technical challenges. The team will be reevaluating the issue and is looking forward to the Firebase 8 release.
Apologies for the delay and thank you again for the report. In the meantime, please keep us in the loop with any further details that may aid in the reproduction/investigation of the issue!


variable $message with blade in emails

when i use Mail::send function and pass array to the second parameter , laravel extract this array to variables but if the array has key "message" laravel duplicate it with Message object
the solution is to change the array key from message to _message .

As I explained on the PR, this is intentional. We're passing this through for people to actually use. It's documented in a red box on our docs. ;P


Support for lazy evaluation of expensive binary annotations

It'd be nice if there was a way to add binary annotations to spans in such a way that the annotation value is only computed if the trace is being sampled.
For example:
localTracer.submitBinaryAnnotation("foo", computeExpensiveValue());

would become:
localTracer.submitBinaryAnnotation("foo", () -> computeExpensiveValue());

Without this feature support, we're forced to do:
if (Boolean.TRUE.equals(tracer.serverSpanThreadBinder().getCurrentServerSpan().getSample())) {
   localTracer.submitBinaryAnnotation("foo", computeExpensiveValue());
}

@schlosna @eirslett @NegatioN @kristofa @ewhauser anybody else interested in this?

I don't think we have many computationally expensive values being traced at this point, but I wouldn't object to having something that solves that problem. 👍
However I'm also curious about what some examples of these computationally expensive values are? On the top of my head I can't think of a use-case where that might happen for us in the near future.

one I hear about often is address resolution, but also interested in the
motivation!

For me, the need arose from having MongoDB queries as a binary annotation for the span.
The MongoDB query would be a BsonDocument (similar to JsonNode from Jackson), which could be a very big structure. To present it as a String, serialization needs to take place and that's the expensive operation I'd like to avoid unless absolutely needed.

For me, the need arose from having MongoDB queries as a binary annotation
for the span.
The MongoDB query would be a BsonDocument (similar to JsonNode from
Jackson), which could be a very big structure. To present it as a String,
serialization needs to take place and that's the expensive operation I'd
like to avoid unless absolutely needed.

thx.. unrelated, but is any of this code on github? I'd like to see if for
vetting a different idea.

I've extracted the relevant bits here: https://gist.github.com/AlejandroRivera/381cd2121462bd3d90f1e5253899c876
Hope it helps :)

I think this design I'm working on should help address this issue https://github.com/openzipkin/brave/issues/284

Brave 4.3 is syncing now and includes HttpTracing component. This lazy tags by default


[FEATURE REQ] Add dogfood Environment as a Test Resource Target

Describe the bug
When trying to create test resources using this script .\eng\common\TestResources\New-TestResources.ps1 on Azure, only environments that are supported are following.
'AzureCloud', 'AzureUSGovernment', 'AzureChinaCloud'
The validation list does not support dogfood environment.
Expected behavior
It would be good to have dogfood supprot as well.
Actual behavior (include Exception or Stack Trace)
When Environment parameter with the value dogfood is added to the script, script returns exception of not supported
To Reproduce
Steps to reproduce the behavior (include a code snippet, screenshot, or any additional information that might help us reproduce the issue)
Run the command below with Environment parameter equal to dogfood
.\eng\common\TestResources\New-TestResources.cmd -TenantId <tenantid> -BaseName <username> -ServiceDirectory <serviceName> -TestApplicationId <testAppId> -Location <region> -environment dogfood -ProvisionerApplicationId <appId> -ProvisionerApplicationSecret <appSecret> -SubscriptionId  <subscriptionid>
Environment:

Name and version of the Library package used: [e.g. Azure.Storage.Blobs 12.2.0]
Hosting platform or OS and .NET runtime version (dotnet --info output for .NET Core projects): Windows 10
IDE and version : powershell command line

Repo: -pr
Branch: feature/communication

PR open: https://github.com/Azure/azure-sdk-for-net-pr/pull/1214/files

PRs allowing this have been merged.


chore(www): create outline_server_repository folder

I've split up the outline_server file into several parts (outline_server_repository, outline_server, and outline_server_access_key) and put them into a subfolder.
Also, I've given all "util" files more meaningful names.

BTW, I started renaming server to service and it quickly became a several thousand line change. I think we should probably save that for the end.


S

Replace this paragraph with a description of your changes and rationale. Provide links to external references/discussions if appropriate.

Resolves SR-NNNN.

Please don't open pull requests that target/originate from branches you are not actively committing to.


Update sample to use compiled framework instead of source files

Other than the comments, if it compiles well then it looks good to me!


[Doc-16732][Development] development-environment-setup improvement

Purpose of the pull request
close #16732
Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

(or)
Pull Request Notice
Pull Request Notice
If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

It's better to add the reason why developer need to add this options.

This is the guidance document for starting the development environment. I don't quite understand why we should add reasons since this is not a code interpretation document..

It's better to add the reason why developer need to add this options.

This is the guidance document for starting the development environment. I don't quite understand why we should add reasons since this is not a code interpretation document...

fine, just nip.


Cyber message is received twice

Describe the bug
I'm sending out cyber proto messages and the message is received in a component's Proc function. Our message has a continuously increasing counter.
In some cases, an old message is received again.
In the receiving function, the received counter will be like:
...
1234
1235
1236
1237
1238
1239
1240
1237 <-- 1237 is received twice
1242 <-- after the duplication of 1237, 1241 is missed.
1243
...

Desktop (please complete the following information):

OS: Ubuntu 20
Version 7.0

I'd like to check if this is related to

Components' Proc() ? Will compoment cache some message?
If I use CreateReader() directly, will it make difference?

Cyber QoS setting?
Cyber use fast-dds, can I change the qos policy in cyber?

Thanks.

On arm platform  “cyber/transport/shm/condition_notifier.cc” has a bug，add "_sync_synchronize()" after indicator->infos[idx] = info;


Modal enhancements

This pull request includes several enhancements to modal:
A variation of Lidbetter's responsive modal pull request that will work with vue.js 1.x.
Allows null width for responsive modals.
Relational widths like '80%' or '5em' are now allowed as well.
Numeric widths will be treated as pixels.
Small and Large options to enable the modal-sm and modal-lg classes respectively.
Clean up of documentation.

Thanks for this!


This 1 line of TS code makes the editor unusable, cpu 100%

Type: Bug
Started noticing the editor was unusable and CPU went to 100%. Narrowed it down to the one commented line below. Confirmed that this happens with --disable-extensions, seems it's an issue with TS itself?
  const formHelpers = createForm({
    // initialValues: startValues,
    onSubmit: (values) => {
      console.log(values);
    },
  });

VS Code version: Code 1.83.1 (f1b07bd25dfad64b0167beb15359ae573aecd2cc, 2023-10-10T23:45:31.402Z)
OS version: Linux x64 6.2.0-35-generic
Modes:

System Info

Item
Value

CPUs
AMD Ryzen 9 7950X 16-Core Processor (32 x 5489)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: disabled_offdirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmultiple_raster_threads: enabled_onopengl: enabled_onrasterization: enabledraw_draw: disabled_off_okvideo_decode: enabledvideo_encode: disabled_softwarevulkan: disabled_offwebgl: enabledwebgl2: enabledwebgpu: disabled_off

Load (avg)
1, 1, 1

Memory (System)
62.53GB (59.15GB free)

Process Argv
--disable-extensions project --crash-reporter-id 099379be-f15c-45a1-a4c7-b487f53f2ffb

Screen Reader
no

VM
0%

DESKTOP_SESSION
ubuntu

XDG_CURRENT_DESKTOP
Unity

XDG_SESSION_DESKTOP
ubuntu

XDG_SESSION_TYPE
wayland

Extensions disabled
A/B Experiments
vsliv368:30146709
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
vswsl492cf:30256860
vslsvsres303:30308271
vserr242cf:30382550
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscoreces:30445986
vscod805cf:30301675
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593cf:30376535
pythonvs932:30410667
py29gd2263cf:30856253
vsclangdf:30486550
c4g48928:30535728
dsvsc012:30540252
pynewext54:30695312
azure-dev_surveyone:30548225
3biah626:30602489
f6dab269:30613381
2i9eh265:30646982
showlangstatbar:30737416
0bi6i642:30841073
03d35959:30757346
pythonfmttext:30731395
fixshowwlkth:30771522
showindicator:30805244
pythongtdpath:30769146
i26e3531:30792625
gsofa:30866041
pythonnosmt12:30797651
pythonidxptcf:30805731
pythonnoceb:30805159
copilotsettingt:30859503
dsvsc013:30795093
dsvsc014:30804076
diffeditorv2:30821572
dsvsc015:30845448
pyreplss1:30865275
pythontbext0cf:30864173

Seems fine for me:

We need to see all the packages you depend on too in this case. Try putting together a small, self contained project which demonstrates the issue

This 1 line of TS code makes the editor unusable

Lock up VS Code using this one weird trick! TypeScript compiler developers hate him!

@RyanCavanaugh @mjbvz Here's a repro you can play with. I've stripped the codebase down significantly. The app is not meant to build, only to show how VS Code goes to 100% CPU usage when uncommenting this line. Tested on 2 different machines.


Memory Leaks

This is a log from Leaks, a program that finds memory leaks in a given process. This may be related to #10.
Rendered 254 characters
Set option 2004 to h
Set option 12 to l
Set option 2004 to h
Set option 2004 to l
Set option 2004 to h
Process:         yterm [20020]
Path:            /Users/USER/*/yterm
Load Address:    0x10f717000
Identifier:      yterm
Version:         0
Code Type:       X86-64
Platform:        macOS
Parent Process:  leaks [20019]

Date/Time:       2023-08-12 11:55:44.759 -0400
Launch Time:     2023-08-12 11:55:39.951 -0400
OS Version:      macOS 13.4.1 (22F82)
Report Version:  7
Analysis Tool:   /usr/bin/leaks

Physical footprint:         38.5M
Physical footprint (peak):  39.8M
Idle exit:                  untracked
----

leaks Report Version: 4.0, multi-line stacks
Process 20020: 20034 nodes malloced for 4107 KB
Process 20020: 2 leaks for 23568 total leaked bytes.

STACK OF 1 INSTANCE OF 'ROOT LEAK: <malloc in SafeMalloc>':
5   dyld                               0x7ff811dc841f start + 1903
4   yterm                                 0x10f71d80c main + 140  main.c:83
3   yterm                                 0x10f71af0a App_Update + 826  app.c:155
2   yterm                                 0x10f72521a TextScreen_Resize + 42  textScreen.c:154
1   yterm                                 0x10f723b05 SafeMalloc + 21  safe.c:6
0   libsystem_malloc.dylib             0x7ff811f70149 _malloc_zone_malloc_instrumented_or_legacy + 88
====
    1 (23.0K) ROOT LEAK: <malloc in SafeMalloc 0x7fbeff20a000> [23552]

STACK OF 1 INSTANCE OF 'ROOT LEAK: <malloc in SafeMalloc>':
7   dyld                               0x7ff811dc841f start + 1903
6   yterm                                 0x10f71d7ad main + 45  main.c:79
5   yterm                                 0x10f71a901 App_Init + 193  app.c:28
4   yterm                                 0x10f71aa5a App_AddTab + 186  app.c:61
3   yterm                                 0x10f72432d Terminal_Init + 77  terminal.c:10
2   yterm                                 0x10f725871 DupString + 33  util.c:38
1   yterm                                 0x10f723b05 SafeMalloc + 21  safe.c:6
0   libsystem_malloc.dylib             0x7ff811f70149 _malloc_zone_malloc_instrumented_or_legacy + 88
====
    1 (16 bytes) ROOT LEAK: <malloc in SafeMalloc 0x7fbefe16d480> [16]  length: 5  "yterm"

...

first leak should be fixed in 5c54634

I'll close the issue.
For whatever reason the title of the application doesn't get handled. The same issue showed with GLFW and I made an issue for them too. It's been about a year since I made it and it still hasn't been resolved yet. Im convinced that it's a MacOS internals thing.


[Question] - Envoy proxy stops if jaeger not running

Hi - I am working of the github example of jaeger native tracing and running a similar example in a container.
https://github.com/envoyproxy/envoy/tree/master/examples/jaeger-native-tracing
Overall it works fine. The problem i am facing is that if jaeger is not running (which is true for my usecase in some of the internal environments) - the envoy stops with this error and no further communication is possible with the main process.
[2019-10-17 20:23:44.173][8][info][config] [source/server/configuration_impl.cc:105]   loading tracing driver: envoy.dynamic.ot [2019-10-17 20:23:45.199][8][critical][main] [source/server/server.cc:93] error initializing configuration 'envoy.yaml': opentracing: invalid configuration: Error resolving address: Name or service not known
Is there any way to to change this behavior so that it doesn't stop envoy in case it doesn't find jaeger?

Figured out by looking at the client here
https://github.com/jaegertracing/jaeger-client-cpp
the value needs to be set as empty string.


Fixed database seeding error, along with page loading error yeet

We did it!

hey, thats pretty good!


LSTM for System Identification task

Hello together,
For a final thesis I am working with Keras on Tensorflow and have to investigate how LSTM Networks can be used to identify dynamical systems.
I have given one dimensional Input Data X(t) and the depending Output Data Y(t).
First of all, I preprocessed the data in terms of normalizing in a range form (-1,1).
From this I made time depending samples, using 10 values of the Input data to get the time dependency in my network.
So da Data looks like:      x(t) = (0......86138)     (Datapoints)
u(0)=(x(t), x(t-1), .... x(t-10))
u(1)=(x(t+1), x(t), x(t-1) .... x(t-10))
....
u(86129)=........
It works like shifting a Window with a length of 10 over the data x(t) and take a snap every step.
My new input data for the NN will then be:            input (86129,10,1)
The output data gets formated to:                          target (86129,1)
--> so basically for each output value, there is a input vector of size 10.
My question about this task is:
As I am relatively new to machine learning and keras, am I doing right so far with preprocessing and do you think that LSTM Networks can leed to better results than general FeedForward ones?
....
model = Sequential()
model.add(LSTM(units=100, activation='tanh', input_shape=(10,1)))
model.add(LSTM(units=100, activation='tanh')
model.add(Dense(units=1, activation='linear')
model.compile(loss='mean_squared_error', optimizer='Adam')
model.fit(input, target, epchs=100, batch_size=1000, verbose=1)
....
The results are quite bad, even worse than using a Dense layers instead of LSTM layers. Is there any thing you would reccomend to change here? Someone recommended to use Sequence2Sequence model structure, but I only can find examples for translation tasks and so on.
I am really greatfull for any help you can give me.
Thanks in advance.

Hi,
I need to develop LSTM networks for SI for my project as well.  I would like to know if you have published your results anywhere.  If not, whats the status?

I am still working on the project. The results are not yet published in any way. If you want I can link you to some resources that I found especially helpful coming from a system identification background.

Thanks. I would like that.   i'm trying the same with Tensorflow using the seq2seq approach--i'm still tinkering with TF as this is my first encounter.
Since you had asked about input data above, what did you finally use?-- I ask in case there is something i missed.  The paper by Wang uses a sum of sinusoids which seems like a standard procedure since the paper by Narendra and Parthasarathy in 1990.

Sry for late answering. I am still working on that Problem.  I haven't had any success so far.
I also found the paper, you are talking about, but its mostly about a convex based LSTM structure.
What i read so far about sequence2sequence models sounds promising, but since I am pretty new to this topic I don't have much expertise. I am using a dataset with about 85000 datapoints. Also tried it with data of a simple dynamic system, without success.
What I am struggeling with, is how to use the internal dynamics of recurrent neural networks (such as LSTM) for the system identification task. Only feed forward neural networks with external dynamics (such as time delay networks) led to quite good results.
But maybe the sequence2sequence idea is the key. Hit me up if it works with your projects.

Is your project academic?
I am pretty much stuck in the same problem, I dont understand how Tensorflow and the LSTM community defines batches and time instances.   This  post looks interesting however.  What are your thoughts on the same?

The explanation in the post you've linked seems right and confirms what I have experienced so far. Nevertheless, batches and stateful recurrent nets in Keras are still a source of confusion for me sometimes.
I found it very helpful to first simulate simple dynamic systems that could easily be represented by a state-space model for example. Then to build a neural net and play around with the parameters and look at the output. You then get a feel for what is going on in the RNN/LSTM and it might help you to come up with more complicated models. In my experience it helps to spend some time figuring out good values for weight initialization and how some constraints could limit the kind of local minima it finds.

I feel the same. Keras and Tensorflow is confusing me a lot. Especially in how to set up the tensors for training. I´m doing this kind of research for a academic project. What i did so far, I only substituted the dense Layers with LSTM-Layers. This doesn´t improove anything exept that te training process lasts longer. And I still need to present the data in form of time vectors to the network (Like Time-Delay-Networks). I think I somehow need to get used to sequence2sequence models, but have not found some code which I can use for that so far.

Was doing some searching today.  Its already been addressed in the form of time series prediction in this link.  The implementation can be optimized though.

Hello,
Interesting questions and answers. Have you made some progress? As a side note, system identification is much more complex than simple time series prediction. It would be really interesting to see what are the advantages of using LSTM and other types of deep NN over the classical system identification techniques (such as prediction error methods and subspace identification methods).
Best Regards,
Aleksandar Haber

Hello again,
Motivated by your questions and due to my interest in this topic, I started a project on the identification of dynamical systems using various recurrent neural network architectures and the Keras toolbox. The project description can be found here, and  the codes can be found here

Very cool project. I did the exact same thing (only with a different dynamical system) when I started working on my project. However, why do you use non-linear activations in this example? This should work with a 2-unit SimpleRNN without non-linearities.
Also, I recently discovered eigensteve.com and his youtube channel. He is a professor at University of Washington and does a lot of research on data driven dynamics and control using machine learning. I found this to be a very interesting resource.

Thank you for the feedback. For brevity, I tried to use the default Keras setting when adding the layers:
model.add(LSTM(32, input_shape=(trainX.shape[1],trainX.shape[2]),return_sequences=True))
I guess that linear activation functions will also work since the model itself is linear. Now, for the specific problem formulation I used, I am not sure that a 2-unit SimpleRNN will give good prediction results (even if linear activation functions are used). Namely, the dynamics here is seen from the input in the form U=(x0, u_{1},u_{2}, \ldots, u_{N}) to the output of the form Y=(y_{0}, y_{1},\ldots, y_{N}). The initial condition needs to be added to inputs since the network needs to know from where to start (to predict the future state of a dynamical system, we need to know an initial state and an input sequence).
This is probably not the "best" problem formulation. I think that it is better to start from an ARX model, which does not need initial conditions, and in the ARX case, the input matrix U contains past system outputs and inputs.
It would be interesting to see your code and the approach you used.
I am now experimenting with different problem formulations and with 1D convolutional neural networks (CNNs) in Keras. I think that CNNs might be more suitable for this application simply because it is very well known that the output of a (linear) system is a convolution of its inputs:
"The fundamental result in LTI system theory is that any LTI system can be characterized entirely by a single function called the system's impulse response. The output of the system is simply the convolution of the input to the system with the system's impulse response."
(https://en.wikipedia.org/wiki/Linear_time-invariant_system)
Finally, I am familiar with the work of the above-mentioned professor. They approach the problems from the dynamical system's perspective, which is an interesting approach.
P.S. The identification problem can be easily solved using the subspace identification methods, that can give the system matrices A,B,C,D up to unknown similarity transformation. However, this method works well only for linear systems and for certain classes of nonlinear systems.

Hi,
I cloned your repo and tried with a 2-unit SimpleRNN. I used the following model:
model = Sequential()
model.add(
    SimpleRNN(
        2,
        input_shape=(trainX.shape[1], trainX.shape[2]),
        use_bias=False,
        activation="linear",
        return_sequences=True,
    )
)
model.add(Dense(1, activation="linear", use_bias=False))

This model contains more than enough parameters to learn the dynamics. The results are very good.

Loss during training:

I also changed sampling rate to 1 and total simulation time to 100 to speed up computations on my (very old) laptop.

Great insight! So it is really necessary to explicitly say to Keras that we want linear activation functions and no bias... Thank you.

Update: Does this mean that nonlinear activation functions actually add an artificial model order?

It depends on the activation function I think. Imagine a sigmoidal for example. Since the underlying system is linear, the non-linear network will have to learn to reproduce that linear behaviour. A single sigmoidal unit for instance would have to learn to stay in its linear regime (around zero), thus pushing the bias close to zero. Any inputs further away from the linear regime however will result in erroneous outputs. You would then have to add many other non-linear units to get better results.
Thats why I chose "linear" activation and no bias. But of course, this is based on some a priori knowledge of the system and can not always be assumed.

For those of you who are still following this post:
I have implemented a subspace identification method for estimating models of dynamical systems. The codes can be found here:
https://github.com/AleksandarHaber/Subspace-Identification-State-Space-System-Identification-of-Dynamical-Systems-and-Time-Series-
and a post with detailed explanations can be found here:
https://aleksandarhaber.github.io/machine_learning/2019/11/13/subspace-identification.html
In my opinion, in the case of linear systems, subspace identification algorithms outperform most of the methods (including neural network methods) for the estimation of dynamical systems. Unfortunately, subspace identification methods are not widely known to a broad scientific community.
In the future, I will compare subspace identification methods with other neural-network-based estimation methods.
Best Regards,
Aleksandar Haber

@AleksandarHaber I read through the post linked above. Thanks for sharing. I wonder if the subspace identification can also solve nonlinear system identification. Say I want to model half car dynamic system which is characteristic as a nonlinear problem.

Hi bballamudi,
Subspace identification of nonlinear systems is an active research area. To the best of my knowledge, stable and experimentally verified methods have been developed for the following classes of nonlinear systems:

Wiener state-space models
Hammerstein state-space models
Wiener+Hammerstein  state-space models
Bilinear state-space models
Linear parameter varying systems

there are methods for the identification of time varying linear systems which are obtained by linearizing nonlinear systems around time-varying state trajectory.

The subspace identification methods work surprisingly well and are experimentally verified on many different system types. Also, these methods have great potential for the estimation of time series, such as the stock market...
To start with, you can look at the following (not so recent) paper,
http://people.duke.edu/~hpgavin/SystemID/References/Palanthandalam-Madapusi-ACC-2005.pdf
and the works of Prof. M. Verhaegen from TU Delft, who is one of the founders of the subspace identification method.
Best,
Alex

Hi,
I also have problems to get my network working. I really don't understand how to use samples (different input sequences of n time-steps each) in keras to train my model.
Actually, I get an error if I train using 2 samples and I validate data on 1 input sequence only.
I started approaching this topic using @AleksandarHaber  code (https://github.com/AleksandarHaber/Machine-Learning-of-Dynamical-Systems-using-Recurrent-Neural-Networks). Then I slightly changed it to my purpose. My system is a non-linear mass-spring system (I added a x^3 term in the equation) and it is forced using a time sequence given by F(t)=F0sin(omegat). Where F0 is a constant value. I use finite differences to simulate data. This part is ok. Results are correct.
Then:
I want to train the NN using 2 input sequences (2 F(t) sequences obtained using 2 different values of omega ). Then I would like to validate on a input sequence generated with a different omega value (only 1 input sequence) and perhaps predict on another one (again, only 1 input sequence). To summary:
training data = (2, 2000, 1)
validation data = (1, 2000, 1)
prediction data = (1, ?, 1) (I say ? instead of 2000 because I am wondering whether I can only make prediction on a time sequence with the same length of  training data; I tried using 2000 for the moment as I already have problems on the sample dimension).
2000 is the time sequence length I get from my simulation.
BTW, I get an error as soon it encounters the validation data (I also tried to use the validation_split=0.2 instead of providing a separate dataset):
tensorflow.python.framework.errors_impl.InvalidArgumentError:  Specified a list with shape [2,1] from a tensor with shape [1,1]
	 [[

I defined input_shape=(2000,1), as I read to do so in many posts. I am not using any complexity, just
model.add(LSTM(32, input_shape=input_shape, return_sequences=True, activation='sigmoid', batch_size=2))
model.add(Dense(1))

Another question is: how should I set my batch_size to warn the NN that I have more than one sample of data?
As far as I understood, batch_size is related to the number of samples used to train the model before an update of hyper-parameters. Because I have only 2, I use them both for the training in 1 batch. There are 2 ways of defining batch_size: one is batch_size kwarg in model.add(LSTM(.., batch_size=xx)) the other one in model.fit(..., batch_size=2). What is the difference between the 2? Where should I specify the number of samples I want to use to train the model? If I don't specify it in LSTM function, I see the code running info:
Train on 1 samples, validate on 1 samples and results are obviously bad (it knows only about 1 input sequence and the result doesn't match the system response).
But if I put batch_size=2 in the model layer definition, then I cannot run with validation as I said.
To clarify, If I remove validation data, put back batch_size=2 in LSTM  and I make "prediction" on the same training set, I get good results. But it doesn't make sense. I want to predict on something else (use different data shape, at least for samples)!
I can share my code if anyone could help me.
Thanks

Can you post the codes?

Hi,
I cloned your repo and tried with a 2-unit SimpleRNN. I used the following model:
model = Sequential()
model.add(
    SimpleRNN(
        2,
        input_shape=(trainX.shape[1], trainX.shape[2]),
        use_bias=False,
        activation="linear",
        return_sequences=True,
    )
)
model.add(Dense(1, activation="linear", use_bias=False))

This model contains more than enough parameters to learn the dynamics. The results are very good.

Loss during training:

I tested this, and there are two important things to consider and to keep in mind:

In my original code, I used the training epochs of 2000. However, to obtain better results for networks with smaller number of units (parameters-model or model order), we need to increase the number of epochs. I obtained similar results to the results that cbow got by increasing the number of epochs to 4000.
Since the data generating model I used in my original code is a linear system, we need to explicitly state that activation functions are linear and that we do not use bias.

I revised my original post by polishing it a bit and by including these important insights. The revised post can be found here:
https://aleksandarhaber.com/using-recurrent-neural-networks-and-keras-tensorflow-to-learn-input-output-behaviour-of-dynamical-systems/


Sync viewers containing images with different sizes and scales

Hi,
first of all thanks for the great work!
My use case is a little bit specific: I have some wsi (whole slide images) slides which I want to show one next to each other. I want to make sure that any operation done on one image is reflected to the other one, like zooming and panning. This would seem simple enough but now I have to consider that the images are not always of the same size and scale.
I've found this codepen https://codepen.io/iangilman/pen/BpwBJe in which the images are the same and pan/zoom accordigly, but modifying it a little bit (https://codepen.io/lcl45/pen/PoyJwyM) gives me a strage result (for me at least): what I want to do is to pan the same point between the two images (or zooming). Like, if I pan clicking on the center of the left image I want the right one to pan also from the center of it. You can see the "error" in particular from the start since the two images get aligned on the top side but i want, instead, to be aligned on the center.
In all of this, I want to make also sure that the same pan is applied across images with different scales. Unfortunatly I can't manage to make it work on codepen: what I get for now, is that one image is moving faster than the other across the screen.
What I've done for now is to transform the center of the dragging one in image coordinates, which will then be re-transformed in viewport coordinates of the other image, still not solving the scale issue. I think, however, that a better solution might exist so I'm asking for a solution to this problem.
Thanks a lot!!

I'm just thinking about this now, but it occurs to me what might work best for this scenario would be to lay the images out so that their viewport coordinates match up. For instance, if image A is 2:1 aspect ratio and image B is 1:2 aspect ratio, you could load them in so viewer1 has:
tileSources: {
    x: -1,
    y: -0.5,
    width: 2,
    tileSource: 'a.dzi'
}

... And viewer2 has:
tileSources: {
    x: -0.5,
    y: -1,
    width: 1,
    tileSource: 'b.dzi'
}

This will line up their centers at (0,0) and make them overlap each other as much as possible.

Thanks a lot! I'll give it a try as soon as possible and will let you know how it goes.

Maybe I'm missing something. I'm declaring my openSeadragon instance with the following code
let openSeaDragon = OpenSeadragon({
    element: containerDiv,
    animationTime: 1,
    zoomPerScroll: 1.25,
    showNavigator: true,
    showNavigationControl: false,
    tileSources: {
        height: imageHeight,
        width: imageWidth,
        tileSize: 512,
        getTileUrl: (level, x, y) => tileUrl(apImage, level, x, y),
    },
});

where tileSources.height and tileSource.width are the actual image sizes (that, just for the sake of the example, are 41216 and 104448).
Setting x and y as you did doesn't do anything and the viewport still remains the same.
What am I missing?

Yeah, the tileSources interface can be pretty confusing! Here's some more info on it:
https://openseadragon.github.io/examples/multi-image/
In your case, this should work:
let openSeaDragon = OpenSeadragon({
    element: containerDiv,
    animationTime: 1,
    zoomPerScroll: 1.25,
    showNavigator: true,
    showNavigationControl: false,
    tileSources: {
        tileSource: {
            height: imageHeight,
            width: imageWidth,
            tileSize: 512,
            getTileUrl: (level, x, y) => tileUrl(apImage, level, x, y),
        },
        x: -imageWidth / 2,
        y: -imageHeight / 2,
        width: imageWidth
        // We don't specify height here because OSD uses the aspect ratio from the tileSource.
    },
});

The entire object put in the tileSources is something we don't have a good name for but maybe you could call it a "open specification". It includes the actual tileSource but also some additional positioning information. Note that the width and height you pass into the tileSource is in image coordinates (the actual pixels of the image), whereas the x, y, and width you put in the open specification are in viewport coordinates, which are arbitrary. You can, as in this case, use the same coordinates for each, but you might not want to; for instance, if you have two images with different sizes, but the same aspect ratio, you might want them to overlap each other exactly, rather than being locked to their original sizes.
More about coordinate systems here:
https://openseadragon.github.io/examples/viewport-coordinates/
I hope all that makes sense!

@lcl45 Great! This seems like an elegant solution. Thank you for sharing your zoom level code :)


Fix insight copy to show all student count, not limit

Who is this PR for?
HS educators
What problem does this PR fix?
insight box copy would show the limit of how many students were returned, not actual real number of students

What does this PR do?
fixes that!
Checklists

[x] Author included specs for new code

selfie


Interactive overview map

Hello,
I have been confronted to the missing interactive OverviewMap control recently: https://github.com/openlayers/ol3/issues/5874
@jonataswalker began some work on the subject, you can find his work here: https://github.com/jonataswalker/ol3-interactive-overview
Therefore I have based my work on his, as you can see in the commit https://github.com/openlayers/ol3/commit/2b7ed7381fb5d353f092b0dee84b3fd44e625f99.
I have tested my modifications thanks to make check and made the requested syntax corrections.
This is my first pull request on OL3 repository so make sure to let me know if something is wrong.

I commited some new changes in order to improve performance.
I was binding necessary functions into the box-updating method instead of the constructor.
Make sure to look at commit https://github.com/openlayers/ol3/pull/5887/commits/7649281900ee8418f6a8d5cba5a914e97a975427 for more information.

@ahocevar Sorry to bug you here but you are the only OpenLayers I had the occasion to work with. Do you know if there is anything to do to push along this pull request ?

Hi, is there a reason, why this PR is not merged yet?!

Would really appreciate having this PR merged too.

I think on mobile devices it should be possible to disable interactivity. Otherwise the mobile user can get confused!?
Is there a definitiv need to let it always be interactive?

@jhwegener I don't see how interactivity would be confusing. It's an improvement that all users should benefit from.

@felixveysseyre, please submit a CLA so we can merge this.

@ahocevar, I just submitted my CLA.


[ERROR INSTALLING] MY ERROR

C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Platforms\x64\PlatformToolsets\v140\Toolset.targets(36,5): error
MSB8036: The Windows SDK version 8.1 was not found. Install the required version of Windows SDK or change the SDK vers
ion in the project property pages or by right-clicking the solution and selecting "Retarget solution". [C:\Users\EMU\AppData\Roaming\npm\node_modules\truffle\node_modules\scrypt\build\copied_files.vcxproj]
C:\Program Files(x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Platforms\x64\PlatformToolsets\v140\Toolset.targets(36,5): error MSB8036: The Windows SDK version 8.1 was not found. Install the required version of Windows SDK or change the SDK version in the project property pages or by right-clicking the solution and selecting "Retarget solution". [C:\Users\EMU\AppData\Roaming\npm\node_modules\truffle\node_modules\scrypt\build\scrypt_wrapper.vcxproj]

Hi there. I recommend attempting to do what the error says:

The Windows SDK version 8.1 was not found. Install the required version of Windows SDK or change the SDK version in the project property pages or by right-clicking the solution and selecting "Retarget solution".

A quick google shows you can get the Windows 8.1 SDK here: https://developer.microsoft.com/en-us/windows/downloads/windows-8-1-sdk
Going to close for housekeeping purposes. Since everyone's installation issues are a little different, especially on Windows, I recommend asking these style of questions in the Truffle gitter: https://gitter.im/ConsenSys/truffle

Ya I had already installed that with npm but will try the setup, thanks.

I thought that may have come with visual studio already but I will try it.
I also installed some build tools package with npm.

Ya, definitely interesting if it doesn't.


Broken link

Link on "choosing PHS" is broken, it gives a 404

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 1bd2f66d-c853-21ff-7f74-ccb1359af4e1
Version Independent ID: 807303e4-fbf8-ad2c-de8a-5ed65ad9216e
Content: Hybrid identity and directory synchronization for Office 365
Content Source: Enterprise/plan-for-directory-synchronization.md
Service: o365-administration
GitHub Login: @JoeDavies-MSFT
Microsoft Alias: josephd

@fbrusa This has been corrected. Thanks so much for letting us know.


I'm using "classNameBindings" with link-to, can I use it here?

I'm using "classNameBindings" with link-to, can I use it here?
Example:
{{#link-to 'post' post.id class='btn' classNameBindings='post.deleted:classDeleted'}}
How to with this?

href-to is just a helper which generates a URL. You should use link-to for this


How to prevent serviceworker files from being injected into "client" when I use serviceworker

How to prevent serviceworker files from being injected into "client" when I use serviceworker

This is only happened at dev, it not caused impact for your build.

but，I need use it in dev。can

That should be worked even it injected hmr helper

import can not worked in service worker。should use importScripts.

We should add a ingoreHmrFiles option to resolve this.
Other, the module rewrite is not worked on importScripts, the alias it will not worked.

This shouldn't happen... since the hot context code is only injected if the code contains reference of import.meta.hot. Need a reproduction.

Closing stale bugs for 1.x. If the issue persists in 2.0, please open a new issue with proper reproduction.

Closing stale bugs for 1.x. If the issue persists in 2.0, please open a new issue with proper reproduction.


fix: Fix support for Svelte 5 snippet blocks

Improves upon #68. I was happy to see that PR, but markup_fmt v0.15.1 still had problems with parsing Svelte 5 snippet blocks when used in combination with typescript 0.93.1:
Unexpected token `/`. Expected identifier, string literal, numeric literal or [ for the computed key at file:///...MyComponent.svelte#.tsx:352:10

        <>{/snippet}</>

I didn't investigate the problem carefully, but I think this has something to do with the fact that the parser was treating the closing {/snippet} tag as text instead of consuming it and re-emitting it in the printer. Maybe this was intentional, since copying the above {/key} parsing logic was the obvious move I did. Perhaps @bartlomieju could comment.
This PR also fixes a typo (ExpectSvelteIfBlock -> ExpectSvelteSnippetBlock) that improves error reporting.

Perhaps @bartlomieju could comment.

I guess it wasn't intentional and that's a good catch.


LDtkUnity.Example already exists

Hello Cammin,
Thanks you and good luck to continue this package on Unity, it will be well received by the community. When we install the package by UPM (modification of the manifest), we get the runtime + Samples. Impossible to get the Samples via package manager/Samples/import, otherwise we have the following message in theconsole Assembly console with name 'LDtkUnity.Example' already exists (Packages/com.cammin.ldtkunity/Samples/Example/LDtkUnity.Example.asmdef)

Ah yes, that was an oversight on my part; will fix that, and thanks for pointing it out :)


[BUG] - cardano-node transaction view doesn't format certificates well

Internal/External
Internal   if an IOHK staff member.
Area
Other   Any other topic (Delegation, Ranking, ...).
Summary
Currently, certificates are a huge dump of types. They aren't easy to glance and and know what's in them. Also, some types are displayed very awkwardly, for example metadata hash of pool registration (it should be hex, since that's what all the inputs and outputs expect it to be).
Steps to reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
System info (please complete the following information):

OS: [e.g. Ubuntu]
Version [e.g. 20.04]
Node version

Screenshots and attachments

If applicable, add screenshots, config files and/or logs to help explain the problem.

Additional context
Add any other context about the problem here.

EX:
    "certificates": [
        "DCertDeleg (RegKey (KeyHashObj (KeyHash \"ba5078dcbfb4a05c36c8ae88334aa1b83553c855433a583e11b9f01f\")))",
        "DCertDeleg (Delegate (Delegation {_delegator = KeyHashObj (KeyHash \"ba5078dcbfb4a05c36c8ae88334aa1b83553c855433a583e11b9f01f\"), _delegatee = KeyHash \"ef789bb19ec1a7e28c0c5f68cb14107828fe878a9483daf14312d903\"}))"
    ],

Ex of pool registration certificate:
    "certificates": [
        "DCertPool (RegPool (PoolParams {_poolId = KeyHash \"bb6c692105fa347b8208b122d1eac295e40246e6fbf8f1c64e56aea5\", _poolVrf = \"cba323339e5f4ac9a9106edbfa76a5aaa23ffcdbbd69dc899e7bd61b03f4c64f\", _poolPledge = Coin<PHONE_NUMBER>0, _poolCost = Coin 340000000, _poolMargin = UnsafeUnitInterval (3 % 400), _poolRAcnt = RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"6452f8fed37ef08407cc6f7e6b843755fbabdeab6512935c5c27141c\")}, _poolOwners = fromList [KeyHash \"6452f8fed37ef08407cc6f7e6b843755fbabdeab6512935c5c27141c\"], _poolRelays = StrictSeq {fromStrict = fromList [SingleHostAddr (SJust (Port {portToWord16 = 4012})) (SJust <IP_ADDRESS>) SNothing,SingleHostAddr (SJust (Port {portToWord16 = 4011})) (SJust <IP_ADDRESS>) SNothing]}, _poolMD = SJust (PoolMetadata {_poolMDUrl = Url {urlToText = \"https://aaaaaaa/meta.json\"}, _poolMDHash = \"AD\\236\\249\\218~M}\\153.R\\185o*S\\ETX\\112\\111\\238b\\DEL\\STXpf\\231}\\SUB\\DC3C\\202\\219\\149\"})}))"


DELTASPIKE-854 - update related artifacts version to fix jacoco profile.

PR merged

This was updated and should be ok. I also added corresponding jacoco property to obtain complete coverage information.


Honor setHidden and setVisible

Thanks for creating such a useful and easy to use library!
I have just started playing around with it and have bumped into something I would like to hear your thoughts about. I am used to applying setHidden or setVisible in my queries to filter out redundant data. For example, consider the following query:
const study = Study
    .where('id', params.id)
    .with('jobs.variables')
    .with('participants')
    .first()

This provides me with the result:
{
  "data": {
    "jobs": [
      {
        "variables": [
          {
            "id": 1,
            "study_id": 1,
            "dtype_id": 1,
            "name": "distractor",
            "created_at": "2020-06-30T12:40:02.000Z",
            "updated_at": "2020-06-30T12:40:02.000Z"
          }
        ],
        "id": 1,
        "study_id": 1,
        "order": 1,
        "created_at": "2020-06-30T12:40:02.000Z",
        "updated_at": "2020-06-30T12:40:02.000Z"
      },
      {
        "variables": [
          {
            "id": 1,
            "study_id": 1,
            "dtype_id": 1,
            "name": "distractor",
            "created_at": "2020-06-30T12:40:02.000Z",
            "updated_at": "2020-06-30T12:40:02.000Z"
          }
        ],
        "id": 2,
        "study_id": 1,
        "order": 2,
        "created_at": "2020-06-30T12:40:02.000Z",
        "updated_at": "2020-06-30T12:40:02.000Z"
      }
    ],
    "id": 1,
    "name": "Attentional Capture",
    "description": "Basic attentional capture experiment",
    "active": 1,
    "osexp_path": "/public/osexp/attentional-capture.osexp",
    "created_at": "2020-06-30T12:40:02.000Z",
    "updated_at": "2020-06-30T12:40:02.000Z"
  }
}

Because the data of jobs is provided as related data for study, I for instance don't see any use for including study_id in each job record (and often I leave away even more data). I used to omit superfluous data by writing my first query as:
const study = await auth.user.studies()
  .where('id', params.id)
  .with('jobs', (builder) => {
    builder
      .setHidden(['study_id'])
      .with('variables')
  })
  .with('participants')
  .first()

This effectively hides away all study_id fields in job records if I simply return {data: study} at the end of the controller. However, if I use a transformer, this study_id field is present again! Thus if I end the controller action with:
return transform
  .include('jobs')
  .item(study, 'StudyTransformer')

where my transformers look like:
class StudyTransformer extends BumblebeeTransformer {
  static get availableInclude () {
    return ['jobs', 'variables', 'participants']
  }

  transform (model) {
    return { ...model.$attributes }
  }

  includeJobs (study) {
    return this.collection(study.getRelated('jobs'), 'JobTransformer')
  }
  ...
}

class JobTransformer extends BumblebeeTransformer {
  static get defaultInclude () {
    return ['variables']
  }

  transform (model) {
    return { ...model.$attributes }
  }

  includeVariables (job) {
    return this.collection(job.getRelated('variables'), 'VariableTransformer')
  }
}

I know the problem probably lies in my usage of return {...model.$attributes} in the transform function, but shouldn't its usage be subjected to or honor the 'setVisible/Hidden' too?

Nevermind, by changing
transform (model) {
    return { ...model.$attributes }
}

to
transform (model) {
    return { ...model.toJSON()}
}

everything works as expected. I think this is the way to go.

Wait, it doesn't... model.toJSON() also includes all relationships. Apologies for the chaotic approach.


Feature Request: set current sciences & skip technologies that require science not being made.

Often, I will unlock a science pack before I am ready to automate it, suppose blue science. If I have a technology in the queue that requires blue science, but items later in the queue that do not, being able to automatically skip the one with blue science would be a nice feature.

I would rather implement proper queue reordering so you could do this yourself. See #3.


[HUDI-2402]Hive Sync supports Kerberos authentication

For Hive with Kerberos enabled, THE HMS of HUDI cannot access the Hive metadata
Hive.get(configuration).getMSC()
Methods before
Perform Kerberos authentication on the Flink client
Set the parameters for enabling Kerberos authentication and the configuration parameters for transmitting hive Principal to Hive
I added two parameters
1.hive_sync.use_kerberos
2.hive_sync.kerberos_principal
Use the following method to enable Hive Kerberos access control
CREATE TABLE t2(
Uuid VARCHAR (20),
The name VARCHAR (10),
The age INT,
Ts TIMESTAMP (3),
partition VARCHAR (20)
)
PARTITIONED BY (partition)
with(
'connector' = 'hudi',
'hive_sync.enable'='true',
'hive_sync.db'='test',
'hive_sync.table'='t2',
'hive_sync.mode'='hms',
'path' = 'hdfs://ip:8020/warehouse/hudi/t2',
'hive_sync.metastore.uris'='thrift://ip:9083',
'hive_sync.use_kerberos' = 'true',
'hive_sync.kerberos_principal' = 'hive/_HOST@BIGDATA'
)

I'm fine with the change, but i'm not familiar with the kerbrous thing, assumes that the author has tested it locally, @vinothchandar , do you want to take a look ?

Sorry, I accidentally deleted the question about kerberos connection failure, I will answer it again here

Sorry, I accidentally deleted the question about kerberos connection failure, I will answer it again here

This is the code I tested to connect to Kerberos-managed Hive in two ways

Add the hive - site. XML
Do not add hive-site.xml
Both methods have been tested to create a database for Kerberos-managed Hive
So there is no problem with either method

To connect to Kerberos-managed Hive, you need to perform the following operations

Perform Kerberos authentication on the client
When calling hive.get (conf).getmsc (),Pass the conf. Set (" hive. Metastore. Sasl. Enabled ", "true") and the conf. Set (" hive. Metastore. Kerberos. Principal ","hive/_HOST@BIGDATA");
Ensure that the Hive client version is the same as the Hive server version

Note that the principal is set to the principal used by the hive being accessed

Note that the principal is set to the principal used by the hive being accessed

Perform Kerberos authentication on the client

How can we do that ? What do you mean by the client ?

@test-wangxiaoyu
I gave it a try, your code doesn't work properly.
kerberos authentication failed and query report not support option.

@test-wangxiaoyu
I gave it a try, your code doesn't work properly.
kerberos authentication failed and query report not support option.
Are you using Flink or Spark? Can you show me the log that reported the error

@test-wangxiaoyu
I gave it a try, your code doesn't work properly.
kerberos authentication failed and query report not support option.
Are you using Flink or Spark? Can you show me the log that reported the error

Sorry, it's works.
Due to maven build cache，lead to deploy code is not up to date.

I see another PR with kerberos support as well https://github.com/apache/hudi/pull/3771. May I know which is the right one. Can we close one of them if they are duplicates please.

@danny0405 : Are we waiting on something wrt this patch. Or is it good to review and take it to landing.

Repeat submit


[Table] Add support for virtualization

This will improve rendering performance on large tables (500+ rows).
To prevent messing the current table implementation I will open a PR adding a new component keeping the same API, look and behaviour.

:tada: This issue has been resolved in version 10.3.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Fix CaretLineVisible default value

Default for this property in Scintilla seems to be true, making it impossible to set it to false in the designer since it doesn't actually generate any code for default values.

This looks pretty good so far in my testing, I have a couple more things to review and then I'll likely approve it soon.


MongoDB oauth User model

TL:DR

class User(models.BaseUser)

should be replaced by

class User(models.BaseUser, models.BaseOAuthAccountMixin)

in models.py for the MongoDB oauth to work properly
Description
The oauth full example (as well as the rest of the documentation) mentions the basic models configuration is adequate for the mongoDB implementation of oauth.
However regarding the User model, in case we do not additionally inherit from the BaseOAuthAccountMixin, which adds a List of BaseOAuthAccount objects, we do not provide for the User a way to store the OAuth information (access tokens, account ids...)
So it seems only reasonable that should we use the code displayed on the https://fastapi-users.github.io/fastapi-users/configuration/oauth/ quoting:

class User(models.BaseUser):
pass

it registers the User document in our collection but it does not store the oauth_accounts array in it that should look like that:

"oauth_accounts": [
{
"id": {
"$binary": {
"base64": "#####################==",
"subType": "04"
}
},
"oauth_name": "google",
"access_token": "############################################################",
"expires_at":<PHONE_NUMBER>,
"refresh_token": null,
"account_id": "people/######################",
"account_email"<EMAIL_ADDRESS>}
]

Hence in case we try to access our API endpoints bearing this user's token it fails with a relevant error message quoting the lack of the oauth_accounts part of the document.
Solution
To overcome this problem i used a User model similar to the Tortoise ORM full example inheriting from both the  BaseUser AND the BaseOAuthAccountMixin:

class User(models.BaseUser, models.BaseOAuthAccountMixin):
pass

There was indeed a mistake in the MongoDB example for OAuth, thank you for raising this 🙏

Thank you for your great project.
It would be fantastic if you considered incorporating some kind of permissions system!


should nuclear envelope be part_of nuclear periphery?

I thought it used to be but apparently not.
Should it be?
One of our Community curators used "nuclear periphery" for  a "nuclear  envelope protein" because the  child wasn't there.

Actually nuclear periphery is defined
The portion of the nuclear lumen proximal to the inner nuclear membrane.


feat(synchecks): adding userId to CloudWatchSynthetics

Goal
Adding optional userId to CloudWatchSynthetics.
Tickets:

https://mozilla-hub.atlassian.net/browse/SOCIALPLAT-386

:tada: This PR is included in version 5.1.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Convert-Path should work with non-existent paths

It seem clear to me that a design mistake was made when Resolve-Path and Convert-Path were created, and they both return only resolved paths (i.e. the path must exist).
The right design would be to have Convert-Path not care if the path exists, and if you care, you use both commands.
Since that ship has sailed, I propose adding a -Unresolved switch to Convert-Path to change it to do what it should have done all along, called GetUnresolvedProviderPathFromPSPath instead of calling GetResolvedProviderPathFromPSPath.
If nobody has a problem with that ... I can send a PR

Great! Make PR.
I'd only made the switch more descriptive: -UnresolvePath?

I'd suggest something along the lines of -SkipPathCheck similar to the other Skip switches as mentioned in #2006

Perhaps @rkeithhill will again offer a good option.

@SteveL-MSFT @mklement0 Can we approve the new behavior and parameter?

The new behavior behind a switch seems fine.  cc @HemantMahawar if he has any good suggestions on naming.

Opting out of behavior is more typically done with -No* parameters (-NoNewline, -NoProxy, -NoClobber, ... - Get-Module and Invoke-WebRequest/RestMethod are, unfortunately, dissenters and use -Skip*)
Therefore: -NoResolve, perhaps?

TBH, -Skip<Something> and -No<Something> are very close, although have very subtle difference.

-Skip<Something> used in cmdlets ((SkipCertificateCheck, SkipEditionCheck, SkipHeaderValidation, SkipNetworkProfileCheck, SkipCACheck, SkipCNCheck,  SkipRevocationCheck) ) is typically for bypassing some check.

-No<Something> used in cmdlets ((NoClobber, NoNewline, NoServiceRestart, NoTypeInformation, NoProxy, NoRecurse, NoQualifier, NoNewWindow, NoNewScope, NoMachineProfile, NoEnumerate, NoEncryption, NoElement, NoCompression, NoWait)) is typically about not taking specific action.

In this case -SkipPathCheck (or maybe -SkipPathResolution) seems more appropriate.

Thanks, @HemantMahawar, I see the difference, but I personally don't think it's strong enough to warrant reflection in distinct parameter names.
Case in point: You could equally argue that the requested behavior here is not taking the action of resolving the path.
To put it differently: I think the existing -Skip* parameters should have been named -No* instead to begin with for consistency (though, should we reach a consensus, we could introduce aliases for consistency going forward).

A couple more thoughts: No is shorter and less ambiguous than Skip, which:

has plausible synonyms such as "bypass", "omit", ...
is also in use with significantly different semantics (Import-Cli -Skip <int>, Select-Object -Skip <int> -SkipFirst <int> -SkipLast <int> -SkipIndex <int[]>)

@PowerShell/powershell-committee reviewed this and recommends -SkipValidation

@SteveL-MSFT: While I see value in the generic aspect of the Validation part of the parameter name, can you explain why the Skip part was considered worth retaining?
In other words: what is the value in retaining the -No* / -Skip* split?

@mklement0 it's unfortunate that we have some inconsistency, however, in discussing this specific issue, we agreed that Skip was more readable than No as in: Convert-Path foo -SkipValidation was slightly more understandable as English than Convert-Path foo -NoValidation

Thanks, @SteveL-MSFT, but there's also this:
PS> Get-ParameterVerb No

Verb Group  Description
---- -----  -----------
No   Common Opts out of a default behavior.

Granted, no is not a verb, but neither are new, foreach, or where - close enough.
And, from the associated help topic:

Verb
Action
Comments

No
Opts out of a default behavior.
For this action, do not use verbs such as Skip, Omit, Bypass, or Drop

Now, you might argue that this is just fan fiction - and you'd be correct - but isn't it a great story?

It seems like the guidance for inverted parameter switches should be along the lines of:

No (Additional Output or Tasks) -- also, use to invert a verb switch
Skip (Additional Tests or Validation)

For what it's worth, if this was being written from scratch, I could see either -NoValidate or -SkipValidation but ...

It needed to be -NoValidate (an inversion of a hypothetical -Validate verb switch), because "Validation" is a noun
In greenfield, we would hopefully have said: change the default behavior, and use -Resolve as the switch 😉

Given the cmdlet has existed, doing what it does, for years, "skip"ing what it normally does makes sense to me.
I liked using "Resolve" in there (-NoResolve ??) rather than validate, because under the covers (and in other cmdlets) that's what we call it. Additionally, we're not (just) skipping validation, we're going to not "resolve" the path to an existing folder -- which means that if we're passed a path with wildcards, we would return a path that still has wildcards in it, right?

I see 28 cmdlets in the repo utilizes the same approach and use GetResolvedProviderPathFromPSPath for both Path and LiteralPath parameters. The fact says about consistency in design - GetResolvedProviderPathFromPSPath resolves glob chars vs GetUnresolvedProviderPathFromPSPath doesn't.
Assumption about using GetUnresolvedProviderPathFromPSPath instead of calling GetResolvedProviderPathFromPSPath in the cmdlet is not correct. GetUnresolvedProviderPathFromPSPath doesn't check existence. I believe we want GetResolvedProviderPathFromPSPath  but internal one with  allowNonexistingPaths=true parameter.
Initial request is "should work with non-existent paths". If original design is "allow only existent paths" the request is obviously "allow non-existent paths". So expected new switch name could be AllowNonExistent.

The Powershell Cmdlet working group reviewed this and agree that adding a switch to skip the path validation is a good idea.
But what should the switch name be? We discussed two alternatives -SkipValidation or -SkipPathValidation.
Comments

Having discussed this further, the parameter name should be -SkipPathValidation

I'm concerned about this -SkipPathValidation parameter name and about what @iSazonov wrote:

... GetResolvedProviderPathFromPSPath resolves glob chars vs GetUnresolvedProviderPathFromPSPath doesn't... I believe we want GetResolvedProviderPathFromPSPath but internal one with allowNonexistingPaths=true parameter.

I thought we were all on the same page that this is going to not resolve paths to existing folders.
As I said before, we're not (just) skipping validation, we're going to not "resolve" the path to an existing folder -- which means that if we're passed a path with wildcards, we would return a path that still has wildcards in it, right?
Right?

We could use a more conventional name -Force, which would make it consistent with Set-Item

I recently needed path validation to be skipped and also think it would he a great addition to Convert-Path. I'll be happy to raise a PR once I can get the code change ready.

I don't see conclusion about what new behavior should be. Obviously "glob expansion", "validation" and "existence" are not the same things.

Hi, I was pointed here because of a similar design Issue, so what's now the difference between No and Skip? When would I use one over the other? Maybe it's because I'm not a native speaker, but all of the examples given back here https://github.com/PowerShell/PowerShell/issues/2993#issuecomment-434784361 don't really help "solve" that mystery for me. All of these sound just perfectly fine:
NoCertificateCheck, NoEditionCheck, NoHeaderValidation, NoNetworkProfileCheck, NoCACheck, NoCNCheck,  NoRevocationCheck, ... also they more clearly indicate being the antonyms to a default behavior. Is it now "best practice" to use these switches instead of using either param([switch]$CertificateCheck=$true) or param([switch]$SkipCertificateCheck=$false)?
The only difference I see is that the "No" ones aren't auto generated for switches right now and Skip is often used instead of a default $true-switch that forces a specific feature to be enabled by default. Where as "No" is currently used either as the antonym to another switch or filtering (basically to avoid to query data that would be dropped by an where-objects call right after, I.E. "query-time where-object") but not features. Like function Get-MyAccounts{ param([switch]$NoServiceAccounts) }. Is this the "best practice" now/still?
Is there a dedicated issue or proposal for this? It is kinda misplaced within an issue about Convert-Path, I think.
Also I'd like keep posted about this topic and I'd argue for have auto generated No switches for every defined one, that'd remove a bunch of validationscript-blocks :D


[Candidate 4v4 Dominion ZH v1] Some Supply Piles can be driven behind, others not

Observed behaviour
Some Supply Piles can be driven behind, others not.
Expected behavior
Mirrored Supply Piles offer equal space for unit mobility.
Screenshots

fixed