Custom BouncyCastle.Security.SecureRandom not available on WP8.1

As the title says. At the moment it is the only class that implements IRandom and can be used with it - and it is not available on Windows Phone.

Reading your book, I see you are suggesting pretty much what I have done, datetime.not + currentthreadID  + system runtime combined with KDF, however I also thought of another neat way to get entropy, read the contents of a section of memory! What do you think?

@Thashiznets while it could be done. mobile platforms would represent a good part of the use of NBitcoin - and neither WinPhones nor Xamarin allows direct memory access.
For better entropy, why not read pseudorandom-defined data from a file? E.g. get a few pseudorandom numbers, based on that, get a path to a file, and get a pseudorandom number for start position to read (e.g. a 2MB file, get a number from ranges 1 - [2MB - 8kB], and read the data from there.

@fonix232 I think you can initialize a defined size array and then because it resides in memory allocated to the mobile app you then access the array and you get the random contents of what was in the memory isn't that right? Because arrays are by reference so you are always dealing with memory spots behind the scenes.
As for the files, remember in the mobile environments you are restricted to a sandbox, and possibly the media library so it may be feasible to sha256 a music file filename or something like that!

Actually WP8.1 allows direct file access in specific places, but yes, browsing into the music library is just as feasible - if it exists. For example, all my music is on Spotify, which uses its own storage.

@fonix232 yea like documents, downloads, music etc, but as you say it depends on user having content in those areas not really a guarantee. There should always be junk data in memory, I actually found ways to get junk memory data when calling some native/unmanaged code when I was doing Unicode NFKD normalisation stuff so it is definitely doable in WP.

@fonix232 @NicolasDorier oh also I use byte[] guidBytes = Guid.NewGuid().ToByteArray(); I get a GUID and use it as well :)

As fonix said, reading random value in memory is problematic, since there is no crossplateform way to do it. If there is no crossplateform way, then the solution is to used plateform specific PRNG (eg: SecureRandom of Java, RNGCryptoProvider for windows desktop)
Adding entropy manually would protect against the underlying PRNG bugs and backdoor though.
As I talk in the book, I added a RandomUtils.AddEntropy method.
The same remark hold for reading a pseudo random file. (not portable)
The Guid.NewGuid() is interesting but the security depends on the underlying implementation of it. And I doubt it use plateform specific PRNG, which mean that the entropy only have clock time, so a very bad one.

Hi @NicolasDorier like you I was skeptical about the entropy of GUID and how it remained random and so I traced the function back to the underlying Win32 functions and I found that it uses the ethernet or token ring MAC address. I imagine it starts as a hash of that and then keeps hashing for each subsequent GUID. It looks good as a source of entropy to me, of course just like you mention in your book, if attacker has access to machine they can get runtime, same would apply fir GUID but I believe it certainly helps bump up the bits of entropy. I am currently writing a method to return a hash of random memory bytes, I found I can exploit the unmanaged NormalizeString function to return bigger byte array than it should which sees it grabbing whatever is in memory so I get nice random byte array after hashing :)

Very interesting, so if the attacker knows your MAC (which is not private info on internal network), it basically has access to your key. In some case, it might be worse than timestamp.
However, yes, it bump up entropy. It protects a bit if there is an underlying bug in the PRNG.
Are you doing the NormalizeString hack for protecting yourself against the RNGCryptoServiceProvider of windows ? Because it already hash a number of internal state like.
Surely enough, adding your own entropy protect against malware messing up the prng.
The current process ID (GetCurrentProcessID).
The current thread ID (GetCurrentThreadID).
The tick count since boot time (GetTickCount).
The current time (GetLocalTime).
Various high-precision performance counters (QueryPerformanceCounter).
An MD4 hash of the user's environment block, which includes username, computer name, and search path. [...]
High-precision internal CPU counters, such as RDTSC, RDMSR, RDPMC

Source

Yea MAC and possibly they use a timestamp as well to ensure uniqueness? Unsure but yes as with anything constant they can get it if machine compromised. RNGCryptoServiceProvider is unfortunately not common between WinRT and WinPhoneRT at the moment, they are putting crypto stuff into .NET Core at the moment they inform me, but in the mean time I am using BouncyCastles Secure random with my own entropy.

What about CryptographicBuffer ?

Problem is CryptographicBuffer is part of Windows.Security.Cryptography and Windows.Security is WinRT only and not available for me in a PCL Targeting Windows 8.1 (WinRT) and Windows Phone 8.1 (WindowsPhoneRT).

I think you should use the CryptographicBuffer for WinRT and the RngCryptoServiceProvider for Windows Phone.
The way you can do that is by having an "Initialize" method in each plateform specific project that affect RandomUtils.Random with the plateform specific implementation.

I don't agree. I think your random should not chop and change between platforms. Given that it is easy to have Hashing algorithm on all platforms, it is very simple to create a common RNG on all platforms and I feel that SHA(your own entropy) is simple, efficient and safe, can be shared across all platforms and you don't have the problem where one RNG may be found insecure on one platform yet fine on another etc. Also you control the RNG and can account for it, as opposed to trusting the platform which is what happened to the Android Wallet if I'm not mistaken.

Well, you can protect against plateform RNG with your own entropy. (and yes, android one was flawed)
Using your own, also mean that if there is a bug in it, then it is still present everywhere.
What I would fear by using a custom RNG is to not have a sufficient entropy, when plateform PRNG give us some assurance about it.
If you want to use your own, then I would advice that at least you use some custom random input from your user. (like moving the mouse randomly)

But what if the entropy adding mechanism is flawed in the platform specific RNG? What if instead of adding seed material it clobbers it instead? So you think you are adding all this entropy but may in fact not be? You see what I mean? And yes I am handling all the entropy and seed generation in my custom random implementation so there is no way a user can go wrong. I have datetime, currentthreadid, systemruntime, guid, threadraceconditionentropy, processoraffinity all rolled into one AND kdf for certainty. This is cross platform and can be used in any WinRT/WinPhoneRT and can easily be ported to Android/iOS

Oh also remember, .NET crypto stuff still lives in the land of Sha1, so Rfc2898DeriveBytes (The .NET PBKDF2) is still using Sha1, but we have been advised now to use Sha256 and stronger so you may find these platform RNGs a bit lacking!

I see, yes it is a valid concern, like I said, android prng was flawed not so long time ago, so that's not paranoid.
I must say that I would not be confident enough about knowing if the entropy you are using is secure.
That's why I like using plateform PRNG which delegate that to the microsoft/apple/google... but well even such can be broken and hacked, so I might be wrong to prefer them instead of your own + KDF.

But my entropy creation is in my open source code so you can see for yourself the entropy creation. You can even look at the SecureRandom class in the BouncyCastle source to check that the entropy is being correctly used, this is what I did and I have absolute confidence in my solution which is using Sha256 and finally sha512 in the kdf. Honestly, Sha2 is the best thing since sliced bread!

It is not the algorithm that I necessarily don't trust.
It is more whether if the entropy is breakable by bruteforcing. But I guess with the KDF it should be fine.

Indeed, I'm confident that I don't need to use KDF, but I do for good measure.

@NicolasDorier hey mate I forgot to mention to you however feel free to put my BIP39 implementation into NBitcoin if you wish. It would be super easy to use your IRandom instead of what I'm doing if you prefer for the cross platform!

Thanks ! I was going to ask you if I can do that at one point. In fact I had several demands for BIP39.
I'll do it.

@Thashiznets I included BIP39. Your implementation helped a lot, but I did lots of refactoring. (I kept your credit on the Mnemonic class)
However the string normalization is not supported in all plateform, so I did not included it in my portable libraries :(
However, it is support on Android.
I've seen your solution to get a portable Normalize, but I suspect it works only on Windows. (so don't work on other plateforms)
I should ship a Windows-only portable lib at one point.

@NicolasDorier nice work! Indeed you are correct, the Normalisation is the reason I removed WP Silverlight support as it is a pinvoked Win32 call and can only be done in the Full .NET or .NET Core project types, the old WP Silverlight don't allow pinvoke so that is why I have ditched the legacy Silverlight and focussing on .NET Core now as it is the future.
Correct, my Normalise solution is reliant on Win32 API so yep you are correct that it is only windows.
I saw you were getting the lists real time, the reason I don't like dynamicly changing wordlists is that if you use the indeces and it points to a changed word, you get totally different derived bytes! I mean the wordlist I wouldn't think should change, but being open source it is a danger!

I opened another #60 issue.
It is impossible that they change that, since it would break existing shipped trezor.
Sometimes the list is updated but I think it does not break old mnemonic.
However, I think I will hardcode the lists in the lib like you did... Because I can't port the GithubWordlistSource (rely on HttpClient, which is not in my profiles)
For the normalized form, you just had to do use String.Normalize(NormalizationForm.FormKD) if you only support WIN32.
I intend to generate a substitution table of characters to their normalized forms and just hardcode it in the portable lib. Since it is potentially big, I'll only do it for well used characters, and just throw an exception for others.

Believe it or not, String.Normalize is another of those not available in .NET Core methods, only full .NET at the moment which is why I need to pinvoke the Win32 NormalizeString

Funny thing that Android support it :D

@Thashiznets I finished by hard coding the dictionaries, as well as a good deal of KD normalization.
Now BIP39 is portable ! :)

@NicolasDorier Nice! Have you tested your KD mapping against the official test vectors just for safety? Particularly it would be good to do with the Jap tests as the Jap tests catch people out when not normalising properly.

Yes I did. You can open NBitcoin.Portable.sln, and run the NBitcoin.Portable.Tests.
I test both, the portable versions and the full windows version.

Ok that's excellent work!

I'm closing that, I will add new Guid at initialization as entropy data only, and provide specific implementation of IRandom for specific plateform when I'll be able to compile on them.


Use cvt to simplify syscall error handling

The CI failure is unrelated to this change.

Thanks for the PR! I think though I'd prefer to avoid picking up a crates.io dependency for this, could the function be inlined into this crate?

Basically, I created the cvt precisely to be able to share this small piece of code between projects. We're already using it in CraneStation/wasi-common (precisely in the winx subcrate so far) and this dependency (however small) would be reused if we also used the crate here.

Makes sense yeah, and seems reasonable! I actually also wrote this support in the standard library :)
I would prefer to not pick up a dependency on a crate, though. I'm ok with inlining a helper function here, but if this is just to add a crate dependency I think it's ok to close this.

If we're not delegating the functionality to a dedicated crate, then I'm unsure of such a PR and I'll basically be  indifferent to this idea. It's up to you if we're not adding the dependency.

Ok, in that case I'm gonna go ahead and close because I'd prefer to not pick up a dependency for this.


Fix broken backwards compat in regards to tmos_version

@jlongstaf  and @mattgreene
Issues:
Fixes #638
Problem:
The tmos_version attribute was changed from being a property to being a
key in the _meta_data dictionary on the bigip container. This change
broken backwards compat without use realizing immediately, and we need
to support both methods until we're ready to bump the release to 2.0.0.
Analysis:
Added the @property back to the f5/bigip/init.py for tmos_version.
Both ways of accessing tmos_version are now supported
Tests:
Added a test to ensure we can access the tmos_version both ways

Ok to commit pending your last investigative test.

Merging. Thanks.


Count distinct floats

Which issue does this PR close?
Closes #199 .
What changes are included in this PR?
Modified try_from_array method on ScalarValue and added tests to physical_plan/distinct_expressions.rs
The tests cover all of the floating point weirdness that I could think of, namely NaN, the infinities, and subnormal numbers, but if I missed any I'm happy to add tests for them.
Are there any user-facing changes?
No

Codecov Report

Merging #252 (774d7b4) into master (e271e4d) will increase coverage by 0.00%.
The diff coverage is 70.00%.

@@           Coverage Diff           @@
##           master     #252   +/-   ##
=======================================
  Coverage   76.80%   76.81%
=======================================
  Files         133      133
  Lines       23284    23294   +10
=======================================
+ Hits        17884    17894   +10
  Misses       5400     5400

Impacted Files
Coverage Δ

datafusion/src/scalar.rs
54.36% <0.00%> (+0.22%)
:arrow_up:

...tafusion/src/physical_plan/distinct_expressions.rs
90.80% <87.50%> (-0.11%)
:arrow_down:

datafusion/src/physical_plan/group_scalar.rs
58.82% <0.00%> (+1.17%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e271e4d...774d7b4. Read the comment docs.

Thanks again @pjmore !

🎉


New Project Proposal - CBT TAPE

Name of project (must be unique within OMP)
CBT TAPE

Project description (what it does, why it is valuable, origin and history)
Enormous repository of z/OS (and MVS) related code, tools, and modifications.  Has existed for 45 years.  This collection is being constantly and proactively expanded on a continuous basis, to keep up with IBM system changes and user code improvements.

Statement on alignment with OMP charter mission
Supporting almost all z/OS installations in the world, and encouraging learning among systems programming practitioners, new and old alike.  Supporting z/OS in a great many companies and other installations worldwide, keeping their expenses lower and greatly aiding in their system support.  Adding enormously to the value of z/OS for all z/OS installations, over a very long period of time.

Sponsor from TAC (sponsor helps mentor projects)

Preferred maturity level (see project stages)
Ongoing, for as long as possible.  This product supports all z/OS installations worldwide, and also MVS-based systems which are older.

License and contribution guidelines (refer to the OMP guidelines)
All contributions have to be usable by anybody, for no cost.  Some contributors maintain ownership by means of a copyright notice, but they must allow anyone to use their materials as submitted to the collection.  There are no "timeouts" allowed for software in this collection.

Source control (GitHub by default)
www.cbttape.org web site

External dependencies (including licenses)
Support for this project needs one or more z/OS LPARs. No additional licenses required.

Initial committers (how long working on project)
Sam Golob (30 years).  Sam Knutson (22 years).  Sam Knutson pays for the website.  Sam Golob controls contents of the collection.

Infrastructure requests (CI / OMP Cluster)

Communication channels (slack, irc, mailing lists)
www.cbttape.org

Issue tracker (GitHub by default)
www.cbttape.org website.  emails to<EMAIL_ADDRESS>are the main input for site maintenance.

Website
www.cbttape.org

Release methodology and mechanics
Download from www.cbttape.org

Social media accounts
None currently.  The word-of-mouth knowledge of this project (the CBT Tape) is known by most z/OS and MVS-based systems support practitioners all over the world.

Community size and any existing sponsorship
At least 3000 or more installations use the service

Project accepted during the meeting.


feat(withLatestFrom): add withLatestFrom and zipLatestFrom

I figured I'd put a stake in the ground and see what happens ... maybe (as a friend of mine says) I'll end up falling on it, lol.  This is definitely POC quality and would need to be cleaned up before merging.  I'm hoping it helps with discussion, though.
This PR removes sample, in favor of adding withLatestFrom and zipLatestFrom, where zipLatestFrom is basically sample with flipped stream arguments.
"Sample" has a clear connotation for continuous values, but not really for events: when an event is not occurring (which is the vast majority of the time), there is no value to sample.  "Sample" is used often in signal processing as analog -> digital.  There is no analog signal in this case, so, in reality, "sample" doesn't make much sense.  We were basically pretending to treat one of the event streams like an analog value.  I think everyone will be better off if we just treat it like what it is, and put some energy into considering how to integrate "real" analog values, maybe something like this into the mostjs ecosystem.

Ah, I did flip the parameter order in this PR as we discussed, so I think its (note return type):
// pickLatestFrom :: Stream a -> Stream b -> Stream a
const pickLatestFrom = (streamWhoseLatestValueWillBePicked: Stream<A>, streamWhoseValuesWillBeReplaced: Stream<B>): Stream<A> => ...


add comment contents

added posts and comment contents for random users

Thank you so much for the contribution you made! It's so nice to see you here @arabacibahadir.
I would like to make a request regarding the operations in our seed.ts file. I was wondering if we could utilize the functions in the models folders, such as post.server.ts, user.server.ts or comment.server.ts to create posts/users/comments. This approach ensures that posts are created with the correct slug and other necessary parameters.

@cansirin tbh i prefer it to be plain prisma calls since we can change something there. seeding is before application is booted, so we ideally shouldn't import any files here, other than some small helpers that are inside prisma folder again.


FSI in VS throws warnings before code is comitted

I was typing (not copy-pasting) some code in the VS FSI:
module Option =
    let tryGet f o =
    match o with<enter>

Unlike in the C# interactive, you can't go back up and edit a different line, so I guess it is ok that it analyzes it instantly, not like you can fix it anyway ...


BC OnPrem Initialization of container x failed with these errors

Hi @freddydk,
A colleague tried to spin up the latest version of NZ on-prem container for BC and got an error.
mcr.microsoft.com/businesscentral/onprem:nz
I thought I'll reproduce it on my local and updated my NavContainerHelper to latest version <IP_ADDRESS>, uninstalled the older one.
Then after trying to spin up the container for BC onPrem NZ version, I got the same error at the very end of the initialization process (Image with details below):

I did setup Docker EE and I do see there is an update but I don't think it's to do with that, current containers run fine on my machine and I was able to spin up containers before.
I did another test, tried to download the latest NZ current version of BC (NOT the OnPrem one) and got a similar issue:

Which leads to one question: is the navcontainerhelper broken and what tags shall I use to get a working image?

I currently use all these options:

New-BCContainer -containerName $ContainerName                 -alwaysPull
-accept_eula                 -auth NavUserPassword
-Credential $credential                 -imageName $image
-licenseFile $LicenseFile                 -isolation process
-updateHosts                 -doNotCheckHealth
-includeAL                 -UseBestContainerOS
-accept_outdated #`
Or...is it to do with the Docker EE which we are using combined with the latest windows updates?
I used these steps and never had a problem until the latest windows updates...
https://www.kauffmann.nl/2019/03/04/how-to-install-docker-on-windows-10-without-hyper-v/
Shall I now use the -ltsc2019 flag?
Cheers,
Robert

@freddydk , nevermind, it seems this was blogged about by you here
Cheers,
Robert


Update to 2.2

In an effort to make everything Swift-ier, I'm proposing some changes to things that were left unchanged from 1.*.
The new Endpoints implementation, other than providing a clearer insight in what's actually going on, allows for both a better imperative and declarative approach. Having a concrete reference to what the user is looking for means we could be able to move the entire APIHandler to an NSOperationQueue (should we decide to, it's just an example) or even bypass it completely (e.g. endpoint.call() using Combine or RxSwift for instance).
Definitely something to discuss for a possible 3.0, since it would mean breaking everything again, more likely than not.
What's new

Endpoints: no more URLs, static lets and funcs, just enums (conforming to an Endpoint protocol), accepting specific arguments (when needed) — I would have loved using 5.1 new syntax, but 5.0 should do for now hahaha — 100% not final, it might change again (for instance holding reference to the response type)

Todo

HttpHelper and PaginationHelper accepting Endpoints directly
more coming soon…

a few endpoints using /v2 api
for example:
https://github.com/TheM4hd1/SwiftyInsta/blob/bfa94f4df3359b8ed2a217cf7a7a8e9c7be2a598/SwiftyInsta/API/Constants/URLs.swift#L641
otherwise they won't work.

Thanks for pointing it out :muscle:
I'll fix it in time for the final commit.

Endpoints returns wrong values for URLs
I applied the fixed for Credentials on current stable version and it worked fine.
please re-check the Endpoints

I've changed the all Endpoints thing again.
It's definitely easier to read, while being a bit more complex implementation-wise.
Do you mind checking if it fixes anything or if I messed up the actual endpoint paths? @TheM4hd1
Still can't test it myself unfortunately 😔

@sbertix will check soon

Done @TheM4hd1 💪


Fix travis configuration + DateTime-bug

After the merge of #36, I saw @steverhoades struggling with the Travis setup. While I'm not experienced with Travis, I decided to dive into it, because I really would like to upgrade my environment to PHP 8.
There where three causes for the failing runs:

The matrix was effectively defined twice, due to the extra environment variable now the tests ran also for incomplete env-setups
Tests where failing because of deprecations in older environments, for which a decission should be made if the will be maintained (and the deprecations should be fixed) or not (deprecations are fine). That's why I have changed the Travis-config to ignore deprecations
After fixing those issues, I ran into an error in the specific combination of oAuth2-server 8.0 and lcobucci/JWT 3.4.*, which I've fixed

And after some failed runs:

Updated Ubuntu-version to Xenial, to support PHP 5.6 as well as 8.0
Allowed to use version 9.5 of PHPUnit, since 5.0 does not support PHP 8.0

Thanks for taking a look at this, I have been traveling and wasn't able to finish that integration. I'll take a look at this ASAP.

LGTM, Thanks for this!

@steverhoades Is it possible to do a release today? It would fit perfectly in our update schedule this week ;)

@tjveldhuizen released as 2.0.0


Fix shield HP updates not being reflected in token resource bars

Fixed based on @ghost91 's suggestion.
Closes #4568

This should actually be done by core, but it's possible the bar redraw is done too early. I'm hesitant to add another blind redraw into the mix without making sure it doesn't break simultaneous re-rendering of other parts of a token, given other updates.

Updated.
This was actually caused by redirecting the Actor#_onEmbeddedDocumentChange call to TokenDocumentPF2e#onActorEmbeddedItemChange which did not redraw the token bars like the super method would do.
Also Token#drawBars was typed as protected which is incorrect.

Test it with a token resize and image swap all on one item.

Like this?
token.webm

Nothing in the console?

Nope, no errors.


[FEATURE REQUEST] Support X-Sendfile Header

Is your feature request related to a problem? Please describe.
When having private files behind a PHP firewall you return them in Symfony as a BinaryFileResponse. When supported by the Webserver in this case the x-sendfile header will be used which just tell the webserver where the file can be found which need to be streamed out.
Describe the solution you'd like
It would be great when roadrunner would also support the x-sendfile header. I could not yet find a reference or a config how this kind of response are handled in roadrunner.
Describe alternatives you've considered
Alternative is not supporting x-sendfile Symfony does then fallback to a normal file stream to php://output.
Additional context
Apache does support x-sendfile of the sendfile mod. Nginx I think does support it out of the box. The state of caddy webserver I don't know.

@alexander-schranz Feature will be available for testing by the end of the week in the v2.5.0-beta.2.


Typo resolved.

Sorry, my OCD kicked in.

any contribution, however small, is appreciated. (I raise PRs for typos like this too.)
Sorry that it took so long to merge. :/


V3.0

I have done a first implementation of types and their validations. Please review the changes and create a new branch (v3.0?) for this request, as was discussed in Issue 108. It is a working in progress for the entire 1.0 specification but for my use case it already works. I have made a quick shortlist of things we should add for a complete 1.0 release in the README. Also we can probably use more unit tests.
Also @AnzeDspot, @therealgambo

Coverage decreased (-18.3%) to 70.892% when pulling 3e74f208930fcf2344e86def8cd8384aba237e0d on InfopactMLoos:v3.0 into 2f85b8de28fdf3276c9f61fdfdcf6862bca3017c on alecsammon:master.

Coverage decreased (-18.3%) to 70.892% when pulling 3e74f208930fcf2344e86def8cd8384aba237e0d on InfopactMLoos:v3.0 into 2f85b8de28fdf3276c9f61fdfdcf6862bca3017c on alecsammon:master.

Coverage decreased (-18.3%) to 70.892% when pulling 3e74f208930fcf2344e86def8cd8384aba237e0d on InfopactMLoos:v3.0 into 2f85b8de28fdf3276c9f61fdfdcf6862bca3017c on alecsammon:master.

@InfopactMLoos , Sorry for the late reply. I'll have a look at this hopefully within the next day or so.
As for whether this eventually gets merged, I'm unsure as it looks like @alecsammon is rather inactive at the moment. I'll be happy to install directly from your fork, assuming it is tagged correctly and won't go away anytime soon :)

What's the status of this? I'd like to use this library, but I'm only use RAML 1.0.
This PR was opened over a year ago, and there's been no update on this at all.

I don't think anything is being merged at the moment. You should read over this issue: https://github.com/alecsammon/php-raml-parser/issues/117
The eLama fork is more inline with the 1.0 spec than this.

The @eLama 's fork with RAML 1.0 support is now backported to master. There is a new version (3.0.0) release that contains it so please do let me know if you run into any issues with it.


IIS cannot access the WSL 2 file system in 20H1

Windows build number:  Microsoft Windows [Version 10.0.19041.207]

What you're doing and what's happening:
Attempting to serve WSL 2 filesystem content through a Windows 10 build 2004 IIS virtual directory with UNC path fails.

What's wrong / what should be happening instead:
e.g. \\wsl$\Ubuntu\Websites\TestWeb
where  \Websites\TestWeb on WSL 2 host has been granted unrestricted access 777 (read/write/executable access for user/group/all)
Testing the connection within IIS reports:

Pass through authentication succeeds.
Authorization fails ( but reports "The application pool identity is valid." )

Exploring the UNC path shows the files but browsing to content reports:

Detailed Error Information

Module
IIS Web Core

Notification
BeginRequest

Handler
Not yet determined

Error Code
0x80070043

Config Error
Cannot read configuration file

Config File
\?\UNC\wsl$\Ubuntu\Websites\TestWeb\web.config

This fails using the built in account or supplying credentials for a local account with access to the path.

This issue persists in build 19041.208

I also have this issue.
And I think it happens because IIS by default uses the ApplicationPoolIdentity built-in account for running the application pool worker process. and since WSL distro is only available per-user and not shared between other accounts, there is no way that the built-in account would work.
So I've tried the  Connet As option with my local username and password, and instead got this Authorization error:
The path does not exist or environment variables in the path could not be expanded to verify whether it exists.
Since the Test Settings takes some time to verify the path, it seems like it does recognize the \\wsl$\ part as a network location, but it has no access rights... But that's just an assumption, and I have no idea why this is happening.
Did you maybe have any progress on this so far?
I think It is really an essential thing and makes WSL useless for any kind of .NET web development...

I have the same issue on build 19041.329.
I worked through the same steps as @stevium and get the same results after changing the AppPool to use my local admin user.

I think (?) this may potentially be fixed in build 20175.
According to the release notes at https://docs.microsoft.com/en-us/windows/wsl/release-notes

Change \wsl$ prefix to \wsl to support URI parsing. The old \wsl$ path is still supported.

This may allow the UNC path resolve correctly. I've not installed the preview version yet to try it as it is a bleeding edge build.

Anyone got this working?

I'm trying to get this working as well. No dice.
Each of these applicationhost.config blocks result in the error below:
applicationhost.config
<virtualDirectory path="/somepath" physicalPath="\\wsl$\Ubuntu-20.04\home\myuser\myproject\somepath" />

applicationhost.config
<virtualDirectory path="/somepath" physicalPath="\\wsl.localhost\Ubuntu-20.04\home\myuser\myproject\somepath" />

Error

HTTP Error 500.19 - Internal Server Error
The requested page cannot be accessed because the related configuration data for the page is invalid.

This is a broad work-around and doesn't work for all scenarios. This may be obvious to some but in-case anyone was racking their brains against this you can always do:
<UseGlobalApplicationHostFile>true</UseGlobalApplicationHostFile>

<UseGlobalApplicationHostFile>true</UseGlobalApplicationHostFile> or mapping to a network drive doesn't seem to work


what would it take to get this compiled on a mac?

Hi,
Read your paper and saw this implementation which is exciting.
Want to use LFTags in a project I'm working on, but cannot get this to compile on a mac.
cargo build uses the v4l2 library somewhere in the dependency chain and this causes the build to fail:
   Compiling rscam v0.5.5
   Compiling deflate v0.7.20
   Compiling stb_truetype v0.3.1
error: rscam (v4l2) is for linux/freebsd only
  --> /Users/Vishesh/.cargo/registry/src/github.com-1ecc6299db9ec823/rscam-0.5.5/src/lib.rs:28:1
   |
28 | compile_error!("rscam (v4l2) is for linux/freebsd only");
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error[E0425]: cannot find value `VIDIOC_STREAMON` in module `v4l2`
   --> /Users/Vishesh/.cargo/registry/src/github.com-1ecc6299db9ec823/rscam-0.5.5/src/lib.rs:690:37
    |
690 |         v4l2::xioctl(self.fd, v4l2::VIDIOC_STREAMON, &mut typ)?;
    |                                     ^^^^^^^^^^^^^^^ not found in `v4l2`

error[E0425]: cannot find value `VIDIOC_STREAMOFF` in module `v4l2`
   --> /Users/Vishesh/.cargo/registry/src/github.com-1ecc6299db9ec823/rscam-0.5.5/src/lib.rs:697:37
    |
697 |         v4l2::xioctl(self.fd, v4l2::VIDIOC_STREAMOFF, &mut typ)?;
    |                                     ^^^^^^^^^^^^^^^^ not found in `v4l2`

   Compiling quote v1.0.3
error: aborting due to 3 previous errors

For more information about this error, try `rustc --explain E0425`.
error: could not compile `rscam`.

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: build failed

is there some alternate way to get this working without rscam/v4l2 - some mac equivalent?

I think the dependency comes from camera_capture, which unfortunately only supports Windows and Linux. I think that actually should be a dev dependency, as its not required for detecting or localizing LFTags in images, and is only required for the example code which detects tags from the webcam.
If you are interested in detecting tags in images/videos, try to move the camera_capture dependency into dev-dependencies (and if it works, a pull request with that modification would be great), but if you want to detect tags from the webcam, there is no rust library for webcam capture as far as I know.

I have moved the dependency to dev as you suggested, and now am encountering a different error (I guess it finished building dependencies and is now trying to build the package)...
error[E0554]: `#![feature]` may not be used on the stable release channel
 --> src/lib.rs:1:1
  |
1 | #![feature(test)]
  | ^^^^^^^^^^^^^^^^^

do you build against nightly/is there some other way to build this?
also as far as reading video frames goes - I notice you have opencv as a dependency. does it's VideoCapture module not work for reading frames from a webcam? (I would like to detect using a webcam, hence I ask)
thanks again for all the help and sure if i get this working I can make a pull request

Hi,
back to trying to get this to work.
I have changed the imageproc depedency to 0.19.0 from 0.18.0 to solve a compiler error (imageproc 0.18 doesn't compile on rust stable > 1.48), and now am seeing:
error[E0277]: the trait bound `Luma<u16>: image::traits::Pixel` is not satisfied
   --> src/decode.rs:316:5
    |
316 | /     pub fn decode_lftag(root: &LFTagComponents, camera: CameraIntrinsics, gradient: &mut Image<Luma<u16>>) -> Option<DecodedLFTag> {
317 | |         if let LFTagComponents::Root { nodes, class, bg } = root {
318 | |             let mut fvs: Vec<_> = nodes.iter().map(|n| {
319 | |                 if let LFTagComponents::Normal {node: fv} = n {
...   |
554 | |         }
555 | |     }
    | |_____^ the trait `image::traits::Pixel` is not implemented for `Luma<u16>`

error: aborting due to previous error; 22 warnings emitted

For more information about this error, try `rustc --explain E0277`.
error: could not compile `fiducial`

How do I go about figuring out what type signature would work? I checked the image crate documentation and found nothing that would explain why Luma<u16> doesn't work and what I could replace it with.
Thanks in advance.

I suspect the issue is that the versions of imageproc and image need to be aligned, try bumping the image crate version too?
For easier discussion, would you mind joining the rust-cv discord to discuss? I often hang out there (I go by @kindiana there) and I think synchronously working through these issues would be easier.

alright, i'm joining the discord. thank you for all the help.
bumping the image crate and image proc to newest versions didn't help - generated more errors, actually.


Racer spawns a large number of one-off threads... why?

On src/racer/ast.rs, there are multiple methods whose implementation looks something like this:
    let thread = Thread::scoped(move || {
        let stmt = string_to_stmt(s);
        let mut v = ImplVisitor { name_path: None, trait_path: None };
        visit::walk_stmt(&mut v, &*stmt);
        return v;
    });
    return thread.join().ok().unwrap();

Since this is using Thread::scoped (which makes the parent thread wait on the thread.join() call), there isn't any gain in performance here - on the contrary, there's a big performance loss. On my linux system, a call to racer complete std::old_io:: spawns 108 threads, and spends almost half of its execution time on the futex syscall (thread synchronization).
On a comment, there's a possible justification for this - handling panic!s from the parser, since Rust provides no way to "catch" panic!s. The problem with this justification is twofold - first, methods as the one above (which are by far the most common on the file I mentioned) aren't actually catching anything, since they unwrap() the final result of thread.join() - meaning that a panic! will be propagated to the parent thread anyway. Secondly, per rust-lang/rust#22435, "catching" panics at the border of a scoped thread won't be supported by the stdlib anymore - which means that racer won't be able to use this technique anymore in a few days.
I'm opening this issue so that we can discuss a possible solution for this - hopefully one that doesn't use threads for control flow :)

Hi Renato,
You are correct in thinking it was originally to catch panics. Racer has got much better at passing parsable blobs of code to libsyntax now so it's not needed and remains mainly because of my lazy copy-paste coding style! Feel free to fix if you want, or I'll repair it when rust-lang/rust#22435 hits.
BTW Thanks for pointing me at rust-lang/rust#22435 - I'll check out the discussion.
Cheers,
Phil

This is great, thanks Renato


When using SLSQP for minima, changes in the order of the constraints can lead to changes in the results

I am solving a minimum problem using SCIPY
When using SLSQP for minima, changes in the order of the constraints can lead to changes in the results
The following is my code, I found that as long as the order of constraint1 to constraint5 in cons is disturbed, the result will change (the range will not be very large)
from scipy.optimize import minimize
def objective(x):
    return x[0]**2 + x[1]**2 + x[2]**2 + x[3]**2 + x[4]**2
def constraint1(x):
    return x[0] + x[1] + x[2] - 1

def constraint2(x):
    return x[0] - x[1] + x[3] - 2

def constraint3(x):
    return x[2] + x[4] - 3

def constraint4(x):
    return x[0] + x[1] - x[2] - 4

def constraint5(x):
    return x[3] + x[4] - 5

x0 = [0, 0, 0, 0, 0]

cons = [{'type': 'eq', 'fun': constraint1},
        {'type': 'ineq', 'fun': constraint2},
        {'type': 'ineq', 'fun': constraint3},
        {'type': 'ineq', 'fun': constraint4},
        {'type': 'ineq', 'fun': constraint5}]

result = minimize(objective, x0, constraints=cons)

The following is my code, I found that as long as the order of constraint1 to constraint5 in cons is disturbed, the result will change (the range will not be very large)
Is it an algorithm problem or a code problem? If I still want to use the SLSQP algorithm, can I keep the result unchanged?

It sounds like you can keep the result unchanged by not changing the order of the constraints, right? You could also choose a guess that is closer to the solution you're interested in, or add constraints that restrict the solution to the one you prefer.

Thank you for your reply.
Yes, I can find a best fit order by trying multiple order of constraints so the result won't change.
But what makes me question is exactly what I asked, why is the order of the constraints changed (of course, the content of each constraint has not changed), will it lead to a change in the local minimum?
Is this a math problem, or is there something wrong with the code?

@mdhaber

It may be too much to expect that changing the problem formulation wouldn't change the result.

As I asked, does merely changing the order of the constraints essentially modify the question?
I think just a change in order should not lead to a change in the solution space? The local minimum should not change.

As I asked, does merely changing the order of the constraints essentially modify the question? I think just a change in order should not lead to a change in the solution space?

No, it doesn't change the mathematical question, and indeed, you always get valid solutions to the original question.

The local minimum should not change.

That doesn't follow.
x = 1e20

x += 1
x += -1e20

x

Mathematically, addition is commutative, but swap the middle lines and you'll get a different result. Numerical algorithms don't always work in ways that are intuitive from the outside.

@liuhuanshuo it's been a few days, so I'll go ahead and close this, but please reopen it it has not resolved your concern. Thanks!


version 1.1.0a2: JSONExportableRootDict() BREAKING CHANGES

JSONExportableRootDict(): do not inherit from JSONExportable()
Increased test coverage

Codecov Report
Attention: 33 lines in your changes are missing coverage. Please review.

Comparison is base (9193bbb) 78.40% compared to head (2357ed9) 80.32%.
Report is 7 commits behind head on main.

Files
Patch %
Lines

src/pydantic_exportables/jsonexportable.py
53.03%
31 Missing :warning:

tests/test_exportable_importable.py
96.96%
1 Missing :warning:

tests/test_pyobjectid.py
96.15%
1 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #18      +/-   ##
==========================================
+ Coverage   78.40%   80.32%   +1.92%
==========================================
  Files          10       11       +1
  Lines        1139     1230      +91
==========================================
+ Hits          893      988      +95
+ Misses        246      242       -4

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Feature/multi webserver support pr

Sorry, take two.

Nice mate, great simplification with getting rid of grunt, et al too!  No issues from me if you want to merge it in

Danke! I'll bump the version and upload it to npm too if you're happy with that. I think this is non-breaking, however, just to be sure I'll go up a major version.

All good mate - go for it


Datalab with Dataproc - Current script doesn't open port 8080

gs://dataproc-initialization-actions/datalab/datalab.sh
I haven't had a chance to test but it looks like this function is missing the -p 8080:8080 parameter.
Current
function run_datalab(){   if docker run -d --restart always --net=host \       -v "${DATALAB_DIR}:/content/datalab" ${VOLUME_FLAGS} datalab-pyspark; then     echo 'Cloud Datalab Jupyter server successfully deployed.'   else     err 'Failed to run Cloud Datalab'   fi }
updated?
function run_datalab(){   if docker run -d --restart always --net=host  -p 8080:8080\       -v "${DATALAB_DIR}:/content/datalab" ${VOLUME_FLAGS} datalab-pyspark; then     echo 'Cloud Datalab Jupyter server successfully deployed.'   else     err 'Failed to run Cloud Datalab'   fi }

dustyny@, it is using --net=host, so -p 8080:8080 is no longer needed. See Use host networking.


Make Library Components Reactive

Pending explicit direction from client on what meta-data makes up the library.

@micchickenburger : not sure when this type of meta will be defined.

This one is in your court @micchickenburger  at defining what the library will be content and meta wise. Let me know once WCASA has defined.

Handled in Pull Request #246


Adding IgnoreUnrecognizedVMOptions to make worker start with jdk-11

Issue:

Many of the default worker JVM parameters is no longer supported in java versions > 11. This will result in the worker process getting crashed. eg: https://github.com/31z4/storm-docker/issues/32
The official storm docker hub (https://hub.docker.com/_/storm) now ships jdk-11 and worker process fails with Error: Could not create the Java Virtual Machine.

What is the purpose of the change
To make storm workers run in jdk-11 and above
(Explain why we should have this change)
Storm workers are not able to start in jdk-11 and above
How was the change tested
Tested by running " storm supervisor -c worker.childopts= .. " with open-jdk11 and see that the topology is working as expected and worker process is getting started.
(Explain what tests did you do to verify the code change)
before fix:
2022-12-07 17:38:27.653 o.a.s.d.s.BasicContainer SLOT_6700 [INFO] Launching worker with command: '/usr/local/openjdk-11/bin/java' '-cp' '/apache-storm-2.4.0/lib-worker/:/apache-storm-2.4.0/extlib/:/conf:/data/supervisor/stormdist/exclaimtopology-1-1670434694/stormjar.jar' '-Xmx64m' '-Dlogging.sensitivity=S3' '-Dlogfile.name=worker.log' '-Dstorm.home=/apache-storm-2.4.0' '-Dworkers.artifacts=/logs/workers-artifacts' '-Dstorm.id=exclaimtopology-1-1670434694' '-Dworker.id=f52f7411-c5dd-4a1f-941f-24a316d0e0fd' '-Dworker.port=6700' '-Dstorm.log.dir=/logs' '-DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector' '-Dstorm.local.dir=/data' '-Dworker.memory_limit_mb=512' '-Dlog4j.configurationFile=/apache-storm-2.4.0/log4j2/worker.xml' 'org.apache.storm.LogWriter' '/usr/local/openjdk-11/bin/java' '-server' '-Dlogging.sensitivity=S3' '-Dlogfile.name=worker.log' '-Dstorm.home=/apache-storm-2.4.0' '-Dworkers.artifacts=/logs/workers-artifacts' '-Dstorm.id=exclaimtopology-1-1670434694' '-Dworker.id=f52f7411-c5dd-4a1f-941f-24a316d0e0fd' '-Dworker.port=6700' '-Dstorm.log.dir=/logs' '-DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector' '-Dstorm.local.dir=/data' '-Dworker.memory_limit_mb=512' '-Dlog4j.configurationFile=/apache-storm-2.4.0/log4j2/worker.xml' '-Xmx512m' '-XX:+PrintGCDetails' '-Xloggc:artifacts/gc.log' '-XX:+PrintGCDateStamps' '-XX:+PrintGCTimeStamps' '-XX:+UseGCLogFileRotation' '-XX:NumberOfGCLogFiles=10' '-XX:GCLogFileSize=1M' '-XX:+HeapDumpOnOutOfMemoryError' '-XX:HeapDumpPath=artifacts/heapdump' '-Djava.library.path=/data/supervisor/stormdist/exclaimtopology-1-1670434694/resources/Linux-amd64:/data/supervisor/stormdist/exclaimtopology-1-1670434694/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/lib64' '-Dstorm.conf.file=' '-Dstorm.options=worker.childopts%3D-Xmx512m+-XX%3A%2BPrintGCDetails+-Xloggc%3Aartifacts%2Fgc.log+-XX%3A%2BPrintGCDateStamps+-XX%3A%2BPrintGCTimeStamps+-XX%3A%2BUseGCLogFileRotation+-XX%3ANumberOfGCLogFiles%3D10+-XX%3AGCLogFileSize%3D1M+-XX%3A%2BHeapDumpOnOutOfMemoryError+-XX%3AHeapDumpPath%3Dartifacts%2Fheapdump' '-Djava.io.tmpdir=/data/workers/f52f7411-c5dd-4a1f-941f-24a316d0e0fd/tmp' '-cp' '/apache-storm-2.4.0/lib-worker/:/apache-storm-2.4.0/extlib/:/conf:/data/supervisor/stormdist/exclaimtopology-1-1670434694/stormjar.jar' 'org.apache.storm.daemon.worker.Worker' 'exclaimtopology-1-1670434694' '21d4cb06-d0bc-44c6-bc73-b56078eadb82-<IP_ADDRESS>' '6628' '6700' 'f52f7411-c5dd-4a1f-941f-24a316d0e0fd'.
2022-12-07 17:38:27.679 o.a.s.d.s.Slot SLOT_6700 [INFO] STATE waiting-for-blob-localization msInState: 183 -> waiting-for-worker-start msInState: 0 topo:exclaimtopology-1-1670434694 worker:f52f7411-c5dd-4a1f-941f-24a316d0e0fd
2022-12-07 17:38:27.679 o.a.s.d.s.Slot SLOT_6700 [INFO] SLOT 6700: Changing current assignment from null to LocalAssignment(topology_id:exclaimtopology-1-1670434694, executors:[ExecutorInfo(task_start:4, task_end:4), ExecutorInfo(task_start:3, task_end:3), ExecutorInfo(task_start:2, task_end:2), ExecutorInfo(task_start:1, task_end:1)], resources:WorkerResources(mem_on_heap:512.0, mem_off_heap:0.0, cpu:40.0, shared_mem_on_heap:0.0, shared_mem_off_heap:0.0, resources:{offheap.memory.mb=0.0, onheap.memory.mb=512.0, cpu.pcore.percent=40.0}, shared_resources:{}), owner:storm)
2022-12-07 17:38:29.077 o.a.s.d.s.BasicContainer Thread-7 [INFO] Worker Process f52f7411-c5dd-4a1f-941f-24a316d0e0fd exited with code: 1
After:
2022-12-07 17:48:06.394 o.a.s.d.s.BasicContainer SLOT_6700 [INFO] Launching worker with command: '/usr/local/openjdk-11/bin/java' '-cp' '/apache-storm-2.4.0/lib-worker/:/apache-storm-2.4.0/extlib/:/conf:/data/supervisor/stormdist/exclaimtopology-1-1670434694/stormjar.jar' '-Xmx64m' '-Dlogging.sensitivity=S3' '-Dlogfile.name=worker.log' '-Dstorm.home=/apache-storm-2.4.0' '-Dworkers.artifacts=/logs/workers-artifacts' '-Dstorm.id=exclaimtopology-1-1670434694' '-Dworker.id=2a1f426e-dbf0-453b-a8bb-d1fe981b5ada' '-Dworker.port=6700' '-Dstorm.log.dir=/logs' '-DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector' '-Dstorm.local.dir=/data' '-Dworker.memory_limit_mb=512' '-Dlog4j.configurationFile=/apache-storm-2.4.0/log4j2/worker.xml' 'org.apache.storm.LogWriter' '/usr/local/openjdk-11/bin/java' '-server' '-Dlogging.sensitivity=S3' '-Dlogfile.name=worker.log' '-Dstorm.home=/apache-storm-2.4.0' '-Dworkers.artifacts=/logs/workers-artifacts' '-Dstorm.id=exclaimtopology-1-1670434694' '-Dworker.id=2a1f426e-dbf0-453b-a8bb-d1fe981b5ada' '-Dworker.port=6700' '-Dstorm.log.dir=/logs' '-DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector' '-Dstorm.local.dir=/data' '-Dworker.memory_limit_mb=512' '-Dlog4j.configurationFile=/apache-storm-2.4.0/log4j2/worker.xml' '-Xmx512m' '-XX:+IgnoreUnrecognizedVMOptions' '-XX:+PrintGCDetails' '-Xloggc:artifacts/gc.log' '-XX:+PrintGCDateStamps' '-XX:+PrintGCTimeStamps' '-XX:+UseGCLogFileRotation' '-XX:NumberOfGCLogFiles=10' '-XX:GCLogFileSize=1M' '-XX:+HeapDumpOnOutOfMemoryError' '-XX:HeapDumpPath=artifacts/heapdump' '-Djava.library.path=/data/supervisor/stormdist/exclaimtopology-1-1670434694/resources/Linux-amd64:/data/supervisor/stormdist/exclaimtopology-1-1670434694/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/lib64' '-Dstorm.conf.file=' '-Dstorm.options=worker.childopts%3D-Xmx512m+-XX%3A%2BIgnoreUnrecognizedVMOptions+-XX%3A%2BPrintGCDetails+-Xloggc%3Aartifacts%2Fgc.log+-XX%3A%2BPrintGCDateStamps+-XX%3A%2BPrintGCTimeStamps+-XX%3A%2BUseGCLogFileRotation+-XX%3ANumberOfGCLogFiles%3D10+-XX%3AGCLogFileSize%3D1M+-XX%3A%2BHeapDumpOnOutOfMemoryError+-XX%3AHeapDumpPath%3Dartifacts%2Fheapdump' '-Djava.io.tmpdir=/data/workers/2a1f426e-dbf0-453b-a8bb-d1fe981b5ada/tmp' '-cp' '/apache-storm-2.4.0/lib-worker/:/apache-storm-2.4.0/extlib/:/conf:/data/supervisor/stormdist/exclaimtopology-1-1670434694/stormjar.jar' 'org.apache.storm.daemon.worker.Worker' 'exclaimtopology-1-1670434694' '832aed9c-a9ef-4100-b74b-0d28069b55e0-<IP_ADDRESS>' '6628' '6700' '2a1f426e-dbf0-453b-a8bb-d1fe981b5ada'.
2022-12-07 17:48:06.420 o.a.s.d.s.Slot SLOT_6700 [INFO] STATE waiting-for-blob-localization msInState: 224 -> waiting-for-worker-start msInState: 0 topo:exclaimtopology-1-1670434694 worker:2a1f426e-dbf0-453b-a8bb-d1fe981b5ada
2022-12-07 17:48:06.421 o.a.s.d.s.Slot SLOT_6700 [INFO] SLOT 6700: Changing current assignment from null to LocalAssignment(topology_id:exclaimtopology-1-1670434694, executors:[ExecutorInfo(task_start:4, task_end:4), ExecutorInfo(task_start:3, task_end:3), ExecutorInfo(task_start:2, task_end:2), ExecutorInfo(task_start:1, task_end:1)], resources:WorkerResources(mem_on_heap:512.0, mem_off_heap:0.0, cpu:40.0, shared_mem_on_heap:0.0, shared_mem_off_heap:0.0, resources:{offheap.memory.mb=0.0, onheap.memory.mb=512.0, cpu.pcore.percent=40.0}, shared_resources:{}), owner:storm)
2022-12-07 17:48:11.440 o.a.s.d.s.Slot SLOT_6700 [INFO] STATE waiting-for-worker-start msInState: 5021 topo:exclaimtopology-1-1670434694 worker:2a1f426e-dbf0-453b-a8bb-d1fe981b5ada -> running msInState: 1 topo:exclaimtopology-1-1670434694 worker:2a1f426e-dbf0-453b-a8bb-d1fe981b5ada

@bipinprasad , can this be merged to master?

@bipinprasad , can you please merge to master if all looks good ?


[No QA] Add green dot in LHN when chat is unread with mention

Details

This PR adds handling to the front-end to show the green dot in the LHN when a user's chat is unread and they have been mentioned. It goes with this back-end PR which handles setting the lastMentionedTime field on the report. The green dot should show up any time a chat is unread and the lastMentionedTime is after the lastReadTime on the chat. In other words, the green dot should show when the user has been mentioned in any of the unread messages.
Fixed Issues

$ https://github.com/Expensify/App/issues/17887
Tests

Sign into any account
Go to a chat and mark the last message as unread
Navigate to a different chat
In the JS console, set the lastMentionedTime field for the unread chat to the same time as the lastVisibleMessageCreated on that report like this Onyx.merge('report_7152000805184733', {"lastMentionedTime": "2023-04-24 20:55:21.858"});
Verify that the green dot shows up in the LHN on the report
Clear the lastMentionedTime field in Onyx like this Onyx.merge('report_7152000805184733', {"lastMentionedTime": ""});
Verify that the green dot goes away
Set the lastMentionedTime field again so that the green dot comes back
Click into the unread chat so that it becomes read
Verify that the green dot goes away
Mark the last message unread and stay in the chat
Verify that the green dot shows up in the LHN on the report

[x] Verify that no errors appear in the JS console

Offline tests

Same as regular tests. This behavior is purely based on the data available in Onyx.
QA Steps

There is nothing for QA to test yet because this field isn't yet being set by the back-end.

[ ] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[ ] I ran the tests on all platforms & verified they passed on:

[ ] Android / native
[x] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[x] MacOS / Chrome / Safari
[x] MacOS / Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
[x] I have checked off every checkbox in the PR author checklist, including those that don't apply to this PR.

Screenshots/Videos

Web

https://user-images.githubusercontent.com/1371996/234722397-bb89e3f3-b5ce-4002-9f19-979456dfa65e.mp4

Mobile Web - Chrome

Mobile Web - Safari

Desktop

https://user-images.githubusercontent.com/1371996/234722875-8a69472c-baba-498a-9f03-60635bc98ece.mp4

iOS

Android

@0xmiroslav @grgia One of you needs to copy/paste the Reviewer Checklist from here into a new comment on this PR and complete it. If you have the K2 extension, you can simply click: [this button]

@0xmiroslav or @grgia do you happen to know how to post to the JS console on iOS and Android in the emulator (it's needed for the test steps)? I was able to figure it out for  android chrome/ios safari, but couldn't figure it out for the native apps.

Ahh, smart @0xmiroslav.
@grgia any other comments? Otherwise, I would love to merge today!

Thanks for the reviews. Gonna merge!

🚀 Deployed to staging by https://github.com/puneetlath in version: 1.3.9-12 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to production by https://github.com/roryabraham in version: 1.3.12-0 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


Typo

https://github.com/buildingSMART/ifc-gherkin-rules/blob/81b07c03b8c9726b4c0b580eb0efd63c1899ab2e/features/ALB002_Alignment-layout.feature#L52-L59
In these Scenarios the word "horizontal" in the agreement statement should be replaced by "vertical" and "cant" respectively

Fixed in https://github.com/buildingSMART/ifc-gherkin-rules/pull/6/commits/66f3c3c4c014422b1816bfd077ed089798ed1dc2


Allow override of certificate management for flexibility and security

Allowing users to override certificate management enables them to use custom certificates as well as not have to store private keys in their module repositories.
The default behavior is retained in this PR as the default value is true.

This would also provide the flexibility requested in PR 71.

Just a note here, the expired certs bit me and broke indexing in one of my clusters a while back.  This PR would prevent this for me and others.


[getopt-win32] should use official repo

Is your feature request related to a problem? Please describe.
https://github.com/microsoft/vcpkg/blob/c6d6efed3e9b4242765bfe1b5c5befffd85f7b92/ports/getopt-win32/portfile.cmake#L3 now the getopt-win32 not using offcial repo https://github.com/libimobiledevice-win32/getopt"
Proposed solution
should using official repo. any change should use patch or summit pr to upstream.
Describe alternatives you've considered
No response
Additional context
No response

Why do you think libimobiledevice-win32/getopt  is the official repo?
Are you aware of the problems with the past state of those libimobiledevice(-win32) ports?

In particular, read https://github.com/microsoft/vcpkg/pull/33246#issuecomment-1710785895 and following.

@jiayuehua, Thanks for posting this issue, I think @xiaozhuai has explained your question in comment, so I will close this issue.


Using DeepExplainer for LSTM : AttributeError: 'Deep' object has no attribute 'masker'

We need SHAP plots for our model following model:

We attempt to use SHAP:

And face the following error:
In [56]: import shap
...: explainer = shap.DeepExplainer(model2, x_train_appended)
...: shap_values = explainer(x_train_appended)
WARNING:tensorflow:Layers in a Sequential model should only have a single input tensor, but we receive a <class 'list'> input: [<tf.Tensor: shape=(49586, 1, 23), dtype=float32, numpy=
array([[[0.40824828, 0.02564103, 0.03370786, ..., 0.4494382 ,
0.43333334, 0.59210527]],
   [[0.        , 0.06410257, 0.05617978, ..., 0.4494382 ,
     0.43333334, 0.59210527]],

   [[0.5400617 , 0.06410257, 0.06741573, ..., 0.4494382 ,
     0.43333334, 0.59210527]],

   ...,

   [[0.5400617 , 0.01282051, 0.05617978, ..., 0.07865169,
     0.01111111, 0.05263158]],

   [[0.        , 0.02564103, 0.05617978, ..., 0.07865169,
     0.01111111, 0.05263158]],

   [[0.        , 0.02564103, 0.05617978, ..., 0.07865169,
     0.01111111, 0.05263158]]], dtype=float32)>]

Consider rewriting this model with the Functional API.
Traceback (most recent call last):
File "", line 3, in
shap_values = explainer(x_train_appended)
File "/home/kiton/.local/lib/python3.8/site-packages/shap/explainers/_explainer.py", line 207, in call
if issubclass(type(self.masker), maskers.OutputComposite) and len(args)==2:
AttributeError: 'Deep' object has no attribute 'masker'
It seems like the explainer(x_train_appended) is responsible for this error.
We are using:
numba                        0.55.1
numpy                        1.18.5
shap                         0.40.0
Keras                        2.4.3

A numpy array should be dense already, if you're not sure that it is, you can call your_data.toarray()

Thanks a lot Remo! Yes it is already dense. Do you have any idea what else
might be causing it? Thanks a lot for the help!
On Wed, Jul 13, 2022, 10:50 AM Remo @.***> wrote:

A numpy array should be dense already, if you're not sure that it is, you
can call your_data.toarray()
—
Reply to this email directly, view it on GitHub
https://github.com/slundberg/shap/issues/2614#issuecomment-1183393422,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AI4RGAAWQZNMMDXENP3XHMTVT3QWBANCNFSM53BEZLVQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

The masker error should be related to the background distribution you pass. I think it might be solved in shap == 0.41
There are a few things I would try:

Pass the models predict function directly, instead of the model
explainer = shap.DeepExplainer(model_2.predict, x_train_append)

You can try not calling the DeepExplainer, but just let SHAP select an explainer, like so:
explainer = shap.Explainer(model_2, x_train_append)

You can test if your problem is explainer specific, by using a model agnostic explainer:
explainer = shap.explainers.Sampling(model_2.predict, shap.kmeans(x_train_append,50))

One last thing, regarding this warning:
WARNING:tensorflow:Layers in a Sequential model should only have a single input tensor, but we receive a <class 'list'>
Are you passing the input as a list somehow? If this is the case, then that would probably break the explainer. Your input should be one array, not a list of arrays.

Thanks a ton, Remo! This is very helpful, Let us try these out.

Yes @remo-help you are right, it has batch size as one dimension. Let me try copying the documentation code and go ahead from there.
Thanks a lot for your help!

use shap_values = explainer.shap_values(X) to get these shap_values. make a try.

for me, the question was caused by the minor inconsistency between the model structure for training and the model structure used in SHAP.
the question got solved after I consistified the structures

use shap_values = explainer.shap_values(X) to get these shap_values. make a try.

This actually solves the issue ... thanks
I was facing the error "AttributeError: 'Kernel' object has no attribute 'masker'" and this recommendation solved it.

@vatsaltailor Hello, I encountered the same problem when training the lstm model. I see that your previous solution is that deepexplainer requires 2D Numpy input, and the model is 3D input. So how to change the input dimension to be suitable for deepexplainer ?  And my current network input shape is ( None, 128, 2 ) thank you~~

@remo-help Thank you very much for your timely help ! I 'll try it right now. I consulted other materials and saw an article explaining the problem of lstm application shap, which should be the problem of input dimension ! Seeing that you have helped solve a lot of problems under the relevant issues, you are so enthusiastic and helpful !

Hi. Does one have a solution to this problem? Currently I am training a RNN and LSTM with a sequence length of 24 and 49 features. Therefore, I have data like this (number of samples, 24, 49). This I can do fine
import shap n_traing_samples = 2 explainer = shap.DeepExplainer(model, X_train) shap_values = explainer.shap_values(X_train[:n_traing_samples]
and if I want to use a summary plot, this works
shap.summary_plot(shap_values[:, :, :, 0].reshape(n_traing_samples*24, 49), X_train[:2].reshape(n_traing_samples*24, 49), plot_type='bar', feature_names=FEATURES, plot_size=(10, 5))
but I can't do a beeswarm-plot or dependence plot since it requires an Explanation object and I won't allow me to do this
explainer = shap.DeepExplainer(model, X_train) shap_obj = explainer(X_train[:n_traing_samples])
or this
explainer = shap.DeepExplainer(model, X_train) shap_obj = explainer(X_train[:n_traing_samples].reshape(n_traing_samples*24, 49))


ChefSpec deprecation warnings in libraries/matchers.rb

Running ChefSpec on a cookbook that includes this cookbook causes deprecation warning spam like so:
[DEPRECATION] `ChefSpec::Runner.define_runner_method' is deprecated. Please use `ChefSpec.define_matcher' instead.

@mwrock Hey! Looks like the relevant PR for this we merged in and released. Can this issue be closed?

Sounds good

Fixed by #132


Always display test coverage; add tests

Hello!
Pull Request overview

When pytest is used:

Always display test coverage.
Always display the 10 slowest tests.

Add passing data.py tests.
Add skipped tests for onnx that fail currently.
Ensure that model.onnx is always deleted, even on test failures.

Details
pytest improvements
This PR introduces .coveragerc, the recommended configuration file for coverage. It's fairly standard, except for the exclusion of if TYPE_CHECKING: branches. As these are never true for users, we don't need to worry about the True branch here, so we can safely exclude these branches.
Additionally, setup.cfg is expanded with:
[tool:pytest]
testpaths = tests
addopts = --cov=setfit --durations=10

This implies pytest will now be equivalent to pytest --cov=setfit --durations=10 ./tests.

The --cov=setfit will print an ascii coverage report at the end of the pytest run, like so:---------- coverage: platform win32, python 3.10.8-final-0 -----------
Name                                 Stmts   Miss Branch BrPart  Cover
----------------------------------------------------------------------
src\setfit\__init__.py                   5      0      0      0   100%
src\setfit\data.py                     114      0     54      0   100%
src\setfit\exporters\__init__.py         0      0      0      0   100%
src\setfit\exporters\onnx.py            95     22     34      8    72%
src\setfit\exporters\utils.py            4      2      0      0    50%
src\setfit\integrations.py              23      0      4      2    93%
src\setfit\logging.py                  142     61     26      2    52%
src\setfit\modeling.py                 313     50    102     20    81%
src\setfit\pipeline.py                   9      9      0      0     0%
src\setfit\trainer.py                  194     25     86     22    82%
src\setfit\trainer_distillation.py      69     17     22      7    67%
src\setfit\utils.py                     55     26      8      0    52%
----------------------------------------------------------------------
TOTAL                                 1023    212    336     61    77%

The --durations=10 will display the 10 slowest tests, like so:====================================================================================== slowest 10 durations ======================================================================================
14.96s call     tests/exporters/test_onnx.py::test_export_onnx_sklearn_head
9.91s call     tests/test_trainer.py::TrainerHyperParameterOptunaIntegrationTest::test_hyperparameter_search
4.96s call     tests/test_trainer_distillation.py::DistillationSetFitTrainerTest::test_trainer_works_with_default_columns
3.80s setup    tests/test_modeling.py::SetFitModelDifferentiableHeadTest::test_max_length_is_larger_than_max_acceptable_length
3.69s call     tests/test_trainer.py::SetFitTrainerTest::test_raise_when_metric_value_is_invalid
3.07s call     tests/test_trainer.py::SetFitTrainerTest::test_trainer_works_with_model_init
2.97s call     tests/test_trainer.py::SetFitTrainerTest::test_trainer_support_callable_as_metric
2.92s call     tests/test_trainer.py::SetFitTrainerMultilabelTest::test_trainer_multilabel_support_callable_as_metric
2.66s call     tests/test_data.py::test_create_fewshot_splits_with_augmentation
2.23s call     tests/test_trainer_distillation.py::DistillationSetFitTrainerTest::test_column_mapping_multilabel

Additionally, pytest --cov-report=html can be used to create a HTML report in a coverage_report_html folder.
New tests
For data.py I have introduced several new tests. Although this code likely isn't frequently used by users of SetFit, I wanted to try and understand it better, and what better way to understand code than to try and verify that it works correctly? 😄
For the ONNX tests I created a simple Trainer that trains a differentiable head, and then I export that model to ONNX in the same way as the existing test. The reasoning here was that exporting a model was only tested for the sklearn head. I discovered that the tests fail regardless of the out_features passed to the differentiable head.
FAILED tests/exporters/test_onnx.py::test_export_onnx_torch_head[1] - ValueError: Using a target size (torch.Size([4])) that is different to the input size (torch.Size([4, 1])) is deprecated. Please ensure they have the same size.
FAILED tests/exporters/test_onnx.py::test_export_onnx_torch_head[2] - assert False
FAILED tests/exporters/test_onnx.py::test_export_onnx_torch_head[3] - assert False

At out_features of 1 there is an issue with training, and at 2 and 3 the following errors pop up, respectively:
            # Compare the results and ensure that we get the same predictions.
>           assert np.array_equal(onnx_preds, pytorch_preds)
E           assert False
E            +  where False = <function array_equal at 0x000001559F262EF0>(array([[7.2937156e-03, 9.9270624e-01],\n       [9.9964154e-01, 3.5840381e-04]], dtype=float32), array([1, 0], dtype=int64))
E            +    where <function array_equal at 0x000001559F262EF0> = np.array_equal

            # Compare the results and ensure that we get the same predictions.
>           assert np.array_equal(onnx_preds, pytorch_preds)
E           assert False
E            +  where False = <function array_equal at 0x000001559F262EF0>(array([[4.0040349e-04, 9.4026297e-01, 5.9336655e-02],\n       [8.8793278e-01, 9.9355258e-02, 1.2711961e-02]], dtype=float32), array([1, 0], dtype=int64))
E            +    where <function array_equal at 0x000001559F262EF0> = np.array_equal

As you can see, the actual model predictions stay 1-dimensional: simply an array with the number of samples to predict for as the length. However, the ONNX predictions are 2-dimensional of the shape (num_samples, out_features). Regardless of whether this is an issue on the PyTorch model, the ONNX model should always be equivalent to the PyTorch model.
I've skipped these tests with the reason of "ONNX exporting of SetFit model with Torch head not yet supported."
Additionally, I've encompassed the body of the ONNX tests with a try-finally, where the model.onnx file is deleted in the finally clause. That way, the file gets deleted even when the test fails.
All in all, this has improved the test coverage from 72% to 77%. It would increase much more when ONNX tests pass, and I've now identified some key code that is never executed, like SetFitBaseModel, SKLearnWrapper, SetFitPipeline and a majority of logging.py code.

Tom Aarsen

My local machine's and the CI's isort seem to disagree on what "correctly formatted" constitutes. I'll investigate.


refactor: no packing inside lib

This PR makes combat much more effective / cheaper in gas and also cleans up the combat library code a bunch.
The main thing the PR does is that it removes the packing & unpacking of Army between combat library calls. It's not necessary. The only packing and unpacking we need to do is when dealing with storage (either writing to or reading from). All internal (library) functions should work on unpacked values only. This rule applies to the whole codebase. It's much more effective this way.
I've also removed the unnecessary defending_army_packed parameter to update_and_pack_army (the function is now called update_army).
Should be merged after #234 as it builds on it.

nice optimization! lgtm

Resolved conflicts using GitHub's UI, hopefully all good now.


RFE: enable encryption at rest where possible

All components of the stack (S3 SSE, Redis, RDS, ...) should use at least encryption at rest since users might be storing sensitive data depending on their legal requirements.

Encryption level status:

S3 => not added, can be done easily through CloudFormation
RDS => already done. check https://github.com/Rungutan/sentry-performance-monitoring/blob/main/cloudformation-template.yaml#L1402-L1403
Redis => cannot be done as it is not supported by CloudFormation -> https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-elasticache-cache-cluster.html
Kafka => already done. check https://github.com/Rungutan/sentry-performance-monitoring/blob/main/cloudformation-template.yaml#L1318-L1324
ClickHouse -> volumes themselves can be encrypted by updating the VolumeConfiguration of the Layer property as defined here -> https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-opsworks-layer-volumeconfiguration.html#cfn-opsworks-layer-volumeconfiguration-encrypted

Encryption level status:

S3 => not added, can be done easily through CloudFormation
RDS => already done. check https://github.com/Rungutan/sentry-performance-monitoring/blob/main/cloudformation-template.yaml#L1402-L1403
Redis => cannot be done as it is not supported by CloudFormation -> https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-elasticache-cache-cluster.html
Kafka => already done. check https://github.com/Rungutan/sentry-performance-monitoring/blob/main/cloudformation-template.yaml#L1318-L1324
ClickHouse -> volumes themselves can be encrypted by updating the VolumeConfiguration of the Layer property as defined here -> https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-opsworks-layer-volumeconfiguration.html#cfn-opsworks-layer-volumeconfiguration-encrypted

This seems to be another +1 for the ElastiCache Replication Group (see #16): https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-elasticache-replicationgroup.html#cfn-elasticache-replicationgroup-atrestencryptionenabled
Let's add S3 and ClickHouse encryption. I'll see if I can get this running and file a PR.

This seems to be another +1 for the ElastiCache Replication Group (see #16): https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-elasticache-replicationgroup.html#cfn-elasticache-replicationgroup-atrestencryptionenabled
Let's add S3 and ClickHouse encryption. I'll see if I can get this running and file a PR.


TODO till launch

[ ] <timer-controls> bundle
[ ] app shell with title and progress bar
[ ] port + event for initial color
[ ] <timer-display> with single color
[ ] switching between screens
[ ] code chunking + prefetching
[ ] controls + events for further colors
[ ] <timer-display> for further colors
[ ] offline support
[ ] PWA goodies
[ ] pause button

@tomekwi Hey, I was at the last React meetup in Wroclaw, I propose adding ESC key handler which will trigger pause action and maybe support swipe gestures on handheld devices. I couldn't find pause button for the first few seconds.
P.S. interesting presentation!

Thanks a lot @hinok, very glad you liked the talk!
Super valid points about the keyboard handlers and UX issues. For us, this has been more of a proof of concept than a polished app – but if you like the app and end up using it regularly, we love PRs! 😄

Closing off this issue for now, because this was a roadmap before the talk rather than something we should actually fix. Most of it is done by now, just short of ✨ offline support.


Rancher-Agent crashes devicemapper with docker daemon

Rancher Version: 1.1.0
Docker Version: 1.10.3
OS and where are the hosts located? (cloud, bare metal, etc): RHEl/7.2/vmware
Setup Details: (single node rancher vs. HA rancher, internal DB vs. external DB) single node with external DB
Environment Type: Cattle
Steps to Reproduce:
Docker environment using devicemapper as storage driver with loopback device.
Docker info:
Server Version: 1.10.3
Storage Driver: devicemapper
Pool Name: docker-253:0-1180431-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop2
Metadata file: /dev/loop3
Data Space Used: 581 MB
Data Space Total: 107.4 GB
Data Space Available: 19.81 GB
Metadata Space Used: 1.434 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.146 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Either use --storage-opt dm.thinpooldev or use --storage-opt dm.no_warn_on_loop_devices=true to suppress this warning.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2015-10-14)
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 3.10.0-327.4.4.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.26 GiB
Results:
Containers went to unresponsive state.
Stopping or deleting container will end container in "dead" state.
Restarting docker daemon failing with below message.
Sep 12 03:58:27 sv1lxdcim04 docker[7434]: time="2016-09-12T03:58:27.172380678-07:00" level=warning msg="devmapper: Usage of loopback devices is strongly discouraged for production use. Please use --storage-opt dm.thinpooldev or use man docker to refer to dm.thinpooldev section."
Sep 12 03:58:27 sv1lxdcim04 docker[7434]: time="2016-09-12T03:58:27.226281019-07:00" level=error msg="[graphdriver] prior storage driver "devicemapper" failed: devmapper: Base Device UUID and Filesystem verification failed.devmapper: Failed to find uuid for device /dev/mapper/docker-253:2-5767170-base:exit status 2"
Sep 12 03:58:27 sv1lxdcim04 docker[7434]: time="2016-09-12T03:58:27.226350604-07:00" level=fatal msg="Error starting daemon: error initializing graphdriver: devmapper: Base Device UUID and Filesystem verification failed.devmapper: Failed to find uuid for device /dev/mapper/docker-253:2-5767170-base:exit status 2"
Sep 12 03:58:27 sv1lxdcim04 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Sep 12 03:58:27 sv1lxdcim04 systemd[1]: Failed to start Docker Application Container Engine.
cadvisor process still showing as zombie process, even after docker daemon is down.
Docker daemon running back only after cleaning devicemapper directory. which clears all the image and container data
Expected:

This issue looks more related to cadvisor, as mentioned in these references.
Ref: https://github.com/docker/docker/issues/9665#issuecomment-137422559
https://github.com/google/cadvisor/issues/771

definitely a known issue... also avoid loopback driver for devicemapper.


Transaction Library for DynamoDB

It´s already developed in java (http://aws.typepad.com/aws/2013/07/dynamodb-transaction-library.html).
Any plans for port this ?

Yes absolutely would love a .NET port for this library.

This is needed if DynamoDB product to be a viable enterprise storage option.

+1 for this feature. Would be very helpful!

Definitely +1 for this.

Added to https://github.com/aws/aws-sdk-net/blob/master/FEATURE_REQUESTS.md

+1.  This is really needed.

+1

+1. I would say that this is pretty much a necessity.

1, really important

Just starting to look at Dynamo and realized this bummer. We have a need to insert to DDB, then insert to SQS, but rollback DDB if the SQS insert doesn't succeed.
Any idea if this will gain any traction?

+1, lack of transaction support is a big hurdle in the project I'm working on.

+1, transaction support in c#  is really needed.

+1 any progress on this?

+1, need transaction support for dynamodb in c#.

+1. Lack of Transaction support in the .NET SDK is a hurdle for us to pick it up for an enterprise project. I'm currently facing issues in a personal project where I'm manually having to delete records from a table based on whether a subsequent write to another table was successful or not.

+1

+1

+1

+1

+1

+1

+1

Very much needed.

+1

+1

+1

+1

+1

+1

+1, hard to believe this has been in the request pipeline for 5 years and nobody has touched it.  This is a massive roadblock to .NET based enterprises using this in anything critical

+1

+1

DynamoDB announced last week for service side transaction support. https://aws.amazon.com/blogs/aws/new-amazon-dynamodb-transactions/
Curious how well this meets the needs of the developers asking for a port of the Java transaction library.

are we able to manipulate transaction boundaries in anyway? Or are the boundaries limited to the call itself?

Is this Transaction Library done for .net core?  I could not find code samples anywhere.

        public async Task VotarAsync(Usuario usuario, Voto voto, EnqueteQueUsuarioVotou enqueteQueUsuarioVotou)
        {
            enqueteQueUsuarioVotou.Id = enqueteQueUsuarioVotou.EnqueteId + "-" + enqueteQueUsuarioVotou.UsuarioHash;
            var enqueteQueUsuarioVotouA = DynamoDbContext.ToDocument(enqueteQueUsuarioVotou).ToAttributeMap();
            var usuarioMap = DynamoDbContext.ToDocument(usuario).ToAttributeMap();
            var votoMap = DynamoDbContext.ToDocument(voto).ToAttributeMap();

            var items = new TransactWriteItemsRequest
            {
                TransactItems = new List<TransactWriteItem>
                {
                    new TransactWriteItem
                    {
                        Put = new Put
                        {
                            TableName = "EL_EnqueteQueUsuarioVotou",
                            ConditionExpression = "attribute_not_exists(Id)",
                            Item = enqueteQueUsuarioVotouA,
                        },
                    },
                    new TransactWriteItem
                    {
                        Put = new Put
                        {
                            TableName = "EL_Usuario",
                            Item = usuarioMap
                        }
                    },
                     new TransactWriteItem
                    {
                        Put = new Put
                        {
                            TableName = "EL_Voto",
                            Item = votoMap
                        }
                    },
                }
            };

            await DynamoDBClient.TransactWriteItemsAsync(items);
        }


Clarify group order 18338

This is a pull request to solve the clarity between functions in issue 18338
Changes made:

added a new test case that checks LeaveOneGroupOut has ordering in the splits
added documentation to clarify the differences between LeaveOneGroupOut and GroupKFold

Thanks for your PR, you probably had not seen it but someone else created a PR tackling the same issue in #22582.
I am going to close this PR, don't hesitate to find another issue to tackle in the github tracker :wink: !

Hmmm I actually realized that this is in your fork and not in scikit-learn/scikit-learn, so you can ignore my message ...


CI: split out the gcc+clang env combo

This way we can use the virtual packages like omp/fc and install all the "clang in ucrt64" stuff separately.
Also move from flang-new to flang as that package now provides that binary too.

clang32 is still special...

clang32 is still special...

how so?  lack of flang for 32-bit?

yeah, I wanted to simplify the install setup, but in the end it looked about the same, so gave up


Some crops are 1px wider/taller than canvas

Some (but not all) of my crops results in a canvas that is 1px wider/taller than the source image.
eg: :

crop is 360 * 466
result is 361 * 467

This could be mitigated somewhat by offering a fill option for crops that extend beyond the image canvas, eg a 'white' fill would remove the black line from the right hand edge of my cropped jpeg.

I'm not sure I understand.
What's the size of the image before cropping?
Can you give an example image and the code you execute on it?

Thanks for your reply -
The first screenshot shows:

at top left: the source image with a pink box around the banner crop
at bottom: the source image inspected for native dimensions (1170x4142)
at right: the LWIP crop from the source image with a 1px line on the right, where I assume the crop has been miscalculated and extended off the edge of the canvas

The second screenshot shows:

at top left: the source image with a pink box around the banner crop
at right: the LWIP crop from the source image with a 1px line on the right, where I assume the crop has been miscalculated and extended off the edge of the canvas
at bottom: the LWIP crop inspected for native dimensions (1171x296) and the filename of the generated image, showing that the expected size is 1px smaller on both axes (1170x295), and the crop is at x:0px, y:50px

Here's the function which runs the cropping and the versions are:

LWIP: "lwip": "0.0.6"
Node: v0.12.2
NPM: 2.7.4
OS: OS X 10.9.5 (Macbook Pro)

Thanks,
Dan

@dotherightthing
Note that crop coordinates are inclusive.
I.e. img.crop(0,0,1,1) will yield an image of 2X2 pixels, starting from pixel (0,0) up to and including pixel (1,1).

Ah, good stuff, thanks @EyalAr - I've adjusted the width and height of my crops by 1 pixel to allow for the 'zero' pixel.


Add magic comment to stop our CI from building a package

@jsingle

Haha is this a normal way to do this?


Select random rows (efficiently) with UUID

I've spoken with Ramsey and got his blessing first before opening this issue (because it's not really an 'issue').
I'm finding it necessary to return multiple random rows while using UUID. Most of the efficient means of doing so require using an auto incrementing primary key. While I can select one record randomly by using an offset clause, I feel like there has to be a better way of selecting a random UUID and then getting multiple random rows at the same time. This will stop the issue of having to COUNT the rows for an offset as well as only having to hit the database once.
Any ideas? I'm stuck using the offset clause for now. I'm currently using MySQL.

@jckeen Just so others don't post this, I'm assuming that your dataset is too large for SELECT * FROM table_name ORDER BY RAND() LIMIT 10; to be performant? Or pulling your dataset into your program and doing the randomization there(array_rand(), etc)?

@moderndeveloperllc Yes, thanks for clarifying that, I should have!
It has the capability of becoming too large for those two solutions. I could cache the dataset (and use array_rand() ) once a day and pull from there, but there's still that feeling of a better way, especially with large amounts of data.

Is there a reason you can't have an auto incrementing key that you use simply for random ordering, and not as the primary key? That way you could use the existing efficient solutions?

I think @jmauerhan's recommendation is probably the best solution for this.

Yes, you could make a column with an auto incrementing key as well.
Messed around a bit. I converted the first six characters of the UUID to a HEX, then used that as the basis of my selection.  It seems to be pretty performant and random, going to do some more testing.
Any thoughts? Credit to http://mysql.rjweb.org/doc.php/random for his post that helped me out.
You could also use an application cache in order to utitilize the benefits of the SET.
SET @max := (SELECT MIN(HEX(LEFT(id, 6))) FROM random_test);
SET @min := (SELECT MAX(HEX(LEFT(id, 6))) FROM random_test);

SELECT id
FROM(
      SELECT id
      FROM (SELECT FLOOR(RAND() *  (@max - @min) + @min) AS start FROM random_test) AS init
      JOIN random_test y
      WHERE HEX(LEFT(y.id, 6)) > init.start OR HEX(LEFT(y.id, 6)) < init.start
      ORDER BY y.id
      LIMIT 50
) z
ORDER BY RAND()
LIMIT 2

@jckeen Depending on the type of UUID you're using, I would not recommend using the first x characters of a UUID to determine identity. In a version 1 UUID, for example, these characters are going to be very similar and even cause collisions. In a version 4 UUID, it might be okay to do this (though the chance for collision goes way up, since you're limiting the amount of randomness you have), and your application will have a dependency on always using version 4 UUIDs, so you'll need to validate the UUID version before inserting into the database.


should not normlize the distance ?

You could, I didn't need it, but some people reported it helped for their task.
You can try.

how to normlize the distance?


Include Equipped and Include Unequipped should be combined as a button group

Currently you can unselect both in the artifact filter, which displays nothing and doesn't really make sense. We should make them connected as a multi select filter

Trying my hardest to get this done 🫡

superseded by https://github.com/frzyc/genshin-optimizer/issues/1569


TFS GIT Pull Request - shortcuts

I don't see the shortcut for adding a comment to a file. There's an icon for doing so, and I know there's a shortcut because I've used it before, but I can't remember it. It seems this page is the place where it should be listed. Please add it.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 6c2fe0af-744d-e535-b300-8d1060945db6
Version Independent ID: 10892c8d-ff0b-3072-46bd-cad31923c240
Content: Web portal Team Explorer keyboard shortcuts - Azure DevOps
Content Source: docs/project/navigation/keyboard-shortcuts.md
Product: devops
Technology: devops-collab
GitHub Login: @KathrynEE
Microsoft Alias: kaelli

@CJBS  Thank you for the feedback, assigning this to the author for review.
For now use the right-click context menu to show additional shortcuts, to add a comment is F4.


[build] Pass target argument to the linker if needed

Following #31125 and #31612, -target is not added automatically to
linker flags when that's needed (e.g. when building for Apple SDKs) --
mimic the logic used to add it for compiler flags.
Addresses rdar://63138761

@swift-ci please test

@drexin  there’s the thread on the variable/predicate that I’d like to see resolved first.

@swift-ci please test

@compnerd Exactly, that's why I was saying I'd like to wait for you to respond :)

Build failed
Swift Test Linux Platform
Git Sha - 8b3464d0b793c4cb0dd70e413aa2b5ce252f89c2

@swift-ci please test Windows

Build failed
Swift Test OS X Platform
Git Sha - 8b3464d0b793c4cb0dd70e413aa2b5ce252f89c2


registration-form: add validation for agreed, add values in template …

…for agreed and gender components
##GithubTaskNumber
Description what actually was done in task:

@MSt1ch thank you for contribution!


Error in VC file

Just noticed this, the third chat in the Sonic chat table
{{{
pickupLifepodSonicChat = SonicChat([ SonicChatVoice(voGoGoGoSound, "Pickup
lifepod"                ) ]);
}}}
This VC isn't used at the moment (if it were it would send a command to pickup
nearest friendly lifepod {I think}), but fact is that it points to the wrong
sound effect (voGoGoGoSound).
Imported from trac ticket #6,  created by juckto on 01-05-2010 at 09:21, last modified: 07-06-2010 at 06:45

Don't have the correct vo wav to replace it with ("Pickup lifeod") ..used "Need rescue help" r671
Trac comment by Imago on 07-06-2010 at 06:45


Calculate check character on diga code before validating against the API

Problem
In the documentation at gkv and bitmarck's FAQ there is some mention of the manufacturer having to check the code is valid before sending it to the api (?). They are likely referring to the check character (last character) of the code. This does not seem urgent to implement as the API will respond if this is incorrect anyway, but it would be a nice to have feature.
Solution
In DigaCodeDefaultParser class there is a method for validating the code
It seems bitmarck is referring to https://base32check.org/ for doing this. Try to implement that (with the check character specification at gkv) and add it to the code validation method.

bitmarck has a repo that does this I think at https://github.com/bitmarck-service/base32check-java


Seeing multiple series after zoom in on x-axis.

When a user zooms in on the x-axis multiple times on a single series, after a certain point the chart displays multiple series. The number of series displayed increases with each subsequent zoom-in.
I'm seeing this behavior in Google Chrome (64.0.3282.186), Windows 10.  I'm seeing it in our application and on the Predix px-vis-timeseries demo page. (see screenshots below.)

Thanks for reporting this issue, we released a new version of the vis framework with a fix for it: 4.7.9


Null JSON to Struct should return an error

Description:
function foo() {
    json j = null;
    var p, e = <Person> j;
}

In the above program, both p and e is null. But since j is a null JSON, it should return an error, and e should be not null.
Affected Versions:
0.962.0

This behaves now as, when the above is run, it will throw a NullReferenceError, since j is null, and it cannot be converted to Person.