eng_devcrypto: add configuration options, misc fixes

This adds some features to the engine:

performing a CIOCCPHASH operation during initialization to decide if a digest is usable or not. The import/export functions used by digest_copy were available since linux 2.6.29, but it was not mandatory for drivers to implement them until linux 4.17.

use CIOCGSESSINFO to query if an algorithm is accelerated or not, disabling those that are software-only.

configuration options to select the list of ciphers digests to enable, and to control the use of unaccelerated drivers

command to dump /dev/crypto driver information and capabilities to stderr

use EVP_MD_CTX_FLAG_ONESHOT flag to save an ioctl call.

There were a couple of fixups as well:

avoid leaking the EVP methods if the engine does not finish initialization

closing the digest session in digest_cleanup instead of in digest_final

Checklist

[x] documentation is added or updated - new options have their usage explained in their description.

Cool!
However, you need to submit a CLA (at least individual), please see here

I have just done that about two minutes ago.

Closing and re-opening to kick the CLA checker

@levitte
Please take a look at this commit https://github.com/cotequeiroz/openssl/commit/651c578b8014b8f72ba11b91f978ad7e34480bbb, in which I add a Configure option to build this engine to be dynamically loaded.
This way you can package the /dev/crypto support separately, and add the /dev/crypto kmod as a dependency as well.
Tell me if you think I should add it here.  I have commits in that branch that add notes to INSTALL and CHANGES, and they would come along as well.

BTW, for positive reinforcement, I do like what I see.  It's just details that need being hammered at.

BTW, for positive reinforcement, I do like what I see. It's just details that need being hammered at.

I appreciate that.

@levitte
I hope I haven't broken anything for moving too fast, but I think I've made all of the changes you requested.

I've added a cipher copy function in the last commit, to make sure the new cryptodev gets initialized.  I haven't been able to see it in action, though.  Do you know under what circumstances it gets called?  Anyway, the initializing it should be the correct behavior anyway.

I've added a cipher copy function in the last commit, to make sure the new cryptodev gets initialized. I haven't been able to see it in action, though. Do you know under what circumstances it gets called? Anyway, the initializing it should be the correct behavior anyway.

Saw it, and while I had never thought of it myself, but it was a good idea.  It's called whenever EVP_CIPHER_CTX_copy is called, which can potentially happen any time, depending on the user.

FWIW, while checking for a real-world case of EVP_CIPHER_CTX_copy usage, I realized that sources in test directory called it, so I decided to run make test again.  I used to try to run it before, but it never passed with any /dev/crypto engine build, so I wasn't even trying anymore.  Now it works:
All tests successful.
Files=152, Tests=1360, 292 wallclock secs ( 1.59 usr  0.15 sys + 155.28 cusr 10.71 csys = 167.73 CPU)
Result: PASS

Never mind, it is still failing.  I had /dev/crypto driver removed.

Hmm, what?  Ok, so more to do?

I haven't really studied why the tests fail.  They have failed since 1.0.2, really.  Let me try to dig into it.

That's very specific...  interesting.  BTW, you do know that you can run individual tests as well as get more verbose test output, right?  For example:
make test TESTS='test_servername test_sslbuffers` V=1

I didn't know that.  I'll try to get some more information.  Tests have changed between when this PR was open and now. I'll rebase first, then try again.

Here's what I'm going to do first:
Run the tests without this PR, compare the results after the PR, and deal with failures introduced here.
Then, I'll see what I can do with the rest of the failures.

Hmmm...  I'm lost regarding configuration and testing, can you clue me in?  I only have software drivers, so would need the USE_SOFTDRIVERS control, but whatever I write in the config file I try to use (and point at with $OPENSSL_CONFIG), nothing happens, it looks like devcrypto isn't even used (I tried adding a bit of tracing).

I'm done with my first assignment: I took care of failures introduced by this PR.
They were related to poor checking in digest_close.  I've added some checks that I saw in crypto/sha/sha512.c as well in a separate commit.
As for testing, in my computer I don't have hw as well, so what I do is copy crypto/cryptodev.h to ./include/crypto/cryptodev.h, comment out the #define CIOCGSESSINFO line, and the defaults will let all algos in.

As for testing, in my computer I don't have hw as well, so what I do is copy crypto/cryptodev.h to ./include/crypto/cryptodev.h, comment out the #define CIOCGSESSINFO line, and the defaults will let all algos in.

Ah, thank you, that was simple!  I just tried with inserting #undef CIOCGSESSIONINFO in eng_devcrypto.c, that works just as well.

just tried with inserting #undef CIOCGSESSIONINFO in eng_devcrypto.c, that works just as well.

I did this, but then I decided to use git rebase -i to run a test at every commit. :stuck_out_tongue_winking_eye:
If there's nothing else I've done wrong, then I think we're back at the point you were saying

This is excellent work

:smiley:
Here are the test results with this PR applied:
All tests successful.
Files=152, Tests=1361, 597 wallclock secs ( 1.59 usr  0.14 sys + 156.90 cusr 345.26 csys = 503.89 CPU)
Result: PASS

And for comparison, current master:
All tests successful.
Files=152, Tests=1361, 229 wallclock secs ( 1.67 usr  0.11 sys + 158.83 cusr  8.21 csys = 168.82 CPU)
Result: PASS

Notice the difference in time, and csys in particular.  That tells me the engine was doing its part using the kernel crypto algos.  Not really efficient using sofware-only, is it?
The tests_ssl_new was the slowest, here's a diff of time -v make test TESTS=test_ssl_new V=1:
        Command being timed: "make test TESTS=test_ssl_new V=1                  Command being timed: "make test TESTS=test_ssl_new V=1
        User time (seconds): 9.85                                 |             User time (seconds): 10.95
        System time (seconds): 0.41                               |             System time (seconds): 260.46
        Percent of CPU this job got: 100%                         |             Percent of CPU this job got: 97%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:10.25      |             Elapsed (wall clock) time (h:mm:ss or m:ss): 4:37.11
        Average shared text size (kbytes): 0                                    Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0                                  Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0                                          Average stack size (kbytes): 0
        Average total size (kbytes): 0                                          Average total size (kbytes): 0
        Maximum resident set size (kbytes): 18508                 |             Maximum resident set size (kbytes): 18156
        Average resident set size (kbytes): 0                                   Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0                                    Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 155425            |             Minor (reclaiming a frame) page faults: 155163
        Voluntary context switches: 2293                          |             Voluntary context switches: 2377
        Involuntary context switches: 160                         |             Involuntary context switches: 11615
        Swaps: 0                                                                Swaps: 0
        File system inputs: 0                                     |             File system inputs: 1960
        File system outputs: 1784                                               File system outputs: 1784
        Socket messages sent: 0                                                 Socket messages sent: 0
        Socket messages received: 0                                             Socket messages received: 0
        Signals delivered: 0                                                    Signals delivered: 0
        Page size (bytes): 4096                                                 Page size (bytes): 4096
        Exit status: 0                                                          Exit status: 0

I know you were not too fond of it when I first proposed it, but can I add a commit adding an option to build the devcrypto engine dynamic?  It simplifies distribution, as you can have the same basic openssl library built for everyone, and then have an add-on package to add the engine for those who want it.

I know you were not too fond of it when I first proposed it, but can I add a commit adding an option to build the devcrypto engine dynamic? It simplifies distribution, as you can have the same basic openssl library built for everyone, and then have an add-on package to add the engine for those who want it.

If you must, I would prefer if that was a separate PR.  Do you mind?

I'll do that after/if this is merged.  Thanks.

One last thing, the bunch of commits could use a cleanup.  Do you want to do it, or do you trust us to do a good editorial job when we merge?

Here's my attempt:  I've added a commit reverting to master, then added the 10 commits that I think complete the cleanup.  Give me your OK, and I'll leave just these 10.

I think @cotequeiroz understood exactly, but he's showing it in a way that doesn't undo the previous commits. An unusual tactic, but I get it.

I found it easier to pull your branch on my machine and look at the new commits with gitk.  Github does mess things up sometimes...
Anyway, the proposed set of commits look good to me.

I think @cotequeiroz understood exactly, but he's showing it in a way that doesn't undo the previous commits. An unusual tactic, but I get it.
It kept the "editorial job" option open.  I'll clean it up  for real now.

2nd reviewer, please

@openssl, it would be nice if this could actually make it into 1.1.1a.  Please?

@openssl, it would be nice if this could actually make it into 1.1.1a. Please?

@cotequeiroz I'm sorry that I don't have the time currently to dig into your patch deep enough for a formal review, but I'd like to say that your pull request looks very well written and documented. I totally agree with @levitte's opinion:

This is excellent work

Thanks!

I wanted to properly review this and run some tests in the debugger to feel comfortable in approving this before tomorrow release, but I realized I don't have a target running BSD readily available for now :(

Please consider adding the fix commits (cipher CTX copy, digest failure, copy of uninitialized digests, and harmless INSTALL notes) in 1.1.1a as the current engine will not work without them.

Unfortunately, timing isn't on our side...  it requires a 2nd reviewer either way.
Also, if only a few of the commits are to be considered for the 1.1.1 branch, it is best expressed as a separate PR.

@levitte
Now that this officially did not make into 1.1.1a, can I make one change:  only offer USE_SOFTDRIVERS if CIOCGSESSINFO is defined.  BSD does not have it, handling the use of softdrivers at the kernel level (through a sysctl call), accomplishing the same goal. Without CIOCGSESSINFO, USE_SOFTDRIVERS won't be so useful anyway: you'd either allow all algorithms (setting it to 1 or 2) or none (0).

Oh, please feel free.

I have changed the rebuild_*_list functions to call ENGINE_unregister_ciphers/digests and then EGINE_register_cipher/digests.  Otherwise the initial lists (all enabled) would still be used.

@levitte
I've noticed that the AES-CTR cipher behaves as a a block cipher with the engine:  update will return no data unless at least at least a full block of input is given, and the final encryption gets padded.  Without the engine, it behaves like a stream cipher, returning data on partial updates, and no padding in the final.
The part about not producing output right away might not be a problem, but I can't ignore the fact that the engine cipher is not able to decrypt unpadded ciphertext.  The ctr tests do not catch this, as the output of enc is fed to be decrypted, and that works.
Is there an easier way to deal with this other than setting EVP_CIPH_FLAG_CUSTOM_CIPHER and doing everything by hand (i.e. duplicating code from evp_lib.c)?

I think I've got it.  Apparently, it was not that difficult after all.

Hi, sorry for the delay, last week was crazy amounts of meetings...  I'm finally catching up on stuff I did before that

I played a lot with this engine stuff meanwhile, mostly trying to make GCM work.  I took a look at #5259 to see what it was doing.  I even wrote an AFALG engine, literally, on top of this one (https://github.com/cotequeiroz/openssl/blob/eng_afalg_test/crypto/engine/eng_devcrypto.c) to see if I could make GCM work.  I got it to mirror 100% this /dev/crypto engine, and it turned out to be much faster than the current AFALG engine.  If you interested, I can open a new PR for it, but it does not use AIO like the current AFALG engine does--probably the reason why it is much faster.
My conclusion is that GCM won't work, unless we do_cipher in one shot.  There's no way to keep sending data and get an updated tag. To mimic what openssl does, it would require state copy, like we did for digests.  However unlike digests, the linux kernel does not have that functionality for GCM.

For the AFALG engine, if you want to play, be my guest.  You might want to talk with @mattcaswell about it, as he's the original author and therefore knows that engine a lot better than I.
Re GCM, let's take that another time, if at all...  and wait for Linux to support partial operations with that mode.  Either way, I'm thinking this PR is already full enough.  Partitioning the work, ya know? :wink:

I have no intention to add GCM (or any new funcionality) to this PR.  It has grown too much, turning it difficult to review.
My conclusion is that the only way to accelerate GCM today would be to break it up and accelerate only AES-CTR part of it.  But that would either duplicate a lot of code, or change some of the internal handling of GCM to allow the use of the engine for the ctr part.  I haven't studied it enough to really discuss this properly, though.
Now the AFALG stuff was really interesting, and I shall proceed after this is done.  I'll take on your review as soon as I find some time.
Cheers

Ok.  The reason I ask is that we who to the merge need to know what to cherry pick back to 1.1.1

(What would normally happen is making two separate PRs, one with the bug fixes, which would be cherry-picked back to 1.1.1 as well, and one with the added features, which would be for the master branch only)

(What would normally happen is making two separate PRs, one with the bug fixes, which would be cherry-picked back to 1.1.1 as well, and one with the added features, which would be for the master branch only)

That's precisely what @cotequeiroz should do IMO. Because only that way the ci's guarantee that the splitting worked.

I could just open a new PR with the fixes for 1.1.1 now, and keep them in sync.  Let me just take care of levitte's review and then I'll do it.

I could just open a new PR with the fixes for 1.1.1 now, and keep them in sync. Let me just take care of levitte's review and then I'll do it.

No problem. You also wait until this one has been approved before creating the other pull request.

@mspncp has the right idea, of course :-)
I'll run a test, but I think this is good to go.

I think I'll split the algo checking commit in two.  I think checking for digest copy usability is crucial to getting this to work without being able to choose the algorithms.  Otherwise, if you have one digest that doesn't implement copy, you won't be able to use the engine at all.
The part about using cryptodev to tell if the driver being implemented in software or not is basically a guess based on the name of the driver.  Unlike BSD, linux kernel has no knowledge if the driver is run by hardware or software (I looked at how cryptodev could tell software from hardware, and AFALG couldn't, and found out it was just  doing strstr with the driver name, and known accelerator strings 😞).  We can't use that information without giving the user means to override it.  So hash-copy check goes with bugfixes, hardware/software check goes with new features.

Done.

So did I get it right, that you'll make a separate PR for 1.1.1?

@openssl, another reviewer needed here.

So did I get it right, that you'll make a separate PR for 1.1.1?

Yes, I am.  Should I do it now, or wait for the second review?
BTW, here are the commits I will use in the second PR:
b0ac041a1d INSTALL: add note about devcrypto engine
8f3855aa1a eng_devcrypto: don't leak methods tables
ba7e6aae1d eng_devcrypto: expand digest failure cases
97338e6095 eng_devcrypto: fix copy of unitilialized digest
9f00061d2c eng_devcrypto: close session on cleanup, not final
ccec8cbf5b eng_devcrypto: add cipher CTX copy function
eeb964a6f2 eng_devcrypto: fix ctr mode
dbf369aa30 eng_devcrypto: make sure digest can do copy

These are all bugfixes, as the engine will not work properly without them.

Yes, you did. Should I do it now, or wait for the second review?

Doesn't really matter to me.

BTW, here are the commits I will use in the second PR:

If that's really just it, I can very well try and cherry pick those.  They will be approved commits, so it seems silly to have to re-approve the exact same code.  Seen like this, it might be wise to wait with a 1.1.1 PR, after all...

If that's really just it, I can very well try and cherry pick those. They will be approved commits, so it seems silly to have to re-approve the exact same code. Seen like this, it might be wise to wait with a 1.1.1 PR, after all...

That's ok for me. BTW: Since you very like have the CI's configured for your personal repo clone, it would be enough to just push the cherry-picked branch to your clone and wait for the CI's to complete (that's how I do it).

As I said previously: form, style, documention, etc. excellent (comme il faut), but I don't have the time and the environment to do some testing or experimenting.

I don't know if you're aware of it or not, but if you want to run-test this, all you need is the kernel crypto algorithms enabled (they're usually enabled for vpn) and cryptodev-linux master (if you use earlier releases, you won't get digest support), which is pretty straightforward.  You don't need to install it, just make sure <crypto/cryptodev.h> can be found by the compiler (-Icryptodev's source dir suffices).  /dev/crypto should pop up (with udev or similar) automatically afeter cryptodev.ko is inserted.  The engine will run with software drivers.  It should not affect testing--once the engine does the ioctl, it's up to cryptodev & the kernel drivers to get it right.
I've tested this pretty extensively with Linux.  I can't test with FreeBSD, the other platform I am aware of where /dev/crypto is available.  @romen, you mentioned earlier about firing up a BSD target.  Can you do this?

Thanks.  I did make the STR change, BTW, without explicitly including opensslv.h, as it is picked by other headers already.

I did make the STR change, BTW, without explicitly including opensslv.h, as it is picked by other headers already.

Okie.

I'm ready to merge this, and will do the cherry pick of the mentioned commits to 1.1.1.

So if @levitte wants to finally merge, he can use my "read only" approval from what I've seen, in order not to delay this pull request unduly.

Much appreciated, @mspncp.  I often do the same myself

Merged to master:
2bafe6cfeda5dc27fbc4c17cd351063872e0c0fd eng_devcrypto: add command to dump driver info
166261a5e9e141cc3850012cbb9106bcd3226393 eng_devcrypto: add configuration options
b2db94d1d0bb79a80e188aaf459698ae5cbabe7d eng_devcrypto: save ioctl if EVP_MD_..FLAG_ONESHOT
16e252a01b754a13e83d5e5e87afbe389997926b eng_devcrypto: make sure digest can do copy
b5015e834aa7d3f0a5d7585a8fae05cecbdbb848 eng_devcrypto: fix ctr mode
6d99e238397859f2df58c60e28905193b2dd6762 eng_devcrypto: add cipher CTX copy function
a67203a19d379a8cc8b369587c60c46eb4e19014 eng_devcrypto: close session on cleanup, not final
ae8183690fa53b978d4647563f5a521c4cafe94c eng_devcrypto: fix copy of unitilialized digest
4d9f99654441e36fdcb49540a1dbc9d4c70ccb68 eng_devcrypto: expand digest failure cases
d9d4dff5c640990d45af115353fc9f88a497a56c eng_devcrypto: don't leak methods tables
eb3fe0ee5411449230cff46b7f13ebef37aa950e INSTALL: add note about devcrypto engine

Merged of fixups to 1.1.1:
488521d77fdc1de5ae256ce0d9203e35ebc92993 eng_devcrypto: make sure digest can do copy
2887a5c8f9a385b3ebee12b98f68e7d1f9cc0ea0 eng_devcrypto: fix ctr mode
a19d1a1d370e2959555fccbafc4e970634840352 eng_devcrypto: add cipher CTX copy function
5378c582c8d3f1130b17abb2950bfd09cde099c6 eng_devcrypto: close session on cleanup, not final
68b02a8ab798b7e916c8141a36ab69d7493fc707 eng_devcrypto: fix copy of unitilialized digest
add2ab1f289c24a1563c5b895d5cd133fe874f12 eng_devcrypto: expand digest failure cases
be5cf61caa425070ec4f3e925d4e9aa484c8315b eng_devcrypto: don't leak methods tables
72b8228a9c0711bdc1f5f2a9f49bef0bb50ffb58 INSTALL: add note about devcrypto engine

... and that concludes this PR.  Great job, and I thoroughly enjoyed the interaction.


Rewrite statusbar when fugitive is loaded

Hello,
First thank you for your job :+1:
I use NeoBundle with NeoBundleLazy for loaded Fugitive Plugin dynamically :
NeoBundle 'vim-airline/vim-airline', {'depends' : 'vim-airline/vim-airline-themes'}
NeoBundleLazy 'vim-airline/vim-airline-themes'
NeoBundleLazy 'tpope/vim-fugitive', {'external_commands' : ['git'], 'disabled' : (!executable('git')), 'autoload' : {'comm    ands' : ['Gstatus', 'Gcommit', 'Gwrite', 'Gdiff', 'Gblame', 'Git', 'Ggrep']}, 'depends' : 'mhinz/vim-signify'}
...
set lazyredraw
autocmd VimEnter * redrawstatus!

There is a way to rewrite status bar for show git branch when plugin Fugitive is loaded ? (Also I use lazyredraw and I need the autocmd below)
Thank you.

no.

Ok thank you for your reply.


Saucelabs Rate Limit

This is a tracking issue where I want to collect everything about the stability of Saucelabs.
There are two different rate limits:

API Rate Limit (currently often causing exceptions; always has been throwing sometimes)
Browser Rate Limit (rarely causing exceptions; already increased to 30 based on feedback)

The API rate limit seems to be described as the following. Within 3600 seconds (1h), we can make with angular-ci license about 3500 requests. At first glance this looks okay because I don't think that we'll run more than 3500 Saucelab jobs within an hour.
Note that the angular-ci license is being used by Angular, AngularJS and Angular Material (previously AngularJS Material and also flex-layout)

Either sauceconnect triggers multiple requests to the Saucelabs REST API, or something spams the API endpoints with our license, or we actually run more than 3500 CI jobs within an hour..

For reference: angular/angular also seems to be affected by flakiness due to the rate limit. The stdout/stderr is just piped to /dev/null for core.
See: https://travis-ci.org/angular/angular/jobs/466233160

Some log output for browser disconnects:
12-17 06:55:04.930  1584  1595 I MediaFocusControl:  AudioFocus  abandonAudioFocus() from android.media.AudioManager@4926910com.android.music.MediaPlaybackService$3@b9cc09
12-17 06:55:04.931  1584  1727 I ActivityManager: Killing 2145:com.android.music/u0a36 (adj 15): empty #17
12-17 06:55:04.931  1584  1727 I am_kill : [0,2145,com.android.music,15,empty #17]
12-17 06:55:04.998  1584  2223 I am_proc_died: [0,2145,com.android.music]
12-17 06:55:05.022  1584  1597 I am_pss  : [2780,10007,com.android.externalstorage,3193856,1789952]
12-17 06:55:05.028  1584  1597 I am_pss  : [2761,10027,com.android.documentsui,4090880,2637824]
12-17 06:55:05.690  2801  2801 I chromium: [INFO:CONSOLE(2)] "WebSocket connection to 'ws://localhost:9876/socket.io/?EIO=3&transport=websocket&sid=Q-UGw0Ul7SlYA5vRAAAO' failed: Connection closed before receiving a handshake response", source: http://localhost:9876/socket.io/socket.io.js (2)
12-17 06:55:06.021  2801  2801 I chromium: [INFO:CONSOLE(248)] "Attempting to configure '__esModule' with descriptor '{"value":true}' on object 'undefined' and got error, giving up: TypeError: Object.defineProperty called on non-object", source: http://localhost:9876/context.js (248)
12-17 06:55:06.169  2801  2801 I chromium: [INFO:CONSOLE(248)] "Running tests with document renderer...", source: http://localhost:9876/context.js (248)
12-17 06:55:08.271  1584  1597 I am_pss  : [1963,10008,com.android.launcher3,39433216,33521664]
12-17 06:55:33.130  1584  1584 I free_storage_changed: 686141440
12-17 06:56:00.085  1232  1312 D hwcomposer: hw_composer sent 64 syncs in 60s
12-17 06:56:00.088  1584  1597 I am_pss  : [1584,1000,system,71268352,61444096]

Update: The Saucelabs support is looking into it. Apparently we acquire a lot of tunnels, but those shouldn't actually reach the rate limit of 3500.


((B O X _ T V))How to watch England vs Australia: TV channel and live

Watch England vs Australia live streams today, as Joseph-Aukuso Sua'ali'i makes his professional rugby union debut. Below we have all the information on how to watch England vs Australia from anywhere with details on worldwide TV channels, broadcasters, live streams and start times.

cvbvb

saa


Investigate failed check in memcmp

This was discovered in #1332, but also occurs in a synchronous context:
#[kani::proof]
pub fn main() {
    let coll: Result<Vec<u32>, ()> = std::iter::empty().collect();
    assert_eq!(Ok(vec![]), coll);
}

Running this with Kani gives the result:
SUMMARY:
 ** 2 of 436 failed
Failed Checks: memcmp region 1 readable
 File: "<builtin-library-memcmp>", line 19, in memcmp
Failed Checks: memcpy region 2 readable
 File: "<builtin-library-memcmp>", line 21, in memcmp

VERIFICATION:- FAILED

Reduced to
#[kani::proof]
pub fn main() {
    let coll: Vec<u32> = std::iter::empty().collect();
    assert_eq!(Vec::<u32>::new(), coll);
}

Reduced further:
#[kani::proof]
pub fn main() {
    assert_eq!(Vec::<u32>::new(), Vec::<u32>::new());
}

The reason seems to be that Vec::new sets its pointer to dangling: https://github.com/rust-lang/rust/blob/aeb5067967ef58e4a324b19dd0dba2f385d5959f/library/alloc/src/raw_vec.rs#L123
Furthermore, in the PartialEq implementation, it calls memcmp: https://github.com/rust-lang/rust/blob/aeb5067967ef58e4a324b19dd0dba2f385d5959f/library/core/src/slice/cmp.rs#L91
But I'm pretty sure that memcmp requires valid pointers, even with size 0.

Nice work reducing the test! We've been bitten by the dangling pointer issue before, which triggered us to turn off some of CBMC's pointer checks (see https://github.com/model-checking/kani/issues/763). Also see this Zulip thread that @celinval created.
The failure does not occur if the vector is created with capacity (to eliminate the dangling pointer):
pub fn main() {
    assert_eq!(Vec::<u32>::with_capacity(1), Vec::<u32>::with_capacity(1));
}

Not sure if there's a way to disable the failing CBMC check. Another possibility is to provide our own override for memcmp.

I've asked on Zulip again, and apparently it is fine to call memcmp on the dangling pointer. I will file a bug with CBMC.

Specifically, Miri permits calling memcp on a pointer without provenance, such as NonNull::dangling.
However Miri will still flag an error on use-after-free with memcp, i.e. when the pointer has provenance and the allocation it points to no longer exists.

@RalfJung I'm not totally sure about this. From the C Standard:
" If an argument to a function has an invalid value (such as a value outside the domain of the function, or a pointer outside the address space of the program, or a null pointer, or a pointer to non-modifiable storage when the corresponding parameter is not const-qualified) or a type (after promotion) not expected by a function with variable number of arguments, the behavior is undefined. "
http://port70.net/~nsz/c/c11/n1570.html#7.1.4
It's not just NULL. I guess it depends if you consider the dangling pointer outside the memory space of the program.

Not sure if that changed in later versions of the standard, but https://en.cppreference.com/w/c/string/byte/memcmp lists the requirements and says

The behavior is undefined if access occurs beyond the end of either object pointed to by lhs and rhs. The behavior is undefined if either lhs or rhs is a null pointer.

Neither of these are the case on a call like memcmp(1, 1, 0).

If only there was an unambiguous spec that would actually precisely spell out all requirements... in lieu of that, we can also argue that

we are calling memcmp via FFI, so really the rules applying are those of the underlying target architecture
there is no evidence and not even a plausible suggestion for how memcmp(1, 1, 0) being UB could be exploited by compilers
this matches our treatment of size-0 accesses in general, and LLVM devs say this will work with their backend (so even cross-lang LTO is covered)

While we argue about the true semantics, here's a plan for mitigation discussed with @danielsn: add a hook for memcmp and check if the size is 0. If so, return true, otherwise actually call memcmp.

A slightly related CBMC issue: https://github.com/diffblue/cbmc/issues/7064.

FWIW, in Miri, memcmp can still be UB when the size is 0. For example:
extern "C" {
    fn memcmp(lhs: *const u8, rhs: *const u8, count: usize);
}

fn main() { unsafe {
    // null ptr is UB
    memcmp(0 as *const u8, 0 as *const u8, 0) ;
} }

and more interestingly:
fn make_uaf() -> *const u8 {
    let b = Box::new(0u8);
    &*b as *const _
}

fn main() { unsafe {
    // use-after-free is UB
    let ptr = make_uaf();
    memcmp(make_uaf(), make_uaf(), 0) ;
} }

The first is probably trivial to detect for Kani, but I do not know if it models provenance accurately enough for the second.

$ cat uaf.rs
extern "C" {
    fn memcmp(lhs: *const u8, rhs: *const u8, count: usize);
}

fn make_uaf() -> *const u8 {
    let b = Box::new(0u8);
    &*b as *const _
}

#[kani::proof]
fn main() { unsafe {
    // use-after-free is UB
    let ptr = make_uaf();
    memcmp(make_uaf(), make_uaf(), 0) ;
} }

$ kani uaf.rs
<snip>
Check 3: memcmp.precondition.1
         - Status: FAILURE
         - Description: "memcmp region 1 readable"
         - Location: <builtin-library-memcmp>:19 in function memcmp

Check 4: memcmp.precondition.2
         - Status: FAILURE
         - Description: "memcpy region 2 readable"
         - Location: <builtin-library-memcmp>:21 in function memcmp
<snip>

Sure, but currently it also rejects code that it potentially ought to accept, see this issue. The interesting part will be to accept that but still reject the UAF example.

It looks like if I declare the dangling pointer as a 0 byte allocated region using the mmio flag to CBMC it works as expected:
// cbmc mmio.c --mmio-regions 16:4,32:4 --pointer-check
// cbmc mmio.c --mmio-regions 16:0,32:0 --pointer-check
#include <assert.h>
#include <string.h>
int main()
{
  int *p=0x10;
  int *q=0x20;

  /* *p=42; */
  /* *q=314; */
  /* assert(*p==42); */
  /* assert(*q==314); */
  assert(memcmp(p,p,0) == 0);
}

@tautschnig may have thoughts on the semantics of C pointers

Similarly
[2](http://port70.net/~nsz/c/c11/n1570.html#7.24.1p2) Where an argument declared as size_t n specifies the length of the array for a function, n can have the value zero on a call to that function. Unless explicitly stated otherwise in the description of a particular function in this subclause, pointer arguments on such a call shall still have valid values, as described in [7.1.4](http://port70.net/~nsz/c/c11/n1570.html#7.1.4). On such a call, a function that locates a character finds no occurrence, a function that compares two character sequences returns zero, and a function that copies characters copies zero characters.

The pointer is completely valid for memory accesses of size 0. So I view that condition as being satisfied.
Pragmatically speaking, IMO it would be a failure of the memcmp specification if a user like Rust's Vec has to add a branch to special-case size 0 here.

It looks like if I declare the dangling pointer as a 0 byte allocated region using the mmio flag to CBMC it works as expected:

Essentially, the Rust stance is that such a 0-byte allocated region exists at every address (except 0). The LLVM devs agree with that interpretation.

It seems we disagree on what allocation means. I'd expect allocation to either refer to something generated by the allocator (e.g. malloc or equivalent), or to a static.  If every location has a 0 sized allocation, what is the provenance of the pointer? How was it allocated?
Question: why not make a shared static as the dangling pointer? That would have an allocation?

It seems we disagree on what allocation means. I'd expect allocation to either refer to something generated by the allocator (e.g. malloc or equivalent), or to a static. If every location has a 0 sized allocation, what is the provenance of the pointer? How was it allocated?

There is an 'invalid' provenance that is the only allowed provenance for these 0-sized 'virtual allocations'.
In Miri, we don't actually consider these allocations as existing. Instead, when we check whether a ptr is valid for a given memory access, then if the ptr has 'invalid' provenance (or "no provenance" or whatever you want to call it -- it is the provenance of a pointer created via ptr::invalid), we only accept the ptr if the access size is 0 (and the address is not 0, and it is sufficiently aligned). If the pointer has any other provenance, we du the full usual check, ruling out use-after-free or out-of-bounds accesses.

Question: why not make a shared static as the dangling pointer? That would have an allocation?

The issue is that NonNull::dangling can be called with types that need arbitrary alignment. Any static allocation we create might be insufficiently aligned.


feat: Improve move_cursor.

This aims to add a "sticky" option for move_cursor, where "sticky" keeps the cursor "stuck" to whatever character the cursor was on before the surround action. See the unit tests in this PR for an idea of how exactly this would work.
One nice use case for this is dot-repeating with a sticky cursor, as one can do ysiw]. to surround a word with double brackets, whereas the old behavior would do something like hello → [[]hello] (since the dot-repeat would surround the open square bracket).
Please react to show interest in this feature!
Replaces #289; is a partial solution to #281
TODO

[x] Unit test inserting where the selection is entirely before the cursor.
[x] Make it work for deletion/changing.
[x] Document the new functionality.

Dot-repeating isn't a feature in Visual mode, because there are no "semantics" to repeat in visual mode---only hard-coded coordinates begin/end of a visual selection.


Generics without generics (or better "Allow infer in every place")

Suggestion
I know the title sounds weird, but it definitely describes what I want.
🔍 Search Terms
generics, generics without generics
✅ Viability Checklist
My suggestion meets these guidelines:

[x] This wouldn't be a breaking change in existing TypeScript/JavaScript code
I don't know because it might remove errors, but that doesn't seem like a breaking change
[x] This wouldn't change the runtime behavior of existing JavaScript code
[x] This could be implemented without emitting different JS based on the types of the expressions
[x] This isn't a runtime feature (e.g. library functionality, non-ECMAScript syntax with JavaScript output, new syntax sugar for JS, etc.)
[x] This feature would agree with the rest of TypeScript's Design Goals.

⭐ Suggestion
In the last time I often saw problems like this, which I actually answered but not in the perfect way. In that question the author wants to create a function with other functions passed to it, where every function takes the output of the last one as it's own input. Sounds easy, right? No, it isn't.
My suggestion has to do something with that. I want to be able to create a type which will infer a type at a specified position and be able to use it elsewhere. This is an example:
type Example = {
	data: infer T,
	process: (data: T) => any
}

Self-explaining, right? So you take the type from one position and infer it at another position. Take a look at this generic function:
function doSomething<T>(data1: T, data2: T) {
	// some actions
}

When you call it with the same types, everything is fine. But when you use different types, T will be a union of those types. Take a look at this. Those problems could be solved with this feature:
function doSomething(data1: infer T, data2: T) {
	// some actions
}

📃 Motivating Example
I'm going to use the first example here. When you want to create a piping function, it is currently really hard. The only possible way is this:

export function asyncPipe<FS extends any[]>(...fns: AsyncParams<FS>): AsyncPipeReturnType<FS>
export function asyncPipe (...fns: AsyncParams<any[]>) {
  if (fns.length === 0) return () => Promise.resolve(null)
  return (x: Parameters<typeof fns[0]>[0]) => fns.reduce(async (y, fn) => fn(await y), x)
}

type Callback = (a: any) => MaybePromise<unknown>;
type FunToReturnType<F> = F extends Callback ? ReturnType<F> extends Promise<infer U> ? U : ReturnType<F> : never;
type EmptyPipe = (a: never) => Promise<never>

type AsyncPipeReturnType<FS extends Callback[], P = Parameters<FS[0]>[0]> =
  FS extends [...infer _, infer Last] ? (a: P) => Promise<FunToReturnType<Last>> : EmptyPipe;

type AsyncParams<FS extends Callback[], P = Parameters<FS[0]>[0]> =
  FS extends [infer H, ...infer Rest] ?
    H extends (p: P) => unknown ?
      Rest extends Callback[] ?
        [H, ...AsyncParams<Rest, FunToReturnType<H>>]
          : [{ error: "__A_PARAMETER_NOT_A_FUNCTION__" }, ...Rest]
            : [{ error: "__INCORRECT_FUNCTION__", provided: H, expected_parameter: P }, ...Rest]
              : FS;

It looks very complicated and it is! Not even the erroring works perfectly. You also need to specify the input type of every function explicitely, it is not inferred. There is an approach like this:
type Pipe<T, K, X = any> = [(i: T) => K] | [(i: K) => X, ...Pipe<X, K>];
function asyncPipe<T, K>(...pipes: Pipe<T, K>): (i: T) => K;

But it isn't even logically correct. When the bug (or feature?) with the recursive things are implemented, you could do something like this with my suggestion:
type Pipe<T, K> = [(i: T) => K] | [(i: T) => infer U, ...Pipe<U, K>];

Doesn't that look good and logical?
💻 Use Cases
As seen before this could be used to implement more complex types than now.
If you want to have an object with the same type twice like this:
type Obj<T> = {
	data: T,
	process: (data: T) => any
}

And want to store it in a variable like this:
const obj: Obj<?> = {
	data: "hello",
	process: (data) => null // what is `data`?
}

It doesn't work. You would need a function call for that. But with this suggestion you just write:
type Obj = {
	data: infer T,
	process: (data: T) => any
}
const obj: Obj = {
	data: "hello",
	process: (data) => null // `data` is of type string!
}

😀 Last words
I know this seems like a small change but it will open the door to much more complex types with a variable length of generic parameters (even if they are not really generic 😀). Please tell me if you have any ideas about this suggestion!
I'm not sure how to store that type data, because when you access obj.data from the last example, it is not directly stored in the type name Obj. I want to end with a question: Where do we get that type data from?

This suggestion commingles some related ideas and needs some untangling.
What you have here is two different things:

Inference of unspecified generics in type positions - this is #26349. That feature stands alone as a useful thing that has the same desirability whether the type was written with explicitly-introduced or implicitly-introduced type parameters. Moreover, it'd be just plain weird if you could only take advantage of this behavior for types written with implicit type parameters.
Implicit introduction of type parameters - more on that

Self-explaining, right?

Very much not 😅
Implicitly introducing type parameters with infer isn't really workable because in practice it's extremely unclear what the implied context of those type variables should be. In your examples here, you've posited them to occur in a function context and in a type alias context. If you wrote something like
type M = {
  (arg: infer T): void;
}

It's not clear if you meant
type M = {
  <T>(arg: T): void;
}

or
type M<T> = {
  (arg: T): void;
}

Similarly, if you wrote
type Obj = {
	data: infer T,
	process: (data: infer T) => any
}

It's not clear if you meant there to be two different inference positions for T, or if you actually meant
type Obj = {
	data: infer T,
	process: <T>(data: T) => any // Shadowing declaration?
}

Once those types exist (in non-call positions), we need to be able to refer to them again later in a deterministic manner for things like declaration emit and tooltips.
Defining the scope of a type parameter is just an absolute necessity for both humans and computers to reason about something. The conciseness gained here just isn't worth it.

@RyanCavanaugh

It's not clear if you meant there to be two different inference positions for T, or if you actually meant

Can't we make a contract for these cases?
type Obj = {
	data: infer T,
	process: (data: infer T) => any
}

would translate to:
type Obj<T> = {
	data: T,
	process: <U>(data: U) => any
}

And
type Obj = {
	data: infer T,
	process: (data: T) => any
}

to:
type Obj<T> = {
	data: T,
	process: (data: T) => any
}

In a context infer keyword should be the precedence for inference.


Change getProtocol => getProtocolVersion

In current("^3.0") amphp/artax version, there is no getProtocol method, instead of getProtocolVersion method exists in artax response. If you try, you will see kind of this error:
"Error: Call to undefined method class@anonymous::getProtocol() in foobar.php on line xxx"

Thanks!

@kelunik you're welcome!


On show and hide of modal, child component inside modal mount and unmount

Environment
Environment:
OS: macOS High Sierra 10.13.5
Node: 6.9.4
Yarn: Not Found
npm: 3.10.10
Watchman: 4.9.0
Xcode: Xcode 9.4 Build version 9F1027a
Android Studio: 2.2 AI-145.3537739
Packages: (wanted => installed)
react: ^16.3.1 => 16.4.1
react-native: ^0.55.0 => 0.55.4
Description
I am using graphQL for data query. There is modal which show listing of data with header, footer and listing in middle. Listing is child component of modal box, challenge is that on each time hide and show of modal box mount and unmount on listing component which will cause to delay response to show.
Could anybody help me about this to prevent this?
Reproducible Demo
            <Modal
            onRequestClose={this.props.onRequestClose}
            visible={this.props.isVisible}>
             <View style={{flex:1}}><ListingComponent/></View>
            </Modal>

Hi @karanjthakkar, I have prevent to new request to server for new data but this actual problem is that we have lots of business logic on data listing like filter, sorting and many more so it is taking process to complete this, due to mount and unmount on every time of show and hide  this is taking to time show.

I have this issue too! I am trying to animate colors in the modal, but when it closes, the animation re-appears!


fix(coverage): Prevent hard failure in case of empty file

What does this PR do?
Adapt the merge_dev_in_main method to prevent hard failure when the main file is empty
Motivation
We sometimes faces failures in coverage uploadm, because some files stored in the cache are empty (in my case it was pkg/trace/stats/oteltest/coverage.out.
If such case can happen we might want to prevent such failure to occur
Describe how to test/QA your changes
No QA but we can monitor we don't have no more occurrence of some logs here
Possible Drawbacks / Trade-offs
Additional Notes

/merge


sequel adapter

Hi, I just updated DatabaseCleaner's README about this Sequel adapter.  Does this adapter support multiple connections?  Looking at the code it appears to but I don't know how it is suppose to be used.  Would someone familiar with this pull request update the README to reflect usage? https://github.com/bmabey/database_cleaner/commit/501025b9dea4bfa7a03b937eb959dd9c9c5aea95#L0R165
I'm also really confused about ::Sequel::DATABASES.  I started to look into Sequel and I saw no such constant mentioned in the docs.  So if this is wrong/outdated I'd appreciate someone familiar with Sequel to submit a patch.  Thanks!


_csrf cookie sometimes generates invalid values which triggers ModSecurity

What steps will reproduce the problem?
activate ModSecurity (always active on shared hostings providers like OVH)
include this in the controller:
public function beforeAction($action) {
if($action->id == 'index'){
$this->enableCsrfValidation = false;
$cookies = Yii::$app->response->cookies;
$cookies->remove('_csrf');
unset($cookies['_csrf']);
}
return parent::beforeAction($action);
}
and refresh the index action a few times to generate new _csrf cookies
after some refreshes (randomly) it creates a "bad cookie"
What is the expected result?
standard index page
What do you get instead?
403 Forbidden
You don't have permission to access / on this server.
nor any other page, ModSecurity stops it before reaching the actual index.php file
(you get the bad cookie about once every 10 refreshes, and need to delete the cookie to start it behave normally again)
example of "bad" cookie content: "964315fe9b4ea193c9706742f641945b5e64db97e407423a84ca465195f684b1a%3A2%3A%7Bi%3A0%3Bs%3A5%3A%22_csrf%22%3Bi%3A1%3Bs%3A32%3A%22%EE3%3C%0B%02%9D%5C%91%5D%E4_Z%96%F9%BB%3F%CE%00%EB%B8%A5n%D6%DE%9D%98%8F%9A%1FJ%E5%60%22%3B%7D"
Additional info

Q
A

Yii version
2.0.12

PHP version
7.0.17

Operating system
Linux 4.4.67-ovh-vps-grsec-zfs-classid

Any idea why it triggers it?

this is one of the error log lines, but i don't really understand the problem... it's not my field...
[Thu Jul 27 08:23:23 2017] [error] [client xx.xx.xxx.xxx] ModSecurity: Access denied with code 403 (phase 2). Found 1 byte(s) in REQUEST_HEADERS:Cookie outside range: 1-255. [file "/usr/local/apache2/conf/modsecurity/base_rules/modsecurity_crs_20_protocol_violations.conf"] [line "252"] [id "960901"] [rev "2.1.1"] [msg "Invalid character in request"] [severity "WARNING"] [tag "PROTOCOL_VIOLATION/EVASION"] [tag "WASCTC/WASC-28"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE3"] [tag "PCI/6.5.2"] [tag "http://i-technica.com/whitestuff/asciichart.html"] [hostname "XXXXXX.net"] [uri "/index.php"] [unique_id "WXmG2woAV1EAAH5oQPYAAAC4"]

Aha. That is helpful. Thanks.

Why we are using binary data as csrf token instead of random string?

We've used to base64() the date before but few releases ago started to use raw binary date. It seemed right at that time cause browsers are fine with that.
When fixed there should be a test that ensures ASCII is written in the cookie.

@Mithrandil please check if ModSecurity is happy with pull request branch. Thanks.


Ping display resets to upper left corner.

I'm not sure when it happens but I notice the ping display keeps resetting to the upper left corner. I have to go in the options all the time to unlock it and drag it back to my preferred position.

Possibly fixed with v2.2.5


Are these threading setup necessary in SS2G_GridComp ?

https://github.com/GEOS-ESM/GOCART/blob/df0e5a80865e73a473f80bef1d198fa55f9baa40/ESMF/GOCART2G_GridComp/CA2G_GridComp/CA2G_GridCompMod.F90#L132-L133
Those corresponding lines and others are not found in SS2G_GridComp. @aoloso

If SS doesn't have thread private data structures, then those lines won't be necessary.


ability to disable get_current_site

getting issue because have to use restful api by IP, in some countries my domain is not resolvable (don't know why)
when requested by IP getting this backtrace:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/exception.py", line 35, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/base.py", line 128, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/base.py", line 126, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/rest_framework/views.py", line 483, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.6/dist-packages/rest_framework/views.py", line 443, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.6/dist-packages/rest_framework/views.py", line 480, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/djoser/utils.py", line 33, in post
    return self._action(serializer)
  File "/usr/lib/python3/dist-packages/djoser/views.py", line 133, in _action
    self.send_password_reset_email(user)
  File "/usr/lib/python3/dist-packages/djoser/views.py", line 150, in send_password_reset_email
    settings.EMAIL.password_reset(self.request, context).send(to)
  File "/usr/local/lib/python3.6/dist-packages/templated_mail/mail.py", line 68, in send
    self.render()
  File "/usr/local/lib/python3.6/dist-packages/templated_mail/mail.py", line 60, in render
    context = make_context(self.get_context_data(), request=self.request)
  File "/usr/lib/python3/dist-packages/djoser/email.py", line 30, in get_context_data
    context = super(PasswordResetEmail, self).get_context_data()
  File "/usr/local/lib/python3.6/dist-packages/templated_mail/mail.py", line 32, in get_context_data
    site = get_current_site(self.request)
  File "/usr/local/lib/python3.6/dist-packages/django/contrib/sites/shortcuts.py", line 13, in get_current_site
    return Site.objects.get_current(request)
  File "/usr/local/lib/python3.6/dist-packages/django/contrib/sites/models.py", line 62, in get_current
    return self._get_site_by_request(request)
  File "/usr/local/lib/python3.6/dist-packages/django/contrib/sites/models.py", line 47, in _get_site_by_request
    SITE_CACHE[domain] = self.get(domain__iexact=domain)
  File "/usr/local/lib/python3.6/dist-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/django/db/models/query.py", line 403, in get
    self.model._meta.object_name
django.contrib.sites.models.DoesNotExist: Site matching query does not exist.

I think at least wrap it in try should be useful feature

I think this is not a problem with django-templated-mail, but rather with your django.contrib.sites configuration. Did you provide SITE_ID in your project settings?

@hawi74 SITE_ID was disabled because I'm processing multiple domains with this snippet
https://stackoverflow.com/questions/25681883/django-multiple-domains-single-app
class MultiSiteMiddleware(object):

    def process_request(self, request):
        try:
            domain = request.get_host().split(":")[0]
            request.site = Site.objects.get(domain=domain)
        except Site.DoesNotExist:
            return http.HttpResponseNotFound()

@theromis this is probably because you are stripping the port while Django's get_current_site function never does it. I highly recommend that you use this function as well, instead of directly accessing Site model. You will also have to include the port in your Site instance. Please let me know if that helps.

@hawi74 Sorry for posting wrong code snippet, here is my actual code used in project:
cat multihost.py
"""

 A simple middleware component that lets you use a single Django
 instance to server multiple distinct hosts.

 IMPORTANT!! Make sure this is the FIRST entry in your MIDDLEWARE_CLASSES

 Revision log:

 v1.2 - 10th January 2012 (Cal Leeming -<EMAIL_ADDRESS>  * Added 'LoadingTime' response header (tells us how long the request took to process)
  * Added 'MultiHost' response header (tells us if multihost was used or not)
  * Added 'HOST_MIDDLEWARE_URLCONF_MAP' example
  * Cleaned up code slightly

https://code.djangoproject.com/wiki/MultiHostMiddleware
# File: settings.py

HOST_MIDDLEWARE_URLCONF_MAP = {
    # Control Panel
    "www.example.com": "webapp.sites.example.urls",
}
"""

import time
from django.conf import settings
from django.utils.cache import patch_vary_headers

class MultiHostMiddleware:

    def process_request(self, request):
        try:
            request.META["LoadingStart"] = time.time()
            host = request.get_host()
            #if host[-3:] == ":80":
            #    host = host[:-3] # ignore default port number, if present

            # best way to do this.
            host_port = host.split(':')
            if len(host_port)==2:
                host = host_port[0]

            if host in settings.HOST_MIDDLEWARE_URLCONF_MAP:
                request.urlconf = settings.HOST_MIDDLEWARE_URLCONF_MAP[host]
                request.META["MultiHost"] = str(request.urlconf)
            else:
                request.META["MultiHost"] = str(settings.ROOT_URLCONF)

        except KeyError:
            pass # use default urlconf (settings.ROOT_URLCONF)

    def process_response(self, request, response):
        return response
        if 'MultiHost' in request.META:
            response['MultiHost'] = request.META.get("MultiHost")

        if 'LoadingStart' in request.META:
            _loading_time = time.time() - int(request.META["LoadingStart"])
            response['LoadingTime'] = "%.2fs" % ( _loading_time, )

        if getattr(request, "urlconf", None):
            patch_vary_headers(response, ('Host',))
        return response

@theromis are you running a single Django app instance that handles multiple domains or separate instance for each domain? Have you tried creating a Site instance for the domain that causes the issue?

@hawi74 Yes, it handles about 4 different domains and it have them in sites db. rest of it in this snippet, I need different domains to handle different localizations as well.


https://www.ucas.com/

Hi @anudeepND
https://www.ucas.com/ is the UK University application website, and track.ucas.com is the section for students to follow their application.
It matches the regex.list entry ^track(ers?|ing)?[0-9]*[-.] by https://github.com/mmotti/pihole-regex that I've implemented.
Thanks
Matt

@mc9662 Thanks for the info and your regex list, been using it for a while. So far so good :)


vendor: Unfork logrus dependency

Vendor the latest official logrus release instead of a custom fork (which now happens to be identical to the fork).

LGTM


Apply futurize stage2

Just for kicks lets see how much this breaks

The answer is "a lot" but from a few problems so far.


fixing maven test making them deterministic (filesystem order) + fixi…

…ng ant matcher setup

Thanks for your contribution to RAT-228 - I applied the changes manually. Pls close the pull request as the SVN<->Github integration does not allow a direct interaction between both worlds.


This issue is actually more of user experience than just option, more like Available Offline option. We didn't want to expose terms like "sync" or "local" to users.


std.env.args does not work

Things to fix

cargo-verify does not expect command line arguments and does not pass to KLEE during verification
argc/argv are just arguments to 'main' in C but in Rust they are special global variables so we need to know their name. And their name is different on OSX from Linux (see the source)
For KLEE, I think we need to pass the --posix-runtime argument (which is not supported on OSX) (This is not true)

The way this works when executing natively on Unix (non-miri) __libc_start_main (in glibc) calls all initializers in the .init_array.* sections. These should be functions taking three arguments argc, argv, envp.
The Rust runtime inserts the function sys::unix::env::imp::ARGV_INIT_ARRAY::init_wrapper(argc, argv, envp) into the .init_array.00099 section. The function just stores argc and argv for later access via sys::env::args(). See this article.
To emulate this behaviour, we need to

collect all data objects from sections called .init_array.* and either generate a sequence of function calls (one per object) or a function that will iterate over the array calling each function.

modify the 'main' function to invoke this initializer before doing anything else.

This could be done in each separate verification tool (eg based on the way that KLEE handles static constructor/destructors).
Or it could be done in the cargo-verify script.

The way this works when executing natively on Unix (non-miri) __libc_start_main (in glibc) calls all initializers in the .init_array.* sections. These should be functions taking three arguments argc, argv, envp.
The Rust runtime inserts the function sys::unix::env::imp::ARGV_INIT_ARRAY::init_wrapper(argc, argv, envp) into the .init_array.00099 section. The function just stores argc and argv for later access via sys::env::args(). See this article.
To emulate this behaviour, we need to

collect all data objects from sections called .init_array.* and either generate a sequence of function calls (one per object) or a function that will iterate over the array calling each function.

modify the 'main' function to invoke this initializer before doing anything else.

This could be done in each separate verification tool (eg based on the way that KLEE handles static constructor/destructors).
Or it could be done in the cargo-verify script.

Closed by #77

Closed by #77


Algolia search results links are broken

I succesfully set up Algolia search for my MinimalMistakes powered blog. But I have one weird issue: the article links from the results page only contain the relative part, not the entire baseurl + relativeurl. When I'm looking into the Algolia index records I see that the url field only contains the relative part - not sure if this is correct in the first place. Any ideas why this might happen?
The config section looks like:
search_provider          : "algolia"
algolia:
  application_id         : "..."
  index_name             : "claudiu.blog"
  search_only_api_key    : "...."
  powered_by             : true

An example of the behavior: https://5d232abe6aaab60007c46223--claudiublog.netlify.com/cautare/
I also submitteed an issue at the jekyll-algolia repo, but there's no activity in there :(

Hmm not sure. The MM demo site has Algolia enabled in the theme and the search links appear to be working fine.
They're just the relative part as well. Only difference I see are all of your links have a double forward slash which could be breaking things... e.g. În țara-n care m-am născut. Not sure how are why it's doing that for you but I'd look into that. It might not have anything to do with Jekyll Algolia so I wouldn't limit your search there.

I have a feeling you may be using baseurl improperly. Without seeing your _config.yml file, but if you have baseurl: / I'd remove that line. That's likely adding the extra /
This is the relevant line in the theme if you want to see how links are being built.

${title}

Bingo. I think the / baseurl was default, I had no idea what that is causing. I removed that and now the links are correct. Thank you so much!


[iOS][Code Maintainence] Provision to provide custom Bundle path

Platform

iOS

Is your feature request related to a problem? Please describe.
It's related to Issue 4167 - No Cheveron on Action.ShowCards
Describe the solution you'd like
The fix provided was good and it involved modifying the ACRButtonExpandable.xib which resides in the AdaptiveCards source code.
It is good to have a custom bundle path that can be provided to the SDK where our own .xib resides, so that we can own the design.

@jwoo-msft please prioritize this for 20.08 after we've shipped 20.07 including Inputs! thanks.

@jwoo-msft please prioritize and ensure all your open issues per this query are closed out and checked into main ahead of this week before taking this one on pls! We are looking to fork and provide privates for testing to Teams by tues, 10/27 EOD

@jwoo-msft is this still on track for this week for 20.10?

This should be solved by using swift package manager, please reopen if this is still an issue for you.


[PLAYGROUND] Dynamic meta tags data for transaction links

🍰 Pullrequest
test vue-meta

it was to understand the interplay


Implement Navigation Between Pages

You may want to change this instruction to include exactly where to put the tags within Page1.xaml and Page2.xaml and the code within the Page1xaml.cs and Page2.xaml.cs.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: be10565a-f999-569f-3314-f1691cfa3d6f
Version Independent ID: 3e862b52-b135-e9af-5c13-424002f038d8
Content: Peer-to-peer navigation between two pages - UWP app developer
Content Source: windows-apps-src/design/basics/navigate-between-two-pages.md
Service: unspecified
Product: windows
GitHub Login: @serenaz
Microsoft Alias: sezhen

The tutorial instructions specify where to place the tags within the XAML pages. As for the CS code, you can place the methods anywhere inside the class!


New Testimonial

Name
dhruvabhat24
Title
Open Source Projects
Description
I had the pleasure of collaborating on a couple of open-source projects, including a Weather App and an E-Commerce App. Working with them has been an enriching experience.
@dhruvabhat24 possesses a remarkable skill set in creating applications and websites. The applications they have worked on showcase an ease of understanding in both the codebase and the overall code workflow. I look forward to future opportunities to work together.

Thank you.


Accessing DbContext from model

Version: EntityFrameworkCore 1.1.0 (SQL)
Hi,
i can't figure out how to access the DbContext from a readonly property in the model.
Maybe with the IEntityMaterializerSource (optionsBuilder.ReplaceService...) and using another constructor  to inject the current DbContext?
Thanks for help.
Sebastian

@sepplK Not really possible right now unless you are willing to hack internal code. Covered by #626 and see my comments on #7124


"Bad Realm file header (#1)" error when opening Realm file

When attempting to read from a Realm DB file, I am occasionally getting a "Bad Realm file header (#1)" error. It appears that the DB files gets corrupted while the app is running in the background (we do perform writes in the background based on push events). The faulty code is as follows (with some omitted, but this likely won't be helpful at all
do {
   let realm = try Realm(configuration: config)
} catch let error as NSError {

}

Below is the corrupted realm file if that helps
corrupted.realm.zip
Realm version: 096.3
iOS: 9.1
Configuration: Debug

@mowens can you provide some more details about what writes are occurring in the background?

@mowens is this Realm file supposed to be encrypted? We just identified an issue which may have caused that assertion to be hit for encrypted files under very specific scenarios (when a file is resized on open).
Also, if you can provide more details about what writes are occurring in the background, that could help us identify what's happening.

@bigfish24 The writes that occur in the background are pretty minimal. We basically update our cache in response to push notifications to sync resources between devices or from location changes. So basically a write in the background would be something like:

Make HTTP request

Write resource to disk (disk)

Publish resource over a socket

Write resource to disk (cache)

Nothing really fancy going on.
@jpsim The realm file is not configured for encryption

@mowens your transaction may seem minimal and unfancy, but you may be attempting a combination of operations in an order which has slipped through our unit tests so far! So we'd appreciate if you could share this code, especially if there's a chance that we can trigger the assertion on our end, which would drastically improve the odds of us finding what's wrong here.
If you'd rather share that code privately, you can email<EMAIL_ADDRESS>referencing this GitHub issue.
And thanks for letting us know that the file isn't encrypted.

@jpsim
There are 2 possible places where this can be happening on our code from a write:

DB compact on start up
Write in background

Here is our code that we run when the app launches to compress our DB file size:
public func compactPersistanceWithConfiguration(config: Configuration, deleteIfCorrupted: Bool = false) {
    dispatch_async(config.writeQueue) {
        autoreleasepool {

            do {

                // force migrate realm
                if let error = migrateRealm(config.createRealmConfiguration()) {
                    Log.error {
                        Log.event("PersistCompactFailed")
                            .message("Failed to migrate realm")
                            .field("path", value: config.path)
                            .error(error)
                    }
                }

                let realm = try Realm(configuration: config.createRealmConfiguration())

                // create random path to copy compact realm to
                let compactPath = NSTemporaryDirectory().stringByAppendingFormat("%0f.realm",
                    NSDate().timeIntervalSince1970 * 1000)

                // copy a compact version of the DB
                try realm.writeCopyToPath(compactPath, encryptionKey: nil)

                var url: NSURL?
                try NSFileManager.defaultManager().replaceItemAtURL(NSURL(string: "file:///\(config.path)")!,
                    withItemAtURL: NSURL(string: "file:///\(compactPath)")!,
                    backupItemName: nil,
                    options: .UsingNewMetadataOnly,
                    resultingItemURL: &url
                )
            } catch let error as NSError {
                Log.error {
                    Log.event("PersistCompactFailed")
                        .field("path", value: config.path)
                        .error(error)
                }
                if deleteIfCorrupted {
                    do {
                        try NSFileManager.defaultManager().removeItemAtPath(config.path)
                    } catch let error as NSError {
                        Log.error {
                            Log.event("PersistCompactCorrupted")
                                .message("Failed to remove corrupted realm file")
                                .field("path", value: config.path)
                                .error(error)
                        }
                    }
                }
            }
        }
    }
}

Our Configuration class is just a object that contains information about which write and callback queues should be used when accessing Realm and the Realm.Configration that we create for each store. We abstract our realm access behind a PersistentStore class. The Realm.Configuration we create is versioned w/ a migration block and specifies which Object types should be managed by the realm instance. (if need you to see this code, let me know)
Our Realm abstraction (e.g. the PersistentStore is a generic class that will manage a single object type. We encode/decode the objects to our own entity models so we do not have to have our resources extend the Realm instance. Here is a basic example:
class Foo: Resource {
    var id: String

    init(id: String) {
       self.id = id
    }
}

class FooPersistent: Object, PersistentType {
   dynamic var id: String = ""

   class func fromModel(model: AnyObject) -> FooPersistent {
       let object = FooPersistent()
       if let model = model as? Foo {
            object.id = model.id
       }
       return object
   }

   func toModel() -> AnyObject {
         return Foo(id: id)
    }
}

Then each PersistentStore will manage a pair of these:
let fooStore = PersistentStore<Foo, FooPersistent>(config: configuration.createRealmConfiguration())

Now we can store instances of Foo using the fooStore and the PersistentStore will handle the encode/decode of to the actual Realm Object that is stored. Here is the code for performing a write using the PersistentStore:
    public func add(object: T, closure: ((T) -> Void)?) {
        dispatch_async(config.writeQueue) {
            autoreleasepool {
                Log.verbose {
                    Log.event("Adding object to database")
                        .field("db", value: NSStringFromClass(T.self))
                        .field("path", value: self.config.path)
                        .field("size", value: self.realmSize())
                        .field("object", value: "\(object)")
                }

                let realm = self.createRealm()
                // convert from Foo to FooPersistent
                let realmObject = U.fromModel(object)

                do {
                    try realm?.write {
                        realm?.add(realmObject, update: true)
                    }

                    self.notify(.Added, model: object)

                    // we convert our objects so we can always invalidate the realm objects in memory
                    realm?.invalidate()

                } catch let error as NSError {
                    Log.error {
                        Log.event("PersistentFailure")
                            .message("Failed to write to realm db")
                            .field("db", value: NSStringFromClass(T.self))
                            .field("object", value: "\(object)")
                            .error(error)
                    }
                }

                // always on the main thread
                if let closure = closure {
                    self.invokeCallback {
                        closure(object)
                    }
                }
            }
        }
    }
```

The code for creating our realm instance:
```
    private func createRealm() -> Realm? {
        do {
            let realm = try Realm(configuration: config.createRealmConfiguration())
            return realm
        } catch let error as NSError {
            Log.error {
                Log.event("PersistRealm")
                    .message("Failed to allocate an instance of realm. Current DB action will not be performed!")
                    .field("db", value: NSStringFromClass(T.self))
                    .error(error)
            }
        }
        return nil
    }
```

So a write would be invoked like so:

```
let foo = Foo(id: "foobar")
fooStore.add(foo) { storedFoo in

}
```

Hope this helps!

Thanks for sharing those details! I'll try to reproduce and get back to you.

@mowens I'm afraid there's too much code that wasn't shared for me to stub in without making so many assumptions as to void the initial conditions that led to the corruption that you've reported to us.
Without having sufficient sample code from you for us to be able to build and run, it'll be impossible to make more progress on this because we won't have anything actionable. Is there any chance you could email your project privately to us at<EMAIL_ADDRESS>so we can attempt to reproduce? We could sign a mutual NDA enforcing that we can only use your code privately and with the sole goal of improving the Realm framework, if that makes you feel more comfortable.
There's nothing that jumps out at me in the snippets you've shared as being able to lead to corruption, although I do wonder why all the wrappers around Realm db connections, writes and models are necessary. If you find yourself wrapping many of our APIs to provide a nicer interface, I'd love to hear more about your rationale to see if our API could be improved.
Thanks!

@jpsim I would have to check with my VP to see if he is willing to travel down the NDA road. We haven't seen this crash in a while as we added recover code that will detect this and rebuild everything from the network. I will continue to monitor and see if I can ever get a better crash dump of this.
As for why we are wrapping Realm API's with our own, this is purely for abstraction reasons. The idea is that we don't let the Realm API bleed out of our PersistentStore API this way if for any reason we needed to switch to a different DB we could just by refactoring the PersistentStoreStore implementation.
We are also using promises (PromiseKit) heavily so we wrap our code (which isn't illustrated in the above code samples), but with wrappers around Realm API's we can provide promise based writes which is very nice for caching purposes:
For example:
firstly {
    // make an HTTP GET at /foo/:id and decode HTTP body into a Foo instance
    httpClient.get("/foo/\(id)", Foo.self)
}.then { foo in
   // cache Foo instance in the store
   return store.add(foo)
}.then { foo in
   // do something with the Foo instance
}.error { error in
   // handle errors
}

So with promises it allows us to do some really nice chaining, but this is really an implementation detail of some decisions we have made :) I think the Realm API is pretty damn awesome :+1:

I understand that you may not be able to share your full code with us, but please also understand that without being able to reproduce this issue ourselves, it's very unlikely we'll be able to fix it. The other option would be for you to take the time to make a clean-room sample project that triggers the same error  without any proprietary information and share that with us.
As for your rationale behind the wrappers, I certainly see the value in having promise helpers. Hiding Realm usage behind an agnostic wrapper, however, will only complicate your codebase without simplifying a potential future move to a different data layer in my opinion, but I understand the motivations. Thanks for sharing!

@jpsim I would have to check with my VP to see if he is willing to travel down the NDA road. We haven't seen this crash in a while as we added recover code that will detect this and rebuild everything from the network. I will continue to monitor and see if I can ever get a better crash dump of this.

Hi @mowens, any news on this? We'd love to continue investigating what may be causing this corruption, but don't have enough to go on at the moment. Let me know if there's anything I can do to help, like talking to your VP for example.

Hey @jpsim I think we figured out what the issue was and it is completely irrelevant to Realm. What I believe to have happened was we had 2 threads running (one doing a write) and one doing something else and that something else crashed. Being that this was running the background at the time, next boot up would show that Realm had been corrupted (which makes sense).
Thanks for looking into this and sorry for taking you down a rabbit hole on this one.

@mowens Realm has measures in place to prevent corruption from ever happening, even when write transactions are interrupted by the process crashing. It seems those safeguards failed in this case, which would still be considered a Realm bug.

Hi there, I still faced this issue on Realm 2.2.0, with message :
EXCEPTION io.realm.exceptions.RealmFileException
LOCATION SourceFile line -2 in SharedRealm.nativeGetSharedRealm()
TIME 12/21/16, 7:32
MESSAGE Unable to open a realm at path '/data/data/com.planday.ninetofiveapp/files/346b6fdd431da7ed53942b847aa7afea531fb9f7.realm': Bad Realm file header (#1). in /Users/cm/Realm/realm-java/realm/realm-library/src/main/cpp/io_realm_internal_SharedRealm.cpp line 92

@NguyenVV please file a new issue taking care to fill out the template.


getState() function

getState() function is implemented , where the user is returned with the value of the key passed.
I have already documented the above function , go through once and accordingly changes can be done if any .

hey @lindelwa122 , if any suggestions are there , I can do the necessary changes.

@lindelwa122 , now I will be taking up the deleteStore  function .

Thank you, @rakshithapatel08, for your dedication. Although the function is set to undergo updates to incorporate localStorage, your current implementation is commendable. Feel free to explore other issues or features and see if something else is of interest to you.


Backport deps updates, refactorings and fixes from master

Fixes #3100
Short description
In order to keep stable aka v2 maintained, we need to backport the updates and refactorings (code quality) from the work done for Arbitrum into the stable branch. This includes the fix for the linked bug, which is also currently affecting Goerli. Keeping dependencies up to date is also important for security, in order to ensure we're not hit by fixed security flaws.
This doesn't pull the schema and contracts changes, which remain in master for arbitrum only, but keep the rest of the codebase somehow closer to master in order to facilitate any later bugfix and updates.
Most of these commits were already reviewed as part of Arbitrum work (sans conflict fixes). Otherwise, just the last 6 commits should require independent review.
Definition of Done

[x] Steps to manually test the change have been documented
[ ] Acceptance criteria are met
[ ] Change has been manually tested by the reviewer (dApp)

Steps to manually test the change (dApp)

To merge this, we need to locally serve the current gh-pages version, install it, create some state etc. Then deploy locally this version and see if it works.

I've tested locally with old state (channels and udc deposits) in goerli, and this version worked fine; performed some direct transfers, everything working as expected.


Repair Nightly Storage Build

azure-storage-file-datalake nightly run is failing
The link above takes you directly to the reason that the test is failing. Namely, that the version of azure-storage-blob installed for latestdependency and mindependency environments is resolving to azure-storage-blob==12.23.0b1.
This is due to the way that python versioning works. When we want to pull in a "dev" version of the package for nightly, we need to update the requirement to an alpha one.
azure-storage-blob>12.23.0 becomes azure-storage-blob>12.23.0a1. However, due to how pip version resolution works, b1 is considered newer than a1. As a result, we are never resolving the alpha version we expect, and are instead picking up a previously published azure-storage-blob==12.23.0b1.
This PR updates our set_dev_version script to add the beta restriction, so any actually intentionally published beta packages don't collide in latest / minimum dependency checks.

This storage build should succeed if my hunch is right

Failure is due to build-extended CG failure. That is an ongoing issue with the cg dris.
Nightly storage checks out


Automatically from release workflow open PR with fresh version of diktat in antrun diktat config

mvn versions:set-property -Dproperty=diktat-check.version etc.
Property is introduced in #111

Covered in #108, #222


Desktop view: Language select menu sometimes missing

For example on /en/users/profile/ or the search landing page.

Please assign this to me


which version support prestodb.io 0.227

hi
i use prestodb0.227,  which version supports it ?

Hi! You can try branch dev_prestodb-0.220_0.1.2, it's developed base on prestodb-0.220, I think it can run on prestodb-0.227. Well, of course some latest functions are not supported, but it runs good.


Andrii: Agile Development, 4 weeks

Learning Objectives

Priorities: 🥚, 🐣, 🐥, 🐔 (click to learn more)

There is a lot to learn in this repository. If you can't master all the material
at once, that's expected! Anything you don't master now will always be waiting
for you to review when you need it. These 4 emoji's will help you prioritize
your study time and to measure your progress:

🥚: Understanding this material is required, it covers the base skills you'll
need for this module and the next. You do not need to finish all of them but
should feel comfortable that you could with enough time.
🐣: You have started all of these exercises and feel you could complete them
all if you just had more time. It may not be easy for you but with effort you
can make it through.
🐥: You have studied the examples and started some exercises if you had time.
You should have a big-picture understanding of these concepts/skills, but may
not be confident completing the exercises.
🐔: These concepts or skills are not necessary but are related to this module.
If you are finished with 🥚, 🐣 and 🐥 you can use the 🐔 exercises to push
yourself without getting distracted from the module's main objectives.

0. Developing HTML & CSS
Practice with the skills, tools, and workflows you will need to efficiently
develop websites written with HTML & CSS.

🥚 VSCode: you can ...

[x] use the LiveServer extension to launch an HTML page in the browser
[x] use autocomplete to begin a new HTML document, create new tags, and much
more
[x] use the HTML CSS Support and IntelliSense fo CSS class names
extensions to make writing HTML & CSS more efficient

🥚 Formatting HTML & CSS: You can ...

[x] use VSCode configurations to format your code on save
[x] use npm run format to format all the code in your repository

[x] 🥚 Linting CSS: You can use the npm run lint:css script to find
mistakes in your CSS, and you can fix the mistakes.
[x] 🥚 Validating HTML: You can use the npm run validate:html script to
find mistakes in your CSS, and you can fix the mistakes.
🥚 Browser Developer Tools: You can ...

[ ] inspect the source code of a web page using the Sources tab
[ ] find any part of the web page in the Elements tab of your Dev Tools DOM
inspector
[ ] find the styles for any element in the Elements tab
[ ] make temporary changes in a web page from the Elements tab to see how
they will look

1. Following Plans
Practice working together in a group to build small websites in incremental
steps according to a plan. You will be given final code, a detailed plan to
rebuild it, and will need to work as a group following the processes in
Planning and Collaborating.

[ ] 🥚 Project Definition and Scope: As a group, you can agree on what you
are trying to build, and can clearly describe the project using a README
and a repository description.
🥚 Project Constraints: As a group you can define project constraints by
discussing your strengths, weaknesses and context. You can set these types of
constraints:

[ ] External: constraints that come from outside your group
[ ] Internal, Involuntary: constraints that come from within your group,
but your group did not decide to have.
[ ] Internal, Voluntary: constraints your group decides on to improve the
quality of your development process and final software.

[ ] 🥚 Communication Plans: You can work with your group mates to agree on
a communication plan to follow during development.
[ ] 🥚 Communicating Challenges: During development you can discuss with
your group when ...

you are blocked and need help the agreed channels of communication.
your schedule or availability changes
you are having trouble getting along with each other

[ ] 🥚 Using a Backlog: You can develop from a backlog of features
organized by priority:

must-have
should-have
could-have

[ ] 🥚 Follow a Development Strategies: You can follow a development
strategy to understand the code required for must-have user stories.
[ ] 🥚 Create Issues From a Dev Strategy: You can create issues for user
stories in your project based on the development strategy.
🥚 Retrospective: Your group can complete a retrospective to ...

[ ] assess how well you worked as a group
[ ] find ways to improve in the next project
[ ] compare your plan to the reality of developing your website

🐣 Claiming and Completing Issues: You can claim issues from the TODO
column of a project board and ...

[ ] track your progress with the project board
[ ] develop your code locally on a separate branch
[ ] send a PR that passes CI checks and meets the requirements

[ ] 🐣 Code Review: You can both review a group mate's PR, and
productively participate in a review of your own code.
[ ] 🐣 Match a Design & Acceptance Criteria: When developing your tasks,
you can write code that matches the project's Wireframe and meets your
issue's Acceptance Criteria.

2. Writing Plans
Practicing writing your own plans as a group. You will be given only the final
code for HTML/CSS web pages, your group will need write a detailed plan for
rebuilding the website in small steps.

[ ] 🥚 Write User Personas: As a group you can describing your site's
users with User Personas, and you can use the personas to plan your
website from the user's perspective.
[ ] 🥚 Write User Stories: Given a working web page, you can describe it
with user stories and acceptance criteria for each user story.
[ ] 🐣 Write a Backlog: After listing all the user stories for a web page,
you can organize them into a Backlog by priority: must-have,
should-have and could-have
[ ] 🐣 Create a Prioritized Design: Using your backlog and the original
site as a guide, you can create a Wireframe of the site your group will
build. You can use different colors and comments to indicate a priority
for each component in the Wireframe.
[ ] 🐣 Write a Development Strategies: You can translate your Wireframe
and Backlog into a Development Strategy for the must-have user stories.

3. Adapting Plans
Even the best plans are not perfect, it's not possible to know everything ahead
of time! You may be able to plan everything ahead of time when the web pages you
build are very small and you have the code ahead of time, but it's a whole
different thing when you need to build a larger web page over a longer period of
time.
In this chapter you will practice full Agile Development by developing a larger,
open-ended web pages and adapting your plans as you go. Your team will need to
make an initial plan, but the plan will change! Practice having regular meetings
with your group to review the website's progress, review your plan, and make any
changes in the plan that are necessary to keep the project on schedule.

[ ] 🐣 Adjusting the Backlog: As a group you can agree to changes in the
backlog when necessary.
[ ] 🐣 Adjusting the Design: As a group you can agree on changes to the
design during the development process when it becomes clear your old
design needs updating.
[ ] 🐣 Updating the Project Board: You can manage the project board by
adding, removing or updating issues when necessary. This includes changing
labels, milestones, or assignees when necessary.

TOP

I Need Help With:

Planning as there so many files and its take time to understand them all and use properly.

What went well?

If don't talking about some conflicts when we was doing PR rest was pretty good.

What went less well?

I had some issue with npm install but it was because I updated my IOS. Also we had some conflicts when we creating PR but we fixed it.

Lessons Learned

On my opinion communication is the most important part of the job in group project.

Sunday Prep Work

Im studying HTML and still trying to be more confident with all planing documents.

Planning as there so many files and its take time to understand them all and use properly.

plenty of time to practice this.  it's the main objective of this module, and you'll use the same process and documents all the way through the final project

@andriivam as Evan said, we will use the exact same process in the next weeks as well in terms of teamwork, HTML, CSS, so you have time to get more familiar with it.

Thank you for your comments and really hope to learn and completely understand they how to planing web page from begging)

#Week 2
Need help in:
Css kind difficult to understand what parameters should I write for each elements.
What went well?

Planning documentation we did much more faster than last time
Now I feel much more confident with HTML
Creating issues and project board.
How to work in the team
How to create classes for css and how to use them

What went less well?

CSS styling/order

Lessons Learned

How to work in the team like in real life. Sometimes it could be difficult and as I say before in the team work probably the most important is COMMUNICATION!!!
To learn HTML and css need to practice a lot. And they which work for me don't try to do huge web page better is to learn parts by parts (css) as for big web page css can really big and its easy to get lost there.

@danielhalasz Thank you for advise. I will try to use more dev tools. Also I understand why so nice to babe big screen when you are coding)

Week 3
I Need Help With:
Im still study css as its not so easy to understand and learn fast. But also I have problem with Bootstrap. I just understand some thinks in css, how I can use them and now I don't know how to apply them in Bootstrap as its ready framework.
What went well?
Our planning documentation. Now, every time its better and better and also new level of quality.
Lessons Learned
Team and communication is the  important in Agile Development. Also need to divide your time for study and project separately.
Sunday Prep Work
Still on the preparation. Trying to understand Bootstrap

@andriivam if your team is more comfortable using regular CSS, then just decide on a naming convention and use that instead of Bootstrap.

Week 4
I Need Help With:
I feel that everything slowly coming and stay in my had. But still need to practice with css values to better understand them. And would be nice also to get some guide for media queries, how to set values for classes in right way.
What went well?
Grid. I  used the grid to create the card and i really liked. Even my css file are really simple as i dint need to set a lot of values. Also the communication and planning goes really well.
Lessons Learned
As always want to mention about the communication as its a key for success. But also i learned that always need to follow some system and before to use any system like bootstrap, gird, flexbox need to learn and understand them.
Sunday Prep Work
Im done with my part so now doind FreeCodeCamping and Scrimba courses to improve my skills.

@andriivam this article shows you more about media queries and the standard sizes:
https://www.freecodecamp.org/news/css-media-queries-breakpoints-media-types-standard-resolutions-and-more/

Thank you @danielhalasz i will take look there))


Add include_pattern rule for itables

Fixes https://github.com/jupyterlite/xeus-python-kernel/issues/126
Otherwise the itables fails at import because some files are missing.

CI failure does not seem to be related:

Opened https://github.com/emscripten-forge/empack/pull/69


Gtk Configuration

Hi @antoniosarosi! very good work on your dotfiles, your dotfiles are very beautiful! congratulations!!!
But I want to know what are your gtk configuration, it's possible? more specifically your cursor theme of
your gtk configuration, you can share it configuration?

https://github.com/antoniosarosi/dotfiles#gtk-theming

Thank you for the fast response!


ES-91: Upgrade Gradle Version

Upgrade gradle version to 8.1 and create new gradle wrapper as part of C5 update.
Added new files that were modified when running gradle with new version
Update check-pr-title.yml workflow to be able to accept ES PR

Confirmed Gradle version bump in on Build scan


refactor: satisfy attrs scenarios with updated types

@probablyup This is looking great. But it doesn't seem to fix the as problem yet:

Here is the snippet for cut-n-paste:
describe('foo', () => {
  const X = (props: { as?: 'div' | 'button' }) => null;
  const StyledX = styled(X)``;
  return (
    <>
      {/* GOOD: X doesn't accept section */}
      <X as="section" />
      {/* BAD: StyledX should accept section */}
      <StyledX as="section" />
    </>
  );
});


.setAutoCollectDependencies(false) still collects dependencies

I've updated the application insights package from 1.8.10 to 2.2.2
Since the update the dependencies are getting collected even when .setAutoCollectDependencies(false) is set on setup.
Is there any other option to turn this off in newer version that I missed?
This function worked correctly before the update.

Can you describe your environment a little more? is possible your dependencies are coming from different place; can you take a look at SDKVersion field in the unexpected telemetry and share it?

I'm using node 14, windows server 2016.
The dependencies started coming the moment I deployed the code to upgrade insights package from 1.8.10 to 2.2.2. There were no dependencies in insights before that. There were only exceptions and some custom events which we logged.
SDK field in insights says node:2.2.2
The application_Version is matching with the latest package version which we automatically increase on commit. So it must be from our latest update.

Hi,
I confirm this behavior if these calls are made on a client:

setup(key)
dispose
again setup(key)

Looking at the code, the variable isEnabled in HttpDependencies is not used here.
... && _this.isEnabled) should be added.
In the meantime, "interceptors" should be removed on http(s).request in dispose() method.

I agree with Benjamin's comment and I am able to reproduce the problem with the calls he indicated.
The developer's choice about whether or not to track dependencies is not really taken into account since the variable containing this information is only read before initialization. If this variable changes over time, the trackRequest  function seems to continue tracking requests regardless of the developer's choice.
This seems to be part of the problem but the real problem is the way dispose works which does not suppress "interceptors" leading to the possibility of overlapping identical "interceptors" when calling the setup method after calling the dispose  method.

@MatejSkrbis are you also disposing the telemetry client at some point?, auto collection of dependencies could also be enabled through configuration file, please check if APPLICATIONINSIGHTS_CONFIGURATION_FILE env variable is present or if applicationinsights.json have the setting on, more info here. @BenjaminBertran  and @Apokalypt  I will investigate the dispose issue, not sure it is the cause of original issue reported here.

auto collection of dependencies could also be enabled through configuration file, please check if APPLICATIONINSIGHTS_CONFIGURATION_FILE env variable is present or if applicationinsights.json have the setting on, more info here.

For my part, the file is completely empty ( file content : '{ }' ). All properties are defined with their respective methods ( e.g. : .setAutoCollectDependencies(true) to collect dependencies automatically )
It is perhaps a different problem which is evoked initially but the result being identical (duplicate of collect) I thought that it was interesting to check and to evoke it 👍

@MatejSkrbis are you also disposing the telemetry client at some point?, auto collection of dependencies could also be enabled through configuration file, please check if APPLICATIONINSIGHTS_CONFIGURATION_FILE env variable is present or if applicationinsights.json have the setting on, more info here. @BenjaminBertran and @Apokalypt I will investigate the dispose issue, not sure it is the cause of original issue reported here.

I am not disposing the telemetry client at any point. APPLICATIONINSIGHTS_CONFIGURATION_FILE is not present and I do not have applicationinsights.json settings file.

Is there any idea when this fix is going to be released, we've recently updated our dependencies and have seen a massive uptick in the amount of information being sent into App Insights instances, two specific event types alone in the Azure Storage SDK have been producing close to 22 million events in a 24 hour period despite us having explicitly turned off the auto-collection of dependencies.
In the meantime, I assume the suggested workaround would be to use the JSON file in the install location of the library?

Is this still an issue? I can see some code was merged, but after grabbing the latest 2.3.3 version from NPM and running a test with the below config, when I make a connection to a test blob storage account, I am still seeing 2 dependency events being logged in AI.

@MatejSkrbis are the dependencies you were seeing related to Azure SDK calls? if that is the case, I can close this issue with related changes.

@hectorhdzg I do not understand what you are asking me. I just described my issue and downgraded application insights to 1.8.10 and have no intention to update it back until this issue is fixed.


AMBARI-25607 disable UI tests with system property

What changes were proposed in this pull request?

Introduce a system property (ambari.web.skip) that could be used to skip the build of frontend artefacts. The build on Linux ARM64 fails because there is no ARM64 binary for PhantomJS.
Updates frontend-maven-plugin to 1.12.0
Fix two unit tests which were failing due to wrong number of parameters passed to PowerMock

How was this patch tested?
The build does not fail now on Linux ARM64 due to the web tests.

Is there any chance this PR could be reviewed/merged ? Thank you!


helmfile not support if else

Operating system
ubuntu 22.04
Helmfile Version
v0.151.0
Helm Version
v3.11.0
Bug description
helmfile not support if else in helmfile.yaml
Example helmfile.yaml
environments:
  default:
    values:
      - serviceType-not-patch:  NodePort
---
repositories:
  - name: bitnami
    url: https://charts.bitnami.com/bitnami
releases:
  - name: nginx
    namespace: nginx
    createNamespace: true
    chart: bitnami/nginx
    values:
    {{ if .Values.serviceType }}
      - service:
          type: {{ .Values.serviceType }}
    {{ end }}

Error message you've seen (if any)
if not supply serviceType  helmfile will give error, but i hope it not give error if user not supply serviceType
root@node35:~# helmfile apply
in ./helmfile.yaml: error during helmfile.yaml.part.1 parsing: template: stringTemplate:10:17: executing "stringTemplate" at <.Values.serviceType>: map has no entry for key "serviceType"

Steps to reproduce
create helmfile.yaml then run helmfile apply
Working Helmfile Version
v0.151.0
Relevant discussion
No response

@willzhang I don't understand the meaning of the title.  but you can use the get func.https://helmfile.readthedocs.io/en/latest/templating_funcs/#get

get function resolved my problems, thanks.
environments:
  default:
    values:
      - serviceType:  NodePort
---
repositories:
  - name: bitnami
    url: https://charts.bitnami.com/bitnami
releases:
  - name: nginx
    namespace: nginx
    createNamespace: true
    chart: bitnami/nginx
    values:
    {{ if .Values | get "serviceType1" "" }}
      - service:
          type: {{ .Values | get "serviceType1" "" }}
    {{ end }}