Configurable heap memory release rate

Expose config option for tcmalloc [memory background release rate](https://github.com/google/tcmalloc/blob/bf4db7e4c85b142ecdd37f8bf388075131c387e9/tcmalloc/malloc_extension.h#L637C15-L637C39, that eases tuning of tcmalloc in Envoy. Gperf tcmalloc is not yet supported in this change, as gperf tcmalloc memory release does not function the same way as tcmalloc does and introduced test flakiness.
Commit Message:
Additional Description:
Risk Level:
Testing: Unit tests
Docs Changes: API docs
Release Notes:
Platform Specific Features:

@mattklein123 would you mind taking another pass for api check?

/wait
for CI

/wait
for CI

Once/if this patch lands the ci in this PR will be fixed after merging main.

/wait

@all above issue is really critical and blocking us in production cases , can you quickly approve and provide fix asap?
thanks

+1 to proceeding with this, it is very useful.

@KBaichoo since switching to gperf tcmalloc for coverage builds turned out to be non feasible due to current code structure, i took time to investigate the original test flakiness in integration tests for gper tcmalloc. Ported the problematic test to gperf tcmalloc unit test suit and debugged the execution to get better understanding of how memory release algorithm works in gperf. It operates on a span level granularity and for each memory release call (please release N bytes <= K tcmalloc pages) gperf tcmalloc will attempt to find suitable spans (the ones that are free and have at least K pages in total) in free list and use it. It also tracks the extra amount of bytes released during previous release call (which is ~(spans size - requested bytes)) and if this amount exceeds the currently requested amount, no memory release will occur. Hence in integration tests for gperf tcmalloc we should not expect memory to be released every time the timer is fired for memory releasing, we should instead make sure that memory has been released at least once. There is another aspect of "flakiness" in gperf memory releasing, e.g. when span was attempted to be released via syscall (e.g. madvise()), gperf tcmalloc will not always bump the stats for unmapped bytes, under certain conditions it will instead increase the page heap free bytes. So in test we should verify that either unmapped bytes have increased or pageheap free bytes have increased. Regular tcmalloc on the other hand is less sophisticated and more predictable/stable and easier to test.
I have fixed the test for original patch (that supported both tcmalloc and gperf tcmalloc) and ran it locally with large enough number of iterations to make sure the flakiness is gone. The fixed tests looks like this:
TEST_F(MemoryReleaseTest, ReleaseRateAboveZeroCustomIntervalMemoryReleased) {
  size_t initial_allocated_bytes = Stats::totalCurrentlyAllocated();
  auto a = std::make_unique<uint32_t[]>(40 * MB);
  auto b = std::make_unique<uint32_t[]>(40 * MB);
  if (Stats::totalCurrentlyAllocated() <= initial_allocated_bytes) {
    GTEST_SKIP() << "Skipping test, cannot measure memory usage precisely on this platform.";
  }
  #if defined(GPERFTOOLS_TCMALLOC)
  size_t initial_free_bytes = Stats::totalPageHeapFree();
  #endif
  auto initial_unmapped_bytes = Stats::totalPageHeapUnmapped();
  EXPECT_LOG_CONTAINS(
      "info",
      "Configured tcmalloc with background release rate: 16777216 bytes per 2000 milliseconds",
      initialiseAllocatorManager(16 * MB /*bytes per second*/, 2));
  EXPECT_EQ(16 * MB, AllocatorManagerPeer::bytesToRelease(*allocator_manager_));
  EXPECT_EQ(std::chrono::milliseconds(2000),
            AllocatorManagerPeer::memoryReleaseInterval(*allocator_manager_));
  a.reset();
  step(std::chrono::milliseconds(2000));
  Stats::dumpStatsToLog();
  b.reset();
  step(std::chrono::milliseconds(4000));
  Stats::dumpStatsToLog();
#if defined(TCMALLOC)
  TestUtility::waitForCounterGe(stats_, "memory_release_test.tcmalloc.released_by_timer", 2UL,
                                time_system_);
  auto final_released_bytes = Stats::totalPageHeapUnmapped();
  EXPECT_LT(initial_unmapped_bytes, final_released_bytes);
#elif defined(GPERFTOOLS_TCMALLOC)
  // Gperf tcmalloc releases memory on a span granularity. If requested amount of memory to release is less than or equal
  // to the extra amount of memory released during previous release request, no memory will be released for the current request.
  TestUtility::waitForCounterGe(stats_, "memory_release_test.tcmalloc.released_by_timer", 0UL,
                                time_system_);
  auto final_released_bytes = Stats::totalPageHeapUnmapped();
  auto final_free_bytes = Stats::totalPageHeapFree();
  EXPECT_TRUE((initial_unmapped_bytes < final_released_bytes) || (initial_free_bytes < final_free_bytes));
#else
  EXPECT_LE(initial_unmapped_bytes, final_released_bytes);
#endif

}

If we decide to support memory releasing for both tcmallocs in Envoy, I would like to be explicit in docs that gperf tcmalloc algorithm is sophisticated and users should not necessarily expect memory to be released each timer fire period. E.g. to avoid the situation when we will get lots of user questions where we need to explain why gperf tcmalloc has acted the specific way.
Am ready to resubmit the original patch with fixed test and user docs.

After discussing offline with @KBaichoo we agreed to proceed with only supporting Google tcmalloc for now due to multiple reasons:

Google tcmalloc is more commonly used compared to gperf tcmalloc and better maintained
There is no known prod use case for gperf tcmalloc in Envoy
Complexity of gperf allocation algorithm, e.g. hard to write deterministic tests for memory release and lack of good project documentation and support, e.g. https://github.com/gperftools/gperftools/issues/1382

/retest

/retest

There is no known prod use case for gperf tcmalloc in Envoy

All Istio derivatives used gperftools until very recently actually - but I flipped it to tcmalloc for all new versions.

@KBaichoo @alyssawilk this is ready for another review.

/retest

@alyssawilk yeap, I will update to throwing an error instead of crashing.

/retest

Which envoy release will this feature be merged?
I found that from the latest v1.30 release the changes were reverted:

@anvo1115 memory releasing for tcmalloc allocator (not gperf) is supported in 1.30, please take a look: https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/bootstrap/v3/bootstrap.proto.html#envoy-v3-api-msg-config-bootstrap-v3-memoryallocatormanager


[process-agent] Migrate check queue settings to global config

What does this PR do?
This PR removes the QueueSize, RTQueueSize  and ProcessQueueBytes fields from the deprecated AgentConfig struct and instead relies on the equivalent settings from the global Config object.
Default values for these settings have been created in the pkg/config/process.go file. The validation once done in LoadProcessYamlConfig() is now performed when these settings are read and used in the Collector constructor.
Motivation
Refactor and improve process-agent config logic.
Additional Notes
process_config.queue_size, process_config.rt_queue_size and process_config.process_queue_bytes currently can't be overriden by env vars. This PR also creates DD_PROCESS_CONFIG_ and PROCESS_AGENT_ env vars for these settings.
Possible Drawbacks / Trade-offs
Describe how to test/QA your changes
Set the following settings in the datadog.yaml file:
log_level: 'debug'

process_config:
  enabled: "true"
  queue_size: 100
  rt_queue_size: 50
  process_queue_bytes: 20000

Start the process-agent and make sure that the following log lines are printed
Creating process check queue with max_size=100 and max_weight=20000
Creating rt process check queue with max_size=50 and max_weight=20000
Creating pod check queue with max_size=100 and max_weight=15000000

Update queue_size, rt_queue_size and process_queue_bytes to non-positive values and make sure that a Warn message is printed showing that process-agent falls back to the default values.
Repeat the tests with positives and non-positives values for
DD_PROCESS_CONFIG_QUEUE_SIZE
DD_PROCESS_AGENT_QUEUE_SIZE
DD_PROCESS_CONFIG_RT_QUEUE_SIZE
DD_PROCESS_AGENT_RT_QUEUE_SIZE
DD_PROCESS_CONFIG_PROCESS_QUEUE_BYTES
DD_PROCESS_AGENT_PROCESS_QUEUE_BYTES

Reviewer's Checklist

[x] If known, an appropriate milestone has been selected; otherwise the Triage milestone is set.
[x] The appropriate team/.. label has been applied, if known.
[x] Use the major_change label if your change either has a major impact on the code base, is impacting multiple teams or is changing important well-established internals of the Agent. This label will be use during QA to make sure each team pay extra attention to the changed behavior. For any customer facing change use a releasenote.
[x] A release note has been added or the changelog/no-changelog label has been applied.
[x] Changed code has automated tests for its functionality.
[x] Adequate QA/testing plan information is provided if the qa/skip-qa label is not applied.
[ ] If applicable, docs team has been notified or an issue has been opened on the documentation repo.
[x] If applicable, the need-change/operator and need-change/helm labels have been applied.
[x] If applicable, the config template has been updated.

@mbotarro Please don't force-push to PRs after the review has started, it makes it impossible to see what was changed.

Hi @vickenty, I'm really sorry for that. I've forced pushed another branch that is based on this one but for now on I'll merge main to it instead. Thanks for pointing it out!


Folding non-constant expressions

Many times there are expressions that can be simplified even if not all of the operands are constants.  A few of these are:
n+0
n*0
n*1
n / 1
b && true
b && false
b || true
b || false

It can be useful to simplify these, especially if they fold to a constant like b && false.
This issue will have a few different parts to it:

[x] Write a utility that folds any instruction if possible.
[x] Make use of this new utility in CCP
[x] Make use of this new utility in GVN

We have done all of the main parts of this.  We still have lots of folding rules we can add, but separate issues should be opened for those that are needed.


feat: Add support for autoscaling group AMI override

PR o'clock
Description
Resolves #1498
Allow users to specify different AMIs based on the instance type ( more details in #1498 )
Since aws has introduced eks support for arm instances  the module should also support the creation of asg with mixed instance type (arm64, x64).
The suggested way to implement this feature is to create multiple launchtemplates and overriding the default launch template with a specific one related to the instance type.
Example
module "eks" {
  ...
  worker_groups_launch_template = [
    {
      name = "spot-1"
      override_instance_types = [
        "m5.large",
        "t4g.medium"
      ]
      override_instance_ami = {
        "t4g.medium" : "<custom_ami>"
      }
    },
  ]
}

This change introduce more complexity in the aws_launch_template creation since we now have to create multiple aws_launch_template for the same aws_autoscaling_group if override_instance_ami is specified.
In order to avoid the duplication of aws_launch_template.workers_launch_template the resource is not created using a for_each which creates a launch template for every node group * n. of specified overrides,
Ideas to reduce the complexity of the features are welcome.
Checklist

[x] README.md has been updated after any changes to variables and outputs. See https://github.com/terraform-aws-modules/terraform-aws-eks/#doc-generation

Related PR - https://github.com/terraform-aws-modules/terraform-aws-eks/pull/1473


Package with Docker

This PR adds .dockerignore and Dockerfile to allow it to be deployed with Docker. The Dockerfile contains instructions & examples of use.
This Dockerfile contains many improvements with respect to #110, see https://github.com/onetimesecret/onetimesecret/pull/110#issuecomment-710808082.
To allow the application to run correctly under docker, some extra changes have been made:

Allow the application to be configured by environment variables too (this is how Docker usually works)
config.ru: Set stdout to be always unbuffered (so logs appear immediately). This is useful for both Docker and other environments.
config.ru: Set PUBLIC_DIR in production environment too, otherwise public files are not served.

If you don't like those changes, tell me and I'll find a workaround.

ruby:2.6 is the latest work version of ruby without errors

Hello @mildsunrise
Could I add a docker-compose.yml directly in your Pull Request ?
I guess I would need some dev access to the repo ? (never done that before)

I think this PR already has many changes as-is... to avoid complicating it further, I'd suggest you start a separate PR based on my branch, and add docker-compose.yml there :)

Will do, thanks Alba :)

is there anything missing to move this pr? i'd like to use it :)

is there anything missing to move this pr? i'd like to use it :)

Thank you! I see the config file question is resolved. Btw, I just noticed a typo in the help comments:

you should protect your Redis instance with authentication or Redis networks

Redis networks -> Docker networks

bump :)
is the docker version usable ?

@isAAAc Sure, why not?


Missing az keyvault set-policy with compound identity

[Enter feedback here]
looking at az keyvault set-policy docs comparing to Set-AzKeyVaultAccessPolicy PowerShell atlernative
since the two calls should be identical I could not find in the docs the option to run the following :
Set-AzKeyVaultAccessPolicy -VaultName $KVNAM -ObjectId $OID -ApplicationId $APP_ID -PermissionsToSecrets Get

the above command adds a permission to COMPOUND IDENTITY on the KV
PS docs :
https://docs.microsoft.com/en-us/powershell/module/az.keyvault/set-azkeyvaultaccesspolicy
Key Vault Security Overview  ( look for the term 'compound identity' )
https://docs.microsoft.com/en-us/azure/key-vault/general/security-overview

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 226e5769-f6c8-a480-db78-727fb96348ec
Version Independent ID: 8ee06a2a-6eda-11be-3c8e-6ad948be207e
Content: az keyvault
Content Source: latest/docs-ref-autogen/keyvault.yml
Service: key-vault
GitHub Login: @rloutlaw
Microsoft Alias: routlaw

add @houk-ms to have a look

+1, we need this @yungezz,  @houk-ms - we would be really appreciate it.


Feat: Create Fibonachicken

Finish Fibonachicken, Checking please

Sorry

Sorry

solved: #2

solved: #2

Solved #2

Solved #2


PcRegisterSubdevice failed with STATUS_ALLOTTED_SPACE_EXCEEDED

### Tasks

Sorry, can you provide more info about what you are requesting?

Sorry,I shoud provide more info previously.
The situation is that I am a newer to windows kernel driver and I want to dynamic add subdevices  a pair of mic and speaker.I have a app to send custom IOCTL code to the driver ,and when driver receive the code will add a pair of subdevices.But when I want to add the sixth pair of subdevices the API PcRegisterSubdevice failed with STATUS_ALLOTTED_SPACE_EXCEEDED.I think i pass valid parameters to PcRegisterSubdevice.

I am not sure is this issue caused by the subdevices name is the same,I wanna change the subdevices name, but like I said I am a newer to Windows kernel I do not faind a way to make subdevices name unique. Do you have any ideas?Thank you so muck and looking forward to your reply.

Hi @SlimSnow,
There could be a limit of 5.  Or you could try unique names. But I do not have specific knowledge in this area, so I am not sure how to do that.  I would recommend posting your question in the following forums, where other driver developers and Microsoft engineers can be found. The community there is quite responsive. This repo is for documentation, so if you discover anything that you think we should add to the docs, feel free to let us know.

Microsoft Q&A
OSR's NTDEV forum


[Docs] Fixing supported Ubuntu versions

This PR closes #608
The supported Ubuntu versions are: 12.04, 14.04 and 15.10.

LGTM


Editing will not recreate projects

I have re-written a few tests to better test their functionality. Additionally, I have build and installed this branch of the gem to ensure it actually works now.
@adamstrickland Thoughts?
Closes #355
Closes #354

Does this fix the bug described?  FWIW, I had a few minutes the other night and was not able to conclusively reproduce the reported failure behavior within a test; I don't see in this changeset anything along those lines?

As for replacing Tmuxinator::Config.exists? with File.exists?...  In general I prefer to use system libraries such as File rather than custom abstractions.  However, the use of Tmuxinator::Config.exists? predates my association with the project, so there may well be reasoning for it of which I am not aware.  @aziz @Soliah may have additional context.

The big issue here is that Tmuxinator::Config#exists? expects a project's name to be passed, but what is being passed here is a full file path to the project file. Tmuxinator::Config#exists? treats this full file path as the project's name, and cannot find such a project. Thus it "creates it", which incidentally overwrites the already-existing one. This is easily mitigated by replacing Tmuxinator::Config#exists? in this instance with File#exists?, which does expect a full file path.

WRT the failing tests, it is only failing for TMUX master, which unfortunately the master branch is as well.

@J3RN I wanted some tests for this specific failing case so that it doesn't slip in again (one of the shortcomings of using mocks, eh?), so I created some, merged your branch on top and now we've got green again.  To wrap it all together, I created a separate PR (side note: anyone know the proper git-iquette for this scenario?).  What do you think?


SmartConfig mac address handling

Extract the mac address from the SmartConfig response
Handle some SmartConfig timeout errors

Please merge - I signed off on the commit.  Maybe I also had to sign off on the PR.

Oh - I see - my commit has the proper Signed-off-by but James' commits don't.   I marked them DCO ok.
James just needs to get into the habit of git commit -s -m where -s adds the Signed-off-by
Anyway this PR is still open.  I'll leave it for James to review/decide if its worthy.

Krook gave me writer access to this repo. In theory I have access to merge the PR.  In practice, I don't like merging my own PRs against other developers' code bases.  I'll merge my own PRs against my code bases or if there are no maintainers with expertise to review.  I've requested a review by James.


Upgrade version of vcdvcd

Hi,
you seem to be depending on a broken version of vcdvcd.py.  After pip installing wal-lang, I get the following error:
(.venv) shareefj@JALAPINO:~/git/wal$ wal
Traceback (most recent call last):
  File "/home/shareefj/git/wal/.venv/bin/wal", line 5, in <module>
    from wal.wal import run
  File "/home/shareefj/git/wal/.venv/lib/python3.10/site-packages/wal/wal.py", line 7, in <module>
    from wal.core import Wal
  File "/home/shareefj/git/wal/.venv/lib/python3.10/site-packages/wal/core.py", line 6, in <module>
    from wal.trace import TraceContainer
  File "/home/shareefj/git/wal/.venv/lib/python3.10/site-packages/wal/trace.py", line 5, in <module>
    from vcdvcd import VCDVCD, StreamParserCallbacks
  File "/home/shareefj/git/wal/.venv/lib/python3.10/site-packages/vcdvcd/__init__.py", line 1, in <module>
    from .vcdvcd import *
  File "/home/shareefj/git/wal/.venv/lib/python3.10/site-packages/vcdvcd/vcdvcd.py", line 355, in <module>
    class Scope(collections.MutableMapping):
AttributeError: module 'collections' has no attribute 'MutableMapping'

and it looks like at some point they've tried to fix this:  https://github.com/cirosantilli/vcdvcd/commit/9c9b7c1aaa7a6bcc4e254e3557cc00a9ec1c7bd0

Installing vcdvcd to 2.3.3 seems to at least enable me to launch wal.

I replaced vcdvcd with pyDigitalWaveTools to resolve this error.
Now WAL should work with the latest Python version.


Maintainer

I am willing to take over maintenance of this feedstock if there is interest.

@hobu In these cases, make a PR to fix the build and then ping conda-forge/core who will happily merge.


Soft Search Model Train

Toy model trained. Will train a larger model tomorrow when the linker script is done.
https://github.com/si2-urssi/eager/blob/main/quick-setfit-model.ipynb


Fix index out of bounds access when using object labels longer than 64 characters

Fixes #3213.
I have not been able to test this change unfortunately since I have been unable to find a combination of wgpu, wgpu-rs and gfx that work without errors.
PR checklist:

[ ] make succeeds (on *nix)
[ ] make reftests succeeds
[ ] tested examples with the following backends:
[x] rustfmt run on changed code

I thought about SmallVec too, however it seems likely that it would be a lot slower. I'm thinking it would first try to copy it to a stack allocated 64 byte buffer and then when it realizes that this will not work it will copy it to the heap allocated buffer. So it might require two copies instead of 1.
Maybe it is smart enough to be compiled to the same thing though. I'll try to see what rustc generates.

The performance difference is even larger than I thought. For small strings (10 characters or so) the custom buffer is about 10 times faster than SmallVec. For large buffers (65 characters) its about 5 times faster.
smallvec_small          time:   38.203 ns
smallvec_large          time:   77.385 ns
custom_buffer_small     time:   2.9266 ns
custom_buffer_large     time:   16.082 ns

Here is the testing code I used: https://pastebin.com/eRtdpMy9
Even smallvec isn't really slow though, so maybe it's just premature optimization.

Wow interesting. Where you measuring with Release?
What I'd ideally expect to happen is that as_bytes().iter() returns an iterator that knows about its length, and so it can hint to SmallVec that the heap allocation is needed before any bytes are copied. Perhaps, it's just something that's missing from SmallVec atm.

Wow interesting. Where you measuring with Release?
Yup. I used the criterion library which does all kinds of warmups and stuff to reduce noise too.

Yeah that's what I would expect in the best case too, given rust's goal of no-overhead-iterators. But it seems like it is not supported by some part of the stack.

We need to back-port this to hal-0.5 branch, to address https://github.com/gfx-rs/wgpu-rs/issues/289


error not find generated_env.pcd

Thank you for publishing your project. I can't find uav_simulator/worlds/generated_env/generated_env.pcd'.

This file is not supposed to be contained in this repo. It was added by accident. I have removed it now.


Radio Buttons need zIndex in their style

Select one ...  (check one with "x")
[X] bug
[ ] feature request
[ ] enhancement

Expected behavior
I expect to be able to add a Clarity radio button inside a modal and add event handlers like (click) and get them to fire.
Actual behavior
The event handlers do not fire.
Reproduction of behavior
I found this bug today while trying to add Radio Buttons inside a modal with event handlers on them. The event handlers do not fire. The reason is the :before element is showing up on top of the input element. The easy CSS solution is to give the radio button inputs a z-index. I am now adding that override locally but you should fix it in the base product.

Include a link to the reproduction scenario you created by forking one of the Clarity Plunker Templates:

Environment details
2.4.1

Angular version: 2.0.X
0.9.2
Clarity version:
Mac OSX
OS and version:
Chrome 58.0.3029.110 (64-bit)
Browser: [all | Chrome XX | Firefox XX | IE XX | Safari XX | Mobile Chrome XX | Android X.X Web Browser | iOS XX Safari | iOS XX UIWebView | iOS XX WKWebView ]

@bbronson: Could you please post a plunkr replicating the issue? https://plnkr.co/edit/uNwwZe?p=info

Even without the plunker, I can already tell you that binding to the click event on radio buttons is a bad idea, because it's not accessible (and as a side effect, it will not work when a label is clicked). You want to listen to the change event, or even better, just use [ngModel)] two-way binding.

Hi Aditya,
I'm not going to upload the code because it is proprietary, and you can easily reproduce this with a clarity modal and a clarity radio button and any even handler, doesn't have to be onclick.
Thank you Youdz, I will keep that in mind, its just that this affects every event binding (mouseover, toggle, whatever) and it should be an easy fix for the Clarity team.

@bbronson: You don't have to upload your code. You just have to provide a simple reproduction of the error that you are facing in a plunkr. Without the plunkr we won't be sure if we are actually looking at the issue that you are facing. Thanks!

Nevermind, I've upgraded to the latest Clarity version and latest clr-model and this issue seems to be resolved. Marking as closed.


Add indexes to search by Product ID or PSD ref

https://trello.com/c/4YDZrTGw/1275-update-search-functionality-to-be-able-to-search-by-unique-product-id
Description
Adds OpenSearch indexes to allow searching products by their ID or PSD Reference.
IMPORTANT: Indexes must be manually deployed:
Product.__elasticsearch__.import force: true, refresh: :wait

Checklist:

[ ] Have you documented your changes in the pull request description?
[ ] Does the change present any security considerations?
[ ] Is any gem functionality overloaded? Eg: Devise controller methods being overloaded.
[ ] Has acceptance criteria been tested by a peer?
[ ] Has the CHANGELOG been updated? (If change is worth telling users about.)

General testing (author)

[ ] Test without JavaScript
[ ] Test on small screen

Accessibility testing (author)

[ ] Works keyboard only
[ ] Tested with one screen reader
[ ] Zoom page to 400% - content still visible
[ ] Disable CSS - does content make sense and appear in a logical order?

Codecov Report

Merging #1743 (0e0a38e) into master (d0bc912) will decrease coverage by 86.07%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master   #1743       +/-   ##
==========================================
- Coverage   89.74%   3.67%   -86.08%
==========================================
  Files         399     399
  Lines        8622   13128     +4506
  Branches     1252     231     -1021
==========================================
- Hits         7738     482     -7256
- Misses        701   12645    +11944
+ Partials      183       1      -182

Impacted Files
Coverage Δ

app/models/product.rb
0.00% <0.00%> (-100.00%)
:arrow_down:

app/models/role.rb
0.00% <0.00%> (-100.00%)
:arrow_down:

app/models/team.rb
0.00% <0.00%> (-100.00%)
:arrow_down:

app/models/alert.rb
0.00% <0.00%> (-100.00%)
:arrow_down:

app/models/contact.rb
0.00% <0.00%> (-100.00%)
:arrow_down:

app/models/country.rb
0.00% <0.00%> (-100.00%)
:arrow_down:

app/models/accident.rb
0.00% <0.00%> (-100.00%)
:arrow_down:

app/models/incident.rb
0.00% <0.00%> (-100.00%)
:arrow_down:

app/models/location.rb
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 357 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d0bc912...0e0a38e. Read the comment docs.


SSL_write fails with error message "error:0A00010F:SSL routines::bad length"

I have an openSSL client that transmits heavy volume of data to openSSL server and occasionally SSL_write fails with error message "error:0A00010F:SSL routines::bad length". Here goes more details:

I'm using OpenSSL 3.0.8 (I believe it is a custom built but unfortunately I don't have an exact configuration)
When transmitting around 1-2Gbps it works fine and it starts failing when it hits over 3-4Gbps.
This failure happens periodically like for 1-2 miuntes working fine and then happens 10-20 seconds and then it doesn't happen and so on.
I'm using non-blocking socket as underlying BIO.
My client is single-threaded. So for openSSL perspective, it is just a simple code that continuously calls SSL_write in a thread and upon returning "WANT_READ" or "WANT_WRITE" it waits epoll for EPOLLOUT. (Pretty standard)

I tried to dig in the openSSL code and learned that the error code 0A00010F can be broken down to three parts

0A --> library, which is ERR_LIB_DSA
000 --> function
10F --> reason, which is SSL_R_BAD_LENGTH

And I wasn't able to find a code part that generates this error, just through the code-reading.
I'd like to consult here in which direction I can further debug this?
Few more updates:

When I search up, most "bad-length" error comes with error code of library "0x14" which is ERR_LIB_SSL and that makes sense when it comes from SSL_write. But this one with "0A" I can't find good material out there so hope here I can get some insights.

Thanks for quick turnaround!
Here goes the content in openssl/err.h
define ERR_LIB_DSA             10
define ERR_LIB_X509            11
/* #define ERR_LIB_METH         12 */
define ERR_LIB_ASN1            13
define ERR_LIB_CONF            14
define ERR_LIB_CRYPTO          15
define ERR_LIB_EC              16
define ERR_LIB_SSL             20
And ERR_LIB_DSA  being 10 and hex 0x0A. And ERR_LIB_SSL  being 20 and hex 0x14.
I think the library code is ERR_LIB_DSA unless there is other way to read the library code.

I think @t8m was looking at this ASCII art:
 * An OpenSSL error packs like this:
 *
 * <---------------------------- 32 bits -------------------------->
 *    <--- 8 bits ---><------------------ 23 bits ----------------->
 * +-+---------------+---------------------------------------------+
 * |0|    library    |                    reason                   |
 * +-+---------------+---------------------------------------------+

So the library code is not the top 8 bits. The first bit is always 0 and its the next 8 bits. Which makes the library code 20 not 10 (i.e. @t8m is correct it is ERR_LIB_SSL)

"error:0A00010F:SSL routines::bad length"

How are you printing that error? You can normally get source line number information from OpenSSL errors which might help to pinpoint which specific instance this is.

ah, that was it! So it is from ERR_LIB_SSL. I think that would be a good starting point for me.
I'm using this line to get that error message:
ERR_error_string_n(ERR_get_error(), msg, sizeof(msg));

Also, I compiled openssl with default configuration (./Configure && make) and installed it. The same issue happens.

I checked that the error happens with positive message length so I can rule out the one from SSL_write and looks like it is coming from:
https://github.com/openssl/openssl/blob/openssl-3.0/ssl/record/rec_layer_s3.c#L368-L372
Which makes sense that in case of SSLfatal it gives 0 for function code.

ERR_error_string_n(ERR_get_error(), msg, sizeof(msg));

If instead of ERR_get_error() here you call ERR_get_error_all you can additionally get hold of the source filename and line number where the error was raised. That would confirm for sure a specific instance.
https://www.openssl.org/docs/man3.0/man3/ERR_get_error_all.html

in case of SSLfatal it gives 0 for function code.

Function codes are not used any more in OpenSSL 3.0, so this applies to all errors.

I verified that the error is coming from https://github.com/openssl/openssl/blob/openssl-3.0/ssl/record/rec_layer_s3.c#L368-L372

I see wnum is set to zero at (https://github.com/openssl/openssl/blob/openssl-3.0/ssl/record/rec_layer_s3.c#L380) after storing its original value to total (https://github.com/openssl/openssl/blob/openssl-3.0/ssl/record/rec_layer_s3.c#L358). And it only writes to wnum only when there were errors happened. So to my understanding, wnum will be always zero unless there were error happened and wonder if I understand this correctly.

Correct. wnum is always reset back to zero, and only set again if we fail to successfully write out all our remaining data.

If my understanding is correct and given the first SSL error is from https://github.com/openssl/openssl/blob/openssl-3.0/ssl/record/rec_layer_s3.c#L368, I guess the exception was triggered by second clause, ((wb->left != 0) && (len < (s->rlayer.wnum + s->rlayer.wpend_tot)) meaning that wpend_tot is larger than the original requested length. And this line (https://github.com/openssl/openssl/blob/openssl-3.0/ssl/record/rec_layer_s3.c#L1143) is most suspicious where it adds up the values coming from pipelines...

Well that actually would be quite interesting to confirm, i.e. exactly which of the two sub-expressions causes the failure and what are the various values that are causing it?
len here represents the length of the data that was originally asked to be written. This should remain the same value in subsequent calls to ssl3_write_bytes in the event of "retries".
wnum represents the number of bytes held in complete records that we have already successfully written out to the network.
wpend_tot represents the number of bytes we have packaged up inside a whole record and stored in the wb buffer to be written.
wb->left represents the number of bytes left in the buffer that are still to be written to the network. This could be a partial record if there were previous writes that only managed to send part of the record. So wb->left could be less than wpend_tot.
So the expression that is failing is: ((wb->left != 0) && (len < (s->rlayer.wnum + s->rlayer.wpend_tot))
The first part says "only do this check if we have a full or partial record waiting to be written to the network". The second part says the total length of the data we are trying to write must not be less than the total amount held in full records that we have already written plus the amount of data held in the record that is currently pending to be written. If that fails then something strange has happened.
I wonder if you might be able to modify that chunk of code to log out the various values (len, wnum, wpend_tot wb->left) in the event that the check fails?

BTW my application is a single-threaded application and wonder if the mis-use of openssl library can cause this behavior or if it is an openssl library side bug?

If you had said your application was multi-threaded then I would have thought the most likely reason was multi-threaded access to the SSL object (which would be an application error). But with a single threaded application this looks more like an OpenSSL bug (although it is curious that no-one else has encountered this).

What about the requirement that in case SSL_write() fails because the socket write returns EAGAIN/EWOULDBLOCK you have to call it again with the same parameters once the socket is ready for writing?

Yes - you're right that could cause this. I was thinking there were earlier checks for that which would have caught it - but actually I was thinking of the checks at the beginning of ssl3_write_pending which isn't called until after this check. So, this probably could be caused by an application error.

Do you mean, if SSL_write fails with SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE from SSL_get_error, I have to retry with the exact same parameters? (same buffer/same length)?
Actually in my application, it discards the buffer and try with a new one.

Yeah, https://www.openssl.org/docs/manmaster/man3/SSL_write.html the WARNINGS section.

Do you mean, if SSL_write fails with SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE from SSL_get_error, I have to retry with the exact same parameters? (same buffer/same length)?

Yes! You must do this. That would explain this problem.

ah got it. Thank you so much for your answers!
BTW I don't see the same warning in SSL_read case, so I assume this only applies to SSL_write?
And I don't set  SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER, so it is by default turn off and I can just retry with the original buffer pointer?

BTW I don't see the same warning in SSL_read case, so I assume this only applies to SSL_write?

Yes, SSL_write only.

And I don't set SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER, so it is by default turn off and I can just retry with the original buffer pointer?

Yes, SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER is off by default so you must retry with the original buffer pointer.

Sorry one last question: Can I use a different buffer pointer but with exact same content/size?
I went over my application code and if I want to maintain the original buffer pointer might take some major surgery so wonder if I can use different buffer pointer with exact same content/size.

Closing since this is marked as resolved.


get_swagger_ui_html does not pass X-API-Key to openapi_url for authorized access

First check

[x] I added a very descriptive title to this issue.
[x] I used the GitHub search to find a similar issue and didn't find it.

similar, yet different, issues or the answers are not satisfactory:

https://github.com/tiangolo/fastapi/issues/2146
https://github.com/tiangolo/fastapi/issues/2005
https://github.com/tiangolo/fastapi/issues/1550

very similar issue, but it does not exactly identify this issue as the main focus

https://github.com/tiangolo/fastapi/issues/1332

requires reinventing the wheel for get_swagger_ui_html

[x] I searched the FastAPI documentation, with the integrated search.
[x] I already searched in Google "How to X in FastAPI" and didn't find any information.
[x] I already read and followed all the tutorial in the docs and didn't find an answer.
[x] I already checked if it is not related to FastAPI but to Pydantic.
[-] I already checked if it is not related to FastAPI but to Swagger UI.
[x] I already checked if it is not related to FastAPI but to ReDoc.
[-] After submitting this, I commit to one of:

Read open issues with questions until I find 2 issues where I can help someone and add a comment to help there.
I already hit the "watch" button in this repository to receive notifications and I commit to help at least 2 people that ask questions in the future.
Implement a Pull Request for a confirmed bug.

Example
See https://medium.com/data-rebels/fastapi-authentication-revisited-enabling-api-key-authentication-122dc5975680 but the use of cookies to hold an API Key circumvents the behavior of the swagger-UI so that all the authorized endpoints are working without using the swagger authenticate entry.  If the cookie is not set (as below) and the api-key auth only allows an X-API-Key header, then the swagger-UI fails to get the openapi.json document; an error appears like "Failed to load API definition.  Fetch Error.  Unauthorized /openapi.json".
It seems like there should be a way to call get_swagger_ui_html so that any swagger JS can use an X-API-Key to authenticate when it tries to fetch the /openapi.json document.  Or, is there a way to define /docs so that the swagger-ui is provided the app.openapi() content without needing to call a /openapi.json endpoint that is protected by an X-API-Key?  (Additional context - a deployment uses an AWS API-Gateway with X-API-Key auth enabled on the API-Gateway with a use policy attached to the API keys - the API-Gateway seems to be tricky to configure to allow unrestricted access to /openapi.json when everything else requires an API key or some other auth, even if the fastapi itself might allow unrestricted access to /openapi.json.). Is it considered safe/secure to allow unrestricted access to /openapi.json?
See the medium post link above but, below, the cookie for an api key is disabled.

    # assume that get_api_key() checks X-API-Key header and returns key (str) when valid
    # see medium post linked above for example

    @app.get("/openapi.json", include_in_schema=False, tags=["documentation"])
    async def get_open_api_endpoint(api_key: str = Depends(get_api_key)):
        # note how this endpoint requires authentication with an X-API-Key
        response = JSONResponse(app.openapi())
        return response

    @app.get("/docs", include_in_schema=False, tags=["documentation"])
    async def get_documentation(api_key: str = Depends(get_api_key)):
        response = get_swagger_ui_html(openapi_url="/openapi.json", title="docs")

        # Note that when cookie is set, it works, but it also circumvents the swagger auth,
        # but when the cookie is not set, the swagger-ui cannot get the openapi.json doc;
        # how can `get_swagger_ui_html` pass an X-API-Key to the /openapi.json endpoint?

        # Can the `response` here be modified so this works with API keys?  The following
        # addition to the response headers is _not_ the way to go for this, it exposes the API
        # in the response to /docs and the swagger-js does not pass it along anyways.
        response.headers["X-API-Key"] = api_key

        # While this cookie approach works, it circumvents the swagger-ui control over auth;
        # any 'Try it out' will use this cookie and work, without setting any swagger-ui auth.
        # response.set_cookie(
        #     API_KEY_NAME,
        #     value=api_key,
        #     # domain=COOKIE_DOMAIN,
        #     httponly=True,
        #     max_age=1800,
        #     expires=1800,
        # )
        return response

Description

Open the browser and call the endpoint /docs.
It returns an error with failure to fetch /openapi.json
But I expected it to return the swagger-ui with the app openapi.json etc

Request
Would it be possible to use
# tell swagger about using the api_key for authenticated access to `/openapi.json`
response = get_swagger_ui_html(openapi_url="/openapi.json", title="docs", api_key=api_key)

The function signature does not yet include such an option, i.e.
def get_swagger_ui_html(
    *,
    openapi_url: str,
    title: str,
    swagger_js_url: str =<EMAIL_ADDRESS>    swagger_css_url: str =<EMAIL_ADDRESS>    swagger_favicon_url: str = "https://fastapi.tiangolo.com/img/favicon.png",
    oauth2_redirect_url: Optional[str] = None,
    init_oauth: Optional[Dict[str, Any]] = None,
) -> HTMLResponse:
    pass ... etc etc

The X-API-Key auth option is not covered by any init_oauth options, is it?  It should not require any OAuth flows.
I'm not so familiar with swagger JS, is it possible to use it with an X-API-Key?

the HTML response uses a default swagger JS from:
<EMAIL_ADDRESS>
Are there fatal security flaws in attempting to use swagger JS in this way?  e.g. it could embed the API key in the JS payload somewhere, basically exposing it in the HTML/JS response payload.
When /docs returns, it is possible to force it to open a modal dialog that would present the swagger-ui auth dialog that requires an API-key auth?  (Assuming that the swagger JS can even get the /openapi.json doc)
Environment

FastAPI Version:  0.63.0

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"

$ python --version
Python 3.7.9

Relevant SO item at

https://stackoverflow.com/questions/42995594/how-to-use-basic-auth-in-swagger-ui-v-3-0
https://github.com/swagger-api/swagger-ui/issues/2793

If I've read those correctly, the proposed solution is a requestInterceptor, e.g.

from https://github.com/swagger-api/swagger-ui/issues/2793#issuecomment-335360204

// The spec URL
const url = "https://www.example.com/authorised-users-only/spec.json";

SwaggerUI({
  url,  // spec url
  requestInterceptor: (req) => {

    // Only set Authorization header if the request matches the spec URL
    if (req.url === url) {
      req.headers.Authorization = "Basic " + btoa("myUser" + ":" + "myPassword");
    }

    return req;
  }
})

For my question, the header details would be an X-API-Key header vs. an Authorization header.  I'm just not sure what the consequences are for putting an API-key value into the swagger-ui JS - assuming it can only get there because a user provides it in the first place, it's only as secure as their browser session or something.

Relevant SO item at

https://stackoverflow.com/questions/42995594/how-to-use-basic-auth-in-swagger-ui-v-3-0
https://github.com/swagger-api/swagger-ui/issues/2793

If I've read those correctly, the proposed solution is a requestInterceptor, e.g.

from https://github.com/swagger-api/swagger-ui/issues/2793#issuecomment-335360204

// The spec URL
const url = "https://www.example.com/authorised-users-only/spec.json";

SwaggerUI({
  url,  // spec url
  requestInterceptor: (req) => {

    // Only set Authorization header if the request matches the spec URL
    if (req.url === url) {
      req.headers.Authorization = "Basic " + btoa("myUser" + ":" + "myPassword");
    }

    return req;
  }
})

For my question, the header details would be an X-API-Key header vs. an Authorization header.  I'm just not sure what the consequences are for putting an API-key value into the swagger-ui JS - assuming it can only get there because a user provides it in the first place, it's only as secure as their browser session or something.

Using a copy-paste and hack-it, the following custom function adds support for a request interceptor that adds an X-API-Key header in order to get the /openapi.json document from an endpoint that is protected by an API key (without further interference with the swagger-ui after that).

import json
from typing import Any
from typing import Dict
from typing import Optional

from fastapi.encoders import jsonable_encoder
from starlette.responses import HTMLResponse

def custom_swagger_ui_html(
    *,
    openapi_url: str,
    title: str,
    swagger_js_url: str =<EMAIL_ADDRESS>    swagger_css_url: str =<EMAIL_ADDRESS>    swagger_favicon_url: str = "https://fastapi.tiangolo.com/img/favicon.png",
    oauth2_redirect_url: Optional[str] = None,
    init_oauth: Optional[Dict[str, Any]] = None,
    api_key: Optional[str] = None,
) -> HTMLResponse:

    # Adapted from fastapi.openapi.docs.get_swagger_ui_html
    # Related to https://github.com/tiangolo/fastapi/issues/2678
    # Related to https://github.com/swagger-api/swagger-ui/issues/2793

    html = f"""
    <!DOCTYPE html>

    <link type="text/css" rel="stylesheet" href="{swagger_css_url}">
    <link rel="shortcut icon" href="{swagger_favicon_url}">
    <title>{title}</title>

    """
    return HTMLResponse(html)

Using a copy-paste and hack-it, the following custom function adds support for a request interceptor that adds an X-API-Key header in order to get the /openapi.json document from an endpoint that is protected by an API key (without further interference with the swagger-ui after that).

import json
from typing import Any
from typing import Dict
from typing import Optional

from fastapi.encoders import jsonable_encoder
from starlette.responses import HTMLResponse

def custom_swagger_ui_html(
    *,
    openapi_url: str,
    title: str,
    swagger_js_url: str =<EMAIL_ADDRESS>    swagger_css_url: str =<EMAIL_ADDRESS>    swagger_favicon_url: str = "https://fastapi.tiangolo.com/img/favicon.png",
    oauth2_redirect_url: Optional[str] = None,
    init_oauth: Optional[Dict[str, Any]] = None,
    api_key: Optional[str] = None,
) -> HTMLResponse:

    # Adapted from fastapi.openapi.docs.get_swagger_ui_html
    # Related to https://github.com/tiangolo/fastapi/issues/2678
    # Related to https://github.com/swagger-api/swagger-ui/issues/2793

    html = f"""
    <!DOCTYPE html>

    <link type="text/css" rel="stylesheet" href="{swagger_css_url}">
    <link rel="shortcut icon" href="{swagger_favicon_url}">
    <title>{title}</title>

    """
    return HTMLResponse(html)

The additional tweaks for this to work with AWS API-Gateway require adding an API-Gateway server to the openapi.json doc, e.g. assume that get_aws_uri(request) is able to use the request event data to get the host and stage from the request to dynamically add the server to the openapi doc.

def get_aws_uri(request: Request) -> Optional[str]:
    # https://mangum.io/adapter/#retrieving-the-aws-event-and-context
    # https://github.com/awsdocs/aws-lambda-developer-guide/blob/master/sample-apps/nodejs-apig/event.json
    aws_event = request.scope.get("aws.event", {})
    context = aws_event.get("requestContext", {})
    domain_name = context.get("domainName")
    domain_stage = context.get("stage")
    if domain_name and domain_stage:
        url = f"https://{domain_name}/{domain_stage}"
        return urlparse(url).geturl()

# now within api factory method:

    def custom_openapi(request: Request):
        if app.openapi_schema:
            return app.openapi_schema
        openapi_schema = app.openapi()
        aws_uri = get_aws_uri(request)
        if aws_uri:
            # AWS API-Gateway is deployed to a baseURL path for a stage
            openapi_schema["servers"] = [{"url": aws_uri}]
        app.openapi_schema = openapi_schema
        return app.openapi_schema

    @app.get("/openapi.json", include_in_schema=False, tags=["documentation"])
    async def get_open_api_endpoint(
        request: Request, api_key: str = Depends(get_api_key)
    ):
        response = JSONResponse(custom_openapi(request))
        return response

    @app.get("/docs", include_in_schema=False, tags=["documentation"])
    async def get_documentation(request: Request, api_key: str = Depends(get_api_key)):
        aws_uri = get_aws_uri(request)
        if aws_uri:
            openapi_uri = f"{aws_uri}/openapi.json"
        else:
            openapi_uri = "/openapi.json"
        response = custom_swagger_ui_html(
            openapi_url=openapi_uri, title="docs", api_key=api_key
        )
        return response

The additional tweaks for this to work with AWS API-Gateway require adding an API-Gateway server to the openapi.json doc, e.g. assume that get_aws_uri(request) is able to use the request event data to get the host and stage from the request to dynamically add the server to the openapi doc.

def get_aws_uri(request: Request) -> Optional[str]:
    # https://mangum.io/adapter/#retrieving-the-aws-event-and-context
    # https://github.com/awsdocs/aws-lambda-developer-guide/blob/master/sample-apps/nodejs-apig/event.json
    aws_event = request.scope.get("aws.event", {})
    context = aws_event.get("requestContext", {})
    domain_name = context.get("domainName")
    domain_stage = context.get("stage")
    if domain_name and domain_stage:
        url = f"https://{domain_name}/{domain_stage}"
        return urlparse(url).geturl()

# now within api factory method:

    def custom_openapi(request: Request):
        if app.openapi_schema:
            return app.openapi_schema
        openapi_schema = app.openapi()
        aws_uri = get_aws_uri(request)
        if aws_uri:
            # AWS API-Gateway is deployed to a baseURL path for a stage
            openapi_schema["servers"] = [{"url": aws_uri}]
        app.openapi_schema = openapi_schema
        return app.openapi_schema

    @app.get("/openapi.json", include_in_schema=False, tags=["documentation"])
    async def get_open_api_endpoint(
        request: Request, api_key: str = Depends(get_api_key)
    ):
        response = JSONResponse(custom_openapi(request))
        return response

    @app.get("/docs", include_in_schema=False, tags=["documentation"])
    async def get_documentation(request: Request, api_key: str = Depends(get_api_key)):
        aws_uri = get_aws_uri(request)
        if aws_uri:
            openapi_uri = f"{aws_uri}/openapi.json"
        else:
            openapi_uri = "/openapi.json"
        response = custom_swagger_ui_html(
            openapi_url=openapi_uri, title="docs", api_key=api_key
        )
        return response


Update tutorials

Ran mktutorial to regenerate tutorial content.

This adds 17 azure-nextgen tutorials, but they aren't added to the TOC page (https://www.pulumi.com/docs/tutorials/azure/).  I'm not sure exactly what we want to do here (we could put them into the existing Azure ToC, but intermixing them there without more work to explain the difference doesn't feel right).  I think it's fine to merge this as-is for now and deal with it as part of transitioning azure-nextgen to be the default in Q1 - but a little odd that these tutorials will have no inbound links at all.  cc @mikhailshilkov


Label issue when the first thing you do is to paste a date

Your demo: http://amirkabirdataminers.github.io/ADM-dateTimePicker/#/
the last field, if you try to paste a date in it's format (like 2016/04/20 00:00) AND then you open the date-picker we have the following problems:

the correct date in the picker is not set by using the date pasted
more important, the label is never updated when selecting the date in the datepicker

You are right,
I think this problem happens only when ngModel is empty.

Nice, if I find the time, I can help you fixing it!


Detox test hanging on device.launchApp command in Release configuration android

Describe the bug
When I run detox test -c android.emu.release the app installs on the emulator and appears to launch when looking at the emulator but in the logs I see it gets stuck at the device.launchApp command. The test then times out. detox test -c android.emu.debug is working well.
To Reproduce

[x] I have tested this issue on the latest Detox release and it still reproduces

This is the first time I have tried running detox on android so is not a regression for me. I set up exactly as in provided docs. detox build -c android.emu.release is successful and I can see the required apk files in the correct folders.
      "android.emu.debug": {
        "binaryPath": "android/app/build/outputs/apk/debug/app-debug.apk",
        "build": "cd android && ./gradlew assembleDebug assembleAndroidTest -DtestBuildType=debug && cd ..",
        "type": "android.emulator",
        "name": "Pixel_2_API_29"
      },
      "android.emu.release": {
        "binaryPath": "android/app/build/outputs/apk/release/app-release.apk",
        "build": "cd android && ./gradlew assembleRelease assembleAndroidTest -DtestBuildType=release && cd ..",
        "type": "android.emulator",
        "name": "Pixel_2_API_29"
      }

detox build -c android.emu.release
detox test -c android.emu.release

I have tried on Pixel_2_API_29 and Nexus_6_API_28 emulators and on a real attached device.
init.js
const detox = require('detox');
const config = require('../../../package.json').detox;
const adapter = require('detox/runners/jest/adapter');
const specReporter = require('detox/runners/jest/specReporter');

// Set the default timeout
jest.setTimeout(120000);
jasmine.getEnv().addReporter(adapter);

// This takes care of generating status logs on a per-spec basis. By default, jest only reports at file-level.
// This is strictly optional.
jasmine.getEnv().addReporter(specReporter);

beforeAll(async () => {
  await detox.init(config);
});

beforeEach(async () => {
  await adapter.beforeEach();
});

afterAll(async () => {
  await adapter.afterAll();
  await detox.cleanup();
});

Test:
  it('should successfully sign up user with phone number', async () => {
    await device.launchApp();
  });

Expected behavior
The tests should run as they do in debug mode and on iOS.
Environment (please complete the following information):

Detox: 14.0.1
React Native: 0.59.4
Node: 12.6.0
Device: MacBook Pro
OS: MacOS Mojave 10.14.5
Jest: 24.5.0

Device and Verbose Detox Logs
detox[31936] INFO:  [DetoxServer.js] server listening on localhost:51839...
detox[31936] DEBUG: [AsyncWebSocket.js/WEBSOCKET_OPEN] opened web socket to: ws://localhost:51839
detox[31936] TRACE: [AsyncWebSocket.js/WEBSOCKET_SEND] {"type":"login","params":{"sessionId":"912056cf-c6f0-7a10-b7e7-63bdcc9b734a","role":"tester"},"messageId":0}
detox[31936] DEBUG: [DetoxServer.js/LOGIN] role=tester, sessionId=912056cf-c6f0-7a10-b7e7-63bdcc9b734a
detox[31936] DEBUG: [DetoxServer.js/LOGIN_SUCCESS] role=tester, sessionId=912056cf-c6f0-7a10-b7e7-63bdcc9b734a
detox[31936] TRACE: [AsyncWebSocket.js/WEBSOCKET_MESSAGE] {"type":"loginSuccess","params":{"sessionId":"912056cf-c6f0-7a10-b7e7-63bdcc9b734a","role":"tester"},"messageId":0}

detox[31936] DEBUG: [exec.js/EXEC_CMD, #0] /Users/lydiagolland/Library/Android/sdk/emulator/emulator -list-avds --verbose
detox[31936] TRACE: [exec.js/EXEC_SUCCESS, #0] Nexus_6_API_28
Pixel_2_API_29

detox[31936] DEBUG: [exec.js/EXEC_CMD, #1] /Users/lydiagolland/Library/Android/sdk/platform-tools/adb  devices
detox[31936] DEBUG: [exec.js/EXEC_SUCCESS, #1] List of devices attached
FPMNW18726002003	device
emulator-5554	device
emulator-5556	device

detox[31936] TRACE: [EmulatorTelnet.js/TELNET_CONNECTING] port: 5554, host: localhost
detox[31936] TRACE: [EmulatorTelnet.js/TELNET_CONNECTING] port: 5556, host: localhost
detox[31936] DEBUG: [exec.js/EXEC_CMD, #2] /Users/lydiagolland/Library/Android/sdk/platform-tools/adb -s emulator-5556 shell "getprop dev.bootcomplete"
detox[31936] TRACE: [exec.js/EXEC_SUCCESS, #2] 1

detox[31936] TRACE: [ArtifactsManager.js/LIFECYCLE] artifactsManager.onBootDevice({ coldBoot: false, deviceId: 'emulator-5556' })
detox[31936] DEBUG: [exec.js/EXEC_CMD, #3] /Users/lydiagolland/Library/Android/sdk/platform-tools/adb -s emulator-5556 shell "getprop ro.build.version.sdk"
detox[31936] TRACE: [exec.js/EXEC_SUCCESS, #3] 28

detox[31936] DEBUG: [exec.js/EXEC_CMD, #4] /Users/lydiagolland/Library/Android/sdk/platform-tools/adb -s emulator-5556 shell "dumpsys power | grep \"^[ ]*m[UW].*=\""
detox[31936] TRACE: [exec.js/EXEC_SUCCESS, #4]   mWakefulness=Awake
  mWakefulnessChanging=false
  mWakeLockSummary=0x0
  mUserActivitySummary=0x1
  mWakeUpWhenPluggedOrUnpluggedConfig=false
  mWakeUpWhenPluggedOrUnpluggedInTheaterModeConfig=false
  mUserActivityTimeoutOverrideFromWindowManager=-1
  mUserInactiveOverrideFromWindowManager=false

detox[31936] DEBUG: [exec.js/EXEC_CMD, #5] /Users/lydiagolland/Library/Android/sdk/build-tools/29.0.1/aapt dump badging "/Users/lydiagolland/Projects/wingsy-rn-poc/packages/mobile/android/app/build/outputs/apk/release/app-release.apk" | grep -e "package: name="
detox[31936] TRACE: [exec.js/EXEC_SUCCESS, #5] package: name='com.XXX' versionCode='1' versionName='1.0.4' platformBuildVersionName='1.0.4' compileSdkVersion='28' compileSdkVersionCodename='9'

detox[31936] TRACE: [ArtifactsManager.js/LIFECYCLE] artifactsManager.onBeforeUninstallApp({ deviceId: 'emulator-5556', bundleId: 'com.XXX' })
detox[31936] DEBUG: [exec.js/EXEC_CMD, #6] /Users/lydiagolland/Library/Android/sdk/platform-tools/adb -s emulator-5556 shell "pm list packages com.XXX"
detox[31936] TRACE: [exec.js/EXEC_SUCCESS, #6] package:com.XXX.test
package:com.XXX

detox[31936] DEBUG: [exec.js/EXEC_CMD, #7] /Users/lydiagolland/Library/Android/sdk/platform-tools/adb -s emulator-5556 uninstall com.XXX
detox[31936] TRACE: [exec.js/EXEC_SUCCESS, #7] Success

detox[31936] DEBUG: [exec.js/EXEC_CMD, #8] /Users/lydiagolland/Library/Android/sdk/platform-tools/adb -s emulator-5556 shell "pm list packages com.XXX.test"
detox[31936] TRACE: [exec.js/EXEC_SUCCESS, #8] package:com.XXX.test

detox[31936] DEBUG: [exec.js/EXEC_CMD, #9] /Users/lydiagolland/Library/Android/sdk/platform-tools/adb -s emulator-5556 uninstall com.XXX.test
detox[31936] TRACE: [exec.js/EXEC_SUCCESS, #9] Success

detox[31936] DEBUG: [exec.js/EXEC_CMD, #10] /Users/lydiagolland/Library/Android/sdk/platform-tools/adb -s emulator-5556 install -r -g -t "/Users/lydiagolland/Projects/wingsy-rn-poc/packages/mobile/android/app/build/outputs/apk/release/app-release.apk"
detox[31936] TRACE: [exec.js/EXEC_SUCCESS, #10] Performing Streamed Install
Success

detox[31936] DEBUG: [exec.js/EXEC_CMD, #11] /Users/lydiagolland/Library/Android/sdk/platform-tools/adb -s emulator-5556 install -r -g -t "/Users/lydiagolland/Projects/wingsy-rn-poc/packages/mobile/android/app/build/outputs/apk/androidTest/release/app-release-androidTest.apk"
detox[31936] TRACE: [exec.js/EXEC_SUCCESS, #11] Performing Streamed Install
Success

detox[31936] TRACE: [ArtifactsManager.js/LIFECYCLE] artifactsManager.onBeforeTerminateApp({ deviceId: 'emulator-5556', bundleId: 'com.XXX' })
detox[31936] DEBUG: [exec.js/EXEC_CMD, #12] /Users/lydiagolland/Library/Android/sdk/platform-tools/adb -s emulator-5556 shell "am force-stop com.XXX"
detox[31936] TRACE: [exec.js/EXEC_SUCCESS, #12]
detox[31936] TRACE: [ArtifactsManager.js/LIFECYCLE] artifactsManager.onBeforeLaunchApp({ deviceId: 'emulator-5556',
  bundleId: 'com.XXX',
  launchArgs:
   { detoxServer: 'ws://localhost:51839',
     detoxSessionId: '912056cf-c6f0-7a10-b7e7-63bdcc9b734a' } })
detox[31936] DEBUG: [exec.js/EXEC_CMD, #13] /Users/lydiagolland/Library/Android/sdk/platform-tools/adb -s emulator-5556 shell "date +\"%m-%d %T.000\""
detox[31936] TRACE: [exec.js/EXEC_SUCCESS, #13] 08-09 16:46:51.000

detox[31936] DEBUG: [exec.js/EXEC_CMD, #14] /Users/lydiagolland/Library/Android/sdk/platform-tools/adb -s emulator-5556 shell "pm list instrumentation"
detox[31936] TRACE: [exec.js/EXEC_SUCCESS, #14] instrumentation:com.XXX.test/androidx.test.runner.AndroidJUnitRunner (target=com.XXX)
instrumentation:org.chromium.webview_shell/.WebViewLayoutTestRunner (target=org.chromium.webview_shell)

detox[31936] DEBUG: [exec.js/SPAWN_CMD, #15] [pid=31963] /Users/lydiagolland/Library/Android/sdk/platform-tools/adb -s emulator-5556 shell am instrument -w -r -e detoxServer ws://localhost:51839 -e detoxSessionId 912056cf-c6f0-7a10-b7e7-63bdcc9b734a -e debug ZmFsc2U= com.XXX.test/androidx.test.runner.AndroidJUnitRunner
detox[31936] TRACE: [exec.js/SPAWN_STDOUT, #15] INSTRUMENTATION_STATUS: class=com.XXX.DetoxTest
INSTRUMENTATION_STATUS: current=1
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: numtests=1
INSTRUMENTATION_STATUS: stream=
com.XXX.DetoxTest:
INSTRUMENTATION_STATUS: test=runDetoxTests

detox[31936] TRACE: [exec.js/SPAWN_STDOUT, #15] INSTRUMENTATION_STATUS_CODE: 1

detox[31936] DEBUG: [exec.js/EXEC_CMD, #16] /Users/lydiagolland/Library/Android/sdk/platform-tools/adb -s emulator-5556 shell "ps | grep \"com\.XXX$\""
detox[31936] TRACE: [exec.js/EXEC_SUCCESS, #16] u0_a99        7107  1791 1471508 134216 0                   0 S com.XXX

detox[7107] TRACE: [ArtifactsManager.js/LIFECYCLE] artifactsManager.onLaunchApp({ deviceId: 'emulator-5556',
  bundleId: 'com.XXX',
  launchArgs:
   { detoxServer: 'ws://localhost:51839',
     detoxSessionId: '912056cf-c6f0-7a10-b7e7-63bdcc9b734a' },
  pid: 7107 })
detox[31936] TRACE: [AsyncWebSocket.js/WEBSOCKET_SEND] {"type":"isReady","params":{},"messageId":-1000}
detox[31936] TRACE: [DetoxServer.js/MESSAGE] role=tester action=isReady (sessionId=912056cf-c6f0-7a10-b7e7-63bdcc9b734a)
detox[31936] DEBUG: [DetoxServer.js/CANNOT_FORWARD] role=testee not connected, cannot fw action (sessionId=912056cf-c6f0-7a10-b7e7-63bdcc9b734a)

Example: should successfully sign up user with phone number
Example: should successfully sign up user with phone number [FAIL]
Example: should successfully sign up user with facebook
Example: should successfully sign up user with facebook [FAIL]

 FAIL  packages/mobile/e2e/tests/onboarding.spec.js (121.196s)
  Example
    ✕ should successfully sign up user with phone number (18ms)
    ✕ should successfully sign up user with facebook (3ms)

  ● Example › should successfully sign up user with phone number

    Timeout - Async callback was not invoked within the 120000ms timeout specified by jest.setTimeout.Error: Timeout - Async callback was not invoked within the 120000ms timeout specified by jest.setTimeout.

      at mapper (../../../node_modules/@jest/core/node_modules/jest-jasmine2/build/queueRunner.js:25:45)

  ● Example › should successfully sign up user with phone number

    TypeError: _detox.device.launchApp is not a function

      58 |
      59 |   it('should successfully sign up user with phone number', async () => {
    > 60 |     await device.launchApp();
         |                  ^

      at launchApp (tests/onboarding.spec.js:60:18)
      at tryCatch (../../../node_modules/@babel/runtime/node_modules/regenerator-runtime/runtime.js:45:40)
      at Generator.invoke [as _invoke] (../../../node_modules/@babel/runtime/node_modules/regenerator-runtime/runtime.js:271:22)
      at Generator.prototype.(anonymous function) [as next] (../../../node_modules/@babel/runtime/node_modules/regenerator-runtime/runtime.js:97:21)
      at tryCatch (../../../node_modules/@babel/runtime/node_modules/regenerator-runtime/runtime.js:45:40)
      at invoke (../../../node_modules/@babel/runtime/node_modules/regenerator-runtime/runtime.js:135:20)
      at ../../../node_modules/@babel/runtime/node_modules/regenerator-runtime/runtime.js:170:11
      at callInvokeWithMethodAndArg (../../../node_modules/@babel/runtime/node_modules/regenerator-runtime/runtime.js:169:16)
      at AsyncIterator.enqueue (../../../node_modules/@babel/runtime/node_modules/regenerator-runtime/runtime.js:192:13)
      at AsyncIterator.prototype.(anonymous function) [as next] (../../../node_modules/@babel/runtime/node_modules/regenerator-runtime/runtime.js:97:21)
      at Object.<anonymous>.exports.async (../../../node_modules/@babel/runtime/node_modules/regenerator-runtime/runtime.js:216:14)
      at Object._callee (tests/onboarding.spec.js:59:60)

  ● Example › should successfully sign up user with facebook

    Timeout - Async callback was not invoked within the 120000ms timeout specified by jest.setTimeout.Error: Timeout - Async callback was not invoked within the 120000ms timeout specified by jest.setTimeout.

      at mapper (../../../node_modules/@jest/core/node_modules/jest-jasmine2/build/queueRunner.js:25:45)

  ● Example › should successfully sign up user with facebook

    ReferenceError: element is not defined

      50 |   };
      51 |
    > 52 |   const continueWithFacebook = async () => {
         |                                ^
      53 |     await element(by.text('Log in / Sign up')).tap();
      54 |     await element(by.text('Continue with Facebook')).tap();
      55 |     await expect(element(by.text('Welcome Lydia!'))).toBeVisible();

      at continueWithFacebook$ (tests/onboarding.spec.js:52:32)
      at tryCatch (../../../node_modules/@babel/runtime/node_modules/regenerator-runtime/runtime.js:45:40)
      at Generator.invoke [as _invoke] (../../../node_modules/@babel/runtime/node_modules/regenerator-runtime/runtime.js:271:22)
      at Generator.prototype.(anonymous function) [as next] (../../../node_modules/@babel/runtime/node_modules/regenerator-runtime/runtime.js:97:21)
      at tryCatch (../../../node_modules/@babel/runtime/node_modules/regenerator-runtime/runtime.js:45:40)
      at invoke (../../../node_modules/@babel/runtime/node_modules/regenerator-runtime/runtime.js:135:20)
      at ../../../node_modules/@babel/runtime/node_modules/regenerator-runtime/runtime.js:170:11
      at callInvokeWithMethodAndArg (../../../node_modules/@babel/runtime/node_modules/regenerator-runtime/runtime.js:169:16)
      at AsyncIterator.enqueue (../../../node_modules/@babel/runtime/node_modules/regenerator-runtime/runtime.js:192:13)
      at AsyncIterator.prototype.(anonymous function) [as next] (../../../node_modules/@babel/runtime/node_modules/regenerator-runtime/runtime.js:97:21)
      at Object.<anonymous>.exports.async (../../../node_modules/@babel/runtime/node_modules/regenerator-runtime/runtime.js:216:14)
      at continueWithFacebook (tests/onboarding.spec.js:52:32)
      at continueWithFacebook (tests/onboarding.spec.js:69:11)
      at tryCatch (../../../node_modules/@babel/runtime/node_modules/regenerator-runtime/runtime.js:45:40)
      at Generator.invoke [as _invoke] (../../../node_modules/@babel/runtime/node_modules/regenerator-runtime/runtime.js:271:22)

Jest did not exit one second after the test run has completed.

This usually means that there are asynchronous operations that weren't stopped in your tests. Consider running Jest with `--detectOpenHandles` to troubleshoot this issue.

I am seeing app hanging on launch for both iOS and Android with RN 0.60.4

Hi, out of interest what happens if you try downgrading to v12 as in #1545

@henrymoulton Thanks for your suggestion but sadly it doesn't work on 12.11.3 either, also my issue is only on android

Sifted through the logs and figured this out (for my app, anyway). In my case, I have a few different build configs (aside from debug and release). One is called releaseStaging, which is basically just as it sounds: A release build but with some internal configs for my organization's staging version of the app.
In AndroidManifest.xml I have:
<application
    ...
    android:usesCleartextTraffic="${uses_clear_text_traffic}"

And in app/build.gradle:
buildTypes {
    ...
    releaseStaging {
        ...
        manifestPlaceholders = [
            uses_clear_text_traffic: "true",
        ]
    }
}

Before, it was set to uses_clear_text_traffic: "true". This all makes sense because, at the end of the day, the local detox server.... needs to communicate with the app. Shaking my head at myself for taking so long to come to this conclusion, but happy to be on my way.
It's important to note that for a TRUE release build, you will definitely want             uses_clear_text_traffic: "true" (unless you're being forced to make http requests or something), so I'd recommend setting up a different buildConfig for, at the very least, testing a build made for release.
Hope it helps!

Thanks for your response @wkoutre. I tried adding uses_clear_text_traffic: "true" to my main manifest and doing a build to test but no difference for me unfortunately.

I am facing the same issue. Anyone found the solution ?

me too, Anyone found the solution ? @LydGol90

@LydGol90 still running Android test in debug mode?

@henrymoulton Yeah, not really had anymore time to look into it recently. I am updating react native to 0.61.1  so then might try updating detox and having another go when I have the time

I have managed to solve this issue. Turns out it was related to these two issues: https://github.com/wix/Detox/issues/1297, https://github.com/wix/Detox/issues/1450.
If you set the targetSdkVersion to 27 it all works fine 🤷♀
Will close this issue in favour of the above


Add an overload for ToAsyncEnumerable that takes a custom TaskScheduler

Current implementation of AsyncEnumerable.ToAsyncEnumerable<TSource>(this IEnumerable<TSource> source) does not allow you to pass a custom TaskScheduler, thus, it provides no control on how the tasks are scheduled.
It would be nice to have an overload that takes a TaskScheduler (or maybe even a TaskFactory<bool>) as a parameter.

@krs500 can you give an example of how this is hard to control/test? For the most part we're using Task.FromResult (at least in 3.1) so I'd love to understand more about the problems you're facing and see what the net effect of this overload will be...

I just realized that in v3.1.0, the ToAsyncEnumerable behaves in a very different way than in v3.0.0. And I am stuck to v3.0.0, because v3.1.0 targets .NET Standard instead of the full .NET Framework (see #295).
In v3.0.0, it uses Task.Factory.StartNew to invoke IEnumerator.MoveNext asynchronously. In v3.1.0, the IEnumerator.MoveNext is invoked synchronously, in a blocking way. This negates all the benefits of IAsyncEnumerable, but that's a different question...
You're right that it makes no sense to use schedulers for Task.FromResult, since it returns a task that is already completed.
In my scenario, I use a custom task scheduler that queues the tasks to run on a specific "working" thread, instead of the .NET thread pool (which is a default behavior of Task.Factory.StartNew when you don't specify a scheduler). Others might want to use a custom scheduler, for example, to target a specific SynchronizationContext, to limit the concurrency level, to target the the I/O thread pool, etc.

cc @onovotny for feels on ☝

@krs500 I responded here...
https://github.com/Reactive-Extensions/Rx.NET/issues/295#issuecomment-261266468 Does that help?

Although NETStandard.Library is fully supported for net45, as you say, I can't afford shipping the whole NETStandard.Library along with my desktop app, especially when it is just a subset of the full .NET, from the API perspective. Why I can't reuse the .NET Framework that is already installed? But this is not the question that I'm asking.
As for the questions posted above, no, that doesn't help.

Closing due to age. Please re-open based on the 4.0 preview behavior and functionality.


Use IML cli inside iml-timer

The iml-timer service currently uses it's own internal CLI for stratagem
scans instead of reusing the iml cli.
If we follow this pattern, it will mean we need to create a separate
binary for each service we want to run as a timer inside of docker.
Instead, we should embed the iml cli inside the container and call it
the same way we do outside the container.
This will allow us to write a single cli command for running snapshot
commands instead of having to maintain two commands.
Signed-off-by: Joe Grund<EMAIL_ADDRESS>

This change is

I've been testing this on a cluster and it looks like everything is working as expected. I ssh'd into the docker iml-timer container and was able to schedule an interval.
[root@iml-timer /]# iml <service> interval add --filesystem fs --interval 2m
✔ Configuring <service> scan interval successful

[root@iml-timer /]# systemctl list-timers --all
NEXT                         LEFT         LAST                         PASSED    UNIT                         ACTIVATE
S
Tue 2020-09-01 20:32:49 UTC  1min 9s left n/a                          n/a       iml-<service>-1.timer        iml-<service>-1.service
Wed 2020-09-02 20:07:43 UTC  23h left     Tue 2020-09-01 20:07:43 UTC  23min ago systemd-tmpfiles-clean.timer systemd-
tmpfiles-clean.service
n/a                          n/a          n/a                          n/a       systemd-readahead-done.timer systemd-
readahead-done.service

I've been testing this on a cluster and it looks like everything is working as expected.

Does it work as expected when you schedule it via the GUI? And does everything work correctly when the timer fires?

The test I just completed involved scheduling the timer using the iml cli inside the iml-timer container. I scheduled the timer after creating a variety of different sized files on the mounted client. After the interval completed I checked the fs detail page and the charts had populated as expected. I will now remove the timer using the cli and then add the timer back using the gui.

Removing the timer:
[root@iml-timer /]# iml <service> interval remove --filesystem fs
✔ Remove <service> for the given filesystem successful
[root@iml-timer /]# systemctl list-timers --all
NEXT                         LEFT     LAST                         PASSED    UNIT                         ACTIVATES
Wed 2020-09-02 20:07:43 UTC  23h left Tue 2020-09-01 20:07:43 UTC  33min ago systemd-tmpfiles-clean.timer systemd-tmpf
iles-clean.service
n/a                          n/a      n/a                          n/a       systemd-readahead-done.timer systemd-read
ahead-done.service

After the interval was removed I waited several minutes to make sure the data didn't update. I then removed some of the files and scheduled a new interval using the gui. After the interval ticket the charts reflected the new change. Checking the timer on the iml-timer container:
[root@iml-timer /]# systemctl list-timers --all
NEXT                         LEFT     LAST                         PASSED       UNIT                         ACTIVATES
Tue 2020-09-01 21:08:50 UTC  31s left Tue 2020-09-01 21:06:50 UTC  1min 28s ago iml-<service>-2.timer        iml-<service>-2.service
Wed 2020-09-02 20:07:43 UTC  22h left Tue 2020-09-01 20:07:43 UTC  1h 0min ago  systemd-tmpfiles-clean.timer systemd-t
mpfiles-clean.service
n/a                          n/a      n/a                          n/a          systemd-readahead-done.timer systemd-r
eadahead-done.service

3 timers listed.


Fix for switch not compiling when inserting a compound statement

Switch statements currently have the following quirkiness:
/*
  Function computing the sum of an array
*/

int sum(int* a, int len) {
  int i, sum = 0;
  switch (i) {
    case 0: {
      printf("Hello World");
      break;
    }
    case 1: {
      printf("Bye bye!");
      break;
    }
  }
  return sum;
}

Results in output:
#!/bin/sh
set -e -u

comp_statement: break not in loop

However, if I now try to omit the break:
/*
  Function computing the sum of an array
*/

int sum(int* a, int len) {
  int i, sum = 0;
  switch (i) {
    case 0: {
      printf("Hello World");
    }
    case 1: {
      printf("Bye bye!");
    }
  }
  return sum;
}

I get the expected
#!/bin/sh
set -e -u

comp_statement: case must be at the beginning of a switch block, and each block must end with a break or return statement

However, compound statements are perfectly legal in C. I understand a limitation is that you can't declare variables in lexical blocks, so there's no real point AFAICT to using scopes in cases, but
#include <stdio.h>
int main() {
  int i = 0;
  {
    printf("%d\n", i);
  }
  return 0;
}

compiles fine, so I don't expect there was a purposeful intent in forbidding the previous form.
I fixed the issue by inserting a special case to parse "top level" lexical blocks. It allows inner breaks to also compile as gcc does as well.

Regarding early breaks, maybe we could detect them and when necessary place the whole conditional block in a while :; do {statements} ; break; done; (same syntax as do...while(); statements). One downside is that it makes the shell code harder to read but maybe with some alias we can have a custom syntax that makes it clear that the loop is only to support early breaks.

Ok, so I copied the if code into the switch statement block and captured the termination from the branches.
This way, the program
int main() {
  int i = 0;

  switch (i) {
    case 0: {
      if (i == 0) {
        break;
      } else if (i >= 0) {
        printf("First statement");
        break;
        printf("Second statement");
      } else {
        printf("Third statement");
        break;
      }
    }
  }

  return 0;
}

Compiles into
#!/bin/sh
set -e -u -f
LC_ALL=C

: $((i = 0))
_main() {
  let i
  i=0
  case $i in
    0)
      if [ $i = 0 ] ; then
      elif [ $i -ge 0 ] ; then
        printf "First statement"
      else
        printf "Third statement"
      fi
    ;;
  esac
  : $(($1 = 0))
  endlet $1 i
}

# Runtime library
# Local variables
__=0
__SP=0
let() { # $1: variable name, $2: value (optional)
  : $((__SP += 1)) $((__$__SP=$1)) # Push
  : $(($1=${2-0}))                 # Init
}
endlet() { # $1: return variable
           # $2...: function local variables
  __ret=$1 # Don't overwrite return value
  : $((__tmp = $__ret))
  while [ $# -ge 2 ]; do
    : $(($2 = __$__SP)) $((__SP -= 1)); # Pop
    shift;
  done
  : $(($__ret=__tmp))   # Restore return value
}

__code=0; # Success exit code
_main __code; exit $__code

Which is exactly your example. I was not able to reproduce a case where break are output directly into the code, leading to a runtime error. This might be because there are now no way to insert a break statement in a statement grouping without going over the if (get_op(node) == BREAK_KW) in comp_switch_block_statement.
Now because the codegen for statements in a compound group (of any kind) are done through comp_switch_block_statements we have the nice property that as soon as provable break is encountered, the generation stops. This means an exhaustive if will prevent what comes after to be compiled as well:
Compiling
#include <stdio.h>
int main() {
  int i = 0;
  switch (i) {
    case 0: {

      if (i == 0) {
        printf("Statement 1");
        break;
      } else {
        printf("Statement 2");
        break;
      }

      if (i == 0) {
        printf("Statement 3");
        break;
      } else {
        printf("Statement 4");
        break;
      }
    }
  }
  return 0;
}

yields
#!/bin/sh
set -e -u -f
LC_ALL=C

: $((i = 0))
_main() {
  let i
  i=0
  case $i in
    0)
      if [ $i = 0 ] ; then
        printf "Statement 1"
      else
        printf "Statement 2"
      fi
    ;;
  esac
  : $(($1 = 0))
  endlet $1 i
}

# Runtime library
# Local variables
__=0
__SP=0
let() { # $1: variable name, $2: value (optional)
  : $((__SP += 1)) $((__$__SP=$1)) # Push
  : $(($1=${2-0}))                 # Init
}
endlet() { # $1: return variable
           # $2...: function local variables
  __ret=$1 # Don't overwrite return value
  : $((__tmp = $__ret))
  while [ $# -ge 2 ]; do
    : $(($2 = __$__SP)) $((__SP -= 1)); # Pop
    shift;
  done
  : $(($__ret=__tmp))   # Restore return value
}

__code=0; # Success exit code
_main __code; exit $__code

Which is an extension of what we have above.
This leaves the case of non-provable breaks, such as a single if:
#include <stdio.h>
int main() {
  int i = 0;
  switch (i) {
    case 0: {

      if (i == 0) {
        printf("Statement 1");
        break;
      }

      printf("Statement 2");
      break;
    }
  }
  return 0;
}

Yields
#!/bin/sh
set -e -u -f
LC_ALL=C

: $((i = 0))
_main() {
  let i
  i=0
  case $i in
    0)
      if [ $i = 0 ] ; then
        printf "Statement 1"
      fi
      printf "Statement 2"
    ;;
  esac
  : $(($1 = 0))
  endlet $1 i
}

# Runtime library
# Local variables
__=0
__SP=0
let() { # $1: variable name, $2: value (optional)
  : $((__SP += 1)) $((__$__SP=$1)) # Push
  : $(($1=${2-0}))                 # Init
}
endlet() { # $1: return variable
           # $2...: function local variables
  __ret=$1 # Don't overwrite return value
  : $((__tmp = $__ret))
  while [ $# -ge 2 ]; do
    : $(($2 = __$__SP)) $((__SP -= 1)); # Pop
    shift;
  done
  : $(($__ret=__tmp))   # Restore return value
}

__code=0; # Success exit code
_main __code; exit $__code

This last case is where we would want to insert an early break mechanism of sort. I think a succinct way is to xor termination_lhs and termination_rhs. We could explore the tree and generate the surrounding while code afterwards, since we can't retroactively add it.

We could explore the tree and generate the surrounding while code afterwards, since we can't retroactively add it.

We can certainly do that! :) See undo_glo_decls and replay_glo_decls_inline which allows the generated code to be moved and append_glo_decl_fixup which can be used to reserve space to be filled later.

We can certainly do that! :) See undo_glo_decls and replay_glo_decls_inline which allows the generated code to be moved and append_glo_decl_fixup which can be used to reserve space to be filled later.

Interesting! I'll take a look. That would be much more efficient than my solution

Regarding early breaks, maybe we could detect them and when necessary place
the whole conditional block in a while :; do {statements} ; break; done;

FWIW, I think we should focus on the easy cases and try to support more
complex cases (such as early breaks) only after we bumped into code
that not only uses them but is cumbersome to change.


After restart service, 500 internal server error occurs repeatedly for more than 10 minutes

Environment：
Install: self hosted (docker Deployment)
version: 8.0
Number of users: about 500 people, 200 of which are online at the same time
Reproduction step:
When I restart zulip services bysupervisorctl restart all or su zulip sh -c /home/zulip/deployments/current/scripts/restart-server, All services will in the running state soon, the web page is also displayed normally. But every two minutes there will be 500 errors on the page. It usually takes more than 10 minutes, even half an hour  to recover automatically.

Temporary solution:
If I manually stop the zulip container for 5 minutes and start it up again, the problem no longer occurs，but users will not be able to use it during that time.
Logs
I  did not see any error messages in /var/log/zulip/errors.log  and /var/log/zulip/django.log, only /var/log/nginx/access.log reports 502 errors，as follows:
<IP_ADDRESS> - - [12/Oct/2024:14:09:36 +0000] "POST /json/users/me/presence HTTP/2.0" 502 2420 "https://test.zulip.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36" test.zulip.com 0.000
<IP_ADDRESS> - - [12/Oct/2024:14:09:36 +0000] "GET /json/messages?anchor=3726040&num_before=0&num_after=1000&client_gravatar=true HTTP/2.0" 502 2420 "https://test.zulip.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36" test.zulip.com 0.000
<IP_ADDRESS> - - [12/Oct/2024:14:09:36 +0000] "GET / HTTP/2.0" 502 2420 "https://test.zulip.com" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36" test.zulip.com 0.000
<IP_ADDRESS> - - [12/Oct/2024:14:09:36 +0000] "GET /user_avatars/2/e15b7c03f6f9ce87e3e194c3502e9e6399157044.png?version=2&s=50 HTTP/2.0" 502 2420 "https://test.zulip.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36" test.zulip.com 0.000

Zulip Server and web app version:

[ ] Zulip Server 8.x

sorry, I fogot paste the /var/log/nginx/err.log
2024/10/12 14:09:36 [error] 131707#131707: *789 connect() to unix:/home/zulip/deployments/uwsgi-socket failed (11: Resource temporarily unavailable) while connecting to upstream, client: <IP_ADDRESS>, server: , request: "POST /json/users/me/presence HTTP/2.0", upstream: "uwsgi://unix:/home/zulip/deployments/uwsgi-socket:", host: "test.zulip.com", referrer: "https://test.zulip.com"
2024/10/12 14:09:36 [error] 131707#131707: *2066 connect() to unix:/home/zulip/deployments/uwsgi-socket failed (11: Resource temporarily unavailable) while connecting to upstream, client: <IP_ADDRESS>, server: , request: "GET /json/messages?anchor=3726040&num_before=0&num_after=1000&client_gravatar=true HTTP/2.0", upstream: "uwsgi://unix:/home/zulip/deployments/uwsgi-socket:", host: "test.zulip.com", referrer: "https://test.zulip.com/"

That is a very long time and definitely not expected.
This sounds like your system doesn't quite have enough CPU or memory to happily restart the processes.
I would watch supervisorctl status and /var/log/zulip/*.log after the upgrade to see if the timestamps or logs show processes being OOM killed and restarted repeatedly.

Also consider upgrading to 9.x; there are some memory usage improvements in that release.

That is a very long time and definitely not expected.
This sounds like your system doesn't quite have enough CPU or memory to happily restart the processes.
I would watch supervisorctl status and /var/log/zulip/*.log after the upgrade to see if the timestamps or logs show processes being OOM killed and restarted repeatedly.

However, after restart,  there is no oom in the log. When I use supervisorctl status all to check status, all processes  are running , the uptime are the same, none of them have been restarted.
machine configuration: 16C16G

Certainly that should be plenty of resources. It kinda sounds like you don't have enough uwsgi processes for your load. The defaults were increased in the 8.2 release; as noted above, I suggest you upgrade to the latest 8.x at least, and ideally to the latest 9.x; it'll likely fix this problem.

Certainly that should be plenty of resources. It kinda sounds like you don't have enough uwsgi processes for your load. The defaults were increased in the 8.2 release; as noted above, I suggest you upgrade to the latest 8.x at least, and ideally to the latest 9.x; it'll likely fix this problem.

When I used version 8.0, changed the number of processes from 6 to 16 in /etc/zulip/uwsgi.ini, and restarted the service, the problem did not recur.
If I use the 9.x version, it doesn't appear again. That solved my problem. Thank you very much.


Remaster changes (identical items from old CRB)

Just in case you guys are still taking information to add to the remaster changes list for item name changes from CRB. Here is a list of items to help others with switches of items from premaster to remaster.
Necklace of Fireballs has changed to the Frozen lave (consumable version) and Eternal Eruption (permanent) items in the GM Core.
Hate of the Magi is now Mage's Hat
Boots of Elvenkind is now Arboreal Boots. There is a slight change to the wording but mechanically they are the same so the item. The wording is changed I think to not just be tried to one ancestry
Cloak of Elvenkind is now Cloak of Illusions
Rod of Wonder became Madcap Top
Arrow-catching Shield is Lodestone Shield
Spell-storing Rune is now Spell Reservoir Rune
Dancing Rune is Animated Rune
Salve of Slipperiness became Tricky Liniment
I know you guys have more important items but just thought I would send some finds.  You guys do amazing work and thank you for all you do.

Hi There,
Thanks again for submitting this info. So that you know, the system still has the following items available:
Boots of Elvenkind, Cloak of Elvenkind, Necklace of Fireballs, Rod of Wonder, and Arrow-catching Shield.
Everything else will be added to the Wiki and the Remaster journal (upon the next release, assuming the change gets approved beforehand).
Thanks again :)


Some property titles don't match the name of the underlying property

Describe the bug
Several of our action properties have titles which don't match the name of the underlying property being set.  This could lead to confusion for developers when dropping to code or trying to read reference docs.  Below is the schema for the BeginDialog action.

Thanks for your valuable feedback. Please see https://github.com/microsoft/BotFramework-Composer/issues/3175.


Add sunjayBhatia to conformance reviewers

What type of PR is this?
/area conformance
What this PR does / why we need it:
Add myself (@sunjayBhatia) to conformance reviewers alias to add to reviewer group as recommended by @robscott.
Which issue(s) this PR fixes:
N/A
Does this PR introduce a user-facing change?:
NONE

Thanks @sunjayBhatia!
/lgtm


draft of new pricing config schema, post and get pricing config from db

Need feedback on the pricing config schema, I have the config in a separate folder to avoid breaking the pipeline for now.

OK I'm going to merge this one for now and we'll adjust the schema if needed with Ben on Monday, after  the IPM meeting.

Sounds good. Thanks!


added german translation

Closing as https://github.com/gitterHQ/gitter-translations/pull/13/files seems more complete.


[New Sample] Bot Streaming in C# for Microsoft Teams

Build Pipeline -
https://github.com/OfficeDev/Microsoft-Teams-Samples/actions/runs/11839526197


Replace all instances of > with Set-Content

Generally we should be making use of Set-Content (or Add-Content) to output data to files. Redirection operators are supported but are more a feature of the console than PowerShell itself, the examples and documentation should be recommending the PowerShell approach over any other method.

Docs Build status updates of commit 724d960:
:clock10: Full build: the average full build time is 14 min(s) 12 sec(s), this is based on the last 45 build(s) for this repository.
:warning: This is a FULL build due to the incremental build being disabled in configuration file (.openpublishing.publish.config.json in 'live' branch). For more detailed info, please click here.

Docs Build status updates of commit 724d960:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

exchange/exchange-ps/exchange/client-access/Test-CalendarConnectivity.md
:white_check_mark:Succeeded
View (exchange-ps)

exchange/exchange-ps/exchange/client-access/Test-EcpConnectivity.md
:white_check_mark:Succeeded
View (exchange-ps)

exchange/exchange-ps/exchange/client-access/Test-ImapConnectivity.md
:white_check_mark:Succeeded
View (exchange-ps)

exchange/exchange-ps/exchange/client-access/Test-PopConnectivity.md
:white_check_mark:Succeeded
View (exchange-ps)

exchange/exchange-ps/exchange/client-access/Test-PowerShellConnectivity.md
:white_check_mark:Succeeded
View (exchange-ps)

exchange/exchange-ps/exchange/client-access/Test-WebServicesConnectivity.md
:white_check_mark:Succeeded
View (exchange-ps)

exchange/exchange-ps/exchange/devices/Test-ActiveSyncConnectivity.md
:white_check_mark:Succeeded
View (exchange-ps)

exchange/exchange-ps/exchange/mail-flow/Get-SystemMessage.md
:white_check_mark:Succeeded
View (exchange-ps)

exchange/exchange-ps/exchange/mail-flow/New-SystemMessage.md
:white_check_mark:Succeeded
View (exchange-ps)

exchange/exchange-ps/exchange/mail-flow/Set-SystemMessage.md
:white_check_mark:Succeeded
View (exchange-ps)

exchange/exchange-ps/exchange/mailboxes/Get-CalendarDiagnosticAnalysis.md
:white_check_mark:Succeeded
View (exchange-ps)

exchange/exchange-ps/exchange/mailboxes/Get-CalendarDiagnosticLog.md
:white_check_mark:Succeeded
View (exchange-ps)

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


added loci

Pull Request Test Coverage Report for Build 387

141 of 142 (99.3%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.2%) to 95.21%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

pyod/test/test_loci.py
82
83
98.8%

Totals

Change from base Build 384:
0.2%

Covered Lines:
2743

Relevant Lines:
2881

💛  - Coveralls

A newer version is merged. This one is obsolete.


Changing width and height for js-modern-calendar

Hi,
I am using these samples and they are great. In regards to the js-modern-calendar web part, wanted to see if width and height can be minimized, it seems to occupying lot of space on modern page. If its possible,  can someone please mention where I need to modify the width-height parameters in the code, thanks in advance.

@mdev23  did you ever get the solution to this issue?
I'm closing issues that haven't had any activity for a while. If you found the solution to this, feel free to add a comment below. If you still have the issue, feel free to re-open it and we'll see if members of the community can help.


Uncaught Error: buffer is either not set or not loaded (Tone.Sampler)

Hi,
I have recently been encountering the issue "Uncaught Error: buffer is either not set or not loaded" on a drum machine app I am building using Tone.Sampler. I am using Vercel to deploy my app and I am storing my audio samples client-side. Everything was working great until I tried to redeploy with my Node back-end. All of a sudden it seems that the audio samples won't buffer (although it works on my local network). Under the Chrome Network tab you can see it making requests for the audio files however it seems to be returning my Index.html file instead of the audio file. You can view a working version of the app here:
https://tr-bot.vercel.app/trbot
And this is the newest version where I am encountering the issue.
https://tr-bot-project.vercel.app/trbot

Is there a minimal example code that you could provide? It seems like most likely the file isn't loaded by the time you're trying to play it. Make sure you wait for the Tone.Sampler's onload callback to resolve before trying to trigger any notes.

Here is the Sampler which I export as a variable into my App component in React.
``export const allSampler =
new Tone.Sampler({
    urls: {
        C4: "trbotkick.mp3",
        D4: "trbotsnare.mp3",
        E4: "trbothh1.mp3",
        F4: "trbothh2.mp3",
        G4: "trbotclap.mp3",
        A4: "trbotperc.mp3",
    },
    onload: function() {console.log('samples loaded')},
    baseUrl: "/audio/",
})

It seems no matter how long I wait the samples won't load. The Network tab in Dev Tools is saying that it is fetching the audio samples but then it returns my index.html file instead of the audio. I haven't had issues with this on previous builds. You can reference the two builds from the links in my initial post if you want to observe the different behaviors.

If you're working with react, make sure you import the links the way that React prefers. I'm not positive, but i think something like this might work:
import Kick from "./audio/trtbotkick.mp3";

new Tone.Sampler({
  C4: Kick
})

Yes this appears to have worked! Thanks for your help!


Avatar not loading on forum

While this seems to have resolved the issue with people who have previously signed up to the forum, it seems like new users avatars are not loading.
See here

Going to ask anyone with this issue to reupload their profile due to resource constraints. It only occurs for a couple of users and it is because they had older profiles.


Delete SAS Token from vault after expiration

Feature Request
This Feature Request aims at delete the SAS Token, once is no longer viable to be used in transfers, to reduce the storage of not needed entries.
Why Is the Feature Desired?
Currently, the SAS Token is created with a fixed timeframe after which the token cannot be used for a transfer (even if it already started).
However, after secret is stored in the respective vault, it is never deleted, even after expiration, so unusable secrets are stored indefinitably.
Solution Proposal
Suggestion would go to delete the SAS Token after a Transfer has been processed (successfully or not). A possible approach would be to delete it during the deprovisioning phase (here), however this has not yet been tested so validity cannot be confirmed.

The delete of the secret token should be already done externally in the TransferProcess state machine by the DeprovisionResponseHandler
this because this behavior should be shared in all the provisioner implementations.
If there's an issue there then I think it could be considered a bug and opened in the Connector repo.

The delete of the secret token should be already done externally in the TransferProcess state machine by the DeprovisionResponseHandler this because this behavior should be shared in all the provisioner implementations.
If there's an issue there then I think it could be considered a bug and opened in the Connector repo.

What we are seeing is that the SAS Token is not being deleted from the vault, so a potential bug is happening. Will update this issue after investigating it. For now will delete and will reopen if needed. Thank you for the feedback, @ndr-brt !