Communicate directly via the app

Current Implementation
Currently, users can find the latest information (e.g. about problems) only through the FAQ.
So f.e. if users under iOS 13.6 wonder why the App is not working, they have to go to the FAQ and find this answer.
If the App would just show an info box on the main screen that this problem is known and solved with iOS 13.6.1, I assume that more users would see this and don't just say the App is not working but upgrade their iOS version.
Suggested Enhancement
I think it would be nice if users would be able to see more informations inside the App via f.e. an info box, etc.
The users should be able to close the box after they read it.
Expected Benefits
It would be much easier for users to get important information, and it could be personalized better (e.g. not showing Android information to someone using iOS)

Related Issues

About showing local regulations: #151
Status page about the App's and the backend's health : #134
Understandable error messages: #112
Report the context of an error: #17

Internal Tracking ID: EXPOSUREAPP-2331

Hello @Ein-Tim,
thanks for sharing your idea! I personally really like it. There is a thread about a similar topic, I like to link here: https://github.com/corona-warn-app/cwa-wishlist/issues/134 this was more about a status board, but similar in some way that both issues are describing ways to inform the users about the latest changes or important information in general.
I will transfer this issue to JIRA so that it gets in the hands of the right people.
Thanks you,
LMM
Corona-Warn-App Open Source Team

Thank you! @GPclips
And thanks for all your quick and nice replies 👍

Related issues:

https://github.com/corona-warn-app/cwa-wishlist/issues/17
https://github.com/corona-warn-app/cwa-wishlist/issues/112

I also really like @vaubaehn's idea here:

Yes, my idea was going into a quite similar direction, I drew a primitive sketch about it -> here, but didn't take time yet for polishing and putting it into the wishlist repo.
In short, I describe a Concept On Viewing important Reliability Information of Device = COVirD (to be pronounced: co-weird) ;)
Sounds like a big thing to implement, but if some very small basics are set up first, it may be extended release by release (quite modular). So, if an error / exception / incompatibility would be detected, a card would show up, and link to in-app information or external FAQ. The 'root' card in main screen could provide genral information, if 'everything works fine' with CWA, and in case of problems it could be the starting point for finding information and solutions.

I second this and like to add something:
pls explain the actual meaning of Timestamps of a Matched Key in "Exposure Checks"-Detail Screen and tell the users, what it really is and what not. Because so many people believe that this timestamp will reflect the timestamp of the concrete Contact! I saw this in several discussions and a simple text label could avoid mis-interpretation here (and save time for those people that have to explain it several times).
see also https://github.com/corona-warn-app/cwa-app-ios/issues/1245 for more information. Thx :)

Something what makes me really sad (there was also a discussion recently in the community slack: https://covidapps.slack.com/archives/C0194ML0MLN/p1608214559132100):
With 1.9.1 CWA updated to version 2 of the ENF.
With that version it is possible that encounters are shown in the log but not in the App anymore. Also after the update user will see a drop in encounters: https://github.com/corona-warn-app/cwa-app-ios/issues/1703. This is very confusing and in combination with the matches shown in the Log, many users will think that this is a bug.
There neither 2 new FAQ entries nor a blog post helps, since the normal user doesn't know about this (since the App does not inform the user that Blog and FAQ does exist... very sad)

@Ein-Tim

There neither 2 new FAQ entries nor a blog post helps, since the normal user doesn't know about this (since the App does not inform the user that Blog and FAQ does exist... very sad)

The FAQ section in the app links to the site of The Federal Government through main_about_link
https://www.bundesregierung.de/corona-warn-app-faq-englisch
https://www.bundesregierung.de/corona-warn-app-faq
The English language page:

prominently displays the telephone number of the technical hotline
refers to version 1.7, not to version 1.9
does not mention the Blog
links to the German-language version of the FAQs https://www.coronawarn.app/de/faq/ (not the English language version) and the link is hidden in some of the individual questions and answers instead of being at the top of the page
was last updated Monday, 19. October 2020

The German language page:

links to the FAQ https://www.coronawarn.app/de/faq/ & Blog https://www.coronawarn.app/de/blog/ at the top of the page
does refer to Version 1.9
says it was last updated on 25. November 2020

So, in summary, the German language page does give the user a route to the Blog and FAQs, even though it is indirect.
The English language page of the Federal Government (https://www.bundesregierung.de/corona-warn-app-faq-englisch) needs some significant improvement. Perhaps the Open Source Team has some direct contact (or indirect contact through RKI) to request some improvement to the English language page? I did try a couple of weeks ago by using the contact form on this site, but I never got any response.
I will add this specifically to the GitHub website issues so it can be passed one-to-one on to the other website owners of the Bundesregierung.

@MikeMcC399

So, in summary, the German language page does give the user a route to the Blog and FAQs, even though it is indirect.

Yes, but be honest: Will the normal user find this? No...

I will add this specifically to the GitHub website issues so it can be passed one-to-one on to the other website owners of the Bundesregierung.

Thank you!

@Ein-Tim

So, in summary, the German language page does give the user a route to the Blog and FAQs, even though it is indirect.

Yes, but be honest: Will the normal user find this? No...

I share your concerns that the level of redirection to find information is too deep.

I will add this specifically to the GitHub website issues so it can be passed one-to-one on to the other website owners of the Bundesregierung.

I have created https://github.com/corona-warn-app/cwa-website/issues/711 now. At the least the available information should be helpful and current.
(BTW: I'm still not seeing CWA 1.9.1 in the Google Play Store.)

This user complains on Twitter about the following:

No information in the app that positive rapid antigen self tests can't be added to the app
and asks the question why these infos are not clearly in the app

@Ein-Tim

No information in the app that positive rapid antigen self tests can't be added to the app

Do you think that the FAQ article https://www.coronawarn.app/en/faq/results/#rat_tan_hotline should be expanded to mention also self-tests?
Also there is no explicit entry in the FAQ for self-tests. They are mentioned in https://www.coronawarn.app/en/faq/results/?search=&topic=glossary#glossary_rapid_antigen_test. Should there be a separate FAQ entry for self-tests (Antigen-Selbsttests)?

@MikeMcC399
Yes, I think it would be good to get this topic covered! Would you like to open an issue in the website repo for this?
However, as this is what the OP of this issue is a request to communicate directly via the app, I must say: It would be better if  this information could be found directly in the app.

@Ein-Tim

Yes, I think it would be good to get this topic covered! Would you like to open an issue in the website repo for this?

https://github.com/corona-warn-app/cwa-website/issues/3000 is now opened (by the way, quite incredible that we reached 3000 issues on the website!)

However, as this is what the OP of this issue is a request to communicate directly via the app, I must say: It would be better if this information could be found directly in the app.

It would be good if the section "Manage Your Tests" also included the information from https://www.coronawarn.app/en/faq/results/#rat_tan_hotline that a positive result from a Rapid Antigen Test, a Rapid PCR Test or a Selftest cannot be shared by TAN. It's essentially the same topic as I raised in https://github.com/corona-warn-app/cwa-documentation/issues/902, so maybe you could add your comment there or open a new topic so that the suggestion does not get lost.

@MikeMcC399
I'll add a comment to the other issue.

As the CWA project went into ramp-down mode, I don't expect this feature to be implemented. I'm therefore closing this issue.


RAM usage

Description
Few seconds after launching the editor, this package makes Atom slowly increasing used RAM until it's consuming about 2,1 GB of memory - while in my case, normally, Atom uses around 700 MB.
This makes whole editor either work amazingly slow, or even stuck at all...

Environment informations
Operating system: Windows 7 Home Premium 64-bit SP1 with 12,0GB RAM
Atom version: 1.23.1
Package version: 1.4.0
Startup time addition: 355ms
Reproduction rate: 100%

That's interesting. I'll make start discord rpc client when the discord exists in the process list.

Any progress on this issue? Still getting it on macOS


Drop Python 2.x support entirely

Pyramda tries to support both Python 2 and 3 in a clever way: it uses pasteurize in setup.py to rewrite the code to be compatible with python 2 while it's being installed. This is a horrible idea and it causes all sorts of problems, like #8 and #28.
Python 2 is being retired globally starting in 2020, so there's not much point to supporting it anymore. Deleting the pasteurize nonsense entirely from setup.py would be a great thing for somebody to do. I doubt I'll get around to it myself, since I don't work in Python professionally anymore.

Hello Jack,
I made the changes to setup.py and .travis.yml and pushed it, Could you please check if it looks fine and merge it.
Build for 3.3 version fails in Travis and it looks like a known issue, i can remove 3.3 from Travis.yml if that is OK with you.
Thanks,
Vijay

@vgangaprasad Thank you! I left a few comments on the pull request. Once those are resolved I will very happily merge it.

Closed by #36 (hooray!)


Carousel design for Scholarship and Inspiring WIT

Description
Created a carousel for the scholarship for WIT and inspiring women in tech screens
https://www.figma.com/file/K6NPMXPPBVfBFf3CHA9nJz/WIT?node-id=1%3A124
Fixes # 27
Type of change

[ ] New feature (non-breaking change which adds functionality)
[ ] This change requires a documentation update

Checklist:

[ ] My contribution follows the style guidelines of this project
[ ] I have performed a self-review of my own contribution
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have checked my code and corrected any misspellings

hi, could you add it in documents -> design -> carousels please.


Error

Hi,
Great paper and thanks for sharing your source code ! It works when using the default input size but I am getting an error when I am trying to use different input sizes. Note that I change the size also in the config file.
Do you have any idea what I am doing wrong ?
Example:
#x = torch.rand((1,2, 160, 192, 224))
x = torch.rand((1,2, 192, 192, 208))
out, flow = model.forward(x)
print(out.size())
print(flow.size())
Error:
x = x.view(B, H // window_size[0], window_size[0], W // window_size[1], window_size[1], L // window_size[2], window_size[2], C)
RuntimeError: shape '[1, 10, 5, 8, 6, 7, 7, 6]' is invalid for input of size 808704

Hi @ppegiosk
Refer to the network architecture diagram below. Because the encoder downsamples the input by a factor of 32, the size of the input must be divisible by 32. As a result, the size 208 is invalid.
Additionally, it is recommended that you configure the window size (for Swin Transformer) to be the same as the input size after 32-fold downsampling. For example, an optimal window size for a 160 x 192 x 192 input would be (160/32, 192/32, 192/32) = (5, 6, 6).
You can set the window size here and the input size here.

Let me know if you have other questions.
Junyu

Thanks a lot for your quick reply. I will have a better look into that.
I have kind of the same problem with your previous work/model vit-v-net. I am getting an error when I tried
the input size mentioned before (192, 192, 208):
Error:
x = torch.cat([x, skip], dim=1)
RuntimeError: Sizes of tensors must match except in dimension 1. Got 12 and 13 in dimension 4 (The offending index is 1)

The same principle applies to ViT-V-Net. Check out the code here and here, where "down_factor" is the downsampling factor of the CNN encoder and "patch_size" is the size of a patch used in the Vision Transformer.
In the default setting, the input size is first downsampled by a factor of 2^down_factor=4 using the CNN encoder, then by a patch size of 8 for self-attention.
Therefore, in total, the image size should be divisible by 4*8=32. As a result, 208 is not a valid size for ViT-V-Net as well. You may zero-pad the image to make it 224.


esp32c3 with embassy support builds, esp32c6 does not

I am trying to get a small sample with embassy running for both the esp32c3 and esp32c6.
The esp32c3 code compiles and runs fine, but the esp32c6 flavour does not.
All I did was replace "esp32c3-hal" with "esp32c6-hal" (and adjust the version number).
I saw no other differences when comparing the esp32c3-hal and esp32c6-hal  modules. Or am I missing something here?
As a minimal working example, see the esp32c3 flavor (working), and the esp32c6 flavour, which fails to build with the following error:
   Compiling embassy-test v0.1.0 (/home/leon/Desktop/ivy/code/embassy-test)
   Compiling esp-hal-common v0.15.0
error[E0599]: no method named `compare_exchange` found for struct `portable_atomic::AtomicU8` in the current scope
  --> /home/leon/.cargo/registry/src/index.crates.io-6f17d22bba15001f/esp-hal-common-0.15.0/src/soc/mod.rs:42:14
   |
41 |           if MAC_OVERRIDE_STATE
   |  ____________-
42 | |             .compare_exchange(0, 1, Ordering::Relaxed, Ordering::Relaxed)
   | |             -^^^^^^^^^^^^^^^^ method not found in `AtomicU8`
   | |_____________|
   |

For more information about this error, try `rustc --explain E0599`.
error: could not compile `esp-hal-common` (lib) due to 1 previous error

Ah, turn's out I missed to change the target from riscv32imc-unknown-none-elf to riscv32imac-unknown-none-elf, my bad.


Modify environment name validation check

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license
Existing environment name check condition allows special characters, however environment name is being used in athena table and other string split (by '-') functions. I modified validation check method and corresponding output messages. That's all.
Below issue has been published. This PR will prevent the user from typing special characters for environment name.
https://github.com/awslabs/aws-saas-boost/issues/231

The downstream effect of the dash character in the environment label is the overly simplistic logic we are using to identify the CodePipeline for a tenant based on meta data available in the eventing system. The proper fix for this isn't to exclude the dash character from the environment label, but to fix the logic that figures out the proper CodePipeline for a tenant. This work is in progress already.

From what I found so far, not only identifying codepipeline, but also identifying athena table name (dash is not allowed in table name). So if your team decided to change the logic, then other downstream effects should be considered and fixed.


Disallow less strict build caches

Your development orb has been published. It will expire in 30 days.
You can preview what this will look like on the CircleCI Orb Registry at the following link:
https://circleci.com/developer/orbs/orb/membraneframework/elixir?version=dev:4a0b30c1e512bae86a6c4e89280251b4aae919fa

Your development orb has been published. It will expire in 30 days.
You can preview what this will look like on the CircleCI Orb Registry at the following link:
https://circleci.com/developer/orbs/orb/membraneframework/elixir?version=dev:40c7e85bf35880a8ef9de1ea3ab97173d20f58f2

Your development orb has been published. It will expire in 30 days.
You can preview what this will look like on the CircleCI Orb Registry at the following link:
https://circleci.com/developer/orbs/orb/membraneframework/elixir?version=dev:cd64fac0fcafb98dacd050b22e7d2c98a9e52691

Your development orb has been published. It will expire in 30 days.
You can preview what this will look like on the CircleCI Orb Registry at the following link:
https://circleci.com/developer/orbs/orb/membraneframework/elixir?version=dev:90f0b8e2ebe077607791fcf9d15e5cfb984ff2a2

Your development orb has been published. It will expire in 30 days.
You can preview what this will look like on the CircleCI Orb Registry at the following link:
https://circleci.com/developer/orbs/orb/membraneframework/elixir?version=dev:e0bfa4687dd1f5bc1ac1ffcc9546d7134c7f264f

Your development orb has been published. It will expire in 30 days.
You can preview what this will look like on the CircleCI Orb Registry at the following link:
https://circleci.com/developer/orbs/orb/membraneframework/elixir?version=dev:e0bfa4687dd1f5bc1ac1ffcc9546d7134c7f264f

Your orb has been published to the CircleCI Orb Registry.
You can view your published orb on the CircleCI Orb Registry at the following link:
https://circleci.com/developer/orbs/orb/membraneframework/elixir?version=1.5.2


Crispr edits 4

More edits for enable:False tutorial. Would be great if someone could approve if tests pass here.

ugh I really need to look into that build failure. It should be fine though.

Thanks again @hexylena !


Remote FTP [Search files and text in files]

Hello, please if possibly, add in Remote FTP package function search files in project and search text in project, or integrate with Atom function search in project.

@Dronax For the text side of things, this would most likely require for all the files to be downloaded so they can be searched through. Meaning there wouldn't be much difference between downloading everything and then using the atom search.


Need to make a variable glossary

Refer to the glossary started in the Overview section.

It also may make sense to create an index.  That way anyone who is interested in when we use the word in the report can find a link or page number.  It think this should be easy to do in Sphinx and will look into it.

@jarrodmillman , I am working on adding variable description. Do you know the meaning of M_AS files under intervals file? I think it may refers to the start and end point of movement in AS. Also, do you know the meaning of C_idx_HB files under txy_coords files.

@HongfeiZhao I'm still unsure what M_AS means. It's one of the questions I was going to ask the Tecott Lab group tomorrow.
As for C_idx_HB--that's a boolean variable that indicates whether the mouse was in the home base or not. To see how all the files in that folder interact, it's helpful to look at/play around with the mousestyles.data.load_intervals() function:
https://github.com/berkeley-stat222/mousestyles/blob/master/mousestyles/data/__init__.py

Thanks, @dylandaniels , it's helpful!

@HongfeiZhao what's the status on this?

I am not sure what variables should I add? It seems there are explanation for 'AS' and 'IS'. Should I give a description for all the data variables like M_AS, C_idx_HB? @dylandaniels

Please close the issue, it has been addressed @pigriver123


workblade and knife cutting saplings into sticks.

I never have enough sticks.

and axes?

makes sense to me, I could use a hatchet for that, and a full axe with a bit more work.


test <IP_ADDRESS>

fixed opt levels everywhere

that PR ready the tree for accepting PR about  postgis / wasi tools / retargeting


Add a README as asciidoc is not recognized by terraform registry

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

New or Affected Resource(s)

Potential Terraform Configuration

# Copy-paste any Terraform configurations for how the requested feature may be used.

References

added in #41


[Esquerda.net]: New Bridge

Another local site, this one with existing broken RSS feeds.
This has a broken pubDate (localized to Portuguese), so with some regex trickery this is easily fixed.
I also massage the content into a nicer HTML.

Looks like everything is in order. Thanks for your contribution :+1:


VM hangs waiting 'for all threads that are not already at a safepoint to check-in'

We're experiencing an intermittent problem where the VM hangs in the midst of running a Flutter test:
https://github.com/flutter/flutter/issues/3735
The bug's swan song is:
+ cd examples/flutter_gallery
+ flutter test

00:00 +0: ...e/Linux/build/flutter/examples/flutter_gallery/test/example_code_parser_test.dart
00:00 +0: ...ld/slave/Linux/build/flutter/examples/flutter_gallery/test/simple_smoke_test.dart
00:00 +0: ... Flutter gallery example code parser test
00:00 +1: ... /b/build/slave/Linux/build/flutter/examples/flutter_gallery/test/smoke_test.dart
00:01 +1: ... /b/build/slave/Linux/build/flutter/examples/flutter_gallery/test/smoke_test.dart
00:01 +1: ... - Flutter Gallery app simple smoke test
00:02 +1: ... - Flutter Gallery app simple smoke test
00:02 +2: ... - Flutter Gallery app smoke test
00:03 +2: ... - Flutter Gallery app smoke test
00:04 +2: ... - Flutter Gallery app smoke test
00:05 +2: ... - Flutter Gallery app smoke test
00:06 +2: ... - Flutter Gallery app smoke test
00:06 +2: loading /b/build/slave/Linux/build/flutter/examples/flutter_gallery/test/smoke_test.dart
Shell: Attempt:1 waiting for 1 threads to check in
Shell: Attempt:2 waiting for 1 threads to check in
Shell: Attempt:3 waiting for 1 threads to check in
...

The message is coming from dart/runtime/vm/safepoint.cc which was updated about a month ago: https://codereview.chromium.org/1886583002/

@a-siva

https://github.com/dart-lang/sdk/commit/79e4792dd609ef2bb1f66588a912e420bc058709


#35 Add basic update event

Codecov Report

Merging #100 (35ab946) into develop (d706153) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##           develop      #100   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           16        16
  Lines          544       559   +15
=========================================
+ Hits           544       559   +15

Impacted Files
Coverage Δ

app/routers/profile.py
100.00% <ø> (ø)

app/routers/event.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d706153...35ab946. Read the comment docs.

Codecov Report

Merging #100 (35ab946) into develop (d706153) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##           develop      #100   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           16        16
  Lines          544       559   +15
=========================================
+ Hits           544       559   +15

Impacted Files
Coverage Δ

app/routers/profile.py
100.00% <ø> (ø)

app/routers/event.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d706153...35ab946. Read the comment docs.


Encoding with options -w 10 -l 5 and then decoding leads to corrupted file

I did a quick test to evaluate heatshrink compressing a GPX file with parameters -w 10 -l 5 but the decoded file is smaller than the original and is has several zeroed out  parts.
I got this result both by compiling heatshrink under linux as by using pip install heatshrink2 under windows:
./heatshrink -e -w 10 -l 5 test.gpx  test.hs
./heatshrink -d test.hs test_decoded.gpx
and
python -m heatshrink2 compress -w 10 -l 5 test.gpx test.gpx_hspython_w10_l5
python -m heatshrink2 decompress test.gpx_hspython_w10_l5 test.gpx_decompress_hspython_w10_l5
If I do not specify the -w/-l parameters under python then the result is correct.
Unless I am missing something in the documentation, w 10 / l 5 should be a valid parameter combination (w between 4 and 15 and l between 3 and 10-1), so I assume I am running into a bug ?
Test files in attachment:
test.zip

I realized the problem : when decoding, you ALSO have the specify the parameters used during the encoding.
That was not clear to me from the documentation.

OK.


Error while building on Windows with lapack 3.6.0

Hello,
It is a good news when OpenBLAS updates to lapack 3.6.0. But I encoutered the following error when building it with GCC and msys 1.0 on Windows 7 64 bit:
make[3]: execvp: ar: Bad file number.
This error was not appeared with lapack 3.4.2.
Could some one suggest a solution for this issue? Thanks in advance

A bit of googling turned up your very own #325 - probably the same changes to the lapacke Makefile need to be made get past the limitation on the number of object files the mingw ar.exe can handle - if this is still a problem for recent versions of mingw.

MSYS2 x64 + x86_64-w64-mingw32 report  make[3]: execvp: ar: Argument list too long error. I guess it's the same issue.

Thanks for your suggestions. I did the trick from #325 but it did not help.

Possibly the addition of other new routines to lapack now causes the same problem in other subdirectories, not just in lapacke. Can you tell from the error messages where it was in the build process that the error occured ? (I.e. what were the message lines immediately before the "make [3]: execvp: ar: Bad file number.")

 Thank you.


Build Error on armv7l (RasperryPI): glib::translate::ToGlibPtr<'_, *mut i8> is not implemented for  std::path::Path

Hi there,
I am trying to build https://github.com/centricular/gstwebrtc-demos/tree/master/sendrecv/gst-rust on a raspberry pi armv7l  and just ran in the following error which i did not get on x86_64.
cargo build
Compiling gio v0.4.0 (https://github.com/gtk-rs/gio#b95e3d5d)
error[E0277]: the trait bound `std::path::Path: glib::translate::ToGlibPtr<'_, *mut i8>` is not satisfied
  --> /home/pi/.cargo/git/checkouts/gio-a4b825b310a073b9/b95e3d5/src/unix_socket_address.rs:56:37
   |
56 |                 Path(path) => (path.to_glib_none().0, path.as_os_str().len()),
   |                                     ^^^^^^^^^^^^ the trait `glib::translate::ToGlibPtr<'_, *mut i8>` is not implemented for
`std::path::Path`
   |
   = help: the following implementations were found:
             <std::path::Path as glib::translate::ToGlibPtr<'a, *const u8>>
             <std::path::Path as glib::translate::ToGlibPtr<'a, *mut u8>>

error[E0308]: mismatched types
  --> /home/pi/.cargo/git/checkouts/gio-a4b825b310a073b9/b95e3d5/src/unix_socket_address.rs:61:84
   |
61 |             SocketAddress::from_glib_full(ffi::g_unix_socket_address_new_with_type(path, len as i32, type_.to_glib()))
   |                                                                                ^^^^ expected u8, found i8
   |
   = note: expected type `*mut u8`
              found type `*mut i8`

error: aborting due to 2 previous errors

Some errors occurred: E0277, E0308.
For more information about an error, try `rustc --explain E0277`.
error: Could not compile `gio`.

To learn more, run the command again with --verbose.

I built the gio library with gstreamers cerbero installation (both on armv7l and x86_64) and pkg-config --modversion gio-2.0 returns version 2.56.1 on both machines.

I just tried to build gstreamer-rtsp-server by itself and i got even more errors:
error[E0308]: mismatched types
  --> /home/pi/.cargo/git/checkouts/gio-a4b825b310a073b9/98d1501/src/desktop_app_info.rs:23:41
   |
23 |                 let tmp: *mut *mut i8 = *out.offset(it);
   |                                         ^^^^^^^^^^^^^^^ expected i8, found u8
   |
   = note: expected type `*mut *mut i8`
              found type `*mut *mut u8`
error[E0277]: the trait bound `std::string::String: glib::translate::FromGlibPtrArrayContainerAsVec<_, *mut *mut i8>` is not satisfied
  --> /home/pi/.cargo/git/checkouts/gio-a4b825b310a073b9/98d1501/src/desktop_app_info.rs:28:38
   |
28 |                 let v: Vec<String> = FromGlibPtrContainer::from_glib_full(tmp);
   |                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `glib::translate::FromGlibPtrArrayContainerAsVec<_, *mut *mut i8>` is not implemented for `std::string::String`
   |
   = help: the following implementations were found:
             <std::string::String as glib::translate::FromGlibPtrArrayContainerAsVec<*const u8, *mut *const u8>>
             <std::string::String as glib::translate::FromGlibPtrArrayContainerAsVec<*const u8, *const *const u8>>
             <std::string::String as glib::translate::FromGlibPtrArrayContainerAsVec<*mut u8, *mut *mut u8>>
             <std::string::String as glib::translate::FromGlibPtrArrayContainerAsVec<*mut u8, *const *mut u8>>
   = note: required because of the requirements on the impl of `glib::translate::FromGlibPtrContainer<_, *mut *mut i8>` for `std::vec::Vec<std::string::String>`
   = note: required by `glib::translate::FromGlibPtrContainer::from_glib_full`
error[E0277]: the trait bound `std::path::Path: glib::translate::ToGlibPtr<'_, *mut i8>` is not satisfied
  --> /home/pi/.cargo/git/checkouts/gio-a4b825b310a073b9/98d1501/src/unix_socket_address.rs:56:37
   |
56 |                 Path(path) => (path.to_glib_none().0, path.as_os_str().len()),
   |                                     ^^^^^^^^^^^^ the trait `glib::translate::ToGlibPtr<'_, *mut i8>` is not implemented for `std::path::Path`
   |
   = help: the following implementations were found:
             <std::path::Path as glib::translate::ToGlibPtr<'a, *const u8>>
             <std::path::Path as glib::translate::ToGlibPtr<'a, *mut u8>>
error[E0308]: mismatched types
  --> /home/pi/.cargo/git/checkouts/gio-a4b825b310a073b9/98d1501/src/unix_socket_address.rs:61:84
   |
61 |             SocketAddress::from_glib_full(ffi::g_unix_socket_address_new_with_type(path, len as i32, type_.to_glib()))
   |                                                                                ^^^^ expected u8, found i8
   |
   = note: expected type `*mut u8`
              found type `*mut i8`
error[E0308]: mismatched types
   --> gstreamer/src/tags.rs:197:39
    |
197 |                 ffi::gst_tag_get_type(tag_name.as_ptr() as *const i8)
    |                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
    |
    = note: expected type `*const u8`
               found type `*const i8`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0308`.
error: Could not compile `gstreamer`.
warning: build failed, waiting for other jobs to finish...
error: aborting due to 4 previous errors

Some errors occurred: E0277, E0308.
For more information about an error, try `rustc --explain E0277`.
error: Could not compile `gio`.

Ok, mismatch between signed and unsigned everywhere. See
https://github.com/rust-lang/libc/blob/master/src/unix/notbsd/linux/musl/b32/arm.rs#L1
vs
https://github.com/rust-lang/libc/blob/master/src/unix/notbsd/linux/musl/b64/x86_64.rs#L1
for example.
We need to consistently use c_char I guess

Hmm but we do: https://github.com/gtk-rs/glib/blob/master/src/translate.rs#L474
Can you check in your Cargo.lock is there are multiple versions of the libc crate linked in?

In the lock file of gstreamer-rs all libc versions are 0.2.42 and in gst-rust's lock file all versions are 0.2.41.

Those definitions also didn't change for years. Ok, someone will have to reproduce and follow the types. Somewhere there must be a u8 or i8 instead of c_char
error[E0308]: mismatched types
   --> gstreamer/src/tags.rs:197:39
    |
197 |                 ffi::gst_tag_get_type(tag_name.as_ptr() as *const i8)
    |                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
    |
    = note: expected type `*const u8`
               found type `*const i8`

This is a bug in gstreamer-rs btw, there might be more like this there. I'll check next week unless you want to send me a PR. @meldron

@sdroege thanks for your input, I opened an issue at sdroege/gstreamer-rs/, and i will try to look into it, but i am not sure where to look.

For gstreamer it's exactly in line 197 there. See my comment in the other ticket :)
For GLib/GIO I don't know either, you'll have to look at each of the errors individually, look at the types of function arguments/return values, local variables, etc and see where exactly the i8 or u8 comes from that should actually be a c_char. Or cast accordingly if needed.

I will look for more and submit a PR if I am successful.

The GStreamer part is fixed now, I'm looking at the GTK part now and the GIO part is here: https://github.com/gtk-rs/gio/pull/139

Oh the other part was also in GIO, added to the existing PR


Check whether to integrate Obsidian into documentation

Obsidian is a powerful knowledge base on top of a local folder of plain text Markdown files, linking notes.

The choice fell on mkdocs due to its simplicity.


tokenAmount issue

https://github.com/saddle-finance/saddle-contract/blob/master/contracts/SwapUtils.sol#L215
require(tokenAmount <= xp[tokenIndex], "Withdraw exceeds available");
It seems that tokenAmount is the amount of LP, But the code compares it with xp of pool token.

@alphastorm @penandlim

The amount of LP tokens you receive can never be greater than the combined amount of tokens you deposit. Hence the totalSupply of LP tokens can never be greater than the token reserves in the pool.
E.g. if you have a fresh pool of e.g. DAI/UST and you add 100 tokens of each to the pool you will receive 200 LP tokens in return.
If you then want to make a single sided withdrawal for DAI and you want to withdraw for 200 LP tokens then that would have to represent 200 DAI, but there is only 100 DAI in the pool.

This is intended as @kyriediculous pointed out. When withdrawing via one token, the amount of LP tokens you burn cannot be more than the available liquidity in the pool.


Zooming chart while new data is added changes visual range

Lightweight Charts Version: 3.7.0
Steps/code to reproduce: / Actual behavior:
I have a chart with data being "lazy loaded" when visible range changes enough. Once the data is loaded, it is added to the existing series using "setData".
Zooming the chart using the mouse wheel while the data is being added, causes the time scale to "jump", similar to what is described in #549
Expected behavior:
The scale should not jump when new data is added, even if the chart is being scrolled / zoomed at the time.
Screenshots:
Below is link to a video of it happening.
https://user-images.githubusercontent.com/45341694/152061788-c4321a46-da99-4263-a271-321e216327b2.mp4
I pan the chart until it is at it's "edge" and start zooming in and out while the data is being loaded. It took multiple tries to demonstrate the bug. You can see it works well for the first 2 attempts, then the bug happens twice after that.
--
I am not sure if it is related, but zooming the chart while also panning it at the same time also appears buggy. Try holding your left mouse button and scroll the wheel while moving. The "anchor" point should stay fixed, but it does not.

To be honest I didn't quite get your issue. Could you create a repro for this issue?

Thank you for the response. After making a simplified example, I am unable to experience the bug with it. I'll have a deeper look into my original implementation where I noticed the bug and report back with what I find.

I managed to create a simplified repro - here's the sandbox:
https://codesandbox.io/s/998-repro-4ytcg
I am using react and in order to see this bug I had to use a state variable and update it inside the "subscribeCrosshairMove" handler.
Repro starts with a random chart of 1000 data points and it will add 10 to the left every second. In order to see the bug - position it so the left edge is visible and start scrolling up and down. Eventually you should notice the timestamp in the bottom jump by 10 seconds - video attached:
https://user-images.githubusercontent.com/45341694/152137841-04adf18a-1e64-4f70-a876-30ec4730e86a.mp4
It was easier to experience this on Firefox, but it happened on Chrome as well.

Hi @bbenejc.
Are you able to reproduce this issue without React?

No, I tried a couple different ways... but the simplest case I can reproduce it is by updating react's state variable in the subscribeCrosshairMove handler - as in the repro posted above.

Experiencing the same issue if dragging the chart with mouse, during which, setData is called

For reproducing, trigger setData inside subsribeToLogicalTimeRange if barsInfo.barsAhead is negative. Initially have the screen fully filled with candles, and then drag the chart with mouse to see ahead

@Effanuel are you using this library with React also?

@Effanuel are you using this library with React also?

Yes, but I dont think there should be any difference here

Unable to reproduce this issue without using React, therefore it is most likely an integration issue with the framework which could be resolved within the React component.


Create an example service page

Create HDFS service page with appropriate component and data fetching.

fixed by #26


ngStomp service mock

Could you please help with some unit test example, on how to mock the ngStomp service.
Thank you.

You can mock the ngStomp as any other service in the angularJs world.
The only different element is you have to deal with the fluent API... which is more verbose to mock.
Could you post an example of a (simple) component you want to test to help you with this ?
BTW, I originally want to provide mock specially for ngStomp (https://github.com/davinkevin/AngularStompDK/tree/master/mock) but I never had others methods... If you want to improve it, PR are welcomed

I wanted to mock ngStomp service and only enter the callback function. But could mock only all methods by callFake, so at the moment the real callback function is not called. Here is how I mocked the service
`
        beforeEach(module(function($provide)` {
    $provide.service('ngstomp',` function() {
        var deferred;
        this.subscribeTo = jasmine.createSpy('subscribeTo').and.callFake(function(message) {
            return this;
        });
        this.callback = jasmine.createSpy('callback').and.callFake(function(message) {
            return this;
        });
        this.connect = jasmine.createSpy('connect').and.callFake(function(message) {});
        this.send = jasmine.createSpy('send').and.callFake(function(message) {});
        this.settings = jasmine.createSpy('settings').and.returnValue({headers : {}});
    });
}));

`
The only other way I am thinking now is to actually call the my callback function in the unit test in order to test the expected results.

I would like to have an example of your code, not your test.
If I take an example coming from the documentation :
angular.controller('myController', function($scope, ngstomp) {

    var items = [];

    ngstomp
        .subscribeTo('/topic/item')
            .callback(whatToDoWhenMessageComming)
        .connect()

    function whatToDoWhenMessageComming(message) {
        items.push(JSON.parse(message.body));
    }
 });

What you want to check is if a callback was registered with ngstomp ?
You can do that by mocking ngStomp on subscribeTo method and return a real SubscribeBuilder. With that, you can access the callback and all the parameter defined during subscription of your app.

In the example above I want to test that the items are the ones expected. So I want to test method whatToDoWhenMessageComming, if it receiving a mock message, parses correctly the data and sets it to the scope (my function whatToDoWhenMessageComming has a lot more logic)

Here is an example of my current code.
In controller:
service.serviceMethod($rootScope.param1, $rootScope.param2);
In service:
`
function serviceMethod(param1, param2) {
            unSubscriber =

                ngstomp
                    .subscribeTo('...')
                    .callback(function(message) {
                        processMessage(message, param1, param2);
                    })
                    .connect();
        }`

I want to test the processMessage method. I can call her separately in unit test with the proper params, but I wanted to also test it included in the controller test.

In unit testing, you test only one part of the app. In the angular world, we separate testing of controller and service.
So in your code example, I will write two test, one for the controller with a mock service.serviceMethod where I only check if the method is called with the right parameter.
And, I will write another test for the service where I will mock the ngstomp service and will use the strategy describe in my last message (returning SubscribeBuilder where I can get the callback and call it). At then end, I will check if the method is called with the right parameters...
I will try to provide you an example for this, but right now, I'm writing this answer with my phone, so this is a little bit tricky to write code 😄

Hello,
Could you please help once more and give me the examples you are referring in the previous message.
Thank you.

I don't have any project setup to write real test like you want, so maybe my example will contain error.
I'm writing it in es2015 for simplicity
export class AService {

	constructor(ngstomp) {
		this.ngstomp = ngstomp;
	}

	serviceMethod(param1, param2) {
        unSubscriber = ngstomp
            .subscribeTo('/path')
            .callback(m => processMessage(m, param1, param2))
            .connect();
    }

    processMessage(a, b c){
    	...
    }

}

Like I said, I would like to test two things :
I will check if the method AService.serviceMethod trigger the connection and in another test if the call to processMessage trigger... what it should trigger.

describe(() => {
	let ngStompMock, service;

	beforeEach(() => {
		ngStompMock = jasmine.createSpyObj('ngStompMock', ['subscribeTo', 'callback', 'connect'])
		ngStompMock.subscribeTo.and.returnValue(ngStompMock);
		ngStompMock.callback.and.returnValue(ngStompMock);
		ngStompMock.connect.and.returnValue({});

		service = new AService(ngStompMock);
	});

	it('should register on the good url with a callback and connect', () => {

		/* When */
		service.serviceMethod('foo', 'bar');

		/* Then */
		expect(ngStompMock.subscribeTo).toHaveBeenCalledWith('/path');
		expect(ngStompMock.callback).toHaveBeenCalled());
		expect(ngStompMock.connect).toHaveBeenCalled());
	})

	it('should have the right callback when message come from ws', () => {
		/* Given */
		service.serviceMethod('foo', 'bar');
		let messageFromWS = "a stomp message you would like to test !"

		/* When */
		let callbackFunction = ngStompMock.callback.calls.argsFor(0)
		callbackFunction(messageFromWS)

		/* Then */
		expect(...).toBe(...);
		// Write your custom expectation around the logic triggerd by the processMessage function
	})

})

This is my point of view about testing this kind of things. We can do better by importing SubscribeBuilder, but this one is simple enougth I think.
Sorry for the delay, I see your message in my inbox since a long time, but I didn't find time to write an answer like this before... but I didn't forget it :D
Hope this help
:smile:

Thank you for the examples above.


Add NanoTDF support

Add the ability to encrypt data in nanotdf format and decrypt a nanotdf.
Acceptance Criteria:

[ ]  Ability to Decrypt nanotdf from CPP Client
[ ]  Example encrypt and decrypt operations

Completed as part of #46


Add LexicalSlashShortcutMenuPlugin and PlaygroundSlashShortcutMenuPlugin

This PR adds the base LexicalSlashShortcutMenuPlugin and a PlaygroundSlashShortcutMenuPlugin example that utilizes it.
As you can see from the video below, this is a powerful design that allows for dynamic menu options to be populated from the query (e.g. 10x5 to create a table of that size). The custom searching logic and UI are decoupled from the base Plugin which allows for any search, cache, and ranking solutions you may need.
https://user-images.githubusercontent.com/13852400/176021961-24266f5e-b3b3-4795-ab7e-2b6f363a749f.mov
Closes #2114

CC @im-adithya

This is awesome. Loving the screencast and can't wait to see this top requested feature merged. I guess a next iteration could add a plugin that would decide which options to show based on the position in the document.

This is awesome. Loving the screencast and can't wait to see this top requested feature merged. I guess a next iteration could add a plugin that would decide which options to show based on the position in the document.

That's a great idea! I should have this PR merged tomorrow by the way. Sorry for the delay, I was on vacation and away from a computer. Excited to get this in though.

@acywatson I rewrote a lot of this to make it much more generic. Now the consumer only need to implement a handful of simple methods to build their very own Typeahead. I refactored our MentionsPlugins to use LexicalTypeaheadMenuPlugin as a proof of concept in addition to building a ComponentPickerMenuPlugin for the Playground.

This looks good to me - once we get the mentions tests passing, it's good to go.


Storing ARHitTestResults in ViroARScene parent state?

Hi!
I would like to:

trigger a performARHitTest from another component than ViroARScene by cliking on a simple button;
store the results in the state of ViroSampleAPP App.js (not in ARHitTestSample.js).

Having studied carrefully the ViroARSample, I then did my best to follow the general React rules on "lifting state up", but I still can't find a way to solve this problem.
Do you have any advice for me?
Thanks in advance for your help!

Hey @MayeulGarreau, the performARHitTest API is actually a property of the ViroARScene and as such needs to be called on a ViroARScene object. If you'd like to have a reference to this control from another part of your code (like the callback of the 2D button), you can simply create a reference to it to call the hit test function:
 <ViroARScene
ref={(component)=>{this.arScene = component}}  >

_yourJavaScriptFunc(){
  this.arPlaneRef.performARHitTest(....);
}

Do let us know if you have problems with the above.

Hi, thanks for your quick awesome answer!
I've implemented what you said on my code but when I run App there is an error:
undefined is not an object (evaluating: '_this.arSceneRef.getCameraOrientationAsync').
Maybe the following code may help you to identifies what went wrong?
In App.js:
`_getARNavigator() {
return (

<ViroARSceneNavigator {...this.state.sharedProps}
initialScene={{scene: InitialARScene}}
viroAppProps={this.state.viroAppProps}
debug={true} />

<TouchableHighlight style={localStyles.buttons}
onPress={this._onCameraARHitTest}
underlayColor={'#00000000'} >
coord

);
}
_onCameraARHitTest = () => {
this.arSceneRef.getCameraOrientationAsync().then((orientation)=>{
this.arSceneRef.performARHitTestWithRay(orientation.forward).then((results)=>{
this._onARHitTestResults(orientation.position, orientation.forward, results);
})
});
}In HelloWorldSceneAR.js:  render() {
return (
<ViroARScene ref={(component) => {this.arScene = component}} onTrackingUpdated={this._onInitialized}  displayPointCloud={true} onAnchorFound={this._setARPlane}>
<ViroText text={this.state.text} scale={[.5, .5, .5]} position={[0, 0, -1]} style={styles.helloWorldTextStyle} />
<ViroARPlane minHeight={.5} minWidth={.5} materials={['planeColor']}>
<ViroBox position={[0, .25, 0]} scale={[.2, .2, .2]} alignment={"Horizontal"} materials={['boxColor']} />

{ this._getCoordinates() }
{ this._drawUI() }

);
}`
Thanks again for your help and sorry I'm not really used to references and stuff!

Hey @MayeulGarreau, no worries, it's all good. In your code example above, you have saved a reference to the ViroARScene in HelloWorldSceneAR through 'this'. However, you are also attempting to access it outside of the HelloWorldSceneAR scope from App.js, causing the undefined behavior.
Another approach we can take, is that we could 'tell' HelloWorldSceneAR to perform the hit test, and then in HelloWorldSceneAR define the _onCameraARHitTest() function correctly accesses ViroARScene's 'this' within the correct scope. However, this doesn't work in this case due to the nature of the Viro Scene Navigator class - The HelloWorldSceneAR may not yet been inflated by the navigator, kind of like React's own SceneNav. As such, we'll have to make a few tweaks to ViroARSceneNavigator:
Within ViroARSceneNavigator, please replace the _renderSceneStackItems() function with this below. Note this effectively enables the referencing of the active scene.
_renderSceneStackItems: function() {
      let views = [];
      var i = 0;
      var sceneDictionary = this.state.sceneDictionary;
      for (var scene in sceneDictionary){
          var Component = sceneDictionary[scene].sceneClass.scene;
          var props = sceneDictionary[scene].sceneClass.passProps;

          if (i == this.state.currentSceneIndex){
            views.push((<Component key={'scene' + i} ref={component => {this.activeScene = component}}
            sceneNavigator={this.sceneNavigator} {...props} arSceneNavigator={this.arSceneNavigator} {...props}/>));
          } else {
            views.push((<Component key={'scene' + i} sceneNavigator={this.sceneNavigator} {...props} arSceneNavigator={this.arSceneNavigator} {...props}/>));
          }

          i++;
      }
      return views;
  },

Then, within your HelloWorldAR.js file you can perform all the AR items you'd like, and confine them to a helper function to be triggered. Note that we also must bind the callback to our class in the constructor:
triggerCustomUpdate(callback){
    // Do AR related items here (ar hit check, etc).

    // Once it's done fire the callback to notify 2D elements
    // We pass back a string for example purposes.
    callback("result params");
  }

constructor() {
    super();
    this.state = {} // Set initial state here
    this.triggerCustomUpdate = this.triggerCustomUpdate.bind(this);
  }

Finally, we can then refer to this triggerCustomUpdate function via the refs we have previously plumbed through the arSceneNavigator: this.nav.activeScene.triggerCustomUpdate. We can then trigger the triggerCustomUpdate() from the click event within App.js:
// Returns the ViroSceneNavigator which will start the VR experience
  _getVRNavigator() {
    return (
      <View style={styles.imageContainer}>
        {/* 3D AR Stuff */}
        <ViroVRSceneNavigator
          ref={(component) => {this.nav = component}}
          {...this.state.sharedProps}
          initialScene={{scene: InitialVRScene}} />

        {/* Mock 2D text to click for triggering triggerCustomUpdate()*/}
        <TouchableHighlight style={{justifyContent:'center',alignItems:'center',alignSelf:'center', position:'absolute'}}
        onPress={()=>{ this.nav.activeScene.triggerCustomUpdate(this._updateUI); }}  >
        <Text style={localStyles.buttonText}>{this.state.testText}</Text>
        </TouchableHighlight>
      </View>
    );
  }

  _updateUI(updateText){
    this.setState({testText:updateText});
  }

I hacked a solution together and this appears to work well. This should get you unblocked for now, please let me know if you are experiencing any difficulties with the above.

Hey! Wah thank you so much for this very

Hey @MayeulGarreau, previously the examples were accessing Viro scene data and components, from the context of your App.js. If instead you want to access App.js data, from the context of your App.js, it would be different - you would have to pass data through to your scene. To do that you can use viroAppProps.
How to achieve that should be thoroughly explained in this issue. Let me know if you have any questions or difficulties with the above.

Hi, Sorry for the delay of my answer!
I fully implemented it, it works fine. Thanks a lot for your help!
I close the issue :)

Sounds good @MayeulGarreau. Note that you will have to re-apply the patch above if you ever upgrade ViroReact in the future. I've filed a bug internally so that we can consider making a more permanent fix. Just an FYI.


Why do we need to take care of num_process in lr_scheduler?

Hi, I am a little bit confused about the behavior of lr_scheduler especially this line of code:
https://github.com/huggingface/accelerate/blob/995563fec9801308b3ddbbfaf4128730182eaa81/src/accelerate/scheduler.py#L73-L82
In my humble opinion, in the DDP mode, gradient_accumulation_steps and more gpus are essentially increase the batch size . Suppose the batch_size parameter we give to dataloader is 10, we have 8 gpus, and gradient accumulation steps is 4. So the real batch size is 10*8*4 = 320. This is i believe when we should do one optimization step as well as one lr schedule step. But in the above line of code, it would do more than one lr schedule step.
Briefly speaking, my question is: why do we need to make lr_sceduler depend on the number of GPUs when we already assign a value to warmup_steps in user config?

a relevant problem is here: https://github.com/huggingface/transformers/issues/18436#issuecomment-1414290934

That's because the lr_scheduler has been defined taking into account the unprepared dataloader which has the original batch size.

But in my case(2 GPUs), the size of train_dataloader doesn't change after the prepare method. I believe the DDP sampler wouldn't change the the size of train_dataloader?
BTW, thanks for the prompt response!

So, if i want to assign warmup_steps to 4000, and i want my lr to exactly peak at 4000. (no matter how many GPUs or batch size i set). What would you suggest to be the best practice?

Sorry, it my mistake. The dataloader size does change after the prepare method.
Could you please explain some conceptual guides on how to set the warmup_steps in accelerate. Take this paper, Attention is all you need as example. In this paper, it said the model was trained on 8GPUs and warmup_steps is 4000. So, i should give 4000*8 to the scheduler function?

from transformers import get_polynomial_decay_schedule_with_warmup,
lr_scheduler = get_polynomial_decay_schedule_with_warmup(
        optimizer,
        num_warmup_steps = 4000*8*cfg.trainer.gradient_accumulation_steps,
        num_training_steps = total_steps
    )

Hi, @sgugger , would you mind spare some time to check this?

Hi @sgugger , really sorry for bothering you. I am new to Accelerate and the prepare method works like a magic black box for me  to handle multi GPUs (which it is intended to be). So the obvious solution of not preparing lr_scheduler is not so obvious for me.
For anyone who is seeking for fixed warmup steps in Accelerate, a simple workaround would be:
lr_scheduler = get_scheduler()

# not prepare lr_scheduler

for batch in dataloader:
	with accelerator.accumulate(model):
		loss = compute_loss()
		if accelerator.sync_gradients and not accelerator.optimizer_step_was_skipped:
			lr_scheduler.step()


Recalculate Equalizer when height of content changes?

Is there a method to ask the equalizer to either monitor the height of a child element or recalculate when the height of a child <div> content changes?  I have some panels which load ajax content and the tables can grow or shrink by several rows and it would be nice to have the Equalizer resize when the content changes?  I haven't been able to figure out how to monitor this though.

This equalizer depends on the resize event to initiate resizing.  See https://developer.mozilla.org/en-US/docs/Web/Events/resize for the event details.  Net result, only a document view resize will fire a change (for example, resizing the browser window), not content changes.

@sjd78 Thank you for your comment. Knowing that the Equalizer component only runs on the resize event, I added this to the callback on my ajax call and the columns resize nicely with the new data.
window.dispatchEvent(new Event('resize'))


IE11: Object doesn't support property or method 'contains'

Getting this error when running the demo page with Internet Explorer 11 (demo page: http://bluekit.blueberry.io/demo).  Click on any element in the sidebar to reproduce what you can see below:

It's pointing to this line of code: https://github.com/blueberryapps/react-bluekit/blob/c81e3af66caa3a634082c67da21f197a8d8f6f3e/src/app/component/PropsSidebar.react.js#L157
On the demo site my debugger claims both area and pickerButton are undefined, but in the app I'm developing on my own, both of these are defined, and pickerButton seems to be an SVG element, which may be the trouble.  IE has supported Node.prototype.contains since IE 5: https://developer.mozilla.org/en-US/docs/Web/API/Node/contains#Browser_compatibility
Thanks in advance!
EDIT: Looks like it is related to pickerButton being an SVGElement.  You can try this in your console on IE11 (while running the demo page linked to above): document.querySelector('svg').contains, which returns undefined.  Try the same in Chrome and it returns the native contains function.

Hi, we do not support IE since it's not officially supported by Microsoft https://www.microsoft.com/en-us/windowsforbusiness/end-of-ie-support
We recommend using different browser than IE.

Thanks for the quick reply.  No worries, but it would be handy to have a disclaimer in the readme somewhere.  Some folks still have users running on IE11 (out of our control), and it would be handy for them to see browser support right on the readme, so they can better evaluate whether to use BlueKit.
Thanks!


Add support for window operations on columns

This patch also adds support for Rolling operations on columns.
Fixes 1 part of https://github.com/dotnet/corefxlab/issues/2815. We should add Expanding APIs too, but that can come in another PR.

using System;

copyright on all code files

Refers to: src/Microsoft.Data.Analysis/DataFrameWindow.Rolling.PrimitiveDataFrameColumn.cs:1 in 83b9e30. [](commit_id = 83b9e3063b11e04827101ef71aa8f1a01e8225d3, deletion_comment = False)

using System;

copyright on all code files

Refers to: src/Microsoft.Data.Analysis/ArrowStringDataFrameColumnRollingWindow.cs:1 in f3ab53b. [](commit_id = f3ab53bc36a4be8830e2c72bbbd923b1d219ae8b, deletion_comment = False)

Not sure if this will make it into 0.3.0. Closing this for now. We can re-open if we have time


Networking breaks if placing 'api' service in 'host' networking mode

Hi there, I'm trying to get bitfeed running on a node that I built using the raspibolt guide, meaning bitcoin core etc. are running on the host machine, not in containers. I made some modifications to docker-compose.yml accordingly but can't seem to get the service to come up.

If I leave the back end port as-is (5000) and do not modify networking setup, the client comes up but the back end can't connect to RPC due to that address being taken. This is expected as lnbits is already using that port.
If I modify port 5000 to something else (6000 in this case), the client comes up but the back end fails because the RPC connection was refused. This is also expected as the default bitfeed network shouldn't have access to the host.
If I then modify the back end to use network_mode: host, the back end comes up, but this time the front end fails to come up:

nginx: [emerg] host not found in upstream "api:6000" in /etc/nginx/conf.d/default.conf:42
I'm sort of at a loss for how to resolve this. While the container is running (i.e. on the desired port but in the wrong network mode), i can see that default.conf does indeed have the expected entry. See below for the configs I'm working with.
A few additional details:

Bitcoin Core v.24.0.1
Ubuntu Server 22.04 running in a Proxmox virtual machine
Deployments being done through Portainer v2.17.1
Docker v23.0.1

Is this a bug or user error?

Values of /etc/nginx/conf.d/default.conf while the web container is running but the back end is unable to connect due to networking settings. Note that when I add network_mode: host to the api stanza in docker-compose, i can no longer access this default.conf because the web container keeps dying over and over.
root@82b17741b141:/etc/nginx/conf.d# cat default.conf
map $sent_http_content_type $expires {
    default                 off;
    text/css                max;
    text/json               max;
    application/javascript  max;
}

server {
    listen 80;

    root /var/www/bitfeed;
    index index.html;

    server_name client;

    location ~* \.(html)$ {
            add_header Cache-Control 'no-cache';
    }

    location /api {
            proxy_pass http://wsmonobackend;
            proxy_set_header        Host $http_host;
            proxy_set_header        X-Real-IP $remote_addr;
    }

    location /ws/txs {
            proxy_pass      http://wsmonobackend;
            proxy_http_version      1.1;
            proxy_set_header        Upgrade $http_upgrade;
            proxy_set_header        Connection "upgrade";
            proxy_set_header        Host $http_host;
            proxy_set_header        X-Real-IP $remote_addr;
    }

    location / {
            try_files $uri $uri/ /index.html;
            expires $expires;
    }
}

upstream wsmonobackend {
    server api:6000;
}

My docker-compose.yml
version: "2.1"

services:
  web:
    image: ghcr.io/bitfeed-project/bitfeed-client:v2.3.4
    restart: on-failure
    stop_grace_period: 1m
    depends_on:
      - "api"
    environment:
      TARGET: "docker"
      BACKEND_HOST: "api"
      BACKEND_PORT: "6000" #modified because port 5000 is already in use by another service
    ports:
      - "3080:80" #modified because port 3000 is already in use by another service

  api:
    image: ghcr.io/bitfeed-project/bitfeed-server:v2.3.4
    user: "1000:1000"
    restart: on-failure
    stop_grace_period: 1m
    network_mode: host #added network_mode: host so that I can access bitcoin RPC on local machine
    environment:
      PORT: "6000" #modified because port 5000 is already in use by another service
      BITCOIN_HOST: "<IP_ADDRESS>" #modified to point to localhost
      BITCOIN_ZMQ_RAWBLOCK_PORT: "28332"
      BITCOIN_ZMQ_RAWTX_PORT: "28333"
      BITCOIN_ZMQ_SEQUENCE_PORT: "28334"
      BITCOIN_RPC_PORT: "8332"
      BITCOIN_RPC_USER: "raspibolt" #using raspibolt RPC user
      BITCOIN_RPC_PASS: "[password]"

I'm going to resolve this because I figured it out and -- wait for it! -- it was user error.
Posting my solution here so that future rookies such as myself can learn from my mistakes as they figure out how to get an application running in docker to talk to bitcoin RPC on the host machine.
Here's the important stuff in bitcoin.conf. The key point is that I needed to rpcbind to <IP_ADDRESS> and then also allow zmq connections on <IP_ADDRESS>:port.
# Bitcoin daemon
server=1
txindex=1

# Network
listen=1
listenonion=1
proxy=<IP_ADDRESS>:9050
bind=<IP_ADDRESS>

# Connections
rpcport=8332
rpcbind=<IP_ADDRESS>     #is this safe? reckless!
rpcallowip=<IP_ADDRESS>     #allow RPC connections from localhost
rpcallowip=<IP_ADDRESS>/16     #allow RPC connections from the docker network on which the containers I'm trying to connect are running
rpcauth=user:passwordhash
zmqpubrawblock=tcp://<IP_ADDRESS>:28332     #need to accept zmq connections on <IP_ADDRESS> instead of <IP_ADDRESS>
zmqpubrawtx=tcp://<IP_ADDRESS>:28333     #ditto
zmqpubsequence=tcp://<IP_ADDRESS>:28334     #ditto again
<EMAIL_ADDRESS>        # for Electrs

Here's the modified docker-compose.yml that works with the above bitcoin.conf configuration:
version: "2.1"

    services:
      web:
        image: ghcr.io/bitfeed-project/bitfeed-client:v2.3.4
        restart: on-failure
        stop_grace_period: 1m
        depends_on:
          - 'api'
        environment:
          TARGET: 'docker'
          BACKEND_HOST: 'api'
          BACKEND_PORT: '6000'
        ports:
          - '3080:80'

      api:
        image: ghcr.io/bitfeed-project/bitfeed-server:v2.3.4
        user: '1001:1001' #running as the bitcoin user (same user that runs bitcoind on the host)
        restart: on-failure
        stop_grace_period: 1m
        volumes:
          - '/home/bitcoin/.bitcoin:/bitcoin:ro'
        extra_hosts:
          - 'host.docker.internal:host-gateway'
        environment:
          PORT: '6000'
          BITCOIN_HOST: 'host.docker.internal'
          BITCOIN_ZMQ_RAWBLOCK_PORT: '28332'
          BITCOIN_ZMQ_RAWTX_PORT: '28333'
          BITCOIN_ZMQ_SEQUENCE_PORT: '28334'
          BITCOIN_RPC_PORT: '8332' #8332
          BITCOIN_RPC_COOKIE: /bitcoin/.cookie
          TARGET: personal


Add tests

May want some unittests on the functions in dds.py
davidmc discovered some existing dds evaluations https://web2.acbl.org/documentLibrary/play/analysis/R030220.pdf. A good test would be to replicate the numbers there (if indeed that is possible.

Note that the dds library itself is very widely used and verified, so bugs are far more likely to be introduced in our code vs the library itself (mocking the library for our tests might be good enough?).

Also note that we only plan on using the calc_dd_table method for the hackathon (the other methods may be used in the future though)

Thanks for the heads-up, Brewing1!
I've assigned myself to this. Anyone who wants to join me is welcome. I'm a fan of pair programming and happy to try mobbing as well.

I guess some tests in the api.py file could be useful now (maybe pytest with the Flask framework? -- i've never used flask so don't really know what's going on). Note that DDScore may not be built upon today

We now have a basic test of the calc_dd_table method. I'll keep this issue open since I plan to add a few more tests there and then look into testing the Flask server per Brewing1's suggestion.
I agree that the place for extensive testing of the underlying DD code is in the dds C++ project, not in our wrapper. They have an extensive library of test deals, though it's not clear to me that their test harness is effective.
I like the idea of mocking the C++ library for our tests, but at least for me the current test runs in under a second so I don't think mocking is necessary.

A javascript unit test framework would be nice too. A few years ago I choose for my personal project Jasmine. I don't remember specifically why I choose it. But I remember I was happy with features it offered. It looks like still maintained. My experience is old and I have only fragmented memory how to use it.
Should javascript tests be a separated issue? (If this is all about server side test then I would say yes.)


Amending paths to match Ark

Treasuredata needs to reference v5 not v4 and the Ark path includes lts
RabbitMQ path should reference noarch rather than x86_64

Found during the upgrade of Graphcore CL0 from Antelope to Caracal

Unfortunately it looks like our CI can't handle external contributors, we don't get very many of these. I've pushed to a new branch under my auth here: https://github.com/stackhpc/stackhpc-kayobe-config/pull/1426
Thanks for the fix @jjo93sa! We'll look into fixing external contributions in the future


Poorly configured legitimate domain mx validation fails

Got a user trying to register with email on agb.lv domain. I am doing an mx_valid? check, and it fails for this domain, even though the domain is legit.
<EMAIL_ADDRESS>#=> true
<EMAIL_ADDRESS>#=> false
<EMAIL_ADDRESS>#=> true

Google domain check here says the domain is poorly configured, but working.
https://toolbox.googleapps.com/apps/checkmx/check?domain=agb.lv
Should I be using ValidateEmail.mx_valid_with_fallback? to ensure these poorly configured domains pass?

Hi yes you should use it


Fix french holidays

🎁 enable assertAllHolidaysChecked for France
🐞 This also aligns the descriptionPropertiesKeys of Holidays_test_fr_2010.xml with the definitions in holiday_description.properties
Thereby the following issue was found and fixed:
🐞 Easter Sunday is not a public holiday in France
TBD: leave it as an UNOFFICIAL_HOLIDAY instead of removing it completely - then the test case would need adjustment.

I also could not find anything that Easter is a valid public holiday or holiday in France.
See https://en.wikipedia.org/wiki/Public_holidays_in_France
I think we should drop it.


Lesson 7: VRF Coordinator version

I added an additional note with respect to the VRF Coordinator version.

Thank you!


use chef_gem and install depends at compile time, version bump

Small changes to use new chef_gem resource. Also, don't assume dependencies exist.
Instead of installed dependencies here, we could make changes to the xml and build-essential cookbooks to ensure both support compile time installation of packages.

haha. back when chef_gem was new. closing zombie PR


No support for client-side TLS

Version: 1.2.1
The current version of device-mqtt only supports server-side TLS, it's not possible to configure a local certificate that the client would use to authenticate to the server instead of using user/password authentication.
It looks like the app-functions-sdk already has support for this, so we should leverage if/when this issue is addressed:
https://docs.edgexfoundry.org/1.2/microservices/application/BuiltIn/#mqtt

Depends on https://github.com/edgexfoundry/device-sdk-go/issues/564


Bd 20200124 mongodb bson parser

First version of ability to parse and encode BSON documents. Includes the Json-Like classes for BSON, a BSON Codec, and small refactorings of the core Json-like interfaces.

This is super cool Bill!
FYI I'm working on getting 4.2.0 out the door this weekend (have been sick this week) but plan on reviewing this as soon as that's happened.

James..
Excellent! I will leave it to you to merge when you think it appropriate. I did want to point out that I have not yet incorporated any of the MongoDB stuff into the master pom build structure. My plan was to have a parent mongodb pom that would build this small first piece along with a more substantial MongoDB server.. maybe somewhat more closely tied to some OSGi framework... maybe looking at existing Karaf stuff but that doesn't look like it has been touched for a while.
cheers, Bill

This PR is no longer compatible with the changes being made in the Jackson parser. I am closing the PR for now

Hmm, that's unfortunate.. I wouldn't have expected the Bson parser to have any dependencies on Gson. Ah well.
I wonder if it makes sense to just explore using this library with Jackson once we're done merging the Jackson stuff:
https://github.com/michel-kraemer/bson4jackson


Release 0.4.0

It's time to release 0.4.0
Here is what needs to get done:

[ ] merge #34
[ ] merge #32
[ ] merge #28
[ ] manually test on a couple different node versions
[ ] update readme with info on patch uploading (part of #32 )
[ ] update readme with changelog info
[ ] update package.json with new version
[ ] publish new version on npm
[ ] patches for WP, bbpress, and buddypress
[ ] announce the change on make/core

If I missed anything, let me know in the comments.

I've created tickets and patches:
• bbPress https://bbpress.trac.wordpress.org/ticket/2914 and working as expected
• WordPress https://core.trac.wordpress.org/ticket/35917 and working as expected (WordPress doesn't need a patch for the actual task, only the package.json change when 0.4.0 is released as the grunt upload_patch:98765 tasks default configuration defaults to WordPress Trac URL's)
• BuddyPress https://buddypress.trac.wordpress.org/ticket/6923 and not working as expected, see https://github.com/aaronjorbin/grunt-patch-wordpress/pull/32#issuecomment-187601648

The following tasks can be marked complete:
• update readme with info on patch uploading (part of #32 )
• patches for WP, bbpress, and buddypress
Cherry picking changelog from https://github.com/aaronjorbin/grunt-patch-wordpress/compare/0.3.0...master -

0.4.0 Added support for uploading a patch from the command line using XML-RPC. CI tests now test on WordPress supported NodeJS versions 0.12.x, 4.x, and 5.x

Someone @nacin-shaped will need to open the attachment upload endpoint to all users on Trac, @iseulde mentioned it doesn't work for users without gardener status.

@aaronjorbin @ntwb @ericandrewlewis Someone @nacin-shaped has opened the attachment upload endpoint to all users on Trac. Sorry, I didn't realize the code had been done for it and that this was ready for my action.

Current release is 0.4.2, so regardless of whether the release got announced on make/core, I think this issue can be closed.


DevConfig "Link List" defekt

Describe the bug
DevConfig "Link List" defekt:
Auch nach 2 Minuten Ladezeit bleibt die Seite weiß und komplett leer.
siehe Homematic Forum
tested with: <IP_ADDRESS>80929 - But problem exists inf RaspberryMatic also in early versions.
To Reproduce
Steps to reproduce the behavior:

Go to 'Startseite > Einstellungen > Systemsteuerung'
Click on 'DevConfig'
Click on 'Link List' - [ccu IP]/tools/devconfig.cgi?cmd=list_links&sid=[sid]
nothing happens

After some investigation of this issue, I found that the XMLRPC server implementation in HMIPServer as well as the used libXmlRpc library has to be partly blamed here. DevConfig is actually using the tclrpc interface to send a getLinks method call via the following tclsh command:
# echo "load tclrpc.so; puts [xmlrpc 'http://<IP_ADDRESS>:2010' getLinks]" | tclsh

If this is executed on a CCU/RaspberryMatic this call never actually returns and results in a stalled tclsh process. That's actually why the DevConfig "Link List" page will never be successfully loaded at all.
After some more deeper analysis it occurred to me that it is not the tclrpc.so library that has to be blamed here, but the XMLRPC Server in HMIPServer not answering in a nice way like the one in rfd or other standard XMLRPC Server implementations. That's why the used XML-RPC client library (libXmlRpc.so) which tclrpc.so (and other processes, like ReGaHss) is using is confused, ending up in a situation where libXmlRpc.so causes a stall.
The same situation can be triggered, for example, by simply sending out an XML-RPC request with some unknown method:
# echo "load tclrpc.so; puts [xmlrpc 'http://<IP_ADDRESS>:2010' unkown]" | tclsh

This call also causes the tclsh to stall forever. By using the following curl call, one can actually see what is happening inside the XMLRPC server in HMIPServer:
# curl -v --silent http://<IP_ADDRESS>:2010/ -H "Content-Type: text/xml" -d "<?xml version='1.0' encoding='UTF-8'?><methodCall><methodName>unkown</methodName><params></params></methodCall>"
> POST / HTTP/1.1
> Host: <IP_ADDRESS>:2010
> User-Agent: curl/7.61.1
> Accept: */*
> Content-Type: text/xml
> Content-Length: 111
>
< HTTP/1.1 200 OK
< Content-Length: 0
<

As you can see, the XML-RPC server returns with an totally empty response with content-length 0 which IMHO violates the XML-RPC protocol and thus libXmlRpc.so being not prepared for such a lazy answer. The same query to the XML-RPC server to rfd returns the following valid response:
# curl -v --silent http://<IP_ADDRESS>:2001/ -H "Content-Type: text/xml" -d "<?xml version='1.0' encoding='UTF-8'?><methodCall><methodName>unkown</methodName><params></params></methodCall>"
> POST / HTTP/1.1
> Host: <IP_ADDRESS>:2001
> User-Agent: curl/7.61.1
> Accept: */*
> Content-Type: text/xml
> Content-Length: 111
>
< HTTP/1.1 200 OK
< Server: XMLRPC++ 0.7
< Content-Type: text/xml; charset=iso-8859-1
< Content-Length: 280
<
<?xml version="1.0" encoding="iso-8859-1"?>
<methodResponse><fault>
	<value><struct><member><name>faultCode</name><value><i4>-1</i4></value></member><member><name>faultString</name><value>unkown: unknown method name</value></member></struct></value>
</fault></methodResponse>

Here the XML-RPC server in rfd always returns with a "" response including content-type and content-length larger 0.
Thus, to conclude: This is a bug/issue in the XML-RPC Server implementation in the HMIPServer component as well as in the libXmlRpc library (since it should never actually happen that it causes a stall!) in the CCU infrastructure. This issue has already been reported to eQ3 and a fix is already being worked on.


Feature Request: Syntacic sugar for looping over a list of viewModels

Hi everyone!
I've got a feature request concerning <compose>. Let's say I have a viewModel/view like so:
somemodel.js
import {SomeOtherModel} from './someothermodel';

export class SomeModel{
    constructor() {
        this.items = [new SomeOtherModel('stuff', 'more stuff'), new SomeOtherModel('stuff', 'more stuff')];
    }
}

somemodel.html
<template>
   <compose view="./someothermodel.html" repeat.for="item of items"></compose>
</template>

And some other viewModel/view like this:
someothermodel.js
export class SomeOtherModel{
    constructor(param1, param2) {
        this.param1 = param1;
        this.param2 = param2;
    }
}

someothermodel.html
<template>

${item.param1}

${item.param2}

</template>

What I would like is to request is some syntactic sugar so that I can shape the models/views into this:
somemodel.js (remains the same)
somemodel.html
<template>
   <someothermodel repeat.for="items"></someothermodel >
</template>

And some other viewModel/view like this:
someothermodel.js (remains the same)
someothermodel.html
<template>

${param1}

${param2}

</template>

Or this:
somemodel.js (remains the same)
somemodel.html
<template>
<template repeat.for="items">
   <someothermodel></someothermodel>
</template>
</template>

someothermodel.js (remains the same)
someothermodel.html (same as the version above)
Or this:
somemodel.js (remains the same)
somemodel.html
<template>
<template repeat.for="item of items">
   <someothermodel view-model="item"></someothermodel>
</template>
</template>

someothermodel.js (remains the same)
someothermodel.html (same as the version above)
Basically it's a static version of <compose> and a form of syntactic sugar. If you'd want dynamic behaviour you would still use compose.
Would love to hear from you.
Greetings, Robert.

Something very close to what you want is possible today. You can accomplish it by requiring the html directly in the view. If you change your view to this:
someothermodel.html
<template bindable="viewModel">

${viewModel.param1}

${viewModel.param2}

</template>

Then you can do this:
somemodel.html
<template>
   <require from="someothermodel.html"></require>
   <template repeat.for="item of items">
      <someothermodel view-model="item"></someothermodel>
   </template>
</template>

Requiring an .html file into a view automatically converts it into a custom element with any bindable properties declared in the bindable attribute on the template.


Run test in Karma, and switch from TravisCI to CircleCI

This PR started as a simple fix for the failing builds on TravisCI... But evolved into a complete refactor of Spine's test runner to use Karma. Changes include:

Builds passing again!
Switched from TravisCI to CircleCI (sponsored by Lightmaker)
Replaced all custom test runner code with Karma

Tests now run in PhantomJS, Chrome, and Firefox by default
$ npm test now runs test suite multiple times against different versions of jQuery (v1.8-v2.2)
CLI test runner will automatically re-run tests when files change
Tests can be run agains specific versions of jQuery using $ JQUERY_VERSION=1.9.0 karma start

Looks pretty slick! Can't think of any reason this isn't a major improvement. I am unfamiliar with circleCI, but had no special love for travisCI either so, no opinion on that really so long as it works.

From my experience, Circle is much nicer. Aside from a generally nicer UI and workflow, the biggest advantage is that you can SSH into build containers for debugging.


Use fluent-bit container instead of filebeat

What
Add err-log and show-lob containers that use fluent-bit by default.
The manifest which contains the filebeat containers(err-log and slow-log) is very long.
And the license of filebeat is complicated.
The err-log and slow-log are needed to check the log of mysqld servers.
So I want to make it easier to use.
How

Check that the fluent-bit is useable for our usage.
Change the moco-controller to create StatefulSet manifest which contains err-log and slow-log containers by default.

Use fluent-bit image in the containers.
Make the containers( err-log and slow-log) customizable.

Update documents.

https://github.com/cybozu-go/moco/blob/v0.4.0/docs/example_mysql_cluster.md#example
https://github.com/cybozu-go/moco/blob/v0.4.0/docs/design.md#how-to-manage-log-files

Checklist

[ ] Finish implementation of the issue
[ ] Test all functions
[ ] Have enough logs to trace activities
[ ] Notify developers of necessary actions

https://docs.fluentbit.io/manual/pipeline/inputs/tail

File rotation is properly handled, including logrotate's copytruncate mode.
Note that the Path patterns cannot match the rotated files. Otherwise, the rotated file would be read again and lead to duplicate records.


Replace WildFlyAwaitStrategy with Arquillian Cube log await strategy

I ended up having to implement a custom AwaitStrategy becuase of some issues I encountered with the Arquillian Cube alternatives:
https://github.com/arquillian/arquillian-cube/issues/502
https://github.com/arquillian/arquillian-cube/issues/499
Ideally we'd use the 'log' await strategy and watch the WildFly Docker container log output for something that confirms the app server is fully initialized.

Blocked on arquillian-cube 1.0.0.Alpha16.


Add a timeline / tracing profiler

PR Details
Description
This PR adds a profiler outputting data in chrome://tracing format (for now) and changes Profiler to make that possible.
ChromeTracingProfileWriter consumes events and dumps them to a file which can then be analyzed with tools like chrome://tracing, perfetto or speedscope.
Usage
//Start
ChromeTracingProfileWriter.Start(string outputPath, bool indentOutput = false)
//Stop
ChromeTracingProfiler.Stop()

//To add another output format/client and receive events use
Profiler.Subscribe() and Profiler.Unsubscribe()

Related Issue
N/A
Motivation and Context
I had trouble pinpointing performance issues with both the existing Profiler and external tools like dotTrace, so I started experimenting with adding something in the vein of chrome://tracing or tracy. In particular the exisiting Profiler is quite heavy-weight and doesn't play well with multithreading, while everything happening inside the ThreadPool / Dispatcher is difficult to understand with sampling profilers as a lot of work happens in anonymous functions.
The TraceProfiler lets you see work on all threads and lets you look at individual frames to analyze spikes instead of just averages. For me this was also very helpful in understanding the structure of the engine.
An example trace from a rendering stresstest opened in perfetto:

The example trace:
tracing_example.json.gz
Open Questions / Todo

[x] Decide what to do with this. I've already found this very useful compared with the existing Profiler, so I'd like to see it added, but of course having multiple internal profilers is not ideal. Merging it with Profiler seemed quite challenging, but maybe it's a bit easier after the refactoring in #1774 and I should take another look.
[ ] Add tests.
[ ] Update outdated comments and add new ones where needed.
[ ] Verify performance.
[ ] Check CPU/GPU timestamp sync.
[ ] Update docs.

Types of changes

[ ] Docs change / refactoring / dependency upgrade
[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[x] Breaking change (fix or feature that would cause existing functionality to change)

Checklist

[x] My change requires a change to the documentation.
[ ] I have added tests to cover my changes.
[x] All new and existing tests passed.

That is very interesting. I want to see this as an opt-in. And given the implementation is not large I think we could merge it into the source code.
What do you think about implementing a hook into the existing Profiler?
Something like
public delegate void EventProcessedHandler(ref ProfilingEvent profilingEvent, ProfilingEventType eventType);
public static event EventProcessedHandler EventProcessed;

// at the end of Profiler.ProcessEvent
EventProcessed?.Invoke(ref profilingEvent, eventType);

// In your implementation a new static method
TraceProfiler.StartCollectingEvents() => Profiler.EventProcessed += OnProfilingEventProcessed;
TraceProfiler.StopCollectingEvents() => Profiler.EventProcessed -= OnProfilingEventProcessed;

And then just add more ProfilingKeys to places where they were missing - like you did with your implementation specifically?
And if there's something in the current Profiler implementation that prevents it from being that simple - maybe we can look at why is that?

We can easily add caller info into the Profiler. Other than that I see mainly that you're adding thread info and maintaining a per thread stack in order to stop the Stopwatch in order for Begin/End pairs (already done by ProfilingState).
The last difference - instead of using a per frame ring buffer to store events you use async channels to stream them to a file.

I'm curious how we ensure profiling call sites get stripped for release/shipping builds?

And then just add more ProfilingKeys to places where they were missing - like you did with your implementation specifically?
And if there's something in the current Profiler implementation that prevents it from being that simple - maybe we can look at why is that?

My first concern here would be performance. How big of a deal that is really depends on what you're profiling: A few hundred scopes spread around should be fine, thousands of scopes in a small function that gets called by worker threads is perf death by synchronization overhead.
As a quick best-cast vs worst-case example this is just collecting 100k events, without any output, either in a regular for loop or in a Parallel.For:

BenchmarkDotNet=v0.13.2, OS=Windows 10 (10.0.19045.3324)
AMD Ryzen 5 3600, 1 CPU, 12 logical and 6 physical cores
.NET SDK=8.0.100-preview.7.23376.3
  [Host]     : .NET 6.0.14 (6.0.1423.7309), X64 RyuJIT AVX2
  Job-JPLRAA : .NET 6.0.14 (6.0.1423.7309), X64 RyuJIT AVX2

InvocationCount=1  UnrollFactor=1

Method
Mean
Error
StdDev
Ratio
RatioSD

StrideProfiler
42.977 ms
0.7726 ms
0.6032 ms
3.85
0.35

Trace
11.706 ms
0.6019 ms
1.7556 ms
1.00
0.00

StrideProfilerParallel
77.534 ms
1.5432 ms
2.8985 ms
6.86
0.75

TraceParallel
2.490 ms
0.1216 ms
0.3566 ms
0.22
0.05

I also thought that adding lower level profiling keys would just clutter the existing profiling UI.
Maybe adding a ProfilingKeyFlags.TracingOnly would be a good option.
I think my other issues were really about Profiler itself and shouldn't affect this PR, your PR has already improved it and the rest can wait for a follow-up.
I'm also not a fan of how adding/removing ProfilingKeys even for temporary profiling is needed, but it's a small price

I'm curious how we ensure profiling call sites get stripped for release/shipping builds?

They don't need to. When the profiler is off the JIT can inline the check and skip the calls to it quite well.
If the Profiler perf is not great it would be good to determine why and see if we can improve it. Maybe it would make sense to gut it even more and use something closer to the runtime? What we need is basically a simple solution to start/stop scopes and gather perf info and output it to some stream once per frame - either consumed by in game UI or by external tools or written to a file, etc.

They don't need to. When the profiler is off the JIT can inline the check and skip the calls to it quite well.
In the current implementation of TracingProfiler/Scope I don't see any is active checks, so how would it avoid that work?

Additionally. what about AOT platforms like iOS, etc?

I think I've finally figured out what I want to do with this. In a first step, replace the Profiler internal buffers with Channels to get rid of the locking and to get support for multiple consumers. Then I can convert the TraceProfiler callsites to the existing functions/datatypes and add the JSON writer as a second consumer. The only functional change I would want to make here is to stop sending Begin events, they just get filtered out on the receiving side (GameProfilingSystem) anyway, while basically doubling the event count for no benefit.
Then, as a second PR, we could take another look at the existing API and datatypes:

Add caller info
I haven't measured the impact, but ProfilingState and ProfilingEvent are currently both 296 bytes, that's bound to have a perf impact. Attributes make up 144 bytes in each and they are never used, Custom messages are 112 bytes, but at least sometimes used. So we can probably slim that down, or split things up, so only what's used is actually recorded.
Mark events are very rarely used and could basically be replaced with 0 duration timespans. Maybe we should add an API for counters instead
Per frame grouping of events (okay, maybe that's a good use for markers)

And then, as a last step, take another look at GPU profiling. Usually I would expect GPU information per queue (on modern APIs), but I don't think that's currently exposed. But I might very well be wrong about that, didn't want to dig to deeply into it for now.
As an alternative to rolling our own we could also look at integrating https://github.com/clibequilibrium/Tracy-CSharp, it's definitely well optimized and feature rich, but ofc it's another (non-C#) dependency (+ imgui) and I don't know what limitations the .NET integration might have.

Other than that I see mainly that you're adding thread info and maintaining a per thread stack in order to stop the Stopwatch in order for Begin/End pairs (already done by ProfilingState).

The stack is only there to support profiling across function boundaries, with ProfilingState you have to pass it along yourself somehow, if begin and end are not in the same scope. (That also means my attempt at limiting event capturing by stack depth was not working, oops. I will remove it, did not seem useful in practice anyway.)

They don't need to. When the profiler is off the JIT can inline the check and skip the calls to it quite well.

In the current implementation of TracingProfiler/Scope I don't see any is active checks, so how would it avoid that work?
Additionally. what about AOT platforms like iOS, etc?

Yeah, no checks in the current version. I will add runtime checks / use the existing ones in Profiler/ProfilingKey. We could add a compile time way of disabling profiling entirely, if it's needed, but I'd prefer having the profiler available in all builds by default.

I think we might be able to separate the message logging from the profiler. There isn't really a good reason for it to be tied together (except being enabled when profiling is enabled) and it will simplify things a bit.
For having variants of the same profiling key attributes should be used.
And for things like the GC counts we may need a counter primitive as you mentioned instead.

I am basically done with my first pass. If the general approach is okay to merge I will add more tests and fix the comments/docs. Everything else can probably be addressed during review or in follow-up PRs.
The remaining problems are mainly around GPU events, but none of it should affect existing functionality.
Currently I map all GPU events to a single thread id, which is certainly not correct, but I also don't see an easy fix for now.
Syncing CPU and GPU timestamps also isn't working correctly, need to investigate if it's just my logic that's wrong or something else:

Then there's this:

Total time is 16 ms, but threadpool work takes 123 ms combined ;) Should I add sorting by avg time?

For having variants of the same profiling key attributes should be used. And for things like the GC counts we may need a counter primitive as you mentioned instead.

Do you have an example? I still don't think I get it.
ProfilingKeys are particularly awkward for me when profiling around polymorphism, but I don't have a suggestion how to improve it:
//Before:
using (Profiler.Begin($"{child.GetType().Name}.Draw"))
{
    child.Draw(drawContext);
}
//Good. Every subclass gets it's own key.

//Now:
using (Profiler.Begin(DrawChildKey, $"{child.GetType().Name}.Draw"))
{
    child.Draw(drawContext);
}
//Sadness.

Doing it properly now requires some Dictionary<Type, ProfilingKey>, which seems like a lot of effort for something that should be quick to add and remove again. I guess I'll do it for these base in-engine cases where it makes sense, but it's not ideal.

I think we might be able to separate the message logging from the profiler. There isn't really a good reason for it to be tied together (except being enabled when profiling is enabled) and it will simplify things a bit.

It doesn't have to be tied together, but having an easy way to log some specific (e.g. asset loading time) profiling events to a file/console is nice. I'd just like to do it without bloating all other profiling in the process.
First look at performance:

BenchmarkDotNet=v0.13.2, OS=Windows 10 (10.0.19045.3448)
AMD Ryzen 5 3600, 1 CPU, 12 logical and 6 physical cores
.NET SDK=8.0.100-preview.7.23376.3
  [Host]     : .NET 6.0.14 (6.0.1423.7309), X64 RyuJIT AVX2
  Job-MBWYOB : .NET 6.0.14 (6.0.1423.7309), X64 RyuJIT AVX2
  Job-YPSZPM : .NET 6.0.14 (6.0.1423.7309), X64 RyuJIT AVX2

Method
Job
NuGetReferences
NUM_ENTITIES
WithProfiling
Mean
Error
StdDev
Median
Allocated

Run
Job-MBWYOB
Stride.Core <IP_ADDRESS>
2000
False
6.412 ms
0.1267 ms
0.2645 ms
6.387 ms
12.08 KB

Run
Job-YPSZPM
Stride.Core <IP_ADDRESS>
2000
False
6.348 ms
0.1249 ms
0.1870 ms
6.262 ms
12.08 KB

Run
Job-MBWYOB
Stride.Core <IP_ADDRESS>
2000
True
18.580 ms
0.3451 ms
0.3059 ms
18.606 ms
12.57 KB

Run
Job-YPSZPM
Stride.Core <IP_ADDRESS>
2000
True
7.184 ms
0.1436 ms
0.2766 ms
7.052 ms
12.39 KB

<IP_ADDRESS> is master with the added profiling keys, <IP_ADDRESS> is this PR, so we can now add profiling to work happening on the threadpool without trashing performance (on desktop at least...).

For having variants of the same profiling key attributes should be used. And for things like the GC counts we may need a counter primitive as you mentioned instead.

Do you have an example? I still don't think I get it.

Probably could have been more explicit here.
Let's take this for an example
var url = newUrl ?? reference.Url;
using (var profile = Profiler.Begin(ContentProfilingKeys.ContentReload, url))
{
    DeserializeObject(reference.Url, url, obj.GetType(), obj, settings);
}

For many assets with different URLs we run the same profiling key called ContentReload. If we're interested in measuring which asset takes the longest to reload we should be looking at variants of that key with for example an attribute ("URL", url).
Whereas in this example
using (processor.UpdateProfilingState = Profiler.Begin(processor.UpdateProfilingKey, "Entities: {0}", entities.Count))
{
    processor.Update(gameTime);
}

There isn't any variation in the parameters. UpdateProfilingKey is annotated with a message on entity count.
This should be potentially a separate counter measurement in a tracing system. It's useful to correlate entity count and how long an Update takes, but the number of entities is not inherently a variation of the update process.
using (var profile = Profiler.Begin(GameProfilingKeys.GameSystemLoadContent, gameSystemName))
    contentable.LoadContent();

Another example where the parameter passed is making this a variant, because we want to distinctly identify the load time of different game systems.

So variants are basically a quick way to (!) dynamically separate a profiling key into a few subkeys.
The way the system is designed currently, there is a parent-child relationship between profiling keys.
In practice, however, that relationship is used more for turning things on and off, rather than grouping results logically.
So each profiling key results in a separate measurement entry in the buffer.
With a more OpenTelemetry like approach, a single measurement with attributes can be looked at either as a whole (not caring about variants) or with division into variant specifics (where each attribute creates a "dimension" the measurement can be partiotioned by).
Hope this makes a bit more sense.

Total time is 16 ms, but threadpool work takes 123 ms combined ;) Should I add sorting by avg time?

This is why there's a concept of a Histogram in meters. It basically looks at the data in terms of duration and allows to view it by max, min, average and percentiles.
I think adding a display mode with average aggregation over a second would be a great option!
And for the threadpool specifically wouldn't the issue also be around it running on multiple threads in parallel? Thus squeezing more compute time into the same ms?

And for the threadpool specifically wouldn't the issue also be around it running on multiple threads in parallel? Thus squeezing more compute time into the same ms?

Yes, exactly that.

This is why there's a concept of a Histogram in meters. It basically looks at the data in terms of duration and allows to view it by max, min, average and percentiles.

GameProfilingSystem is already aggregating by RefreshTime (by default KP_Plus/KP_Minus to change) and then rescaling to per-frame where it makes sense.

That should be it for now.
I've marked it as breaking change, but unless I've missed something, it would only be breaking if someone replaced GameProfilingSystem with their own solution.
Performance is okay, not amazing, but better than before, especially when capturing events from the ThreadPool.
CPU and GPU timelines are not synced correctly, but that only affects the JSON output so far. D3D12 and Vulkan have dedicated APIs for clock calibration, on older APIs  we'd need to do it ourselves.
And I imagine the option to output tracing .json and view it externally should also be mentioned in the profiling section of the docs.

I ran Starbreach project to test those changes. I'm happy to merge it.
However, you may need to look into performance of ChromeTracingProfileWriter. Having it enabled had a big impact on the system with the frame rate dropping by around 10FPS (from 95-100 to 85-90).

It seems some build are broken.
FYI, builds don't auto-run on PR from people who are not yet marked as collaborator or contributors (for security reasons).

It would be nice to mention this new ability in the documentation somewhere.