Tabbing with radio fieldsets

Summary:
Steps to reproduce:

Create a modal with a fieldset inside, where the fieldset contains at least two radio inputs and there are no focusable elements after the fieldset in the modal.
Open the modal and attempt using tab to navigate
When focus is within the fieldset, tab will move focus outside the modal. If the modal contained the final focusable element in the DOM, this means the window will blur; if it was not, then focus will move to the next element outside of it.

Expected behavior:
Focus should stay within the modal.
Link to example of issue:
Behavior when final focusable element in DOM is in modal: https://codesandbox.io/s/q4lk918jww
Behavior when focusable elements in DOM follows modal: https://codesandbox.io/s/j2jjvljp7w
Additional notes:
Radio groups have tabbing behavior that is not being accounted for by the tab intercept logic in react-modal. That logic bails and defers to the default behavior except when the active element is the final focusable element in the modal. However, radio inputs within a fieldset have keyboard behavior for accessibility with a bit more nuance. When you tab into a radio group "from above", you land on the first input, and when you tab into a radio group "from below", you land on the last input, yet within the group, you navigate with arrow keys, not tab; instead, tab exits the whole group. So when the first input is focused and you press tab, react-modal sees correctly that it is not the final focusable element, but it assumes this also means that the native tab behavior will be to go to the next input element — which is not true. Natively, the next focusable element in this situation is whatever follows the whole group. So by bailing out at this point, focus is able to escape the modal.

@bathos Check it out #521. It is a very sketch implementation and I'll refactor it before merge. Can you help me validate if this is the correct TAB behavior?


Remove now default 'sudo: false'

'sudo: false' is now default on Travis CI

@bors-servo: r+
Thanks!

:pushpin: Commit 7161a4f has been approved by jdm

:hourglass: Testing commit 7161a4f with merge b8ee011...

:sunny: Test successful - travis


Long audio delay on Quest 2

Hello ! I noticed a very long audio delay with OculusKiller (~1 second).
I'm using Quest 2 with Air Link at 200Mbps with an Asus AX5400 router.

@Skynova-FR Have you verified that this issue is only happening with OculusKiller, cause this sounds like an issue with your router...

@UnusualNorm I confirm that this issue is only happening with OculusKiller. Only the audio is delayed, not the video. I never had this issue without OculusKiller (I played every day for months now).

I faced the same issue. The feeling that the point is that the picture is transmitted to the helmet without sound, and the sound is transmitted as an additional one in accordance with the "Play sound from computer in VR" configuration in the Oculus desktop application.


making prediction

Hi,
I saw your work when I checked for the literature online. I am trying to make predictions on the protein solubility using your model. I wonder if the original models can be uploaded for making the predictions.
I also tried to train the models on my own but could not find all the necessary files for training.

Thank you for your attention to our work, we upload a predicted workflow folder and update the README.md.


rust-bindgen: 0.35.0 -> 0.36.0

Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

Success on x86_64-linux (full log)
Attempted: rust-bindgen
Partial log (click to expand)
cp: -r not specified; omitting directory 'target/release/deps'
cp: -r not specified; omitting directory 'target/release/native'
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/bbncdjk892pkdz0pwvfrw6h41gwcj1c5-rust-bindgen-0.36.0
shrinking /nix/store/bbncdjk892pkdz0pwvfrw6h41gwcj1c5-rust-bindgen-0.36.0/bin/.bindgen-wrapped
strip is /nix/store/j75dgadrff2d1fyc4fczmcgqkid2imdx-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/bbncdjk892pkdz0pwvfrw6h41gwcj1c5-rust-bindgen-0.36.0/bin
patching script interpreter paths in /nix/store/bbncdjk892pkdz0pwvfrw6h41gwcj1c5-rust-bindgen-0.36.0
checking for references to /build in /nix/store/bbncdjk892pkdz0pwvfrw6h41gwcj1c5-rust-bindgen-0.36.0...
/nix/store/bbncdjk892pkdz0pwvfrw6h41gwcj1c5-rust-bindgen-0.36.0

Failure on aarch64-linux (full log)
Attempted: rust-bindgen
Partial log (click to expand)
cannot build derivation '/nix/store/pyg2w3yj1aywni78jjj4jk6wxvbw706y-rust_serde_derive-1.0.18.drv': 6 dependencies couldn't be built
cannot build derivation '/nix/store/wh8v1y72n99c6sid6h4s6blmm6yd2gr8-rust_crates-io-0.11.0.drv': 30 dependencies couldn't be built
cannot build derivation '/nix/store/5mvqzzqd2lqc4qap0mcrp7p38f0cm6hq-rust_docopt-0.8.1.drv': 19 dependencies couldn't be built
cannot build derivation '/nix/store/v1bmm0z1yvajs6zmlcz3jnz0xifd3qmg-rust_git2-curl-0.7.0.drv': 20 dependencies couldn't be built
cannot build derivation '/nix/store/211ykpbhf7wljp94c408b61rsd5cn69y-rust_ignore-0.2.2.drv': 18 dependencies couldn't be built
cannot build derivation '/nix/store/3g1afpn8ylzk8vfnpag751hbn42b5kxr-rust_cargo-0.22.0.drv': 78 dependencies couldn't be built
cannot build derivation '/nix/store/6kfkgfripry5s4999bwfqis3fppanfsh-rust_cargo-vendor-0.1.13.drv': 79 dependencies couldn't be built
cannot build derivation '/nix/store/wmnrx0alk0zk8lpl86yjy3qlkqdqiiv3-rust-bindgen-0.36.0-vendor.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/nmzy9mdzj70sddrw1mi2fi72sglalqnf-rust-bindgen-0.36.0.drv': 3 dependencies couldn't be built
error: build of '/nix/store/nmzy9mdzj70sddrw1mi2fi72sglalqnf-rust-bindgen-0.36.0.drv' failed


[Feature Request] Resume downloads? Is it possible?

I was wondering if it could be possible to allow the program to resume downloads instead of deleting it. I tried to look in the files, but I couldn't figure out what it uses to download files, how it does it, or if this is possible.
I am asking because I was having problems downloading my purchase. It was way too slow and it kept failing to download yesterday, both in the browser and also this program. I was using this program hoping it would have this feature, but it doesn't.
Thankfully, it got fixed today. I guess I just had to wait or maybe it's because I set max-retries to 100. Either way, it would be nice if this program had this feature.

resume downloading is not supported whatever version of AWS they are using doesn't support a range header for file downloads.


Proofread and improve the whole class reference

I finished proofreading the entire class reference, so here it is :slightly_smiling_face:

Document a few more properties and methods
Add more information to many classes
Fix lots of typos and gramar mistakes
Use [code] tags for parameters consistently
Use [b] and [i] tags consistently
Put "Warning:" and "Note:" on their own line to be more visible, and make them always bold
Tweak formatting in code examples to be more readable
Use double quotes consistently
Add more links to third-party technologies

Awesome work! There's a lot of content to review, but I'll merge it as is already to prevent having too many merge conflicts with current and future documentation PRs.
Please continue reviewing post-merge anyway, and further edits can be made in another PR.

Thanks!


chore: forward port of #18742, no simps lemmas for Category.Hom

This is the forward port of https://github.com/leanprover-community/mathlib/pull/18742. That PR hasn't landed yet, so this PR still needs to be updated with the new commit SHA.

Cool, potentially this could help in https://github.com/leanprover-community/mathlib4/pull/2482

bors merge


Update injection_script.js

Bypasses for some sites requiring several "next" clicks like forexmab.com, forexlap.com, forex-articles.com.
Tested and working fine.
 NOTICE
I dedicate any and all copyright interest in this software to the
public domain. I make this dedication for the benefit of the public at
large and to the detriment of my heirs and successors. I intend this
dedication to be an overt act of relinquishment in perpetuity of all
present and future rights to this software under copyright law.

Fixes: Link to issue

Added 3 bypasses to sites which require one to click a link at the top of page to send us to bottom and click another NEXT link. These 3 sites are often chained together.
Example: http://linkjust.com/ackNeTSd

[X] I made sure there are no unnecessary changes in the code*
[X] Tested on Chromium- Browser OS
[ ] Tested on Firefox

* indicates required

So what's next ? who's going to incorporate this patch in the update ? Forgive my ignorance.


Refactor project tree

This PR :

rename the scripts folder into lib
creates a views folder to hold all the views of our application. A view is basically a high level component that you import in views/routes.ts
moves routes.ts and router.ts from scripts to views since they are related to views
delete the Main.vue component. This component does nothing and it's neither an atomic component nor a real view. So...

Do you suggest any other change ?

Current dependencies on/for this PR:

main

PR #9   👈

This comment was auto-generated by Graphite.

I have a suggestion for the tree:
Instead of
...
|__views
   |__Home.vue
   |__router.ts
   |__routes.ts

we can do:
...
|__views
|  |__Home.vue
|  |__Contact.vue
|  |__ ...
|__router
   |__index.ts (router.ts + routes.ts)

As here

I knew you would said that. I was going to suggest the same except that I don't want a router folder lol. How about we do as you suggested but we use a single file called router.ts ?


"Not a valid add-on repository

Hi! I am new to HAOS, and I have tried to download by typing in the exact repository link, and it shows the error in the image below (If it cannot be seen, it says "https://github.com/schmittx/home-assistant-eero/ is not a valid add-on repository"

This is a custom component and repository. You can manually place the files in the custom_components folder, or Use HACS. Follow the instructions for installing HACS first (link), and then add this repo as a custom repo within HACS (link).


Wrongly bump other package's version if they're in same file and version number are same with current package version

If current_version same with a requirement package's version in a meta file(like setup.py), bumpversion will wrongly bump requirments's version.
In following example, "package>=0.0.1" shouldn't be changed:
(yasaka)mitsukeru@Miku:~/project/test-proj$ bumpversion patch --dry-run --verbose
Reading config file .bumpversion.cfg:
[bumpversion]
files = setup.py test-proj/__init__.py
commit = True
tag = True
current_version = 0.0.1

Parsing version '0.0.1' using regexp '(?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)'
Parsed the following values: major=0, minor=0, patch=1
Attempting to increment part 'patch'
Values are now: major=0, minor=0, patch=2
Dry run active, won't touch any files.
New version will be '0.0.2'
Asserting files setup.py, test-proj/__init__.py contain the version string:
Found '0.0.1' in setup.py at line 11:     "package>=0.0.1"
Found '0.0.1' in test-proj/__init__.py at line 2: __version__ = '0.0.1'
Would change file setup.py:
--- a/setup.py
+++ b/setup.py
@@ -9,7 +9,7 @@
 readme = open('README.rst').read()

 requirements = [
-    "package>=0.0.1"
+    "package>=0.0.2"
     # TODO: put package requirements here
 ]

@@ -19,7 +19,7 @@

 setup(
     name='test-proj',
-    version='0.0.1',
+    version='0.0.2',
     description='feature generators used inside BosonData.',
     long_description=readme,
     author='gestapolur',
Would change file test-proj/__init__.py:
--- a/test-proj/__init__.py
+++ b/test-proj/__init__.py
@@ -1,3 +1,3 @@
 # -*- coding: utf-8 -*-

-__version__ = '0.0.1'
+__version__ = '0.0.2'
Would write to config file .bumpversion.cfg:
[bumpversion]
files = setup.py test-proj/__init__.py
commit = True
tag = True
current_version = 0.0.2

So, bumpversion is agnostic to the file format by design (i.e. it doesn't know about what version is a dependency and which is the software that is versioned).
Does https://github.com/peritus/bumpversion/issues/76#issuecomment-75938030 work for you or is this about something else ?

@peritus Not really, I edited my .bumpversion.cfg like this:
[bumpversion]
files = setup.py
commit = True
tag = True
current_version = 0.0.1
[bumpversion:file:setup.py]
search = version='{current_version}',
replace = version='{new_version}',

but it still did change like this:
 requirements = [
-    "package>=0.0.1"
+    "package>=0.0.2"
     # TODO: put package requirements here
 ]

@@ -19,7 +19,7 @@

 setup(
     name='test-proj',
-    version='0.0.1',
+    version='0.0.2',
     description='blablabla',
     long_description=readme,

Strange. What if you remove
files = setup.py

?

Oh, it works somehow:
(yasaka)mitsukeru@Miku:~/project/test-proj$ bumpversion patch --dry-run --verbose
Reading config file .bumpversion.cfg:
[bumpversion]
commit = True
tag = True
current_version = 0.0.1
[bumpversion:file:setup.py]
search = version='{current_version}',
replace = version='{new_version}',

Parsing version '0.0.1' using regexp '(?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)'
Parsed the following values: major=0, minor=0, patch=1
Attempting to increment part 'patch'
Values are now: major=0, minor=0, patch=2
Dry run active, won't touch any files.
New version will be '0.0.2'
Asserting files setup.py contain the version string:
Found 'version='0.0.1',' in setup.py at line 21:     version='0.0.1',
Would change file setup.py:
--- a/setup.py
+++ b/setup.py
@@ -19,7 +19,7 @@

 setup(
     name='test-proj',
-    version='0.0.1',
+    version='0.0.2',
     description='blablabla',
     long_description=readme,
     author='gestapolur',
Would write to config file .bumpversion.cfg:
[bumpversion]
commit = True
tag = True
current_version = 0.0.2

[bumpversion:file:setup.py]
search = version='{current_version}',
replace = version='{new_version}',

Would prepare Git commit
Would add changes in file 'setup.py' to Git
Would add changes in file '.bumpversion.cfg' to Git
Would commit to Git with message 'Bump version: 0.0.1 → 0.0.2'
Would tag 'v0.0.2' in Git

But such configure seems werid.

But such configure seems werid.

I'm committed to making bumpversion as simple as possible (though, not simpler). I'd like your opinion on what could be done better (one action point for me is pointing this out in the docs because you're not the first person to ask about this since it was implemented).
One idea would be to make bumpversion language-specific (so that it knows about setup.py or setup.cfg and to differentiate between deps and versions of the package at hand), but then again there are a few hundred other software package specifications.

Also happy for you that it works :)


Build errors

Hi, nail looks really interesting, but I'm having problems building. They seem to be "environmental". I'm happy to split this into multiple issues if that's better for you.
The first problems are the implicit dependencies on boost and hammer. I worked around them by soft linking boost -> /opt/local/include/boost and hammer -> ../hammer/src/ in the workspace root. From what I can tell the hammer dependency is unneeded, but still included by grammer.c and parser_test.c.
The problem I haven't worked out is that I get error from the way ld used for parser_template, like so:
ld -r -o parser_template.c.o --format=binary parser_template.c
ld: unknown option: --format=binary
make: *** [parser_template.c.o] Error 1

A reasonable amount of googling hasn't turned up what version of ld support --format. I'm running on OSX Mavericks 10.9.5. Ld version says:
$ ld -v
@(#)PROGRAM:ld  PROJECT:ld64-241.9
configured to support archs: armv6 armv7 armv7s arm64 i386 x86_64 x86_64h armv6m armv7m armv7em
LTO support using: LLVM version 3.5svn

I'm happy to help any way I can (splitting issues, pull requests, etc.) Thanks.

Hey! Sorry for the late response, I've been off at a conference. I pushed a few new changes (unrelated, halfway done code in /network, and the new C++ backend - use the old C style Nail with ./cnail ).
There is a dependency on boost (I use Arch linux and ubuntu, and both support system wide installs of boost), and removed the (last remaining, code-smelly and unused) include of hammer -- let me know if it still does not work.
As to ld, what ld --format=binary does is create an object file that has a single symbol which contains the file given as an argument - which was the quickest way at the time to include the parser templates as a string. When I have a moment, I'll look into a way to do this more portably - probably using XXD and inc files...

hey ,
i get this error , ld -r -o parser_template.c.o --format=binary parser_template.c
ld: unknown option: --format=binary
make: *** [parser_template.c.o] Error 1
my ld -v shows
@(#)PROGRAM:ld  PROJECT:ld64-236.3
configured to support archs: i386 x86_64 x86_64h armv6 armv7 armv7s armv7m arm64
LTO support using: LLVM version 3.5.2
let me know how to proceed .
Thanks

@gopi-pmk
This error seems to be caused by using the Mac OS X compiler instead of GCC. I haven't worked around it myself. Julian says above that he plans to make a portable way to handle it.
In the meantime, you could try installing GCC and building with it. I can't give steps to do that other than google.

It took me a while to get around to the second half of the issue -- see the last few commits I pushed. I now embed the templates with XXD as opposed to LD (which turns out to be a GNU ld + ELF specific feature).

Thank you guys , compilation worked !! will try out some examples and see how it goes.


fix(website): Fix typo

website/docs/api/misc/logger/logger.md:32:45

Pre-flight checklist

[x] I have read the Contributing Guidelines on pull requests.
[ ] If this is a code change: I have written unit tests and/or added dogfooding pages to fully verify the new behavior.
[ ] If this is a new API or substantial change: the PR has an accompanying issue (closes #0000) and the maintainers have approved on my working plan.

Motivation

Fixing typo
Test Plan

Manual test
Test links

Deploy preview: https://deploy-preview-8162--docusaurus-2.netlify.app/docs/api/misc/@docusaurus/logger/
Related issues/PRs

Preview page: https://deploy-preview-8162--docusaurus-2.netlify.app/docs/api/misc/@docusaurus/logger/


RTL, clear screen (with no icon) support

It will be helpful to have those option also:

Adding a screen without presence of an icon - just to show the text messenge scrolling.
icon will scroll also as an option how to present a messenge
Support for Right to Left scroll (not just left to right direction)
Ability to import icons with bitmap code
Ability to rainbow the color text (per word or character)

And one last thing if you can add on the readme example how to use the draw_clock service.

Hi,
Adding a screen without presence of an icon - just to show the text messenge scrolling
this will come in 20234.4.0
icon will scroll also as an option how to present a messenge
not yet planned
Support for Right to Left scroll (not just left to right direction)
good idea i will consider this
Ability to import icons with bitmap code
good idea but hard to implement in esphome, since you can access the displaybuffer directly
Ability to rainbow the color text (per word or character)
this is not planned
And one last thing if you can add on the readme example how to use the draw_clock service.
in 2023.5.0 this will change and will be much clearer, now you should not use draw_clock directly

Hi,
"Support for Right to Left scroll (not just left to right direction)"
When i change the scroll direction should the icon also be on the other side? I think this is a request for arab language?

For my use case I prefer the icon as it is on the left.  But, it's  a nice feature to have the ability to choose for each screen where to place the icon.
BTW,  will the new version plan to have the ability to create a full screen with text only (no icon)?

"BTW, will the new version plan to have the ability to create a full screen with text only (no icon)?"
Yes, of course


ALT_SEARCH_KENDRA_FALLBACK_CONFIDENCE_SCORE not honored when LLM_QA_USE_KENDRA_RETRIEVAL_API=true

Describe the bug
When leveraging the Kendra fallback index, and use of the Kendra Retrieval API is enabled, it appears that the ALT_SEARCH_KENDRA_FALLBACK_CONFIDENCE_SCORE setting is not honored as a threshold. To use
To Reproduce

LLM_QA_ENABLE = true
LLM_QA_USE_KENDRA_RETRIEVAL_API = true
ALT_SEARCH_KENDRA_FALLBACK_CONFIDENCE_SCORE = VERY_HIGH
Perform a search that should result in a fallback to Kendra, and verify that only documents with a VERY_HIGH confidence are included

Expected behavior
The ALT_SEARCH_KENDRA_FALLBACK_CONFIDENCE_SCORE setting should be honored whether the Kendra Query API is used OR the Kendra Retrieve API. I could not find documentation stating that this otherwise would not work.
Please complete the following information about the solution:

Version: 5.4.3
Region: us-east-1
Was the solution modified from the version published on this repository? No
Have you checked your service quotas for the services this solution uses? Yes
Were there any errors in the CloudWatch Logs? None of relevance

Thanks for reporting this. Let us take a look and get back to you.

Hi @shieldsjared - currently QnABot does not filter results using the Kendra Retrieval API. I can update the documentation to clarify where the ALT_SEARCH_KENDRA_FALLBACK_CONFIDENCE_SCORE is used. With that being said, will this solve your problem? Do you still want a mechanism to filter Kendra retrieval results?

We will update the documentation during the next release. If you want a new setting for filtering Kendra Retrieval results then please submit a feature request, since this functionality does not currently exist.


Spring Boot Starter for Azure AD

For a SPA application the previous recommendation was to use OAuth implicit flow
(https://docs.microsoft.com/... AND
https://azure.microsoft.com....
If Spring Boot Starter for Azure AD is used is OAuth implicit flow no longer recommended when registering your application with your Azure Active Directory tenant?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: be0884d3-263f-9913-7b8c-170d8092baa2
Version Independent ID: 42c5b06a-81a7-97f7-a4cf-7ca77a30ec37
Content: Secure single page applications using the Azure AD v2.0 implicit flow
Content Source: articles/active-directory/develop/active-directory-v2-protocols-implicit.md
Service: active-directory
Product: unspecified
GitHub Login: @CelesteDG
Microsoft Alias: celested

@robert-cleric Thanks for bringing this to our attention. Your feedback has been shared with the content owner for further review.

Hello @robert-cleric - Thank you for your comment. I checked with our feature team and received the following info: SpringBoot Azure AD has the user generate a client secret, so it’s not using implicit flow and isn’t a “true” SPA – it still has a backing service, it still handles routes, etc. it’s doing back-end auth.
Please let us know if you have any follow up comments or questions. Thank you.

@femsulu @MohitGargMSFT @SaurabhSharma-MSFT No follow up questions from customer and no changes made to the doc. #please-close


Default colors need improvement

Using asciidoctor-epub3 1.5.0.alpha.17
Contrary to the claims in the readme, the colors of the default theme don't always give good results across the different viewing modes. In some cases, text can be completely unreadable. Here's a few observations with the current stylesheet on some apps/devices:

For best readability, when viewing with white background the main text should ideally be black (0x000000). This is what you will typically see with commercially produced ebooks (as far as I've been able to tell). The current default for asciidoctor-epub3 appears to be 0x333332 instead, which is dark but not as dark as it should be. This was one of the first things a reader noticed on Kindle (text not as dark and clear as other ebooks). I see the same in calibre and iBooks on a mac desktop, so I presume it would be the same everywhere. Since this is likely to be the default/most commonly used viewing mode, it's an important one to give the optimal readability.

For black background (Night mode), the text in calibre is still quite dark and hard to read. On iBooks, it's a good contrast and doesn't have the same issue, so the problem seems to be reader-specific.

For sepia dark, on calibre the text color is the same as the background color, so it disappears completely! Again, iBooks is okay for sepia dark.

FWIW, the default theme for asciidoctor-pdf has a similar issue for the white background case, but asciidoctor-pdf supports being able to import the default theme and override only the bits you want. There is no equivalent functionality for asciidoctor-epub3 releases, so the only alternative is to copy the entire stylesheet (all three files). That means you'd also miss any changes to these stylesheets that are made from one release of asciidoctor-epub3 to another, so it is not an ideal workaround.

I believe that Calibre is not a good tool for testing dark/night modes. If I change text color from #333332 to #000000 and enable "Black" theme in Calibre, text remains black, so it is invisible on black background. Calibre has "Override all book colors = In dark mode" that fixes text color in both "Black" and "Dark sepia" modes, however all text coloring is lost for obvious reasons.
So, while general thesis that asciidoctor-epub3 text isn't black enough still might be true, changing it to completely black does not fix readability across all Calibre modes and only changes "text is unreadable in dark sepia mode" into "text is unreadable in black mode".

I think the solution is to not set the font color. The reason we had set it before is because we did not want black.
Once again, the readers are supposed to be changing the colors uniformly, but we've learned that we cannot trust that contract.

@mojavelinux Hmm. Avoiding font colors at all is a rather hard restriction. We have syntax highlighting, colored icons, shades of grey for various things... I don't quite understand where is the boundary between coloring and not-coloring.

I meant for the main text. The other stuff is out of our control since you're right, we need to use colors.

This is how our README looks in Calibre after I removed all #333332 from CSS  and turned on dark sepia theme.

It is simply awful. I'm still not convinced whether this mode should influence our decisions at all, maybe it's just broken. I'll take a look at how other readers handle night/dark modes.

And black theme is not going to work too. I dunno what they're thinking about. It is impossible to pick colors that would look OK on both white and black background.

My spouse uses Calibre to read books. In her experience, the dark mode is totally broken. So I don't think this should be driving decisions. I think iBooks or Kindle should be our main reference.

@mojavelinux So. Do we want to only change main text color or to rescale all our palette into darker shades? Theoretically thinking, the latter should make more sense.

I think it's both. All colors that are used likely need to be darker, but the main text shouldn't have a color defined (which will default to black). But its never going to be perfect because EPUB 3 readers are imperfect.

Related: #337, #67.

For best readability, when viewing with white background the main text should ideally be black (0x000000).

Funnily, GitHub uses #24292e for main text.

So. I've created #347 that changes asciidoctor-epub3 to a darker palette.


Port config-reloader logic back into prometheus-config-reloader

We currently have two sidecars for each Prometheus Pod.

Prometheus config reloader: Templating and reloading Prometheus configuration
Rule config map reloader: Reloading rule configuration

The logic of the latter should be merged back into the former, reducing the amount of sidecars by one.

With https://github.com/coreos/prometheus-operator/pull/3128 merged, this should now be trivial to configure with one container.

Done in #3457


Fix minor mistakes in AndroidScopes documentation

Android scoping has been entirely rewritten.
This PR is no longer relevant.
Thanks anyways, and sorry :(


fix: do not panic when failed to parse rustc commit-hash

What does this PR try to resolve?
In some situation the commit-hash in rustc -vV output is "unknown".
The debug assertion must not block any progress on others like miri,
How should we test and review this PR?
See https://github.com/rust-lang/miri/issues/3160

@bors r+

:pushpin: Commit 7ca122c00947c15169a97bd881350a9bae16f7a9 has been approved by epage
It is now in the queue for this repository.

:hourglass: Testing commit 7ca122c00947c15169a97bd881350a9bae16f7a9 with merge 4e02f11faf4dd75e3c39c237b59ca94016ca7f0e...

:sunny: Test successful - checks-actions
Approved by: epage
Pushing 4e02f11faf4dd75e3c39c237b59ca94016ca7f0e to master...


Upgrade email-encoding to v0.3

Fixes some technical debt plus includes the fix for #949

(this should have been a DRAFT PR since we're using the git dependency until v0.3 is released)


Created/updated timestamps are saved in JST

To support overseas client, it should be UTC.
For example, in Wakaba Client App's  lm.isExpired, we tried to compare the expiry date of the timestamp with current time. If the application is executed in machine that use US time, we will have the wrong result.

I think it is saved in UTC format internally.  Probably you are talking about PROPFIND result.
PROPFIND response rendering somehow converts to local time format.

https://github.com/personium/personium-core/blob/93105d9491c6366e3c6f2686034ebc9bee204f33/src/main/java/io/personium/core/model/DavRsCmp.java#L506
Plz see above, Yes, It is handled in unix time interanally. but org.apache.wink.webdav.model.Getlastmodified automatically change to local time format.
I remember we took i18n (timezone) in consideration on initial implementation, also.
We found out that wink library automatically converts to local date and could not find the way to force it to return in utc time.  And considering the fact that it still returns timezone info in the format, we gave up and made it clients' responsibility to properly handle the info obtained from the server (time w/ timezone info) and present to user in preferable way (covert in accordance with client timezone)

Sorry i am not the one to close this issue.
Dixonsiu, if you are satisfied with the above answer, plz close this issue.

Which machine's local time do you refer to?

Personium Unit
User PC

Correct me if I am wrong. We will have problem with the following cases.
Personium Unit using Japan UTC and convert to Japan local time when replying to PROFIND. However, user is in London and the PC's time is London time. There will be problem comparing London local time to Japan local time.


Bootstrap carousel

First slide on BS4 carousel gets lazy loaded just fine. Other images not. (You can inspect elements and see that the first image is data-loaded="true", other images not)
See https://codepen.io/kickbk/pen/qyLOeN
How do we apply lazy loading to images in slides when they enter into view?

You're only selecting the first img with querySelector in your JS, you should be all set just changing that to querySelectorAll as in the example.


Added missing variable in code sample.

See https://github.com/cloudcreativity/laravel-json-api/issues/257.

Thanks!


Question: is it possible to transform the already cloned object?

Hey,
Sorry if it's already mentioned in the docs, but I don't understand how to achieve what I want.
I have normalized data in my store:

a list of enemy
a list of spell

Each enemy has 2 spells, so I'm trying to do a simple join:
const selectSpells = (enemy, state) => (
  select([
    'spells',
    $each,
    $apply(({ key }) => (
      find(['data', 'spells', key], state)
    ))
  ], enemy)
)

Until this point everything work as expected, I've got enemies having their spells filled in instead of ids. Everything has been cloned to I'm not mutating the store.
Each spell has a predefined cooldown and each enemy has a cdr attribute. The cdr is simply a ratio that lowers the predefined cooldown. I would like to apply this transformation so each user has its spells with the adjusted cooldown.
Here is one of my attempts:
select([
    'spells',
    $each,
    $apply(({ key }) => (
      find(['data', 'spells', key], state)
    )),
    $apply(spell => {
      spell.cooldown *= (1 - ennemy.cdr)
      return spell
    })
  ], enemy)

No matter what I try, my normalized data is mutated, so the initial predefined cooldown for each spells gets mutated.
So my question is, is it possible to $apply on the result object (already cloned) instead of the source one? I reckon that I'm probably missing something somewhere :)
Thanks for the help.

Hey @ngryman, thanks for trying out qim! :-) Looks like you started composing things together which is cool, but you just have to go one step further:
select([
  'spells',
  $each,
  $apply(({ key }) => (
    find(['data', 'spells', key], state)
  )),
  $apply(
    update(['cooldown', $apply(cooldown => cooldown * (1 - enemy.cdr))])
  )
], enemy)

I'm using a curried version of update there. Note there's no second parameter, so it just returns a function that takes data. I plug that into the $apply so it gets used to update the spell immutably.

@jdeal Thanks! Now I see your solution I feel dumb :) I expected the whole input object to be clones somewhere in between the 2 $apply, but I guess that makes sense. Thanks for your time :)
Oh and all functions of qim are curryable? 👏

@ngryman Ah, yeah, cloning only happens when an object is mutated (by qim). To allow for mutations like that, we'd have to deeply clone, which would be expensive and remove any ability to do equality comparisons on unchanged branches.
And yep, all functions listed in the API section are curried.

@jdeal 👏 Oh then it would be awesome to highlight it more because I didn't get it while reading the docs.
Not necessary the right place to ask about that (if you have a better place tell me), I have 2 questions for you:

Do you plan to add memoization? I mainly use qim as a selector to compute derived data, and similar libraries (like reselect) offer this feature.

This library is really neat and navigators is a truly powerful tool, but they impose a kind of "painful" API why may turn some new users away (I know that's silly but...). The main pain-point would be path defined by arrays. It makes perfect sense but it's also sometimes painful to use, error prone, and make the code harder to read.

Would you consider also supporting var args so it becomes:
select('path', 'to', $nav, 'property', state)

Added a quick note to the readme about currying, for now.
@ngryman Do you mind adding those questions as two separate issues?


Don't assume code is always "stripe"

Corresponding change coming to the theme file

The data-trigger-condition on #stripePaymentForm needs changed to have value of ="value[code; ?>]", but it seems you've updated the approach to this?

I don't think this is a necessary change as the process.stripe.js should only work for the stripe payment. value=stripe should work fine except you save the payment with a different code in the admin.

Yep that’s our use case.
We have a restaurant group with multiple locations - each with a different stripe account but under the same TI instance.
Seems like something we should cover off as it’s a small fix?

On 27 May 2020, at 20:18, Sam Poyigi<EMAIL_ADDRESS>wrote:

I don't think this is a necessary change as the process.stripe.js should only work for the stripe payment. value=stripe should work fine except you save the payment with a different code in the admin.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

I've been able to use the current payment form with Stripe Connect by listening on the 'payregister.stripe.extendFields` event, you could follow the same approach if that's what you're trying.
Hardcoding the payment code into the partials and JS files was intentional to force developers to either extend the partials or JS file when extending a payment to avoid conflicting issues. The idea is that each payment gateway has a single unique payment code, instead of having multiple payment codes link to a single payment gateway.
I know it seems a small fix and doesn't change much but I'd rather leave it since its working as expected.

No problem - thanks for the explanation.

On 27 May 2020, at 21:19, Sam Poyigi<EMAIL_ADDRESS>wrote:
I've been able to use the current payment form with Stripe Connect by listening on the 'payregister.stripe.extendFields` event, you could follow the same approach if that's what you're trying.
Hardcoding the payment code into the partials and JS files was intentional to force developers to either extend the partials or JS file when extending a payment to avoid conflicting issues. The idea is that each payment gateway has a single unique payment code, instead of having multiple payment codes link to a single payment gateway.
I know it seems a small fix and doesn't change much but I'd rather leave it since its working as expected.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/tastyigniter/ti-ext-payregister/pull/4#issuecomment-634917151, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAMVO2RZ3E6H3A7BNJPAM3RTVYUHANCNFSM4NL2IWBA.


Support cancellation in the Http2StreamChannelBootstrap

Motivation:
Right now you can cancel the Future returned by
Http2StreamChannelBootstrap.open() and that will race with the
registration of the stream channel with the event loop, potentially
culminating in an IllegalStateException and potential resource leak.
Modification:
Ensure that the returned promise is uncancellable.
Result:
Should no longer see IllegalStateExceptions.

@netty-bot test this please

@bryce-anderson good catch!

@bryce-anderson thanks!


Fix bug in recent StringBuilder perf improvement

cc @bbowyersmyth

I appreciate that @jkotas, thanks for the fix


Integrate invenio-stats into invenio-app-rdm

[ ] how to handle cleaning of indices (to avoid growing the cluster)
[x] investigate: replace the signals: do we use the message bus? If yes, it is a pre-requisites. Or, where to hook in? part of uow? part of the resource/service?
[x] Verify the latest version of OpenSearch if there is any change how aggregation works, comparing to ES7
[ ]  can we exclude admins/curators/owners from the stats calculation? how do we pass the current identity to the service correctly?

Investigate:

[ ] migration to Counter v5
[ ] adding more aggregations (country of visit, others)
Nice to have:
statistics per community (can be done as a part of the MS5+)

frontend mockups:
https://github.com/inveniosoftware/invenio-stats/issues/120

Regarding the event bus as replacement for signals: According to last Friday's discussion, the event bus isn't as high priority as the general integration of the Invenio-Stats package.
As such, we will for now emit the relevant events right after the UoW committed (or as a post-action service component).
This means we're essentially skipping one step of event handlers for now, and shouldn't be too hard to add later.

Stats events & event bus: discussion recap
A brief summary of our discussions in Discord (February 7th, 2023) regarding the potentially upcoming event bus and the stats events:
Where should events be emitted?
While the units of work (uow) for service methods would be a somewhat natural candidate for emitting events (via a kind of TaskOp), it should be noted that not all service methods have uows right now: Read operations don't.
Adding units of work to read operations just for the sake of having a nice spot to emit events seems bad practice.
We don't intend to go down that route until there's further reasons for a uow there.
Also, the question of who emitted the event is less important than the fact that it is being emitted – where it's emitted depends on the event itself.
For the initial integration of statistics events, we simply go for direct event emission in the resources and view functions.
Should we count events for all endpoints?
There's multiple endpoints that touch the "reading" of a record (similar for file downloads), including:

UI landing page
UI export of record in various metadata schemas
API endpoint for record metadata
etc.

Collecting "record view" events for the UI seems like a no-brainer, while it's questionable whether we'd wanna count API endpoint calls.
For file downloads however, both UI and API need to be counted.
We decided to limit the record-view event emission to the landing page UI (view function in app-rdm) and the "read record" API endpoint (resource in records-resources) for now.
Events from these sources can be distinguished by a boolean flag in the event, called via_api.
This way, we can track record views from both the UI and the API, and filter out unwanted reads when querying for the statistics.
Similar for file downloads: resources in app-rdm and view functions in records-resources, distinguished by via_api.
Single-Page Applications: out of scope (for now)
If some third party writes an SPA as an alternative InvenioRDM frontend, we don't have control over how that SPA interacts with the Invenio's REST API (e.g. they might cache API results, and thus "falsify" statistics, etc.).
These considerations are out of scope for now.

The OpenSearch changelog doesn't mention any changes regarding to aggregations, and I haven't noticed anything weird either.


RouteLocator to use object instead of string literal for matching route.

This is to open ways for fine-grained route matching instead of only
path. e.g. matching route based on path and http method or adding
header values into the mix in the future. This will also give the ability
to separate out GET, PUT, POST etc and scale accordingly if needed.
Also, header values can be added to the mix later to route to
different endpoints.

Codecov Report

Merging #2169 into master will increase coverage by 0.05%.
The diff coverage is 91.42%.

@@            Coverage Diff             @@
##           master    #2169      +/-   ##
==========================================
+ Coverage   70.37%   70.43%   +0.05%
==========================================
  Files         229      230       +1
  Lines        7467     7484      +17
  Branches      933      933
==========================================
+ Hits         5255     5271      +16
  Misses       1838     1838
- Partials      374      375       +1

Impacted Files
Coverage Δ

...ingframework/cloud/netflix/zuul/filters/Route.java
96.55% <100%> (+0.25%)
:arrow_up:

.../netflix/zuul/filters/pre/PreDecorationFilter.java
88.88% <100%> (+0.19%)
:arrow_up:

...tflix/zuul/filters/post/LocationRewriteFilter.java
89.09% <100%> (ø)
:arrow_up:

...ud/netflix/zuul/filters/CompositeRouteLocator.java
100% <100%> (ø)
:arrow_up:

...ork/cloud/netflix/zuul/filters/ZuulProperties.java
88.18% <80%> (+0.18%)
:arrow_up:

...cloud/netflix/zuul/filters/SimpleRouteLocator.java
81.91% <88.88%> (ø)
:arrow_up:

...ork/cloud/netflix/zuul/filters/RequestedRoute.java
90.9% <90.9%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e5b14e4...8506bd7. Read the comment docs.

Can you provide some examples of how this will be used?

Sure, this is the first step to enable a fine-grained route matching - planning to do changes in small PRs to keep it simple.
Next step will be:

In SimpleRouteLocator, use bean for PathMatcher so matching logic can be customized.

With the above in place, we can

have the ability to limit routing to back-end for specific http methods.

zuul:
  routes:
    user-service:
      path: /user/**
      # Only GET and PUT are allowed
      methods: GET, PUT
      url: http://localhost:8080

have the ability to route to different endpoints based on http methods.

zuul:
  routes:
    user-service-read:
      path: /user/**
      # Only GET is allowed
      methods: GET
      url: http://localhost:8080
    user-service-write:
      path: /user/**
      # Only PUT and POST are allowed
      methods: PUT, POST
      url: http://localhost:8081

More complex example would be to route based on header values.

zuul:
  routes:
    user-service-read-debug-server:
      path: /user/**
      # Only GET is allowed
      methods: GET
      header:
        debugserver: true
      url: http://localhost:8090
    user-service-read:
      path: /user/**
      # Only GET is allowed
      methods: GET
      url: http://localhost:8080

Also, JWTs can be used to further filter the routing but don't want to complicate it too much at this point - that is for another day.
Hope the examples above make sense, let me know if you want me to put more examples.
Cheers!

Thanks.  I have labeled this as "for team discussion" so I can get the opinions of others.  We will probably talk about it later this week.
In the mean time can you add some documentation about how this will be used to the docs?  Also can you merge in the latest changes from master into your branch, it should fix the circleci build.

Thanks. The circleci build is passing now.

Hi @ryanjbaxter,
About the docs, I still need to do the second part of the implementation. I was thinking of doing that in another PR, the docs would not make much sense without that. Do you still want me to the docs now or I can do them with the second PR?
Or, I can do the full change in this PR - a complete set with the docs?
Your views?

After discussing with the team today, the general consensus was that we were in favor of this functionality :)
The one thing we wanted to see as far as an implementation is to create a new interface instead of changing RouteLocator.

Thanks @ryanjbaxter . I'll do a new interface but could you please elaborate a bit more on that. The idea was to get it working in SimpleRouteLocator hence the change in RouteLocator. How do you see the new interface working with SimpleRouteLocator? Just have it extend the new interface or do you have something else in mind?

If you extend RouteLocator and add the new method and then have SimpleRouteLocator implement that new interface shouldnt that work?

I don't think SimpleRouteLocator needs to be involved necessarily. The new interface should use the actual Request rather than an object that gathers information from a request.

I like the idea of using the actual Request and think using RequestContext will give more freedom. It will allow to even do some manipulation (if needed) before the request matching happens.

By the way, there is another matching happening in ZuulHandlerMapping.lookupHandler - PatternMatchUtils.simpleMatch(ignored, urlPath) - this one is just a check though
@Override
	protected Object lookupHandler(String urlPath, HttpServletRequest request) throws Exception {
		if (this.errorController != null && urlPath.equals(this.errorController.getErrorPath())) {
			return null;
		}
		String[] ignored = this.routeLocator.getIgnoredPaths().toArray(new String[0]);
		if (PatternMatchUtils.simpleMatch(ignored, urlPath)) {
			return null;
		}
		RequestContext ctx = RequestContext.getCurrentContext();
		if (ctx.containsKey("forward.to")) {
			return null;
		}
		if (this.dirty) {
			synchronized (this) {
				if (this.dirty) {
					registerHandlers();
					this.dirty = false;
				}
			}
		}
		return super.lookupHandler(urlPath, request);
	}

Hi guys,
I am done for now, have updated the docs as well. Could anyone please review and see if I've made a mess of all or have actually done something that will work? :)
All yours now.
Cheers!

Hi @spencergibb,
I've made the change to use HttpSerlvetRequest instead of RequestContext. I squashed my commit with the old one to keep it in 1 commit and that has made your comment outdated, didn't know it will do that. Will try avoiding squashing next time :)
Cheers!

And not sure why circleci failed this time, all tests are passing on my local.

Squashing is fine. I can get to the outdated code.

Hi @spencergibb,
So I've taken your suggestion of creating a new set of classes to do the matching. I've committed a change, it is just a draft commit. I want to get your opinion on it before I go ahead and shine it a bit and write further tests. I'll fix/squash the commit later - easier to discuss this way.
Now

Everything works as it was as a default implementation i.e. matching using the path. It uses the existing RouteLocator.
zuul.route-matcher.fancy property has been introduced which when set to true disables the simple matching using RouteLocator and enables the fancy matching using RouteMatcher.  I had to do a bit more to get all this working together so you would see some classes being created, some being extended. It was all because the RouteLocator was quite embedded in those classes and had to create new ones to use RouteMatcher.

FancyZuulProxyApplicationTests class would be the one to see how it works.
Yaml will look like below
zuul:
  routes:
    users:
      path: /myusers/service
      url: http://default-user-service
      routeOptions:
        allowedMethods: GET, PUT, POST, PATCH
        methodOptions:
          GET:
            url: http://read-user-service
          PUT:
            url: http://write-user-service
          POST:
            url: http://write-user-service

GET will end up with http://read-user-service
PUT and POST will end up with http://write-user-service
PATCH will end up with http://default-user-service i.e. default url since a specific one is not specified.

I'll later combine all the beans in one class to enable and disable simple and fancy route matching using a single property instead of current implementation where I've put the property on individual beans. Just in case you notice that. :)
Cheers!

Cool, I'll fix the above mentioned issues, just wanted to see if there are any major concerns about the approach - no big concerns raised so I'll take it as a yes :)

Hi @spencergibb,
I've committed some changes that you asked for - you will see some code copied from the spring-cloud-gateway as well :). Wasn't able to spend much time on it last two weeks but hoping to get it all sorted this coming weekend.
I've got rid of duplicate code and made route options dynamic instead of hard coded. I hope I am on a right track now - let me know what you think?
By the way, I've squashed the commits again to keep it to 1 single commit.
Cheers!

Thanks @ryanjbaxter, I've got it back to java 1.7 now. Have updated the docs and added some tests to make sure it can read the routes properly from the yml file.
Let me know if you guys want me to fix anything else. I'll squash all commits into one once you guys are happy with it.
Cheers!

Thanks @ryanjbaxter. I've updated my branch, resolved conflicts and squashed all my commits into one.
This is the end result. :)
zuul:
  route-matcher:
    advanced: true
  routes:
    advance-routing-compact:
      path: /advanceroutescompact/**
      url: http://user-default-service
      routeOptions:
        allowedMethods: PUT, POST, GET
        routeConditions:
        - user-get-operation=http://user-get-service,Method=GET
        - user-post-operation=http://user-post-service,Method=POST
    advance-routing-expanded:
      path: /advanceroutesexpanded/**
      url: http://user-default-service
      routeOptions:
        allowedMethods: PUT, POST, GET
        routeConditions:
        - id: user-get-operation
          url: http://user-get-service
          predicates:
          - Method=GET
        - id: user-post-operation
          url: http://user-post-service
          predicates:
          - Method=POST

Cheers!

Hi @spencergibb, your thoughts?

Hi @spencergibb @ryanjbaxter,
Is there anything I need to do to have this merged please?

Looks like another rebase is in order. I'll review it next week.

Hi @spencergibb
Rebase done.

Hi @spencergibb
Could you please review this one? I really want to get this one in before the end of the year.

Closing the pull request as it's been too long and the modules are now in maintenance mode.


haskellPackages: update stackage and hackage

This Merge
This PR is the regular merge of the haskell-updates branch into master.
This branch is being continually built and tested by hydra at https://hydra.nixos.org/jobset/nixpkgs/haskell-updates. You may be able to find an up-to-date Hydra build report at cdepillabout/nix-haskell-updates-status.
We roughly aim to merge these haskell-updates PRs at least once every two weeks. See the @NixOS/haskell team calendar for who is currently in charge of this branch.
haskellPackages Workflow Summary
Our workflow is currently described in pkgs/development/haskell-modules/HACKING.md.
The short version is this:

We regularly update the Stackage and Hackage pins on haskell-updates (normally at the beginning of a merge window).
The community fixes builds of Haskell packages on that branch.
We aim at at least one merge of haskell-updates into master every two weeks.
We only do the merge if the mergeable job is succeeding on hydra.
If a maintained package is still broken at the time of merge, we will only merge if the maintainer has been pinged 7 days in advance. (If you care about a Haskell package, become a maintainer!)

This is the follow-up to #182019. Come to #haskell:nixos.org if you have any questions.

@robx hasql-dynamic-statements broke again, is this a pure constraints issue? Do we just need to downgrade hasql-dynamic-statements?

@robx hasql-dynamic-statements broke again, is this a pure constraints issue? Do we just need to downgrade hasql-dynamic-statements?

It looks like we just got new hasql releases. hasql-1.6 and hasql-dynamic-statements-<IP_ADDRESS> were uploaded to hackage on 2022-07-20, and the latter requires the former. From the error it looks like nixpkgs kept the old hasql version while pulling in the new hasql-dynamic-statements?

@robx Thanks for your diagnosis, I've addressed the problem in ff7abeabe9c2b8e9ef0d47d8ef8cafc64b6d609f – we can just downgrade hasql-dynamic-statements until we switch Stackage version (hasql 1.6 will never make it into LTS 19 sadly).


(Bug fix) Slices in correct dimension (audio dim), so batch size is not reduced

I noticed that my batchsize was reduced from 100 to 34 if the model received audio with sample_rate=48000. The line where downsamplling to sample_rate=16000 happens, we slice in dim 0, so if the batch size is greater than 1, we will slice the batch dimension instead.

Thanks for your contribution! I fixed this bug in the jit model also
https://github.com/snakers4/silero-vad/commit/b88084c7ed4b7c1050548be5af129fb4ea98a47a


expose helm values for yurthub cacheagents

Signed-off-by: huiwq1990<EMAIL_ADDRESS>
What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespace from that line:
/kind bug
/kind documentation
/kind enhancement
/kind good-first-issue
/kind feature
/kind question
/kind design
/sig ai
/sig iot
/sig network
/sig storage

What this PR does / why we need it:
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:

Does this PR introduce a user-facing change?

other Note

@huiwq1990 Thanks for posting pull request, it's a good idea to expose cache agents field in helm values.

/lgtm
/approve


New update

Did u just add unit to new verison?
and i have line of code to spawn 4 units at start of battle if u want it

sure, I'll have a look.


[SMALLFIX]A COMMIT

change some thing

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/19350/
Test PASSed.

@S847659045 please use a more descriptive message as the title

LGTM. Thanks.


Proposal for renaming notaryproject\notaryproject to notaryproject\specifications

As part of the discussions we are having about cleaning up the branding (#32 and #35) there is a proposal to archive the notaryproject repository as it is confusing and replace it with a new repository. This issue is intended to describe the proposal for the new repository, its purpose and structure.
The proposal is to create a new repository with the name specifications that will contain specifications shared across "subprojects" under Notary Project as well as used by other OSS projects and/or vendor tools that want to interoperate with Notary Project tooling. Here the details:

Repository name: specifications
Repository purpose: Cross tooling and interoperability specifications
Structure:

specifications repository

oci-signature-specification folder
Those specifications are OCI specific and describe how the Notary Project Signing Scheme is applied to signatures stored in OCI registries. The folder contains the following documents:

Signature Specification referred to as "Notary Project OCI Signature Specification" or just "OCI Signature Specification" if the Notary Project reference is implied.
Signing and Verification Workflow referred to as "Notary Project OCI Signing and Verification Workflow" or just "OCI Signing and Verification Workflow" if the Notary Project reference is implied.
Signing Envelope COSE referred to as "Notary Project OCI COSE Envelope" or just "OCI COSE Envelope" if the Notary Project reference is implied. Ideally, we would like to have COSE envelope spec that is not specific to OCI.
Signing Envelope JWS referred to as "Notary Project OCI JWS Envelope" or just "OCI JWS Envelope" if the Notary Project reference is implied. Ideally, we would like to have JWS envelope spec that is not specific to OCI.

signing-scheme folder
This specification is not specific to any storage and describes the supported signing schemes. Ideally any tool that supports those schemes should be able to produce signatures that are understood by Notary Project tools. Ideally, we would like to have non-OCI specific envelope specifications in this folder also. This folder contains the following documents:

Signing Scheme referred to as "Notary Project Signing Scheme" or just "Signing Scheme" if the Notary Project reference is implied.

In the future, this repository may contain specifications about identities, attestations, counter receipts, etc. depending on how the Notary Project evolves.

Adding @gokarnm in this conversation since the comment - https://github.com/notaryproject/notary/pull/1685#issuecomment-1610321068 is related to this proposal.

@FeynmanZhou those specifications may be referred from specifications, blog posts, articles and other documents that are not in that folder and without the context of the folder name. That is why it is important to define a clear name for them to avoid ambuiguity.

@toddysm   - Is there a proposal for the other two specifications currently present in the https://github.com/notaryproject/notaryproject/tree/main/specs - The plugin-extensibility.md and trust-store-trust-policy.md ?
Also, what about the other sub folders in the notaryproject repo  - Security, Requirements, media

@iamsamirzon thanks for asking.

plugin and trust store are very specific to Notation, so those should be specs under Notation IMHO
regarding the security audits, we should have those in the .github repo if they are across subprojects or in the specific repos if they are specific to a subproject

I believe this issues needs to be retitled as renmae notaryproject\notaryproject to notaryproject\specficiation
Just to confirm is it specification or specfications? the repository could be specification and not plural similar to image-spec/distribution-spec tuf spec etc.

We may need to separate the signature storage from signature specification and write a new specification for signature storage, since the signature can be stored in OCI compliant registries or in a filesystem as a file, see https://github.com/notaryproject/notation/issues/741

As one work item from the approved proposal https://github.com/notaryproject/.github/issues/35#issuecomment-1642938826 and alignment in community meeting on Jul 24 PDT, this proposal requires a two-thirds supermajority of the maintainer votes, please reply LGTM to this proposal. As the one writing this proposal, my vote is LGTM.
@SteveLasker @justincormack @NiazFK @gokarnm @toddysm @FeynmanZhou @vaninrao10 @priteshbandi @iamsamirzon

Just a question: Based on proposal https://github.com/notaryproject/.github/issues/35 , folder "requirements" will not be maintained any more.

I'd agree with @TheFoxAtWork for this comment:

Requirements may be captured as GH issues however it may also be prudent to record high level 'requirements' or design principles as Scoping Goals/ Guide that the project and its community members may continuously refer back to when determining overall features, design criteria, and the acceptable characteristics for PRs.

Keeping the requirements and scenarios as a single doc, provides context to the purpose of the project and it's goals. Which should continue to evolve. Converting these to issues or archiving would lose that context in a meaningful way.

Thanks @SteveLasker .
I will create a separate issue for addressing the folder structure. This issue will be used for Proposal for renaming notaryproject\notaryproject to notaryproject\specifications

@SteveLasker @justincormack @NiazFK @gokarnm @toddysm @FeynmanZhou @vaninrao10 @priteshbandi @iamsamirzon
Please reply LGTM to if you agree on renaming notaryproject\notaryproject to notaryproject\specifications
As the one writing this vote, my response is LGTM.
Regarding to folder structure discussion, please go to this issue https://github.com/notaryproject/.github/issues/48

I created the proposal, so my +1 should be implicit 😀

LGTM

LGTM

LGTM, except for Trust Store and Trust Policy should be part of the Notary Project specifications, it has the core verification logic that is references from the OCI specific steps in Signing and Verification Workflow.

LGTM for renaming, we will have separate issue for restructure of pkg.

LGTM

The renaming and purpose of repository were updated, see https://github.com/notaryproject/specifications
@toddysm Maybe we can close this issue and refer to https://github.com/notaryproject/.github/issues/48 for folder structure and content discussion.

Close as the renaming was completed. Please go to https://github.com/notaryproject/.github/issues/48 for folder structure and content discussion.


Replace easywatch with something else

That project doesn't ship wheels, and sdist builds emit setuptools warnings because it uses deprecated features. When running python -Werror pip install staticjinja, it turns said warnings into errors.
I'm suggesting to replace the dependency as that project was last updated 5 years ago and might now issue a fix in the near future.
While I can opt out of turning these warnings into errors for now, on my side, those warnings will turn into errors for everyone this fall, in just a few months. I think, it's wise to act proactively.

Filed an upstream issue too: https://github.com/Ceasar/easywatch/issues/12.

Thanks for the issue! I agree something must be done. I think the best path is to just vendor in https://github.com/Ceasar/easywatch/blob/master/easywatch/easywatch.py, that appears to be the entirety of the code! The underlyting lib, watchdog, appears to be well-maintained.

@webknjaz This should be fixed for you in 5.0.0. If you do pip install staticjinja==5.0.0 do you still get the warning?

@NickCrews wow, thanks! I'll alert my colleague who is integrating this lib.
By the way, I noticed that some ecosystem projects integrated Watchman:
https://adamj.eu/tech/2021/01/20/efficient-reloading-in-djangos-runserver-with-watchman/ / https://www.youtube.com/watch?v=IghyoR6ld60. FYI in case you ever end up needing to rethink this feature.
P.S. I noticed that you are using a long-outdated version of my pypi-publish GitHub Action (many other actions look outdated too, for that matter) and are relying on long-living PyPI API tokens. May I invite you to upgrade the action and switch to tokenless trusted publishing? The doc is here https://docs.pypi.org/trusted-publishers/ and the change is pretty straightforward: https://github.com/marketplace/actions/pypi-publish#trusted-publishing.

Oh, yes! I've used trusted publishers before, but I just haven't done that here. Will do.

Hi @NickCrews Thanks a mill for the quick fix. I've upgraded staticjinja to 5.0.0 but still notice there's an exception related to setuptools. Here are some logs that might give you some info.

@oraNod that's not related to this project anymore: https://github.com/ansible/docsite/pull/182/files#r1297191664

Thanks for the heads up @webknjaz and apologies for the noise here @NickCrews cheers!


Scheduler Fix

The scheduler tried to upload the files even when the FTP function was deactivated

Fixed with if function


Enhanced Tracing for Workflows via Python SDK

In what area(s)?
/area runtime
Describe the feature
Would like to enhance the existing tracing for workflows with two main additions:

Be able to add attributes to existing activity spans. For instance add the name of the pod that executed the activity
Add child spans to the activity span.  If the activity interacts s3 for instance we would like to capture that as a sub-span.

Seems like the WorkflowActivityContext in the python sdk would be an obvious place to add that additional info.

Thanks @cchanley2003 for this suggestion. You mentioned wanting to do this in Python - can you expand on your feature description by providing some sample code that illustrates what it might look like to add new attributes to existing spans? That would help further clarify things for us.

So ideally this would be an example:
def model_properties(ctx: WorkflowActivityContext, input):
  ctx.span.set_attribute("pod_name", os.getenv("POD_NAME")
  tracer = trace.get_tracer(__name__)

  # Start a child span using the parent span object
  with tracer.start_as_current_span("child-span", parent=ctx.span) as child_span:
    # talk to s3

I think this is a good proposal that would greatly improve the tracing/diagnostics story for workflows. We would ideally do this for each SDK. In terms of runtime changes, I think we'll need to update the gRPC protocol used by durabletask-go to flow the distributed trace information as part of orchestration and activity work items so that the workflow SDKs can pick them up.
/cc @rabollin


Conflict when the app already contains a Transactions collection

When an app already defines a MongoDB collection named transactions, the developers are forced to change its name after installing the package. I think that the collection created by the package should be renamed to something that is not a plain dictionary word to avoid any conflict.

Good point. There might be some backwards compatibility concerns doing this, but you're right - the collection name should be prefixed with an underscore at least.
Sent from my iPhone

On Aug 24, 2015, at 11:25 PM, Pier-Luc Brault<EMAIL_ADDRESS>wrote:
When an app already contains a collection named Transactions, the developers are forced to change its name before installing the package. I think that the collection created by the package should be renamed to something that is not a plain dictionary word to avoid any conflict.
—
Reply to this email directly or view it on GitHub.

How about a config setting to chg name?

Good idea @ericoe

Agreed. Will make this option available in the next release.
Sent from my iPhone

On Aug 25, 2015, at 9:34 PM, Steve Bourgoin<EMAIL_ADDRESS>wrote:
Good idea @ericoe
—
Reply to this email directly or view it on GitHub.

In 0.7.5, the transactions collection can be specified in a settings.json file in the project root:
{
  transactionsCollection: "transactionRecords"
}

Thanks!

Does not work. Client still shows error: "ncaught Error: There is already a collection named "transactions".
I think it's because the settings in settings.json is not accessible from client (it it is not in the "public" area).

Yes, I believe you're right.  I'll accept a PR and republish if there's a solution for this.
On another note, if you're using this package to deal with money type transactions, I'd have to warn against it.  Using this package is a step up from no transactions, but it's not mature enough to have important data relying on its functionality.

Thank you
Yes, I've read your warning about using it in money treating apps.
I want it for simpe purpose, for my own risk.


Complete refactor of TableView and table.html template

Split from #878. The current TableView class is by far the most complex part of Datasette, and the most difficult to work on: https://github.com/simonw/datasette/blob/0.59.2/datasette/views/table.py
In #878 I started exploring a new pattern for building views. In doing so it became clear that TableView is the first beast that I need to slay - if I can refactor that into something neat the pattern for building other views will emerge as a natural consequence.
I've been trying to build this as a register_routes() plugin, as originally suggested in #870 - though unfortunately it looks like those plugins can't replace existing Datasette default views at the moment, see #1517. [UPDATE: I was wrong about this, plugins can over-ride default views just fine]
I also know that I want to have a fully documented template context for table.html as a major step on the way to Datasette 1.0, see #1510.
All of this adds up to the TableView factor being a major project that will unblock a whole flurry of other things - so I'm going to work on that in this separate issue.

Here's where I got to with my hacked-together initial plugin prototype - it managed to render the table page with some rows on it (and a bunch of missing functionality such as filters): https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2

Ideally I'd like to execute the existing test suite against the new implementation - that would require me to solve this so I can replace the view with the plugin version though:

#1517

I was wrong about that, you CAN over-ride default routes already.

A (likely incomplete) list of features on the table page:

[ ] Display table/database/instance metadata
[ ] Show count of all results
[ ] Display table of results

[ ] Special table display treatment for URLs, numbers
[ ] Allow plugins to modify table cells

[ ] Show interface for filtering by columns and operations
[ ] Show search box, support executing FTS searches
[ ] Sort table by specified column
[ ] Paginate table
[ ] Show facet results
[ ] Show suggested facets
[ ] Display schema for table

[ ] Maybe it should show the SQL for the query too?

My goal is to break up a lot of this functionality into separate methods. These methods can be executed in parallel by asyncinject, but more importantly they can be used to build a much better JSON representation, where the default representation is lighter and ?_extra=x options can be used to execute more expensive portions and add them to the response.
So the HTML version itself needs to be re-written to use those JSON extras.

Right now the HTML version gets to cheat - it passes through objects that are not JSON serializable, including custom functions that can then be called by Jinja.
I'm interested in maybe removing this cheating - if the HTML version could only request JSON-serializable extras those could be exposed in the API as well.
It would also help cleanup the kind-of-nasty pattern I use in the current BaseView where everything returns both a bunch of JSON-serializable data AND an awaitable function that then gets to add extra things to the HTML context.

... and while I'm doing all of this I can rewrite the templates to not use those cheating magical functions AND document the template context at the same time, refs #1510.

Very confused by this piece of code here: https://github.com/simonw/datasette/blob/1c13e1af0664a4dfb1e69714c56523279cae09e4/datasette/views/table.py#L37-L63

I'm going to try leaning into the asyncinject mechanism a bit here. One method can execute and return the raw rows. Another can turn that into the default minimal JSON representation. Then a third can take that (or take both) and use it to inflate out the JSON that the HTML template needs, with those extras and with the rendered cells from plugins.

This may be an argument for continuing to allow non-JSON-objects through to the HTML templates. Need to think about that a bit more.

I can definitely support this using pure-JSON - I could make two versions of the row available, one that's an array of cell objects and the other that's an object mapping column names to column raw values.

To new requirements inspired by work on the datasette-table (and datasette-notebook) projects:

#1533
#1534

I'm also going to use the new datasette-table Web Component to help guide the design of the new API, which relates directly to this issue too:

#1532

I'm also going to use the new datasette-table Web Component to help guide the design of the new API, which relates directly to this issue too:

#1532

Aside: is there any reason this work can't complete the long-running goal of merging the TableView and QueryView, such that most of the features available for tables become available for arbitrary queries too?
I had already mentally committed to implementing facets for queries, but I just realized that filters could work too - using either a CTE or a nested query.
Pagination is the one holdout here, since table pagination uses keyset pagination over a known order. But maybe arbitrary queries can only be paginated off you order them first?

(I could experiment with merging the two tables by adding a temporary undocumented ?_sql= parameter to the in-progress table view that sets an alternative query instead of select cols from table - added bonus, this will force me to use introspection against the returned columns rather than mixing in the known columns for the specified table)

If I break this up into @inject methods, what methods could I have and what would they do?

resolve_path: Use request path to resolve the database and table. Could handle hash URLs too (if I don't manage to extract those to a plugin) - would be nice if this could raise a redirect, but I think that will instead have to be one of the things it returns
build_sql: Builds the SQL query based on the querystring (and some DB introspection)
execute_count: Execute the count(*)
execute_rows: Execute the limit 101 to fetch the rows
execute_facets: Execute all requested facets (could this do its own asyncio.gather() to run facets in parallel?)
suggest_facets: Execute facet suggestions

Are there any plugin hooks that would make sense to execute in parallel? Actually there might be: I don't think extra_template_vars, extra_css_urls, extra_js_urls, extra_body_script depend on each other so it might be possible to execute them in a parallel chunk (at least any of them that return awaitables).

I have a hunch that the conclusion of this experiment may end up being that the asyncinject trick is kinda neat but the code will be easier to maintain (while still executing in parallel) if it's written using asyncio.gather directly instead.
It's possible asyncinject will end up being neat enough that I'll want to keep it though.

Rebuilding TableView from the ground up is proving not to be much fun. I'm going to explore starting the refactor of the existing code by separating out the bit that generates the SQL query from the rest of it.

The tests for TableView are currently mixed in with everything else in tests/test_api.py and tests/html.py - might be good to split those out into test_table_html.py and test_table_api.py since they're such a key part of how Datasette works.

I don't think this code is necessary any more: https://github.com/simonw/datasette/blob/492f9835aa7e90540dd0c6324282b109f73df71b/datasette/views/table.py#L396-L399
That dates back from when Datasette was built on top of Sanic and Sanic didn't preserve those query parameters the way I needed it to:
https://github.com/simonw/datasette/blob/1f69269fe93e4cd42e56890126cc0dbcf719c6cb/datasette/views/table.py#L202-L206

No, removing that gave me the following test failure:
tests/test_table_api.py::test_table_filter_queries[/fixtures/simple_primary_key.json?content__exact=-expected_rows2] FAILED                                       [100%]

=============================================================================== FAILURES ================================================================================
______________________________________ test_table_filter_queries[/fixtures/simple_primary_key.json?content__exact=-expected_rows2] ______________________________________

app_client = <datasette.utils.testing.TestClient object at 0x10d45d2d0>, path = '/fixtures/simple_primary_key.json?content__exact=', expected_rows = [['3', '']]

    @pytest.mark.parametrize(
        "path,expected_rows",
        [
            ("/fixtures/simple_primary_key.json?content=hello", [["1", "hello"]]),
            (
                "/fixtures/simple_primary_key.json?content__contains=o",
                [
                    ["1", "hello"],
                    ["2", "world"],
                    ["4", "RENDER_CELL_DEMO"],
                ],
            ),
            ("/fixtures/simple_primary_key.json?content__exact=", [["3", ""]]),
            (
                "/fixtures/simple_primary_key.json?content__not=world",
                [
                    ["1", "hello"],
                    ["3", ""],
                    ["4", "RENDER_CELL_DEMO"],
                    ["5", "RENDER_CELL_ASYNC"],
                ],
            ),
        ],
    )
    def test_table_filter_queries(app_client, path, expected_rows):
        response = app_client.get(path)
>       assert expected_rows == response.json["rows"]
E       AssertionError: assert [['3', '']] == [['1', 'hello'],\n ['2', 'world'],\n ['3', ''],\n ['4', 'RENDER_CELL_DEMO'],\n ['5', 'RENDER_CELL_ASYNC']]
E         At index 0 diff: ['3', ''] != ['1', 'hello']
E         Right contains 4 more items, first extra item: ['2', 'world']
E         Full diff:
E           [
E         -  ['1',
E         -   'hello'],
E         -  ['2',
E         -   'world'],
E            ['3',
E             ''],
E         -  ['4',
E         -   'RENDER_CELL_DEMO'],
E         -  ['5',
E         -   'RENDER_CELL_ASYNC'],
E           ]

/Users/simon/Dropbox/Development/datasette/tests/test_table_api.py:511: AssertionError

Idea: in JSON output include a warnings block listing any _ parameters that were not recognized.

Built a new plugin to help with this work by improving the display of ?_trace=1 output: https://datasette.io/plugins/datasette-pretty-traces

Useful old comment here: https://github.com/simonw/datasette/issues/617#issuecomment-552253893

As noted in #621 (comment) a common pattern in this method is blocks of code that append new items to the where_clauses, params and extra_human_descriptions arrays. This is a useful refactoring opportunity.
Code that fits this pattern:
* The code that builds based on the filters: `where_clauses, params = filters.build_where_clauses(table)` and `human_description_en = filters.human_description_en(extra=extra_human_descriptions)`

* Code that handles `?_where=`: `where_clauses.extend(request.args["_where"])` - though note that this also appends to a `extra_wheres_for_ui` array which nothing else uses

* The `_through=` code, see [Syntax for ?_through= that works as a form field #621](https://github.com/simonw/datasette/issues/621) for details

* The code that deals with `?_search=` FTS

The keyset pagination code modifies where_clauses and params too, but I don't think it's quite going to work with the same abstraction that would cover the above examples.

There are actually four forms of SQL query used by the table page:

from_sql - just the from table_name where ...
sql_no_order_no_limit - used for faceting, "select {select_all_columns} from {table_name} {where}"
sql - the above but with order and limit clauses: "select {select_specified_columns} from {table_name} {where}{order_by} limit {page_size}{offset}"
count_sql used for the count, built out of from_sql: "select count(*) {from_sql}"

Should facets really not be displayed on pages past page one (where ?_next= is set)? That made sense to me at the time, but I'm now having second thoughts about it.
I guess it's a good performance boost for when crawlers keep hitting the ?_next= link.

A fundamental operation of this view is to construct the SQL query and accompanying human description based on the incoming query string parameters.
The human description is the bit at the top of https://latest.datasette.io/fixtures/searchable?_search=dog&_sort=pk&_facet=text2&text2=sara+weasel that says:

1 row where search matches "dog" and text2 = "sara weasel" sorted by pk

(Also used in the page <title>).
The code actually gathers three things:

Fragments of the where clause, for example  "text2" = :p0
Parameters, e.g. {"p0": "sara weasel}
Human description components, e.g. text2 = "sara weasel"

Some operations such as ?_where= don't currently provide an extra human description component.
_where= also doesn't populate a parameter, but maybe it could? Would be neat if in the future ?_where=foo+=+:bar worked and added a bar input field to the screen, as seen with custom queries.

I added a ton of comments to the data() method which really helps get a better feel for how this all works: https://github.com/simonw/datasette/blob/2c07327d23d9c5cf939ada9ba4091c1b8b2ba42d/datasette/views/table.py#L322

Is there an opportunity to refactor things using a new plugin hook here? Maybe the register_filters hook from #473, where the hook becomes responsible for building where clauses (and human descriptions of them) based on the incoming query string.

I think that plugin hook would get given the request object (and datasette and the name of the database and table) and returns a list of SQL fragments, a dictionary of lookup arguments and a list of human-description fragments - or an awaitable.

The hook could return a named tuple like this one:
from typing import NamedTuple, List, Optional, Union, Dict

class FilterArguments(NamedTuple):
    where_clauses: List[str]
    params: Dict[str, Union[str, int, float]]
    human_descriptions: List[str]

Ran into a problem prototyping that hook up for handling ?_where= - that feature also adds a little bit of extra template context in order to show the interface for removing wheres - the extra_wheres_for_ui variable: https://github.com/simonw/datasette/blob/0663d5525cc41e9260ac7d1f6386d3a6eb5ad2a9/datasette/views/table.py#L457-L463

I managed to extract both _search= and _where= out using a prototype of that hook. I wonder if it could extract the complex code for ?_next too?

Got a TIL out of this: https://til.simonwillison.net/pluggy/multiple-hooks-same-file

Documentation of the new hook in the PR: https://github.com/simonw/datasette/blob/54e9b3972f277431a001e685f78e5dd6403a6d8d/docs/plugin_hooks.rst#filters_from_requestrequest-database-table-datasette

These changes are now in the 0.60a0 alpha: https://github.com/simonw/datasette/releases/tag/0.60a0

I sketched out a chained SQL builder pattern that might be useful for further tidying up this code - though with the new plugin hook I'm less excited about it than I was:
class TableQuery:
    def __init__(self, table, columns, pks, is_view=False, prev=None):
        self.table = table
        self.columns = columns
        self.pks = pks
        self.is_view = is_view
        self.prev = prev

        # These can be changed for different instances in the chain:
        self._where_clauses = None
        self._order_by = None
        self._page_size = None
        self._offset = None
        self._select_columns = None

        self.select_all_columns = '*'
        self.select_specified_columns = '*'

    @property
    def where_clauses(self):
        wheres = []
        current = self
        while current:
            if current._where_clauses is not None:
                wheres.extend(current._where_clauses)
            current = current.prev
        return list(reversed(wheres))

    def where(self, where):
        new_cls = TableQuery(self.table, self.columns, self.pks, self.is_view, self)
        new_cls._where_clauses = [where]
        return new_cls

    @classmethod
    async def introspect(cls, db, table):
        return cls(
            table,
            columns = await db.table_columns(table),
            pks = await db.primary_keys(table),
            is_view = bool(await db.get_view_definition(table))
        )

    @property
    def sql_from(self):
        return f"from {self.table}{self.sql_where}"

    @property
    def sql_where(self):
        if not self.where_clauses:
            return ""
        else:
            return f" where {' and '.join(self.where_clauses)}"

    @property
    def sql_no_order_no_limit(self):
        return f"select {self.select_all_columns} from {self.table}{self.sql_where}"

    @property
    def sql(self):
        return f"select {self.select_specified_columns} from {self.table} {self.sql_where}{self._order_by} limit {self._page_size}{self._offset}"

    @property
    def sql_count(self):
        return f"select count(*) {self.sql_from}"

    def __repr__(self):
        return f"<TableQuery sql={self.sql}>"

Usage:
from datasette.app import Datasette
ds = Datasette(memory=True, files=["/Users/simon/Dropbox/Development/datasette/fixtures.db"])
db = ds.get_database("fixtures")
query = await TableQuery.introspect(db, "facetable")
print(query.where("foo = bar").where("baz = 1").sql_count)
# 'select count(*) from facetable where foo = bar and baz = 1'

I think I might be able to clean up a lot of the stuff in here using the render_cell plugin hook: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L87-L89

I think I can move this much higher up in the method, it's a bit confusing having it half way through: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L414-L436

Also the whole special_args v.s. request.args thing is pretty confusing, I think that might be an older code pattern back from when I was using Sanic.

New short-term goal: get facets and suggested facets to execute in parallel with the main query. Generate a trace graph that proves that is happening using datasette-pretty-traces.

It looks like the count has to be executed before facets can be, because the facet_class constructor needs that total count figure: https://github.com/simonw/datasette/blob/6b1384b2f529134998fb507e63307609a5b7f5c0/datasette/views/table.py#L660

New short-term goal: get facets and suggested facets to execute in parallel with the main query. Generate a trace graph that proves that is happening using datasette-pretty-traces.

I wrote code to execute those in parallel using asyncio.gather() - which seems to work but causes the SQL run inside the parallel async def functions not to show up in the trace graph at all.
diff --git a/datasette/views/table.py b/datasette/views/table.py
index 9808fd2..ec9db64 100644
--- a/datasette/views/table.py
+++ b/datasette/views/table.py
@@ -1,3 +1,4 @@
+import asyncio
 import urllib
 import itertools
 import json
@@ -615,44 +616,37 @@ class TableView(RowTableShared):
         if request.args.get("_timelimit"):
             extra_args["custom_time_limit"] = int(request.args.get("_timelimit"))

-        # Execute the main query!
-        results = await db.execute(sql, params, truncate=True, **extra_args)
-
-        # Calculate the total count for this query
-        filtered_table_rows_count = None
-        if (
-            not db.is_mutable
-            and self.ds.inspect_data
-            and count_sql == f"select count(*) from {table} "
-        ):
-            # We can use a previously cached table row count
-            try:
-                filtered_table_rows_count = self.ds.inspect_data[database]["tables"][
-                    table
-                ]["count"]
-            except KeyError:
-                pass
-
-        # Otherwise run a select count(*) ...
-        if count_sql and filtered_table_rows_count is None and not nocount:
-            try:
-                count_rows = list(await db.execute(count_sql, from_sql_params))
-                filtered_table_rows_count = count_rows[0][0]
-            except QueryInterrupted:
-                pass
-
-        # Faceting
-        if not self.ds.setting("allow_facet") and any(
-            arg.startswith("_facet") for arg in request.args
-        ):
-            raise BadRequest("_facet= is not allowed")
+        async def execute_count():
+            # Calculate the total count for this query
+            filtered_table_rows_count = None
+            if (
+                not db.is_mutable
+                and self.ds.inspect_data
+                and count_sql == f"select count(*) from {table} "
+            ):
+                # We can use a previously cached table row count
+                try:
+                    filtered_table_rows_count = self.ds.inspect_data[database][
+                        "tables"
+                    ][table]["count"]
+                except KeyError:
+                    pass
+
+            if count_sql and filtered_table_rows_count is None and not nocount:
+                try:
+                    count_rows = list(await db.execute(count_sql, from_sql_params))
+                    filtered_table_rows_count = count_rows[0][0]
+                except QueryInterrupted:
+                    pass
+
+            return filtered_table_rows_count
+
+        filtered_table_rows_count = await execute_count()

         # pylint: disable=no-member
         facet_classes = list(
             itertools.chain.from_iterable(pm.hook.register_facet_classes())
         )
-        facet_results = {}
-        facets_timed_out = []
         facet_instances = []
         for klass in facet_classes:
             facet_instances.append(
@@ -668,33 +662,58 @@ class TableView(RowTableShared):
                 )
             )

-        if not nofacet:
-            for facet in facet_instances:
-                (
-                    instance_facet_results,
-                    instance_facets_timed_out,
-                ) = await facet.facet_results()
-                for facet_info in instance_facet_results:
-                    base_key = facet_info["name"]
-                    key = base_key
-                    i = 1
-                    while key in facet_results:
-                        i += 1
-                        key = f"{base_key}_{i}"
-                    facet_results[key] = facet_info
-                facets_timed_out.extend(instance_facets_timed_out)
-
-        # Calculate suggested facets
-        suggested_facets = []
-        if (
-            self.ds.setting("suggest_facets")
-            and self.ds.setting("allow_facet")
-            and not _next
-            and not nofacet
-            and not nosuggest
-        ):
-            for facet in facet_instances:
-                suggested_facets.extend(await facet.suggest())
+        async def execute_suggested_facets():
+            # Calculate suggested facets
+            suggested_facets = []
+            if (
+                self.ds.setting("suggest_facets")
+                and self.ds.setting("allow_facet")
+                and not _next
+                and not nofacet
+                and not nosuggest
+            ):
+                for facet in facet_instances:
+                    suggested_facets.extend(await facet.suggest())
+            return suggested_facets
+
+        async def execute_facets():
+            facet_results = {}
+            facets_timed_out = []
+            if not self.ds.setting("allow_facet") and any(
+                arg.startswith("_facet") for arg in request.args
+            ):
+                raise BadRequest("_facet= is not allowed")
+
+            if not nofacet:
+                for facet in facet_instances:
+                    (
+                        instance_facet_results,
+                        instance_facets_timed_out,
+                    ) = await facet.facet_results()
+                    for facet_info in instance_facet_results:
+                        base_key = facet_info["name"]
+                        key = base_key
+                        i = 1
+                        while key in facet_results:
+                            i += 1
+                            key = f"{base_key}_{i}"
+                        facet_results[key] = facet_info
+                    facets_timed_out.extend(instance_facets_timed_out)
+
+            return facet_results, facets_timed_out
+
+        # Execute the main query, facets and facet suggestions in parallel:
+        (
+            results,
+            suggested_facets,
+            (facet_results, facets_timed_out),
+        ) = await asyncio.gather(
+            db.execute(sql, params, truncate=True, **extra_args),
+            execute_suggested_facets(),
+            execute_facets(),
+        )
+
+        results = await db.execute(sql, params, truncate=True, **extra_args)

         # Figure out columns and rows for the query
         columns = [r[0] for r in results.description]

Here's the trace for http://<IP_ADDRESS>:4422/fixtures/compound_three_primary_keys?_trace=1&_facet=pk1&_facet=pk2 with the missing facet and facet suggestion queries:

The reason they aren't showing up in the traces is that traces are stored just for the currently executing asyncio task ID: https://github.com/simonw/datasette/blob/ace86566b28280091b3844cf5fbecd20158e9004/datasette/tracer.py#L13-L25

Did this in the 1.0 alphas.


Ignore stop_timezone completely for parsing stop_times.txt - stop_timezone is only relevant for user info

(Just dropping by making a comment, not sure if this is the right place...)
I had been under the impression that stop_timezone is relevant for interpreting stop_times rows, but apparently it is currently assumed that it is not.

There's a discussion about this in google/transit#322.
relevant discussion in the OTP chat: https://matrix.to/#/!oXNNoHKzbaSOlFzLEt:gitter.im/$4vki4gvL6-4n66VlpKbuNSj4WiYL2NdjknX0D4OKnY8?via=gitter.im&via=matrix.org&via=vern.cc
I have changed gtfs-via-postgres's behaviour in public-transport/gtfs-via-postgres#38.

https://github.com/motis-project/nigiri/commit/60d256a410a69d0f832c1e4d6910228a77a54f62#r108449094

https://github.com/motis-project/nigiri/pull/30/files#diff-cb424468b06b79a533d7fd62f197228d8b2b3ef7c0ddd0ff90e4178e80eaa355R87-R89
Thank you @derhuerst for reviewing the GTFS parser code. Very good catch! I hope it is correct now - it takes only the agency timezone into consideration when parsing stop_times.txt. If I understood it correctly, this was the problem you were pointing to. Please let me know if you spotted other issues or if I still got it wrong.
The PR #30 also introduces several speedups for the GTFS loader such as precomputing the noon offsets and switching to a faster hash map implementation.


Deprecated constructors of ZstdOutputStream fix

The zstd-jni was updated to 1.4.4-7, and some constructors of ZstdOutputStream were deprecated.
This PR fix them as the comment was recommended :
/**

@deprecated
Use ZstdOutputStream() or ZstdOutputStream(level) and set the other params with the setters
**/

Coverage increased (+0.03%) to 87.033% when pulling e5625d080bcf1678c14c80a7f5d9f7242b676914 on PeterAlfreadLee:zstd_deprecated_fix into 1e8d131ec08ae418f0140d8166d452dff6739937 on apache:master.

many thanks


Create a plan of Ms thesis

@andkorsh Please drop a line with your google-docs with plan of thesis. :exclamation:


Empty database?

Not sure if I did it in the right way but it showed the line: memory
after conversion is done.
However, when open the file with sqlite3, it doesn't show any tables. Am I missing anything?

I can't unfortunately tell what the issue might be from this sparse information. Could you please follow the instructions in https://github.com/dumblob/mysql2sqlite/issues/9#issuecomment-284164828 so that I can track down the issue? Thank you.