Fix TextureLoaderStore not always appending extensions to lookups

Closes https://github.com/ppy/osu/issues/18989

Tournament texture storage uses a StorageBackedResourceStore, which is not of type ResourceStore, therefore TextureLoaderStore can't append the image extensions and fail lookups:
https://github.com/ppy/osu-framework/blob/e918d33889a39c4895c01bcca384e88c89b52a57/osu.Framework/Graphics/Textures/TextureLoaderStore.cs#L24-L29
This is a regression from https://github.com/ppy/osu-framework/pull/5240, since prior to that PR, TextureStore appends extensions as well (this is also the reason why previously texture store sometimes looks up textures with double extensions, i.e. texture.jpg.png).
Instead, I've changed TextureLoaderStore to explicitly construct a resource store and add the given store along with the right extensions there, rather than safely cast and silently fail.
Note that SampleStore and TrackStore suffer from the same issue, which would fail similarly if constructed with a non-ResourceStore<T> store (e.g. StorageBackedResourceStore).

Tests need attention here.

Turns out in master, TextureLoaderStore populates the texture extensions to game.Resources, due to...
https://github.com/ppy/osu-framework/blob/604724157b84250bdf6ea02a7f1d124fd58ed8b5/osu.Framework/Game.cs#L181
https://github.com/ppy/osu-framework/blob/604724157b84250bdf6ea02a7f1d124fd58ed8b5/osu.Framework/Game.cs#L234-L235
(notice Resources passed to addFont, which creates a texture loader store of it)
that is also the reason why we see lookups like file.mp3.jpg all over the place game-side.
This is relevant in the test failure because the test was never supposed to work without specifying the extension, but it was actually working because of the png/jpg extensions being added to game.Resources:
https://github.com/ppy/osu-framework/blob/4fef626f03dbf53f8954aa896526563dd1f0e42a/osu.Framework.Tests/Visual/Sprites/TestSceneTextures.cs#L264-L266
(notice game.Resources there in the store flow)

I'll run tests game-side and confirm this doesn't regress skins and storyboards just to be sure.

Can't seem to find any issues so far, this is ready for review.


SAM causing certain unplayed games/software to be listed as "already in your Steam library" on store pages

Once the list of games for SAM populates with Steam open, certain games/software end up being listed as "already in your Steam library" on their respective store pages despite having never played/used them.  Some of the games/software include:

Alien Swarm
Dota 2
Estranged: Act 1
Eternal Silence
Source Filmmaker

Example of what I see for Source Filmmaker:

The above games/software do not appear in my Steam library (not hidden either), nor do I have licenses under my account for them, which is a cause for concern.  The issue persists when logging into Steam via browser.  Is there any way to fix this problem?  Any help would be appreciated!

Ok, so I've looked into this, and it seems that some games are tied to all Steam accounts, regardless of whether or not you've played them.  If anyone else can chime in and confirm that this is the case, then the issue can be closed.


Pillow/PIL still doesn't work

Installed Pillow 6.0.0, shows error on both local runtime, and Google Colab
Attribute error: module scipy.misc has no attribute 'imresize'
I'm using python 3.6.2

Instead reinstalled scipy==1.0.0 using pip.
That worked.


Update pixels.json

Added my pixel to the file.

Checklist

[x] I ran npm test locally and it passed without errors.

[x] I only edited the _data/pixels.json file.

[x] I entered the username in the pixels.json that I'm also using to create this pull request.
[x] I acknowledge that all my contributions will be made under the project's license.

I've already merged.


chore: delete useless buffered activation

For QLoRA models, we do not need to update the $\mathbf{W}$, so the buffered activation of $\mathbf{A}$ is useless. It is suggested not to save $\mathbf{A}$ in ctx to save the memory.

Great catch! This is the bug that caused a lot of memory issues with QLoRA. I cannot believe I missed this bug for such a long time. This will very significantly reduce the memory footprint of QLoRA fine-tuning with larger batch sizes or longer sequence lengths.

Jup, great work @Ther-nullptr
Just out of curiosity, how did you become aware of this, track this down?

I'm aiming to do a patch release this week to make sure this reaches all users asap. We've had this on our backlog for a bunch of time, but it's been a bit tricky to track down and then we had to deprioritize it as our limited resources were tied up in the multi-backend-refactor stuff (and being sick)..
Thank you so much for this extremely helpful contribution, @Ther-nullptr !

Ah, just saw your repo and this commit, really interesting!
Please tag me on any topics that supports your use-case, helps unblock you. We've had a bit of an impasse with maintenance recently, this way you can be sure it doesn't slip our radar.

Thank you so much for this extremely helpful contribution, @Ther-nullptr ! Saw your repo and this commit, really interesting! Please tag me if there's anything else we can help you with in that effort.
Release with this fix just went live!

I'm quite new to qlora's repo, what is the $A$ and $B$ in this context? Are they from LoRA's $LoRA_A$ and $LoRA_B$? If yes why that we do not need to store the activation of $A$ since $A$ and $B$ is trainable?


Merge campaign, keyword, other ref updates

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

BingAds/scripts/concepts/bid-strategies.md
:white_check_mark:Succeeded

BingAds/scripts/includes/final-url.md
:white_check_mark:Succeeded

BingAds/scripts/includes/reads-limit.md
:white_check_mark:Succeeded

BingAds/scripts/reference/AdGroup.md
:white_check_mark:Succeeded

BingAds/scripts/reference/AdGroupBuilder.md
:white_check_mark:Succeeded

BingAds/scripts/reference/AdGroupIterator.md
:white_check_mark:Succeeded

BingAds/scripts/reference/AdGroupOperation.md
:white_check_mark:Succeeded

BingAds/scripts/reference/AdGroupSelector.md
:white_check_mark:Succeeded

BingAds/scripts/reference/AdGroupUrls.md
:white_check_mark:Succeeded

BingAds/scripts/reference/Campaign.md
:white_check_mark:Succeeded

BingAds/scripts/reference/CampaignIterator.md
:white_check_mark:Succeeded

BingAds/scripts/reference/CampaignSelector.md
:white_check_mark:Succeeded

BingAds/scripts/reference/CampaignUrls.md
:white_check_mark:Succeeded

BingAds/scripts/reference/Keyword.md
:white_check_mark:Succeeded

BingAds/scripts/reference/KeywordBidding.md
:white_check_mark:Succeeded

BingAds/scripts/reference/KeywordBuilder.md
:white_check_mark:Succeeded

BingAds/scripts/reference/KeywordIterator.md
:white_check_mark:Succeeded

BingAds/scripts/reference/KeywordOperation.md
:white_check_mark:Succeeded

BingAds/scripts/reference/KeywordSelector.md
:white_check_mark:Succeeded

BingAds/scripts/reference/KeywordUrls.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Add more documentation about overriding the Datagrid styles

The Datagrid component expect an object with specific keys to override its styles. We should add more documentation about it.

Fixed by #165


support clustering on BigQuery

Feature
Feature description
BigQuery recently announced support for table clustering. With this new feature, a list of up to four "clustering columns" can be supplied to a create table as (...) statement. BigQuery will use the clustering columns to colocate similar data. This should speed up filters and aggregations, and BigQuery can both skip scanning whole blocks of data, as well as avoid shuffling data between nodes for certain aggregations. Docs: https://cloud.google.com/bigquery/docs/creating-clustered-tables
Who will this benefit?
This feature will benefit users of BigQuery. It will work in concert with table partitioning.
Proposed Implementation
This feature can be implemented in the BigQuery implementation of create_table_as(). It's implementation will look similar to the partition_by implementation.

cluster_by can't be used without partition_by. Should we enforce that in the macro, or let the database return the error when a user doesn't specify both?

Yeah, I'm fine with letting BQ return an error in that scenario

Fixed in #978


Add maus>=0.4.1 as a dependency

To sort files into format specific folders instead of Future/Current/Past, we first need to add maus>=0.4.1 as a dependency. This is done here.

to be explicit: add maus>=0.4.1 also to this list:
https://github.com/Hochfrequenz/edi_energy_scraper/blob/55b12e1872b1b40db6be2be47b0f78a44fc122cc/pyproject.toml#L36


Background Colors Still White In Dark Mode

Describe the bug
Background of fields are still white though in dark mode
To Reproduce
Steps to reproduce the behavior:

Change to dark mode

Expected behavior
Background colors of the fields should change to a darker color or be derived from app theme
Screenshots

Smartphone (please complete the following information):

Device: Samsung Note 10 Lite
OS: Android 11

hi @nivla360 , we are working on this feature, you will get this feature in next version


I met a login network error problem

I'm in China, i'm integrating with line, used a line login for our JP customers, but when i test line login, some phones are just doing fine, on Mi4 phone always met this problem(see screenshot above), please help me or give me some suggestion on this.  thanks a lot!

@Kevin-Misfit
Sorry our servers IP are blocked by the great firewall....


Add PaLM-2 tokenizer when available

The tokenizer is most likely not correct but there is no official tokenizer for this model yet. The paper (See report) mentions the mT5 tokenizer, but it is not clear if this is the same tokenizer that is used for the mT5 model.
This question has been asked in the Google Cloud Community: link
As of 2023-12-05, there is no answer yet.

Done by #2138.


Github Actions versions bump

Updated actions/upload-pages-artifact to v2
Updated actions/deploy-pages@v3
Fixed README

Thanks, the fixes resolved the issue 👍


Running python src/test/s3.py list-buckets gives me an error.

Hi TWCC-CLI team,
Our project is deducted by 「雲端資源」 service:

But on the web interface, I can't find anything related to that deduction:

So, now I'm trying to use TWCC-CLI to find out more information, but when I ran the command python src/test/s3.py list-buckets, I ran into the following bug:

Is it normal?
Or should I wait for v0.5 before further investing that deduction problem?

hi, there,

TWCC-CLI v0.1 is no longer maintain. please try v0.5 series
and try to install pip install TWCC-CLI==0.5.12rc1 and sync all objects by using twccli cp cos -bkt $BUCKET_NAME -dir $LOCAL_DIR -sync from-cos
you also can use s3cmd to access TWCC COS service.

TWCC COS is compatible for all object storage services, thanks for using TWCC.

Hi @Nash2325138 ,
It's a message from TWCC-CLI team.
Is everything goes right?
The version v0.5.12 will be released this friday. And this issue will be closed in next week Jun. 6th.
Let us know, if you have any suggestion or question! Thank you!!

I was working on some important deadlines last week and haven't tested it yet.
I will use the v0.5.12 tomorrow and will let you know if I encounter any problem

Hi,
I installed the new version twcccli via pip (inside a container) following this doc. The installation is successful and the following texts showed up:

, but when I typed twccli config init, this error popped out

The same error happens on other commands like twccli ls ccs and twccli ls cos
It persists after I
export LANG=C.UTF-8
export LC_ALL=C.UTF-8
export PYTHONIOENCODING=UTF-8

What should I do to get rid of this error?

Hi @Nash2325138 ,
Thank you for using the latest version v0.5.12!
We've found that there are python package problem in your error message.
Please tried:
rm -rf ~/.twcc_data
rm -rf ~/.twcc_data
pip install TWCC-CLI
to re-install the CLI tool.
Let us know, if you have any suggestion or question!

If everything goes well,  #7 will be closed on June 15th.
Thanks!

I just removed ~/.twcc_data and re-install TWCC-CLI via pip. The previous error is solved, but another popped out:

Is there any possible mistake I've made?

Hi @Nash2325138
Sorry about the command mistake I wrote in this block. Please tried:
rm -rf ~/.twcc_data
rm -rf ~/.local
pip install TWCC-CLI

to solve the package error.
And for the error message above, please check out our User Manual (⚠ draft!! ⚠ ) in #1-3, to see if your project code been typed correctly.
Let us know, if we solved your issue and problems above!
Also, if there is any progress, please let us know promptly!! Thank you!

Hi, I solved the problem after using the project code shown on the page "TWCC API 金鑰管理" instead of the one shown in "iService 計畫資訊".
I'll let you know if I have any other problem

Hi, I've successfully executed twccli ls cos, and no Cloud Object Storage's object shows up.
We suspect the mysterious deductions mentioned in the first post are caused by the not-suspended containers when TWGC services were closed. We will ask about this by another email.
Anyway, thanks for the troubleshooting to help me use TWCC-CLI! (You can close this issue now)

Hi @Nash2325138
Thank you very much for letting us know your problem using TWCC-CLI!
Hope you get the solution for the deduction problem, after connect to our Support team!
All the questions, concerns, or suggestions are very welcome! :)


Add subcommand to convert to/from various result formats

This can be very useful when comparing results from experiments when the outputs were stored in different formats.

This is a good start, but there are several more use cases that I think need to be covered. Extend this to include the following use cases:

I have the DEM, not the circuit, and want to convert detection event data.
I have neither the DEM nor the circuit but I know all the values that would be derived from them like the bits per shot
I know how many bits per shot, but I don't know if it's detection event data or measurement data (and I'm not using the DETS format which is the only one where that distinction matters)
The input file has observable data appended to the detector data, and I want to split the observable data out into a separate file

These use cases all kind of differ in what information they need. In general the command should infer everything it can from whatever the user provides, and if it can't infer a needed quantity it should raise an error identifying that quantity and saying various ways it can be provided.

Thanks, that is a very good point. I naively assumed the user would always have the circuit on hand. I added support for all of these cases, but there is another I'm wondering if we should add. For DETS files we can also attempt to parse the text looking for the value prefixes, but this may not be reliable as the file would need to include the max index of each type, which may not always occur. What do you think?

Because it's not reliable to infer sizes or types from a det file, I would not attempt to do so.


Sort

I think that it's more convenient if we could sort article by our will?

Yes sir, absolutely. Drag and drop would be nice. There's a nice library here to handle resorting of lists (could be modified)
https://github.com/bgrins/nativesortable

Drag and drop! It's fascinating!
I thought about 'sort by date', 'sort by title', ... etc.
But drag and drop is better!


Is spack-stack 1.7.0 GNU broken on Orion/Hercules after maintenance on 2024/07/24?

Describe the bug
People are reporting what looks like a broken MPI-related environment for spack-stack 1.7.0 GNU broken on Orion/Hercules after maintenance on 2024/07/24.
To Reproduce

Orion:

$ module purge
$ module use /work/noaa/epic/role-epic/spack-stack/orion/modulefiles
$ module load ecflow/5.8.4

$ module use /work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.7.0/envs/ue-gcc/install/modulefiles/Core
$ module load stack-gcc/12.2.0
$ module load stack-openmpi/4.1.6

$ mpicc
mpicc: error while loading shared libraries: libslurm_pmi.so: cannot open shared object file: No such file or directory

System:
MSU machines

@AlexanderRichert-NOAA and @RatkoVasic-NOAA, I've seen your involvement on https://github.com/JCSDA/spack-stack/issues/981, so I was wondering if ate least one of you should be aware of this one.

@fabiolrdiniz, thanks for reporting this. I can get it to work by adding /opt/slurm/lib/slurm to my LD_LIBRARY_PATH. For example:
(venv) orion-login-2[60] herbener$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/slurm/lib/slurm
(venv) orion-login-2[61] herbener$ mpicc --version
gcc (GCC) 12.2.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

(venv) orion-login-2[62] herbener$

Not sure why this is necessary, or what changed. Any ideas about what has happened, and the proper way to fix?  Thanks!

The sysadmins probably removed/changed the slurm installation - they've done that in the past and that broke the OpenMPI builds.

You could add the workaround to https://github.com/JCSDA/spack-stack/wiki and be done with it. Future spack-stack releases may need to include the LD_LIBRARY_PATH modification in the meta modules (for example by adding them in the compiler config), or in the openmpi modulefile (ourselves if we built them, or ask the sysadmins if we use their OpenMPI).

Thanks for the pointer to the spack-stack wiki page Dom. I've added the LD_LIBRARY_PATH workaround so all the tips are in one spot.

I added new path to LD_LIBRARY_PATH in openmpi/4.1.6.lua.
@fabiolrdiniz can you please try with same loading procedure.

Thank you for the quick replies.
@srherbener and @climbfuji, thanks for the suggestions.
@RatkoVasic-NOAA, yes, I can confirm it now runs fine with the comments above, and the path is augmented (please, notice the two locations... should be fine):
$ echo -e ${LD_LIBRARY_PATH//:/\\n}
/opt/slurm/lib/slurm
/work/noaa/epic/role-epic/spack-stack/orion/installs/openmpi/4.1.6/lib
/work/noaa/epic/role-epic/spack-stack/orion/installs/gcc/12.2.0/lib
/work/noaa/epic/role-epic/spack-stack/orion/ecflow-5.8.4/lib64
/work/noaa/epic/role-epic/spack-stack/orion/ecflow-5.8.4/lib
/usr/lib64
/usr/lib
/opt/slurm/lib

Closing this one, but feel free to re-open if needed.

@fabiolrdiniz is that same thing like on Orion?

Unfortunately, yes. Thanks for asking.

Yes, I still have the same MPI issue on the hercules

I'm looking into that.

Spack has a behavior where if an external package is a link, then it fully resolves the link when it goes to the linking stage of compiling, i.e. /opt/slurm -> /opt/slurm-XX.yy at build time gets the full path hard coded. You can see this by ldd -ing the executable. When we upgrade slurm for security patches and remove the old slurm, it breaks mpi builds.
Our helpdesk/admin team has created a non-production node where slurm isn't a link, and that has fixed this spack behavior. If you need a different openmpi, send us a ticket and we'll get it created. gcc-12.2.0+openmpi-4.1.4 should be working now, following the previous rebuild issues.

@fabiolrdiniz and @huishao-r
Please check if this change works for you:
I renamed required openmpi from 4.1.6-gcc-12.2.0-spack to 4.1.4 in gcc/12.2.0/stack-openmpi/4.1.6.lua modulefile.

I didn't have the original MPI issues. But I got the following message during build:
CMake Warning at /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.7.0/envs/ue-gcc/install/gcc/12.2.0/ecbuild-3.7.2-nnjboq2/share/ecbuild/cmake/ecbuild_add_executable.cmake:209 (add_executable):
  Cannot generate a safe runtime search path for target
  test_fv3jedi_variablechange.x because there is a cycle in the constraint
  graph:

    dir 0 is [/work2/noaa/jcsda/hshao/hercules/JEDI_ROOT_gnu/ufs-bundle.HailingVersion+ROPP_202407/build/Depends/lib]
    dir 1 is [/work2/noaa/jcsda/hshao/hercules/JEDI_ROOT_gnu/ufs-bundle.HailingVersion+ROPP_202407/build/lib]
    dir 2 is [/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.7.0/envs/ue-gcc/install/gcc/12.2.0/ecmwf-atlas-0.36.0-tgmbvjc/lib64]
    dir 3 is [/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.7.0/envs/ue-gcc/install/gcc/12.2.0/fckit-0.11.0-njbdjuc/lib64]
    dir 4 is [/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.7.0/envs/ue-gcc/install/gcc/12.2.0/eckit-1.24.5-a4uan4t/lib64]

@RatkoVasic-NOAA, it seems to be working fine now. Please, see below:
$ module purge
$ module use /work/noaa/epic/role-epic/spack-stack/orion/modulefiles
$ module load ecflow/5.8.4
$ module use /work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.7.0/envs/ue-gcc/install/modulefiles/Core
$ module load stack-gcc/12.2.0
$ module load stack-openmpi/4.1.6

$ mpicc
gcc: fatal error: no input files
compilation terminated.

$ echo -e ${LD_LIBRARY_PATH//:/\\n}
/opt/slurm/lib/slurm
/work/noaa/epic/role-epic/spack-stack/orion/installs/openmpi/4.1.6/lib
/work/noaa/epic/role-epic/spack-stack/orion/installs/gcc/12.2.0/lib
/work/noaa/epic/role-epic/spack-stack/orion/ecflow-5.8.4/lib64
/work/noaa/epic/role-epic/spack-stack/orion/ecflow-5.8.4/lib
/usr/lib64
/usr/lib
/opt/slurm/lib

Thanks again to everyone involved. Closing it... reopen if needed.


images taken from iphone 6(maybe others too)  webcam doesnt work

there is a recomended resolution or anything?

I'd double check if photo orientation is right - iphones rotates photos, js-zxing-pdf417 does not recognise photo orientation from exif.


Date issue with draft data

Steps to reproduce:
(NB: this occurs in date fields where validation is set to disable user from selecting date after current date)

Load the form https://enketo.ona.io/x/#Yp7k, set the date of interview to the current date and fill up some of the form and then save the data as draft
Change the computer date to the next day and load the draft data
Change the date of interview to the current date . The validation error "Date cannot be greater than current date" shows up making it impossible to select the current date
Wondering what could be the cause of this error?

moving this question to the forums instead

Thanks. It probably would an enketo-core issue, so it's good to put it here.  I am aware of some issue with timezones, but if you do step 1 and 3 in quick succession and the error doesn't show up with #1, it must be something else.

from forum:

The validation error "Date cannot be greater than current date" shows up making it impossible to select the current date
My guess is this is happening because in the xlsform the constraint has been set to .<=${today} whereby ${today}==today. What are some of the workarounds for this?

If you could investigate further, that would definitely speed up the resolution.  Does the same occur with . <= now() or . <=  today()?

Also please add an XLSForm or XForm to this thread. It can be tricky to reproduce otherwise.

I've tried both . <= now() or . <= today() and still doesn't work.  When I use . <= today() with my computer date as 30-11-2016, the date error shows up and only accepts dates <= 29-11-2016.
Sorry for taking long to test this out...
Here is a sample xlsform https://drive.google.com/open?id=0B1A3BnI7x8VObl84blFnYUVzOXM

Thanks!

in Denver, set computer time to around 4am
form with date input and constraint . < today()
today's date should NOT be acceptable but it is

in Denver set time to around 3:50pm
form with date input and constraint . >= today()
todays' date should be acceptable but isn't

I think this thread started to cover 2 separate issues. The constraint logic should now be working (e.g. ". < today(). That issue was split off into #522.
The issue with loading an older record that used to pass a date-cannot-be-greater-than-today constraint at the date is was collected, but no longer passes that constraint a few days later, is something else.
I'm not quite sure what to do there. We cannot selectively exclude validation checks of some types of constraints.


Paired t-test Table can't render

A clear and concise description of what the bug is.
Expected results
It should show table values like p-value、lift.
Actual results
show empty table.
Screenshots

How to reproduce the bug

create Paired t-test Table chart
select metrics and group fields
click run

Environment
(please complete the following information):

superset version: 0.17.12
python version: Python 3.7.9
node.js version: v14.0

Checklist
Make sure to follow these steps before submitting your issue - thank you!

[x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[x] I have reproduced the issue with at least the latest released version of superset.
[x] I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context

Closing this as I don't believe it's an issue on the latest version of Superset. Feel free to reopen if this is incorrect.

We currently use v1.3.0 and the problem still seems to be present. Doesn't matter which settings, metrics or group by we choose, no data is actually rendered. Also the chart type isn't in the deprecated category.
Is it supposed to work? Does anyone have more information on that? Couldn't really find any current information on that.

Im using latest version v2.1.0 and this issue persists. I see empty graph


How do i clone jetcaster/jetchat in android studio?

when i try to do this via git, I get an error

Can you share what steps you're following and what the error is?

get from VCS - link
clone failed - repository not found

Why closed lmao? You don't help me

урод


Fix the issue of case sensitivity when matching keys with cache enabled

What's the purpose of this PR
Fix the issue of case sensitivity when matching keys with cache enabled

Add a configuration config-service.cache.key.ignore-case to control whether the cache key is case-sensitive.

Which issue(s) this PR fixes:
Fixes #3529

This attempt is commendable, but a problem may arise if the database is set to be case-sensitive. In such a scenario, the cache loader would fail to fetch data from the database, since the appid, cluster, and namespace have all been converted to lowercase.

but a problem may arise if the database is set to be case-sensitive. In such a scenario, the cache loader would fail to fetch data from the database, since the appid, cluster, and namespace have all been converted to lowercase.

There is indeed such a situation, and this PR also considers it. It is currently not enabled by default, and needs to be enabled through a configuration switch. This has also been prominently explained in the documentation.


⚠ ClamAV has degraded performance

In 286d82e, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 194 ms

Resolved: ClamAV performance has improved in 9651c66.


hydra: Add extracopy.sh script

Please consider fixing shellcheck reported issues:
$ shellcheck containers/hydra/extracopy.sh containers/hydra/hydra_conf.sh

In containers/hydra/extracopy.sh line 94:
      export NIX_SSHOPTS="-i /home/hydra/.ssh/key -p $(cat /home/hydra/upload_port.txt)"
             ^---------^ SC2155 (warning): Declare and assign separately to avoid masking return values.

In containers/hydra/extracopy.sh line 100:
    cd "$SDIR"
    ^--------^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

Did you mean:
    cd "$SDIR" || exit

For more information:
  https://www.shellcheck.net/wiki/SC2155 -- Declare and assign separately to ...
  https://www.shellcheck.net/wiki/SC2164 -- Use 'cd ... || exit' or 'cd ... |...

You might also want to use this opportunity to fix the shellcheck issues for all other shell scripts under containers/hydra.


Add Google AI/Gemini AI Extension

Extension built to add support for Google's Gemini AI series of models by Google AI studio, supported models are

Gemini 1.5 Pro 002 (Latest)
Gemini 1.5 Flash 002 (Latest)
Gemini 1.5 Pro Experimental 0801 (Experimental Model)
Gemini 1.5 Pro Experimental 0827 (Experimental Model)
Gemini Experimental 1114 (Experimental Model)
Gemini 1.5 Flash Experimental 0827 (Experimental Model)
Gemini 1.5 Flash 8b
Gemini 1.5 Flash 8b Experimental 0924 (Experimental Model)
Gemini 1.0 Pro (Legacy Model)

Hey @ProdeusUnity! Thanks for submitting another PR.
A few comments on this one:

In the most recent version of the master branch, config.py has been moved out of the /extensions directory and into the root directory (as config.py now includes several non-extension-related variables). Could you update your PR to match this?
In your new requirements.txt, you've included both "google.generativeai" and "genai". Since you're importing google.generativeai as genai, I think only the "google.generativeai" line (in requirements.txt) is needed.
Regarding the model selection, I like the inclusion of the models with the "latest" tag, as these stay up to date without requiring us to make code changes in the future. Google seems to be preparing to remove support for at least one of these models in the near future, and I imagine they won't support the experimental models as long as the core ones. What do you think about including only those models with the latest tag? Maybe just gemini-1.5-pro-latest, gemini-1.5-flash-latest,  and gemini-1.5-flash-8b-latest?

Hi, I can certainly do that, the reason for "GenAI" and "google.generativeai" in the requirements file is that I found it doesnt seem to work right if I don't have both installed via pip. Regarding config.py, I'm not sure what you mean by it being outside the directory, it is on my side, do you mean the code needs to be updated to reflect this? As for models, yes I can change the choices to just the latest ones, experimental models were purely there just since they were offered by the API.

Alright, Config issue has been fixed, and experimental/legacy models have been removed. The only thing that won't be changed is requirements.txt as it does not work without both

One second I broke something, ugh

I'll have to look at what's breaking it, the original config in the older version works fine, but as soon as I update it to use the config in the root directory, suddenly responses arent sent anymore, I'll update when its fixed


test config before reloading [untested]

⚠ This is an untested change for now ⚠
Question/discussion:
In general, wouldn't it be better to test the config before reloading nginx.
The expected outcome would be to have nginx still running, with your old config - instead of loosing connection through reverse-proxy to all your sites.
Related to

https://github.com/jwilder/nginx-proxy/issues/502
https://github.com/jwilder/nginx-proxy/issues/438

CC: @handcode

even with this, if you restart the nginx-container, nginx will fail to start. This just delays the problem IMHO.
I'm afraid users will then wonder why their new container is not considered by the nginx-proxy and this will end up in a support request here.

@thomasleveil Yes that's correct.
But the issue was more like, if there's a broken config somehow and you try to reload, nginx will fail and all your virtual-hosts will be offline, i.e. it's related to https://github.com/jwilder/nginx-proxy/pull/585
I'd say to original behavior might be more desirable in development, while the patch is definitely the way to go in production.
What's about using an ENV variable for this?

Any news here? Without a fix for this we end up loosing connection to all sites, just because of one broken config/setup.


Optimize position of login popup for multi-monitor setups

Recently I changed where we're positioning the login popup as part of this commit.
While this will ensure the popup is displayed in the center of the screen for typical single-monitor setups, it fails to position it in a multi-monitor setup.
This behavior is also not consistent between browsers,

In Chrome, if you open the popup from your primary monitor, it correctly positions it. But if you do it from the secondary monitors, it will fail to center the popup (it shows the popup on the same monitor).

In Firefox, as long as you have more than one monitor, it fails to position the popup properly. It also opens the popup on a different monitor than the one you're looking at.

Therefore, we should look into other ways to position the popup in multi-monitor setups. I tried some of the solutions mentioned here and here. But all of them are making assumptions about the number of monitors or their orientation.

Current Implementation :
We are subtracting screen width and popup width and dividing it by two and giving that as the left position to centre the popup. This approach will not work in multi-monitor setups as we need to add primary screen width to the left offset.
const popupWidth = Math.min(800, screen.availWidth);
const popupHeight = Math.min(750, screen.availHeight)
const topOffset = 50;
const popupLeft = (screen.availWidth - popupWidth) / 2;
// top should just have some small offset if there's available space
const popupTop = (topOffset + popupHeight) < screen.availHeight ? topOffset : 0;
const win = window.open('', '_blank', width=${popupWidth},height=${popupHeight},left=${popupLeft},top=${popupTop});
// focus on the opened window
win?.focus();
// keep track of the window, so we ensure only one is opened at a time.
popupWindowRef.current = win;
_logInHelper(win, postLoginCB, 'popUp', null, logAction);
Different Approaches I tried:

Approach: Tried calculating popup position based on window resolutions.
Problems Faced: It was causing problems when it is a single screen and if window is minimised the pop takes up the entire screen area.
Link referred: https://stackoverflow.com/questions/4068373/center-a-popup-window-on-screen

Approach:

We need to add left offset to centre the popup on secondary monitor. Left offset is the width of the first monitor. Tried calculating based on screen resolution. Used screen. availLeft - (Returns the first available pixel from the left side of the screen.)
const left = Math.max(0, (screen.availWidth - popupWidth) / 2 + screen.availLeft);
const openedWindow = window.open('', '_blank', width=${popupWidth},height=${popupHeight},left=${left},top=${popupTop});
if (openedWindow && openedWindow.focus) {
openedWindow.focus();
}
This approach gave the expected behaviour in Chrome and Firefox. It centres the popup and opens the popup in the right window.
Problems Faced: .screen. availLeft is non-standard API and we cannot use it
Link referred: https://developer.mozilla.org/en-US/docs/Web/API/Screen/availLeft

Approach: Used window.screenLeft instead screen.availLeft
Problems Faced: In a single screen scenario, if your current window is in the middle of the screen, the popup will show up to the right of it.

Approach: Tried to build a wrapper for screen.availLeft.
we have to calculate the width of the primary screen as well to centre it on the secondary monitor and add that width to the horizontal offset.
Now the challenge is if the secondary screen is on the right or left of laptop. Based on which we have to calculate right and left offset.

screen.availLeft does the same:
If you work with two screens this property, evaluated on the right screen, returns the width of the left one in pixels (thereby indicating the X coordinate of the left edge of the screen on the right).
[1] [2] - on left screen availLeft returns 0, on the right screen it returns the width of the left one
[2] [1] - on left screen availLeft returns -width of that screen, on the right screen, it returns 0
For this we don't have a built-in approach to calculate width of primary screen if the window is on secondary monitor. There are third-party solutions like:
npm packages screen-info. It is a very old repo and uses python2.
window.getScreenDetails() - This gives the screen details and details of screens connected to the monitor. But this is non-standard and works only for chrome.
Overall, in my investigation there are solutions more based on window parameters than screen


Created additional indices

Changes
Looks like Foreign key required index setup. Also added additional indices for feature.
How to test these changes?

Related issues

Closing this because we need to change schemas (to make numeric values numeric).


Refresh on ErlPort mechanisms

What comes with ErlPort:

https://github.com/hdima/erlport/blob/master/src/python.hrl
https://github.com/hdima/erlport/blob/master/src/python.erl
https://github.com/hdima/erlport/blob/master/src/python_options.erl
https://github.com/hdima/erlport/tree/master/priv/python3/erlport

What needs to be created for CL? Probably something analogous:

[ ] erlang.lisp
[ ] erlproto.lisp
[ ] erlterms.lisp
[ ] stdio.lisp

With the Common Lisp project https://github.com/flambard/cl-erlang-term much of this should already be done.


Release v2.1.0 is not backward compatible

Environment details

OS: Linux
@google-cloud/common version: v2.1.0

While compiling https://github.com/googleapis/cloud-profiler-nodejs, we see the error message:
Tue Aug  6 00:07:36 UTC 2019: ts/src/profiler.ts:444:9 - error TS2345: Argument of type '{ method: string; uri: string; body: { deployment: Deployment; profileType: string[]; }; json: boolean; timeout: number; }' is not assignable to parameter of type 'DecorateRequestOptions'.
Tue Aug  6 00:07:36 UTC 2019: Types of property 'method' are incompatible.
Tue Aug  6 00:07:36 UTC 2019: Type 'string' is not assignable to type '"GET" | "HEAD" | "POST" | "PUT" | "OPTIONS" | undefined'.
I believe https://github.com/googleapis/nodejs-common/pull/448 changed the interface for DecorateRequestOptions.method. Release v2.1.0 probably shouldn't be a minor release, as the change is not backward compatible.

This should be resolved, as we rolled back the dist tag on @google-cloud/common to a compatible version 👍

v2.1.1 is also not backward compatible for typescript compilation.
We see the same errors again with the latest release.


Add support for MSX disk formats

For MSX disk images I currently use atarist.360 and atarist.720. While this works MSX does support more formats (40 track based) and also I would suspect MSX format is more like IBM's (given Microsoft's involvement). I would propose the following to diskdefs.cfg. I am not sure about iam, cskew and hskew used in atarist formats.
`disk msx.1d
cyls = 40
heads = 1
tracks * ibm.mfm
secs = 9
bps = 512
gap3 = 84
rate = 250
end
end
disk msx.1dd
cyls = 80
heads = 1
tracks * ibm.mfm
secs = 9
bps = 512
gap3 = 84
rate = 250
end
end
disk msx.2d
cyls = 40
heads = 2
tracks * ibm.mfm
secs = 9
bps = 512
gap3 = 84
rate = 250
end
end
disk msx.2dd
cyls = 80
heads = 2
tracks * ibm.mfm
secs = 9
bps = 512
gap3 = 84
rate = 250
end
end`
I will create some flux dumps of freshly formatted disks (I will gw erase them prior to formatting on real hardware). I am not sure I have a machine that supports the 40 track format but will check.

msx1dd.zip
msx2dd.zip

Found a National FS-5500 which supports the 40 track option during formatting (all MSX's can read such disks afterwards). When issueing the call fornat command the following is printed on screen:
[code]Drive name? (A,B)
1 - 1 side
2 - 2 sides
3 - 1 side,  double track
4 - 2 sides, double track[/code]
The following two flux dumps are from option 1 (msx1d.scp) and 2 (msx2d.scp). The 3.5" HD drive seems to double step automatically upon reading (so I dumped cylinders 0 to 41 for both). Or do you think the drives are not double stepping (and only cylinders 0-39 are used)? That would seem a bit odd though...
msx1d.zip
msx2dd.zip

There's no interleave or skew. So your config lines are correct.

Thx!


Proprose docvim inclusion

New to Stackage, so hopefully not doing anything too egregiously wrong here.

I guess it doesn't build with ghc-8.0:
base-<IP_ADDRESS> is out of bounds for:

[ ] docvim-<IP_ADDRESS> (==4.8.*). Greg Hurrell<EMAIL_ADDRESS>@wincent

@juhp: I relaxed the base dependency in <IP_ADDRESS>, confirmed that it builds with GHC 8.0.1 at least (tested with nightly-2016-06-15 resolver).


How can I allocate different AMD gpu device to different process?

Suppose, on the machine, there are 2 AMD gpu devices. How can I make process 1 use device0, process2 use device1?  Is there an environment variable like "CUDA_VISIABLE_DEVICES" to set visible AMD GPU devices for processes?
Dy default, are AMD GPU devices shared by all processes?

If it is OpenCL application, set the GPU_DEVICE_ORDINAL environment parameter could help.  If it is HC or HIP application, does this GPU_DEVICE_ORDINAL  take effect to allocate gpu device to application?

There're a couple of methods to expose only selected GPUs to the user process for hip/hcc path:

Use HIP_VISIBLE_DEVICES environment variable to select the target GPUs for the process from the HIP level. e.g. use the following to select the first GPU:

export HIP_VISIBLE_DEVICES=0

Use ROCR_VISIBLE_DEVICES environment variable to select the target GPUs from the ROCr (ROCm user-bit driver) level. e.g. the following to select the first GPU:

export ROCR_VISIBLE_DEVICES=0

Pass selected GPU driver interfaces (/dev/dri/render#) )to Docker container. e.g. use the following docker run command option to select the first GPU:

sudo docker run -it --network=host --device=/dev/kfd --device=/dev/dri/renderD128 --group-add video
Note you should see the following four interfaces if you have a 4xGPU system:
$ ls /dev/dri/render*
/dev/dri/renderD128  /dev/dri/renderD129  /dev/dri/renderD130  /dev/dri/renderD131

@sunway513  many thanks for your response. It is real helpful to me.  I am new for AMD GPU world, and I have no real environment for testing and experiment. Could I know more about ROCR_VISIBLE_DEVICES?
For ROCR_VISIBLE_DEVICES environment variable, does it work for both HCC, HIP and openCL applications?
Is there any environment pre-condition to use the ROCR_VISIBLE_DEVICES environment variable for exposing selected devices? What is difference between ROCm user-bit driver and driver installed from rocm-dkms primary meta-package?
If I install rocm platform via rocm-dkms primary meta-package way,  then I could leverage this variable to expose selected gpu devices for different application process?

Hi @nanguanqi , you are welcome :-)
ROCR_VISIBLE_DEVICES operates on the ROCm ROCr runtime, that's under the layer below hip/hcc/math-libs etc; therefore, I'd assume it'll work equally for OCL path in ROCm stack.
ROCm user bit drivers include ROCr and THUNK.
If you use docker as an example, those two user bit drivers shall be included inside docker container.
Rock-dkms, on the other hand, includes kernel driver (amdgpu) and device firmware, those must be installed on the bare metal.
THUNK and amdgpu kernel driver talks via /dev/dri and /dev/kfd interfaces, you can consult with the following docker run command for ROCm containers:
docker run -it --network=host --device=/dev/kfd --device=/dev/dri --group-add video
And to your last question, yes, that would work.

@sunway513 but how do i confirm that my process is using the right device?

Hi, you can open another terminal and watch for the GPU activities using the following command:
watch -n 0.1 rocm-smi

@sunway513, I got Radeon WX4100(node 1) and MI100(node 2), I've set export HIP_VISIBLE_DEVICES=2; export ROCR_VISIBLE_DEVICES=2 and did rocminfo which show only node 1(wx4100) while I'm expecting node 2((MI100). I tried all the combinations 0,1,2 and "0,1,2" but the MI100 card will never be active except for "0,1,2".


Export job start/end times

Firstly, RQ Exporter has been an excellent!
Do you think it'd be possible to include an option to export Job start/end times? I believe this should be accessible at job.started_at and job.ended_at per rq docs.

Hi,
Thank you for your suggestion, how do think these job start/end times should be exposed?
The only thing that I can think of right now is exposing the job duration but I haven't checked how to do it yet, if you have any suggestions about how we should implement this please provide us with more details.
I'm not a Prometheus expert but I think getting this amount of details about the jobs is maybe more suitable for another monitoring system, for example check out:

RQ Dashboard
RQ Monitor (Newer project)

But I haven't checked if these projects expose the job start/end times.

I think there's been two kind of monitoring systems I've seen so far for RQ:

Snapshots: what's going on right now - for example, using RQ Dashboard to check what workers are registered to which queue, getting errors, etc. I haven't looked at RQ Monitor before, but on first look it seems like it'd fall in this category + a bunch of nice new features
Load monitoring: what's the current processing capacity at - using RQ Exporter's Grafana dashboard to see the % of worker usage over time

The start/end time isn't as important as the job duration like you suggested. The use case I'm thinking would be:

Say I have a set of queues with some running at high priority where I care about the response time. Is at some point in time, for whatever reason, the jobs on these specific queues completing slower than normal?
Another point would be: is for whatever reason my job.func_name job suddenly running much slower than normal?

So kind of an average job duration per queue name view and/or per job.func_name.
I'm also not a Prometheus expert though, so happy to hear your thoughts.

Actually I thought that we can calculate the duration from the start and end times of the job.
As per the Prometheus documentation on histograms and summaries:

Histograms and summaries both sample observations, typically request durations or response sizes. They track the number of observations and the sum of the observed values, allowing you to calculate the average of the observed values. Note that the number of observations (showing up in Prometheus as a time series with a _count suffix) is inherently a counter (as described above, it only goes up). The sum of observations (showing up as a time series with a _sum suffix) behaves like a counter, too, as long as there are no negative observations. Obviously, request durations or response sizes are never negative.

The thing is that RQ is not like Celery which uses events, the RQ exporter queries Redis on each scrape by Prometheus to get the jobs and workers at the time of the scrape, so say for example a job was finished and deleted before this scrape, we won't know anything about it, also this means that we need to fetch all the jobs in the finished and failed registries and calculate their duration from their start and end times on each scrape.
That's why I can't think of a good way to track the job duration without events dispatched from the jobs.

Hmm, BaseRegistry.get_job_ids() does have a start/end range calling ZRANGE. Thoughts on scraping the last 10 jobs or so on each queue.finished_job_registry? I'm thinking it'd have the same effect for the use case with a histogram, though this could get expensive very fast with one call to get the IDs per finished queue and another per job to get start/end times.
If I remember correctly, the finished_job_registry includes the failed_job_registry.
The jobs should also be ordered with the score by timestamp + ttl, with jobs that never expire set to +inf.

I could be wrong but I still don't think we can record this data using a histogram/summary metric accurately, also as you said this operation could get very expensive.
About the job registries, the finished and failed registries hold the successful and failed jobs respectively, the finished_job_registry does not include the failed jobs.
Honestly, I really don't have time to start working on this feature, of course pull requests are welcome, but if the operation is expensive you need to implement it behind a flag and disable it by default.

That works, I'll have to take a further look on a good way to record this. Thanks for the insight.
We'll have a use case for this in Q4 this year, so I'll open a PR if it works out.


5.0 rc-[1-3]: running clean check together causes resources to be missing at test runtime

In a multiproject Kotlin build, running ./gradlew clean check with 4.10.2 works perfectly, and with 5.0 rc-[1-3], has test failures because a particular resource file (in src/main/resources) is not present in the classpath. However, with any version, running ./gradlew check by itself succeeds.
Expected Behavior
Running clean check together should produce the same behavior as clean, check separately.
Current Behavior
SomeClass::class.java.getResourceAsStream(path) returns null for paths that are most definitely in src/main/resources on 5.0 rc's when using clean check as one invocation, causing tests to understandably fail.
Steps to Reproduce (for bugs)
I haven't been able to isolate this in a simple test case (yet). Let me know if the info below doesn't produce any insight, and I'll give it some further effort.
In this project, one particular subproject (svc) is the subproject where the tests load resources and fail as mentioned above. svc also has the most build customization; the other subprojects are very simple kotlin projects with a few dependencies specified.
svc has db-related logic and uses the jooq plugin to generate jOOQ sources, which adds a task called generateApiSvcJooqSchemaSource.
It also uses the flyway plugin, which adds flywayClean, flywayMigrate, and other tasks I don't use.
These are hooked in like so:
tasks.compileKotlin.dependsOn(generateApiSvcJooqSchemaSource)
tasks.generateApiSvcJooqSchemaSource.dependsOn(flywayMigrate)
tasks.generateApiSvcJooqSchemaSource.mustRunAfter(clean)

For test db usage, I also have these to allow tests to use their own database and Flyway tasks:
task(makeTestDb) {
  doLast {
    // do some plain sql stuff to make a test user and database, if they're not present
  }
}

task(migrateTestDb, type: FlywayMigrateTask, dependsOn: makeTestDb) {
  url = testJdbcUrl
  user = testDbUser
  password = testDbPass
}

task(cleanTestDb, type: FlywayCleanTask, dependsOn: makeTestDb) {
  url = testJdbcUrl
  user = testDbUser
  password = testDbPass
}

and some more wiring
tasks.migrateTestDb.mustRunAfter(cleanTestDb)
tasks.flywayMigrate.mustRunAfter(flywayClean)

tasks.test.dependsOn(migrateTestDb)
tasks.clean.dependsOn(cleanTestDb)

In the above setup, the erroneous behavior is observed.
However, if this line tasks.generateApiSvcJooqSchemaSource.mustRunAfter(clean) is removed, ./gradlew clean check works correctly on 5.0 rc3 (didn't test other rcs).
I don't recall exactly what usage pattern needed that task ordering, but it doesn't occur in my day to day usage, so I'm ok with removing that line. However, it seems like it's probably undesirable that the behavior changed in 5.0... I don't see why that task ordering should cause resources to not show up for tests, at the very least.
Your Environment
openjdk 11.0.1
x64 Linux 4.14

@marshallpierce Could you create a build scan when it doesn't work?
If you can't do that, could you post the log of the build?  I'm trying to tell if clean tasks are running in the middle of the build.
Thanks!

@marshallpierce we have not been able to reproduce this issue. Do you still experience it? Could it be an issue with your build specifically? It would be really helpful if you could provide a reproducible example or at least a build scan.

Removed 5.0 RC5 as target since there is no indication that this is a regression so far.

Here's a scan of the build failing with 5.0 rc3: https://scans.gradle.com/s/3hlp7bwex33wi/
And, with the exact same project succeeding, having run ./gradlew wrapper --gradle-version=4.10.2 first: https://scans.gradle.com/s/irj2czdcgp3qs

It appears that with 5.0rc3, processTestResources and other similar tasks run before clean and therefore are not very useful, whereas with 4.10, it's the other way.
It looks like 5 is interleaving dependency tasks of check with those of clean, leading to a situation where part of the work of check is done, and then cleaned.

Tests pass if run without parallel.
The task wiring is described in the first comment in the issue. If you need more detail it shouldn't be a big deal on my end to get you NDA'd so you can access the real project.

Sorry I missed a part of that original description.
This might be related to https://github.com/gradle/gradle/issues/2488.
Although you do not have a clean task depending on something else but only something else (generateApiSvcJooqSchemaSource) depend on clean, right?
Independent of this issue, I don't see why you would need something like tasks.generateApiSvcJooqSchemaSource.mustRunAfter(clean). If removing this fixes the problem for you in 5.0, you should do that.
We still should explore why the beavior here changed with 5.0. Maybe @big-guy can make some sense of it.

Clean must also clean the test database: tasks.clean.dependsOn(cleanTestDb)
tasks.generateApiSvcJooqSchemaSource.mustRunAfter(clean) was an attempt to solve an issue where the generated source wasn't present by the time compilation tasks ran. I don't remember which combination of tasks I was specifying on the command line at the time I added that, though. With the mustRunAfter removed (so clean check now at least has its resources present at test runtime), there are still task ordering issues that mean I need to run more than one ./gradlew invocation to successfully clean everything, have jooq generate sources again, and compile / test everything successfully. That may well be simply more cases of #2488 causing problems, and that's what the mustRunAfter was trying to address.

Closing this as a dupe of https://github.com/gradle/gradle/issues/2488


Add a threadpool to Qualification tool to process logs in parallel

Add a threadpool and parameters to configure timeout and number of threads (defaults to cores on host/4).
Updated some log messages so it shows path or something useful since now the log messages don't come out in sequential order.
Fixed a bug with printing out the text file when there aren't any apps in QualOutputWriter.
Times to process 105 log files went from about 6.5 minutes to 45 seconds, interestingly enough, processing 210 log files only takes a bit longer at like 50 seconds, it looks like a few of the log files are skewed and take much longer than others.

build


Save to store after X seconds of no further changes.

Clear and concise description of the problem
Hi, and Thanx for this great code!
I use pinia for software with many mutations, right now I would want 2 things:

Store to dynamic path to store to listitem-1 and a seperate storage path for listitem-2

Only store to storage after X seconds of inactivity of the path (to avoid many storage moments)

Suggested solution
A configuration property to allow for deferred storage (only store the last known value)
Alternative
No response
Additional context
No response
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guide.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

Hi, I think this is out of scape of the package. It is meant to be basic and sync a storage with the store, adding a debounce would add a lot of complexity and unstability to the code atm.
Maybe the debounce belongs before the data hits the store. You can also look into plugins like https://github.com/posva/pinia-plugin-debounce


Add final row that is the sum of what is currently displayed by the filter

Would be really neat to have a "sum of all jobs currently shown" somewhere in the UI.  Or I could see how much of the cluster I'm using by searching for my username. etc... Thoughts @parente ?

Datatables has support for this

Agreed. I had the same thought today. We were probably debugging the same problem.


Function Docstrings Obscured By Decorators That Return Protocols

Current Behavior
When implementing decorators it's often convenient to use a Protocol and ParamSpec to annotate the type they return. However, using a Protocol causes Pylance to hide the docstring of the decorated function even though, in most cases where a Protocol is generalized by a ParamSpec, it would make sense to display the docstring of the function the ParamSpec came from.
Consider the example below:
from typing import Callable, Concatenate, ParamSpec, Protocol, TypeVar

P = ParamSpec('P')
R = TypeVar('R')

class Options:
    """Options for a function"""

class RequiresOptions(Protocol[P, R]):
    """A protocol for functions that require options to be passed in."""

    def __call__(self, options: int, *args: P.args, **kwargs: P.kwargs) -> R: ...

# annotated with a protocol
def use_options(func: Callable[P, R]) -> RequiresOptions[P, R]: ...

@use_options
def foo(x: int) -> int:
    """Some docstring"""
    ...

foo(1)  # docstring hidden

Here, Pylance just shows the type of foo. One could argue that this is sensible given that the decorator changed the type of foo. That might imply that the foo's docstring wouldn't necessarily be meaningful anymore. However, if that's true, Pylance isn't consistent in its behavior since, if we change this decorator to use Concatenate instead of a Protocol, the docstring is preserved:
# annotated with concatonate
def use_options(func: Callable[P, R]) -> Callable[Concatenate[Options, P], R]: ...

@use_options
def bar(x: int) -> int:
    """Some docstring"""

bar(1)  # docstring visible

Expected Behavior
When Pylance encounters decorators of the form:
P = ParamSpec('P')

class SomeProtocol(Protocol[P]):
    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> Any: ...

def some_decorator(func: Callable[P, Any]) -> SomeProtocol[P]: ...

Pylance should preserve the docstrings of the functions it decorates. Further, the details of __call__ should not matter beyond the fact that it uses the ParamSpec. So for example, any of the following should still preserve docstrings:
class SomeProtocol(Protocol[P]):
    def __call__(self, added_arg: int, *args: P.args, **kwargs: P.kwargs) -> Any: ...

class SomeProtocol(Protocol[P]):
    @overload
    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> Any: ...
    @overload
    def __call__(self, added_arg: int, *args: P.args, **kwargs: P.kwargs) -> Any: ...
    def __call__(self, *args: Any, **kwargs: Any) -> Any: ...

Open Questions
It's a little unclear whether any Protocol instances that are generalized by a function's ParamSpec should adopt the docstring of that function. Perhaps only a Protocol that implements a __call__ that uses the ParamSpec should have this behavior.
For example, it might not make sense to copy the docstring in a case like:
class HasSomeMethod(Protocol[P]):
    def some_method(self, *args: P.args, **kwargs: P.kwargs) -> Any: ...

def has_some_method(func: Callable[P, Any]) -> HasSomeMethod[P]: ...

It's not particularly clear what the best thing to do in this situation is.

I don't think there's a "correct" answer here. I can point to cases where either choice is correct. The current behavior is at least internally consistent. If a callable object has a __call__ method with a docstring, that docstring takes precedence over any docstring that is captured by a ParamSpec used within that __call__ method's signature.
You can omit the docstring for the __call__ method if you don't want it to take precedence. Pyright/pylance will then use the docstring captured by the ParamSpec.

I see what you mean. The docstring does appear in the signature help (when typing the arguments for the foo call), but it doesn't appear when hovering over foo. That's a current limitation of the hover provider — the language server module that provides the hover text. I agree that it would make sense for the hover provider to include the "adopted" decorator in this case in the same way that the signature help provider does.
Here's the signature help for foo. You can see that the "adopted" docstring does appear here.

Great. Thanks for looking into it. Look forward to having this fixed.

when we rework how we do doc string this sprint, we should make sure all features that use doc string use same code to make sure we have consistent behavior regardless where it is shown to users.
currently, we have slightly different code depends on each features.


Model zoo style

Fix model zoo related style issues.

@kevinthesun i wrote a script that first converts a markdown table into the rst format when make html. it modifies the .md files directly. so please don't commit the new generated .md files which contain '```rst' after running make html.

lgtm


NodeJs Emitter - Local Testing Exploration

Run NodeJs Emitter scopes against actual repos to ensure data is gathered correctly and discover potential errors or situations  that we aren't accounting for in the code.

We've done this against a few real large-scale repositories, and we're doing this by way of our __fixtures/projects directory during e2e unit tests.


Error: Status 403 trying to pull repository sath89/oracle-xe-11g: "Account is not Active"

Please make sure that you activate your Hub account prior to pulling any repository.
Just visit http://hub.docker.com


Texture vertical flipping

This adds a toggle on texture tabs to correctly display textures that look vertically mirrored.
By default new tabs are opened with the flipping turned off, but we can change this depending on usability vs data accuracy.

Nice, thanks!


Deprecation banner for Diff and other tools

Add deprecation warnings to the different tools we are going to remove from the portal:

Specs diff & change log
Preview docs for specs
Publish pull request
Update existing OpenAPI specs

On the warning message, include links to the docs, alternative tools, and contact info.

PR: https://devdiv.visualstudio.com/DevDiv/_git/openapi-portal/pullrequest/536675


v1.0.0

Changes and bugfixes for cross-team cohort analysis.

Looks great!  I have not inspected all of the changes, but I assume that the merging code has successfully executed the workflow for the MVP release.


the client connect(nursery) API is odd

Here too we should use the trio start() protocol.  E.g.:
await connection = nursery.start(client.connect)
...

I suspect there no users yet so this isn't a concern, but it's possible to make this API change in a backwards compatible way:
async def connect(self, nursery=None, *, task_status=trio.TASK_STATUS_IGNORED):
    ...

I found that accepting a nursery as an argument also limits the implementation unfairly.  Specifically, it precludes connect() implementation placing a try-catch around the _reader_task().  This is something I actually need to do for #17.

I don't like this API either, but I'm new to Trio and not really sure what else to do.
If the client starts its own nursery, then AFAIK the caller needs to provide a handler coroutine that we will launch as a new task in our own nursery. In this hypothetical, the client code would need to look like this:
async def client():
    c = WebSocketClient('foo.com', 80, 'resource', use_ssl=True)
    # Rename connect() to run(), b/c it doesn't resume until after the
    # connection closes:
    await c.run(handler)

async def handler(conn):
    # Do some stuff with connection. When I finish #3, this will be able
    # to use an async context manager.
    await conn.close()

Is this something we like? Is it "Trionic" (the Trio version of Pythonic)? cc: @njsmith

John pointed out that Fuyukai/asyncwebsockets doesn't take a nursery as an argument. Indeed, it doesn't use a nursery at all. It doesn't need a background task because it doesn't read network data until you call next_event():
    async def next_event(self):
        """
        Gets the next event.
        """
        with trio.open_cancel_scope() as scope:
            self._cancel_scopes.add(scope)
            try:
                while True:
                    for event in self._connection.events():
                        if isinstance(event, events.DataReceived):
                            # check if we need to buffer
                            if event.message_finished:
                                return self._wrap_data(self._gather_buffers(event))
                            else:
                                self._buffer(event)
                                break  # exit for loop
                        else:
                            return event

                    data = await self._sock.receive_some(4096)
                    self._connection.receive_bytes(data)
            finally:
                self._cancel_scopes.remove(scope)

This next_event() API feels too low level. It can return any event type: text frame, binary, ping, pong, etc., and the caller has to figure out if and how to handle each event. Compare to the trio-websocket API, where a call to get_message() is guaranteed to return str or bytes.
I also think that this will break if two different tasks call next_event() at the same time. If there are no pending events, both tasks will call self._sock.receive_some() at the same time, which will throw an exception in the second task.

I agree that trio-websocket has a higher level API and we want it to handle ping and hide wsproto details.  So the issue is that if the user is only blocking on next_event() sometimes, then we'd miss a ping.
Regarding client usage by simultaneous tasks, I don't think it should be supported.  It's straightforward to wrap a single-task websocket client to support multiplexing, and I've done exactly that with asyncwebsockets.   I'm not sure what the semantics of allowing multiple listeners would be.  In my case responses need to get paired with requests based on a proprietary scheme, so custom multiplexing is needed regardless.
Regarding passing in nurseries, I've got over a month of using Trio full time now and am forming an opinion that it's an anti-pattern.  The usual pattern when you have an object which needs to run something in the background is to give full control to the user about what scope that is run in, most importantly of which is the ability to wrap the task in a try-catch.  This error handling part, along with scoped cancellation, are Trio's main selling points, and passing in a nursery foils them.
async def do_session():
    try:
        async with open_websocket(host, port, handler) as ws:
            # At this point we know that there was successful ws handshake with peer.
            # We can check ws.port if it was automatically assigned.
            # ping requests will be handled automatically.
            # We can call ws.send() and ws.receive().  Neither support
            # simultaneous task access, so use a lock for send() and multiplexing
            # on receive() as necessary.
            # If we're defining an asynccontextmanager, we would
            # yield ws here.
        # this point is never reached since open_websocket always raises
    except ClosedConnection:
        # pass or something based on close reason
    except SomeSevereHandlerError:
        # ...

# do_session would normally be run in parallel with other tasks via nursery.
#
# Another use case is to wrap the run in "with trio.move_on_after()", e.g. if
# session can't complete in 30 seconds, give up.

My suggestion would be to have a "low level" API where you pass in a nursery, plus a "high level" API like async with open_websocket(url): ...
The high level API makes things convenient, since most users just want to open a websocket connection and don't care about the details; the low level API makes more exotic cases possible, like when you want to have an unpredictable number of connections with unpredictable lifetimes, and there's no clear mapping between connections and tasks. There's some more discussion of this (even using websocket as the example :-)) here: https://stackoverflow.com/questions/48282841/in-trio-how-can-i-have-a-background-task-that-lives-as-long-as-my-object-does


Dependabot configuration to update actions in workflow

Noticed the actions used in https://github.com/RiotGames/key-conjurer/blob/main/.github/workflows/ci.yml are outdated, proposing a Dependabot configuration to update - reference https://docs.github.com/en/actions/security-guides/using-githubs-security-features-to-secure-your-use-of-github-actions#keeping-the-actions-in-your-workflows-secure-and-up-to-date
Resolves warning on executions ex. https://github.com/RiotGames/key-conjurer/actions/runs/11964511082

The following actions use a deprecated Node.js version and will be forced to run on node20: actions/checkout@v3 ...

Suggest enabling https://docs.github.com/en/code-security/dependabot/working-with-dependabot/about-dependabot-on-github-actions-runners#enabling-or-disabling-for-your-repository as well

@punmechanic Spare a moment to review?

Hi @ScottBrenner, sorry for the late reply, I am actually on PTO until the new year. This looks good to me. Thanks for the contribution.

Thanks, one more in https://github.com/RiotGames/key-conjurer/pull/120 !


Fix comment in PerformanceTestDirectoryProvider

The comment was lost in https://github.com/gradle/gradle/commit/1af03ab6925aca7ed3d05831c6d0223408a83b67.
Kudos to @lacasseio for finding this.

@bot-gradle test and merge

OK, I've already triggered a build for you.


DOC: add combine_by_coords in merge see also

Keewis mentioned other options to merge are concat and combine_by_coords (https://github.com/pydata/xarray/discussions/5462#discussioncomment-861492). This PR adds combine_by_coords in the see also docstring of merge

Hello @raybellwaves! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file xarray/core/merge.py:

Line 883:1: W191 indentation contains tabs
Line 883:1: E101 indentation contains mixed spaces and tabs

Thanks @raybellwaves


Sent message through OH once graphs are finished?

This is dependent on the Open Humans API allowing for such things without having to update the master token once a day. If the latter is the case this feature is not possible right now. @madprime just a reminder for looking into this.


Update package.json

updated to latest sass

@mohitsud Build is failing  ¯_(ツ)_/¯

Closing this. As of 4.0.0 we don't use ember-cli-sass.


Run-time error

Subject of the issue
I am getting the following error:
angular.min.js:127 TypeError: angular.lowercase is not a function
at L (angular-translate.min.js:6)
at Function.R.use (angular-translate.min.js:6)
After the this line of the code:
$translate.use(lang);
Your environment

2.18.1

1.7.8

which browser and its version
Chrome
Am I using  correct version of angular-translate with AngularJS 1.7.8?

Are you sure, that you're using version 2.18.1 from angular-translate?

/*!

angular-translate - v2.18.1 - 2018-05-19

Copyright (c) 2018 The angular-translate team, Pascal Precht; Licensed MIT
*/

This code helped with the problem:
angular.uppercase = function (text) {
return text.toUpperCase();
}
angular.lowercase = function (text) {
return text.toLowerCase();
}

I'm asking, because the calls to angular.lowercase have already been removed with 2.17.0 and within the release-files I download from/for 2.18.1, there is no call to angular.lowercase in the min.js - files.

Well, I downloaded it today. Anything else you want me to check?
On Tue, Sep 17, 2019 at 3:12 PM Thorsten Späth<EMAIL_ADDRESS>wrote:

I'm asking, because the calls to angular.lowercase have already been
removed with 2.17.0 and within the release-files I download from/for
2.18.1, there is no call to angular.lowercase in the min.js - files.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/angular-translate/angular-translate/issues/1889?email_source=notifications&email_token=ABYF22HLSNO22RLMTLJ2IKTQKE2ZPA5CNFSM4IXTCEK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD65YGMA#issuecomment-532382512,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABYF22DIJQSD5PTCXCIGWETQKE2ZPANCNFSM4IXTCEKQ
.

Well, as described in the issue template, best would be if you setup a complete example so others can check what goes wrong.

Hello,
I'm closing this issue due inactivity. We have asked for additional details, but you could not helped us. Therefore we close this issue as not reproducible.
If you still have an issue, please create a new issue because we cannot be sure to track closed issues.


File::readdir fails for filename longer than 128 bytes

If a directory contains a file with a name longer than 128 bytes, that filename is not returned when iterating over the directory contents using readdir().
Example program:
use std::{net::TcpStream, path::Path};
use ssh2::Session;

fn main() {

    let tcp = TcpStream::connect("<IP_ADDRESS>:22").unwrap();
    let mut sess = Session::new().unwrap();
    sess.set_tcp_stream(tcp);
    sess.handshake().unwrap();
    sess.userauth_password("xxxx","xxxx").unwrap();
    let sftp = sess.sftp().unwrap();

    let mut dir = sftp.opendir(Path::new("/home/someuser/tst")).unwrap();
    loop {
        match dir.readdir() {
            Ok((dir,..)) => println!("Found: {}", dir.display()),
            Err(e) => {
                println!("Err: {}", e);
                break;
            },
        }
    }
}

In the test directory, create a file with name 128 chars long:
mkdir tst
touch tst/12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678

Output:
Found: .
Found: ..
Err: [Session(-16)] no more files

The problem appears to be the way in which the buffer is managed:
let mut buf = Vec::<u8>::with_capacity(128);
...
if rc == raw::LIBSSH2_ERROR_BUFFER_TOO_SMALL {
    let cap = buf.capacity();
    buf.reserve(cap);
}  ...

Ie. create a Vec with capacity 128, then pass it in to libssh2 readdir_ex. If we get back ERROR_BUFFER_TOO_SMALL, then double the buffer's capacity.
The problem is that Vec::reserve reserves enough capacity in the vec for at least additional elements, and does nothing if capacity is already sufficient. Since the actual length of the vec is still 0, buf.reserve(cap) does nothing because the Vec already has enough space for that.
A simple solution would be:
buf.reserve(cap * 2);

Unfortunately, in the process of testing this, we found a bug in libssh2, where if it returns LIBSSH2_ERROR_BUFFER_TOO_SMALL, then it returns one less that the actual number of directories. See this libssh2 bug report.
Until this is fixed, perhaps we should be allocating a bigger buffer from the start.

Any feedback on this? It is blocking us at the moment.
I have not had much movement on the associated libssh2 bug report but I was thinking that maybe in the interim we could improve the situation by

Changing the way the buffer is being resized, as described above, which would prepare us if libssh2 fixes their bug
Allocating a bigger bufffer to begin with, which would lessen the chance of having this issue, at the cost of more memory being used for all cases. Maybe double the initial buffer to 256.

Would a PR that implemented these two suggestions be accepted?

I think this is not yet published to crates.io? If there are now new active maintainers who can push a new crate release may I ask that this be pushed?


Connection failure type

Hi H2zero,
some time ago, I had a modification in NimBLEClient which detected the rc from ble_gap_connect() and allowed me to read it in the event of connection failure.
After moving to 1.20, I removed this.
However, have just got a report of a device getting into a state where connect() is failing immediately, and never again succeeds.
this was my code outside the client, run after a connect failure:
#ifdef NIMBLE_CLIENT_HAS_GETRESULT
    int rc = pClient->getResult();

    switch (rc){
      case (0x0200+BLE_ERR_CONN_LIMIT ):
#ifdef BLE_ESP32_DEBUG
        AddLog(LOG_LEVEL_ERROR,PSTR("BLE: Hit connection limit? - restarting NimBLE"));
#endif
        BLERestartNimBLE = 1;
        BLERestartBLEReason = BLE_RESTART_BLE_REASON_CONN_LIMIT;
        break;
      case (0x0200+BLE_ERR_ACL_CONN_EXISTS):
#ifdef BLE_ESP32_DEBUG
        AddLog(LOG_LEVEL_ERROR,PSTR("BLE: Connection exists? - restarting NimBLE"));
#endif
        BLERestartNimBLE = 1;
        BLERestartBLEReason = BLE_RESTART_BLE_REASON_CONN_EXISTS;
        break;
    }
#endif

This type of failure was not uncommon in 1.00; do you think I should re-instate the code, at least temporarily, to see if we have the same issue in 1.20?
It seems that it may be good to have something in the client which would enable a user to find out why the connect failed - the two above in 1.00 were effectively un-recoverable without restarting the BLE stack....
your advice appreciated..
br,
simon
p.s. these are those pesky EQ3 thermostats which have given both me and @fightforlife such grief.

my new mod:
https://github.com/btsimonh/Tasmota/commit/31afd59175b96bd6d49869eb4df0a7e52fa70021
will let you know if we see it.

The changes made to the client in 1.2.0 should handle all of the possible error codes and print a debug message with the error message and code.

that's what I thought...  in this case we only have syslog messages, not serial.
What could result in never being able to make another connection and immediate 'cannot connect'?

There are a few possibilities depending on the peer device:

It could be in a connection with something else already
It could be in a non-connectable state (check flags)
It could be advertising with a white list and only allowing connections to certain peers
You could already be connected but that should not be possible any longer.
You or the device could be out of memory.
Some bug that we don't know of yet.
It doesn't like you anymore and doesn't want to talk to you 😄

Without log info it's pretty hard to say, I would be inclined to use a sniffer to trace this.

Some observations:

It's not device specific, it simply refuses to send commands to any device
It's still able to process messages from other devices which are broadcasting messages (pvvx sensors)
A reboot of the ESP32 clears the situation immediately

So I'm inclined to say that it's not something on the target device, but more on 'our' side

@h2zero - so the of two specific rc values I saw previously - we should see neither, as we've always asked it to disconnect previously...  That was my impression of your work on 1.20, hence why I removed the test :).
I'll add a log to display the rc for all failure to connect(), as it looks like my code will only log those two.

Ok, so if the problem is on "our side", which seems to be the case, then there are a few possibilities.
The reasons for this could be:

the scanner is active and not being stopped
the previous connection is still active somewhere in the stack
A previous attempt to connect is still ongoing (not yet completed or timed out)
not enough ram

The scanner stops automagically?
Previous connections SHOULD have been closed, or failed.
(but hence the two things I previously looked for - already connected and too many connections).
'timed out' - the failure occurred ~2:30 am, and every connection attempt after that failed immediately - up until reboot at 9:30am.
ram is probably ok, between 120-140k free heap reported.

The scanner stops automagically?

It does if the call succeeds, however it can fail to stop and will result in an immediate return from connect() - impossible to know without a log.

Previous connections SHOULD have been closed, or failed.

They should if all went well, however there could be a bug dangling a connection somewhere.

'timed out' - the failure occurred ~2:30 am, and every connection attempt after that failed immediately - up until reboot at 9:30am.
ram is probably ok, between 120-140k free heap reported.

Probably not the issue, and narrows down the list.

@ysbrand now has a version which logs the rc on every connect failure; so assuming that may tell us something, we will know when it happens again.  However, we've been running this for some months, and this is the first time we've noticed it doing this specific thing - so we may be in for a long wait! :(

I guess we'll have to be patient, maybe some early logs will give a clue though.

Closing as stale, please reopen if the issue still persists.


Add FileRecordSourceNamedWriter

Create a File Record Writer which uses Header source for the filename
and accept a parameter for output suffix.
new FileRecordSourceNamedWriter("out")
Reading File1.csv --> Write File1.out
Reading File2.csv --> Write File2.out
Another option would be to set a record limit per file
Reading File1.csv --> Write [ File1-part1.out , File1-part2.out ]

Hi @bgfeldm,
Thank you for opening this issue. I believe we don't need a new concept FileRecordSourceNamedWriter for this use case since we have control over file names when creating the record reader/writer. In your example, you can create a FlatFileRecordReader("File1.csv") and a corresponding FileRecordWriter("File1.out").
I'm closing this issue for now, but feel free to add a comment if needed.


When creating a DataTree from a Dataset with path-like variable, subgroups are expected to be created

When creating a DataTree from a Dataset with path-like variable, subgroups are expected to be created
from pathlib import Path

import numpy as np
import xarray as xr
import datatree as dt
from xarray import open_zarr

# Set to True to get rich HTML representations in an interactive Notebook session
# Set to False to get textual representations ready to be converted to markdown for issue report

INTERACTIVE = False

# Convert to markdown with
# jupyter nbconvert --to markdown notebooks/unexpected-behaviour-dt-from-ds.ipynb

Test Data Initialization
A Dataset containing a single variable, with a name containing slashes, representing a path.
xda = xr.DataArray(
    np.arange(3 * 18).reshape(3, 18),
    coords={"label": list("abc"), "z": list(range(18))},
)
xda = xda.chunk({"label": 2, "z": 4})
xds = xr.Dataset({"group/subgroup/my_variable": xda})
xds if INTERACTIVE else print(xds)

<xarray.Dataset>
Dimensions:                     (label: 3, z: 18)
Coordinates:
  * label                       (label) <U1 'a' 'b' 'c'
  * z                           (z) int64 0 1 2 3 4 5 6 ... 11 12 13 14 15 16 17
Data variables:
    group/subgroup/my_variable  (label, z) int64 dask.array<chunksize=(2, 4), meta=np.ndarray>

This flat Dataset containing path-like variable name is expected to produce groups and subgroups
once injected into a DataTree.
Unfortunately, it does not happen. Instead it produces a flat DataTree with a single variable,
with an illegal name (containing slashes).
xdt = dt.DataTree(xds)
xdt if INTERACTIVE else print(xdt)

DataTree('None', parent=None)
    Dimensions:                     (label: 3, z: 18)
    Coordinates:
      * label                       (label) <U1 'a' 'b' 'c'
      * z                           (z) int64 0 1 2 3 4 5 6 ... 11 12 13 14 15 16 17
    Data variables:
        group/subgroup/my_variable  (label, z) int64 dask.array<chunksize=(2, 4), meta=np.ndarray>

This is not only cosmetic. Indeed, trying to access this malformed variable name will result in an error:
try:
    xdt["group/subgroup/my_variable"]
except KeyError as err:
    print(err)

'Could not find node at group/subgroup/my_variable'

The expected behaviour would be the one of using __setitem__:
xdt = dt.DataTree()
for varname, xda in xds.items():
    xdt[varname] = xda
xdt if INTERACTIVE else print(xdt)

DataTree('None', parent=None)
└── DataTree('group')
    └── DataTree('subgroup')
            Dimensions:      (label: 3, z: 18)
            Coordinates:
              * label        (label) <U1 'a' 'b' 'c'
              * z            (z) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
            Data variables:
                my_variable  (label, z) int64 dask.array<chunksize=(2, 4), meta=np.ndarray>

Technical Hints
__setitem__ wraps the key into a NodePath:
https://github.com/xarray-contrib/datatree/blob/0afaa6cc1d6800987d8b9c37a604dc0a8c68aeaa/datatree/datatree.py#L923
Probably this section of the DataTree initialization logic would need to be adapted:
https://github.com/xarray-contrib/datatree/blob/0afaa6cc1d6800987d8b9c37a604dc0a8c68aeaa/datatree/datatree.py#L408

Note: this issue was generated from a notebook. You can use it to reproduce locally the bug.

Thank you for spotting this!

This flat Dataset containing path-like variable name is expected to produce groups and subgroups once injected into a DataTree.

Yes, or a simpler solution would be just to raise an error if someone tries to pass a Dataset which has path-like variable names. We are then basically assuming that (a) any Dataset passed is supposed to represent a group, and (b) a single group cannot have path-like variable names.
I think there is also an argument to be made with round-tripping. I think we want this property to hold always:
ds  # start with a dataset
dt = DataTree(ds)
new_ds = dt.to_dataset()  # expect new_ds == ds

If ds contains path-like variable names, that are then automatically turned into different groups, this property won't work. If we just forbid it, we can make sure it works in all other cases.

https://github.com/xarray-contrib/datatree/issues/311#issuecomment-1944364934
Hello, thanks for your answer!
Indeed the round-trip capability seems the cleanest default behaviour. A Dataset is supposed to be a group. I think what I needed already exists: it's DataTree.from_dict .
https://github.com/xarray-contrib/datatree/blob/0afaa6cc1d6800987d8b9c37a604dc0a8c68aeaa/datatree/datatree.py#L1035
I can have a look to fix the bug (forbidding Datasets containing path-like variable names), and I can suggest an error message like this to help the user that probably wanted this "auto-creating-group" behaviour but did not knew about .from_dict:
Given Dataset contains path-like variable names.
A Dataset represents a group, and a single group cannot have path-like variable names.
Consider `DataTree.from_dict` to automatically create groups from a mapping of paths to data objects

(if you have an idea to shorten and improve it I am interested!)
Here is what I really wanted to do when creating the issue:
from pathlib import PurePosixPath

# Note: paths represent variable names ; the mapping should only contain groups, hence `.parent`
# Note 2: the named DataArray is renamed to only keep the name part of the path.
mapping_path_to_dataarray = {
    PurePosixPath(str(varname)).parent: xda.rename(PurePosixPath(str(varname)).name)
    for varname, xda in xds.items()
}
mapping_path_to_dataarray

{PurePosixPath('group/subgroup'): <xarray.DataArray 'my_variable' (label: 3, z: 18)>
 dask.array<xarray-<this-array>, shape=(3, 18), dtype=int64, chunksize=(2, 4), chunktype=numpy.ndarray>
 Coordinates:
   * label    (label) <U1 'a' 'b' 'c'
   * z        (z) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17}

Though this logic will work, it seems a little bit over-complicated to me.
However, a Dataset carrying variables names containing paths is a little bit
a misuse of a Dataset. A DataTree should be used in the first place
if possible.
xdt = dt.DataTree.from_dict(mapping_path_to_dataarray)
xdt if INTERACTIVE else print(xdt)

DataTree('None', parent=None)
└── DataTree('group')
    └── DataTree('subgroup')
            Dimensions:      (label: 3, z: 18)
            Coordinates:
              * label        (label) <U1 'a' 'b' 'c'
              * z            (z) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
            Data variables:
                my_variable  (label, z) int64 dask.array<chunksize=(2, 4), meta=np.ndarray>

Note: Pylance is not happy of usage of dict of PurePosixPaths. It only expects str as keys.
Maybe it could be worth it to make it accepts Pure Paths too?
Argument of type "dict[PurePosixPath, DataArray]" cannot be assigned to parameter "d" of type "MutableMapping[str, Dataset | DataArray | DataTree[Unknown] | None]" in function "from_dict"
  "dict[PurePosixPath, DataArray]" is incompatible with "MutableMapping[str, Dataset | DataArray | DataTree[Unknown] | None]"
    Type parameter "_KT@MutableMapping" is invariant, but "PurePosixPath" is not the same as "str"
    Type parameter "_VT@MutableMapping" is invariant, but "DataArray" is not the same as "Dataset | DataArray | DataTree[Unknown] | None"PylancereportArgumentType

Reconverting the DataTree gives an empty Dataset, as the root contains only groups
but no concrete variables:
reconverted = xdt.to_dataset()
reconverted if INTERACTIVE else print(reconverted)

<xarray.Dataset>
Dimensions:  ()
Data variables:
    *empty*

Reconverting the subgroup containing the variable is successful
reconverted = xdt["group/subgroup"].to_dataset()
reconverted if INTERACTIVE else print(reconverted)

<xarray.Dataset>
Dimensions:      (label: 3, z: 18)
Coordinates:
  * label        (label) <U1 'a' 'b' 'c'
  * z            (z) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Data variables:
    my_variable  (label, z) int64 dask.array<chunksize=(2, 4), meta=np.ndarray>

(if you have an idea to shorten and improve it I am interested!)

That looks good - I think long error messages are good if they are adding more context, like yours is. Only thing I would add is an explicit list of the offending variable names.

Note: Pylance is not happy of usage of dict of PurePosixPaths. It only expects str as keys.
Maybe it could be worth it to make it accepts Pure Paths too?

The Posix part is to forbid backslashes, only allowing forward slashes. Maybe we can forbid those at runtime whilst relaxing the type though.

After re-reading all of this, I think the most natural behaviour from a user perspective is the one of __setitem__:

Why? What do you dislike about the "one dataset per group" logic?

Okay for adding the offending variable names!
I completely agree with the "one dataset per group" logic. This is more from a usage perspective where I find these 3 lines to be the simplest to convert a Dataset containing paths to a DataTree:
xdt = dt.DataTree()
for varname, xda in xds.items():
    xdt[varname] = xda

The user does not have to think that much: the assignment will take care of creating groups and renaming the DataArrays from the source Dataset accordingly. The only thing I quite dislike is having to make the for-loop and assignments, not having a declarative comprehension-like approach to achieve what these lines do, without becoming verbose. But I must admit this is mainly about aesthetics rather than a true need, so I don't think anything more needs to be done on this topic!

I personally think this little loop is fine - it's clear, explicit and won't be buggy. If many other people appear to say that they would like a special function for ingesting this type of non-compliant dataset then we could revisit the idea.

Closing in favour of https://github.com/pydata/xarray/issues/9339


[FEA] in AutoTuner provide suggestions about file caching

Is your feature request related to a problem? Please describe.
https://github.com/NVIDIA/spark-rapids/pull/8212 has added in support for file caching. It is not a silver bullet and needs fast disks to be worth while. But in the tool we should be able to detect if data is coming from a blob store, and if the data has overlap in what is being read.  (Same files, paths, and/or tables and overlapping columns). We should even should be able to see the scan time for the reads that appear to have overlaps so we can know if there is some potential for improvement here. Sadly we don't have a good way to tell if the disks are going to be fast. Without that we cannot make a hard recommendation to turn on the cache, but we can make some kind of a recommendation along with a comment explaining about disk speeds preferable linking to some documentation explaining the potential issues.

Proposed requirements:

From data_source_information.csv in profiler, we get various metrics regarding the reads in the job -- including data size, scan time, buffer time, data location, and schema.
We can determine the ratio of data that is redundantly read by grouping the reads by data location and schema.
If the ratio of data that is distinct versus the total reads is < X%, then we should recommend that the file cache is turned on.  For X%, we can start with 50%.

Also, disclaimers regarding recommending the file cache:

Not recommended for slow disks (e.g. xlarge/2xlarge instance types in AWS).
Only available for Parquet and ORC (coming soon).


Preprocessing

Hello! I have a question, why do you rotate the image in the MRI (prostate) preprocessing? In case I don't rotate my images or "flatten" them or use 5 slices per slice, could the code work? (I am segmenting an area of the heart)
Another question, in the preprocessing a csv appears for validation, but I think there is no validation in the training, right?
Thanks so much for reading,
Greetings

I used rotation on CT images to have a better visualization of organs, and the rotation on MRI images is just because I want to have a consistency of preprocessing on different datasets. The rotation is not necessary.
Yes, there is no validation in the training.


Have Travis build the doc

Some changes can break the doc or the way it is built. Having Travis
build the doc could allow to detect these changes and to fix them before
merging them.
Travis CI relies on an old version of Ubuntu (12.04) that only provides
an old version of pandoc (1.9). This version of pandoc does not allow to
build the doc. Therefore, we must install a more recent version of
pandoc in Travis. As this install takes about 20 minutes, I set up a
cache. See Travis doc on cache:
http://docs.travis-ci.com/user/caching/#Arbitrary-directories
As a cache was set up, I extended it to pip. This is mostly useful for
the install of matplotlib that got speed up from several minutes to
about 4 seconds.
The first Travis run after this commit will be long (more than 20
minutes), but the subsequent run should be faster than what they were
previous to it. A python 3 test job is speed up from about 6 minutes to
about 4 minutes to to the faster install.
As for now, the doc is only built for python 2.7. Indeed, the doc
requires MDAnalysis that is not compatible with python 3. To avoid to
increase further the duration discrepancy between test runs for python 2
and python 3, the doc is built on a separate process. This is done using
build matrix with two different environment: test, and doc. See travis
doc for more details:
http://docs.travis-ci.com/user/customizing-the-build/.
Note that the version of pandoc we install in Travis is more recent than
the one used by Read the doc. This version discrepancy is to take into
account! If a change introduces the use of a feature not available in
the Read the docs version of pandoc, the Read the doc failure will not
be predicted by Travis.
Travis is planning a move to Ubuntu 14.04. This upgrade should allow us
to streamline the install of pandoc, as a recent enough version of
pandoc will be available through APT. It should also fix the version
discrepancy between Travis and Read the doc.

Ready to merge.


Bump version: 1.0.2 → 1.0.3

Codecov Report

Merging #68 (78b3e15) into master (8d34a60) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##            master       #68   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            3         3
  Lines          507       507
  Branches        99        99
=========================================
  Hits           507       507

Impacted Files
Coverage Δ

tomli/__init__.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8d34a60...78b3e15. Read the comment docs.


Menu: add menu-trigger prop

Please makes sure these boxes are checked before submitting your PR, thank you!

[ ] Make sure you follow Element's Contributing Guide.
[ ] Rebase before creating a PR to keep commit history clear.
[ ] Add some descriptions and refer relative issues for you PR.

#619


Profiling

What's the purpose of this pull request?
This PR adds profiling capabilities to theme store.
just change the envvar and start profiling in either dev tools and react profiler
How it works?
Tell us the role of the new feature, or component, in its context.
How to test it?
btglobal
marinbrasil
storecomponents
References
Spread the knowledge: is there any content you used to create this PR that is worth sharing?
Extra tip: adding references to related issues or mentioning people important to this PR may be good for the documentation and reviewing process

As @lbebber said, install the new version of react@experimental and react-dom@experimental solves the problem that profiling and the tree of componente doesn't work. What do you think about bump the version of it?


add lb/ip blocks for eu-west3 and us-west4 | OPS-16210

preview
main docs

Description
Add new IP blocks for GKE dc (EU, and US-west).
It is to help with changes coming in OPS-16210 and customer communication.
Merge once got representative signoff.
Motivation and Context
Some customers who do IP white/blacklist their network should include new IP blocks ( and later clean-up old IP blocks  - see comments )
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[x] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Documentation fix (typos, incorrect content, missing content, etc.)

Checklist

[x] I have read the contributing document.
[x] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

see [#coreservices-slack thread|https://saucelabs.slack.com/archives/C0L0J10V7/p1666808125549759] for some more info

@lauramthomas Can you help with this PR?


gparyani: Watch ribbon.readTimeout

gparyani requests the watch of the watch_keyword ribbon\.readTimeout. See the MS search here and the Stack Exchange search here.

ribbon\.readTimeout has been seen in 1 true positive, 0 false positives, and 0 NAAs.

This appears to be something that the OP copied into the post, which appears associated with Netflix / Hystrix. While it shouldn't hurt to watch this, I don't see a lot of benefit at this time.
If the question this is from [MS] was clear and in English, it might be a real question on SO.

Coverage remained the same at 64.618% when pulling cfd6f170a33f5fdd7ff50ee26e1d976e01c5b30e on auto-blacklist-1539681239 into e25e082bc4cb4fc60744305a3894bfe252827b29 on master.