WIP: Reduce memory usage in doc build

CircleCI can't complete building because we're running out of memory (again), so I'm investigating why some things can't be cleaned up (again). Locally on this branch running some 3D examples I see that some vtkPolyData objects stick around:

$ PATTERN="\(35_eeg_no\|50_background_free\)" xvfb-run make html_dev-pattern-memory
...
Handler <function generate_gallery_rst at 0x7f474340db80> for event 'builder-inited' threw an exception (exception: 2 BackgroundPlotter:
dict: len=14, 3 referrers: dict/list
dict: len=14, 3 referrers: dict/list
dict: len=2, 5 referrers: dict/list
dict: len=55, 4 referrers: BackgroundPlotter/dict/list
QFrame: <PyQt5.QtWidgets.QFrame object at 0x7f4700c521f0>
method: <bound method QVTKRenderWindowInteractor.paintEvent of <pyvistaqt.plotting.BackgroundPlotter object
method: <bound method QVTKRenderWindowInteractor.CursorChangedEvent of <pyvistaqt.plotting.BackgroundPlotter
method: <bound method QVTKRenderWindowInteractor.CreateTimer of <pyvistaqt.plotting.BackgroundPlotter object
method: <bound method QVTKRenderWindowInteractor.DestroyTimer of <pyvistaqt.plotting.BackgroundPlotter objec

@GuillaumeFavelier I'll keep looking into it this week, but feel free to have a look if you want and/or have ideas about what's causing this.

I  cannot say for all references but I know that Renderers, ScalarBars and RenderWindowInteractor all hold a reference to the plotter.  I'll continue to investigate.

Thanks for looking into it, feel free to push commits here @GuillaumeFavelier . No need to push with circle full until the builds succeed here without it, as I touched an example that already shows the problem.

045fbff will be problematic on local builds (not on Circle IIRC since Mayavi shouldn't be installed there?) because just doing import mayavi.mlab will cause a lingering PolyData. Also the new code was cleaner. Why revert?

Trying to digest the changes of the PR, I realised that the figures was hanging during doc building. I believe that the scrapers were not called.

will cause a lingering PolyData.

I see, this is the cause of :
Traceback (most recent call last):
  File "/home/guillaume/source/anaconda3/envs/mne-py38/lib/python3.8/site-packages/sphinx/events.py", line 110, in emit
    results.append(listener.handler(self.app, *args))
  File "/home/guillaume/source/anaconda3/envs/mne-py38/lib/python3.8/site-packages/sphinx_gallery/gen_gallery.py", line 433, in generate_gallery_rst
    generate_dir_rst(src_dir, target_dir, gallery_conf,
  File "/home/guillaume/source/anaconda3/envs/mne-py38/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 361, in generate_dir_rst
    intro, title, cost = generate_file_rst(
  File "/home/guillaume/source/anaconda3/envs/mne-py38/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 834, in generate_file_rst
    clean_modules(gallery_conf, fname)
  File "/home/guillaume/source/anaconda3/envs/mne-py38/lib/python3.8/site-packages/sphinx_gallery/scrapers.py", line 476, in clean_modules
    reset_module(gallery_conf, fname)
  File "/home/guillaume/source/mne-python/doc/conf.py", line 312, in __call__
    _assert_no_instances(vtkPolyData, 'vtkPolyData')
  File "/home/guillaume/source/mne-python/mne/utils/misc.py", line 363, in _assert_no_instances
    assert n == 0, f'{n} {when}:\n' + '\n'.join(ref)
AssertionError: 1 vtkPolyData:
tuple: ((vtkmodules.vtkCommonDataModel.vtkPolyData)0x7f905da0c340, 1)
dict: len=11, 4 referrers: PolyData/dict/list

It might be trickier than I thought

I realised that the figures were hanging during doc building. I believe that the scrapers were not called.

For this I would just locally add assert 'pyvista' in scrapers or something, it should be in there. If it's not, we should fix the new logic I tried to implement

it should be in there.

I just used logging to confirm

we should fix the new logic I tried to implement

I'll focus on the reverted patch

It works for me locally and Circle seems to be happy for now. Next step is [circle full]?

Locally can you reproduce this problem with 50_beamformer_lcmv.py? If so we should fix it first
$ PATTERN="\(30_forward\|35_eeg_no\|50_background_free\|35_dip\|40_mne\|50_beam\|60_visualize_stc\)" xvfb-run make html_dev-pattern-memory
   6.6 s : 433.1 MB    auto_tutorials/forward... [ 42%] 30_forward.py
  59.5 s : 877.4 MB    auto_tutorials/forward... [ 57%] 35_eeg_no_mri.py
  79.7 s : 969.5 MB    auto_tutorials/forward... [ 71%] 50_background_freesurfer_mne.py
generating gallery for auto_tutorials/forward... [100%] 90_compute_covariance.py
  96.7 s : 990.5 MB    auto_tutorials/inverse... [ 36%] 35_dipole_orientations.py
 119.0 s : 1.21 GB     auto_tutorials/inverse... [ 45%] 40_mne_fixed_free.py
generating gallery for auto_tutorials/inverse... [ 54%] 50_beamformer_lcmv.py
Traceback (most recent call last):
  File "/home/larsoner/python/sphinx/sphinx/events.py", line 101, in emit
    results.append(listener.handler(self.app, *args))
  File "/home/larsoner/python/sphinx-gallery/sphinx_gallery/gen_gallery.py", line 454, in generate_gallery_rst
    generate_dir_rst(src_dir, target_dir, gallery_conf,
  File "/home/larsoner/python/sphinx-gallery/sphinx_gallery/gen_rst.py", line 381, in generate_dir_rst
    intro, title, cost = generate_file_rst(
  File "/home/larsoner/python/sphinx-gallery/sphinx_gallery/gen_rst.py", line 861, in generate_file_rst
    clean_modules(gallery_conf, fname)
  File "/home/larsoner/python/sphinx-gallery/sphinx_gallery/scrapers.py", line 474, in clean_modules
    reset_module(gallery_conf, fname)
  File "/home/larsoner/python/mne-python/doc/conf.py", line 305, in __call__
    _assert_no_instances(Brain, 'Brain')  # calls gc.collect()
  File "/home/larsoner/python/mne-python/mne/utils/misc.py", line 363, in _assert_no_instances
    assert n == 0, f'{n} {when}:\n' + '\n'.join(ref)
AssertionError: 8 Brain:
method: <bound method Brain.save_image of <mne.viz._brain._brain.Brain object at 0x7fbe632d2a30>>
cell: <cell at 0x7fbee1c9c130: Brain object at 0x7fbe632d2a30>
cell: <cell at 0x7fbeacd19820: Brain object at 0x7fbe632d2a30>
dict: len=2, 4 referrers: UpdateLUT/dict/list
method: <bound method Brain.save_image of <mne.viz._brain._brain.Brain object at 0x7fbee1e72af0>>
...

The leaks were actually in 40_mne_fixed_free.py and detected afterwards by the Resetter.
There was a reference to plotter remaining in _Figure so I set it to None in _window_clean(), a method called when the plotter is closed.
I'm building locally and on Circle simultaneously to check if there is somewhere else to fix.

Looks like there is still a ~300 MB loss in mne_fixed_free (I'm pretty sure that the memory usage at the beginning of a given example is printed after its name due to how things work with sphinx-gallery's reset_modules):
generating gallery for auto_tutorials/inverse... [ 45%] 40_mne_fixed_free.py
1753.3 s : 1.39 GB
generating gallery for auto_tutorials/inverse... [ 54%] 50_beamformer_lcmv.py
1824.7 s : 1.63 GB

Not sure there is much we can do about it, though.

Awesome, thanks @GuillaumeFavelier !!!


Initial Kotlin Multiplatform Support

This adds a Kotlin Multiplatform library people asked for months.
Closes #49 and closes #54
May require more improvements, however it's current state is at least the same state as the java (+ further methods e.g. create Theme from Image on Android and JVM).

I know this repo normally doesn't accept PRs currently, however I think it's better to merge and add this now, as it's more useful to maintain this than improving the java library

IIRC Google doesn't accept pull requests to this repo.

@mon-jai read description

@DATL4G if you want them to consider your pull reques tyou need to sign the CLA as per https://github.com/material-foundation/material-color-utilities/pull/76#issuecomment-1428784962 otherwise I doubt they'll even check the submission


platform-external: simplifying steps to extract openshift-tests

Optimizing the steps of extracting openshift-tests from release image.
x-ref: https://redhat-external.slack.com/archives/C05N3PY1XPH/p1724860915034919

/assign @elmiko @rvanderp3

/lgtm
/approve


Add readiness probe to example deployment and e2e test

We have readiness endpoint in operator now. We can make use of that to set readiness probe.

I am looking for a readiness endpoint, for running etcd-operator (0.3.2). Is there something I can already place in deployment.yaml? I don't know how important a readiness probe for the operator would be, but it feels like the "right thing" to have one.


[spi_agent] Read commands response addition

Added device agent response for Read Status commands and Read JEDEC and Read SFDP commands. Assumption is these responses are each 32 bit long.
Added enumeration for given command opcodes.
Signed-off-by: Kosta Kojdic<EMAIL_ADDRESS>

Just to clarify: Read SFDP grabs data from a RAM, and it could be up to 256 bytes long. Read JEDEC could technically also be that long, since the 3 bytes that aren't continuation codes could have up to 255 of the 0x7F continuation codes in front.

Just to clarify: Read SFDP grabs data from a RAM, and it could be up to 256 bytes long. Read JEDEC could technically also be that long, since the 3 bytes that aren't continuation codes could have up to 255 of the 0x7F continuation codes in front. Last I checked, the current set of JEDEC manufacturer IDs hasn't reached higher than 15 continuation codes, though.

Thanks, if I understand well, if Read SFDP gets data from RAM, is it RAM of spi_device? In that case downstream device agent shouldn't respond to this command?

Just to clarify: Read SFDP grabs data from a RAM, and it could be up to 256 bytes long. Read JEDEC could technically also be that long, since the 3 bytes that aren't continuation codes could have up to 255 of the 0x7F continuation codes in front. Last I checked, the current set of JEDEC manufacturer IDs hasn't reached higher than 15 continuation codes, though.

Thanks, if I understand well, if Read SFDP gets data from RAM, is it RAM of spi_device? In that case downstream device agent shouldn't respond to this command?

OK, I see that SW should prepare SFDP data, so I will remove it from device agent responses.

Sorry, I missed this PR. Please ping me if I forget to review some PRs.


ci: update renovate configuration and fix mergify rebase conditions

:tada: This PR is included in version 3.0.15 :tada:
The release is available on:

3.0.15
GitHub release

Your semantic-release bot :package::rocket:


HTTP/2 DelegatingDecompressorFrameListener return bytes to flow control

Motivation:
If a single DATA frame ends up being decompressed into multiple frames by DelegatingDecompressorFrameListener the flow control accounting is delayed until all frames have been decompressed. However it is possible the user may want to return bytes to the flow controller which were not included in the onDataRead return value. In this case the amount of processed bytes has not been incremented and will lead to negative value for processed bytes.
Modifications:

Http2Decompressor.incrementProcessedBytes should be called each time onDataRead is called to ensure all bytes are accounted for at the correct time

Result:
Fixes https://github.com/netty/netty/issues/5375

@nmittler - FYI
@npordash - Please verify this addresses this issue.

LGTM

Confirmed that calls to Http2LocalFlowController.consumeBytes(..) are no longer throwing exceptions. Thanks again!

thanks for confirming @npordash! This should make it into the next release.

@npordash thanks a lot for verify!

@Scottmitch took a first pass

@nmittler - Comments resolved. PTAL

4.1 (6aa5f76)


enhance

add getRules: return all the rules
add hasRule: avoid add a same rule twice

so do you still need _state in the lib/engine.js? @pivstone

so do you still need _state in the lib/engine.js? @pivstone

Nope. So I expose the internal rules.


is it supposed to disable the "search bar" ?

After using it I can't use the "windows search bar" it won't just work

I can also confirm, but I'm more inclined to say this is a bug of the new cumulative update.
TL;DR - Probably you should blame Microsoft again.

+1 It seems to have broken in mine too after the latest update.

Found the fix,
change the following key in the registry.

HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Search]
"BingSearchEnabled“=dword:00000001

source

did it work? I already reinstalled windows (fresh) to fix the problem

It did work in resolving the issue. I'd ask the developers in investigating this:
https://github.com/Sycnex/Windows10Debloater/blob/528ed9590963447617044388365313faa2d128fc/Windows10Debloater.ps1#L188-L195
Until this update, this key flipped to zero would only not pop up a bing search page.
But this update likely bound the whole search feature to this. Either intentional or an accident, and definitely not documented.

Anyone find a fix to this? I want to use the debloater but dont want cortana using 20-40% CPU all the time.

Microsoft supposedly pushed out an update to resolve this. This isn’t due to my script but a big by Microsoft.
Sent from my iPhone

On Oct 14, 2019, at 11:08 AM, aalar2<EMAIL_ADDRESS>wrote:

Anyone find a fix to this? I want to use the debloater but dont want cortana using 20-40% CPU all the time.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.


Deployment

Planning to setup CI to push to Dockerhub, and then using something like Terraform for deployment to Vultr.

Would that work with our secrets being in docker-compose.yml?

@erfanio Yup! We can either bake in the secrets or set them with CI.
I'm actually keen to give Dokku a go, because it's a lot more lightweight than Rancher or Kubernetes, and handles Postgres out of the box!
http://dokku.viewdocs.io/dokku/deployment/application-deployment/
http://dokku.viewdocs.io/dokku/configuration/environment-variables/

@hugomd That looks awesome!

@erfanio I've added a todo list to the description 🚀


Alternate action to point to staging image

As outlined in this conversation we want the development process to use two images, a staging and production/latest. This new action will pull from the the staging for testing

definitely worth having a way to use the staging image pushed to dockerhub from the db-data-library repo
but since action releases use semantic versioning, we likely want to be more explicit with our release versions (e.g. v1.1-beta) and image/action build process
since we can't change a published action version on the Github Marketplace, we wouldn't say "oh we like v1.2, time to update v1.1." nor should we say "oh we like v1.2, time to release it as v1.3"


Load https images from cdn instead of http

Previously, we were loading only http images as background-image for asc, sort and other actions, while best practice is to load HTTP/HTTPS based on requesters preference.

As previously, images were always loading over HTTP, even if your site is fully secure and HTTPS protected, some modern browsers will show that your site is not fully secure and which may lead to doubt in user's mind.

Yep, very sensible. Thank you!


Tricky #2

Tricky #2 from sync/atomic - atomic.Uint64

https://github.com/pedrobertao/golang-interview-prep/commit/46cd0a360e2802c96f4a77e503cc33ef64f9e84b


strictNullCheck Errors caused by accessing array attributes like .pop in a while loop

TypeScript Version: 2.9.0-dev.20180412

Search Terms: .pop, strictNullChecks while(), loop, array, possibly, undefined
Code
const queue = [
    () => { console.log("hello") }
];

while (queue.length) {
    queue.pop()();
}

Expected behavior:
That because the while loop is checking the length of an iterable is first > 0, that array methods are callable and valid with out errors.
Actual behavior:
Error message:  Cannot invoke an object which is possibly 'undefined'.
Playground Link:
https://www.typescriptlang.org/play/#src=const queue %3D [
    () %3D> { console.log("hello") }
]%3B

while (queue.length) {
    queue.pop()()%3B
}
Relevant Information
This is a pretty common practice in webpack to use a queue to perform breadth first search traversals of our dependency graph in the fastest means possible. So it would be nice to not have to @ts-ignore this line

Seems like a good use-case for a definite assignment assertion.
const queue = [
    () => { console.log("hello") }
];

while (queue.length) {
    queue.pop()!(); // <--- !
}

Sean didn't mention it, but this was in a JavaScript file, and we don't have non-null assertion syntax there.

Just opened up #23405.

Ah sorry. Then my suggestion is redundant.

Seems like there are two ways to go here, either change the definition of pop to be "optimistic" and not return undefined, that will be inline with array indexing not containing undefined, or go with the more explicit option in #23405. The first approach seems rather targeted, and i am sure this will come up again. so #23405 seems like a better investment here.
The underlying issue is discussed in https://github.com/Microsoft/TypeScript/issues/9998.
Closing in favor of #23405.


upload_file returns 0 byte file with no mime_type (high severity)

Description of the bug:
genai.upload_file(path=path, mime_type=mime_type)
Any code that was once working downstream of this, including generate_content now fails
Looks like this was from a production push? Started today 11/11/24 around noon PDT.
Also #627 likely related
Actual vs expected behavior:
Expected: a file with proper size_bytes and mime_type
Actual: size_bytes always zero and mime_type always ''
Any other information you'd like to share?
Looks like this was from a production push? Started today 11/11/24 around noon PDT.
Also #627 likely related

We are seeing this as well (also Hi Mike!)

Same issue here, even manually setting the mime_type doesn't do anything

Even the example notebooks on Gemini docs are broken
e.g. https://ai.google.dev/gemini-api/docs/vision?lang=python

same here.

Following up for visibility. I reached out on Twitter and got a response, so they should be aware.
https://x.com/OfficialLoganK/status/1856130496058495019

+1 seeing this as well

is this only happening with the python SDK? can we call the API directly as a temp fix here?

is this only happening with the python SDK? can we call the API directly as a temp fix here?

failing with the node sdk as well @adboio

related post on google cloud forums: https://www.googlecloudcommunity.com/gc/AI-ML/Gemini-API-method-models-generateContent-returns-error-code-400/m-p/831749

So happy to not just be the only one. Who changed the server side of file_upload?

found a potential workaround! :rocket:
instead of calling upload_file, use the Blob type and pass file contents directly to the generate_content:
model = genai.GenerativeModel("gemini-1.5-pro")

with open(media_filepath, 'rb') as fp:
    file_contents = fp.read()
    media_blob = protos.Blob(data=file_contents, mime_type=mime_type)

    response = model.generate_content([media_blob, prompt_text],
        generation_config=genai.GenerationConfig(
            response_mime_type="application/json", response_schema=Recipe
        )
    )

I'm having the same issue

my use-case is for video files, and the workaround i posted above does not work if the files are too large... i tried splitting the file into multiple parts and uploading the blobs individually (like generate_content([ blob1, blob2, blob3, "prompt" ])), but it seems the payload limit is on the entire payload, not individual parts... so i'm not sure of any workaround for larger files at this point

My work around is to wait. Is this the first major outage of this beautiful product?

working for me now too!

It’s finally working again!
But, Why?

Hey! We rolled out a fix for this issue, apologies for the delay. I am going to also get a status page setup to make it easier to follow this stuff in the future.


Ui/TRA-4607 - replay mizu requests

why it was removed?

wasnt used.
used in Enterprise

why do you need to open the request and response accordions onModalClose?

i tought its good idea to reset the state bedore the next opening (which have the same state)


Being able to modify names of Model when using prisma db pull

Problem
I've got an existing database (Generated using Laravel conventions) where table names follow snake_case pluralized pattern. (eg: 'voice_search_sessions', 'weather_city_translations', etc.).
And when I use prisma db pull then Prisma writes schema file with model name as it is with the table name. but when using snake_case plurlaized name in TypeScript it doesn't look good, such as writing:
let session: voice_search_sessions = prisma.voice_search_sessions.findFirst();
// OR
let sessions: voice_search_sessions[] = prisma.voice_search_sessions.findMany();

What I want is to write something which looks good and meaningful, such as.
let session: VoiceSearchSession = prisma.VoiceSearchSession.findFirst();
// OR an Array of VoiceSearchSession which is meaningful.
let sessions: VoiceSearchSession[] = prisma.VoiceSearchSession.findMany();

Suggested Solution
We can add a flag when running prisma db pull which will use the PascalCase Singular pattern with @@map for model naming (eg: --use-pascal-case-singular-model-naming).
Or we can give user flexibility of name resolving class which will extend user and implements two methods, one for database table name to model name and another for model name to database table name.
Thanks.

Similar requests are collected under the term "introspection configuration": https://github.com/prisma/prisma/labels/topic%3A introspection-configuration


prevent form-error from shifting form-group [fix #138]

Task
see issue #138
Screenshots

CSS SubGrid would provide a much more elegant solution..
I can imagine a .form-row taking over for .form-group with it's own grid area.  So, given .form-row > .form-group*n > .form-label+.form-control+.form-error each label/control/error would occupy a grid-row with their respective siblings in neighboring .form-groups.  Alas

I pushed some commits to simplify the fix. It had to do with how it was handling the layout of the inner content of the grid. I use min-content for the rows instead of absolutely positioning and translating the error message.
This also revealed an existing bug with Checkboxes and Radios where they were not laying out correctly.


Tailwind CSS v4 Beta

📋 Explain your issue
Hello,
Is there any current docs on the new v4 beta in tailwind?
There are breaking changes that the @astrojs/tailwind package does not seem to support

Hi! We have not updated @astrojs/tailwind for the beta because we would normally wait for a stable release to update our own integration. So there is not yet any documentation to update because our integration has not updated.
However, Astro Tips has written some helpful advice on this topic, so we suggest reading that! 🙌


Add database and save details for every installation

Tasks:

listen to installation events to get the data
Add mongoDb to the app and make the required models for saving the installation details

Models:
Installation:

installationId
account{ id: number, login: string }
permissions: {}
events: []

The bot will follow these steps to comment on a PR:

It will listen for every PR open event. This event will be the starting trigger.
Now, to comment on files with associated risk scores, the bot would need the file names that are modified in the PR.

How to get the file names:

The bot got the file names, now It will fetch the file names in the DB to get the associated risk scores.
The bot will construct a good-looking comment and finally will comment on the PR.

Feature: risk scores would be more refined: The bot will follow these steps after every new installation

It needs to prepare all the files with associated risk scores beforehand to react to PR events.
To do so, it needs to fetch all repositories on which the bot has been installed.
Then, iterate on each repository and get all files.
For each file, first calculate the risk score and then save it.
Now, the bot can give the files with their respective scores in every PR.

how to uniquely identify an installation on receiving a PR open event?

on every PR event, we get an installation object.

How to give each file risk score before the bot triggers to comment:

First, the bot will fetch the issues labeled as bugs or fixes for a repository.
The response payload would have the pull requests associated with the issue, so the bot will get all the PRs related to the issue by getting the PR URL.
From the PR URL the bot will get the the pull number. Using this pull number, the bot will get all the affected files and the files will be scored before saving them in the DB.

If the bot saves a file that already exists in the DB, it will make a new entry with the same fields but a different score(if any) to keep the historical data.

The bot will repeat the process from step 1 for all the repositories for every installation.


⚠ Jenkins Plugins Artifactory has degraded performance

In 7e218b6, Jenkins Plugins Artifactory (https://get.jenkins.io/plugins/artifactory/) experienced degraded performance:

HTTP code: 200
Response time: 1301 ms

Resolved: Jenkins Plugins Artifactory performance has improved in 62607c8 after 47 minutes.


crash and disappear after open

windows 10 x64
.net framework 4.8
microsoft visual c++ 2015-2019 x64 v14.28.29325
simple dnscrypt 0.7.1 x64 dnscrypt-proxy 2.0.42
in resolvers tab:
no resolvers have been loaded yet
in log folder:
2021-01-24 00:56:59.3402 ProcessHelper System.Exception: Timed out
at SimpleDnsCrypt.Helper.ProcessHelper.ExecuteWithArguments(String filename, String arguments)
2021-01-24 00:57:08.4662 ProcessHelper System.Exception: Timed out
at SimpleDnsCrypt.Helper.ProcessHelper.ExecuteWithArguments(String filename, String arguments)

after 1 minutes app crash and close automatically

Same trouble with Win 10 LTSC, not open configuration window after install, i try both x64-x32 versions..


Adds Notification molecule

Additions

Adds Notification molecule

Testing

Visit /browse-filterable/

Review

@KimberlyMunoz
@jimmynotjim
@sebworks

Screenshots

Todos

https://github.com/cfpb/cfgov-refresh/pull/1185 should be merged in first.

LGTM

It would be suuuuuper cool if these were in Capital Framework. hinthintnudgenudge

It would be suuuuuper cool if these were in Capital Framework. hinthintnudgenudge

@ascott1 Sorry to lazyweb this, but if these were in Capital Framework, which CF repo should they be added to?

There's already a cf-notifications repo, I just never finished the initial work due to time :(

:+1:

:+1:


Is there's a way to custom error message for each field?

Package version eg. v8, v9:
Issue, Question or Enhancement:
Code sample, to showcase or reproduce:
type User struct {
        Name     string `validate:"required, errMessage"`
	Password string
}

I can get the errMessage  in error when validate failed.
PS: I don't need i18n support, just user friendly error message.

@coolerfall Currently there is no way to pull the error message from the tags, and not sure if/when I might add that.
The error message itself has all the information surrounding the error and can be used to create your error message if not using the translations logic.
i18n support is provided, optionally, through registering translations. Currently, translations are using the following packages to provide i18n/l10n:

https://github.com/go-playground/universal-translator
https://github.com/go-playground/locales

Thanks for sharing this.
But I need to go through all the FieldErrors to convert to my custom message which seems not friendly/simple to use. It will be very nice if add this feature.

Just adding to @coolerfall's suggestion.
Extracting error messages into a yaml file (or any other format file) will make like easier for users to customize their error messages.
One example is from the rails world: https://github.com/rails/rails/blob/master/activemodel/lib/active_model/locale/en.yml

Another option is to use govalidator's solution to the problem.
type User struct {
        Email     string `validate:"required~email is required, email"`
	Password string
}

I find this solution really clean for the user. If we want more fine grain control we can use FieldErrors

From now, is there any plans to implement this feature？

@threeaccents what if you need english, russian and french languages?

@threeaccents solution would be perfect for me. In my case I want the validation errors to provide an error translation key. For example:
type User struct {
        Email     string `validate:"required~error-validation-email-invalid,email"`
	Password string
}

Then our frontend team has a list of translation tables with mapping like:
    "error-validation-email-invalid": "Please check your email is set correctly",

Our reason for doing it this way is we'd like all translations to be held in one place, rather than a mismatch between some translations happening on the backend and others on the frontend.


experiment_id inconsistencies with GMD and ES-DOC

Here's a correspondence from the UKMO:
There are still a few small differences between es-doc and the CMIP6 Controlled Vocabulary (CV):
CV: version 1.00.06 published 10th April
Tier 1:
hist-1950HC, hist-piNTCF, histSST, histSST-1950HC, histSST-piCH4, histSST-piNTCF, piClim-CH4, piClim-HC, piClim-NTCF, piClim-aer, piClim-control, ssp370, ssp370-lowNTCF, ssp370SST, ssp370SST-lowCH4, ssp370SST-lowNTCF, ssp370SST-ssp126Lu
Tier 2:
hist-piAer, histSST-piAer, histSST-piN2O, histSST-piO3, piClim-2xdust, piClim-2xss, piClim-BC, piClim-N2O, piClim-O3, ssp370SST-lowAer, ssp370SST-lowBC, ssp370SST-lowO3
Tier 3:
piClim-2xDMS, piClim-2xNOx, piClim-2xVOC, piClim-2xfire, piClim-NH3, piClim-NOx, piClim-OC, piClim-SO2, piClim-VOC
ES-Doc: version <IP_ADDRESS>
Tier 1:
piClim-control | hist-piNTCF | hist-1950HC | histSST | histSST-piNTCF | histSST-1950HC | histSST-piCH4 | ssp370-lowNTCF | ssp370SST | ssp370SST-lowNTCF | ssp370SST-lowAer | ssp370SST-lowBC | ssp370SST-lowO3 | ssp370SST-lowCH4 | ssp370SST-ssp126Lu | piClim-NTCF
Tier 2:
hist-piAer | histSST-piAer | histSST-piO3 | histSST-piN2O | piClim-aer | piClim-BC | piClim-O3 | piClim-CH4 | piClim-N2O | piClim-HC | piClim-2xdust | piClim-2xss
Tier 3:
piClim-NOX | piClim-VOC | piClim-2xDMS | piClim-2xfire | piClim-2xNOX | piClim-2xVOC
Differences:
piClim-CH4, piClim-HC, piClim-aer:
Tier 1 in CV
Tier 2 in ES-Doc
ssp370:
Tier1 in CV
“other” in ES-DOC
ssp370SST-lowAer, ssp370SST-lowBC, ssp370SST-lowO3
Tier 2 in CV
Tier 1 in ES-Doc
Tier 3:
           “piClim-NOX” in ES-Doc, “piClim-NOx” in CV (case of final character)
           “piClim-2xNOX” in ES-DOC, “piClim-2xNOx” in CV (case of final character)
            “piClim-NH3”, “piClim-OC”, “piClim-SO2”,  in CV missing from ES-DOC

The data request at version 1.00.07 still has no AerChemMIP experiments listed.

@momipsl @eguil @markelkington @charliepascoe @bnlawrence @matthew-mizielinski @martinjuckes pinging you all here.
I note the CV version 1.00.06 noted above doesn't relate to this (CMIP6_CVs) repo directly. Rather the version information is contained in the version_metadata entry, with the latest being:
"version_metadata":{
   "author":"Paul J. Durack<EMAIL_ADDRESS>   "creation_date":"Wed Apr 26 09:19:24 2017 -0700",
   "institution_id":"PCMDI",
   "latest_tag_point":"3.2.3 (29; g9be7426)",
   "note":"Revise source_id BESM-2-7",
   "previous_commit":"66ea1f100a70977e3400c65e14f0c91fc6590ac6"
}

To answer the question about experiment_id versioning, the last substantive change was a revision of the piClim-aer experiment (https://github.com/WCRP-CMIP/CMIP6_CVs/commit/bf062d96944c8bbbad2eec9e31c9addf07ed4450) on 29th March this year. Any subsequent changes have been to the versioning history (which is updated each time a change is made to any contents of the repo) or changing the internal format of contents (not the contents themselves) within the json file for consistency

@momipsl @eguil @markelkington @charliepascoe @bnlawrence @matthew-mizielinski @martinjuckes
I have consulted the GMD papers on MIP experiments and found that the CMIP6 experiment_id CV is currently consistent (and ES-DOC) is inconsistent in all cased listed above except:  "piClim-NOX" and "piClim-2xNOX" are listed with the "X" as upper case.
Since "NO" is a chemical compound, use of upper case makes sense, but the "x" is a lower case subscript, which I think should not be capitalized even when it doesn't appear as a subscript.  I think the Aerchem leaders made a mistake when they wrote their GMD paper, which they subsequently corrected when submitting the experiment info. to the CMIP6 CV.
I conclude that ES-DOC will need to make small changes for consistency with the CV.

Also note that piClim-aer is considered Tier 2 by Aerchem but Tier 1 by RFMIP, so the higher Tier is indicated in the CV.  (the difference in priority between the two activities is noted in the CV's "description" of piClim-aer.

I've made the following updates to the latest version of the ES-Doc experiment spreadsheet for CMIP6.  You should see these changes online when Mark next does his update.

Changed the case of the final "x" in the piClim-NOx and piClim-2xNOx experiments.
Changed the tier from 2 to 1 for piClim-CH4, piClim-HC and piClim-aer experiments.
Changed the tier from 1 to 2 for ssp370SST-lowAer, ssp370SST-lowBC and ssp370SST-lowO3.

I still need to add the piClim-NH3, piClim-OC and piClimSO2 experiments, I've upped the priority on our ticket for that task https://github.com/ES-DOC/esdoc-docs/issues/240

ssp370 is listed as a tier 1 ScenarioMIP experiment.
It appears as an "other" experiment under the AerChemMIP listing because it is governed by an "other" MIP.
If we need to retain the tier information from MIP to MIP then we will need to make an update to the way we render the lists of experiments associated with each MIP.

I believe it is not always the case when a MIP re-uses an experiment from another MIP that the experiment keeps the same tier level (but I'd need to check that).   For the time being I suggest that we re-word the "other" list of experiments in the ES-Doc interface to make it clear that these are experiments that are governed by another MIP and not experiments in an "other" tier.

From: Elkington, Mark
Date: Tuesday, May 2, 2017 at 1:18 AM
To: Taylor, Karl E., charlotte.pascoe, Eric.Guilyard, martin.juckes
Cc: Mizielinski, Matthew, Durack, Paul J.
Subject: RE: FW: AerChemMIP experiments for input into CREM

Hi Karl

Thanks for checking this – it is clearly important that the CVs, ES-DOC and DREQ are kept in synch as the modelling groups start their simulations.  I can see that at this stage it might be hard to do but it is really important.  It would be easier if we could just assume that one source is the master but  to take the MetOffice as an example this is not realistic:
-	We rely on the CVs to be correct as we will use CMOR to produce out output data
-	We rely on ES-DOC being correct with regard to experiment names as we will be using their experiment definition exports to populate the structure of our metadata database
-	We rely on DREQ to establish what variables we need to produce for each experiment.

I expect other groups will be in a similar position to us.

It would help a lot if there was a clear milestone set when the three sources would be checked for consistency and from then on consistency will be maintained across each release.

Regards

Mark

@charliepascoe @durack1 @martinjuckes @momipsl @markelkington @eguil @bnlawrence @matthew-mizielinski    You are right, Charlotte, that the tier of an experiment can depend on the MIP.  If there are 2 MIPs that include the same experiment, the priority might  (and in I think 2 cases does) differ.  Perhaps Mark E. can suggest a way to handle this more gracefully.  Currently in the CV, there is a single tier given.   As I recall for the 2 cases where there are different tiers depending on the MIP, I've recorded the highest priority, which in both cases was set by the prime sponsor of the experiment (the MIP that appears in the DRS identifier).

Karl/Charlotte
Before suggesting an approach I think I need to understand what the reaction of a modelling group should be to an experiment being defined (wrt tier) in different MIPS.  Lets assume experiment ABC is defined at tier1 for  MIP#1 and tier2 for MIP#2.  A modelling group might only have a policy of doing tier1 experiments and would do experiment ABC if it is participating in MIP#1 but would not do it if it is participating in MIP#2 but not in MIP#1.  Is this a correct interpretation?
Clearly this doesn’t account for the fact that the modelling group may make exceptions to its policy in certain cases.
I think it is OK to have just the highest tier in the CVs – we will not be using the CVs to determine what experiments we will run.  However I do think that ES-DOC and the DREQ need to allow modelling groups to understand the full MIP ‘dependency’ for a particular experiment  But as Charlotte mentioned yesterday, the current allocation of experiments to the “other” tier category on a MIP page doesn’t really make it clear what is meant.   Perhaps if that category was relabelled something like “Other MIP dependencies” – and each experiment in that list indicated the tier for “this” MIP.  So taking the case of ssp370 as an example – it is shown by its “owning” MIP (scenario) as Tier 1.  In AerChemMIP it is shown in the “other” category.  If it was instead shown as:
            Other MIP dependency:    picontrol(tier 1)|abrupt-4xCO2(tier 1)|historical(tier 1)|ssp370(tier 2)

ssp370 could then clearly be interpreted as tier1 in scenarioMIP, but tier2 in AerChemMIP.
In this case the decision would be very simple for us – we are doing ScenarioMIp, it is tier 1 so we would do it whether or not we are planning to do AerChemMIP.  However if the tier allocation were reversed then we would only do it if we were planning to do AerChemMIP.
I think this would be consistent with DREQ which records tier allocation for an experiment in each MIP
This approach doesn’t cover all possibilities but is probably good enough for the few cases we have.
Mark
From: taylor13<EMAIL_ADDRESS>Sent: 02 May 2017 23:54
To: WCRP-CMIP/CMIP6_CVs
Cc: Elkington, Mark; Mention
Subject: Re: [WCRP-CMIP/CMIP6_CVs] experiment_id inconsistencies with GMD and ES-DOC (#316)
@charliepascoehttps://github.com/charliepascoe @durack1https://github.com/durack1 @martinjuckeshttps://github.com/martinjuckes @momipslhttps://github.com/momipsl @markelkingtonhttps://github.com/markelkington @eguilhttps://github.com/eguil @bnlawrencehttps://github.com/bnlawrence @matthew-mizielinskihttps://github.com/matthew-mizielinski You are right, Charlotte, that the tier of an experiment can depend on the MIP. If there are 2 MIPs that include the same experiment, the priority might (and in I think 2 cases does) differ. Perhaps Mark E. can suggest a way to handle this more gracefully. Currently in the CV, there is a single tier given. As I recall for the 2 cases where there are different tiers depending on the MIP, I've recorded the highest priority, which in both cases was set by the prime sponsor of the experiment (the MIP that appears in the DRS identifier).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/WCRP-CMIP/CMIP6_CVs/issues/316#issuecomment-298783752, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFTIJ5lt3xI9T_GxDyoq75QZnK8j5gJjks5r17PxgaJpZM4NKkm5.

Folks it seems that we're now working toward a more refined versioning system in #28, for this reason I will close this issue - please reopen if things have not been resolved completely


Added changes to sign artifacts

@kumaritanushree Was this tested on a fork? (Just curious because I don't see the "Setup cosign" step)

@kumaritanushree Was this tested on a fork? (Just curious because I don't see the "Setup cosign" step)

Thank you @praveenrewar  for pointing this. I have tested the process in fork but this step got missed in this PR. Added new PR to fix this
https://github.com/carvel-dev/vendir/pull/352


⚠ Bofrost.be has degraded performance

In 4256309, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1934 ms

Resolved: Bofrost.be performance has improved in bf42adf.


[BEAM-3545] Support Metrics in Go SDK CombineFns

It was determined that CombineFns should reasonably have metrics. This PR does two things:

Populate the CombineFn PTransform id into a context.
Fix an infinite recursive loop in the context code, and added a proper test case for it.

These were caught by a stackoverflow from the latter point, due to the former point.
While it's unlikely for a future transform to be missing it's bundleID (that's applied very early in the stack) it's likely to be missing it's ptransformID, if some new operation is added (eg. SplittableDoFns, timers, etc). Now it will clearly indicate (ptransformID unset) in this case.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

R: @youngoli Please take a look, and ask questions as needed.

R: @aaltay Please review and merge. Thanks!


[295] Implement diversity disclosure section

Context

https://trello.com/c/ljSVTV3Z/295-s-record-if-the-trainee-has-shared-diversity-details-or-not

Changes proposed in this pull request

Create the start of the diversity section (disclosure)

Guidance to review

 [S] Record if the trainee has shared diversity details or not


[500] You may forget to apply app.use(head)

Describe the bug

but working well is exclude extends: ['nuxt-seo-kit']
I can't figure out which way to look (
{
  "name": "website",
  "private": true,
  "type": "module",
  "scripts": {
    "build": "nuxi build",
    "build:static": "nuxi generate",
    "dev": "nuxi dev --port 3010",
    "generate": "nuxi generate",
    "preview": "nuxi preview",
    "start": "node .output/server/index.mjs"
  },
  "dependencies": {
    "@segment/analytics-next": "^1.51.0",
    "@unocss/transformer-variant-group": "^0.49.4",
    "@vueuse/core": "^9.12.0",
    "@walletconnect/web3-provider": "^1.8.0",
    "axios": "^1.3.3",
    "lottie-web": "^5.10.2",
    "nuxt-swiper": "^1.0.0",
    "std-env": "^3.3.2",
    "unocss-preset-scrollbar-hide": "^1.0.1",
    "vite-compatible-readable-stream": "^3.6.1",
    "vue-loading-overlay": "^6.0.3",
    "vue-use-web3modal": "workspace:*",
    "web3": "^1.8.2",
    "ws": "^8.12.1"
  },
  "devDependencies": {
    "@nuxt/devtools": "^0.1.2",
    "@nuxt/image-edge": "1.0.0-27919678.2f5b64b",
    "@types/node": "^18.13.0",
    "@unocss/nuxt": "^0.49.4",
    "@unocss/transformer-directives": "^0.49.4",
    "@vueuse/nuxt": "^9.12.0",
    "fast-glob": "^3.2.12",
    "nuxt": "3.2.0",
    "nuxt-seo-kit": "^1.2.3",
    "nuxt-unhead": "^1.3.4",
    "postcss": "^8.4.21",
    "postcss-nested": "^6.0.0",
    "web3-utils": "^1.8.2"
  },
  "engines": {
    "node": ">=18",
    "pnpm": ">=7"
  }
}

Reproduction
No response
System Info
System:
    OS: macOS 13.2.1
    CPU: (8) arm64 Apple M2
    Memory: 1.07 GB / 24.00 GB
    Shell: 5.8.1 - /bin/zsh
  Binaries:
    Node: 18.14.0 - ~/Library/Caches/fnm_multishells/42035_1676455696779/bin/node
    Yarn: 1.22.19 - ~/Library/Caches/fnm_multishells/42035_1676455696779/bin/yarn
    npm: 9.3.1 - ~/Library/Caches/fnm_multishells/42035_1676455696779/bin/npm
  Browsers:
    Chrome: 110.0.5481.96
    Safari: 16.3

Used Package Manager
pnpm
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guide.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A, please open a GitHub Discussion instead.
[X] The provided reproduction is a minimal reproducible of the bug.

Faced the same problem today

Hi @reslear and @DmitrySkakun
You both seem to have a dependency on the old version of @vueuse/head (0.9.8).
I'm guessing one of your other dependencies has it hardcoded to this version. The reason why nuxt-seo-kit throws an error with it is nuxt-seo-kit uses the latest features within @vueuse/head, unlike Nuxt itself.
Could you run pnpm why @vueuse/head or npm why @vueuse/head and paste the output?

hi @harlan-zw thx for fast answering :)

this seems to have happened after upgrading to nuxt 3.2 and adding nuxt devtools...

Hm this is really strange, doesn't say you have had the old vueuse/head as a dependency. :thinking:
Could you try clearing your node_modules folder and re-install deps with as pnpm i --shamefully-hoist

@harlan-zw also you right, is click to error i'm jumping to @vueuse/head (0.9.8).
so i install in deps @vueuse/head as pnpm add @vueuse/head@latest - and work fine (    "@vueuse/head": "^1.0.25")
thx for support :) have a good day 😀


Make queue manager configurable.

Fixes #2990
Also, invoke HTTPClientConfig.validate manually, as it won't be called automatically.

@brancz can I take the "looking good" as an LGTM?

yep, sorry for not following up. lgtm :+1:


feat: add last_activity_at to feedback datasets

Description
This PR adds a new DateTime column to datasets table with name last_activity_at.
This new column will be updated for the following cases:

When a dataset is created or updated.
When a response for a record belonging to the dataset has been created, updated or deleted.

The idea is that we can use last_activity_at to record the last time that an activity associated with a dataset happened and use it instead of updated_at to show recent activity on the UI.
Closes #3981
Type of change

[x] New feature (non-breaking change which adds functionality)

How Has This Been Tested

[x] Adding new tests and running them locally.
[x] Running tests using PostgreSQL locally.

Checklist

[ ] I added relevant documentation
[x] follows the style guidelines of this project
[x] I did a self-review of my code
[ ] I made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[ ] I filled out the contributor form (see text above)
[ ] I have added relevant notes to the CHANGELOG.md file (See https://keepachangelog.com/)

@damianpumar please feel free to add here the frontend changes, using this new dataset attribute instead of updated_at for feedback datasets.
Once it's done please set the PR as ready for review.


Initial Commit to store job_ids info

@antonyr please close this PR. Created another PR #3  for the same.

closed on dev's request


Create translation.json

ru

@vanja-san we changed the structure of the repo, so we do not force push the main branch no more. Could you please possibly make a new pull request with those changes? That would be very appreaciated! Thank you for your contribution and sorry for the inconvinience.


fix: add missing main in package.json

Which problem is this PR solving?

Add missing main to package.json

Short description of the changes

Missing main in package.json makes developer unable to do require("@opentelemetry/shim-opentracing");

Codecov Report

Merging #577 into master will decrease coverage by 0.04%.
The diff coverage is n/a.


tests: run tests on Node 10, 14 & 15

As the engines are set to >=10, it would be a good idea to run tests against all supported Node versions
https://github.com/kgnugur/formik-chakra-ui/blob/7647e9552b4a3be7b54cf9ad15155c474be150c7/package.json#L21-L23
I also added running tests on PRs

Thank you


fix: use OpenGraph image URL and remove server request from user profile

Description

Removes the OpenGraph image request from getServerSideProps
Returns the OpenGraph image directly as the OpenGraph request is fast enough

Related Tickets & Documents

Related to: https://github.com/open-sauced/app/issues/3066
Mobile & Desktop Screenshots/Recordings

Steps to QA

Visit user profile in OpenGraph image viewer
Note OpenGraph image is displayed
Note no function timeout

Tier (staff will fill in)

[ ] Tier 1
[ ] Tier 2
[ ] Tier 3
[x] Tier 4

[optional] Are there any post-deployment tasks we need to perform?
[optional] What gif best describes this PR or how it makes you feel?

Fixed the OpenGraph image display issue on beta, so it should display now


Namespace creation for custom app on linkerd

When we try to deploy a custom app using yaml on a new namespace, the app deployment fails because the new namespace is not created.
Steps to reproduce the behavior:

Configure Linkerd
Go to Playground -> Linkerd
Give a new namespace and click on custom Yaml
Provide a custom yaml file and click Submit

Expected behavior
The provided app should be deployed.
We see error message: meshery-linkerd_1  | time="2019-04-28T21:24:02Z" level=warning msg="unable to create the requested resource, attempting operation without namespace: namespaces "linkerdapp" not found"

@girishranganathan is this an issue in Meshery or the Linkerd adapter?

It is not a bug. I was just explaining to @prateeksahu.
When using custom yaml, if a namespace needs to be created, it should be included in the yaml.
Reason: a custom yaml could contain any valid k8s yaml which Meshery and its adapters today don't try to interpret today. Their goal, atm, is just to apply the given yaml as it is.
When @prateeksahu updated his yaml to include the namespace block in it, it worked as expected.

Updated issue with "docs" label.

The reported message is just a warning. I will close this issue for now. We can reopen if it is really an issue.


Compilation of old errors that need to be checked by people

Hi guys, I just did a purge of all of the old inactive errors. The ones I did not immediately know a fix for I've compiled in the list below;

Option to resume failed downloads
Adding some way to filter eg filler lists that @ArjixGamer mentioned in the discord
disable ssl not carrying to aria
downloading from watch re-downloads instead of skipping previously downloaded eps
can't get weird episode names, like censored and uncensored
anime dl https://kisscartoon.is/Cartoon/Mia-and-Me-Season-3 -e 15:26 -ll DEBUG -xd "{aria2}"  fix this command
animepahe NoneType groups error is back
Handling special characters in anime names, K-ON! and K-ON!!
Adding tracking to watch
Improve quality handling
KissAnime picks lowest quality?
KissAnime using RapidVideo when RapidVideo is broken, Unfortunately this must be delegated to Arjix and Red Guy as they re-implemented KissAnime. They'll know this answer

If anyone has any free time, consider working on these issues.

anime dl https://kisscartoon.is/Cartoon/Mia-and-Me-Season-3 -e 15:26 -ll DEBUG -xd "{aria2}" fix this command

Isn’t -ll DEBUG in the wrong place? Shouldn’t it be before dl?

anime dl https://kisscartoon.is/Cartoon/Mia-and-Me-Season-3 -e 15:26 -ll DEBUG -xd "{aria2}" fix this command

Isn’t -ll DEBUG in the wrong place? Shouldn’t it be before dl?

Yes, put it in the correct place and it still errors. The line was from a really old error where -ll DEBUG could be called like that.

so back then the log level was handled by dl.py?

so back then the log level was handled by dl.py?

There was no dl.py back then.

KissAnime picks lowest quality?
Should be using hydrax which got fixed to use quality based on config
Either way doesnt really matter as kissanime implemented hcaptcha which pops up pretty often. Kissanime should be disabled or selenium adds support for hcaptcha (brings up non headless for users).

* Option to resume failed downloads

If we can pass arguments for aria2c, we can pass --always-resume=true to always resume the download from .aria2 file.

:heavy_check_mark: Option to resume failed downloads
Pretty sure this is already the case
:heavy_check_mark: Adding some way to filter eg filler lists that @ArjixGamer mentioned in the discord
--skip-fillers got added
:heavy_check_mark: disable ssl not carrying to aria
Pretty sure aria2 downloads regardless of ssl
downloading from watch re-downloads instead of skipping previously downloaded eps
I think this is due to different filenames, will investigate
can't get weird episode names, like censored and uncensored
I'll need some further explaining
anime dl https://kisscartoon.is/Cartoon/Mia-and-Me-Season-3 -e 15:26 -ll DEBUG -xd "{aria2}" fix this command
Will do
:heavy_check_mark:  animepahe NoneType groups error is back
Deactivated, but will be investigated once m3u8 downloader is back
Handling special characters in anime names, K-ON! and K-ON!!
Willfix
Adding tracking to watch
Leaving this to @IguanasInPyjamas
Improve quality handling
Too vauge, it seems to work fine already
:heavy_check_mark:  KissAnime picks lowest quality?
KissAnime using RapidVideo when RapidVideo is broken, Unfortunately this must be delegated to Arjix and Red Guy as they re-implemented KissAnime. They'll know this answer
Dead


Issue with new SafeCtrTransfer steps

After the CtrTransfer completes, you are told to reboot into 2.1.0, however, the SD card at that point is still inserted and you will black screen on boot. This issue refers to the gm9 branch.

https://github.com/Plailect/Guide/commit/30ef6c588d15c69cbb160babaf8950aa85dbace0


.NET Standard 2.0 target should avoid a System.Text.Json 9.0.0 dependency

The package dependency only shows up in the netstandard2.0 target framework anyway, and by referencing 9.0.0, even .NET 8 test projects were required to reference STJ 9 rather than the one that comes with the runtime.
Given tests should be testing the library in their natural environment, we should avoid forcing a test to update dependencies beyond what the library under test would naturally require.
Referencing STJ 9 is fine once .NET 9 is the oldest supported runtime by Microsoft (in about a year).

Nit: net9 will never be the oldest supported framework. Net8 support will continue until after net10 is released.

Are these libraries actually tightly coupled to the .NET version? If it's only in .net standard anyway it won't be affecting net core runtimes.

Are these libraries actually tightly coupled to the .NET version? If it's only in .net standard anyway it won't be affecting net core runtimes.

Correct. STJ is not considered tightly coupled to the net sdk, and developers are encouraged to use newer STJ versions with older sdk.

If it's only in .net standard anyway it won't be affecting net core runtimes.

It does, actually.
Consider a .NET Standard 2.0 library that consumes this one. It now requires STJ 9.
Now a .NET 8 application project comes along and references the .NET Standard 2.0 library. Now that .NET 8 app must ship STJ 9, instead of relying on the one that .NET 8 ships. This makes the app larger. It also increases the servicing onus on the app author because now the (relatively regular) servicing updates to STJ have to ship in new app servicing releases instead of just when the app user keeps their .NET 8 runtime current.

It also increases the servicing onus on the app author because now the (relatively regular) servicing updates to STJ have to ship in new app servicing releases instead of just when the app user keeps their .NET 8 runtime current.

I'm not sure what you mean here, since both net8 and net9 have monthly servicing releases, and STJ follows the SDK release schedule.

@viceroypenguin A .NET 8 app that does not ship System.Text.Json.dll (because it uses v8 which is included in the runtime) has no servicing responsibility for that dll, because the user that installed the app is responsible to keep their runtime updated.
In contrast, a .NET 8 app that does ship System.Text.Json.dll v9 now has to service the app itself to update the STJv9 DLL whenever a security vulnerability is found in it.
And it makes the app larger.
I'm assuming a FDD (framework-dependent deployment) app here. If it's an SCD (self-contained deployment) app which includes the runtime, then there's a servicing onus either way.


Convergence criteria

Is your feature request related to a problem? Please describe.
Currently, only fixed number of iteration can be set to the optimizer. However, a convergence criteria would be useful.
Describe the solution you'd like
By using a convergence criteria, a maximum number of steps OR the convergence criteria could be used to stop the iteration (whichever happens first). The reason why the iteration stopped should be available after the execution.
Are you able and willing to implement this feature yourself and open a pull request?

[X] Yes, I can provide this feature.

If anyone can do this issue, that would be great, however, I'm also fine with doing it myself in a form of PR.

I wouldn't mind adding optional convergence criteria as an argument that can be passed at instantiation.
At the moment, there is only really one - number of iterations. But we could also add:

If x iterations have occured without at least y improvement (y should probably include both relative and absolute tolerance?)
if a solution is found better than some value
others?

* If `x` iterations have occured without at least `y` improvement (y should probably include both relative and absolute tolerance?)

This is exactly what we need. However, your other mentioned option also looks great, and might be useful.

I wouldn't mind adding optional convergence criteria as an argument that can be passed at instantiation. At the moment, there is only really one - number of iterations. But we could also add:

If x iterations have occured without at least y improvement (y should probably include both relative and absolute tolerance?)
if a solution is found better than some value
others?

From my project experiences, here are some termination criteria:

input variables(X) tolerance based termination
target function value tolerance based termination
max iteration / evaluation
time based termination


Added section for lp

@xyutech  please use the fromLocalPath() API to call the lightPipeline, that's the sole purpose of using it.


feat: remove pulumi

Description

remove pulumi dependencies

Testing required outside of automated testing?

[x] Not Applicable

Screenshots (if appropriate):

[ ] Not Applicable

Rollback / Rollforward Procedure

[x] Roll Forward
[ ] Roll Back

Reviewer Checklist

[ ] Description of Change
[ ] Description of outside testing if applicable.
[ ] Description of Roll Forward / Backward Procedure
[ ] Documentation updated for Change

:tada: This PR is included in version 2.6.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


updates astro + back to top button

adds selection color

Thanks.
This minimal template is meant to work with JS disabled except on two pages: search and map. I prefer not to introduce additional mandatory JS. So I will not merge this change into my repo.


bug in ODEFnc()

There is a bug in ODEFnc(), for the Parameter can not be augmented assignment directly, or else it would have an error "cannot assign ‘torch.cuda.FloatTensor’ as parameter ‘self.w’ (torch.nn.Parameter or None expected)"
it should be :
        self.w.data = (1 + self.beta) * self.w - self.beta * torch.mm(torch.mm(self.w, torch.t(self.w)), self.w)
        xw = torch.einsum('ijkl, lm->ijkm', x, w)

        d2 = torch.clamp(self.d2, min=0, max=1)
        w2 = torch.mm(self.w2 * d2, torch.t(self.w2))
        self.w2.data = (1 + self.beta) * self.w2 - self.beta * torch.mm(torch.mm(self.w2, torch.t(self.w2)), self.w2)

You are right, but we found such operation led to suboptimal convergence. And a simple alternative is to just delete this line, which means we don’t constrain $self.w$ to be orthogonal, and it works in practice. We have fixed the bug in this way.
Thanks for your attention.

tks


combine timeout as error with RecvError ?

I'm not sure if can post this as an issue, but here it goes:
I'm trying to use Receiver::recv() together with a timeout, similar like this example.
However, as Receiver::recv() returns RecvError, not std::io::Error, things got complicated. I ended up to use anyhow::Error but it really feels clumsy.  (see code below).
Does RecvError itself support converting from std::io::ErrorKind? or, is there a plan to support users to create RecvError for timeouts?
Here is my function complicated by error handling:
async fn chan_recv_with_timeout(chan: &Receiver<Vec<u8>>, timeout: Option<Duration>) -> anyhow::Result<Vec<u8>> {
    let chan_f = chan.recv();
    if let Some(duration) = timeout {
        let time_f = Timer::after(duration);
        futures::pin_mut!(chan_f);
        match future::select(chan_f, time_f).await {
            Either::Left((c, _)) => match c {
                Ok(t) => Ok(t),
                Err(r) => Err(anyhow::Error::new(r)),
            },
            Either::Right(_) => Err(anyhow::Error::new::<io::Error>(io::ErrorKind::TimedOut.into())),
        }
    } else {
        match chan_f.await {
            Ok(t) => Ok(t),
            Err(r) => Err(anyhow::Error::new(r)),
        }
    }
}

You could reuse my timeout() function and do:
timeout(duration, async { io::Result::Ok(chan.recv().await.ok()) }).await;

This way the result is io::Result<Option<Vec<u8>>, where Ok(None) means the channel is closed and empty.
Or, alternatively:
timeout(duration, async {
    match chan.recv().await {
        Ok(v) => v,
        Err(err) => io::Error::new(io::ErrorKind::Other, err),
    }
})
.await;

The return type is io::Result<Vec<u8>>, and Err(_) will represent both timeouts and channel being closed and empty.


investigate adding a visited set to the skeleton generator in sesame

we're not 100% sure if anything would break, but I expect that in some cases, planning would get faster

turns out this doesn't work on stick_button and coffee (they both get around 25/50 with oracle), the reason is related to the general fact that predicates are a lossy abstraction for the purposes of bilevel planning, so in general we don't want a visited set for skeleton generator in sesame
specifically, in stick_button:

the predicates can't distinguish whether a button is reachable by the stick or by the robot
and so a correct sequence of presses would end up with the same atoms as an incorrect sequence of presses
for example, press 0, pick stick, press 1, press 2, press 3 versus press 1, pick stick, press 0, press 2, press 3. if only the second one is correct, we'll never try it, because after press 2 the atoms are the same as in the first skeleton


Z5: quick-copy & automatic journal abbreviation

continuation of additional posts in https://github.com/retorquere/zotero-better-bibtex/issues/604
Thanks for a quick reply there! I am afraid however that it is still not working for me.

I installed pre-relase v5.0.22 (through the automated update) and updated Zotero to 5.0.20 (through the automated update too).
I see the new option in Better BibTeX settings Pane: Journal abbrevations, and I have the only option there checked.
Unfortunately still when I do quick copy of an entry with ctrl+shift+c the journal names don't get abbreviated. When I click it however with RMB and select export, then it does---just in case you'd be worried that the entry on which I am testing it is corrupted.
I will be happy to provide any additional details you may need.

What quick-copy translator do you have set?

It is Better BibTeX. This is the option I set. Is there any other place I should change settings in?

Can you right-click the reference you were quick-copying and select "Send BBT error"?

Here it comes IZ8RPBY6

Ah, that's a different matter. You're not looking for auto-abbreviation (which is what the preference controls) but to have the abbreviated title exported to the reference journal field instead of the full title.
What you want is not possible; during regular export you can check Use journal abbreviation, but there's no way to communicate this intent to quick-copy, and Zotero doesn't tell translators whether they're being used for quick-copy or regular exports, so I can't enforce it for quick-copy only.

Ah... I see... Do you know maybe about some extension that could go over all my entries and abbreviate them? Anyway, this issue can be closed then.

You want all your reference titles replaced by their abbreviated titles? Or do you want the Abbreviated Title field filled for all your references?

I want to have Quick-Copy export abbreviated name as a title so I hope just substituting all names by abbreviated form would do the trick. I could always export first to file (checking abbreviate option) and then copy paste it wherever I want to but it seems like annoying hassle to do.

If Zotero had export options for quick-copy I could make use of those, but as it is there's nothing I can do about this that would not also affect regular exports.
I don't know of any extensions that do what you want, sorry.

For reference if anyone comes across this issue:
A workaround to get journal abbreviations with Better BibTeX quick-copy is to edit Zotero\translators\Better BibTeX.js and set "useJournalAbbreviation" to true.

Which will get overwritten the next time BBT updates.


cannot find latest release

I have:
build.yml
- uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
   with:
     version: '272.0.0'
     service_account_key: ${{ secrets.GOOGLE_CLOUD_SERVICE_ACCOUNT_JSON_KEY }}

When I look at the logs, I see:
(node:7088) UnhandledPromiseRejectionWarning: Error: Failed to find release, os: linux arch: x64 version: 272.0.0

Interestingly some of my commands do seem to run okay, though maybe it's the previously cached version?
Cloud run, which was made GA in this version, does not run:
build.yml
- name: Deploy
   run: gcloud run deploy --image gcr.io/"$GOOGLE_CLOUD_PROJECT"/"$CONTAINER_IMAGE":"$GITHUB_SHA" --platform managed

Result:
ERROR: (gcloud) Invalid choice: 'run'.
This command is available in one or more alternate release tracks.  Try:
  gcloud alpha run
  gcloud beta run

when I have it outputting the version it shows as 268.0.0

271.0.0 works - just not 272.0.0

I think I see the problem - 272 is not on the public bucket: https://console.cloud.google.com/storage/browser/cloud-sdk-release?prefix=google-cloud-sdk-27

ah, according to https://cloud.google.com/sdk/docs/downloads-versioned-archives from 272 onwards it needs to pull from new location, not the cloud storage...

Made a PR - fixes it for me and all tests pass. Can now deploy to cloud run with 272.0.0 :D
Please merge so I can depend on the official repo instead of my fork. Thanks!

Thanks for raising the issue as well as the PR. To clarify, every version except the latest release is contained within the archive bucket. You do raise a good point though that we should allow for specification of the latest release within the action.

Fixed by: https://github.com/GoogleCloudPlatform/github-actions/pull/20


Featured articles

What is this PR trying to achieve?
Adding the featured articles block to the top of the explore landing page. I've introduced a grid--scroll class, but the behaviour and styles are relatively custom to this particular case (e.g. two different types of borders are necessarily coupled to the grid). I'm wondering if a less abstract e.g. grid--featured for this would be better for use here (and only here).
What does it look like?

For interface components

[x] Browser testing - Have you checked the component in our supported browsers
[x] No javascript - Have you checked the component with javascript disabled

I'm wondering if a less abstract e.g. grid--featured for this would be better for use here (and only here).

I would think so, or even a higher order featured-grid.
We can try to abstract it out if we get another case of this, that way at least we have multiple examples to test it against.


Create config file

Set binary size threshold.

AAA=BBB
CCC=DDD
threshold=1M

threshold.firefox=500K

priority: command option -> threshold. -> threshold


JBDS-3878 pull installer version from...

JBDS-3878 pull installer version from package.json

This PR is not quite perfect.... ran fine, built fine, even the tests ran OK.
But on launch of the 1.6G installer, I get:
system.js:4 Uncaught (in promise) Error: Error: ENOENT: no such file or directory,
open 'C:\Users\nboldt\AppData\Local\Temp\7zS82BB.tmp\package.json'(…)

So either need a different way to depend on package.json, or need to copy that file over into the .asar file somehow. @dgolovin @joshuawilson @jrichter1 @ppitonak any suggestion how to do that?


PCC issue for matmul 2d and in falcon40b attention

Describe the bug
In falcon40b we see an issue with the two attention matmuls (QKˆT) and (scoresV) when using a custom program config for sharding.
Obervations:

Unit test for the op in isolation is fine
Running the whole model end to end with one layer is fine too
Running the model with multiple layers (>=12 layers) in the end to end test shows significant drop in PCC, most apparent for the V Cache
Using interleaved inputs and the default program config has good PCC
Using interleaved inputs and the custom program config has bad PCC
Using sharded inputs (that's what we ultimately want) and the custom program config has bad PCC
We can't run sharded with the default program config

To Reproduce
Steps to reproduce the behavior:

Checkout branch jrock/prefill-perf2@8b1618f8beb4875730af7e27be48457ac806c39e
Run the the falcon40b end to end test
pytest models/experimental/falcon40b/tests/test_falcon_end_to_end.py::test_FalconCausalLM_end_to_end_with_program_cache[BFLOAT8_B-DRAM-falcon_40b-layers_12-prefill_seq128-8chips]
You'll see bad PCC

Output: Max ATOL Delta: 3.618520736694336, Max RTOL Delta: inf, PCC: 0.9980098587307713
K Cache Layer 11: Max ATOL Delta: 1.4973526000976562, Max RTOL Delta: inf, PCC: 0.9921958122243109
V Cache Layer 11: Max ATOL Delta: 0.6698611974716187, Max RTOL Delta: inf, PCC: 0.9474645013210232, PCC check failed

Go to models/experimental/falcon40b/tt/falcon_attention.py:550 and comment to use the default program config
Run the test again
You'll see good PCC

fyi @s-jovic

The issue is that we're using LoFi for the 1D config, and HiFi4 for the default program config.
Even tho you specify that you're using LoFi fro the default program config (that is how it is done in the code via compute_kernel_config), default program config makes the underlying matmul use HiFi4.
Switching the 1D compute kernel config to use HiFi2/HiFI3/HiFi4 yields better PCC then the default one (since packer_l1_acc and FP32 dest accumulation are turned on as well). PCC of the V cache in that case becomes 0.9884457046


Update old members.js script for theme handling

The old members.js script for theme data attributes handling for members enabled themes that was ported has not been updated for a while and has new additions since like reading name with data-member-name.

[x] Update and verify the members.js theme handling script is up-to-date

Closed with https://github.com/TryGhost/Portal/commit/2740805fdf8762e2802271c360ec5fe9cdf16f06


Add shell command to load plugin

Closing as I had forced pushed some derp fixes without noticing you had started work on this repo. I included those mods in another force push. I'll stop that now.


fix/fetch: when fetching a MD the resourceType value was 'NFS' instead of 'MD'

When calling the fetch experimental API to fetch a MD the value returned in the resourceType was 'NFS' instead of 'MD'. I'm also adding some tests for fetching different type of resources, i.e. MD, IMMD, RDF and NFS.

Coverage decreased (-7.3%) to 88.988% when pulling dc6bbed742f1245777a7aef89768599460463596 on bochaco:fix-type-fetch-md into bdad4da5dc511c9d79394c42f6997d7ddb28643b on maidsafe:master.


bug(slider): Slider doesn't render values in SSR (Universal)

Is this a regression?

[ ] Yes, this behavior used to work in the previous version

The previous version in which this bug was not present was
No response
Description

When rendered on the server with Angular Universal, the slider thumb always renders at the left-most position, regardless on the actual value
What's more, a range slider only shows the "start" thumb, but not the "end" thumb.

Reproduction
(don't know where to host a Universal app)
Expected Behavior

Two thumbs are rendered in SSR for a range slider
Slider thumbs positions in SSR reflect the actual slider values

Actual Behavior

Only the "start" thumb is rendered in SSR
The the start thumb is rendered in the left-most position in SSR, regardless on the actual value

Environment

Angular: 15
CDK/Material: 15
Browser(s):
Operating System (e.g. Windows, macOS, Ubuntu):

@amakhrov I believe this should be fixed in v17. I created this example:
https://github.com/wagnermaciel/demo
Let me know if things appear fixed for you as well

Nope, the demo looks as if nothing changed:

Only the "start" thumb is rendered in SSR
The the start thumb is rendered in the left-most position in SSR, regardless on the actual value

It's easier to repro if you block main.js in the browser network console, to make sure you're only looking at the server-rendered content

Ah I see, I was getting tricked with main.js quickly hiding the issue. I'll take a closer look at this and see where things are going wrong
A quick side note on the bug description: Both thumbs are being rendered, but they are just displayed overlapping at a broken/default position. This is likely because we're using something like getBoundingClientRect, etc.

Update
The root problem is that we cannot call getBoundingClientRect() on the server.
Slider thumbs behavior explained
The slider thumbs are positioned using transform: translateX(${someValue}px). In order to calculate someValue we need to know the size of the whole mat-slider which depends on us calling getBoundingClientRect(). Since we are on the server, getBoundingClientRect() does not exist and therefore we cannot determine someValue. That is why both thumbs are just overlapping at the 0th position - they both just have transform: translateX(0px)
Theoretical fix?
We'd need to do a lot of targeted overrides on mdc's styles. From what I can see, there are 3 main places we'd need to tweak: track, thumb, and tick marks. This change would be very hacky but it is theoretically possible

Is it possible to define all positions relatively in percents instead of absolutely in px? This way, we wouldn't need getBoundingClientRect()

Yeah, that's the part that's theoretically possible, but if we did it on our end it'd require a bunch of hacks due to how material design has structured the html/css for these components. We'd need to override a bunch of their css in order to get things working
Ideally this would be changed on their end, but that would add a ton of churn to getting this fixed


fix(react-virtual): add missing react-dom peer dependencies

closes #612

thanks. can we publish the release to npm?

thanks. can we publish the release to npm?

Yes, it’s already there.

Can you recheck?

https://www.npmjs.com/package/@tanstack/react-virtual

Ooo looks like something went wrong while auto publishing 😞

@simPod checkout the 3.0.0-beta.68

CI is green now thanks!


Typo Fix

Fix a small typo for the page on troubleshooting reporting services installation.

@snhardin : Thanks for your contribution! The author, @markingmyname, has been notified to review your proposed change.

Thank you @snhardin for catching this.  Approved change.
#sign-off


RuntimeError

when inferring on my own imgs, RuntimeError: Given groups=1, weight of size 192 192 3 3, expected input[1, 384, 4, 4] to have 192 channels, but got 384 channels instead. Any idea how to fix it?

What is the shape of the input tensor that you are feeding into the model?

What is the shape of the input tensor that you are feeding into the model?

It is 3x116x55

Please resize and pad your input to match the expected input dimensions of the model. Refer to infer_single.py:
https://github.com/anibali/margipose/blob/master/src/margipose/bin/infer_single.py#L58-L66

Please resize and pad your input to match the expected input dimensions of the model. Refer to infer_single.py:
https://github.com/anibali/margipose/blob/master/src/margipose/bin/infer_single.py#L58-L66

Does the input shape has to be 3x256x256?

That size is guaranteed to work, I think that other spatial dimensions could result in rounding errors. Also, make sure that you add a batch dimension (even if it is just 1). So your input should be 1x3x256x256. Again, refer to the script.

That size is guaranteed to work, I think that other spatial dimensions could result in rounding errors. Also, make sure that you add a batch dimension (even if it is just 1). So your input should be 1x3x256x256. Again, refer to the script.

Thank you for your help. Closing...


XML behavior with parameters with children but without group type

Parameters in pyqtgraph can have children without the necessity for them to be of group type.
This behavior is currently not being adressed as the type of parameters define how it will be explored and written.
With this patch, it is now possible to correctly write and read an XML file which uses parameter with:

non-group type
children

Wrong branch

could you add such a parameter in the parameter_ex? We use that guy for all the testing so your stuff would be tested also!

could you add such a parameter in the parameter_ex? We use that guy for all the testing so your stuff would be tested also!

This is in place, I added it in the group category

actually are you using this functionnality? I'm a bit worried it would create issues in some things I programmed assuming only a group would have children...

we should therefore be particularly cautious with previous filtering using the 'group' string in type.
For instance, in ioxml line 45:
if 'group' not in param_type:  # covers 'group', custom 'groupmove', 'groupai' ...
    add_text_to_elt(parent_elt, param)

this could trigger an issue. add_text_to_elt should be used when there is a value to be written as text in the xml element. Therefore the  if clause should be:
if 'value' in param.opts:

I know this one is not directly related to your PR but we should properly review all the locations of such use of the check for 'group' in param.type() kind of clause

we should therefore be particularly cautious with previous filtering using the 'group' string in type. For instance, in ioxml line 45:
if 'group' not in param_type:  # covers 'group', custom 'groupmove', 'groupai' ...
    add_text_to_elt(parent_elt, param)

this could trigger an issue. add_text_to_elt should be used when there is a value to be written as text in the xml element. Therefore the if clause should be:
if 'value' in param.opts:

I know this one is not directly related to your PR but we should properly review all the locations of such use of the check for 'group' in param.type() kind of clause

I agree with you, I wanted to do minimal changes to avoid problems with the previous implementation but I'd much rather change
it as you suggest. I will look through the codes to check if I find other cases like this one.

I agree, there is actually no reason to not include an extra entry with keyword value. You could say that it is redundant with the text attribute but it gets much clearer in the management. It should be quite straightforward to adapt the functions in the ioxml file.
I think it is good for a go, the tests are all getting passed and the behavior with loading/saving/updating in the parameter manager is ok too.
Btw, I realize we were not handling well the label in the bool_push parameter item.
commit 2f4533ae7b1c44276f76ec28defce6c91dbbee23 and commit a053153948baa991fda66f87601e61b2d1f271f8 fixes it.

I added in this PR some small changes in the iter_children_params function (parameter/utils.py) that allows one to filter parameters from the list.
For now, I added a filter for the 1) type or for the 2) name
The user can decide if he wants to keep or not keep the filtered parameters with a boolean argument.
For example, this can be convenient to remove the group type which does not have any value or also to select some specific parameter types or names
Without any inputs, the behavior is the same as before.
I added a test (test_iter_children_params) in param_utils_test.py checking that the behavior is consistent

The utils module also has an iter_children method that gives a list of children names. It would be better if both methods: iter_children and iter_children_param have the same signature and functionality! Could you add the implementation?

Is it ok if I merge the two? I think we can just add an extra agument to the iter_children_param to get out all the relevant values that one is interested in?
If so, do I just add a deprecation signal for the iter_children function which redirects to the iter_children_param method?

well no deprecation, it is heavily used in commit_settings in the plugins.
It is meant to quickly check if a given name is within a Parameter structure. So it should only return a list of string not a list of Parameter. However it could have the same filtering (make more sense having similar functions having the same kind of signature). But internally you could just use your new iter_children_param and extract the list of names out of it!


[R] Add as_scalar()

There's as_everything_else but not as_scalar.
Reporter: Neal Richardson / @nealrichardson
Note: This issue was originally created as ARROW-17607. Please see the migration documentation for further details.

Is there more info we can add to this issue? Add as_scalar() where?


Problem with a cat

$ parquet cat train-00000-of-00001-15a05aeec7726f9d.parquet
Unknown error
shaded.parquet.org.apache.avro.SchemaParseException: Illegal character in: original-instruction
at shaded.parquet.org.apache.avro.Schema.validateName(Schema.java:1607)
at shaded.parquet.org.apache.avro.Schema.access$400(Schema.java:92)
at shaded.parquet.org.apache.avro.Schema$Field.(Schema.java:556)
at shaded.parquet.org.apache.avro.Schema$Field.(Schema.java:595)
at org.apache.parquet.avro.AvroSchemaConverter.convertFields(AvroSchemaConverter.java:295)
at org.apache.parquet.avro.AvroSchemaConverter.convert(AvroSchemaConverter.java:279)
at org.apache.parquet.cli.util.Schemas.fromParquet(Schemas.java:89)
at org.apache.parquet.cli.BaseCommand.getAvroSchema(BaseCommand.java:405)
at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:66)
at org.apache.parquet.cli.Main.run(Main.java:163)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.parquet.cli.Main.main(Main.java:193)
the data set in question is: https://huggingface.co/datasets/argilla/databricks-dolly-15k-curated-en/tree/main/data
Reporter: Rémy Léone / @remyleone
Original Issue Attachments:

image-2023-11-16-21-40-07-628.png

Note: This issue was originally created as PARQUET-2378. Please see the migration documentation for further details.

Gang Wu / @wgtmac:
Thanks for reporting the issue! I can reproduce it on my end. Let me investigate it.

Jiashen Zhang / @zhangjiashen:
This Cat supports multiple file formats - Parquet, Avro, Text, it internally needs to converts MessageType to Avro Schema if it is a parquet format, which doesn't support '-' . What do you think we add a new command to cat parquet file format without converting to Avro Schema and we can directly print content given a parquet file? Below is some code sample:

  String input = <parquet file>;

  ParquetReader<SimpleRecord> reader = null;
  try {
    PrintWriter writer = new PrintWriter(Main.out, true);
    reader = ParquetReader.builder(new SimpleReadSupport(), new Path(input)).build();
    ParquetMetadata metadata = ParquetFileReader.readFooter(new Configuration(), new Path(input));
    JsonRecordFormatter.JsonGroupFormatter formatter = JsonRecordFormatter.fromSchema(metadata.getFileMetaData().getSchema());

    for (SimpleRecord value = reader.read(); value != null; value = reader.read()) {
      value.prettyPrint(writer);
      writer.println();
    }
  } finally {
    if (reader != null) {
      try {
        reader.close();
      } catch (Exception ex) {
      }
    }
  }
}

Output sample:

.......

id = 15012
category = open_qa
original-instruction = What is the difference between a road bike and a mountain bike?
original-context =
original-response = Road bikes are built to be ridden on asphalt and cement surfaces and have thin tires, whereas mountain bikes are built to be ridden on dirt and have wider tires. Road bikes also have more aerodynamic handle bars while mountain bike handle bars a built for less responsive steering while bouncing around off the road.
new-instruction:
.user_id:
..list:
.value:
..list:
...item = What is the difference between a road bike and a mountain bike?
.status:
..list:
...item = submitted
new-context:
.user_id:
..list:
.value:
..list:
...item =
.status:
..list:
...item = submitted
new-response:
.user_id:
..list:
.value:
..list:
...item = Road bikes are built to be ridden on asphalt and cement surfaces and have thin tires, whereas mountain bikes are built to be ridden on dirt and have wider tires. Road bikes also have more aerodynamic handle bars while mountain bike handle bars a built for less responsive steering while bouncing around off the road.
.status:
..list:
...item = submitted

id = 15013
category = general_qa
original-instruction = How does GIS help in the real estate investment industry?
original-context =
original-response = Real estate investors depend on precise, accurate location intelligence for competitive insights about the markets and locations where they do business. Real estate investment teams use GIS to bring together location-specific data, mapping, and visualization technology. This enables them to provide the latest insights about real estate markets and their investments, now and in the future. Using thousands of global datasets, investors can quickly understand how their real estate investments are performing across town or around the world, quickly access precise local data about real estate assets, on any device, anywhere, anytime, including information on occupancy, building maintenance, property valuation, and more.Real estate companies and investors use GIS to research markets, identify new opportunities for growth and expansion, and manage their investments at the market and neighborhood levels. They can also use GIS to create professional digital and printed materials—such as 3D renderings and virtual walk-throughs—to help market investments across platforms. Real estate investors can use mobile data collection tools to gather property information directly from the field and analyze and share insights across their organizations in real time. Investors can leverage precise local knowledge about their assets across geographies. GIS maps and dashboards help investors see, in real-time, relevant data that can affect properties, and streamline investment management with access to all relevant data about every asset in any portfolio.
new-instruction:
.user_id:
..list:
.value:
..list:
...item = How does GIS help in the real estate investment industry?
.status:
..list:
...item = submitted
new-context:
.user_id:
..list:
.value:
..list:
...item =
.status:
..list:
...item = submitted
new-response:
.user_id:
..list:
.value:
..list:
...item = Real estate investors depend on precise, accurate location intelligence for competitive insights about the markets and locations where they do business. Real estate investment teams use GIS to bring together location-specific data, mapping, and visualization technology. This enables them to provide the latest insights about real estate markets and their investments, now and in the future. Using thousands of global datasets, investors can quickly understand how their real estate investments are performing across town or around the world, quickly access precise local data about real estate assets, on any device, anywhere, anytime, including information on occupancy, building maintenance, property valuation, and more.Real estate companies and investors use GIS to research markets, identify new opportunities for growth and expansion, and manage their investments at the market and neighborhood levels. They can also use GIS to create professional digital and printed materials—such as 3D renderings and virtual walk-throughs—to help market investments across platforms. Real estate investors can use mobile data collection tools to gather property information directly from the field and analyze and share insights across their organizations in real time. Investors can leverage precise local knowledge about their assets across geographies. GIS maps and dashboards help investors see, in real-time, relevant data that can affect properties, and streamline investment management with access to all relevant data about every asset in any portfolio.
.status:
..list:
...item = submitted

id = 15014
category = general_qa
original-instruction = What is the Masters?
original-context =
original-response = The Masters Tournament is a golf tournament held annually in the first week of April at Augusta National Golf Club in Augusta, Georgia.  The Masters is one of four Major golf tournaments and the only one to be played at the same course every year.  The course is renowned for its iconic holes, impeccable groundskeeping, and colorful flowers that are typically in bloom.  The winner earns a coveted Green Jacket and a lifetime invitation back to compete.  Many players and fans consider The Masters to be their favorite tournament given these traditions and the historical moments that have occurred in past tournaments.
new-instruction:
.user_id:
..list:
.value:
..list:
...item = What is the Masters?
.status:
..list:
...item = submitted
new-context:
.user_id:
..list:
.value:
..list:
...item =
.status:
..list:
...item = submitted
new-response:
.user_id:
..list:
.value:
..list:
...item = The Masters Tournament is a golf tournament held annually in the first week of April at Augusta National Golf Club in Augusta, Georgia.  The Masters is one of four Major golf tournaments and the only one to be played at the same course every year.  The course is renowned for its iconic holes, impeccable groundskeeping, and colorful flowers that are typically in bloom.  The winner earns a coveted Green Jacket and a lifetime invitation back to compete.  Many players and fans consider The Masters to be their favorite tournament given these traditions and the historical moments that have occurred in past tournaments.
.status:
..list:
...item = submitted

Or Print with Json format, such as:

{"id":"15012","category":"open_qa","original-instruction":"What is the difference between a road bike and a mountain bike?","original-context":"","original-response":"Road bikes are built to be ridden on asphalt and cement surfaces and have thin tires, whereas mountain bikes are built to be ridden on dirt and have wider tires. Road bikes also have more aerodynamic handle bars while mountain bike handle bars a built for less responsive steering while bouncing around off the road.","new-instruction":{"user_id":{"list":[{}]},"value":{"list":[{"item":"What is the difference between a road bike and a mountain bike?"}]},"status":{"list":[{"item":"submitted"}]}},"new-context":{"user_id":{"list":[{}]},"value":{"list":[{"item":""}]},"status":{"list":[{"item":"submitted"}]}},"new-response":{"user_id":{"list":[{}]},"value":{"list":[{"item":"Road bikes are built to be ridden on asphalt and cement surfaces and have thin tires, whereas mountain bikes are built to be ridden on dirt and have wider tires. Road bikes also have more aerodynamic handle bars while mountain bike handle bars a built for less responsive steering while bouncing around off the road."}]},"status":{"list":[{"item":"submitted"}]}}}
{"id":"15013","category":"general_qa","original-instruction":"How does GIS help in the real estate investment industry?","original-context":"","original-response":"Real estate investors depend on precise, accurate location intelligence for competitive insights about the markets and locations where they do business. Real estate investment teams use GIS to bring together location-specific data, mapping, and visualization technology. This enables them to provide the latest insights about real estate markets and their investments, now and in the future. Using thousands of global datasets, investors can quickly understand how their real estate investments are performing across town or around the world, quickly access precise local data about real estate assets, on any device, anywhere, anytime, including information on occupancy, building maintenance, property valuation, and more.\n\nReal estate companies and investors use GIS to research markets, identify new opportunities for growth and expansion, and manage their investments at the market and neighborhood levels. They can also use GIS to create professional digital and printed materials—such as 3D renderings and virtual walk-throughs—to help market investments across platforms. Real estate investors can use mobile data collection tools to gather property information directly from the field and analyze and share insights across their organizations in real time. Investors can leverage precise local knowledge about their assets across geographies. GIS maps and dashboards help investors see, in real-time, relevant data that can affect properties, and streamline investment management with access to all relevant data about every asset in any portfolio.","new-instruction":{"user_id":{"list":[{}]},"value":{"list":[{"item":"How does GIS help in the real estate investment industry?"}]},"status":{"list":[{"item":"submitted"}]}},"new-context":{"user_id":{"list":[{}]},"value":{"list":[{"item":""}]},"status":{"list":[{"item":"submitted"}]}},"new-response":{"user_id":{"list":[{}]},"value":{"list":[{"item":"Real estate investors depend on precise, accurate location intelligence for competitive insights about the markets and locations where they do business. Real estate investment teams use GIS to bring together location-specific data, mapping, and visualization technology. This enables them to provide the latest insights about real estate markets and their investments, now and in the future. Using thousands of global datasets, investors can quickly understand how their real estate investments are performing across town or around the world, quickly access precise local data about real estate assets, on any device, anywhere, anytime, including information on occupancy, building maintenance, property valuation, and more.\n\nReal estate companies and investors use GIS to research markets, identify new opportunities for growth and expansion, and manage their investments at the market and neighborhood levels. They can also use GIS to create professional digital and printed materials—such as 3D renderings and virtual walk-throughs—to help market investments across platforms. Real estate investors can use mobile data collection tools to gather property information directly from the field and analyze and share insights across their organizations in real time. Investors can leverage precise local knowledge about their assets across geographies. GIS maps and dashboards help investors see, in real-time, relevant data that can affect properties, and streamline investment management with access to all relevant data about every asset in any portfolio."}]},"status":{"list":[{"item":"submitted"}]}}}
{"id":"15014","category":"general_qa","original-instruction":"What is the Masters?","original-context":"","original-response":"The Masters Tournament is a golf tournament held annually in the first week of April at Augusta National Golf Club in Augusta, Georgia.  The Masters is one of four Major golf tournaments and the only one to be played at the same course every year.  The course is renowned for its iconic holes, impeccable groundskeeping, and colorful flowers that are typically in bloom.  The winner earns a coveted Green Jacket and a lifetime invitation back to compete.  Many players and fans consider The Masters to be their favorite tournament given these traditions and the historical moments that have occurred in past tournaments.","new-instruction":{"user_id":{"list":[{}]},"value":{"list":[{"item":"What is the Masters?"}]},"status":{"list":[{"item":"submitted"}]}},"new-context":{"user_id":{"list":[{}]},"value":{"list":[{"item":""}]},"status":{"list":[{"item":"submitted"}]}},"new-response":{"user_id":{"list":[{}]},"value":{"list":[{"item":"The Masters Tournament is a golf tournament held annually in the first week of April at Augusta National Golf Club in Augusta, Georgia.  The Masters is one of four Major golf tournaments and the only one to be played at the same course every year.  The course is renowned for its iconic holes, impeccable groundskeeping, and colorful flowers that are typically in bloom.  The winner earns a coveted Green Jacket and a lifetime invitation back to compete.  Many players and fans consider The Masters to be their favorite tournament given these traditions and the historical moments that have occurred in past tournaments."}]},"status":{"list":[{"item":"submitted"}]}}}

Gang Wu / @wgtmac:
Can we get rid of the schema conversion via AvroSchemaConverter? This file is created by the C++ parquet writer from Apache Arrow. So it does not have to do the conversion.

Jiashen Zhang / @zhangjiashen:

This error is expected because some names using '-' in schema are invalid, it is thrown from validateName in https://github.com/apache/avro/blob/branch-1.11/lang/java/avro/src/main/java/org/apache/avro/Schema.java, please double check?

Gang Wu / @wgtmac:
Sorry for the late reply. I'm not sure if it is a good idea to add a new command. Since both head and cat commands have the same issue, can we try/catch the exception and use the JsonRecordFormatter you proposed as a fallback solution if the avro schema conversion fails?