Possibility of a World::retain() method?

Hi, first off, just want to say I love this library, it's absolutely fantastic and exactly what I've been looking for. Also the documentation is amazing, so thank you so much for putting the effort in there.
I think a retain(..) interface would be incredibly handy to have, I find myself writing the following boilerplate quite a lot and there are definitely optimisations that could be made under the hood:
let mut entities_to_remove = vec![];
for result!(identifier, data) in world.query(query).iter {
	if data.health < 0.0 {
		entities_to_remove.push(identifier);
	}
}
for identifier in entities_to_remove {
	world.remove(identifier);
}

Also, do you take pull requests? I might take a shot at implementing the above myself if you think it wouldn't be too difficult of a change to make.

Thanks for the kind words, I'm glad to hear you're enjoying the library :) yes, I think World::retain() is something that would fit well into this library. I think a method that looks similar to Vec::retain() would work well. Probably something that looks like retain(query: Query, f: F) where F: FnMut(Views) -> bool (more or less, there would definitely need to be more generics stuck in there) would work.
I definitely welcome pull requests! I'm not sure the difficulty of this change, as I haven't had a chance to look at it closely yet, but feel free to give it a go. I'd recommend looking at how World::query() is implemented and trying something similar. I can look at this a bit closer later and try to give a bit more guidance if needed.
If you do end up forking and submitting a PR, please work off of the dev branch to avoid merge conflicts :)

I gave implementing it a shot, but got quickly bogged down in all the generics - this is the most generic-dense code I have ever seen 😄
Special shout-out to ContainsViewsOuter in sealed.rs which has 77 lines straight of code in a where clause.
The main thing that held me up though was I couldn't figure out how to get the entity identifier from the query - the end user may or may not already have an entity::Identifier in their query so I can't just add one to the end.
Perhaps I'll start with one of the easier issues, like an example or two 😄

No worries! I have tried my best to keep the public API as clean as I can, but I will be the first to admit that the stuff under the hood can get a bit messy, especially when it comes to traits. I sometimes get confused myself when looking at it, and I wrote the dang thing lol. I am making efforts to clean up some of the code, but it's dependent on having enough time to work on it. And yes, ContainsViewsOuter in src/registry/contains/views/sealed.rs is one of the harder-to-parse examples. Some of the scheduler logic is actually worse in that regard, but I am working to fix it along with my in-progress fix for #222.
As far as the entity::Identifier thing, that's actually the entire purpose of having separate ContainsViewsOuter and ContainsViewsInner traits. ContainsViewsOuter is implemented for two distinct cases: impl<'a, I, IS, P, V, R, Q> ContainsViewsOuter<'a, V, (Contained, P), (I, IS), Q>     for (EntityIdentifierMarker, R) and impl<'a, I, P, R, V, Q> ContainsViewsOuter<'a, V, (NotContained, P), I, Q>     for (EntityIdentifierMarker, R). The single letter generic names are awful to read here, but basically they're both implemented for (EntityIdentifierMarker, R), where R is the registry of components, and the only difference is the first one is implemented when the entity identifier is Contained, and the second one is implemented when it is NotContained. The definition adds EntityIdentifierMarker onto the registry list, because an entity::Identifier is a special-cased view that doesn't actually view a component in the registry. In the "canonical" form of a set of views, the entity::Identifier always comes at the beginning of the list, so this basically handles pulling out the entity::Identifier from the views before letting ContainsViewsInner handle the canonicalizing of the rest of the component views.
That probably sounds a bit confusing, which is understandable. From a high level, a query is basically handled as follows:

Filter only the archetypes that are actually viewed by the query (https://github.com/Anders429/brood/blob/dev/src/query/result/iter.rs#L109-L118)
Call archetype.view() to actually iterate over the viewed components (https://github.com/Anders429/brood/blob/dev/src/query/result/iter.rs#L124-L130). Note that these components are stored in canonical order within the archetype.
The archetype defers to ContainsViews to actually view the components (https://github.com/Anders429/brood/blob/dev/src/archetype/mod.rs#L260-L267). ContainsViews allows converting the Views into their canonical form, so they can actually be viewed.
ContainsViews uses the sealed trait pattern, so the method is actually defined on the Sealed trait. From there, Sealed defers to ContainsViewsOuter::view(). Like I mentioned before, this is implemented in two cases: the entity::Identifier being present, or not. When present, the identifiers are iterated over. When not provided, we don't iterate over them.
ContainsViewsInner is used to just define the canonical shape of the remaining components. The where clauses specify that the registry should implement CanonicalViews with respect to this canonical set of views. When we view the rest of the components (https://github.com/Anders429/brood/blob/dev/src/registry/contains/views/sealed.rs#L262), it is calling into CanonicalViews::view(), which is where the iterators over the components are actually created.
Finally, since view is now in "canonical order", it is reshaped back to the order specified by the user.

The whole point is to allow users to specify components in any order they wish (similar logic exists for inserting components and other cases, but querying is far more complicated). This is how you can switch the order of components in queries and it doesn't break anything. This is the full power of heterogeneous lists :) if you want to see similar generic-heavy code in the Rust ecosystem, the frunk library's hlist module is full of this kind of stuff, though this is a much more specific application.
I can definitely take a look at more of the specifics of implementing retain(), but it will be a few weeks at least. My free time is limited, and I'm first working to resolve #222. The deletion aspect of this method might actually get a bit complicated, since we will actually need to delete all components in the entity, not just the ones used in the function f. We'll also need to make sure the entity::Allocator is aware of any changes to locations that are made as a result, but that part should be fairly straightforward :)

Thanks for the in-depth response, that's really very helpful, I might go give it another bash with all this new knowledge in mind and see what I can come up with. For the deletion, I was planning just to defer to self.remove(..), at least initially for a POC, as performance-wise the entities to remove will be likely scattered all over the archetypes so there's possibly less to gain than I'd first hoped by treating it as a batch (although we can probably avoid the overhead of maintaining a separate entity::Identifier vec at least).
I was doing some fiddling with these heterogeneous entity views and with a bit of magic you can define a get_component() trait and put methods on them. I'm not quite sure of how useful it is yet, but it's certainly a cool thing you can't do in any other ECS system I've ever seen.
struct A1;
struct A2;

trait GetComponent<T, A> {
	fn get_component(&mut self) -> &mut A;
}

impl<T> GetComponent<A1, T> for T {
	fn get_component(&mut self) -> &mut T {
		self
	}
}

impl<T, Q, A: GetComponent<Q, T>, B> GetComponent<(A1, Q), T> for (A, B) {
	fn get_component(&mut self) -> &mut T {
		self.0.get_component()
	}
}

impl<T, Q, A, B: GetComponent<Q, T>> GetComponent<(A2, Q), T> for (A, B) {
	fn get_component(&mut self) -> &mut T {
		self.1.get_component()
	}
}

(I'm using the generic Q here as a discriminatory to prevent my impl definitions from overlapping)
Which can then be used as follows, for example, to define get_aabb() on all entity views that have both a Size and a Pos component:
trait GetAABB<T> {
	fn get_aabb(&mut self) -> AABB;
}

impl<Q, T: GetComponent<Q, Size> + GetComponent<Q, Pos>> GetAABB<(A1, Q)> for T {
	fn get_aabb(&mut self) -> AABB {
		let pos: Pos = *self.get_component();
		let size: Size = *self.get_component();

		AABB {
			min: pos - size / 2.0,
			max: pos + size / 2.0,
		}
	}
}


Support rendering Table elements

First, thanks for this small and nice renderer and the description in the README and provided examples!
After parsing a document with table extension, like this:
gm := goldmark.New(goldmark.WithExtensions(extension.Table))
...the resulting document is a "little bit destroyed", this seems to be caused by the missing "KindTable" for the renderer.
Is there a plan to implement it or would this be possible by a custom extension, e.g. with this PR #16 ?

maybe this would help with the implementation:
https://github.com/Kunde21/markdownfmt/blob/master/markdown/renderer_table.go

Hi @gen2thomas thanks for the suggestions and glad you found the repo to be nice!
I'll look into adding the missing test coverage for #16 as well as adding support for the table extension directly. I was planning on adding support for all extensions in GitHub Flavored Markdown but got sidetracked with other projects.

#16 has been merged. One step closer!


Add missing const modifier

Missed the const modifier last time. Can we get this in quickly?

Coverage remained the same at 100.0% when pulling 692ccf046c5986f785d4142dc3a471317bdb2eb2 on kamilritz:const into 674bd99f3b9284e1088d100d765b3a09cf0c192f on PX4:master.


Added ability to use package.json in different dir

Added a parameter which allows to define the directory of the package.json if it's not in root.
The system will work by copying such package.json to the root of the cloned repository inside the action.

@ScottBrenner it seems that the Lint Code Base problem is failing for problems which I didn't cause and where there before. Should I change something there?

Cool! Thanks for putting this together. Do you by chance have a sample workflow run that demonstrates this functionality?

Do you by chance have a sample workflow run that demonstrates this functionality?

I did this same modification in the past https://github.com/pascalgn/npm-publish-action/pull/15 for getting a custom directory for but it was in JS.
In the case of Shell I used $1 as in the Github official tutorial they are using it too for the only input that is being set up.


Adapt Dockerfile to use JAVA_OPTIONS

Describe the bug
Currently no JAVA_OPTIONS are used during startup.
Adapt Dockerfile like the following one.
FROM registry.access.redhat.com/ubi8/openjdk-11:latest

EXPOSE 8080

COPY target/*.jar /deployments/spring-boot-application.jar

CMD "java" ${JAVA_OPTIONS} "-jar" "/deployments/spring-boot-application.jar"

Additionally i suggest to use registry.access.redhat.com/ubi8/openjdk-11:latest as the base image.
The same image is used within the default lhm pipelines.
Furthermore the image is tested and regular updated by red hat.

Is it ok to use JAVA_TOOL_OPTIONS (openjdk) or should we add JAVA_OPTIONS to every service?

I suggest to use both options.
The JAVA_TOOL_OPTIONS  are pulled automatically.
The JAVA_OPTIONS are part of the CMD to start the application.
If the JAVA_OPTIONS  are not set, the application can start either.

Already implemented


Works with Regular Mailer, not with custom

Hello! I'm using maildown and premailer-rails, I have the normal rails mailer (app/mailers/mailer.rb) and I have a custom mailer that inherits from that mailer notification_mailer.rb (app/mailers/notification_mailer.rb) class NotificationMailer < Mailer
All the views are in the app/views/mailer directory and the layout is rendering fine for each, just no html is parsed into inline css for any emails coming from notification_mailer.rb, only mailer.rb.
Do I have to enable it differently for a custom mailer file?

All emails are processed. Are you sure the CSS is linked in the email?
You can try running the hook manually to see if the parsing works. From the README:

Emails are only processed upon delivery, i.e. when calling #deliver on the email, or when previewing them in rails. If you wish to manually trigger the inlining, you can do so by calling the hook:
mail = SomeMailer.some_message(args)
Premailer::Rails::Hook.perform(mail)

When I look at the emails in mailhog, I see that the mails sent from mailer.rb have no stylesheet link in the rendered html of the email and all styles are inlined, but with notification_mailer.rb, the stylesheet
link is still in the source of the email and no inline styles have been applied.

Can you please try the snippet I posted above and inspect the body of the mail afterwards? If it's still not inlined it means that something is off with your stylesheet link.


Network Device in Cross-Node Communication Profiling

Thank you for the helpful profiling data for the simulation. The "send_recv" metric for pipeline parallelism indicates cross-node communication, but I fail to find documentation on the specific network device used for this. Knowing whether the communication happens over Ethernet, InfiniBand, or another device is important for accurately understanding the simulation results, as different devices can significantly impact latency and throughput.
Could you please provide details about the network device used for cross-node communication?

Hello Rongzhi,
The network type depends on the cluster type. Here is the mapping:
A100 80GB DGX: 8x200Gbps Infiniband
H100 80GB DGX: 8x400Gbps Infiniband
4xA100 80GB Pairwise NVLink Node: 1x100Gbps Ethernet
8xA40 Pairwise NVLink Node: 1x100Gbps Infiniband
Thanks!


crtime: add MonoFromTime method

This is useful when we have a time.Time that we can't change to use
Mono, for example ctx.Deadline().

This change is

TFTR!


i18n: Harvesting client-side strings

This is part of the i18n project described in #5345. I'd love to see some progress on this :smile:
Harvesting server side strings is already in progress (see issue #5617), but we also need to start isolating strings inside the Ember app. This should be done using the ember-intl addon.
This is a big job, as we have lots of templates and .js files that have English strings which need to be moved to a translations file and hooked up with ember-intl.
So, the first thing to do is to setup the Ghost Ember app to work with ember-intl. I believe we should use the 2.0 branch of the project (as @morficus showed in his proof of concept). Once this is done, we can start working on translating individual files.
I've checked which files contain strings needing translating and this is what I found:

[ ] /core/client/app/assets/lib/uploader.js
[ ] /core/client/app/components

[ ] gh-search-input.js
[ ] gh-upload-modal.js
[ ] gh-user-invited.js

[ ] /core/client/app/controllers/modals (multiple files)
[ ] /core/client/app/controllers/settings/labs.js
[ ] /core/client/app/controllers/setup (multiple files)
[ ] /core/client/app/controllers/team/user.js
[ ] /core/client/app/controllers/error.js
[ ] /core/client/app/controllers/post-settings-menu.js
[ ] /core/client/app/controllers/signin.js
[ ] /core/client/app/controllers/signup.js
[ ] /core/client/app/html (multiple files)
[ ] /core/client/app/mixins

[ ] editor-base-controller.js
[ ] pagination-route.js

[ ] /core/client/app/models/user.js
[ ] /core/client/app/routes (most routes have a titleToken that should be translated + some have alerts/other strings)
[ ] /core/client/app/services/notifications.js
[ ] /core/client/app/templates (pretty much all template files have English strings that need to be isolated)
[ ] /core/client/app/utils

[ ] ajax.js
[ ] titleize.js (how would we handle this util in regards to i18n?)

[ ] /core/client/app/validators (multiple files)

As you can see, there's a lot to do here. If you're willing to work on this, feel free to leave a comment stating the files you're translating.
If there's anything I missed - please let me know!
To the Ghost Core team - if this is not the right time for translating client-side strings, then we could close this issue for now and open it up again later.

Thanks @kevinkucharczyk! I'm definitely up for starting work on this sooner rather than later as we're only going to gather more strings to translate as time goes by.
As you say the first step is to get ember-intl set up - this is something I'd like to do once #5970, #6020 and a forthcoming ember-cli upgrade are merged and confirmed working. There is also on-going work on the client-side acceptance tests (#6039) which I'd like to get in place before we start moving too far with i18n as they should provide more confidence that we're not breaking anything as we go.
Once ember-intl has been properly set up I'm fine with pulling out strings and switching to i18n helpers incrementally. I think we'll also want to come up with a method of testing our internationalisation - if you or anyone has any experience or ideas surrounding that I'd definitely be interested in hearing them.
Finally, in order to add full internationalisation on the client we also need to consider #6050 and how we want to handle translation of API-provided messages.
In conclusion, there's a lot to be done but nothing which can't be done incrementally, especially if it's taken into consideration when other changes are made.

#6050 is an interesting one. I kinda like the idea of providing an i18n key to the client, which would look up its own translations.
However, we are also translating the API messages themselves (in #5617), so I'm worried that this might lead to duplicate strings - they'd exist in both the client and server translation jsons and we would always need to be sure to update both files whenever we change one of the strings.

Yeah, it's an interesting one - my initial thoughts on #6050 were that the API should be language-agnostic otherwise for every API request we also have to pass which language we expect back. However, if that is the case we would then expect any other clients of the API to use the default english response, provide their own translations, or offer some way for clients to retrieve language packs from the API that they could use for local translations.

We could add an Accept-Language header to all requests by modifying the Adapter in Ember. The server could use that to determine which translation it should send back and fall back to english if no header is provided.

I started this job.
This is my first contribution so fell free to warn me if I'm doing it wrong ;).
https://github.com/matitalatina/Ghost/tree/i18n-client

i'd be glad to help on that, just point me in the right direction @kevinkucharczyk

@qLb I think we're still waiting for the acceptance tests (#6039) to be finished before moving on with the harvesting. Perhaps you could help out with the tests? :smiley:

                                                                                ember is not my pony to be honest but sure, i'll do my best to push it further :) pzdr                                                                                qL.b                                                                                From: Kevin P. KucharczykSent: środa, 27 stycznia 2016 18:56To: TryGhost/GhostReply To: TryGhost/GhostCc: qLbSubject: Re: [Ghost] i18n: Harvesting client-side strings (#6075)@qLb I think we're still waiting for the acceptance tests (#6039) to be finished before moving on with the harvesting. Perhaps you could help out with the tests?

—Reply to this email directly or view it on GitHub.

The list of files requiring translation is out of date and the proposed addon for handling i18n also needs review - it looks like ember-i18n is now preferred option.
I'm going to close for now and mark as later to clean up and focus our issue list.


Move field_name from runtime to schema generation time

Pydantic should have this information when it is building the schema so this removes a bit of branching that happens at runtime, including a runtime error if the schema is misconfigured and a field from the poor catch-all Extra struct.
If we take this one step further we could replace field_name with field_info which could be an arbitrary generic type, giving us more flexibility on the Pydantic side to e.g. make it a FieldInfo instead of a string which you then need to look up in the model class. This change in particular would allow us to pass FieldInfo into Annotated validators if we wanted to.
Selected Reviewer: @dmontagu

Just an idea I got from discussion in #714 . We could move the whole FieldInfo here or at least the field_name. I don't see any reason why we can't supply that to the schema in pydantic, and it would save us the runtime work of copying field_name around in `Extra, some branching, etc.

please review

I measured https://github.com/pydantic/pydantic-core/pull/726 locally and am getting 22.09±0.5us on this branch vs. 26.17±0.362 on main.

Benchmarks are now reflecting a 13.5% speedup :)

@samuelcolvin Adrian and I discussed, and I think Adrian was going to try to implement the pydantic side before merging this to make sure there were no surprises, but we also want to make sure you don't have objections before merging.
(As far as I'm concerned this is good, but figure there's a chance I'm not considering something important.)

I created https://github.com/pydantic/pydantic/pull/6392, it looks pretty good I think. If that looks good to you David lets merge this and I can update that PR once we have a new core release


Use latest versions in cmctl install instructions

It looks like in some of the past releases we've forgotten to update the cmctl version in the installation instructions.
I've chosen to use latest for the current docs, since  latest is now configurable:

https://github.blog/changelog/2022-10-21-explicitly-set-the-latest-release/

I also added -f to the curl command so that people get early warning if the URL is broken and curl returns 404.

/lgtm


fixed the split screen issue by closing sidebar and courseViewToolbar…

… when screenwidth is less than 900
PR Fixes:

1 Responsiveness issue when we view the webpage in a split screen was resolved using resize event listener in useEffect hook

Resolves #1094
Checklist before requesting a review

[x] I have performed a self-review of my code
[x] I assure there is no similar/duplicate pull request regarding same issue

Before:

After:

Closing in the favour of keshav's pr


Report separate running time for RSA key generation and scanning

Even though they happen in parallel, it's interesting to know how long
each of them took separately

/test baremetalds-sno-recert-cluster-rename
/test e2e-aws-ovn-single-node-recert-parallel
/test e2e-aws-ovn-single-node-recert-serial

e2e-aws-ovn-single-node-recert-serial  OK


Allow reading block-size of Rasters

This PR allows efficient streaming of raster by reading (using already existing read_as) along block boundaries.  To get block size, call block_size on a RasterBand.
Additionally, we also add feature flag gdal_2_2 that enables the corresponding feature in gdal-sys crate to pull in features from GDAL >= 2.2.  Using this, we provide actual_block_size function behind this feature gate to be used if required.
List of what is implemented:

implemented GetBlockSize
added feature flag "gdal_2_2" to enable recent GDAL functions.
implemented GetActualBlockSize (behind "gdal_2_2" flag)

Thanks for your contribution, @rmanoka! Code looks good to me. Would you mind adding some tests for the new functions and resolve the conflicting files?

@pka Sure, I'll add tests and resolve Cargo.toml conflict over the weekend.
There are a couple of minor issue with #68 that has been merged recently:

The PR introduces block_size in RasterBand, similar to this PR.  The code however is identical, so I will fix it in when updating this PR.

However, read_block function added by the PR seems to have a bug.  Suppose the raster dimensions are not perfectly divisible by the block-size, then the right, and bottom edge blocks will not have the size as provided by block_size (as it will be smaller).   This was the reason why we had to include actual_block_size in this PR.  To the best of my understanding, the read_block function would error when reading these edge blocks.  The fix is to use actual_block_size in the beginning of read_block implementation;  however, the two functions are under different feature gates, so I don't know how to proceed here.

Any ideas?

Actually, the GDAL docs say it already yields an array of block-size dimensions; so the user only needs to use actual_block_size to correctly use the output from read_block.  Seems we should be in good shape with present implementation.  Sorry about the confusion.

Ping @pka?

cargo doc fails on nightly. Will hopefully work again in a few days?

Works for me. Let's wait for the master build to finish.


fix(socket): err message checking

While debugging an application. I came over the .message property does not actually exist. The error code itself is the root err thus the If statements are never actually working.
Not sure how this one was missed, or if something changed elsewhere.

Due to refactors this is no longer valid.


[Python Docs]: i want to add File Handling section in python

Is there an existing issue for this?

[X] I have searched the existing issues

Feature Description
i want to file handling section in python
i hope this finds helpful

Use Case
use know more about python file handling
Benefits
use know more about python file handling
Add ScreenShots
No response
Priority
High
Record

[X] I have read the Contributing Guidelines
[X] I'm a GSSOC'24 contributor
[X] I have starred the repository

hey @Ajay-Dhangar , please look over my issues i have stated collecting info from resources please assign it so that i can start work on this


[Bug]: Compilation fails with "error: invalid organization name" although orgname is valid

Description
I am trying to build this project workspace in the order of fares->inventory->booking->reservation (based on dependencies)
Each of the dependent packages have been packed and pushed to local central using bal push --repository=local
While the compilation and pushing of the first three modules are successful, the compilation of the reservation module fails with the following error (attaching the full log):
Compiling source         SumayyaTestOrg/ReservationAPI:1.0.0 Error while pulling package [SumayyaTestOrg/FaresAPI:1.0.0]: error: invalid organization name. Organization name can only contain alphanumeric values with lower case letters and cannot start with numeric valuesorg_name = SumayyaTestOrg ballerina/observe:1.0.7 [central.ballerina.io -> home repo]  100% [===========================================================] 206/206 KB (0:00:00 / 0:00:00)          ballerina/observe:1.0.7 pulled from central successfully ERROR [SumayyaTestOrg/InventoryAPI/1.0.0::service.bal:(3:1,3:45)] cannot resolve module 'SumayyaTestOrg/FaresAPI as fares_api' ERROR [SumayyaTestOrg/InventoryAPI/1.0.0::service.bal:(20:5,20:19)] undefined module 'fares_api' ERROR [SumayyaTestOrg/InventoryAPI/1.0.0::service.bal:(20:5,20:19)] unknown type 'Fare' ERROR [SumayyaTestOrg/BookingAPI/1.0.0::service.bal:(1:1,1:45)] cannot resolve module 'SumayyaTestOrg/FaresAPI as fares_api' ERROR [SumayyaTestOrg/BookingAPI/1.0.0::service.bal:(19:6,19:20)] undefined module 'fares_api' ERROR [SumayyaTestOrg/BookingAPI/1.0.0::service.bal:(19:6,19:20)] unknown type 'Fare' ERROR [SumayyaTestOrg/BookingAPI/1.0.0::service.bal:(38:5,38:19)] undefined module 'fares_api' ERROR [SumayyaTestOrg/BookingAPI/1.0.0::service.bal:(38:5,38:19)] unknown type 'Fare' ERROR [SumayyaTestOrg/BookingAPI/1.0.0::service.bal:(78:9,78:23)] undefined module 'fares_api' ERROR [SumayyaTestOrg/BookingAPI/1.0.0::service.bal:(78:9,78:23)] unknown type 'Fare' ERROR [SumayyaTestOrg/BookingAPI/1.0.0::service.bal:(78:37,78:109)] incompatible type for parameter 'targetType' with inferred typedesc value: expected 'typedesc<(ballerina/http:2.6.0:Response|anydata)>', found 'typedesc<other>' ERROR [SumayyaTestOrg/BookingAPI/1.0.0::service.bal:(138:31,138:49)] invalid field access: 'fare' is not a required field in record 'SumayyaTestOrg/BookingAPI:1.0.0:BookingRecord', use member access to access a field that may have been specified as a rest field ERROR [SumayyaTestOrg/BookingAPI/1.0.0::service.bal:(139:37,139:61)] invalid field access: 'flightDate' is not a required field in record 'SumayyaTestOrg/BookingAPI:1.0.0:BookingRecord', use member access to access a field that may have been specified as a rest field ERROR [SumayyaTestOrg/BookingAPI/1.0.0::service.bal:(143:35,143:57)] invalid field access: 'flightNo' is not a required field in record 'SumayyaTestOrg/BookingAPI:1.0.0:BookingRecord', use member access to access a field that may have been specified as a rest field WARNING [ballerinax/twitter/3.1.0::types.bal:(23:5,23:17)] undocumented field 'auth' ERROR [SumayyaTestOrg/InventoryAPI/1.0.0::service.bal:(3:1,3:45)] cannot resolve module 'SumayyaTestOrg/FaresAPI as fares_api' ERROR [SumayyaTestOrg/InventoryAPI/1.0.0::service.bal:(20:5,20:19)] undefined module 'fares_api' ERROR [SumayyaTestOrg/InventoryAPI/1.0.0::service.bal:(20:5,20:19)] unknown type 'Fare' ERROR [SumayyaTestOrg/BookingAPI/1.0.0::service.bal:(1:1,1:45)] cannot resolve module 'SumayyaTestOrg/FaresAPI as fares_api' ERROR [SumayyaTestOrg/BookingAPI/1.0.0::service.bal:(19:6,19:20)] undefined module 'fares_api' ERROR [SumayyaTestOrg/BookingAPI/1.0.0::service.bal:(19:6,19:20)] unknown type 'Fare' ERROR [SumayyaTestOrg/BookingAPI/1.0.0::service.bal:(38:5,38:19)] undefined module 'fares_api' ERROR [SumayyaTestOrg/BookingAPI/1.0.0::service.bal:(38:5,38:19)] unknown type 'Fare' ERROR [SumayyaTestOrg/BookingAPI/1.0.0::service.bal:(78:9,78:23)] undefined module 'fares_api' ERROR [SumayyaTestOrg/BookingAPI/1.0.0::service.bal:(78:9,78:23)] unknown type 'Fare' ERROR [SumayyaTestOrg/BookingAPI/1.0.0::service.bal:(78:37,78:109)] incompatible type for parameter 'targetType' with inferred typedesc value: expected 'typedesc<(ballerina/http:2.6.0:Response|anydata)>', found 'typedesc<other>' ERROR [SumayyaTestOrg/BookingAPI/1.0.0::service.bal:(138:31,138:49)] invalid field access: 'fare' is not a required field in record 'SumayyaTestOrg/BookingAPI:1.0.0:BookingRecord', use member access to access a field that may have been specified as a rest field ERROR [SumayyaTestOrg/BookingAPI/1.0.0::service.bal:(139:37,139:61)] invalid field access: 'flightDate' is not a required field in record 'SumayyaTestOrg/BookingAPI:1.0.0:BookingRecord', use member access to access a field that may have been specified as a rest field ERROR [SumayyaTestOrg/BookingAPI/1.0.0::service.bal:(143:35,143:57)] invalid field access: 'flightNo' is not a required field in record 'SumayyaTestOrg/BookingAPI:1.0.0:BookingRecord', use member access to access a field that may have been specified as a rest field WARNING [ballerinax/twitter/3.1.0::types.bal:(23:5,23:17)] undocumented field 'auth' HINT [service.bal:(39:5,39:5)] concurrent calls will not be made to this method since the service is not an 'isolated' service error: compilation contains errors
However, the organization name is as follows.

Steps to Reproduce

Clone https://github.com/SumayyaZiyad/SampleChoreoProject
Build the packages in the order of fares->inventory->booking->reservation. Between each build, push the required dependent packages to local central

Affected Version(s)
Reproducible on both 2201.3.2 and 2201.4.0 RC2
OS, DB, other environment details and versions
MacOS Ventura
Related area
-> Compilation
Related issue(s) (optional)
No response
Suggested label(s) (optional)
No response
Suggested assignee(s) (optional)
No response

I was able to reproduce the issue locally. The bug is two folds.

When building ReservationAPI, FaresAPI which is already packed and in the local repo fails to be fetched from the local repository. Reasons for this needs to be investigated.
Due to the above failure, when FaresAPI is tried to fetch from the central repository, it fails due to an inconsistency in organization name validation between the CLI & toml validation VS central. The central requires the org name to be entirely lowercase while the CLI commands allow to have uppercase chars.

@SumayyaZiyad Have you added the local dependency to Ballerina.toml?

@SumayyaZiyad Have you added the local dependency to Ballerina.toml?

Yes.

Shall we add the transitive dependency too?

Adding that worked. :)

Based on how local dependencies work, we need all dependencies specifically mentioned to be pulled from the local repository otherwise resolution engine tries to pull from central. :)

Ack, thank you!

Closing this issue since point 2 in https://github.com/ballerina-platform/ballerina-lang/issues/39721#issuecomment-1442854347 is addressed.
The issue of inconsistent org field validation in CLI and central is tracked by https://github.com/ballerina-platform/ballerina-lang/issues/39723.


Process B fails quickly results in "process A depends on B, but it isn't running"

Defect
When process A depends on B completing successfully, process A runs if B fails quickly. It should not run if B fails, regardless how quickly.

[x] Included the relevant configuration snippet
[x] Included the relevant process-compose log (log location: process-compose info)
[x] Included a [Minimal, Complete, and Verifiable example] (https://stackoverflow.com/help/mcve)

Version of process-compose: v1.34.0
OS environment: MacOS 14.7.1
Steps or code to reproduce the issue:
process-compose.json:
{
  "processes": {
    "bundle-install": { "command": "sleep 10", "namespace": "default" },
    "configure": { "command": "diff a b", "namespace": "default" },
    "mysql": {
      "availability": { "max_restarts": 5, "restart": "on_failure" },
      "command": "sleep 100",
      "namespace": "mysql.mysql",
      "readiness_probe": {
        "exec": {
          "command": "echo mysql"
        },
        "failure_threshold": 5,
        "initial_delay_seconds": 2,
        "period_seconds": 10,
        "success_threshold": 1,
        "timeout_seconds": 4
      }
    },
    "mysql-configure": {
      "command": "sleep 3",
      "depends_on": { "mysql": { "condition": "process_healthy" } },
      "namespace": "mysql.mysql"
    },
    "prepare-db": {
      "command": "echo PREPARE DB",
      "depends_on": {
        "bundle-install": { "condition": "process_completed_successfully" },
        "configure": { "condition": "process_completed_successfully" },
        "mysql-configure": { "condition": "process_completed_successfully" }
      },
      "namespace": "default"
    },
    "web": {
      "command": "sleep 100",
      "depends_on": {
        "bundle-install": { "condition": "process_completed_successfully" },
        "configure": { "condition": "process_completed_successfully" },
        "prepare-db": { "condition": "process_completed_successfully" }
      },
      "namespace": "default"
    }
  },
  "shell": { "shell_argument": "-c", "shell_command": "/bin/bash" }
}

Expected result:
24-11-07 16:04:39.275 ERR Error: process prepare-db depended on configure to complete successfully, but it exited with status 2
24-11-07 16:04:39.275 ERR Error: process prepare-db won't run

Actual result:
24-11-07 16:04:49.281 ERR Error: process web depends on configure, but it isn't running
24-11-07 16:04:49.281 ERR Error: process web depends on prepare-db, but it isn't running
24-11-07 16:04:49.286 INF Started command=["/bin/bash","-c","sleep 100"] process=web

Full log:
This run had both the good case and the bad case happening:
24-11-07 16:04:39.260 INF Process Compose v1.34.0
24-11-07 16:04:39.261 INF Loaded project from nix/process-compose-enterprise.json
24-11-07 16:04:39.261 INF Global shell command: /bin/bash -c
24-11-07 16:04:39.261 INF start http server listening :8080
24-11-07 16:04:39.261 DBG Spinning up 6 processes. Order: ["configure" "mysql" "mysql-configure" "bundle-install" "prepare-db" "web"]
24-11-07 16:04:39.261 INF web is waiting for bundle-install to complete successfully
24-11-07 16:04:39.262 INF mysql-configure is waiting for mysql to be healthy
24-11-07 16:04:39.262 INF prepare-db is waiting for configure to complete successfully
24-11-07 16:04:39.264 INF Started command=["/bin/bash","-c","sleep 100"] process=mysql
24-11-07 16:04:39.264 INF Started command=["/bin/bash","-c","sleep 10"] process=bundle-install
24-11-07 16:04:39.265 INF Started command=["/bin/bash","-c","diff a b"] process=configure
24-11-07 16:04:39.275 INF Exited exit_code=2 process=configure
24-11-07 16:04:39.275 ERR Error: process prepare-db depended on configure to complete successfully, but it exited with status 2
24-11-07 16:04:39.275 ERR Error: process prepare-db won't run
24-11-07 16:04:41.264 DBG mysql_ready_probe started monitoring
24-11-07 16:04:41.275 INF Started command=["/bin/bash","-c","sleep 3"] process=mysql-configure
24-11-07 16:04:44.288 INF Exited exit_code=0 process=mysql-configure
24-11-07 16:04:49.281 INF Exited exit_code=0 process=bundle-install
24-11-07 16:04:49.281 ERR Error: process web depends on configure, but it isn't running
24-11-07 16:04:49.281 ERR Error: process web depends on prepare-db, but it isn't running
24-11-07 16:04:49.286 INF Started command=["/bin/bash","-c","sleep 100"] process=web
24-11-07 16:06:19.279 INF Exited exit_code=0 process=mysql
24-11-07 16:06:29.315 INF Exited exit_code=0 process=web
24-11-07 16:06:29.315 INF Project completed
24-11-07 16:06:31.008 DBG Shutting down 0 processes. Order: []
24-11-07 16:06:32.009 DBG Logs monitoring canceled
24-11-07 16:06:32.011 DBG TUI Wait stopped
24-11-07 16:06:32.011 INF Thank you for using process-compose

It looks like waitIfNeeded only looks for the depended on processes in the list of running processes, from which processes are removed once they exit. It should probably check a list of completed processes as well.

Thanks for the quick fix @secobarbital.
Added in v1.40.0


Merging "Fix materialized view with column defaults"

Technical PR to merge #3796

Merged in #3796, #6324.


Incease default hugepage size for v2 data engine to 2GiB

Which issue(s) this PR fixes:

Issue https://github.com/longhorn/longhorn/issues/7606
What this PR does / why we need it:
Special notes for your reviewer:
Additional documentation or context

See if we need to update longhorn/longhorn accordingly, also doc.


Forms not showing

I have created a table as you said in the models with two more fields : country and city
and registered it in admin but not showing in the newsletter subscription form it shows just the email and the action
can you show how to do to a custom form for my additional feilds and thanks

https://github.com/matthiask/django-newsletter-subscription/blob/master/newsletter_subscription/templates/newsletter_subscription/form.html
You have to add your own copy of newsletter_subscription/form.html to your project containing those fields. The default template only shows email and action..

I did, I tried to add a {% form_item form.country %} and it gives me a 404 error
Can you tell me how to put an additional field?

The 404 cannot have anything to do with the new form item. There's something else amiss, and it's really hard to debug this from a distance.

OK thanks I will try my best to fix it

I have enabled the debug and I have end up with this error
'str' object has no attribute 'field'

This happens when you use towel's {% form_item %} template tag with a form field which does not exist on the form itself. (i.e. {% form_item form.field_which_does_not_exist_on_the_form %})


secure boot: update CAAM config on close devices

Docs for b2494228ed71b00df6a644f88b43dbe391185d85 are browesable at: https://ci.foundries.io/projects/fio-docs/builds/401/docs/artifacts/html/index.html

Docs for aa6c028d26ac9636870a987a42b78170351d4b1f are browesable at: https://ci.foundries.io/projects/fio-docs/builds/402/docs/artifacts/html/index.html


add colab notebook & update docs

What does this PR do?

Add a section for Colab notebooks links
Clean up old getting started
[?] not included: how to merge with zero-to-hero?

Test Plan

Sources
Please link relevant resources if necessary.
Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[ ] Ran pre-commit to handle lint / formatting issues.
[ ] Read the contributor guideline,
Pull Request section?
[ ] Updated relevant documentation.
[ ] Wrote necessary unit or integration tests.

I am wondering if we should group Notebooks and Cookbooks under a single section -- something like Tutorials? Could we absorb the eval cookbook into a notebook (along with all the diagrams you had @yanxi0830)? Curious to know your thoughts.

I am wondering if we should group Notebooks and Cookbooks under a single section -- something like Tutorials? Could we absorb the eval cookbook into a notebook (along with all the diagrams you had @yanxi0830)? Curious to know your thoughts.

Good idea! I was thinking we: merge Notebooks + Cookbooks + Building AI Applications into a single Tab called Tutorials?
/tutorials

index.md --> include table of content for the Colab + doc link

Llama Stack Showcase (or Llama Stack - Building Agentic Application)
Llama Stack Evals

evaluations.md --> contents of evals flow + link to "Eval" notebook
agents.md --> contents of Building AI Applications agents + link to "Showcase" notebook


fix: make SpaceBetween with one element Stretch 🐛

When there's just one element, SpaceBetween should do the same thing as Stretch.

I don't think converting SpaceBetween to Stretch is a good idea.
I agree the current behavior (sometimes on the left sometimes on the right) is a bug, but instinctively, setting SpaceBetween with one element I expect it to respect its constraint and stay either left or right. (Looking at this page, it's probably left).

In CSS the default when you use flex is justify to the start. So when there's just one element that's what they do.
For us, our default is Stretch.
Additionally, there's two reasons I think SpaceBetween should be Stretch.

Intuitively it makes sense because we are saying for every element add a spacer between adjacent element but make the start of the first element at the start of the area and make the end of the last element at the end of the area.
This above is exactly is what constraints are added in the SpaceBetween match but we are using tuple_combinations and windows so when there's just one element and no spacers, it doesn't do anything. If I make that code work for one element, it'll do the same thing as Stretch.

In CSS the default when you use flex is justify to the start. So when there's just one element that's what they do.
For us, our default is Stretch.

Alright this convinced me. I see your reasoning.

If we changed our default layout to do Start, I can see the rationale for making this do Start as well.

Can we then make it Flex::default() so it expresses that intent? It'll be StretchLast but this should produce the same output right?

Can we then make it Flex::default() so it expresses that intent? It'll be StretchLast but this should produce the same output right?

I like this idea and almost made the change to do it that way. But I'd like to leave it as Flex::Stretch for now. I want to play around with all the constraint weights in this PR https://github.com/ratatui-org/ratatui/pull/788 because I think that might affect what we choose here as well. If I don't see any issues after messing around with the weights, then we can do that.


Move CDN files to /cdn/.

Fixes #1140

Hey this change shouldn't break anything. Test failures aren't related to
the files move IIRC.
But maybe one of you guys needs to look into the symlink because thats the
only thing that can break.
On Sat, May 26, 2018, 20:42 Joshua Mervine<EMAIL_ADDRESS>wrote:

@jmervine commented on this pull request.
This breaks
https://www.bootstrapcdn.com/bootstrap/4.1.1/css/bootstrap.min.css and
the like. While it's not ideal, we currently have a significant amount of
traffic to the static endpoints.
@jdorfman https://github.com/jdorfman I'm of two minds on this. If we
break these, we're going to get some blow back, but at the same time, we'll
significantly reduce the traffic -- and undesirable traffic at that -- to
www.
I'll leave this to you to decide. I don't mind the general approach
though, so 👍 on that.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/MaxCDN/bootstrapcdn/pull/1146#pullrequestreview-123562819,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAVVtTjroaT-m_dlLcKgxV_jM8nSN__8ks5t2ZQQgaJpZM4UO-XD
.

Yup, the symlink is broken. Since I'm on Windows I need one of you guys to fix this and append the patch to mine.
develop: 21923 passing (38s)
this brandh: 14851 passing (34s)

OK, I think I got it right now. I used my Linux Mint VM.
So, as long as the tests number is 21923 we are all good.

Hmm, now they are even more :P
BTW there are 2 more symlinks in the repo apart from the bootstrap one.

I see your point now @jmervine. I hadn't even thought of that, well, because it shouldn't happen in the first place...

If we have a significant amount of traffic, can't we redirect those? I mean, as far as I can remember, we never exposed these endpoints. So if people used it, it's their fault, but I agree we should handle this gracefully.

@jdorfman That's a tough call. About 2/3 of our traffic is to cdn assets, but it's only about 100 per minute, so not a ton. Additionally, a lot of them are from the same folks, best I can tell.
I'd lean towards merging this and breaking for those few, but I'm 🤷♂

This needs resolving the conflicts, but if I do it on Windows, the symlinks will break.
Can one of you guys do it for me?

OK, I did the rebase on my Linux Mint VM.


Add i18n for most UI strings in AI Lab

This PR makes most of the UI text in AI Lab be translatable.

JIRA

Formatting changes
Changes I've made that aren't strictly related to translating text, but I think are more universal / simple and will be easier for all students to understand
Changing A: B
It many places in the code I opted to remove or replace colons (:) placed between a label and text because the character is being used for structural purposes and may not be common in other languages. For example
Last updated: 1997

becomes
Last updated
1997

Removing commas from lists of items.
Not every languages uses commas for list of items, so I have changed the UI to instead separate items by whitespace instead.
apples, oranges, lemons, peaches

becomes
apples oranges lemons peaches

How to review this PR
There are a lot of changes in this PR, so here are some steps you can take to get started:

Read through all the strings in ailab.json. These are all the English strings which can be translated. Does something look wrong? Is there a string missing?
Check out the "Formatting changes" section above to see if they are acceptable.
Watch the demo video below to see where in the UI the strings are used.
Look through each individual commit so you can see a smaller set of changes.
Review the "Future work" below and note if there is anything else which should be done.

Demo
It would be way too many screenshots so I have opted for a video. Every string with a ! at the beginning is a string which can be translated. The exclamation point is a manual change I am doing to demo this change.
https://user-images.githubusercontent.com/1372238/196313540-435f0816-7d3a-41b0-8469-331681f94b81.mov
Future work

A different PR will address the  "Predict ___ based on ____" strings.
A different PR will address translating the CSV content.

I'm popping in because I'm curious about international dataset curation - will Code.org only offer the English, US-centric sets we have now? Folks can always upload their own data in whatever language they chose, but it was a bit of an undertaking to find datasets that had potentially interesting findings and I don't know what's out there in other languages.

@Erin007 , non-US-centric datasets are out of scope for this push to make CSD translatable, but we agree we need to diversify our datasets. Many of these datasets are not interesting to students outside the US.  That will be something the Student Experience team will need to resolve.

Left a couple inline comments. Also curious whether the app will need other changes to support RTL layouts?

I did a quick check by setting <html dir="RTL"> in the chrome developer console, and it looks like the UI behaves well in RTL!


Remove trackers on selected torrents

Please make it possible to select many torrents, right click and remove their trackers, to stay seeding in DHT/PEX/LSD only. Similar rule to "After seeding to 300%, limit upload rate to N kb/s", but for tracker selection.
It'd be nice if DHT/PEX/LSD were shown and treated as trackers, so they could also be en/disabled at will for many torrents at once, for whatever reason.
^^^ only this is really important, below some related ideas vvv
Much more could be done with the help of labels:
Label A, do not use DHT/PEX/LSD, like a private tracker.
Label B, after seeding to 300%, remove all trackers except DHT/PEX/LSD, like large scale public seeding to avoid overloading the tracker. And/or some rules relaxing the announce intervals.
A good label may incorporate a path, like TRACKER\SECTION\TYPE to be able to categorize a lot of various torrents. If we could refer to these as $1$2$3, it'd be easy to write all kinds of conditions for a wide variety of torrents. Things like priority, up/down limits, times when active, a lot could be done.
Labels on the other hand are most useful with RSS, which would be very necessary despite not being an internal part to torrents as dht etc are. Expose full regex library for labeling cases. It shouldn't be impossible for an average user to learn to do regex MOVIE.1080p.x264. instead of dos style wildcard MOVIE1080px264
An important part of RSS would be a checkmark to download .torrent files for all (matched) items in the feed, even if they were not selected to be started. This could be important for the health of the DHT cloud, if there was a way to keep the torrent actively seeding the torrent file itself despite the client not having any actual data. Nice people could use this to backup and seed every single .torrent file encountered.
This in turn would lend developent towards a DHT crawler. Catalogue hashes encountered, try to find out their contents/identity, download .torrent files, enable downloading according to label rules.

Hi! Cool suggestions - there are some related thoughts on labels and such in the Presets project. I'm adding this to that project as well.


Defining cell types and states with single-cell genomics

https://doi.org/10.1101/gr.190595.115
Authors
Trapnell, C.
Year
2015
Abstract

A revolution in cellular measurement technology is under way: For the first time, we have the ability to monitor global gene regulation in thousands of individual cells in a single experiment. Such experiments will allow us to discover new cell types and states and trace their developmental origins. They overcome fundamental limitations inherent in measurements of bulk cell population that have frustrated efforts to resolve cellular states. Single-cell genomics and proteomics enable not only precise characterization of cell state, but also provide a stunningly high-resolution view of transitions between states. These measurements may finally make explicit the metaphor that C.H. Waddington posed nearly 60 years ago to explain cellular plasticity: Cells are residents of a vast “landscape” of possible states, over which they travel during development and in disease. Single-cell technology helps not only locate cells on this landscape, but illuminates the molecular mechanisms that shape the landscape itself. However, single-cell genomics is a field in its infancy, with many experimental and computational advances needed to fully realize its full potential.

Discusses data generation, cell state identification/trajectory, computational methods, theory, and limitations.


Type pd.Timestamp not enforced

Initial Checks

[X] I have searched GitHub for a duplicate issue and I'm sure this is something new
[X] I have searched Google & StackOverflow for a solution and couldn't find anything
[X] I have read and followed the docs and still think this is a bug
[X] I am confident that the issue is with pydantic (not my code, or another library in the ecosystem like FastAPI or mypy)

Description
When using the BaseModel, pydantic ignores the specified type pd.Timestamp and converts to python's native datetime.
Uncommenting the custom init makes the code print "Success!", but then what's the point, right?
Example Code
import pandas as pd
from pydantic import BaseModel

object_sample = {
  "created_at": "2023-02-13T20:14:00.936721506Z"
}

class SamplePandasTimestamp(BaseModel):
    created_at: pd.Timestamp

    # def __init__(self, **data):
    #     data["created_at"] = pd.Timestamp(data["created_at"])
    #     super().__init__(**data)

instance_object = SamplePandasTimestamp(**object_sample)
if isinstance(instance_object.created_at, pd.Timestamp):
    print("Success!")
else:
    print("Failure")

Python, Pydantic & OS Version
pydantic version: 1.10.4
            pydantic compiled: True
                 install path: /Users/pichurri/miniforge3/envs/py3_10-pydantic/lib/python3.10/site-packages/pydantic
               python version: 3.10.9 | packaged by conda-forge | (main, Feb  2 2023, 20:26:08) [Clang 14.0.6 ]
                     platform: macOS-13.2-arm64-arm-64bit
     optional deps. installed: ['typing-extensions']

Affected Components

[ ] Compatibility between releases
[X] Data validation/parsing
[ ] Data serialization - .model_dump() and .model_dump_json()
[ ] JSON Schema
[ ] Dataclasses
[ ] Model Config
[X] Field Types - adding or changing a particular data type
[ ] Function validation decorator
[ ] Generic Models
[ ] Other Model behaviour - model_construct(), pickling, private attributes, ORM mode
[ ] Plugins and integration with other tools - mypy, FastAPI, python-devtools, Hypothesis, VS Code, PyCharm, etc.

The problem comes from the fact that pd.Timestamp is a subclass of datetime.datetime. The validator that is parsing as a datetime.datetime is looked up by subclass relationship; @samuelcolvin I wonder if, in the case where a validator is retrieved by a subclass lookup, if the type is a strict subclass, we could (perhaps optionally?) add a validator that applies the class to the returned value. To prevent issues like this where the value is parsed with an incorrect type.
I'm not sure if this has been resolved in v2 or not.
For what it's worth, you may be able to work around this with:

class PandasTimestamp(pd.Timestamp):
    @classmethod
    def convert_from_datetime(cls, v: Any) -> pd.Timestamp:
        return PandasTimestamp(v)  # could also return pd.Timestamp(v)

    @classmethod
    def __get_validators__(cls):
        # one or more validators may be yielded which will be called in the
        # order to validate the input, each validator will receive as an input
        # the value returned from the previous validator
        yield cls.convert_from_datetime

class SamplePandasTimestamp(BaseModel):
    created_at: PandasTimestamp

I don't think we can do anything for this in v1.
In v2, I think we should add documentation on how to work around this. We could even raise an error for subtypes to force the user to decide what to do manually.

Thank you for the update and for the workaround 👍

I don't think we can do anything for this in v1.

Oh yes, you neglect 500 open issues because of v2.
pydantic gets rusty
Regardless of how super-duper the V2 is going to be, the V1 is going to be keep in use for years by thousands of developers. You cannot stop maintaining v1.

V2 existed precisely because many of the open issues can't be solved in v1.
We're not ignoring open issues, we're working flat out to build a new version of pydantic that addresses them.
We are continuing to maintain v1, example - the patch release last week.
What we're not doing is implementing changes of behaviour in v1 which might break existing code, however loudly some people might demand those changes.

@adriangb I think this should be possible now in v2 with a custom annotation, but I'm not sure if there's currently a clean way to do it that works nicely with annotations/etc. Not sure if you have any good ideas. (Obviously if we ignore annotations, we can just produce a core schema that does a after function validator to convert to pd.Timestamp)

Need to add docs for handling custom types like this in v2 following tests.test_types.test_third_party_type_integration or similar.

Hello, sorry to bump here, I haven't found a better place to ask.
I got a similar problem with :
class MyModel(BaseModel):
    ...
    reception_date: datetime | None = Field(...,  gt=DEFAULT_MIN_DATETIME,  lt=DEFAULT_MAX_DATETIME)

Omitting the Field and everything is fine (I got a custom decorated validator to replace the validation) however, I don't have the gt and lt in the schema...
Doing as above I ran into a
ValueError: On field "reception_date" the following field constraints are set but not enforced: lt, gt.
For more details see https://docs.pydantic.dev/usage/schema/#unenforced-field-constraints

I could use exclusiveMaximum to enforce my schema and keep the decorated validation function but I was wondering if what I was trying to do is possible.
Thanks in advance,
Have a great day.

@PierreSnell I don't think this issue is the right place.
That said, on main at least I can't reproduce:
from datetime import datetime

from pydantic import BaseModel, Field

class MyModel(BaseModel):
    reception_date: datetime | None = Field(..., gt=datetime(year=1999, month=1, day=1))

MyModel(reception_date=datetime(year=2000, month=1, day=1))  # no error
MyModel(reception_date=datetime(year=1990, month=1, day=1))  # error

Tests with V2

Same code as initial report
With "arbitrary_types_allowed=True"

Conversion fail (no conversion str -> pd.Timestamp)
Validation success (passing a pd.Timestamp value)

Workaround to achieve validation and conversion (manually convert the string to pd.Timestamp)

Tested version

Tests with V2
Same code as initial report
import pandas as pd
from pydantic import BaseModel, ConfigDict

object_sample = {"created_at": "2023-02-13T20:14:00.936721506Z"}

class SamplePandasTimestamp(BaseModel):
    created_at: pd.Timestamp

    # def __init__(self, **data):
    #     data["created_at"] = pd.Timestamp(data["created_at"])
    #     super().__init__(**data)

instance_object = SamplePandasTimestamp(**object_sample)
if isinstance(instance_object.created_at, pd.Timestamp):
    print("Success!")
else:
    print("Failure")

Output:
❯ python timestamp_bug_validation.py
Traceback (most recent call last):
  File "/Users/pichurri/pydantic_bug/timestamp_bug_validation.py", line 7, in <module>
    class SamplePandasTimestamp(BaseModel):
  File "/Users/pichurri/pydantic_bug/.venv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 173, in __new__
    complete_model_class(
  File "/Users/pichurri/pydantic_bug/.venv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 463, in complete_model_class
    schema = cls.__get_pydantic_core_schema__(cls, handler)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pichurri/pydantic_bug/.venv/lib/python3.11/site-packages/pydantic/main.py", line 549, in __get_pydantic_core_schema__
    return __handler(__source)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/pichurri/pydantic_bug/.venv/lib/python3.11/site-packages/pydantic/_internal/_schema_generation_shared.py", line 82, in __call__
    schema = self._handler(__source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pichurri/pydantic_bug/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 426, in generate_schema
    return self._generate_schema_for_type(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pichurri/pydantic_bug/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 458, in _generate_schema_for_type
    schema = self._generate_schema(obj)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pichurri/pydantic_bug/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 684, in _generate_schema
    return self._model_schema(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pichurri/pydantic_bug/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 528, in _model_schema
    {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pichurri/pydantic_bug/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 528, in <dictcomp>
    {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pichurri/pydantic_bug/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 848, in _generate_md_field_schema
    common_field = self._common_field_schema(name, field_info, decorators)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pichurri/pydantic_bug/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 900, in _common_field_schema
    schema = self._apply_annotations(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pichurri/pydantic_bug/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1586, in _apply_annotations
    schema = get_inner_schema(source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pichurri/pydantic_bug/.venv/lib/python3.11/site-packages/pydantic/_internal/_schema_generation_shared.py", line 82, in __call__
    schema = self._handler(__source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pichurri/pydantic_bug/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1550, in inner_handler
    schema = self._generate_schema(obj)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pichurri/pydantic_bug/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 689, in _generate_schema
    return self.match_type(obj)
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/pichurri/pydantic_bug/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 776, in match_type
    return self._unknown_type_schema(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pichurri/pydantic_bug/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 375, in _unknown_type_schema
    raise PydanticSchemaGenerationError(
pydantic.errors.PydanticSchemaGenerationError: Unable to generate pydantic-core schema for <class 'pandas._libs.tslibs.timestamps.Timestamp'>. Set `arbitrary_types_allowed=True` in the model_config to ignore this error or implement `__get_pydantic_core_schema__` on your type to fully support it.

If you got this error by calling handler(<some type>) within `__get_pydantic_core_schema__` then you likely need to call `handler.generate_schema(<some type>)` since we do not call `__get_pydantic_core_schema__` on `<some type>` otherwise to avoid infinite recursion.

For further information visit https://errors.pydantic.dev/2.1.1/u/schema-for-unknown-type
❯

With "arbitrary_types_allowed=True"

[ ] Works for conversion (as it does when passing a string with an int to an int property)
[X] Works for validation, only accepts objects with field of the right type

Conversion fail (no conversion str -> pd.Timestamp)
import pandas as pd
from pydantic import BaseModel, ConfigDict

object_sample = {"created_at": "2023-02-13T20:14:00.936721506Z"}

class SamplePandasTimestamp(BaseModel):
    model_config = ConfigDict(arbitrary_types_allowed=True)

    created_at: pd.Timestamp

    # def __init__(self, **data):
    #     data["created_at"] = pd.Timestamp(data["created_at"])
    #     super().__init__(**data)

instance_object = SamplePandasTimestamp(**object_sample)
if isinstance(instance_object.created_at, pd.Timestamp):
    print("Success!")
else:
    print("Failure")

Output:
❯ python timestamp_bug_validation.py
Traceback (most recent call last):
  File "/Users/pichurri/pydantic_bug/timestamp_bug_validation.py", line 17, in <module>
    instance_object = SamplePandasTimestamp(**object_sample)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pichurri/pydantic_bug/.venv/lib/python3.11/site-packages/pydantic/main.py", line 159, in __init__
    __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__)
pydantic_core._pydantic_core.ValidationError: 1 validation error for SamplePandasTimestamp
created_at
  Input should be an instance of Timestamp [type=is_instance_of, input_value='2023-02-13T20:14:00.936721506Z', input_type=str]
    For further information visit https://errors.pydantic.dev/2.1/v/is_instance_of

Validation success (passing a pd.Timestamp value)
import pandas as pd
from pydantic import BaseModel, ConfigDict

object_sample = {"created_at": pd.Timestamp("2023-02-13T20:14:00.936721506Z")}

class SamplePandasTimestamp(BaseModel):
    model_config = ConfigDict(arbitrary_types_allowed=True)

    created_at: pd.Timestamp

    # def __init__(self, **data):
    #     data["created_at"] = pd.Timestamp(data["created_at"])
    #     super().__init__(**data)

instance_object = SamplePandasTimestamp(**object_sample)
if isinstance(instance_object.created_at, pd.Timestamp):
    print("Success!")
else:
    print("Failure")

Output:
❯ python timestamp_bug_validation.py
Success!

Workaround to achieve validation and conversion (manually convert the string to pd.Timestamp)
import pandas as pd
from pydantic import BaseModel, ConfigDict

object_sample = {"created_at": "2023-02-13T20:14:00.936721506Z"}

class SamplePandasTimestamp(BaseModel):
    model_config = ConfigDict(arbitrary_types_allowed=True)

    created_at: pd.Timestamp

    def __init__(self, **data):
        data["created_at"] = pd.Timestamp(data["created_at"])
        super().__init__(**data)

instance_object = SamplePandasTimestamp(**object_sample)
if isinstance(instance_object.created_at, pd.Timestamp):
    print("Success!")
else:
    print("Failure")

Output:
❯ python timestamp_bug_validation.py
Success!

Tested version
>>> print(pydantic.version.version_info())
             pydantic version: 2.1.1
        pydantic-core version: 2.4.0
          pydantic-core build: profile=release pgo=false mimalloc=true
                 install path: /Users/pichurri/pydantic_bug/.venv/lib/python3.11/site-packages/pydantic
               python version: 3.11.3 | packaged by conda-forge | (main, Apr  6 2023, 08:58:31) [Clang 14.0.6 ]
                     platform: macOS-13.5-arm64-arm-64bit
     optional deps. installed: ['typing-extensions']

What would be more proper fix/workaround?

We’re no longer actively developing Pydantic V1 (although it will continue to receive security fixes for the next year or so), so we’re closing issues solely related to V1. If you really think this shouldn’t be closed, please comment or create a new issue 🚀.


[feature request] consistent YAML ordering

I find it useful to have consistent ordering on some sections of dbt, and default alphabetical ordering on some other sections.
The example below are sources, but the idea can be applied to other yaml documents too.
version: 2
sources:

# main keys are ordered consistently, where tables is the last key to ease file reading
- name: A
  description: 'description'
  database: 'DB'
  schema: 'SCHEMA'
  quoting:
   database: false
   schema: false
   identifier: false
  meta:
   key_1: 'a'
   key_2: 'b'

  freshness:
   warn_after: { count: 9999, period: day }
  loaded_at_field: updated_at::timestamp

  tags: []

  tables: # list of tables ordered Alphabetically
  - name: table_A
    description: 'description'
    columns:
    - name: col_a
      description: ' '

    - name: col_b
      description: ' '
      tests:

  - name: table_B
  - name: table_C

# if other sources in the file, then sources list is also ordered alphabetically
- name: B

Also, I would challenge the fact of ordering columns as the order coming from the DB. I would say alphabetical order makes reading easier.
I am not very familiar with dbt-osmosis code base yet, but happy to take a look at how could the be implemented if it is of any interest

I think configurability here would make sense -- there is one place in the code base where we do the yaml dumping.
I am fairly strongly of the opinion that DB ordering is better for a number of reasons. But I also think that folks will have differing opinions and needs to my own. So maybe a CLI flag is sufficient.


feat: render skeleton of repo with a jsonnet lib

Rendered from Jsonnet:

config.yml for k8s-gen
LICENSE
root/docs READMEs
mkdocs config and github action + requirements for the action

Removed the rendered files from the source to avoid confusion.
This depends on a change to docsonnet to render the docs in a directory tree instead of a flat file
structure: https://github.com/jsonnet-libs/docsonnet/pull/20
After this, rendering a new Jsonnet lib & docs from a CRD would look like this:
local config = import '../../jsonnet/config.jsonnet';

config.new(
  name='cert-manager',
  specs=[
    {
      output: '1.3',
      openapi: 'http://localhost:8001/openapi/v2',
      prefix: '^io\\.cert-manager\\..*',
      crd: 'https://github.com/jetstack/cert-manager/releases/download/v1.3.1/cert-manager.crds.yaml',
      localName: 'cert_manager',
    },
  ]
)

Next PR will automate the push to the remote repository.

I have tested this on https://github.com/jsonnet-libs/cert-manager-lib.

@xvzf and @julienduchesne also interested in reviewing this?


Use specific "llama.cpp" version

.package(url: "https://github.com/ggerganov/llama.cpp.git", branch: "master")

It is better to use a special version of the package, because master is constantly updated and different revisions may work unstable or not work at all even

The llama.cpp package contains unsafeFlags, which prevents from using semantic versions.
Please refer to https://forums.swift.org/t/override-for-unsafeflags-in-swift-package-manager/45273/5 for detailed explanation.


Assemble list contributor matrices and cell type annotation for MVP DCP2 release

We need to provide a list of contributor matrices and cell-type annotation for the MVP DCP2 release so they can be part of the first snapshot.
We need to create a spreadsheet/tsv file which has the following four columns.

project UUID
filename
file type (gene expression matrices, cell type annotation file, other)
file source (GEO, ArrayExpress, Contributor)

The files themselves should be uploaded to a GCP bucket, but we don't know where yet.
This task must be completed in this sprint, but no heroic measures should be taken to acquire this data.
There is no expectation of file format standardization or metadata harmonization within the files.
The expected source of files

Cell type annotation files produced for the March 2020 release
Expression files from GEO/ArrayExpress
Responses to emails sent to contributors to ask for cell-type annotation/matrix files in loom/h5ad format (single emails to be sent with a clear deadline, no expectation of chasing)

Existing documents associated with this task
The plan overview https://docs.google.com/document/d/184dWgyqPUW2Kirt5C1VF9EXVRR58wTgFXtxHFPHel_0/edit#
The list of possible datasets (needs to be crosschecked with what will be uploaded) https://docs.google.com/document/d/1af-E-sGAvrTNyYdp_nsRDFka_q4BSG4djV4CQ2AfTjs/edit
Checklist:

[ ] Cross check what new DCP2 datasets will be uploaded
[x] What is the location of the GCP bucket the files need to be uploaded to

gs://hca-prod-ebi-matrices
https://console.cloud.google.com/storage/browser/hca-prod-ebi-matrices

[ ] Where are all the expression and cell-type annotation files
[ ] Emailing all contributors if needed for loom/h5ad files and/or cell type annotation

Acceptance criteria for the task:

[ ] Files uploaded to specified dropbox
[ ] Spreadsheet produced listing four columns specified above
[ ] A list of projects which should not have contributor matrices/cell type annotation associated with them

Bucket setup is being established here https://github.com/DataBiosphere/azul/issues/2136
I have asked for access for @rolando-ebi and @rdgoite as the dev people on operations this sprint  and next, more people can always be requested

I created a sheet as specified here in the brokering folder: https://docs.google.com/spreadsheets/d/1Te1FTUbdn-tFS3eLVTB7uMEApoFU0Il6krSoiq_LAAw/edit#gid=0
The housing folder for files and the sheet is here: https://drive.google.com/open?id=1FMjJwYamMyuCtNJoTiA30kM3vsL6Q8LD
@lauraclarke I have added the tissue sensitivity study as a pilot, do you think the files and the way the spreadsheet is filled looks suitable?

That spreadsheet looks good
Isn't c4077b3c-5c98-4d26-a614-246d12c2e5d7.TissueSensitivity-Lung-10x_cell_type_2020-03-12.csv cell type annotation though rather than other?
If those files could be uploaded to the GCP bucket and we can give UCSC both this file and tell them the actual files are in the bucket that would be great, then they could give us feedback

yes that makes more sense. do we think it is a good idea to prefix with the project uuid? I don't like having such long names but at least it makes the files unambiguous.
I had thought of also having subdirectories per project, but not sure if that is needed/wanted but will ask ucsc

I think the UUID is a good way to ensure filename uniqueness without having do a massive amount of checking, I guess the project short name might also work if you would rather have something human-readable but that might create as many challenges as it resolved.
For sub-directories I don't know which is easier from a downstream perspective, all the files or subdirectories so checking sounds like a good plan
I will try and remember to figure out who other than Hannes is on point for this, it might be Daniel Sothos but I will confirm at the leadership MVP catch up today at 4 pm.

ok thanks for confirming
I just tried uploading some test files to the bucket and it doesn't seem to work, I will post on the ticket referred to above

I think Hannes already closed that ticket so it might be better to create a new one or also ping Daniel and Hannes on slack

I am now trying with the gsutil command line tool instead and it might be working, it is fairly slow and the h5ad files are quite large so won't know for a while.

Trevor confirms Daniel Sothos is the right contact for their needs on this. I guess pointing him to this ticket is probably a good idea

yep he has already fixed for me

From trevor on how to indicate files which fall into more than one category
"We suggest 3 columns (1 for each file type), and check all that are applicable to that file (i.e. that row)."

@danielsotirhos just letting you know I have started uploading files to gs area, so you could use for testing for what needs to be done in terms of building sub-graphs for these files. Let me know if there is anything unexpected with the folder structure or naming.
Also don't consider these the 'final' files as yet, we want to get confirmation from contributors.
Also wanted to point out some files are from new projects that weren't a part of DCP1 so won't be in the system as yet.

UCSC team said they want all files in the staging bucket by 2 weeks before the release deadline, the deadline is 18th of September so our deadline for this task is now end of business, Friday 4th September

@lauraclarke just to confirm, if for a project I have found a gene expression matrix and cell type annotations I still need to email the contributor to check if that's what they want released?
Or do I only need to email people if I can't find appropriate files?

I have emailed authors of the projects I worked on to ask for cell type annotations and expression matrices. They were not available through the publication supplements. The raw expression matrix if consistently available in GEO.

All files we have received have been uploaded to the provided google storage area and the UCSC team have been informed about them on HCA slack dcp2 channel
I consider this one done, I am confused by a couple of the acceptance criteria above but if someone disagrees feel free to reopen

@mshadbolt The last one was so we knew the numbers and when UCSC post their summary table for the release we could confirm that nothing had been misattributed

@mshadbolt @lauraclarke According to the spreadsheet the following files should exist but were not found:
gs://hca-prod-ebi-matrices/abe1a013-af7a-45ed-8c26-f3793c24a1f4-KidneySingleCellAtlas/abe1a013-af7a-45ed-8c26-f3793c24a1f4.Fetal_full.h5ad
gs://hca-prod-ebi-matrices/abe1a013-af7a-45ed-8c26-f3793c24a1f4-KidneySingleCellAtlas/abe1a013-af7a-45ed-8c26-f3793c24a1f4.Mature_Full_v2.1.h5ad
gs://hca-prod-ebi-matrices/f83165c5-e2ea-4d15-a5cf-33f3550bffde-FetalMaternalInterface/f83165c5-e2ea-4d15-a5cf-33f3550bffde.FetalMaternal-Decidua-10x_cell_type_2020-03-12.csv
gs://hca-prod-ebi-matrices/f2fe82f0-4454-4d84-b416-a885f3121e59-FetalLiverHaematopoiesis/f2fe82f0-4454-4d84-b416-a885f3121e59.supp_table_s6.csv
gs://hca-prod-ebi-matrices/42d4f8d4-5422-4b78-adae-e7c3c2ef511c-HumanAdiposeTissue/42d4f8d4-5422-4b78-adae-e7c3c2ef511c.GSE128890_RAW.tar
gs://hca-prod-ebi-matrices/7027adc6-c9c9-46f3-84ee-9badc3a4f53b-Pitx2DevelopingHeart/7027adc6-c9c9-46f3-84ee-9badc3a4f53b.GSE131181_e10.5.meta.data.csv.gz
gs://hca-prod-ebi-matrices/7027adc6-c9c9-46f3-84ee-9badc3a4f53b-Pitx2DevelopingHeart/7027adc6-c9c9-46f3-84ee-9badc3a4f53b.GSE131181_e10.5.raw.data.csv.gz
gs://hca-prod-ebi-matrices/7027adc6-c9c9-46f3-84ee-9badc3a4f53b-Pitx2DevelopingHeart/7027adc6-c9c9-46f3-84ee-9badc3a4f53b.GSE131181_e10.5.scale.data.csv.gz
gs://hca-prod-ebi-matrices/7027adc6-c9c9-46f3-84ee-9badc3a4f53b-Pitx2DevelopingHeart/7027adc6-c9c9-46f3-84ee-9badc3a4f53b.GSE131181_e13.5.meta.data.csv.gz
gs://hca-prod-ebi-matrices/7027adc6-c9c9-46f3-84ee-9badc3a4f53b-Pitx2DevelopingHeart/7027adc6-c9c9-46f3-84ee-9badc3a4f53b.GSE131181_e13.5.raw.data.csv.gz
gs://hca-prod-ebi-matrices/7027adc6-c9c9-46f3-84ee-9badc3a4f53b-Pitx2DevelopingHeart/7027adc6-c9c9-46f3-84ee-9badc3a4f53b.GSE131181_e13.5.scale.data.csv.gz
gs://hca-prod-ebi-matrices/b4a7d12f-6c2f-40a3-9e35-9756997857e3-Lung_Fibroblasts/b4a7d12f-6c2f-40a3-9e35-9756997857e3.GSE129605_RAW.tar
gs://hca-prod-ebi-matrices/2086eb05-10b9-432b-b7f0-169ccc49d270-DevelopingCardiacSystem/2086eb05-10b9-432b-b7f0-169ccc49d270.GSE132658_RAW.tar
gs://hca-prod-ebi-matrices/2ef3655a-973d-4d69-9b41-21fa4041eed7-HumanAdultKidneyLiaoMiao/2ef3655a-973d-4d69-9b41-21fa4041eed7.GSE131685_RAW.tar

@danielsotirhos apologies for missing those, currently uploading, will let you know when complete

For the FetalLiverHaematopoieis there was a typo in the file name, it is an upper case S i.e.
https://console.cloud.google.com/storage/browser/_details/hca-prod-ebi-matrices/f2fe82f0-4454-4d84-b416-a885f3121e59-FetalLiverHaematopoiesis/f2fe82f0-4454-4d84-b416-a885f3121e59.supp_table_S6.csv
some have inconsistencies in the folder naming which I will harmonise

@danielsotirhos can you have another check and see that you can find all the files ok?

just clarifying, 3 files were outright missing, the others had different naming which I largely matched above, though I didn't fix the upper case S in the fetal liver file in the cloud storage, I just updated the name in the google sheet

@mshadbolt Thank you, using the latest spreadsheet (minus rows missing a file name) I found just one missing file.
gs://hca-prod-ebi-matrices/42d4f8d4-5422-4b78-adae-e7c3c2ef511c-HumanAdiposeTissue/42d4f8d4-5422-4b78-adae-e7c3c2ef511c.GSE128890_RAW.tar

Looks like it was placed in an extra subdir?
42d4f8d4-5422-4b78-adae-e7c3c2ef511c-HumanAdiposeTissue/42d4f8d4-5422-4b78-adae-e7c3c2ef511c_GSE128890_HumanAdiposeTissue/42d4f8d4-5422-4b78-adae-e7c3c2ef511c.GSE128890_RAW.tar

d'oh I thought I fixed those, fixing...

I think I fixed it @danielsotirhos

Perfect, all files on the spreadsheet found in the bucket. Thank you @mshadbolt

Thanks for confirming  @danielsotirhos !


Serialization Error: Failed to deserialize: expected end of object, but found field id: 103

I am using duckdb version 0.10.2, and using DBeaver 24.1.0 with org.jkiss.dbeaver.ext.duckdb_<IP_ADDRESS>406021658.jar.
DBeaver could connected to :memory: and a file-backed duckdb database with an empty table, which are created using duckdb 0.10.2.
The connection test from within DBeaver shows:
Server: DuckDB v0.9.1
Driver: DuckDBJ 1.0

I did the same connection testing to larger duckdb (30 MB). It gave me this error:
eclipse.buildId=unknown
java.version=17.0.6
java.vendor=Eclipse Adoptium
BootLoader constants: OS=win32, ARCH=x86_64, WS=win32, NL=en_SG
Framework arguments:  -eclipse.keyring C:\Users\<user>\AppData\Roaming\DBeaverData\secure\secure_storage
Command-line arguments:  -os win32 -ws win32 -arch x86_64

org.jkiss.dbeaver.model
Error
Sun Jun 09 09:34:43 SGT 2024
Serialization Error: Failed to deserialize: expected end of object, but found field id: 103

java.sql.SQLException: Serialization Error: Failed to deserialize: expected end of object, but found field id: 103
	at org.duckdb.DuckDBNative.duckdb_jdbc_startup(Native Method)
	at org.duckdb.DuckDBConnection.newConnection(DuckDBConnection.java:48)
	at org.duckdb.DuckDBDriver.connect(DuckDBDriver.java:38)
	at org.jkiss.dbeaver.model.impl.jdbc.JDBCConnectionOpener.run(JDBCConnectionOpener.java:109)
	at org.jkiss.dbeaver.model.impl.jdbc.JDBCConnectionOpener.run(JDBCConnectionOpener.java:83)
	at org.jkiss.dbeaver.model.impl.jdbc.JDBCDataSource.openConnection(JDBCDataSource.java:221)
	at org.jkiss.dbeaver.ext.generic.model.GenericDataSource.openConnection(GenericDataSource.java:159)
	at org.jkiss.dbeaver.model.impl.jdbc.JDBCExecutionContext.connect(JDBCExecutionContext.java:106)
	at org.jkiss.dbeaver.model.impl.jdbc.JDBCRemoteInstance.initializeMainContext(JDBCRemoteInstance.java:102)
	at org.jkiss.dbeaver.model.impl.jdbc.JDBCRemoteInstance.<init>(JDBCRemoteInstance.java:61)
	at org.jkiss.dbeaver.model.impl.jdbc.JDBCDataSource.initializeRemoteInstance(JDBCDataSource.java:119)
	at org.jkiss.dbeaver.ext.generic.model.GenericDataSource.<init>(GenericDataSource.java:124)
	at org.jkiss.dbeaver.ext.duckdb.model.DuckDBDataSource.<init>(DuckDBDataSource.java:38)
	at org.jkiss.dbeaver.ext.duckdb.model.DuckMetaModel.createDataSourceImpl(DuckMetaModel.java:155)
	at org.jkiss.dbeaver.ext.generic.GenericDataSourceProvider.openDataSource(GenericDataSourceProvider.java:106)
	at org.jkiss.dbeaver.registry.DataSourceDescriptor.openDataSource(DataSourceDescriptor.java:1381)
	at org.jkiss.dbeaver.registry.DataSourceDescriptor.connect0(DataSourceDescriptor.java:1245)
	at org.jkiss.dbeaver.registry.DataSourceDescriptor.connect(DataSourceDescriptor.java:1035)
	at org.jkiss.dbeaver.runtime.jobs.ConnectJob.run(ConnectJob.java:78)
	at org.jkiss.dbeaver.runtime.jobs.ConnectionTestJob.run(ConnectionTestJob.java:103)
	at org.jkiss.dbeaver.model.runtime.AbstractJob.run(AbstractJob.java:117)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)

I have raised a similar issue in https://github.com/dbeaver/dbeaver/issues/21353
Update 1:
I have updated duckdb python client to 1.0.0 and duckdb.exe to 1.0.0 too. The issue persists even if I have re-created a new database file.

Hey, could you please share the exact steps to reproduce this?

I could not share the data and views. Are there ways for me to trace the code for you?

If you can provide us with a sequence of commands and SQL queries to reproduce the issue, that would be best

I use the following SQL command to create or replace a view comp:
CREATE OR REPLACE VIEW comp AS
-- Latest semester for every module
WITH _a AS (
    SELECT
        module_id,
        MAX(semester) AS max_semester
    FROM nsrq966
    GROUP BY module_id
),
_others AS (
    SELECT DISTINCT
        n.semester,
        n.module_id,
        n.catalog,
        n.class,
        n.component,
        n.pattern,
        n.day,
        n.start_time,
        n.duration,
        n.load,
        (n.day = 'SAT') AS oal
    FROM
        nsrq966 n,
        _a
    WHERE
        n.semester = _a.max_semester AND
        n.module_id = _a.module_id AND
        n.class != 'SSP' AND   -- ignore summer school program (SSP)
        -- Ingore INT and FYP which have many different durations
        n.catalog NOT LIKE '%INT6' AND
        n.catalog NOT LIKE '%INTO6' AND
        n.catalog NOT LIKE '%INTY1' AND
        n.catalog NOT LIKE '%INTY2' AND
        n.catalog NOT IN ('41FYP', '43PDD', '44FYP', '93FYP')
),
-- Manually put INT module back
_int AS (
    SELECT DISTINCT
        n.semester,
        n.module_id,
        n.catalog,
        'XX' AS class,          -- Set fixed class
        'PRA' AS component,     -- Set fixed component
        '' AS pattern,          -- Set fixed pattern
        '' AS day,              -- Set fixed day
        '08:00' AS start_time,  -- Set fixed start_time
        2.0 AS duration,        -- Set fixed duration
        2.0 AS load,            -- Set fixed load
        false AS oal            -- Set oal to false
    FROM
        nsrq966 n,
        _a
    WHERE
        n.semester = _a.max_semester AND
        n.module_id = _a.module_id AND
        (n.catalog LIKE '%INT6' OR
         n.catalog LIKE '%INTO6' OR
         n.catalog LIKE '%INTY1' OR
         n.catalog LIKE '%INTY2')
),
-- Manually put FYP module back
_fyp AS (
    SELECT DISTINCT
        n.semester,
        n.module_id,
        n.catalog,
        'XX' AS class,          -- Set fixed class
        'PRA' AS component,     -- Set fixed component
        '' AS pattern,          -- Set fixed pattern
        '' AS day,              -- Set fixed day
        '08:00' AS start_time,  -- Set fixed start_time
        1.0 AS duration,        -- Set fixed duration
        1.0 AS load,            -- Set fixed load
        false AS oal            -- Set oal to false
    FROM
        nsrq966 n,
        _a
    WHERE
        n.semester = _a.max_semester AND
        n.module_id = _a.module_id AND
        n.catalog in ('41FYP', '43PDD', '44FYP', '93FYP')
),
-- Combine _others, _int and _fyp
_combined AS (
    SELECT * FROM _others UNION ALL
    SELECT * FROM _int    UNION ALL
    SELECT * FROM _fyp
),
-- Append sequence to repeated component in the
-- same module_id, class
_numbered AS (
    SELECT
        *,
        CASE
            WHEN ROW_NUMBER() OVER (
                PARTITION BY
                    module_id,
                    class,
                    component
                ORDER BY duration DESC) = 1 THEN component
            ELSE component || CAST(ROW_NUMBER() OVER (
                PARTITION BY
                    module_id,
                    class,
                    component
                ORDER BY duration DESC) AS VARCHAR)
        END AS component_1
    FROM _combined
)
-- Finally
SELECT DISTINCT
    module_id,
    catalog,
    component_1 AS component,
    pattern,
    duration,
    load,
    oal
FROM
    _numbered
ORDER BY
    catalog,
    class,
    component,
    load,
    oal;

If the SQL is executed via duckdb CLI, it would give me the beautified SQL code:
CREATE VIEW comp
AS
  WITH _a
       AS (SELECT module_id,
                  Max(semester) AS max_semester
           FROM   nsrq966
           GROUP  BY module_id),
       _others
       AS (SELECT DISTINCT n.semester,
                           n.module_id,
                           n."catalog",
                           n."class",
                           n.component,
                           n.pattern,
                           n."day",
                           n.start_time,
                           n.duration,
                           n."load",
                           ( n."day" = 'SAT' ) AS oal
           FROM   nsrq966 AS n,
                  _a
           WHERE  ( ( n.semester = _a.max_semester )
                    AND ( n.module_id = _a.module_id )
                    AND ( n."class" != 'SSP' )
                    AND ( n."catalog" !~~ '%INT6' )
                    AND ( n."catalog" !~~ '%INTO6' )
                    AND ( n."catalog" !~~ '%INTY1' )
                    AND ( n."catalog" !~~ '%INTY2' )
                    AND ( n."catalog" NOT IN ( '41FYP', '43PDD', '44FYP',
                                               '93FYP'
                                             ) ) )
          ),
       _int
       AS (SELECT DISTINCT n.semester,
                           n.module_id,
                           n."catalog",
                           'XX'                 AS "class",
                           'PRA'                AS component,
                           ''                   AS pattern,
                           ''                   AS "day",
                           '08:00'              AS start_time,
                           2.0                  AS duration,
                           2.0                  AS "load",
                           Cast('f' AS BOOLEAN) AS oal
           FROM   nsrq966 AS n,
                  _a
           WHERE  ( ( n.semester = _a.max_semester )
                    AND ( n.module_id = _a.module_id )
                    AND ( ( n."catalog" ~~ '%INT6' )
                           OR ( n."catalog" ~~ '%INTO6' )
                           OR ( n."catalog" ~~ '%INTY1' )
                           OR ( n."catalog" ~~ '%INTY2' ) ) )),
       _fyp
       AS (SELECT DISTINCT n.semester,
                           n.module_id,
                           n."catalog",
                           'XX'                 AS "class",
                           'PRA'                AS component,
                           ''                   AS pattern,
                           ''                   AS "day",
                           '08:00'              AS start_time,
                           1.0                  AS duration,
                           1.0                  AS "load",
                           Cast('f' AS BOOLEAN) AS oal
           FROM   nsrq966 AS n,
                  _a
           WHERE  ( ( n.semester = _a.max_semester )
                    AND ( n.module_id = _a.module_id )
                    AND ( n."catalog" IN ( '41FYP', '43PDD', '44FYP', '93FYP' )
                        )
                  )),
       _combined
       AS (((SELECT *
             FROM   _others)
            UNION ALL
            (SELECT *
             FROM   _int))
           UNION ALL
           (SELECT *
            FROM   _fyp)),
       _numbered
       AS (SELECT *,
                  CASE
                    WHEN (( Row_number()
                              over (
                                PARTITION BY module_id, "class", component
                                ORDER BY duration DESC) = 1 )) THEN
                    ( component )
                    ELSE ( component
                           || Cast(Row_number()
                                     over (
                                       PARTITION BY module_id, "class",
                                     component
                                       ORDER BY duration DESC) AS VARCHAR) )
                  END AS component_1
           FROM   _combined)
  SELECT DISTINCT module_id,
                  "catalog",
                  component_1 AS component,
                  pattern,
                  duration,
                  "load",
                  oal
  FROM   _numbered
  ORDER  BY "catalog",
            "class",
            component,
            "load",
            oal;

I could not attach the database via DBeaver.
However, if I execute the first SQL code via DBeaver, it gives me the following beautified schema:
CREATE VIEW comp
(module_id, "catalog", component, pattern, duration, "load", oal)
AS
  WITH _a
       AS (SELECT module_id,
                  Max(semester) AS max_semester
           FROM   nsrq966
           GROUP  BY module_id),
       _others
       AS (SELECT DISTINCT n.semester,
                           n.module_id,
                           n."catalog",
                           n."class",
                           n.component,
                           n.pattern,
                           n."day",
                           n.start_time,
                           n.duration,
                           n."load",
                           ( n."day" = 'SAT' ) AS oal
           FROM   nsrq966 AS n,
                  _a
           WHERE  ( ( n.semester = _a.max_semester )
                    AND ( n.module_id = _a.module_id )
                    AND ( n."class" != 'SSP' )
                    AND ( n."catalog" !~~ '%INT6' )
                    AND ( n."catalog" !~~ '%INTO6' )
                    AND ( n."catalog" !~~ '%INTY1' )
                    AND ( n."catalog" !~~ '%INTY2' )
                    AND ( n."catalog" NOT IN ( '41FYP', '43PDD', '44FYP',
                                               '93FYP'
                                             ) ) )
          ),
       _int
       AS (SELECT DISTINCT n.semester,
                           n.module_id,
                           n."catalog",
                           'XX'                 AS "class",
                           'PRA'                AS component,
                           ''                   AS pattern,
                           ''                   AS "day",
                           '08:00'              AS start_time,
                           2.0                  AS duration,
                           2.0                  AS "load",
                           Cast('f' AS BOOLEAN) AS oal
           FROM   nsrq966 AS n,
                  _a
           WHERE  ( ( n.semester = _a.max_semester )
                    AND ( n.module_id = _a.module_id )
                    AND ( ( n."catalog" ~~ '%INT6' )
                           OR ( n."catalog" ~~ '%INTO6' )
                           OR ( n."catalog" ~~ '%INTY1' )
                           OR ( n."catalog" ~~ '%INTY2' ) ) )),
       _fyp
       AS (SELECT DISTINCT n.semester,
                           n.module_id,
                           n."catalog",
                           'XX'                 AS "class",
                           'PRA'                AS component,
                           ''                   AS pattern,
                           ''                   AS "day",
                           '08:00'              AS start_time,
                           1.0                  AS duration,
                           1.0                  AS "load",
                           Cast('f' AS BOOLEAN) AS oal
           FROM   nsrq966 AS n,
                  _a
           WHERE  ( ( n.semester = _a.max_semester )
                    AND ( n.module_id = _a.module_id )
                    AND ( n."catalog" IN ( '41FYP', '43PDD', '44FYP', '93FYP' )
                        )
                  )),
       _combined
       AS (((SELECT *
             FROM   _others)
            UNION ALL
            (SELECT *
             FROM   _int))
           UNION ALL
           (SELECT *
            FROM   _fyp)),
       _numbered
       AS (SELECT *,
                  CASE
                    WHEN (( Row_number()
                              over (
                                PARTITION BY module_id, "class", component
                                ORDER BY duration DESC) = 1 )) THEN
                    ( component )
                    ELSE ( component
                           || Cast(Row_number()
                                     over (
                                       PARTITION BY module_id, "class",
                                     component
                                       ORDER BY duration DESC) AS VARCHAR) )
                  END AS component_1
           FROM   _combined)
  SELECT DISTINCT module_id,
                  "catalog",
                  component_1 AS component,
                  pattern,
                  duration,
                  "load",
                  oal
  FROM   _numbered
  ORDER  BY "catalog",
            "class",
            component,
            "load",
            oal;

CREATE TABLE nsrq966
  (
     career     VARCHAR,
     module_sch VARCHAR,
     semester   INTEGER,
     acadprog   VARCHAR,
     stud_id    VARCHAR,
     stud       VARCHAR,
     module_id  VARCHAR,
     subject    VARCHAR,
     "catalog"  VARCHAR,
     "class"    VARCHAR,
     component  VARCHAR,
     instr_id   VARCHAR,
     instr      VARCHAR,
     venue      VARCHAR,
     "day"      VARCHAR,
     pattern    VARCHAR,
     start_time VARCHAR,
     end_time   VARCHAR,
     duration   DOUBLE,
     "load"     DOUBLE
  );

I could attach the database via DBeaver even I close and open DBeaver after creation of the view comp.
Hope the information is good for you.