Performance questions

I'm keen to "run onto the spike" and find any big potential performance improvements in pydantic-core while the API can be changed easily.
I'd therefore love anyone with experience of rust and/or pyo3 to have a look through the code and see if I'm doing anything dumb.
Particular concerns:

The "cast_as vs. extract" issues described in https://github.com/PyO3/pyo3/issues/2278 was a bit scary as I only found the solution by chance, are there any other similar issues with pyo3?
generally copying and cloning values - am I copying stuff where I don't have to? In particular, is input or parts of input (in the case of a dict/list/tuple/set etc.) copied when it doesn't need to be?
Similarly, could we use a PyObject instead of PyAny or visa-versa and improve performance?
here and in a number of other implementations of ListInput and DictInput we do a totally unnecessary map, is this avoidable? Is this having a performance impact? Is there another way to give a general interface to the underlying  datatypes that's more performance
The code for generating models here seems to be pretty slow compared to other validators, can anything be done?
Recursive models are slowing than I had expected, I thought it might be  the use of RwLock that was causing the performance problems, but I managed to remove that (albeit in a slightly unsafe way) in #32 and it didn't make a difference. Is something else the problem? Could we remove Arc completely?
lifetimes get pretty complicated, I haven't even checked if get a memory leak from running repeat validation, should we/can we change any lifetimes?

I'll add to this list if anything else comes to me.
More generally I wonder if there are performance improvements that I'm not even aware of? "What you don't know, you can't optimise"
@pauleveritt @robcxyz

I tried doing some performance profiling, see
https://github.com/samuelcolvin/pydantic-core/blob/7c9058e641549e816614a097148019c11feae95e/benchmarks/minimal.py#L3-L15
But I didn't find anything particularly interesting.

Hi :) First of all, thank you for Pydantic!
Here are some ideas about allocations:
truncate - as it allocates in format!, but it could be avoided:
use std::fmt::Write;

macro_rules! truncate {
    ($out: expr, $value: expr) => {
        if $value.len() > 50 {
            write!($out, "{}...{}", &$value[0..25], &$value[$value.len() - 24..]);
        } else {
            $out.push_str(&$value);
        }
    };
}

Similarly, some push_str calls together with &format! could be replaced by write! (clippy should warn about it from Rust 1.61 btw):
output.push_str(&format!(", input_type={}", type_));

to
write!(output, ", input_type={}", type_);

There are a few more to_string calls that could be avoided:
let loc = self
    .location
    .iter()
    .map(|i| i.to_string())
    .collect::<Vec<String>>()
    .join(" -> ");

to:
let mut first = true;
for item in &self.location {
    if !first {
        output.push_str(" -> ");
        first = false
    }
    match item {
        LocItem::S(s) => output.push_str(&s),
        LocItem::I(i) => {
            output.push_str(itoa::Buffer::new().format(*i))
        },
    }
}

Required the itoa crate though - it would be helpful in a few more places (ryu would help with floats as well)
StrConstrainedValidator::_validation_logic if strip_whitespace and to_lower are true, then it will allocate twice, but only the latter actually requires allocating a new string.
Also, the signature implies an allocation, but it is not needed if e.g. min_length / max_length validation fails, so it might be better to use Cow instead.
py_error! uses format!, so I assume that calling to_string on its argument should not be necessary (like this one).
Are you ok with me submitting a PR for such things?
Static vs Dynamic dispatch. Is there any particular reason for using &'data dyn Input in the Validator trait? I'd assume it is possible to use &impl Input instead and avoid vtable indirection in cost of some compilation time.

The code for generating models here seems to be pretty slow compared to other validators, can anything be done?

inside with_prefix_location:
self.location = [location.clone(), self.location].concat();

One allocation could be avoided:
let mut new_location = location.clone();
new_location.extend_from_slice(&self.location);
self.location = new_location;

I'll take a look at other places during the weekend :)

That's so much, this is really interesting.
I didn't know allocation was particularly slow, I'll research it.
My initial guess is that most of these things will have a pretty small effect, e.g. so the truncate stuff is only called if you get an error and if print repr. I don't want to add more dependencies unless they allow a significant improvement.
The most significant thing is dyn Input Vs impl Input - if that makes a difference, it could make a big change.
PR very welcome.

Unfortunately using impl Input on Validators doesn't work because the trait needs to be object safe:
error[E0038]: the trait `Validator` cannot be made into an object
   --> src/validators/union.rs:12:22
    |
12  |     choices: Vec<Box<dyn Validator>>,
    |                      ^^^^^^^^^^^^^ `Validator` cannot be made into an object
    |
note: for a trait to be "object safe" it needs to allow building a vtable to allow the call to be resolvable dynamically; for more information visit <https://doc.rust-lang.org/reference/items/traits.html#object-safety>
   --> src/validators/mod.rs:189:8
    |
181 | pub trait Validator: Send + Sync + fmt::Debug {
    |           --------- this trait cannot be made into an object...
...
189 |     fn validate<'s, 'data>(
    |        ^^^^^^^^ ...because method `validate` has generic type parameters
...
196 |     fn validate_strict<'s, 'data>(
    |        ^^^^^^^^^^^^^^^ ...because method `validate_strict` has generic type parameters
    = help: consider moving `validate` to another trait
    = help: consider moving `validate_strict` to another trait

I would like to help too (been keen about helping this project for some time 😃 ). I would take a look and if I see some low hanging fruit will be posting (small) PRs. that is ok?
Are stylistic changes welcome too (thinks like iterators vs loops etc)?

Amazing yes please.

Are stylistic changes welcome too (thinks like iterators vs loops etc)?

yes, but my third priority (after safety and speed) is readability, particularly for python developers - e.g. me. So if changes make it technically more correct rust but make it harder to read for notices, and have no other impact, I might be inclined to refuse them.
Best to create some small PRs and I'll comment.

Are stylistic changes welcome too (thinks like iterators vs loops etc)?

That's not just a stylistic choice - avoiding bound checks means iterators can often be faster than index based loops.

Similarly, could we use a PyObject instead of PyAny or visa-versa and improve performance?

PyObject is just the owned version of PyAny - use the former whenever you need to keep around references to Python objects.

generally copying and cloning values - am I copying stuff where I don't have to? In particular, is input or parts of input (in the case of a dict/list/tuple/set etc.) copied when it doesn't need to be?

Where is this?

here and in a number of other implementations of ListInput and DictInput we do a totally unnecessary map, is this avoidable? Is this having a performance impact? Is there another way to give a general interface to the underlying datatypes that's more performance

If you just want to provide an interface, consider using a visitor rather than returning an iterator. This also avoids the boxing, but it doesn't really matter for performance I think.
For example, you have:
impl<'data> ListInput<'data> for &'data PyList {
    fn input_iter(&self) -> Box<dyn Iterator<Item = &'data dyn Input> + 'data> {
        Box::new(self.iter().map(|item| item as &dyn Input))
    }
    // ...
}

But you could also accept a closure to run over each item:
impl<'data> ListInput<'data> for &'data PyList {
    fn visit(&self, visitor: impl FnMut(???))  {
        self.iter().for_each(visitor)
    }
}

Or if you prefer something more typed:
trait Validator{
    fn validate(???)
}

impl<'data> ListInput<'data> for &'data PyList {
    fn visit(&self, visitor: impl Validator)  {
        self.iter().for_each(|item| visitor.validate(item))
    }
}

More generally I wonder if there are performance improvements that I'm not even aware of? "What you don't know, you can't optimise"

I mainly see some papercuts, not any actual performance crimes.
For example, in apply_validator, you have:
let loc = if key_loc {
    vec![key.to_loc(), "[key]".to_loc()]
} else {
    vec![key.to_loc()]
};
for err in line_errors {
    errors.push(err.with_prefix_location(&loc));
}

But if you fix the with_prefix_location to take a reference to a slice, rather than a reference to a Vec, you can avoid creating the vecs altogether:
if key_loc {
    let locs = [key.to_loc(), "[key]".to_loc()];
    for err in line_errors {
        errors.push(err.with_prefix_location(&locs));
    }

} else {
     let loc = key.to_loc();
     for err in line_errors {
        errors.push(err.with_prefix_location(std::slice::from_ref(&loc)));
    }
};

Note that the above comment holds true in general: because taking &Vec has more indirection and requires the caller to provide a vec rather than any continuous sequence you don't want this
fn foo(blahs: &Vec<Blah>){}

but this:
fn foo(blahs: &[Blah]){}

Recursive models are slowing than I had expected, I thought it might be the use of RwLock that was causing the performance problems, but I managed to remove that (albeit in a slightly unsafe way) in https://github.com/samuelcolvin/pydantic-core/pull/32 and it didn't make a difference. Is something else the problem?

    fn set_ref(&mut self, name: &str, validator_weak: ValidatorWeak) -> PyResult<()> {
        unsafe { Arc::get_mut_unchecked(&mut self.validator_arc).set_ref(name, validator_weak) }
    }

Functions like these are incredibly dangerous, they enable unsynchronized shared mutability. At least make set_ref unsafe. You could try using parking_lot's synchronization primitives - they are a bit faster.

Could we remove Arc completely?

Arc is just for sharing things. If you want to share things, you will need it or Rc.

lifetimes get pretty complicated, I haven't even checked if get a memory leak from running repeat validation, should we/can we change any lifetimes?

Do you mean Rust's lifetime annotations?

Also, I recommend to do benchmarking as part of your CI workflows, so you can identify performance regressions (or improvements :) ) quickly.

cc @woodruffw

I got some really helpful suggestions from @hoodmane, noting them here before i forgot:

Use an enum for validators, then use enum_dispatch, instead of dyn Validator, same for InputList
Use a slot table instead of weak references for recursive references
use tp_new to create classes.

This is a small thing, but getattr with constant strings can be slightly optimized by using intern!: https://docs.rs/pyo3/latest/pyo3/prelude/struct.Py.html#method.getattr
For example, this:
https://github.com/samuelcolvin/pydantic-core/blob/7c9058e641549e816614a097148019c11feae95e/src/validators/model_class.rs#L29
could become:
let new_method = class.getattr(intern!("__new__"))?;

...to save an allocation. I can make a small PR for these, if you'd like.

I didn't know allocation was particularly slow, I'll research it.
My initial guess is that most of these things will have a pretty small effect, e.g. so the truncate stuff is only called if you get an error and if print repr. I don't want to add more dependencies unless they allow a significant improvement.

Those are indeed micro-optimizations for some specific scenarios, but sometimes they may yield some nice cumulative effect :) I.e. if there is an opportunity to avoid heap allocations I usually prefer to take it
Here is another small Idea that may help a bit in some scenarios (with some cost of slowing down the others) - using a bitvector (e.g. u64) instead of a set for storing what fields were hit inside ModelValidator::validate (fields are ordered there). It might work for validators with <65 fields (which I think is quite common), for others, it could fall back to a regular set. The check_extra branch will require an actual set as those field names are arbitrary. The final Ok branch will require creating a set. It might not improve the happy path case when there are no validation errors, but maybe it could be considered if those other scenarios would need some extra performance (bitwise operations should be orders of magnitude cheaper than set operations)

@woodruffw thanks so much for the input. I think on this occasion this is superseded by #43.
I'll look at all the other suggestions here when I have time.

posted on r/rust for visibility: https://www.reddit.com/r/rust/comments/ugniwv/creator_of_pydantic_is_looking_for_help_in_moving/

I didn't know allocation was particularly slow, I'll research it.

An individual allocation is pretty quick, but it can add up.  You know how in python, you try to avoid patterns like:
s = "Something"
for line in lines():
    s += "\n" + line

and instead do something like:
s = "Something\n" + "\n".join(get_lines())

The main reason the first pattern is slow is because each iteration through the loop allocates a new string for s in addition to allocating each of the lines.  The second one only allocates a single extra string for the whole join function.
In rust, you can mutate strings, and strings amortize reallocations (in the same way dicts do in python) which means that
let mut s = String::from("Something\n");
for line in get_lines() {
    s.push_str(&line);
    s.push('\n')
}

is relatively performant, as it only has to reallocate roughly every time the string doubles in size.
If you know the size of all your inputs, you can often make it even faster by allocating the right size up front for the string:
let lines = get_lines();
let mut s = String::with_capacity(lines.iter().map(|s| s.len()).sum() + "Something\n".len());
s.push_str("Something\n");
for line in lines {
     s.push_str(&line);
     s.push('\n');
}

In this case the string never needs to reallocate, since it never outgrows its initial allocation.  As you said, readability for python programmers is a high concern, so the performance boost of the with_capacity version may not be worth the added bookkeeping.  Your call.  The point is that rust gives you more tools for managing allocations, which can help with performance.

impl<'a> ValLineError<'a> {
    pub fn with_prefix_location(mut self, location: &Location) -> Self {
        if self.location.is_empty() {
            self.location = location.clone();
        } else {
            // TODO we could perhaps instead store "reverse_location" in the ValLineError, then reverse it in
            // `PyLineError` so we could just extend here.
            self.location = [location.clone(), self.location].concat();
        }
        self
    }
}

This one uses needless clones. Looking at the code, it seems that almost all usages involve taking a vec![single_item], with a single vec![first, second] in apply_validator.rs. I suggest doing away with taking a slice entirely. Instead, make it a function which takes a single LocItem by value:
impl<'a> ValLineError<'a> {
    pub fn with_prefix_location(mut self, location: LocItem) -> Self {
        self.location.insert(0, location);
        self
    }
}

The old code involved unconditionally cloning a vector, which is a strong hint that you should have taken it by value (cloning a vector also involves cloning each of its elements, and LocItem can contain a String, so that's actually 2-3 needless allocations per call). The new approach eliminates those temporary vectors entirely.
If you expect that in the future you may need vectors longer than 2, or even ones of indeterminate length, you can consider taking an ArrayVec instead (see arrayvec, smallvec or tinyvec crates, I recommend tinyvec since it has no unsafe code).
While I have not benchmarked the crate and don't know of specific issues, in general excessive allocations can significantly harm the performance. For that reason I would try to avoid all allocations unless necessary: collecting into vectors, cloning Box'es, needlessly returning String's.
For example, Validator::get_name returns a String, even though all its uses just take its return value by reference, so could use a &str. Moreover, many impls of that method just create a String out of a static slice, needlessly allocating. I suggest making that method return Cow<'_, str>.
impl Clone for Box<dyn Validator> is a huge performance trap. Not only does it allocate new memory for the validator (this can actually be reasonably fast on modern allocators), but it also calls clone recursively on the validator (which in turn will call clones on inner validators, causing new allocations and clones). This is exponential in tree depth and should be strongly avoided. I would suggest using Arc<dyn Validator> where you are using Box<dyn Validator>, but unfortunately Validator::set_ref takes &mut self. This prevents calling it on Arc<dyn Validator>. While we could use Arc<RwLock<dyn Validator>> and acquire locks as needed, it would still likely be incorrect since currently all cloned validators are entirely independent, while with the Arc approach modifying the validator will also modifies everything which references it.
Unfortunately, there is no simple solution to this problem. It's something that should have been considered during the design phase.
You are also returning a lot of boxed trait objects. Besides extra allocations, those also cause extra indirection on method calls. Some of those trait objects probably can't be removed entirely, but some (where there are just a few implementing types, e.g. DictInput) can be converted into enumerations. This would allow to do away both with excessive boxing and with indirect trait calls, and it would also somewhat simplify the API.
If you want, I can make a PR with my suggestions.

For general performance ideas The Rust Performance Book may be useful. It's pretty short, and full of suggestions about things like how to avoid unnecessary allocations.

From readme:
https://github.com/samuelcolvin/pydantic-core/blob/e50eecc6853047adc478d82b7c2992fdc1a6c3d3/README.md?plain=1#L103-L104
This is not true at least for jsonschema-rs.
It can load dict and also json string (this lib has python bindings).
jsonschema-rs is also quite performat, so I think it may worth it to check how it achieves it.

Also please take a look at this issue from jsonschema-rs repo: simd support? (many ideas in it!)

jsonschema-rs is also quite performant, so I think it may worth it to check how it achieves it.

As the author of jsonschema-rs, I am inclined to think that fast validation is about having a packed, cache-friendly data layout that enables fast iteration and pre-computes as much as possible. The current approach in jsonschema-rs is building a serde_json::Value-like structure that contains all data need for validation (e.g. unwrapping Value::Number, using bitvecs for type combos, etc). The downside is that having such a fragmented layout with many boxes leads to cache misses. The current version doesn't resolve references upfront, which leads to synchronization overhead when locks are used :( I'd expect that a WIP version I haven't published yet might yield nice improvements because it stores the graph in the compressed sparse row format (so, 3 vectors total) and basically gives a nondeterministic finite automaton to execute the validation process. All references are also resolved upfront + partially inlined (the idea is borrowed from ajv)
Not sure how much it is applicable to this library, though it might be helpful.

Thanks both for your input, I'll definitely the simd-support issue and think more about what you've said here @Stranger6667.
@karolzlot, in terms of using jsonschema-rs (or any other JSON Schema library) directly in pydantic-core, I don't think it would ever be possible for the other reasons listed in the readme:
https://github.com/samuelcolvin/pydantic-core/blob/e50eecc6853047adc478d82b7c2992fdc1a6c3d3/README.md?plain=1#L96-L101
Most of all:

pydantic does coercion as well as validation
pydantic mostly works with python objects as input, not JSON

@samuelcolvin Yes, I agree with those points you mentioned.
What I mean is that it's possible that jsonschema-rs implemented this well ( "load dict and also load json string"), so it may be worth to check the code.
I don't know rust enough to give any more specific tip unfortunately.

Yes, agreed, thank you.

What I mean is that it's possible that jsonschema-rs implemented this well ( "load dict and also load json string"), so it may be worth to check the code.

I'd say that is suboptimal there - it converts a Python object into serde_json::Value, which sometimes takes up to 70% of all validation time. Having inputs behind an opaque wrapper that implements some trait should be much better, i.e. something like Input doesn't have that conversion overhead

I'm going to close this issue as the codebase has moved a long way since this discussion.
Thank you all for your input, it's been really useful and interesting
Any more feedback (probably best via new issues or PRs) is very welcome.
A lot of the suggestions here have now been implemented and from some quick check the code base is around 2x faster than when I created this issue (as well as significantly more complex and powerful).


az extension add --name aks-preview fails to install

If the issue is to do with Azure CLI 2.0 in-particular, create an issue here at Azure/azure-cli

Extension name (the extension in question)
aks-preview
Description of issue (in as much detail as possible)
Cannot update or install the aks-preview extension
Debug error shows issue for checksum
Downloaded to /var/folders/4y/p5llxl6d2x5gwhdnfbjzxr0w0000gn/T/tmpwkxks6xm/aks_preview-0.4.10-py2.py3-none-any.whl
Validating the extension /var/folders/4y/p5llxl6d2x5gwhdnfbjzxr0w0000gn/T/tmpwkxks6xm/aks_preview-0.4.10-py2.py3-none-any.whl
Invalid checksum for /var/folders/4y/p5llxl6d2x5gwhdnfbjzxr0w0000gn/T/tmpwkxks6xm/aks_preview-0.4.10-py2.py3-none-any.whl. Expected '412e936f0505550e1f2130ec45e412dcfbff61b6f2b0443295ab15c313341cfe', computed '156174affc440ebe2373007166b737e7807cf59719a51fed917129ff3be377ca'.

This appears to have been resolved. I was able to install the extension.
azure-cli                         2.0.71 *

command-modules-nspkg               2.0.3
core                              2.0.71 *
nspkg                              3.0.4
telemetry                          1.0.3

Extensions:
aks-preview                       0.4.12

Python location '/usr/local/Cellar/azure-cli/2.0.71/libexec/bin/python'
Extensions directory '/Users/jkucan/.azure/cliextensions'

Python (Darwin) 3.7.4 (default, Jul  9 2019, 18:13:23)
[Clang 10.0.1 (clang-1<IP_ADDRESS>)]

Legal docs and information: aka.ms/AzureCliLegal

It seemed that this issue was resolved. I will close it right now.

Please let us know if any further question, thanks.


Collaborators Pane

Show the list of collaborators on the side under the total space component.

No review link

The UI has been checked to see that it corresponds to the design.
the testing link when live is  link and in development is dev
Design - design


add to AbstractConverter method map to allow use another converters f…

In many use cases when use modelMapper in converter need access to MappingEngineImpl for re use already present converters.
But it not comfortable to use in converters this structure
mappingEngine.map(source, Types.deProxy(source.getClass()), null, TypeToken.<Destination>of(destinationType), null)
I think it will be useful feature when you can to reuse registered converters in any converter just using
map(Object source, Class<Destination> destinationType)

Thanks for the pull request. This pull request looks good to me. Can you use 2-space-indent instead of 4-space-indent to align with our current coding style? Thanks.

Thanks for the pull request. This pull request looks good to me. Can you use 2-space-indent instead of 4-space-indent to align with our current coding style? Thanks.

changed.

Hi recently I created pr for new feature that I think will be useful (I hope because I use it). But after solve discussion but I didn't see any respond, maybe now it will be useful ?Thats why I think this small feature will be useful ?

It's a good idea to have a method map in the AbstractConverter, but I think it will be better to remove the local variable MappingEngineImpl mappingEngine.
Any idea if  we can remove the local variable? Thanks!

thanks I'm do link static.

Hi @Merdoc97 ,
Sorry for the lately reply :(
And thanks for your contribution, I'm really appreciate it.
I still think it's not a good idea to introduce mappingEngine as a local variable of AbstractConverter.
For those cases which needs the mappingEngine to convert child inside the implementation of Converter, I suggest to implement Converter directly instead of extend AbstractConverter.
class ParentConverter implements Converter<ParentEntity, ParentDTO> {

    @Override
    protected ParentDTO convert(MappingContext<ParentEntity, ParentDTO> context) {
      ParentEntity source = context.getSource();
      ParentDTO dto = new ParentDTO();
      dto.setSecondName(source.getSecondName());
      dto.setName(source.getName());
      dto.setChild(context.getMappingEngine().map(source.getChild(), ChildDTO.class));
      return dto;
    }
  }

What do you think?


Adding new polkadot parachains

Efinity

Spoiler!

Centrifuge

Spoiler!

HydraDX
This network is blocking connection to wss://rpc-01.hydradx.io by Origin header which must equal https://polkadot.js

Spoiler!

Interlay

Spoiler!

Interlay tokens:
https://github.com/interlay/interbtc/blob/master/primitives/src/lib.rs#L473
Interlay existential deposit:
https://github.com/interlay/interbtc/blob/34c24961384208d8a22cef3446fea22d585c3ef7/parachain/runtime/interlay/src/lib.rs#L942

Interlay uses orml

fixed!

Interlay uses orml

Could you resolve your change request?


Say you can index as .ListTemplates("Name")

"Cannot" contradicts ListTemplates.Item Method (Word), which says that the index "Can be . . . the name of the individual object."
To confirm, I ran the following code without errors.
Set objA = ActiveDocument.ListTemplates.Add(OutlineNumbered:=True, Name:="TrialName")
Set objB = ActiveDocument.ListTemplates(Index:="TrialName")
Debug.Print objB.Name

italicize,
Thanks to your contribution to ensuring the validity of the VBA documentation. And thank you for testing the code as well.
Office Scale Team


change relative path in lenkf.j to abs path

This PR  addresses issue #742

Thanks, @weiyuan-jiang.  I tested successfully for LDAS_CONUS and also verified that I can submit the batch job from a directory other than the experiment ./run dir.

Approving, but let's coordinate with @mathomp4 and his work to re-org the GEOSldas repo (see #717) before merging

My moving-GEOSldas_App should be fairly compatible with this once I re-do it. I think Git will happily merge.


Feature request: sort the accounts

Maybe on the pane on the left that lists all the accounts, you can enable it to make the accounts sortable via drag-and-drop or some mechanism
Otherwise, nice job on 2.x

@chrisgo It's a nice to have feature (we had that in 1.x). I started implementing that with this PR #58.
Still have some small SwiftUI problem but it works as expected. I'll include this in a future 2.1 release.

Merged.

Thank you sir!

@chrisgo FYI v2.1.0 has been approved and will be available soon.


Merge master branch

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/5815/
Test PASSed.


Missing main entry in package @sveltejs/adapter-node

Describe the bug
I think someone forgot to publish the main entry of package @sveltejs/adapter-node in the version 1.0.0-next.13.

Logs
After I ran npm run dev, I got the following error.
Cannot find module<EMAIL_ADDRESS>Please verify that the package.json has a valid "main" entry
Error: Cannot find module<EMAIL_ADDRESS>Please verify that the package.json has a valid "main" entry
    at tryPackage (node:internal/modules/cjs/loader:344:19)
    at Function.Module._findPath (node:internal/modules/cjs/loader:557:18)
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:910:27)
    at Function.Module._load (node:internal/modules/cjs/loader:769:27)
    at Module.require (node:internal/modules/cjs/loader:996:19)
    at require (node:internal/modules/cjs/helpers:92:18)
    at Object.<anonymous> (/path/to/my/project/svelte.config.cjs:2:14)
    at Module._compile (node:internal/modules/cjs/loader:1092:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1121:10)
    at Module.load (node:internal/modules/cjs/loader:972:32)
 ERROR  Command failed with exit code 1.

To Reproduce
This is very easy to reproduce with @sveltejs/adapter-node 1.0.0-next.13. Just run dev mode and you will get this error.
Expected behavior
@sveltejs/adapter-node should be imported without any errors.
Stacktraces
N/A
Severity
This bug makes the package unusable.
Additional context
N/A

That bug was hotpatched about 2 minutes before you opened the issue, but currently we've hit a CI bug where it's not publishing the updated package. 😬

Published.

Great! Thanks for your quick response!


bug: lazydev.nvim notifies the same thing on every nvim launch

Did you check docs and existing issues?

[X] I have read all the lazydev.nvim docs
[X] I have searched the existing issues of lazydev.nvim
[X] I have searched the existing issues of plugins related to this issue

Neovim version (nvim -v)
0.10.0
Operating system/version
Archlinux
Describe the bug
i get this notification whenever i launch neovim it is going on since last 1 or 2 updates in lazydev.nvim
~/.config/lazyvim

/usr/share/nvim/runtime/lua
LazyVim ~/.local/share/lazyvim/lazy/LazyVim/lua
~/.local/share/lazyvim/lazy/luvit-meta/library

Steps To Reproduce

Open neovim if you have this plugin enabled
Lazy Sync
restart nvim
try it with multiple lazyvim configurations and all will have this issue

i Lazy sync'ed  in my main config and i had this notification popup everytime, then i tried to reproduced in my Lazyvim starter templete and it happened there too
Expected Behavior
it shouldn't pop up this notification
Repro
No response

Yeah, I accidentally pushed an update where debug was enabled. Sorry!

So just update and it will be gone.


Support returning any from callbacks

Feel free to take this over or close this or make suggestions or whatever!
Closes #1045

Codecov Report

Merging #1046 (bd773ec) into master (aa1e904) will decrease coverage by 0.18%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #1046      +/-   ##
==========================================
- Coverage   46.04%   45.85%   -0.19%
==========================================
  Files          11       11
  Lines        1490     1496       +6
==========================================
  Hits          686      686
- Misses        667      673       +6
  Partials      137      137

Impacted Files
Coverage Δ

callback.go
56.43% <0.00%> (-1.73%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update aa1e904...bd773ec. Read the comment docs.

@rittneje I'm okay to merge.

Thanks folks!


Remove Gaylord from list of names

Because the domain faker uses this list, I ended up with gaylord.com – which out of context can look deliberate and when our tests are client-facing, looks derogatory.

That's the beauty of randomness! Gaylord IS a valid english name, see https://en.wikipedia.org/wiki/Gaylord. Besides, what's wrong about http://www.gaylord.com?


Check explicitly for equals 0 in packed division kernel.

This PR fixes https://github.com/tensorflow/tfjs/issues/1321
I don't understand why straightforward equality check fails, and I also don't understand which op is bringing in unused channels from the result of calling div - it's not matmul because we already have a test for that: https://github.com/tensorflow/tfjs-core/blob/master/src/ops/binary_ops_test.ts#L23
I created an issue (https://github.com/tensorflow/tfjs/issues/1324) for better understanding the root cause of this. I tried a fix where we check whether a channel is out of bounds and simply set it to zero if so but that didn't seem to fix Shanqing's bug.
Hopefully this is enough of a band-aid for now.

This change is

cc @caisq

@nsthorat yes - environment epsilon is not available at that point :/

@nsthorat I ended up removing the test because (1) it doesn't fail on master so does not actually expose the root cause and (2) I was struggling to get readPixels to work with bytes for Safari (somehow just evaluating the environment flag 'is_download_float_enabled' within the test was enough to cause Chrome to fail?)

@annxingyuan thanks for investigating the issue and putting together a fix quickly. To make sure I understand the problem in the lstm-text-generation example: do division-by-zero actually happen in that example? From the fact that this works with older versions of TF.js and with the WEBGL_PACK flag gets rid of the problem, it seems that the division-by-zero isn't due to that particular model or the data, is it? (Just making sure there isn't a bug in the tfjs-example)

@caisq No it does not appear that division by 0 occurs in the lstm-text-generation example. Or at least, if the denominator is 0 somewhere, the numerator is also 0 (which means they are equal and we return 1). So the example seems fine.

Let's close this one and use https://github.com/tensorflow/tfjs-core/pull/1605 instead.


lack of documentation on how to add components

It would be great if you could document that it may need to copy the symbol (*.asy) to Resource/sym/ with changing the extension to txt.
% cp ~/Library/Application\
% Support/LTspice/lib/sym/OpAmps/LF000.asy \
  ~/Library/Python/3.8/lib/python/site-packages/ltspice2svg/Resource/sym/OpAmps/LF000.txt

Hey @SaitouToshihide, please allow some time. I will go through and update the code. Thank you for indicating problem.


Fullscreen icon missing in deployed pages

https://cdn.rawgit.com/rajsite/i3-docs/c8c2ac622f2b54d17ac69dbfb5572d088b983baf/docs/index.html

self-closing stale issue 👍


LinkFix: windows-driver-docs-ddi (2022-09)

This PR is the result of running a Link Repair script on the included content. This script is primarily being run to clean up links to function correctly in Air Gapped Clouds (AGC). Here's a breakdown of what the script that generated the changes in this PR fixes:

docs.microsoft.com Fully Qualified Domain Name (FQDN) Links to either Relative (if in same Repo/DocSet/ContentSet) and Site Relative if not

azure.microsoft.com/documentation/articles that reflect to Docs to either Relative (if in same Repo/DocSet/ContentSet) and Site Relative if not

All known flavors of MSDN/TechNet domains (we know of 8 or 9 now) that reflect to Docs to either Relative (if in same Repo/DocSet/ContentSet) and Site Relative if not

This also includes cleanup of the appended query string for MSDN (redirectedfrom=MSDN).

Fixes articles that are redirected in .openpublishing.redirection.json and updates the link to the current final location (this has a lot of customer usability fixes).

Please note the Contributor Guide: Links has been updated with the following link-type preference order:

By order of preference, links hosted on Docs should be Relative if they are in the same repository and docset. If they are in a different docset, even if in the same repository, they should be Site Relative. Links to content hosted on Docs shouldn't use a complete URL, otherwise known as Fully Qualified Domain Names (FQDN). Using a complete URL from Docs to other content on Docs will cause that link to be non-functional in air-gap environments.

#sign-off

Hi @cmcclister I'm seeing zero modified files in this PR.  Is that expected?

Hi @tedhudek I am closing this PR and replacing with this new one: https://github.com/MicrosoftDocs/windows-driver-docs-ddi/pull/1358


ERROR in Generating Greek language images

Traceback (most recent call last):
File "C:\Users\Muhammad Hamza\AppData\Roaming\Python\Python39\site-packages\synthtiger\gen.py", line 121, in _generate
data = template.generate()
File "E:\datasetPADDLE\examples\synthtiger\template.py", line 113, in generate
fg_image, label, bboxes, glyph_fg_image, glyph_bboxes = self._generate_text(
File "E:\datasetPADDLE\examples\synthtiger\template.py", line 217, in _generate_text
label = self.corpus.data(self.corpus.sample())
File "C:\Users\Muhammad Hamza\AppData\Roaming\Python\Python39\site-packages\synthtiger\components\wrapper\selector.py", line 29, in sample
idx = meta.get("idx", self._sample_idx())
File "C:\Users\Muhammad Hamza\AppData\Roaming\Python\Python39\site-packages\synthtiger\components\wrapper\selector.py", line 58, in _sample_idx
idx = np.random.choice(len(self.components), replace=False, p=self._probs)
File "mtrand.pyx", line 932, in numpy.random.mtrand.RandomState.choice
ValueError: 'a' and 'p' must have same size
kindly someone help

Hi,
It seems that the number of corpus weights is different from the number of corpus.
Please check config file.
https://github.com/clovaai/synthtiger/blob/master/examples/synthtiger/config_horizontal.yaml#L16
This example has two corpus (length augmentable corpus, char augmentable corpus) and two corpus weights (1, 0).

Hi,
It seems that the number of corpus weights is different from the number of corpus. Please check config file.
https://github.com/clovaai/synthtiger/blob/master/examples/synthtiger/config_horizontal.yaml#L16 This example has two corpus (length augmentable corpus, char augmentable corpus) and two corpus weights (1, 0).

Sorry, can you guide what to change, im trying to generate greek images from corpus, im using this config file, i change the weight but i dont know why still getting the same error
coord_output: true
mask_output: true
glyph_coord_output: true
glyph_mask_output: true
vertical: false
quality: [50, 95]
visibility_check: true
midground: 0.25
midground_offset:
percents: [[-0.5, 0.5], [-0.5, 0.5]]
foreground_mask_pad: 5
corpus:
weights: [1]
args:
# length augmentable corpus
- paths: [resources/corpus/ge.txt]
weights: [1]
min_length: 1
max_length: 25
textcase: [lower, upper, capitalize]
augmentation: 0.1
augmentation_length: [1, 25]
# char augmentable corpus
- paths: []
weights: []
min_length: 1
max_length: 25
textcase: [lower, upper, capitalize]
augmentation: 0
augmentation_charset: resources/charset/alphanum_special.txt
font:
paths: [resources/font]
weights: [1]
size: [40, 80]
bold: 0.5
texture:
prob: 0.5
args:
paths: [resources/image]
weights: [1]
alpha: [0, 1]
grayscale: 0
crop: 1
colormap2:
paths: [resources/colormap/iiit5k_gray.txt]
weights: [1]
k: 2
alpha: [1, 1]
colorize: 1
colormap3:
paths: [resources/colormap/iiit5k_gray.txt]
weights: [1]
k: 3
alpha: [1, 1]
colorize: 1
color:
gray: [0, 255]
alpha: [1, 1]
colorize: 1
shape:
prob: 1
args:
weights: [1, 1]
args:
# elastic distortion
- alpha: [15, 30]
sigma: [4, 12]
# elastic distortion
- alpha: [0, 2]
sigma: [0, 0.6]
layout:
weights: [4, 1]
args:
# flow layout
- space: [-2, 5]
line_align: [middle]
# curve layout
- curve: [20, 40]
space: [-2, 5]
convex: 0.5
upward: 0.5
style:
prob: 0.25
args:
weights: [1, 2, 2]
args:
# text border
- size: [1, 12]
alpha: [1, 1]
grayscale: 0
# text shadow
- distance: [1, 6]
angle: [0, 360]
alpha: [0.3, 0.7]
grayscale: 0
# text extrusion
- length: [1, 12]
angle: [0, 360]
alpha: [1, 1]
grayscale: 0
transform:
prob: 1
args:
weights: [1, 1, 1, 1, 1, 1, 2]
args:
# perspective x
- percents: [[0.5, 1], [1, 1]]
aligns: [[0, 0], [0, 0]]
# perspective y
- percents: [[1, 1], [0.5, 1]]
aligns: [[0, 0], [0, 0]]
# trapezoidate x
- weights: [1, 0, 1, 0]
percent: [0.75, 1]
align: [-1, 1]
# trapezoidate y
- weights: [0, 1, 0, 1]
percent: [0.5, 1]
align: [-1, 1]
# skew x
- weights: [1, 0]
angle: [0, 30]
ccw: 0.5
# skew y
- weights: [0, 1]
angle: [0, 10]
ccw: 0.5
# rotate
- angle: [0, 10]
ccw: 0.5
pad:
prob: 1
args:
pxs: [[0, 10], [0, 10], [0, 10], [0, 10]]
postprocess:
args:
# gaussian noise
- prob: 1
args:
scale: [4, 8]
per_channel: 0
# gaussian blur
- prob: 0.9
args:
sigma: [0, 2]
# resample
- prob: 0.1
args:
size: [0.4, 0.4]
# median blur
- prob: 1
args:
k: [1, 1]

Please change this line.
from
corpus:
  weights: [1]

to
corpus:
  weights: [1, 0]

Or, remove this lines.
    # char augmentable corpus
    - paths: []
      weights: []
      min_length: 1
      max_length: 25
      textcase: [lower, upper, capitalize]
      augmentation: 0
      augmentation_charset: resources/charset/alphanum_special.txt

Please change this line.
from
corpus:
  weights: [1]

to
corpus:
  weights: [1, 0]

Thank you soo much it worked
Also it gives this error can we prevent this
RuntimeError: Text is not visible

You may ignore this message RuntimeError: Text is not visible.
Please see https://github.com/clovaai/synthtiger/issues/35.

oky thank you soo much now i can make greek recognition model

You may ignore this message RuntimeError: Text is not visible. Please see #35.

@Ham714  I am having trouble incorporating a new font for the language: "Gujrati" ; font name: Shruti
Could you please guide me on how did you add a new language Greek and its font? :)


Bug: Copying data from Postgres via Pandas crashing the entire PC

Kùzu version
V0.6.0
What operating system are you using?
Windows 11
What happened?
I was trying a COPY operation on a pandas data frame.
First my monitor size changed, enlarging everything, then the screen went black.
The data frame was very large about 21 GB in size, filling up 96% of my RAM.
The issue doesn’t go away even if I chunk down the data frame, just delays it.
The PC automatically restarts. The screen aspect is permanently changed.
Edit: further investigation shows a DWM crash followed by the PC no longer detecting the GPU.
Are there known steps to reproduce?
No response

Hi @Gilnore , sorry for the crash of your machine. This is weird to me. We haven't encountered any cases where the running/crash of Kuzu leads to changed settings of monitors, or restart, or crash of GPU. We can investigate into this a bit more to see where went wrong. Could u kindly provide a schema of the data frame? We can try to populate a large data frame to that size to reproduce the issue.
Also, if you can provide the hardware configuration of your machine would be great.
@benjaminwinger can help look into this issue when he got time.

Hi @Gilnore , sorry for the crash of your machine. This is weird to me. We haven't encountered any cases where the running/crash of Kuzu leads to changed settings of monitors, or restart, or crash of GPU. We can investigate into this a bit more to see what went wrong. Could u kindly provide a schema of the data frame? (if it's possible to dump the data frame to file and ship to us would be helpful) We can try to populate a large data frame to that size to reproduce the issue. Also, if you can provide the hardware configuration of your machine would be great.
@benjaminwinger can help look into this issue when he got time.

Thanks for the reply. I loaded an entire SQL table with pandas read sql table. The table was in Postgres. I can provide the sqlalchemy code that generated the table. (I’ll need to quickly switch devices first, got to reboot the PC, I’ll attach the code in a separate comment).

here's the code that generated the sql table:
@architect.install('partition',
type='range',
subtype='date',
constraint='100000',
column='date_time',
db = postgres_url)
class history(postgres_base):
tablename='history'
identity: Mapped[stock_id] = composite(mapped_column('date_time',
sa.TIMESTAMP,
index=True),
mapped_column('symbol',
sa.String,
index=True))
Open:Mapped[float] = mapped_column('Open')
Close:Mapped[float] = mapped_column('Close')
High:Mapped[float] = mapped_column('High')
Low:Mapped[float] = mapped_column('Low')
Volume:Mapped[float|None] = mapped_column('Volume',nullable=True)
Dividends:Mapped[float|None] = mapped_column('Dividends',nullable=True)
Splits:Mapped[float|None] = mapped_column('Splits',nullable=True)
table_args = (sa.PrimaryKeyConstraint('date_time','symbol',name='identity'),
# {'postgresql_partition_by': 'RANGE (date_time)'}
)
here's the code that generates the graph:
def create_node_tables(self, table, kuzu_conn):
#make nodes of various types by putting together some tables in sql
columns = []
table_lable = table.tablename
#we need to add primary key statments
#we can try to decypher the identity arg of table
primaries = [key.name for key in table.mapper.primary_key]
typed_primaries = []
# Loop through columns in the table
for col in table.table.columns:
col_name = col.name
col_type = col.type
# Map SQLAlchemy types to Kuzu types (expand as needed)
kuzu_type = self.map_sa_type(col_type)
if col_name in primaries:
typed_primaries.append(f"{col_name} {kuzu_type}")
columns.append(f"{col_name} {kuzu_type}")
else:
columns.append(f"{col_name} {kuzu_type}")
#we need to pack the primary key columns into a struct
#if the primary key is a composite key, ie if len(primaries) > 1
if len(typed_primaries)>1:
# Join columns into the CREATE NODE statement
columns_str = ", \n".join(columns)
#then in the statment a composite key refers to a struct
# primaries_str = ', '.join(typed_primaries)
#unfortunatly the id must be string or numerical, so we're using
#a seregant key
create_node_statement = f"""CREATE NODE TABLE IF NOT EXISTS
{table_lable}
(id SERIAL,
{columns_str},
PRIMARY KEY (id)
);"""
else:
# Join columns into the CREATE NODE statement
columns_str = ", \n".join(columns)
#
create_node_statement = f"""CREATE NODE TABLE IF NOT EXISTS
{table_lable}
({columns_str},
PRIMARY KEY ({primaries[0]})
);"""
kuzu_conn.execute(create_node_statement)
def create_edge_tables(self,source, target, kuzu_conn):
    #source and target need to be specified if table is not
    #the only table on edge is DOT
    a = source.__tablename__
    b = target.__tablename__
    create_edge_statement = f"""
    CREATE REL TABLE IF NOT EXISTS {a}_effect_on_{b} (FROM {a}
                                   TO {b},
                                   source_time TIMESTAMP,
                                   target_time TIMESTAMP,
                                   correlation_index DOUBLE,
                                   exchange_index DOUBLE);"""
    kuzu_conn.execute(create_edge_statement)

def read_to_node(self,table, kuzu_conn, chunksize = 0, sample = False):
    #pass in a sql table object and a kuzu connection
    table_name = table.__tablename__
    if table_name == 'history':
        #problem, the table is not detected this way.
        con = self.history_session.connection()
    else:
        con = self.session.connection()
    if chunksize > 0:
        df = pd.read_sql_table(table_name, con,chunksize=chunksize)
        #df should now be an iterator each returning 50000 rows
        columns = kuzu_conn._get_node_property_names(table_name)
        columns = list(columns.keys())
        if 'id' in columns:
            columns.remove('id')
        for data in df:
            #reordering dataframe
            data = data[columns]
            try:
                kuzu_conn.execute(f"COPY {table_name} FROM data")
            except Exception as e:
                print(f'\n\n\n error when dealing with {table_name}')
                raise e
            if sample:
                break
    else:
        df = pd.read_sql_table(table_name, con)
        columns = kuzu_conn._get_node_property_names(table_name)
        columns = list(columns.keys())
        if 'id' in columns:
            columns.remove('id')
        df = df[columns]
        try:
            kuzu_conn.execute(f"COPY {table_name} FROM df")
        except Exception as e:
            print(f'\n\n\n error when dealing with {table_name}')
            raise e

    print(f'{table_name} nodes allocated.')

also it seems kuzu.Database('econ_graph') encounters a unknown error and crashes, even if I were to exclude the problematic table, but everything works perfectly fine if i were to reduce the number of nodes to 100 using the sampling function of the read_to_node method.

most of this code is modified from chat-gpt, so there might be some bad notations and habits like using f strings and underscored, i don't know if that had to do with anything.

Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz   2.59 GHz with integrated UHD graphics 630
RAM	32.0 GB (31.8 GB 可用)
Nvidia GTX 1660Ti GPU
my PC is in Chinese, I don't know if the language of OS have anything to do with it, but i can't change the language settings on Windows 11 home.

I should also mention when using chunk size of 50,000 , the PC crashes after running for 48 hours. The COPY operation would probably take longer. It is also somehow eating space on my C drive, when kuzu is installed completely on a microconda virtual environment in E drive. The PostgreSQL database was in D drive. Docker is in C for running kuzu-explorer, but it wasn't running.

I went and ran a SELECT COUNT (*) for the problematic table, there are 132123860 rows.

I tried again after clearing out some space (40Gb) in C drive in case there’s some type of memory leak or something that causes all the space to fill up.
Problem persists, first task manager crashes, then 10Gb of C drive memory is instantly occupied, then a mosaic coloured square appears in the screen centre, then the computer becomes locked and is unable to be unlocked.
I restarted the computer at that point trying to prevent another disconnection of GPU.

Problem persists

That’s what I managed to get as it crashed, screen did go black, but GPU didn’t disconnect this time.

Further reduction of cpu count to 1/4 results only in crash of IDE and program. I’ll now try no multithreading.

Never mind, it seems reducing cores didn’t help. What did was not opening task manager. It seems that as long as you have task manager open when the program runs, the crash is always bad enough to cause black screen.

Hi @Gilnore, as an alternative to using Pandas read_sql which is entirely in memory, have you considered using Kùzu's Postgres extension to directly scan the contents of a Postgres table and copy it into Kùzu?
https://docs.kuzudb.com/extensions/attach/rdbms/
Looking at your code, all the operations to move data between your Postgres source and Kùzu is done from within Python, which explains the high memory usage that leads to the crash.
We haven't done extensive benchmarking on scanning performance from Postgres tables, but it should be better than relying on Pandas for that large a dataset, not to mention that it should required far less glue code. Curious to know if this approach solves your crash and memory issues.

Hi @Gilnore, as an alternative to using Pandas read_sql which is entirely in memory, have you considered using Kùzu's Postgres extension to directly scan the contents of a Postgres table and copy it into Kùzu? https://docs.kuzudb.com/extensions/attach/rdbms/
You can copy subsets of your Postgres table into Kùzu by using a LOAD FROM subquery, as shown in the docs:
COPY Person FROM (LOAD FROM pg_table.person RETURN name);

Looking at your code above, all the operations to move data between your Postgres source and Kùzu is done from within Python, which explains the high memory usage that leads to the crash.
We haven't done extensive benchmarking on scanning performance from Postgres tables, but it should be better than relying on Pandas for that large a dataset, not to mention that it should require far less glue code and Python loops. Curious to know if this approach solves your crash and memory issues.

 (Chat GPT really let me down XD)

I keep running into "postgres_scanner.duckdb_extension not found, Extension "postgres" is an existing extension." when running the attach statement for Postgres, and Unicode error when running install statements for SQLite.
the same error occurs for Postgres if I didn't run the install statement for it, but with another line like the following:
"postgres_scanner.duckdb_extension" not found.
Extension "postgres" is an existing extension.
Install it first using "INSTALL postgres".

Hi @Gilnore
Extensions are currently not working for v0.6.0.
To use the postgres extension in kuzu, you can install our latest dev build by:
pip install kuzu==0.6.1.dev58

I think this will be impossible for us to replicate on our side and I also don't think even if this was an issue on Kuzu's side, I cannot think of how we would be crashing other processes (and I doubt we are, the problem can be somewhere else).
The only thing I can think of right now is to not use Kuzu or any other DBMS in a system if a large chunk of RAM is already occupied by other processes. So if 21GB is already used by pandas, Kuzu won't be able to get enough basic memory to operate (Kuzu's buffer manager by default tries to grab 80% of your RAM). So the setting is not ideal. You can try to set the buffer manager to a smaller size but not to a few GB if your data is tens of GB.

I see, thanks for everything. I'll continue with the extensions.


Fix headline bug and add image support

Add image tags support
Add DIV tags support (will be tricked as general paragraph)
Fix a problem with the lack of \n\n before the headline so that the headline may be rendered to the last paragraph

BTW. The Xcode 16.0 reports an error indicating that the firstMatch(in: _) function may not be available in lower than iOS 16 (Sources/SwiftHTMLtoMarkdown/BasicHTML.swift:98). I have no idea if it is a critical issue because I have no errors encountered when I use this library with network reference. The error appears when I clone the library to the local and import it as a local library.
Here is a suggested fix authored by Xcode:

Thank you for making this PR! What are the changes to the Headers? I am seeing tests for those failing so I am wondering if the tests are incorrect because output was originally flawed.


Fast trainer "no metadata" error?

I decided to try the fast kohya trainer colab, but got a strange error:
Traceback (most recent call last):   File "/content/kohya-trainer/train_network.py", line 507, in <module>     train(args)   File "/content/kohya-trainer/train_network.py", line 69, in train     train_dataset = FineTuningDataset(args.in_json, args.train_batch_size, args.train_data_dir,   File "/content/kohya-trainer/library/train_util.py", line 828, in __init__     raise ValueError(f"no metadata / メタデータファイルがありません: {json_file_name}") ValueError: no metadata / メタデータファイルがありません: /content/drive/MyDrive/training_dir/meta_lat.json Traceback (most recent call last):   File "/usr/local/bin/accelerate", line 8, in <module>     sys.exit(main())   File "/usr/local/lib/python3.8/dist-packages/accelerate/commands/accelerate_cli.py", line 45, in main     args.func(args)   File "/usr/local/lib/python3.8/dist-packages/accelerate/commands/launch.py", line 1104, in launch_command     simple_launcher(args)   File "/usr/local/lib/python3.8/dist-packages/accelerate/commands/launch.py", line 567, in simple_launcher     raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/usr/bin/python3', '/content/kohya-trainer/train_network.py', '--output_name=OFF', '--pretrained_model_name_or_path=/content/pretrained_model/anything-v3-fp32-pruned.safetensors', '--vae=/content/vae/anime.vae.pt', '--train_data_dir=/content/drive/MyDrive/Goku', '--in_json=/content/drive/MyDrive/training_dir/meta_lat.json', '--output_dir=/content/drive/MyDrive/training_dir/output', '--network_dim=128', '--network_alpha=128', '--network_module=networks.lora', '--unet_lr=0.0001', '--text_encoder_lr=5e-05', '--optimizer_type=AdamW8bit', '--learning_rate=2e-06', '--lr_scheduler=constant', '--lr_warmup_steps=250', '--dataset_repeats=10', '--resolution=512', '--lowram', '--noise_offset=0.1', '--mixed_precision=fp16', '--save_precision=fp16', '--save_n_epoch_ratio=3', '--save_model_as=safetensors', '--train_batch_size=4', '--max_token_length=225', '--max_train_epochs=20', '--clip_skip=2', '--logging_dir=/content/training_dir/logs', '--log_prefix=OFF', '--shuffle_caption', '--xformers']' returned non-zero exit status 1.
What could this "no metadata" error be about?
How might i fix it?
I certainly connected my drive and i am sure i put the folder name and path correct, but i keep running into this issue.
Also, my images have their own .txt.

Having the same issue since the new update with the 1-click kohya trainer and the dreambooth trainer lora using almost the default options, i suppose is a general issue, hope they could say how fix this

I believe this is where the error is:
Status Legend:
(OK):download completed.
Skipping directory 5_Goku
2023-03-05 16:07:40.898578: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-03-05 16:07:41.951879: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.8/dist-packages/cv2/../../lib64:/usr/lib64-nvidia
2023-03-05 16:07:41.952017: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/python3.8/dist-packages/cv2/../../lib64:/usr/lib64-nvidia
2023-03-05 16:07:41.952038: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
found 0 images.

I have no idea why, but the colab is skipping my training folder.

I think I'm able to make it by

changing the code

if install_xformers:
!pip install xformers==0.0.17 triton==2.0.0
!pip install torch==2.0.0
!pip uninstall torchvision
!pip install torchvision -f https://download.pytorch.org/whl/cu111/torch_stable.html

Make sure the image file in put in the training_dir folder
enable Auto-captioning
use_wd_tagger (checked)
use_blip (checked)

I'm sorry, I don't have any plans to continue the Fast Kohya Trainer project (for now).

The code is getting longer in one cell, and my device is not powerful enough to handle it, (I run most of things in small laptop), which is causing lag.

It's really hard to maintain the 1-click cell colab.

There is another good 1-click cell colab project maintained by HollowStrawBerry, which is integrated with Google Drive. It based on my notebook (?) so you may get the same experience, or even better.You can check it out here: https://github.com/hollowstrawberry/kohya-colab


feature projection node for Traffic light enablement : Help needed

I am trying to setup the feature projection node:
The node uses camera intrinsic parameters [ fx,fy,cx,cy] to translate signal points in map coordinates to pixel units and computes u,v.  It also computes position, orientation of camera. I had to add a static transform between velodyne and camera to ensure pose of camera wrt map was computed using tf package. Is this the right approach? Also my signal pose has values like [-154,-168,13]. As you can see they are negative and so my u,v values are computed negative and fail the project2 method on the condition. How do I make this work?
Also does feature projection only work if camera is calibrated and the ros camera driver loads the calibration yml file ?

I did publish the calibration file and now I do not have to setup the static transform. However, although my camera is forward  facing and translation parameters between velodyne and camera are very low, I do not get matching camera pose as that of lidar. Example when lidar is at -139, -160, camera is at -139, -87. Can you throw some light on this? Should I use the adjust x, y parameters to shift u,v values. ? They compute as 9000, 831 where my image width is 600 * 800 at my stopline. My camera can view the signal at the stop line.  Will the adjustments ensure the signal centroid is visible on image plane.? I do see the transform between /velodyne and /camera being computed correctly using rosrun tf tf_echo /camera_id /velodyne .


ProfilerHook and loading libcupti.so cause Ubuntu to completely freeze

OS Platform and Distribution: Ubuntu 14.04 LTE
TensorFlow version: 1.14
Bazel version: 0.9.0
CUDA/cuDNN version: 8.0/7.0.5
GPU model and memory: GeForce GTX1060 - 6070MB
Exact command to reproduce: python3.4 -m music_modeling
--

I added "ProfilerHook" to Estimator for recording GPU memory consumption; but it always causes my Ubuntu to freeze indefinitely and Ubuntu never makes away with it.
Here is the source code:
import tensorflow as tf

from tensorflow.python.layers.core import dense
from tensorflow.python import debug as tf_debug

from data.music_data_reader import MusicDataReader
from model.tf_msa_rnn import dynamic_msa_rnn

FLAGS = tf.app.flags.FLAGS
tf.app.flags.DEFINE_string("mode", tf.estimator.ModeKeys.TRAIN,
                           """"Is training or testing mode""")
tf.app.flags.DEFINE_string("model_dir", './msa_model',
                           """"Directory in where checkpoints are stored""")
tf.app.flags.DEFINE_integer("batch_size", 20,
                            """Number of samples in a batch""")
tf.app.flags.DEFINE_integer("num_epochs", 100,
                            """"How many times the whole training set has to be fed into network""")
tf.app.flags.DEFINE_string("log_directory", './log_dir',
                           """"Directory in where logs and checkpoints are stored""")
# *************************************
# *************************************Configuration options for the network
# *************************************
tf.app.flags.DEFINE_integer("num_units", 30,
                            """"# of hidden units in an LSTM cell""")
tf.app.flags.DEFINE_integer("num_msa_feats", 10,
                            """"# of MS features to be learned""")
tf.app.flags.DEFINE_integer("signal_len", 100,
                            """"Length of the signal at a time to be processed by
                                multi-scale analyzer
                            """)
tf.app.flags.DEFINE_integer("dim_pitch", 88,
                            """"# of hidden units in an LSTM cell""")
# *************************************
# *************************************Configuration options for dataset
# *************************************
tf.app.flags.DEFINE_string("dir_path",
                           './music_samples/MuseData',
                           """"Absolute path to the music files for reading training/testing samples""")
tf.app.flags.DEFINE_integer("pitch_low",
                            21,
                            """"Low pitch value""")
tf.app.flags.DEFINE_integer("pitch_high",
                            109,
                            """"High pitch value""")
tf.app.flags.DEFINE_float("dt",
                          0.3,
                          """"Not sure yet...""")

def loss_fn(y_pred, y_true):
    '''

    :param y_pred: Logits predicted by the model
    :param y_true: Correct values corresponding each prediction
    :return:
    '''

    y_pred = tf.log(tf.nn.softmax(y_pred, name="probs_tensor"))  # [BSxMTxOS] -- Probabilities
    p_trun = y_pred[:, 0:-1, ...]  # x'[1], x'[2], ..., x'[N-1]
    t_trun = y_true[:, 1:, ...]  # x[1], x[2], ..., x[N-1]
    loss = tf.reduce_sum(p_trun*t_trun, axis=2)  # [BSxMT] -- Dot product between 3rd dimensions
    loss = tf.reduce_mean(loss, name="piano_roll_loss")  # loss function -- returns a scalar
    tf.summary.scalar("loss_fn", loss)  # Add summary for the loss
    loss = tf.Print(loss, [loss], "Loss: ")
    return loss

def model_fn(features,
             mode=tf.estimator.ModeKeys.TRAIN,
             params=None):

    print("Creating Model...")
    input_ph = tf.reshape(features['input_ph'], [FLAGS.batch_size, params['max_seq'], FLAGS.dim_pitch])
    seq_len_ph = tf.reshape(features['seq_len_ph'], [FLAGS.batch_size])
    outputs, state = dynamic_msa_rnn(FLAGS.batch_size,
                                     input_ph,
                                     seq_len_ph,
                                     params['max_seq'],
                                     FLAGS.signal_len,
                                     [20, 10],  # Number of filters per layer
                                     [11, 13],  # Kernel size for each layer
                                     [3],  # Pooling size for each layer
                                     FLAGS.num_msa_feats,
                                     FLAGS.num_units,
                                     activation=tf.nn.tanh,
                                     initializer=tf.glorot_normal_initializer())  # [BSxMTxOS], [BSxSS]
    # Create fully connected layer to generate output for piano keys
    outs = dense(outputs,
                 FLAGS.dim_pitch,
                 kernel_initializer=tf.glorot_normal_initializer())  # BSxMTxID
    loss = loss_fn(outs, features['input_ph'])
    print("Creating Estimator Spec for %s ..." % mode)
    # For training
    if mode == tf.estimator.ModeKeys.TRAIN:
        optimizer = tf.train.AdamOptimizer()
        train_op = optimizer.minimize(loss, global_step=tf.train.get_global_step())
        return tf.estimator.EstimatorSpec(mode=mode,
                                          loss=loss,
                                          train_op=train_op)
    # For evaluation
    eval_metric_ops = {
        "accuracy": tf.metrics.accuracy(
            labels=tf.argmax(features['input_ph'][:, 1:, ...], axis=-1),
            predictions=tf.argmax(outs[:, 0:-1, ...], axis=-1)
        )
    }
    return tf.estimator.EstimatorSpec(mode=mode,
                                      loss=loss,
                                      eval_metric_ops=eval_metric_ops)

def do_train(tr_data, vl_data):

    # Create Estimator
    sess_conf = tf.ConfigProto(allow_soft_placement=True, log_device_placement=False)
    sess_conf.gpu_options.allow_growth = True
    config = tf.estimator.RunConfig(model_dir=FLAGS.model_dir,  # CheckpointSaverHook
                                    save_checkpoints_steps=100,  # CheckpointSaverHook
                                    log_step_count_steps=10,  # SummarySaverHook
                                    session_config=sess_conf)
    music_classifier = tf.estimator.Estimator(model_fn,
                                              config=config,
                                              params={'max_seq': tr_data.max_seq})
    # Prepare input data
    tr_input_fn = tf.estimator.inputs.numpy_input_fn(
        {'input_ph': tr_data.data, 'seq_len_ph': tr_data.seq_len},
        batch_size=FLAGS.batch_size,
        num_epochs=FLAGS.num_epochs,
        num_threads=1,
        shuffle=True
    )
    # Extra Hooks
    logging_hook = tf.train.LoggingTensorHook(
        tensors={'probabilities': 'probs_tensor'},
        every_n_secs=60
    )
    # debugging_hook = tf_debug.LocalCLIDebugHook(thread_name_filter="MainThread$", dump_root="./dump")
    profiler_hook = tf.train.ProfilerHook(save_steps=1,
                                          output_dir="./profile",
                                          show_dataflow=False,
                                          show_memory=True)
    # Train
    music_classifier.train(tr_input_fn, hooks=[profiler_hook])
    print("Training is over...")

def do_test(te_data):

    print("Start testing...")

def main(_):

    if FLAGS.mode == tf.estimator.ModeKeys.TRAIN:
        tr_data = MusicDataReader(FLAGS.dir_path,
                                  'train',
                                  (FLAGS.pitch_low, FLAGS.pitch_high),
                                  FLAGS.dt)
        vl_data = MusicDataReader(FLAGS.dir_path,
                                  'valid',
                                  (FLAGS.pitch_low, FLAGS.pitch_high),
                                  FLAGS.dt)
        print("Number of training samples: %d" % tr_data.data_num)
        print("Number of validation samples: %d" % vl_data.data_num)
        do_train(tr_data, vl_data)
    else:
        te_data = MusicDataReader(FLAGS.dir_path,
                                  'test',
                                  (FLAGS.pitch_low, FLAGS.pitch_high),
                                  FLAGS.dt,)
        print("Number of testing samples: %d" % te_data.data_num)
        do_test(te_data)

if __name__ == "__main__":
    tf.app.run(main=main)

I cannot attach the output of my console for it is impossible due to the indefinite freezing of my Ubuntu. All I can say is that it tells me that it loads libcupti.so and computes a step of training process. Then, everything totally messes up.
Thank you for your support in advance.

Nagging Awaiting TensorFlower: It has been 14 days with no activity and the awaiting tensorflower label was assigned. Please update the label and/or status accordingly.

Hi,
I have the same problem in Tensorflow 1.5, when I try to use ProfilerHook, and I launch:
my_model.train(...)
my jupyter notebook kernel dies instantaneously. Is there a work around I can use ?
Thanks

Nagging Awaiting TensorFlower: It has been 14 days with no activity and the awaiting tensorflower label was assigned. Please update the label and/or status accordingly.

Nagging Awaiting TensorFlower: It has been 14 days with no activity and the awaiting tensorflower label was assigned. Please update the label and/or status accordingly.

Nagging TensorFlower: It has been 14 days with no activity and the awaiting tensorflower label was assigned. Please update the label and/or status accordingly.

Assigning to @ispirmustafa who worked on ProfilerHook.
Also @iliTheFallen, can you provide a short example that reproduces the problem? It's very difficult to debug long examples, so try making the example as short as possible.

Nagging Assignee @ispirmustafa: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

It has been 14 days with no activity and the awaiting response label was assigned. Is this still an issue?

It has been 36 days with no activity and the awaiting response label was assigned. Is this still an issue?

We are closing this issue for now due to lack of activity. Please comment if this is still an issue for you. Thanks!


Page 10 (3/3) Attribution: The fifth tip "not the one for driving" is imho rather confusing.

Quiz 4/7, "What does the term 'attribution' mean in the context of CC licenses?"


LicensePlate: Allow access to normalized value (even for invalid values)

For LicensePlates only the normalized value is now accessible to users (via #toString), i.e. this is also done for LicensePlates that are not valid.
This allows users of this class to continue working with the normalized version of the LicensePlate, e.g. for searches in a database.
The class was also made immutable, which allows precomputing the normalized value and the validation state.

Superseded by PR #5


Two of the tests appear to be susceptible to race conditions

Two of the tests appear to be susceptible to race conditions.
With the current head of master – the tree at 8b1698e – on macOS Catalina with Rust 1.41.0, after initial cloning of the fsevent-rust repository and first run of cargo test, two of the tests failed.
The tests that failed were observe_folder_async and observe_folder_sync. The test output from that initial invocation of cargo test was:
   Compiling libc v0.2.68
   Compiling getrandom v0.1.14
   Compiling cfg-if v0.1.10
   Compiling ppv-lite86 v0.2.6
   Compiling bitflags v1.2.1
   Compiling remove_dir_all v0.5.2
   Compiling fsevent-sys v3.0.0
   Compiling time v0.1.42
   Compiling fsevent v2.0.1 (/Users/erikn/src/github.com/octplane/fsevent-rust)
   Compiling rand_core v0.5.1
   Compiling rand_chacha v0.2.2
   Compiling rand v0.7.3
   Compiling tempfile v3.1.0
    Finished test [unoptimized + debuginfo] target(s) in 10.76s
     Running target/debug/deps/fsevent-ce679260b5d6056f

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Running target/debug/deps/fsevent-a3719c7d2dc1d40b

running 4 tests
test observe_folder_sync ... FAILED
test observe_folder_async ... FAILED
test validate_watch_single_file_sync ... ok
test validate_watch_single_file_async ... ok

failures:

---- observe_folder_sync stdout ----
thread 'observe_folder_sync' panicked at 'Some expected events did not occur before the test timedout:
		[("/private/var/folders/97/2k44164n2s9c3_c9zydgdz_80000gn/T/durj2526W/dest1", ITEM_CREATED | IS_DIR), ("/private/var/folders/97/2k44164n2s9c3_c9zydgdz_80000gn/T/durj2526W/dest2", ITEM_CREATED | IS_DIR)]', tests/fsevent.rs:45:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

---- observe_folder_async stdout ----
thread 'observe_folder_async' panicked at 'Some expected events did not occur before the test timedout:
		[("/private/var/folders/97/2k44164n2s9c3_c9zydgdz_80000gn/T/duru38162/dest1", ITEM_CREATED | IS_DIR)]', tests/fsevent.rs:45:5

failures:
    observe_folder_async
    observe_folder_sync

test result: FAILED. 2 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out

error: test failed, to rerun pass '--test fsevent'

Notice that the failure output of one of the tests mentions "dest1" and "dest2", and the failure output of the other test mentions "dest1".
Looking at the tests, we have:
#[test]
fn observe_folder_sync() {
    internal_observe_folder(false);
}

and
#[test]
fn observe_folder_async() {
    internal_observe_folder(true);
}

Looking then at internal_observe_folder, we have:
fn internal_observe_folder(run_async: bool) {

Inside of which, first a bunch of directories are created
    let dir = tempfile::Builder::new().prefix("dur").tempdir().unwrap();
    // Resolve path so we don't have to worry about affect of symlinks on the test.
    let dst = resolve_path(dir.path().to_str().unwrap());

    let mut dst1 = dst.clone();
    dst1.push("dest1");

    let ddst1 = dst1.clone();
    fs::create_dir(dst1.as_path().to_str().unwrap()).unwrap();

    let mut dst2 = dst.clone();

    dst2.push("dest2");
    let ddst2 = dst2.clone();
    fs::create_dir(dst2.as_path().to_str().unwrap()).unwrap();

    let mut dst3 = dst.clone();

    dst3.push("dest3");
    let ddst3 = dst3.clone();
    fs::create_dir(dst3.as_path().to_str().unwrap()).unwrap();

Then a channel is created, and the directories are observed
    let (sender, receiver) = channel();

    let mut async_fsevent = fsevent::FsEvent::new(vec![]);
    let fsevent_ref_wrapper = if run_async {
        async_fsevent
            .append_path(dst1.as_path().to_str().unwrap())
            .unwrap();
        async_fsevent
            .append_path(dst2.as_path().to_str().unwrap())
            .unwrap();
        async_fsevent
            .append_path(dst3.as_path().to_str().unwrap())
            .unwrap();
        Some(async_fsevent.observe_async(sender).unwrap())
    } else {
        let _t = thread::spawn(move || {
            let mut fsevent = fsevent::FsEvent::new(vec![]);
            fsevent
                .append_path(dst1.as_path().to_str().unwrap())
                .unwrap();
            fsevent
                .append_path(dst2.as_path().to_str().unwrap())
                .unwrap();
            fsevent
                .append_path(dst3.as_path().to_str().unwrap())
                .unwrap();
            fsevent.observe(sender);
        });
        None
    };

Then, it proceeds to attempt to validate that it received the ITEM_CREATED events for the directories it was watching (and then after that there is the shutdown stuff for the async observer).
    validate_recv(
        receiver,
        vec![
            (
                ddst1.to_str().unwrap().to_string(),
                StreamFlags::ITEM_CREATED | StreamFlags::IS_DIR,
            ),
            (
                ddst2.to_str().unwrap().to_string(),
                StreamFlags::ITEM_CREATED | StreamFlags::IS_DIR,
            ),
            (
                ddst3.to_str().unwrap().to_string(),
                StreamFlags::ITEM_CREATED | StreamFlags::IS_DIR,
            ),
        ],
    );

    match fsevent_ref_wrapper {
        Some(r) => async_fsevent.shutdown_observe(r),
        None => {}
    }
}

But keep in mind that the directory creation was initiated prior to watching those paths.
Upon subsequently running the tests again a few times, both with and without doing a rebuild of the tests themselves in-between, all four tests passed.
But from the initial failure that I saw, and from the code I quoted above, this looks like a case of susceptibility to race conditions to me.
Also, the test case seems a little odd to me. Is there a particular reason for subscribing to events on all three of those the paths "dest1", "dest2" and "dest3" rather than subscribing to none of those and instead subscribing to the initial temporary directory that is created (dst) before creating dst1, dst2 and dst3 and then verifying that the creation of dst1, dst2 and dst3 is seen from the observer of dst? I am thinking that I might be missing something here.

I have a couple of suggestions for making some changes to the tests, and can submit a PR for that, but wanted to get your feedback on my initial impression and questions about these two tests first.

Hey @ctsrc , thanks for the feedback. From my analysis, the fact:

That they observe folders which have been created before the start of the test seems curious
And that they seem to test 3 times the same IS_DIR ITEM_CREATED feels a bit redundant

So if you have something in mind, I'll gladly have look and integrate your changes!


Can't open help in OSX

Pressing F1 to open help in osx does not work.
This world has been deactivated to prevent cascading errors.
gl-error-tuple
function f
code     1286
string   "Invalid framebuffer operation"

That is an error in the 0.97 release, but it was fixed in the current git master which will become 0.98 when we are ready to do the next release.
See #1379


Uniformize extension of proposed download

Describe the bug
Fix file extension in all pages.
To Reproduce
Steps to reproduce the behavior:

Go to https://adoptium.net/temurin/releases/
Look at listed extension (.pkg, .zip, .msi)
The only wrong is tag.gz because it's computed (wrong way)

Expected behavior
We should have the .tag.gz extension
Screenshots

Note: I'm going to make the Pull Request.

Yep, I'm on it ;-)


Fix sphinx warnings

Standard GPflow issue template
Bug case
Sphinx currently raises a large number of warnings when docs are built. So that doc builds are easier to debug, and errors are more likely to be caught, fix or filter these warnings.

Some of them are fixed in https://github.com/GPflow/GPflow/pull/1391 - most of the others are due to misformatted (invalid reST) docstrings.


Updated ESDoc to 0.5.x.

Hi,
I changed the version of the esdoc dependency to 0.5.x. There are no breaking changes in this version of ESDoc. I also adjusted the dev dependency grunt to match the new version style.
I think it's much easier to understand the semantics of using x instead of ^ or ~. If you prefer the old style, I'll change it back. Just say the word!

Closed in favour of #8.


Multi language

Thank for this great tool!
Excuse me for bad english!
I want to translate Description AbstractGenetator.php into Chinese.
It is not good to change directly on the basis of the original text. Can be changed into the form of a configuration file

Yeah - this would be a great enhancement!


odd issue creating embeddings with Ollama

Hi, I am trying to use the Ollama EmbeddingModel with PgVectorStore and it's failing with
Caused by: org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [CREATE INDEX IF NOT EXISTS spring_ai_vector_index ON vector_store USING HNSW (embedding vector_cosine_ops)
]; SQL state [XX000]; error code [0]; ERROR: column cannot have more than 2000 dimensions for hnsw index
	at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1549) ~[spring-jdbc-6.1.8.jar:6

the same application works with OpenAI
here's a sample https://github.com/joshlong/bootiful-spring-boot-2024/tree/main/service
and there's a PgVector Docker Compose file here https://github.com/joshlong/bootiful-spring-boot-2024/blob/main/compose.yaml
help, please

@joshlong unfortunately, Ollama has a fixed 4096 size for the embeddings and it's not currently possible to customise the value. There is a feature request to do that: https://github.com/ollama/ollama/issues/651
I have opened a PR to improve the Spring AI PGvector documentation and mention the limitation of the HNSW indexing strategy, which can't support dimensionality above 2000: https://github.com/spring-projects/spring-ai/pull/825/files
My current workaround to get applications running fully locally is to use Ollama for the chat model and one of the ONXX Transformers for the embedding model: https://docs.spring.io/spring-ai/reference/api/embeddings/onnx.html
I hope Ollama addresses this issue asap!

Thanks @ThomasVitale Closing the issue

In case it's useful to people landing on this issue and having the same problem, I'll share another tip to use Ollama for embeddings paired with PGVector.
The multi-purpose models in the Ollama library (like mistral and llama3.1) seem to have been configured with 4096 dimensionality, which cannot be changed as of now, and doesn't work with PGVector.
However, dedicated embedding models like nomic-embed-text (full list here: https://ollama.com/search?q=&c=embedding) are configured with different dimensionality. In many cases, it's lower than 2000, so they all can be used with PGVector (and Spring AI lets you configure the dimensions via spring.ai.vectorstore.pgvector.dimensions).

@joshlong  - I did "ollama pull all-minim" the used this in my properties file:
  ai:
    ollama:
      base-url: http://${OLLAMA_HOST}:11434
      chat:
        model: llama3.1:8b
      embedding:
        enabled: true
        model: all-minilm

You can find the code here: https://github.com/ByteworksHomeLab/spring-ai-lab

Thanks all

이 문제에 직면해 같은 문제를 겪고 있는 사람들에게 유용할 경우, PGVector와 함께 Ollama를 임베딩에 사용하는 또 다른 팁을 알려드리겠습니다.
Ollama 라이브러리의 다목적 모델( mistral및 llama3.1)은 4096차원으로 구성된 것으로 보이며, 현재로서는 이를 변경할 수 없고 PGVector와 작동하지 않습니다.
그러나 nomic-embed-text(전체 목록은 여기: https://ollama.com/search?q=&c=embedding )과 같은 전용 임베딩 모델은 다른 차원으로 구성됩니다. 많은 경우 2000보다 낮으므로 모두 PGVector와 함께 사용할 수 있습니다(그리고 Spring AI는 를 통해 차원을 구성할 수 있습니다 spring.ai.vectorstore.pgvector.dimensions).

In case it's useful to people landing on this issue and having the same problem, I'll share another tip to use Ollama for embeddings paired with PGVector.
The multi-purpose models in the Ollama library (like mistral and llama3.1) seem to have been configured with 4096 dimensionality, which cannot be changed as of now, and doesn't work with PGVector.
However, dedicated embedding models like nomic-embed-text (full list here: https://ollama.com/search?q=&c=embedding) are configured with different dimensionality. In many cases, it's lower than 2000, so they all can be used with PGVector (and Spring AI lets you configure the dimensions via spring.ai.vectorstore.pgvector.dimensions).

@joshlong
like below?
ai:
ollama:
base-url: http://${OLLAMA_HOST}:11434
chat:
model: llama3.1:8b
embedding:
enabled: true
model: nomic-embed-text


Camera orientation handling is added for jpeg files

Merge with opencv_extra: https://github.com/Itseez/opencv_extra/pull/276
This pull request is for resolving the issue #4344.
Separate class ExifReader is introduced that prepares exif directory structure to read from.
Currently, only Orientation tag is being processed, however the rest tags are also built.
The solution is tested on Ubuntu 14.04.
Tests are placed into opencv_extra/testdata/highgui/readwrite/ directory (separate pull request is created).

@micalan Thanks! It is awesome patch!
There are a lot of unnecessary clone() operations.
flip accepts negative mode argument with mean of flip both axes.
My suggestion is to use something like this:
    Mat dst;
    switch(m_orientation)
    {
        case    JPEG_ORIENTATION_TL: //0th row == visual top, 0th column == visual left-hand side
            dst = img;
            break;
        case    JPEG_ORIENTATION_TR: //0th row == visual top, 0th column == visual right-hand side
            flip(img, dst, 1);
            break;
        case    JPEG_ORIENTATION_BR: //0th row == visual bottom, 0th column == visual right-hand side
            flip(img, dst, -1);
            break;
        ...
    }
    img = dst; // or img.swap(dst);

Dear @alalek,
Thank you for such warm response.
Should I squash fix for your request into existing commit or prepare new one?
There are some minor fixes I'd like to add as well.

Just update your current branch in any way (we prefer squashed commits before merging only, because intermediate changes history is not needed in the most cases - but it is not required during WIP/review stage).
Thanks!

Great! Looks good to me.
Could you please squash commits into one (it is unnecessary to store intermediate fixups history in Git).

I do this for the first time so could you please give me advice.
Is it necessary to squash commits locally and push them into origin JpegExif branch or I have to do squash remotely?
Maybe there is some instruction?
Thank you

All is right! :+1:


docs: release 5 blogpost

replaced by: https://github.com/operator-framework/java-operator-sdk/pull/2598


NatsKVContext.CreateStoreAsync (with defaults) fails on a NATS KV store created with defaults from CLI

Observed behavior
Create a KV store from the CLI using defaults and then try to create the same store from nats.net/v2 with defaults, it fails (10058, "stream name already in use with a different configuration").
When I compare the config from a CLI created store with defaults vs a store created from code the only difference is Placement. One observation is the cli created store has "placement": {} where the c# generated store does not.
There is no way to specify placement when creating a KV store from code.
nats kv add cli vs. await kvContext.CreateStoreAsync("csharp", cancellationToken);
CLI created store:
    "config": {
        "name": "KV_cli",
        "subjects": [
            "$KV.cli.>"
        ],
        "retention": "limits",
        "max_consumers": -1,
        "max_msgs": -1,
        "max_bytes": -1,
        "max_age": 0,
        "max_msgs_per_subject": 1,
        "max_msg_size": -1,
        "discard": "new",
        "storage": "file",
        "num_replicas": 1,
        "duplicate_window":<PHONE_NUMBER>00,
        "placement": {},
        "compression": "none",
        "allow_direct": true,
        "mirror_direct": false,
        "sealed": false,
        "deny_delete": true,
        "deny_purge": false,
        "allow_rollup_hdrs": true,
        "consumer_limits": {}
    }

c# created store:
    "config": {
        "name": "KV_csharp",
        "subjects": [
            "$KV.csharp.>"
        ],
        "retention": "limits",
        "max_consumers": -1,
        "max_msgs": -1,
        "max_bytes": -1,
        "max_age": 0,
        "max_msgs_per_subject": 1,
        "max_msg_size": -1,
        "discard": "new",
        "storage": "file",
        "num_replicas": 1,
        "duplicate_window":<PHONE_NUMBER>00,
        "compression": "none",
        "allow_direct": true,
        "mirror_direct": false,
        "sealed": false,
        "deny_delete": true,
        "deny_purge": false,
        "allow_rollup_hdrs": true,
        "consumer_limits": {}
    }

Expected behavior
I should be able to create a store with defaults from CLI and then call NatsKVContext.CreateStoreAsync with defaults. Also, I should be able to specify placement NATS.Client.JetStream.Models.Placement via NatsKVConfig when creating a KV store from code.
Server and client version
nats-server: v2.10.18
CLI client: 0.1.5
NATS.NET 2.3.1
Host environment
Windows 11 Pro - 23H2
Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz   2.90 GHz
Installed RAM	64.0 GB (63.8 GB usable)
Steps to reproduce
nats kv add cli
from code:
await kvContext.CreateStoreAsync("cli", cancellationToken);

Fixed in https://github.com/nats-io/natscli/pull/1124
# made before the fix
$ nats s info KV_X --json|jq .config.placement
{}

# now we make another with the fix
$ nats kv add BOB
$ nats s info KV_BOB --json|jq .config.placement
null


Company room events missing

We will need to add company room events. The company descriptions should be included there.

No need for that, only in the agenda, apparently.


FirebaseDynamicLinks.instance.getInitialLink() only works after using a Timer to delay the call

I'm using
firebase_dynamic_links: 0.5.0+11
and the current version of the FirebaseDynamicLinks.instance.getInitialLink() method does not find the initial link unless I wait 700+ milliseconds to try retrieving it.
I'm referring to the sample code at these 2 links:
https://pub.dev/packages/firebase_dynamic_links
https://github.com/FirebaseExtended/flutterfire/blob/master/packages/firebase_dynamic_links/example/lib/main.dart
Here is what it looks like when I simply add a Timer to the sample code:

  @override
  void initState() {
    super.initState();
    _timerLink = new Timer(const Duration(milliseconds: 700), () {
      print('700 ms later, going to init dynamic links');
      this.initDynamicLinks();
    });
  }

This successfully finds the link, but if I do not use the Timer it does not find the link.  To be clear, this is when I do a fresh install of the app on the iPhone after tapping a dynamic link (and it gets automatically copied to the clipboard).  For the record, onLink works when I try to open a link while the Flutter app is already running in the background on my iPhone.
flutter doctor -v
[✓] Flutter (Channel stable, v1.17.1, on Mac OS X 10.15.4 19E287, locale en-US)
    • Flutter version 1.17.1 at /Users/stephen/wardrobe/flutter
    • Framework revision f7a6a7906b (13 days ago), 2020-05-12 18:39:00 -0700
    • Engine revision 6bc433c6b6
    • Dart version 2.8.2

[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.3)
    • Android SDK at /Users/stephen/Library/Android/sdk
    • Platform android-29, build-tools 29.0.3
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_212-release-1586-b4-5784211)
    • All Android licenses accepted.

[!] Xcode - develop for iOS and macOS (Xcode 11.5)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 11.5, Build version 11E608c
    ! CocoaPods 1.7.5 out of date (1.8.0 is recommended).
        CocoaPods is used to retrieve the iOS and macOS platform side's plugin code that responds to your plugin usage on the Dart side.
        Without CocoaPods, plugins will not work on iOS or macOS.
        For more info, see https://flutter.dev/platform-plugins
      To upgrade:
        sudo gem install cocoapods

[✓] Android Studio (version 3.6)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin version 45.1.1
    • Dart plugin version 192.8052
    • Java version OpenJDK Runtime Environment (build 1.8.0_212-release-1586-b4-5784211)

[✓] VS Code (version 1.45.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.10.2

[✓] Connected device (3 available)
    • Nokia 6 1                  • PL2GARM870103356                     • android-arm64 • Android 10 (API 29)
    • iPhone11                     • 00008030-000139113A30802E            • ios           • iOS 13.4.1
    • iPhone SE (2nd generation) • 45D76A5A-C10A-42B7-9753-51B3B0DF66B9 • ios           • com.apple.CoreSimulator.SimRuntime.iOS-13-5 (simulator)

! Doctor found issues in 1 category.

Is this the same issue mentioned by
https://github.com/FirebaseExtended/flutterfire/pull/2580
and
https://github.com/FirebaseExtended/flutterfire/issues/100
?

I believe this is a duplicate of https://github.com/FirebaseExtended/flutterfire/issues/100
Closing as duplicate of https://github.com/FirebaseExtended/flutterfire/issues/100


Fix #899 - Remove nvim-qt from bundled binary

Pick up latest oni-neovim-binaries and oni-release-downloader
oni-neovim-binaries now deletes qt related dlls / exes

Fixes #899


Add to the demo to show using a placeholder for the phone numbers

Close #201

This is fine but I don't think it fixes the original issue. Probably the default when no "hint" is provided should be to have no placeholder.
I can't actually think of a time when we want to use placeholder text. It's actively discouraged in our style guide.


Update docs for deploying to Heroku

Summary
Update Heroku deployment documentation and update the required version of Node to 12.x (Heroku looks at this value when choosing a build pack).
This PR should take care of #25 -- Heroku no longer has a free MongoDB add-on. The documentation now points readers to search for one or use MongoDB Atlas which does currently have a free tier/option.
Requirements (place an x in each [ ])

[x] I've read and understood the Contributing guidelines and have done my best effort to follow them.
[x] I've read and agree to the Code of Conduct.

Follow up PR to #30 which addresses issue #25
cc @seratch for when you have a chance

Great idea @seratch. Done!


chore: make samples depend on the current version

Auto-fixing samples dependency.

Codecov Report

Merging #24 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff          @@
##           master    #24   +/-   ##
=====================================
  Coverage     100%   100%
=====================================
  Files           1      1
  Lines           4      4
=====================================
  Hits            4      4

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 65c022a...daa28c2. Read the comment docs.


Move error body object to *Response constructor param to allow deconstruction in Scala generated client

From hipchat:
Evan Wall
can the error object be added to the constructor of the error response case class so it can be deconstructed with the other parameters?
for example:
case class UserSubscriptionConflictErrorResponse(
  response: play.api.libs.ws.WSResponse,
  message: Option[String] = None
) extends Exception(message.getOrElse(response.status + ": " + response.body)){
  lazy val userSubscriptionConflictError = root.com.gilt.api.user.registration.v0.Client.parseJson("com.gilt.api.user.registration.v0.models.UserSubscriptionConflictError", response, _.validate[com.gilt.api.user.registration.v0.models.UserSubscriptionConflictError])
}
should be:
case class UserSubscriptionConflictErrorResponse(
      response: play.api.libs.ws.WSResponse,
      message: Option[String] = None,
      userSubscriptionConflictError: UserSubscriptionConflictError
    ) extends Exception(message.getOrElse(response.status + ": " + response.body))
and the logic to populate userSubscriptionConflictError can be moved outside of the case class
Michael Bryzek
the lazy val makes the json parsing error more obvious to the caller if you choose to call it - not sure I completely understand why you care that it's a val (vs in the signature)?
Evan Wall
I wanted it as a constructor parameter so I can deconstruct it in a pattern match
like case UserSubscriptionConflictErrorResponse(_, _, UserSubscriptionConflictError(someMessage)) => //do something with someMessage
Michael Bryzek
oh i see - would it be okay if it's an option? BTW - I'm not sure hip chat is going to work for me post
Evan Wall
yeah sure

This ticket should be moved to apidoc-generator


README has redirect loop to titan homepage

Just so you know, the link to the homepage on the README is a redirect loop for
http://titan.thinkaurelius.com/
Thanks

still happening sadly .. just wanted to read up on titan .. but i am not sure whether a project that doesnt fix a simple thing like this for 4 days ? is titan reliable ? are there enough people in the community behind it ?

Yep, I'm concerned about exactly that perception.  I don't know what the holdup is, but this is definitely on the radar for fixing.

we're aware of the issue and as of friday, someone was looking into it. sorta surprised that it is still a problem as of this morning.

The DNS issue appears to have been resolved sometime yesterday:
http://titan.thinkaurelius.com/