Feature request: read data from Hadoop files / Custom data sources

In my application, I have to read data in a lot of exotic formats, sometimes binary, but never usable with the possibilities Seahorse offers (CSV, JSON, Parquet). It is measuring data from proprietary measuring devices that isn't to be converted prior to analyzing.
For this, I use Hadoop's FileInputFormat classes, Spark's newAPIHadoopFile function and some mapping to get the stuff into Spark.
It would be nice if Seahorse supported something like this. My proposal would be:

Specify a custom JAR file with the reading logic as when using JDBC
Have a well-defined method somewhere in the JAR (similar to the main method) which takes the SparkContext and additional parameters, and spits out a DataFrame

For now, this can be done using SDK - https://github.com/deepsense-ai/seahorse-sdk-example/tree/1.4.1

The SDK is unfortunately out of date. See the issue I posted there.

SDK should work now. @rabejens can you confirm that?

I am currently not near one of my computers but I try it tomorrow.

OK, I just tried it and the example compiled fine (apart from tests failing).


CSV export of morphemes is broken

"'Morpheme' object has no attribute 'language'"

Hi, the error is a line in dictionary file adminviews.py:
signlanguages = [signlanguage.name for signlanguage in gloss.language.all()]
should be:
signlanguages = [signlanguage.name for signlanguage in gloss.signlanguage.all()]
(Updated on my local version fixes the issue.)
I looked through the archive and it looks like this code -- field language -- has been this way for a very long time (perhaps always).
Perhaps the Gloss field signlanguage used to be language.

Ah thanks. Yes, I think it was 'language', @vanlummelhuizen recently changed it.

Works.


[good first issue - intermediate/advanced] Moar data adapters (i.e. materializers)!

Is your feature request related to a problem? Please describe.
We have a few default ones, and could use a ton of extensions.
Describe the solution you'd like
Moar data adapters!
Describe alternatives you've considered
These should be small/lightweight extensions. Similar to the rest of the extension framework.
Some ideas:

postgresql (needs some SQL stuff, might be complicated. Also connection management...)
duckdb (ditto)
arrow integrations
s3-specific ones
big query
image
image groups
geopandas shape file
Pandas on spark for ^^^ + the defaults
Polars for ^^^ + the defaults

Additional context
Will need extension help -- see, for example, https://github.com/DAGWorks-Inc/hamilton/blob/main/hamilton/plugins/pandas_extensions.py to get started. The TL;DR is that the extensions are responsible for registering, and raise a NotImplementedError if the required dependency isn't present (which then gets caught downstream). They're all loaded here: https://github.com/DAGWorks-Inc/hamilton/blob/bfc2300bf06cd83ab28b514474732c7a27698dd3/hamilton/function_modifiers/base.py#L26. That way you don't have to have extra setup.py, etc... for each one.

postgresql (needs some SQL stuff, might be complicated. Also connection management...)

How about tackling SQLite first to side-step the connection management complexity?

postgresql (needs some SQL stuff, might be complicated. Also connection management...)

How about tackling SQLite first to side-step the connection management complexity?

@Scorpil Yep that would simplify things, but to narrow the issue down, it's also a question of what the object type is coming out?  We have Pandas SQL (#355 ) that covers this for Pandas dataframes, but not for other object types. Did you have an object type in mind to serialize to/from SQLite?

Closing this issue in favor of creating more specific ones.


Ability to use dupl as a library

For some historical reason golanci/golangci-lint uses for of mibk/dupl. I am in the process of moving golangci-lint away from forks towards upstream versions of all used linters. To achieve this we must be able to use dupl as a library and this PR introduces such an ability.

Hi, sorry for the delay.

To achieve this we must be able to use dupl as a library […]

I think that's a great idea and I'm open to make this module available as a library. I have a slight problem with the pull request: there's too many, too chaotic commits. (I understand there's a history behind the commit chain.) Would you be willing to collaborate with me on turning these commits into a reasonable commit chain? I've already merge one of your commits, the one fixing the unchecked error.
I want dupl to stay compatible as much as possible. So far, this concerned only the CLI binary itself. If we're about to make some library packages part of the public interface (thus covered by the compatibility rules), I want to make a clear distinction between those: all private packages should go under internal/, making the rest public.
As the first step, I'd like to define how exactly should the public API look like. I tried inferring the API from the client perspective from this code block, but I'm not sure that's all of it. Could you reply with your vision of how the public API should look like (something like a brief package API with types and function signatures)? The goal is to come up with a small public API I'd feel comfortable staying compatible with in the following years. Thank you!

To elaborate further, this is the API implemented in your fork (if I understood correctly):
package api

type Issue struct {
	From, To clones.Clone
}

func Run(files []string, threshold int) ([]Issue, error)

package clones

type ByNameAndLine []Clone

type Clone struct {
	Filename  string
	LineStart int
	LineEnd   int
	Fragment  []byte
}

func PrepareClonesInfo(fread ReadFile, dups [][]*syntax.Node) ([]Clone, error)

type ReadFile func(filename string) ([]byte, error)

It's questionable why e.g. ByNameAndLine or ReadFile are part of the interface. Also, I'd prefer the public interface to be a single package. There's probably no need to split them into two.

internal package sounds great :) I will find some time to work on the PR during the weekend.

@mibk does it look acceptable?

First let's discuss the package interface. Originally dupl was design to find clone groups, not necessarily pairs. Would that be a problem? I was thinking about names as well and this is I've come up with:
package clone

type Pos struct {
	Line   int
	Column int
}

type Clone struct {
	Filename string
	From, To Pos
}

type Group struct {
	Clones []Clone
}

// SearchFiles searches files for clone groups.
func SearchFiles(files []string, threshold int) ([]Group, error)

I named the package "clone" because "api" felt too vague. What do you think?

@mibk does it look acceptable?

I'd prefare there be no unnecessary changes, e.g. s/mibk/golangci/ and back, but please, let's talk first and implement second so we're not wasting each other's time. Thank you!
BTW I have also considered implementing dupl as Go Analyzer, which seems to be a more standard way of writing Go linters nowadays.

I think that we can swap
type Group struct {
  Clones []Clone
}

for
type Group []Clone

(Group would be a container for []Clones and nothing more).
I think that dropping Fragment from Clone is not a good idea as golangci-lint uses it to output meaningful error message.

First of all, sorry for my late response.

I think that we can swap
type Group struct {
  Clones []Clone
}

for
type Group []Clone

(Group would be a container for []Clones and nothing more).

That's true, but having the container gives us more power in the future should a new group field (in addition to Clones) be added.

I think that dropping Fragment from Clone is not a good idea as golangci-lint uses it to output meaningful error message.

One argument would be that the client can always obtain the fragment by reading the file at the specified position. I'm not entirely against keeping the Fragment field, though.

@iwankgb Would it be possible to you to finish this work?

@SVilgelm yes.

thank you!


Feat: Improved GLTF loader, implement typed asset loading

Two features:
(1) Issue #367 : I updated asset loading and implemented load_typed() to use alongside load_untyped(). I did this by changing the extensions-to-loader hashmap to have a key of (TypeId, String) instead of the previous String. This cleanly supports having multiple AssetLoader with different T but the same file extension.
(2) Implemented texture reading for GLTF.
(2a) Retouched the GLTF loading procedure so that instead of checking just the first node, it reads all nodes, meshes and textures. Then it returns just the first valid mesh/texture, depending on what you're requesting. This opens the door for potentially loading multiple assets from a GLTF file, but that is a much larger scope than this PR, because it is not possible to create and return multiple handles from within an AssetLoader at the moment.
Future work:

This work was mostly orthogonal to loading GLTFs as scenes.
A GLTF has to be parsed multiple times in order to get everything needed to load a model. This is highly inefficient -- as you can see in the 3d/load_model example, it takes a moment for the cubes to appear -- but as per (2a) it's not possible to easily solve right now.
We still only allow returning the first mesh or texture from a GLTF, when it can contain any number of files.

Would this PR solve #367 ?

Would this PR solve #367 ?

I think it should. You can't request, say, a Texture from a hypothetical OBJ loader because the list of extensions to loaders is keyed by type as well as extension.

Merge conflict resolved.

One of the possible workarounds for scene loading could be a load_scene method in AssetServer that will work similar to load_from_folder, but instead of the recursive directory walk, it can open a GLTF file and walk through assets, adding them.
I have suggested to use URI-like notation to access the assets and @cart liked it. For example, if you load spider.gltf, you can access assets using artificial paths like:

spider.gltf#model -> mesh
spider.gltf#skin -> texture
spider.gltf#walk -> walk animation
spider.gltf#jump -> jump animation

Alternative to # characters could be ? or :

I was going to implement the URI-like notation as well, way back when I first wrote this PR, but I ran out of steam, partially because it didn't seem like it would get accepted soon.

Yeah sorry for not responding sooner. I intentionally put this off because the asset system is either going to be replaced by atelier-assets or modified to account for cases like this better.
I don't want to merge anything that extends and/or hacks around the current asset system, as it will almost certainly be replaced in short order.
If we go with atelier, aspects of the implementation in this pr will need to change. If we go with extending the current asset system, aspects of this pr will need to change.
Again, I'm really sorry for not keeping you in the loop. I really appreciate the work you put in here and leaving you in the dark on my current thoughts for so long was disrespectful.

It's just that Ateliers been happening soon:tm: for ages, and in the
meanwhile people can't load GLTF files properly. I feel like there's value
in merging this PR for the here and now.
On Fri, Sep 18, 2020, 00:26 Carter Anderson<EMAIL_ADDRESS>wrote:

Yeah sorry for not responding sooner. I intentionally put this off because
the asset system is either going to be replaced by atelier-assets or
modified to account for cases like this better.
I don't want to merge anything that extends and/or hacks around the
current asset system, as it will almost certainly be replaced in short
order.
If we go with atelier, aspects of the implementation in this pr will need
to change. If we go with extending the current asset system, aspects of
this pr will need to change.
Again, I'm really sorry for not keeping you in the loop. I really
appreciate the work you put in here and leaving you in the dark on my
current thoughts for so long was disrespectful.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/bevyengine/bevy/pull/378#issuecomment-694532415, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AA22VOZRQCZTTJNCM7JJ4FTSGKEINANCNFSM4QNPPSIA
.

I believe current bevy_asset can stay as  a part of the engine for longer, even as bevy_asset_legacy or bevy_legacy/bevy_asset, being deprecated, whatever path you choose to go either rewrite or switch to atelier.
Another thought on the topic, is that it could be optional choice to stay with atelier, or native bevy_asset, or even a 3rd party bevy_asset package, because for some projects atelier could be kind of an overkill.


fix "Public key not valid (Point not on curve)" error

I don't get your comment.
The enclave's logger (I think you refer to PERR/golog) is not used here.
The return statement simply creates the error message which tlcc#77 expects and logs/dumps as wanted.

in enclave_stub.go there is just one logger instance called tl-enclave. See code
var logger = flogging.MustGetLogger("tl-enclave")

So far this logger is used exclusively by the method golog to print enclave logging statements.
//export golog
func golog(str *C.char) {

enclave_stub.go is otherwise log free and all errors are just returned.
I suggest that we don't use the tl-enclave logger to perform any other logging in enclave_stub.go as this may result in difficulties reading the logs later. Therefore, I also suggest that we use the tlcc logger in tlcc.go to output this error before we return an error to the client (as shim error).
I guess the idea behind his wired tl-enclave logging construct is not ideal and may be improved.

This looks good to me. Just a minor. Let's not use the logger here as it is used for the enclave logging only; just return the error is sufficient here. If you want to add an error log do so in tlcc.go
return nil, nil, fmt.Errorf("error requesting local attestation from tlcc")

What do you think?

Alright, now I see you were referring to the "previous" line.
Fixed!

Awesome thanks!


fix(cleanup): remove unused code and cookies

@QuincyLarson this should bring us to only three cookies.


⚠ Jenkins Plugins Artifactory has degraded performance

In b925af7, Jenkins Plugins Artifactory (https://get.jenkins.io/plugins/artifactory/) experienced degraded performance:

HTTP code: 200
Response time: 1354 ms

Resolved: Jenkins Plugins Artifactory performance has improved in 777868f after 21 minutes.


Any chance this can be bumped to 5.15?

There's a feature there that I'd really like to have.
Thanks.

To be frank, this repo isn't really maintained anymore as far as I can tell. Why not just use npm? Bower doesn't provide any advantage over npm at this stage.

Thanks for the quick response.
Unfortunately, its not hat easy - this is a small component on our site, and I don't have time to rework everything.

It would be awesome if this could get updated to 5.15.x Project I'm working on unfortunately depends on bower, and we would love to be able to use fine uploader inside a <table> element, which was just recently added.

I'm likely going to just delete this repo and not update bower anymore. This repo clearly isn't being maintained and it was never really setup correctly in the first place. Second, no one should be using bower anymore. There just isn't a solid reason to use this over npm or yarn, or...?

My big apologize about not maintain this repo for too long, I've been too busy with my own business.
I'll update in few minutes.

If still accepter 😢

If #10 is completed, this can be fully automated and we won't have these issue going forward.

I hope I'm still have chances to do that..


Memory usage in Flurry Hashmap

Hi I am trying to use flurry hashmap in one of my projects and want to understand memory usage and garbage collection.
I am trying a simple actix web appplication with flurry hashmap by inserting same key/value again and again. Few observation are

On first add memory usage ~15GB while it is 6.7GB with std hashmap behind the lock
Calling add -> clear -> add  repeatedly on same key/values eventually leads to OOM error and application is killed.

Output of program
curl --location --request GET '<IP_ADDRESS>:8080/add'
count 50331648! from thread ThreadId(18) - stats Length: 50331648, Capacity: 50331648, Memory: 15.7 GiB, Virtual: 17.3 GiB

curl --location --request GET '<IP_ADDRESS>:8080/clear'
count 0! from thread ThreadId(19) - stats Length: 0, Capacity: 0, Memory: 15.7 GiB, Virtual: 17.3 GiB

curl --location --request GET '<IP_ADDRESS>:8080/add'
count 50331648! from thread ThreadId(20) - stats Length: 50331648, Capacity: 50331648, Memory: 29.1 GiB, Virtual: 30.6 GiB

curl --location --request GET '<IP_ADDRESS>:8080/clear'
count 0! from thread ThreadId(21) - stats Length: 0, Capacity: 0, Memory: 29.1 GiB, Virtual: 30.6 GiB

curl --location --request GET '<IP_ADDRESS>:8080/add'
curl: (52) Empty reply from server

main.rs
use actix_web::web::Data;
use actix_web::{get, web, App, HttpServer};
use std::sync::Mutex;

use bytesize::ByteSize;
use sysinfo::{get_current_pid, ProcessExt, ProcessRefreshKind, RefreshKind, System, SystemExt};

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    let data = web::Data::new(AppState {
        data: flurry::HashMap::<String, String>::new(),
        sys: Mutex::new(System::new_with_specifics(
            RefreshKind::new().with_processes(ProcessRefreshKind::new()),
        )),
    });

    HttpServer::new(move || {
        App::new()
            .app_data(data.clone())
            .service(add)
            .service(clear)
            .service(stats)
    })
    .bind(("<IP_ADDRESS>", 8080))?
    .run()
    .await
}

struct AppState {
    data: flurry::HashMap<String, String>,
    sys: Mutex<System>,
}

#[get("/stats")]
async fn stats(data: web::Data<AppState>) -> String {
    stats_2(data)
}

fn stats_2(data: Data<AppState>) -> String {
    let pid = get_current_pid().unwrap();
    let mut sys = data.sys.lock().unwrap();
    sys.refresh_process(pid);

    let proc = sys.process(pid).unwrap();
    let map = &data.data;
    let string = format!(
        "Length: {}, Capacity: {}, Memory: {}, Virtual: {}\n",
        map.len(),
        map.len(),
        ByteSize::b(proc.memory()).to_string_as(true),
        ByteSize::b(proc.virtual_memory()).to_string_as(true)
    );

    string
}

#[get("/add")]
async fn add(data: web::Data<AppState>) -> String {
    let size;
    {
        let max_entries = 100663296 as u64;
        let m = &data.data;
        for i in 0..max_entries / 2 {
            m.pin().insert(format!("str-{i}"), format!("str-{i}-{i}"));
        }

        size = m.len();
    }
    let stats1 = stats_2(data);
    format!(
        "count {size}! from thread {:?} - stats {stats1}\n",
        std::thread::current().id()
    )
}

#[get("/clear")]
async fn clear(data: web::Data<AppState>) -> String {
    let size;
    {
        let m = &data.data;
        m.pin().clear();
        // unsafe { malloc_trim(0) };
        size = m.len();
    }

    let stats1 = stats_2(data);
    format!(
        "count {size}! from thread {:?} - stats {stats1}\n",
        std::thread::current().id()
    )
}

Cargo.toml
[package]
name = "skiptest"
version = "0.1.0"
edition = "2021"

[dependencies]
actix-web = "4"

flurry = "0.4.0"

sysinfo = "0.28.4"
bytesize = "1.2.0"

Huh, that's really interesting indeed, and suggests that the memory reclamation isn't kicking in at some point here. @JackThomson2 if you're looking for a project to get you back into more depths of flurry, this may be a good candidate!

Hi Jon, sorry been away for a while. This looks an interesting problem, and does seem something to do with the reclamation not running for some reason. I'll definitely take a look at this, I'll update in this issue here what I find.
Thanks ss025 for the example really helpful to get going.

@JackThomson2 would you be able to find anything on above issue ?

Hey sorry I didn't update here, been pretty busy at work.
So a couple of new findings:

Replicating this in Axum webserver the issue is gone and memory behaves as expected
I also tried this with xitca which is a 100% alternative to actix and the same memory issue occurs.

My suspicion is that its something to do with how actix uses multiple single threaded runtimes, but I haven't been able to investigate further yet. I'll keep you posted as I find more

Thanks, @JackThomson2 for finding.  My original problem started with hyper. I am not sure if there is an issue with Hyper or not. Let me try that

Thanks for taking a look, when you test with hyper can you use a custom allocator such as jemalloc so we can try eliminate heap fragmentation being a potential cause

Sure. WIll you be able to share the Axum example code you tried so I can compare it with hyper on the same machine and settings?

Sure this is the code I was using
use lazy_static::lazy_static;

use axum::{response::Html, routing::get, Router};
use std::net::SocketAddr;
use std::sync::Mutex;
use bytesize::ByteSize;
use sysinfo::{get_current_pid, ProcessExt, ProcessRefreshKind, RefreshKind, System, SystemExt};
use tikv_jemallocator::Jemalloc;

#[global_allocator]
static GLOBAL: Jemalloc = Jemalloc;

lazy_static! {
    static ref APP_DATA: AppState = AppState::new();
}

struct AppState {
    data: flurry::HashMap<String, String>,
    sys: Mutex<System>,
}

impl AppState {
    pub fn new() -> Self {
        AppState {
            data: flurry::HashMap::<String, String>::with_capacity(100663296),
            sys: Mutex::new(System::new_with_specifics(
                RefreshKind::new().with_processes(ProcessRefreshKind::new()),
            )),
        }
    }
}

fn stats_2(data: &AppState) -> String {
    let pid = get_current_pid().unwrap();
    let mut sys = data.sys.lock().unwrap();
    sys.refresh_process(pid);

    let proc = sys.process(pid).unwrap();

    let string = {
        let map = &data.data.pin();
        format!(
            "Length: {}, Capacity: {}, Memory: {}, Virtual: {}\n",
            map.len(),
            map.len(),
            ByteSize::b(proc.memory()).to_string_as(true),
            ByteSize::b(proc.virtual_memory()).to_string_as(true)
        )
    };

    string
}

#[tokio::main]
async fn main() {
    // build our application with a route
    let app = Router::new()
        .route("/add", get(add_handler))
        .route("/clear", get(clear_handler))
        .route("/", get(handler));

    // run it
    let addr = SocketAddr::from(([127, 0, 0, 1], 3000));
    println!("listening on {}", addr);
    axum::Server::bind(&addr)
        .serve(app.into_make_service())
        .await
        .unwrap();
}

async fn handler() -> Html<&'static str> {
    Html("
Hello, World!
")
}

async fn add_handler() -> String {
    let size;
    let init_size;
    {
        let max_entries = 100663296 as u64;
        let m = &APP_DATA.data;
        init_size = m.len();
        let adding = m.pin();
        for i in 0..max_entries / 2 {
            adding.insert(format!("str-{i}"), format!("str-{i}-{i}"));
        }
        size = m.len();
    }
    let stats1 = stats_2(&APP_DATA);
    format!(
        "initial count {init_size}, count {size}! from thread {:?} - stats {stats1}\n",
        std::thread::current().id()
    )
}

async fn clear_handler() -> String {
    let size;
    let init_size;
    {
        let m = &APP_DATA.data;
        init_size = m.len();
        m.pin().clear();
        // unsafe { malloc_trim(0) };
        size = m.len();
    }

    let stats1 = stats_2(&APP_DATA);
    format!(
        "initial count {init_size}, count {size}! from thread {:?} - stats {stats1}\n",
        std::thread::current().id()
    )
}

I don't exact numbers, but what I found was memory increased the first 2 cycles then eventually dropped and stayed consistent

So some exciting news. I've made a little breakthrough and figured out the exact location of our issue.
It appears to be with the delayed allocations in the shield / waiting for it to drop.
I was able to change the line here: https://github.com/jonhoo/flurry/blob/main/src/reclaim.rs#L152 to be:
self.collector().unwrap().retire(shared.ptr, seize::reclaim::boxed::<T>); Calling retire directly against the collector is a way to bypass the use of the TLS implementation and directly drop if possible. With this change we see consistent memory usage throughout write / clear cycles!
I will now dig further into the implementation of Seize to figure out how we're seeing this issue. But I feel we're close to a fix for this! Thanks again @ss025 for flagging this up.

Just saw this issue. If you need any help or guidance regarding seize please let me know!

So I've come full circle here. I think the issue is actually just memory fragmentation. I switched to Jemalloc and hooked into their API to retrieve actual bytes allocated and it's consistent between clear() and remove() so it seems with the huge amounts allocated the allocator can't keep up. I tried manually calling tikv_jemalloc_ctl::epoch::advance().unwrap(); to try force a de-fragmentation with no luck.
The question is why the remove() loop is friendlier on the allocator, I tried re-writing the clear function to look more like the remove function. I managed to get it pretty close to the dellocation order but still ended up with the heap fragmentation.
I think the real solution may be too implement the preloading of nodes and reusing them as suggested by @ibraheemdev  here: https://github.com/jonhoo/flurry/issues/80#issuecomment-1082309088
I've been having a play with implementing this to see how it affects the memory profile (at the moment it's very hacky and a lot slower but looking at memory usage)

So I had some time to work on this, and I implemented some memory optimizations on an experimental branch. It cuts the node memory overhead in half by moving batches to a separate allocation instead of the linked list. crossbeam-epoch implements this batches this way as well, and I think it's a good tradeoff in general because the allocation is still amortized to once per-batch. If either of you want to run your tests against the branch and report any findings here that would be helpful.

The seize improvements are released in 0.3.3 (https://github.com/jonhoo/flurry/pull/123). The memory overhead per object is now just one word (vs. four). However, I'm not sure if this will fix the issue because the additional memory is still incurred after retiring and it's unclear what the actual issue is here.


Annotating regulation and making regulation_of terms on the fly -> (ref: gomodel:59dc728000000324)

Hello,
I am seeking advice re: capturing regulation in Noctua and making new regulation terms in GO and/or in Noctua on the fly.
It was decided at the GOC meeting in Cambridge that for assertions of type “Gene product involved_in regulation of X”, we will use default ‘involved in’ (1, 2).
It was also decided that Noctua will create regulation_of GO terms on the fly based on 'regulates' relations in GO-CAM (1).
I was wondering whether this has started being implemented yet?
I recently intended to create two new regulation_of GO terms (3, 4) based on PMID:17139249, but during an ontology call I was advised to annotate in Noctua instead. The initial advice was to use the relation causally_upstream_of_or_within but I have since then re-read the paper, and I think it should actually be 'regulates'. (Or part_of regulation_of GO term??). Details below.
Issue 1: Making regulation_of terms of the fly
I have started making the model, but I noticed that the regulation_of terms do not get created based on the 'regulates' relations, as I expected. Instead the GPAD output uses what would have been the is_a regulation_of parent term, of either of the terms that I intended to create (please compare the screenshots below).
GO-CAM:

Info that goes into the GPAD file:

Issue 2: 'Regulates' relation + 'MF/BP X'   or   'part_of' relation + 'regulation of MF/BP X'?
Relating to the above, I have just checked the GO-CAM documentation for regulation_of models. This document actually recommends using part_of relations + regulation_of GO terms. I suppose this is probably out of date?
Issue 3: Making relations between MF and regulated MF/BP or BP that the MF is part_of and regulated MF/BP --> annotation output very different
The reason why I wanted to take a look at the GO-CAM documentation in the first place was that I seem to recall from a Noctua training that the 'regulates' relation should only be used between molecular functions/activities (here 'protein serine/threonine kinase activity') and the biological processes, which they regulate (here 'dynein intermediate chain binding'). However, if the molecular function ('protein serine/threonine kinase activity') is a part of another process (here: 'peptidyl-serine phosphorylation'), I noticed that the 'regulates' relation needs to be created between 'peptidyl-serine phosphorylation' and 'dynein intermediate chain binding', or the regulation_of GO annotations do not appear in what would be the GPAD output. (Please compare the screenshots below with the ones that I pasted in above under issue 1 header).
GO-CAM:

Info that goes into the GPAD file:

Summary
So, overall my questions are:

Has Noctua started making regulation_of GO terms on the fly? Or what is the expected timeframe for this to be implemented? Should I go ahead and make my required regulation terms for now?
What is the most up-to-date documentation for capturing regulation in Noctua?
If I make the regulations terms mentioned above (3, 4), and then use the 'regulates' relation + process X in Noctua, will my output annotation be 'regulation of process X'? Or should I use the 'part of' relation with the new regulation terms, as the GO-CAM curation guidelines suggest? (But I suspect these are of date?).

Please advise!
Thanks,
Barbara
(optional back ref: http://noctua.berkeleybop.org/editor/graph/gomodel:59dc728000000324)

cc
@RLovering
@rachhuntley
https://github.com/geneontology/go-annotation/issues/1690

Hi Kimberly
Please could you comment on the questions that Barbara has raised above?
Thanks
Ruth
@vanaukenk

Hi,
Although I agree that if you create a "regulates" relationship a term should be created, dynein is required for microtubule anchoring, rather than 'regulating' (the kinase is regulating). The dynein complex moves the  components required for anchoring to the centrosome, and is definitely part_of the process.
val

Hi @BarbaraCzub
Kimberly is on vacation, I will try to answer:

Has Noctua started making regulation_of GO terms on the fly? Or what is the expected timeframe for this to be implemented? Should I go ahead and make my required regulation terms for now?

This is not yet implemented, so best is to request the term until it is.

What is the most up-to-date documentation for capturing regulation in Noctua?

The documentation on relations is here: http://wiki.geneontology.org/index.php/Annotation_Relations
but there isn't very much on 'regulates'.
However, the relations for Molecular Function to Biological Process are:

part of
causally upstream of or within, positive effect
causally upstream of or within, negative effect
causally upstream of, positive effect
causally upstream of negative effect

If I make the regulations terms mentioned above (3, 4), and then use the 'regulates' relation + process X in Noctua, will my output annotation be 'regulation of process X'? Or should I use the 'part of' relation with the new regulation terms, as the GO-CAM curation guidelines suggest? (But I suspect these are of date?).

I think you should use 'part of' relation with the new regulation terms, until we can do differently.
(That is, if there really is regulation - see Val's comment).
Thanks, Pascale

Thanks @pgaudet, I'll add the regulation term then.
Thanks @ValWood for your comment that only the kinase is regulating. I'll change the relation between 'dynein intermediate chain binding' and 'microtubule anchoring at centrosome' from 'regulates' to part_of and hopefully this will fix the output annotations.
Thanks,
Barbara

Updated:

But Noctua seems to have lost my ARUK-UCL attribution...

I'm still confused by this:
part of
causally upstream of or within, positive effect
causally upstream of or within, negative effect
causally upstream of, positive effect
causally upstream of negative effect

When I speak to @ukemi  he says that "causally upstream of" can be used for things way, way upstream (this is the feedback I got when querying GPs like this in matrix intersections).
If this is the case, this leaves us with no way to distinguish  "way, way upstream" from direct upstream regulation (i.e. the signalling that directly turns a process on).
This is a problem.....because it means that we have not really improved from the previous situation where we could say "regulates" or "part of" except that we can now say "regulates OR part of" explicitly.
I really don't think the meaning of "causally upstream of" is precise enough with regard to how far upstream it can be used. There will be huge problems later on unless this is made explicit somewhere. At the moment, it seems to me that there is actually a loss of specificity  compared to what we had previously because causally upstream is less stringent than "regulates" (however this depends who I speak to).
One thing is for sure. Everyone is not on the same page about this.

@ValWood The idea is if you are regulating, you would be 'part of' regulation of x, and not 'upstream from x'.
Ideally we would move away from 'causally upstream of', this is mostly to deal with legacy annotations when experiments did not allow us to annotate very precisely (or when guidelines were a bit different).
Pascale

OK, I think I understand. So if something is causally upstream, but not regulating, which relationship would you use?

causally upstream of, positive effect
OR
causally upstream of negative effect
dependent on the direction of the effect.

So in this case  these are not the only relations to connect MF  to BP?
part of
causally upstream of or within, positive effect
causally upstream of or within, negative effect
causally upstream of, positive effect
causally upstream of negative effect
the regulation terms are additional?

There are no regulation relations - we should use 'part of regulation', AFAIK.

OK so
protein kinase activity  has_input  BICD1 part_of regulation of microtubule anchoring at the centrosome
(as above)

Sounds right to me.


SLIM-268 update firmware

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

All is well.

All is well.


Refresh

Updating an old feedstock.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.


Simswap not works

Getting an error with two Simswap collab (pictures/videos)

What is the error message?

Related:

https://github.com/neuralchen/SimSwap/issues/445

I can confirm that editing the following file works.
File "/usr/local/lib/python3.10/dist-packages/insightface/model_zoo/model_zoo.py", line 23

Before:
    def get_model(self):
        session = onnxruntime.InferenceSession(self.onnx_file, None)

After:
    def get_model(self):
        session = onnxruntime.InferenceSession(self.onnx_file, None, providers=['AzureExecutionProvider', 'CPUExecutionProvider'])

This works as well, but leads to another issue.
!pip install -U insightface

This happens with insightface==0.7 or above.
ValueError: This ORT build has ['AzureExecutionProvider', 'CPUExecutionProvider'] enabled. Since ORT 1.9, you are required to explicitly set the providers parameter when instantiating InferenceSession. For example, onnxruntime.InferenceSession(..., providers=['AzureExecutionProvider', 'CPUExecutionProvider'], ...)

This happens with insightface==0.6.2 or below.
TypeError: RetinaFace.detect() got an unexpected keyword argument 'threshold'

The version history is at https://pypi.org/project/insightface/#history

Fixed for now with a fork of SimSwap which removes the threshold for Retinaface.

Working now! thanks a lot pal

You are welcome! Thank you for reporting this issue!
Regarding my solution, I had the choice between fixing ValueError: This ORT build has... and TypeError: RetinaFace.detect() got .... I went with the latter and I believe there should be no issue with this solution. Indeed, the default value is the same (0.5):
class RetinaFace:
    def __init__(self, model_file=None, session=None):
        [...]
        self.det_thresh = 0.5

vs.
https://github.com/woctezuma/SimSwap/blob/061329137d6d978d44ae2dc7c4700b825180c2ea/insightface_func/face_detect_crop_single.py#L51-L52
and this value is used inside detect():
    def detect(self, img, input_size = None, max_num=0, metric='default'):
        [...]
        scores_list, bboxes_list, kpss_list = self.forward(det_img, self.det_thresh)

https://github.com/deepinsight/insightface/blob/c2db41402c627cab8ea32d55da591940f2258276/python-package/insightface/model_zoo/retinaface.py#L207-L224


Fix behavior of PublicOnlyProxy in setattr, wrapped methods, and calling

Fixes/Closes
Closes #5767
Description
This PR makes a few tweaks to the behavior of PublicOnlyProxy:

in setattr, if we want to set an attribute of a wrapped object, if the attribute value is itself an object wrapped with PublicOnlyProxy, we unwrap it. This has two benefits: (1) checking the attribute later incurs a significant cost in _is_called_from_napari, which involves frame inspection. This way we don't incur that cost. And (2) Certain equality checks fail when PublicOnlyProxy's interact with Qt. This is #5767 and this PR fixes that.
in PublicOnlyProxy.create, we now check whether the object we want to wrap is a method, because then the module we need to check is not that of the method (this is always builtins), but rather the module of the object that that method is bound to. This is now fixed.
When a PublicOnlyProxy'd object is callable, we unwrap the arguments, call the unwrapped function with the unwrapped arguments, then wrap the result. This avoids the performance issues detailed above.

@Czaki I'm sorry, I don't really understand this PR... I trust that you know what you're doing 😅 but as a reviewer I'd like to understand it. Do you think you could explain it in a bit more detail? Something like, "When passing the viewer to plugins, we wrap it in a PublicOnlyProxy, to avoid plugins accessing private attributes and methods. This has worked fine in most cases but fails when X, because Y. To fix it, we remove the proxy before doing Z."
It's also unclear to me whether this fixes the "whole class of issues" exemplified by #5767 as you commented there, or just two (__call__ and __setattr__) — but are there more?

It's also unclear to me whether this fixes the "whole class of issues" exemplified by #5767 as you commented there, or just two (__call__ and __setattr__) — but are there more?

The pointed issue shows the problem with one function. I mean that same problem may happen in any place where we pass napari objects to Qt, as it uses something like id(), not is to compare objects. So it does not treat wrapped and unwrapped objects as identical.

@Czaki I'm sorry, I don't really understand this PR... I trust that you know what you're doing 😅 but as a reviewer I'd like to understand it

Ok. I will add more explanation.

@jni I have added docstrings. Could you check now?

@Czaki I've updated the comments, updated the PR description, and made the variables in the tests more descriptive because I was having trouble following the logic. Could you please double-check them and thumbs-up here if all good? Otherwise I think this is ready to merge! 🚀 Thanks for the pair session which made things clear. 😊


Slow windows drawing/update if i move with the mouse a powerpoint box - some seconds lag

Hello, i think i have found an odd behavior in freerdp.
Terminal Server 2008 R2
commandline:
xfreerdp -sec:rdp /f /multimon:force /v:servername
If i draw in Powerpoint an box with the mouse the box appears verry slowly on the screen. if i move the mouse ( still pressed ) and the box gets bigger and smaller i have a timelag of 2-5 seconds.
Same behavior in the freeplane mindmap manager.. moving a node item has also this lag..
I have tried to find an older working Version which hasn´t the problem.  This one is working for me..
GOOD: bfdab0ed4aff4ea09966e93aa73e31b186fe78ef.
First Version which doesn´t work is this one..
BAD: 06a04a0199b2c53bfdf121e94e885d7919ed3911
I checked also Master from today and still the problem is around.
i can upload a video if someone wants to see it 8-)..
Anyone around who can verify this ?

@ropetry could you provide a command line to get an idea which operations may be at fault?

sorry, wrong issue, question for you was could you find out which operations are involved in your example? bitblt may have a debug print, but not sure and nowhere near a proper computer.

@ropetry Could you also check, if the issue persists when using /gdi:sw?

@akallabeth I will try it as soon as possible.. today no access to the system. But perhaps you could help me in the meantime by explaing your debug wish..
i searched for bitblt in the code and some log lines near it, but didn´t find something i can use.

@ropetry: You connect with 16bit color depth (default) and legacy GDI operations (no RFX/GFX/H264 modes active).
In this mode the drawing is done by gdi_BitBlt, gdi_LineTo and a few others.
gdi_BitBlt implements a whole lot of subfunctions, which are of different importance for different Windows Server versions.
The current implementation is falling back to a generic (slow) decoding to implement all possible subfunctions, but uses optimized versions for heavily used ones.
Since you encountered slow performance, it would be interesting to know, which one is responsible.

@ropetry If you add a log line in BitBlt_process printing rop should be sufficient. Be aware that that will slow down the whole client and write a whole lot of logging.

@akallabeth : spent some time this morning to answer your question but i think i can´t answer your questions deeper 8-(..for the moment. perhaps you can take me a little bit further..
i started with a fresh checkout from master ( last commit e60d0d5) and added some wlog debug stuff in the bitmap.c and gdi.c
gdi_bitBlt gets a new line with
WLog_INFO(TAG,  "DEBUGDH initialize gdi_BitBlt function");
WLog_INFO(WLog_Get(TAG), WLOG_INFO, "DEBUG DH ROP=%s", rop);
but i don´t see them with my normal command line
xfreerdp -sec:rdp /v:servername
if i use the /gdi:sw switch i see those Loglines, but i get a segfault and freerdp crashes directly. So normally i don´t use gdi:sw.
i added also some lines to gdi.c like
WLog_INFO(TAG,  "DEBUG DH initialize gdi_bitmap_update");
in the gdi_bitmap_update function.. those lines appear verry often 8-).
If i don´t use the /gdi:sw switch i think it will use some hw gdi functions. Is there an other place to debug those lines ? as in the bitmap.c file
my configure script looks like this
#!/bin/bash
cmake
-DCMAKE_BUILD_TYPE=Debug
-DCMAKE_C_FLAGS="$RPM_OPT_FLAGS"
-DCMAKE_INSTALL_PREFIX=/opt/freerdp
-DCMAKE_SKIP_INSTALL_RPATH=ON
-DSTATIC_CHANNELS=OFF
-DWITH_ALSA=OFF
-DWITH_FFMPEG=ON
-DWITH_GSTREAMER_0_10=ON
-DWITH_GSTREAMER_1_0=OFF
-DWITH_PULSE=ON
-DWITH_SSE2=ON
-DBUILD_SHARED_LIBS=OFF
Coudl you please give me some more assistance on this. Thanks for you time
bye
roman

@akallabeth : sorry i have to re-edit this 3 times. now the comments and logs are valid.

@ropetry Please compile with some optimisations (just use Release build). This speeds things up a lot. As for your debug messages, better place is (as mentioned before)  BitBlt_process printing rop (which is already a string there) is sufficient.
As you mentioned that it is also quite slow when using default settings, you can try the following options:

Different color depths /bpp:32 or /bpp:24
/rfx, /gfx, ```/gfx-h264````(requires OpenH264) and a combination of these

@akallabeth : thanks again.. sorry i missed the old line with your hint to the right function to add the debug line. Did it now and i see this behavior:
I have to use the gdi:sw switch again to see the debugs..
here are the outputs with gdi:sw enabled and doing powerpoint box drawing.
[16:13:01:910] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:02:572] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:02:572] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:02:723] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:02:724] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:02:843] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:02:844] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:02:987] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:02:988] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:02:060] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:02:060] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:03:236] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:03:236] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:03:343] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:03:344] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:03:580] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:03:580] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:03:852] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:03:852] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:05:262] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:05:263] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:05:263] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:05:263] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:07:372] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:07:372] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:07:372] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:07:372] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:18:812] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DSa
[16:13:18:813] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DSo
[16:13:18:814] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DSa
[16:13:18:815] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DSo
[16:13:18:862] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DSa
[16:13:18:863] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DSo
[16:13:45:234] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:45:560] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:45:560] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:45:655] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:45:656] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:46:520] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:46:521] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:46:521] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:46:521] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
[16:13:46:521] [29890:29892] [INFO][com.freerdp.gdi.bitmap] - DEBUGDH initialize BitBlt_processROP= DPx
Test with /gfx fails with error loadling lib librdpgfx-client.so .not a real static build i think with my compile options.
Test with /rfx works and looks good i think..
/bpp:32 or 24 doesn´t help
if i use /gdi:hw or /gdi:sw it is slow. with /rfx it is o.k
The old Version bfdab0e is at same speed for alloptions i think..
Anythink i can do further to help ?
bye
roman

@ropetry Ok, so it is not an issue with the BitBlt options. The one you posted in the logs is using X11 accelerated operation with /gdi:hw (default if /gdi:sw is not specified)
So to further debug this:

Could you post the output ofxfreerdp /buildconfig?
Could you add debug options in gdi_opaque_rect, gdi_multi_opaque_rect, gdi_line_to, gdi_polyline and gdi_surface_bits

@akallabeth O.K. Thanks for those infos. If you want to see the problem have a look at //videobin.org/+ex4/iqu.html .
What do you want to add.. Just a info from the function ?like
WLog_INFO(TAG, "DEBUGDH initialize gdi_opaque_rect");
in the beginning of each function ?
The output of the /buildoptions and debug session is here: i hope pastebin is o.k..
http://pastebin.com/seRwN8nm

@ropetry Ok, that's it. No I know which functions are invoked. gdi_opaque_rect definitely needs some speedup, but you have the laggy behavior with /gdi:hw too? That is indeed buzzling, because the X11 version uses XFillRectangle which should be really fast...

@akallabeth Glad to hear that you now know the place where the sloweness could appear in gdi:sw mode.. normally we don´t use this gdi:sw mode so you are right it should work via the faster XFillRectangle function.
Anything we can try to debug the gdi:hw mode ? if i use the old git version and gdi:hw it is fast. with the new git version it is slow.. same thinclient, same os.. just an other xfreerdp session.
Where there some changes in the logic for gdi:hw and gdi:sw in the commits between both versions. Perhaps i have to bisect again to find a closer commit..But there was a really huge 8-). pull request between those 2 git version i have mentioned in the first comment 8-)..
bye and thanks again for your help
roman

@ropetry Yes, I know about the huge pull (that one is on me). With /gdi:hw nothing should have changed in regard to performance. /gdi:sw on the other hand has been (mostly) rewritten and therefore some issues were expected. So maybe we're missing something here.

@akallabeth 2 hours later i have some news 8-).. I checked again several commits and i think i have found the issue for the /sw:hw lag.
i compiled the following commits and did my test:
267dea9
071b79a
06a04a0
f5a6645
All o.k . The bad thing happened with commit
a6f4117
it lags.. xfreerdp /sec:rdp /f /multimon:force /gdi:hw /v:servername
if i add this option -max-loop-time:1000 it´s better.
perhaps it´s a to low default value for the max-loop-time ? If i use 100 instead of the 1000 i have the same lag.
See #3364 ... Sadly i´m also one of the people who sad this old pull request solved some other issue (excel endless scroll problem) 8-)..but introduced perhaps this new one ?
bye
roman

@ropetry Ouch, that hurts. So we have contradicting requirements, great :/

@akallabeth yes.. perhaps someone will have a good idea over the wekend 8-).. at the moment my users use the /rfx and the max-loop-time:1000 options and i will see if i get complains next week.

Well I really think there's no contradiction that https://github.com/FreeRDP/FreeRDP/pull/3364 gives that behaviour. If we treat inputs at a higher rate but the client side rendering is too slow, then we see the observed behaviour. When treating inputs generates a lot of rendering then we get stuck (like in the video when dragging makes the server re-render), I'm not sure how we can deal with this. Perhaps it's just a matter of setting the right value...

Some thoughts about that bug: latency is the delay that is observed between an action and its feedback.
In our case we would like to have a measurable feedback of the mouse move action, in the ideal case we would render what is the consequence of the mouse move, and while doing this we would stop treating inputs (so not flooding the server with mouse moves, that generates frames that we don't want to render client side). The problem is that we don't have such feedback.
So I guess that solving this problem will be only based on smart heuristics.

@hardening @akallabeth
Just to give an update.. My users (30 thinclients at the moment) are working now for 1 week with the new delay option and 1000 as the max-loop time.. at the moment no complains about performance or hanging sessions..
so at the moment i will push this option to all other thinclients.
Thanks for your help so far.
bye
roman

@ropetry thanks for the update. I'll close the issue then for now. Just re-open it or create a new one in case. Thanks.


Data loss

Assuming a huge stream of events:
100_000.times { |i| puts i; MyWorker.debounce }

It takes a while to enqueue and if you hit CTRL+C while this is running, you might lose the debounced job.
What happened is that it deleted the previous scheduled job but didn't create the new one.
I would expect at least to the previous one to continue there.
Maybe a Redis.multi to avoid that?

That's a good point. Feel free to open PR if you have already prepared a solution for that, if not we will take care of this sooner or later. cc @tallica

Should be solved in 2.0.0

Awesome, thanks! ❤


[php] Update 8.1.5, 8.0.18, 7.4.29

Tags are created:
https://github.com/php/php-src/releases/tag/php-8.1.5
https://github.com/php/php-src/releases/tag/php-8.1.5
https://github.com/php/php-src/releases/tag/php-7.4.29
But not officially released on homepage:
https://www.php.net/ChangeLog-8.php
https://www.php.net/ChangeLog-7.php

@BiNZGi  my 2 cent is to wait until official announcement

Looks like it is official now


Cannot use leaflet directive multiple time at once.

From @wayofspark on March 19, 2014 15:58
Hello,
The _leafletMap directive variable singleton in the root directive cause leaflet to crash in various way if more than one map is used at the same time in the application. ( all maps use the variable, causing previous maps to loose reference to their own map at one point )
Fix should be easy using a scope variable instead of directive singleton to store the map promise.
Copied from original issue: tombatossals/angular-leaflet-directive#331

From @tombatossals on March 20, 2014 13:27
Hi, have you tried passing an id to the "leaflet" tag? The leafletData object stores a leaflet object based on the id, so it will be able to fetch the required map object when you ask for it.
For example:
<leaflet id="first" center="london"></leaflet>
<leaflet id="second" center="paris"></leaflet>

Then you can use:
leafletData.getMap("first").then(function(map) {
    console.log(map);
});

Take a look at this example code:
http://tombatossals.github.io/angular-leaflet-directive/examples/double-map-example.html
I think this will solve the behaviour you are experiencing.

From @wixo on March 21, 2014 5:36
This is kind of happening to me too with async loaded views, when I go back and forth between these views the current leaflet map inherits the center of the previous leaflet map.
I am using the ids as advised.

From @ugaz on April 23, 2014 22:48
There is an issue with the _layersControl when you have multiple maps: it is also a singleton. I have fixed it replacing "var _layersControl;" by "var _layersControl = [];" and every occurrence of the variable "_layersControl" by "_layersControl[mapId]". Seemed to work for me, though I am sort of newbie both in angularjs and leaflet, and I'm not sure if this is the best solution. Hope you hackers could take it from here, maybe...
By the way, thanks for the good job!!

From @tombatossals on October 8, 2014 14:28
Hey guys, can anyone make a jsfiddle where I can see this behaviour?

From @dnazarov on March 21, 2014 8:27
Is this example OK? You may change some of scope preperties directly through the scope methods.
http://jsfiddle.net/4PhzC/36/

From @michaelryancaputo on October 8, 2014 14:16
Having this issue as well. Having an issue building a dynamic map that has the ability to toggle various geojson and popup data. I've settled on having multiple maps on the page, and show/hiding them, but now that i've built that, I'm finding that I can't have more than one map showing different data on one page. I've tried passing the id solution as above with no luck unfortunately.

From @Splaktar on September 24, 2014 19:3
I'm seeing this as well. All of the markers which are on layers (feature group) are applied to the first map on the page. This includes using marker clustering. If the markers are not on layers, then the markers get applied to the proper map. But our use case requires that all of the markers be in separate layers/feature groups.
So this limits us to using only one map per page, which is a significant limitation that we would like to see removed.

From @tombatossals on March 21, 2014 6:46
Can you guys code a simple case where I can see this behaviour?
You could fork this jsfiddle example as a starting point:
http://jsfiddle.net/tombatossals/4PhzC/

From @michaelryancaputo on October 8, 2014 19:30
I'm using multiple maps with different geojson data on each one (not markers) - also including data, style, and onEachFeature popups for each geojson boundary.

From @tombatossals on October 8, 2014 14:32
@flashpunk, do you mean that you can't show two maps on the web page with different markers on each one? Is there some more complexity on your code?
I'm going to try to make a jsfiddle myself

From @tombatossals on October 8, 2014 14:39
http://jsfiddle.net/4PhzC/168/
This jsfidle could be an starting point. Could anyone ellaborate a more complex example where we can see the failure?

From @michaelryancaputo on October 9, 2014 14:27
Ok, so i've been playing with your example above, with some of the extra features i'm using and it seems to be working...
Going to go back again and try and re implement it again.
http://jsfiddle.net/flashpunk/vosru6t9/

From @tombatossals on October 9, 2014 17:10
Fantastic example. Would you mind if I put this code in the project as an example of how to use multiple maps and geojson data at the same time?

From @Splaktar on October 22, 2014 20:59
That example is helpful but it still isn't quite what I need. I debugged this a lot more today.
Basically, I need to pass in dynamic names to the directive via the attribs. But since the directive uses isolated scope with = (two way data binding) you cannot use data binding in the attribs. Do you really need to be bound in both directions? Should the directive really be modifying the event-broadcast, markers, paths, or layers objects? It seems like these should be input only?
Here is what I need to work:
  <leaflet id="{{map.uuid}}" center="'{{ 'leaflet.center.' + map.name }}" markers="{{ 'leaflet.markers.' + map.name }}"
           layers="{{'leaflet.layers.' + map.name}}" controls="controls" paths="{{'leaflet.paths.' + map.name}}"
           event-broadcast="{{'leaflet.events.' + map.name}}"
           class="leaflet-map"></leaflet>

But this fails with Angular exceptions since you can't use data binding for these attributes.

From @michaelryancaputo on October 16, 2014 15:33
FYI, I ended up scrapping this method, and using this gist: https://gist.github.com/pospi/85c31530c4a93cf091ce
It destroys the layer and recreates a new one. I've also had to hack a way to destroy the legend and create a new one whenever the button was created (that was done with angular.element('.legend.leaflet-control').remove();)
Can probably create a fiddle at some point with an example for anyone else needing this type of functionality.

From @michaelryancaputo on October 9, 2014 17:38
Just found this bug with ng-show/ng-hide - the map on the show/hide doesn't seem to want to draw: http://jsfiddle.net/flashpunk/vosru6t9/1/
Sure feel free to use the example.

From @tombatossals on October 17, 2014 15:33
Hi @flashpunk, just added your example here:
http://tombatossals.github.io/angular-leaflet-directive/examples/double-map-sharing-attributes.html
Thanks for sharing

From @tombatossals on October 16, 2014 6:38
Yes, the ng-show/ng-hide behaviour is related with leaflet, which needs to call "invalidateSize" when the size of the div is changed. You can easily solve this using ng-if instead of ng-show:
http://jsfiddle.net/bL69rzch/1/
I'll add your example to the project, thanks.

From @michaelryancaputo on October 22, 2014 23:16
@Splaktar This isn't the full answer you're looking for, but in the end I was able to render multiple different maps by modifying and using this directive: https://gist.github.com/pospi/85c31530c4a93cf091ce

From @kuanb on June 5, 2015 21:26
Wondering if anyone has any insight on the following TypeError (seems to likely be related to this issue). I have two maps, map_left and map_right. When adding new layers, using leafletData.getMap('map_xxx').then(function(map) { ... } works great but whichever map I attempt to var.addTo(map) second (after one map has already been interacted with), results in the following TypeError:
Uncaught TypeError: Cannot read property 'min' of undefined
o.LineUtil._getBitCode @ leaflet.js:8
o.LineUtil.clipSegment @ leaflet.js:8
o.Polyline.o.Path.extend._clipPoints @ leaflet.js:8
o.Polyline.o.Path.extend._updatePath @ leaflet.js:8
o.Mixin.Events.fireEvent @ leaflet.js:6
o.Util.bind @ leaflet.js:6

I should not this occurred when employing leafletData.getMap('map_left').then(function(map) { ... }.

Did this ever get fixed? I am still having this issue.

Related #261 #250 #87


File size is too big

I have just installed react on windows using npx create-react-app my-app version 16.13.1
I have not written any code, but if you check it's size on disk it's 181 megabytes
Why is it so big?

Hi @Elham13,
Create React App contains a great amount of crucial development tooling including but not limited to:

React, JSX, ES6, TypeScript and Flow syntax support.
Language extras beyond ES6 like the object spread operator.
Autoprefixed CSS, so you don’t need -webkit- or other prefixes.
A fast interactive unit test runner with built-in support for coverage reporting.
A live development server that warns about common mistakes.
A build script to bundle JS, CSS, and images for production, with hashes and sourcemaps.
An offline-first service worker and a web app manifest, meeting all the Progressive Web App criteria. (Note: Using the service worker is opt-in as of<EMAIL_ADDRESS>and higher)
Hassle-free updates for the above tools with a single dependency.

When you build your app with npm run build, the built size will be a fraction of that as only the relevant bits and pieces in your app are included to the built product and all the development additions are stripped off.


Honor application env in globbed configs

Added a check to see if the application env exists in the loaded
configuration; if so, resets the configuration to the env-specific
configuration.

Rob -- I think the changes in my last commit (c1b605a) address the concerns you have.


add example catalogsource

This example catalogsource can be applied to an OpenShift cluster to be
able to install the operator via the web console.
Signed-off-by: Jens Freimann<EMAIL_ADDRESS>- Description of the problem which is fixed/What is the use case
- What I did
This example catalogsource can be applied to an OpenShift cluster to be
able to install the operator via the web console.
- How to verify it
oc apply -f config/samples/catalogsource.yaml
Go to web console and try to install via Operatorhub
- Description for the changelog
Added an example for a working catalogsource that can be used for installing the Operator via the web console

/lgtm


[BUG]: App launcher hardcodes Public Preview cloud runtime URI, breaking regions other than Preview

Describe the bug
App launcher page linking to the five apps hardcodes the Public cloud runtime URI, such that one cannot get to any of the apps in deployments other than the Preview stamp.
To Reproduce
Steps to reproduce the behavior:

Deploy to anywhere other than a Preview stamp via the public instructions, https://docs.microsoft.com/en-us/powerapps/sample-apps/emergency-response/deploy-configure, including inputting the five Canvas appids
Import sample data
Run the Canvas root app
Click on any individual app, which navigates to
https://apps.preview.powerapps.com/play/980e2c01-2bad-4ad0-8ebf-4946aecc1713?FacilityID=c738f79c-e06f-ea11-a811-001dd801898c

Expected behavior
Navigation instead to
https://apps.gov.powerapps.us/play/980e2c01-2bad-4ad0-8ebf-4946aecc1713?FacilityID=c738f79c-e06f-ea11-a811-001dd801898c for GCC, or apps.powerapps.com for Public, or ...

We will be rolling out a fix for this issue.

Issue resolved with latest update.


e2e: Killed gossip e2e tests.

:fire: :fire:
Signed-off-by: Bartek Plotka<EMAIL_ADDRESS>

Just out of curiosity what is the timeline to drop the gossip?

@FUSAKLA

Just out of curiosity what is the timeline to drop the gossip?

Not sure yet, I know people still use it. I think we will do next release as the last one with gossip.


Placed the credits bar at the bottom and Fixed the footer Date

Resolved the issue raised in #16 and #18.
You can verify it.

It fixes the problem for bigger screens, but it bugs with smaller ones.

@Fonta22 Yeah I looked it over and made some changes. Can you review it and let me know.

Looks good! Thanks!


Check for previous/existing GitHub issues/module proposals

[X] I have checked for previous/existing GitHub issues/module proposals.

Check this module doesn't already exist in the module indexes

[X] I have checked for that this module doesn't already exist in the module indexes.

Bicep or Terraform?
Terraform
Module Classification?
Resource Module
Module Name
avm-res-network-virtualnetworkgateway
Module Details
AVM Resource Module for Virtual Network Gateway
Do you want to be the owner of this module?
No
Module Owner's GitHub Username (handle)
No response
(Optional) Secondary Module Owner's GitHub Username (handle)
No response

I'm interested in being an external contributor once a module owner has been found :)

@PmeshramPM i think that this is covered by this module https://github.com/Azure/Azure-Verified-Modules/issues/157
agree?

Closing as we have existing pattern.


Currently it's not buildable and not working properly with Opencart

I think the mysqli module is not enabled at all and it can not connect to the database.
I need to check again, but I think the file placement (/www, /upload etc.) are also not consistent between the containers.

This PR contains the commits needed to build the images https://github.com/DockerIt/opencart-docker/pull/13. Regarding file placements, you need to adjust WEB_DOCUMENT_ROOT in the ../apache/Dockerfile and specify correct paths into volumes of docker-compose.yml. As example:
volumes:

../src:/var/www


running the LSP on a remote machine / using a remote LLM LSP server: documentation / if necessary general implementation (?)

Many thanks again for an awesome tool :) .
If I understand correctly, it is possible to run the LSP on a remote machine (for example running the LSP on a GPU server, while coding on a laptop, at least for a specific LLM backend https://github.com/SilasMarvin/lsp-ai/pull/15 ).
This is actually a common user case I think - for example at companies, it would be much more convenient to have a shared GPU server running the LLMs, and employees connecting their LSP editor client to it, rather than each employee getting a GPU and running a LLM server on their own machine.
It is not clear to me if the use of a remote server to actually run the LLM is only supported for a specific model after the PR above, or if it is a general feature. If it was possible to support it as a general feature, this would be very convenient :) . Not sure what the architecture of lsp-ai is at now, and not sure for example if the server vs "client" part of the LSP are decoupled and if this would be as simple as ssh-piping from a machine running the LLM to the local machine on a given port. Could this be documented / explained as an example of deployment? :)
This would have a lot to say for adoption, since I am afraid at present it is not realistic to hope that each developer would have a powerful enough GPU in their machine running a LLM, and without such hardware, the models that can be run are very small due to both latency and overall memory use considerations :) .

I'm glad you like it!
LSP-AI is a language server: https://en.wikipedia.org/wiki/Language_Server_Protocol It is designed to be run locally on your computer alongside the editor you are using. You should not host LSP-AI remotely.
GPUs  become useful when doing text-generation with large LLMs. For this use case, LSP-AI supports making API requests. LSP-AI can make API requests to:

OpenAI or OpenAI compatible APIs like fireworks, VLLM, etc...
Anthropic or Anthropic compatible APIs
Ollama

The API you use is dependent on the model you want to use. The endpoints are all configurable. See: https://github.com/SilasMarvin/lsp-ai/wiki/Configuration
Let me know if you have any other questions. I really do need to improve the general documentation around configurations and how to properly use it.


[YZHSDImageCache queryDiskCacheForKey:done:]_block_invoke

New Issue Checklist

[x] I have read and understood the CONTRIBUTING guide
[x] I have read the Documentation
[x] I have searched for a similar issue in the project and found none

Issue Info

Info
Value

Platform Name
e.g. ios

Platform Version
e.g. 8.0

SDWebImage Version
e.g. 3.7.6

Integration Method
e.g. carthage / cocoapods / manually

Xcode Version
e.g. Xcode 8.3.3

Repro rate
e.g. all the time (100%) / sometimes x% / only once

Repro with our demo prj
e.g. does it happen with our demo project?

Demo project link
e.g. link to a demo project that highlights the issue

Issue Description and Steps
Please fill in the detailed description of the issue (full output of any stack trace, compiler error, ...) and the steps to reproduce the issue.
Decompressing images that are downloaded and cached can improve performance but can consume lot of memory.I will canche. Config. ShouldDecompressImages and downloder shouldDecompressImages set to no, but have no effect. Still images decompression method.

You need to let us know which version of the library are you using. Why did you not fill these info in the issue template and just left them with default values?
More to that, probably your setting is not done properly, please debug and stop on that line, check the setting.


s/camelCase/snake_case/g

PEP8 plz :scream:

:+1:

Where do you saw problems with pep8 ? Being more explicit would be helpful to the maintainers.

@ramnes: AutobahnPython is PEP8-compliant except for variable naming and long lines.
For more information: https://groups.google.com/forum/?hl=fr#!searchin/autobahnws/PEP8/autobahnws/EMKF4a0aXKI/_wBcjGixXNcJ

@ramnes: AutobahnPython is PEP8-compliant except for variable naming and long lines.

So basically, it's not PEP8-compliant, right?

@ramnes Short answer, yes.

@ramnes there are hundreads of PEP8 rules .. and AB conforms to all but 2.

Yeah, well, those are the two most important rules of the PEP8 to me (and to a lot of developers). I won't argue on why they are important, the PEP8 is there for a reason. Why not just following them?

So basically, it's not PEP8-compliant, right?

This is plan wrong.

those are the two most important rules of the PEP8 to me (and to a lot of developers)

There are different opinions on that. But we have that discussion .. I'd be ready .. thing is: it'll break user's apps.

The more the project will grow, the more you'll have unhappy Python developers giving some hate like me because of the bad naming convention, so if I were you, I'd really think it would be better to change sooner than later.

@ramnes I can follow that thinking. if any, now (<1.0) is the time to break stuff https://github.com/tavendo/AutobahnPython/issues/463

That's it.

related: https://github.com/tavendo/AutobahnPython/issues/463

It's worth noting that changing some sections of Autobahn (the Twisted portion, for example) to snake case actually breaks PEP8 (due to this section: https://www.python.org/dev/peps/pep-0008/#a-foolish-consistency-is-the-hobgoblin-of-little-minds). Consistency with the framework it's written against (first Twisted, now Twisted and asyncio) is better than consistency with the default PEP8 rules.
A truly PEP8 compliant Autobahn|Python would use snake_case when interfacing with asyncio (eg. on_join()), and camelCase when interfacing with Twisted (eg. onJoin()). PEP8 is about readability, and that section says that the default rules should not apply when:

When applying the guideline would make the code less readable, even for someone who is used to reading code that follows this PEP.

That is -- a Twisted user using Autobahn would find it less readable to call a snake_case API, as an asyncio user using Autobahn would find it less readable to call a camelCase API.

(however the long lines are a problem to be fixed, Twisted and PEP8 are 79 characters. That being said, PEP8 says 72 characters for docstrings, and Twisted says 79. I'm not sure why they have reason to be different, and I'm nearly certain that Twisted's is better.)

@hawkowl this is interesting. I didn't knew about this rule. that makes it even more complicated for us, since a goal of AB is to make it possible to write user code that is source compatible over tx and asyncio. it's even more of a problem with https://github.com/tavendo/txaio, as we definitely MUST have a single API on that (as we are using it in AB to make single-source over tx/aio possible)

@hawkowl You're entirely right. But as you just told, Autobahn deals with multiple libraries written with different rules, so rather than choosing one to follow randomly, I would rather choose the official Python rules.

so rather than choosing one to follow randomly,

I think what @hawkowl ment is: have PEP8 on the asyncio API of AB, and tx standards on the tx flavor of AB's API ..

@oberstet One possible solution would be to have a canonical API (which is PEP8 default snake_case), and then the Twisted ones expose a Twisted-style API (therefore making users codebases consistent, and making PEP8 happy).
In this case -- internal Autobahn is completely reasonable to make uniformly snake_case -- but then we selectively expose APIs which match the target platform. txaio is not a public interface in Autobahn, so it remains snake case.

The voice of Truth.

we're gonna address that. pls see https://github.com/tavendo/AutobahnPython/issues/496


Create authentication guard

Follow guides at

https://docs.nestjs.com/security/authentication
https://docs.nestjs.com/microservices/guards
https://www.npmjs.com/package/nest-keycloak-connect

Integration with Keycloak is advised
Wrap the new functionality in new AccountModule

Done


Ability to execute IPython (Jupyter) Notebook Code

Issue: Academic users will often frame their python code within a Jupyter notebook. Examples:

Notebook Estimate (simpler code)
Deep Blind Watermark Removal (more complex code)
Meta AI Protein Folding (more complex code)

Proposed solution: enable Bacalhau to execute common python code (and supporting libraries) used in academic papers
Technical implementation: tbd, love to hear your guys suggestions. Perhaps wrapping the jupyter notebook in a container would simplify execution?
Prior art:
Zero to Binder
Code Ocean Capsules

Inspiration:
Zero to Binder

Let's overlap this work with the The SAME project.


⚠ Telly has degraded performance

In b9f384f, Telly ($TLY) experienced degraded performance:

HTTP code: 200
Response time: 1523 ms

Resolved: Telly performance has improved in 51db8c2.


Updated a Link

A link was deprecated. So, updated it

Summary

Ticket Link

Hello @Sayanta66,
Thanks for your pull request! A Core Committer will review your pull request soon. For code contributions, you can learn more about the review process here.


Update RightFont to 2.1.0

Thank you for the contribution. It was merged directly as 3016f732681736a4830ac78e2e78f628aed9db9d to keep commit history cleaner. Your contribution is still credited to you.


Code license?

Could you provide details of the license the code in this repo is published under?
Some of the files have explicit MIT headers, but some don't and there's no repo-level LICENSE file. Specifically I'm interested in using the low-level ov7670.{c,h} register settings in my project.
Thanks!

Added LICENSE file and SPDX headers throughout. All MIT, knock yourself out, thank you for asking!


Created file structure

The file structure has been explained in the README file.

All seems good. Just one thing - @FlyingSaturn please put all the flask code into a backend folder, as the frontend is gonna be in react. We will have a frontend folder for it.
We can merge it then.

folder structure changes have been made in #4

Closing this as we planned to shift to a JS backend only.


Evaluate configuration management tools

Trying out ansible now (finally in year 2020).

Tutorial https://serversforhackers.com/c/an-ansible-tutorial
$ ansible all -m ping
[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to allow bad characters in
group names by default, this will change, but still be user configurable on deprecation. This feature
will be removed in version 2.10. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details

130s-mac1.local | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh<EMAIL_ADDRESS>Permission denied (publickey,password,keyboard-interactive).",
    "unreachable": true
}
130s-rpi2.local | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh<EMAIL_ADDRESS>Permission denied (publickey,password).",
    "unreachable": true
}

I don't have ssh keys set up on the server for both of these nodes.
https://docs.ansible.com/ansible/latest/user_guide/intro_getting_started.html#action-check-your-ssh-connections

Confirm that you can connect using SSH to all the nodes in your inventory using the same username. If necessary, add your public SSH key to the authorized_keys file on those systems.

I'm honestly not good at memory enough to remember how to do these. Looking at https://www.youtube.com/watch?v=d6jTzve7mFY I ended up doing this on the ansible server side (the host where ansible runs on):
ssh-copy-id %HOST_NAME_TARGET%

Then, running ansible all -m ping started asking differently:
Enter passphrase for key '/home/usaaaaa/.ssh/id_rsa':

Typing in the passphrase lets ping worked.
$ ansible all -m ping
[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to allow bad characters in
group names by default, this will change, but still be user configurable on deprecation. This feature
will be removed in version 2.10. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details

[WARNING]: Platform darwin on host 130s-mac1.local is using the discovered Python interpreter at
/usr/bin/python, but future installation of another Python interpreter could change this. See
https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more
information.

130s-mac1.local | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": false,
    "ping": "pong"
}
[WARNING]: Platform linux on host 130s-rpi2.local is using the discovered Python interpreter at
/usr/bin/python, but future installation of another Python interpreter could change this. See
https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more
information.

130s-rpi2.local | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": false,
    "ping": "pong"
}

Then I started to wonder whether https://serversforhackers.com/c/an-ansible-tutorial is outdated as some commands fail:
$ ansible all -s -m shell -a 'apt-get install nginx'
usage: ansible [-h] [--version] [-v] [-b] [--become-method BECOME_METHOD]
               [--become-user BECOME_USER] [-K] [-i INVENTORY] [--list-hosts]
               [-l SUBSET] [-P POLL_INTERVAL] [-B SECONDS] [-o] [-t TREE] [-k]
               [--private-key PRIVATE_KEY_FILE] [-u REMOTE_USER]
               [-c CONNECTION] [-T TIMEOUT]
               [--ssh-common-args SSH_COMMON_ARGS]
               [--sftp-extra-args SFTP_EXTRA_ARGS]
               [--scp-extra-args SCP_EXTRA_ARGS]
               [--ssh-extra-args SSH_EXTRA_ARGS] [-C] [--syntax-check] [-D]
               [-e EXTRA_VARS] [--vault-id VAULT_IDS]
               [--ask-vault-pass | --vault-password-file VAULT_PASSWORD_FILES]
               [-f FORKS] [-M MODULE_PATH] [--playbook-dir BASEDIR]
               [-a MODULE_ARGS] [-m MODULE_NAME]
               pattern
ansible: error: unrecognized arguments: -s

Then also I do see errors with playbook sample:
$ more nginx.yml
---
- hosts: local
  tasks:
   - name: Install Nginx
     apt: pkg=nginx state=installed update_cache=true

$ ansible-playbook nginx.yml
[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to allow bad characters in group names by default, this will change, but still be user configurable on deprecation. This feature will be
removed in version 2.10. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details

PLAY [all] ********************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ********************************************************************************************************************************************************************************************
Enter passphrase for key '/home/n130s/.ssh/id_rsa': [WARNING]: Platform darwin on host 130s-mac1.local is using the discovered Python interpreter at /usr/bin/python, but future installation of another Python interpreter could change this. See
https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information.

ok: [130s-mac1.local]

[WARNING]: Platform linux on host 130s-rpi2.local is using the discovered Python interpreter at /usr/bin/python, but future installation of another Python interpreter could change this. See
https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information.

ok: [130s-rpi2.local]

TASK [Install Nginx] **********************************************************************************************************************************************************************************************
fatal: [130s-mac1.local]: FAILED! => {"changed": false, "msg": "value of state must be one of: absent, build-dep, fixed, latest, present, got: installed"}
fatal: [130s-rpi2.local]: FAILED! => {"changed": false, "msg": "value of state must be one of: absent, build-dep, fixed, latest, present, got: installed"}

PLAY RECAP ********************************************************************************************************************************************************************************************************
130s-mac1.local            : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
130s-rpi2.local            : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Switched to a tutorial ansible.com.
I started seeing Enter passphrase for key multiple times whenever ansible command is run, but I noticed it's a common issue https://stackoverflow.com/questions/40735635/ansible-using-ssh-key-authentication-but-asked-multiple-times-for-passphrase. I just have to run ssh-add once per terminal, same as I do with git using ssh.
failure

Seems to be working:
$ more apt_distupgrade.yml
---
- hosts: ubuntu
  tasks:
    - name: Update and upgrade apt packages
#      become: true
      become: yes
      apt:
        update_cache: yes
        upgrade: 'yes'

$ ansible-playbook -i inventory_southlite.yml apt_distupgrade.yml -k -K
SSH password:
BECOME password[defaults to SSH password]:

PLAY [ubuntu] *************************

:

[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host noodlers-serval.local should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible
release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information. This feature will be
removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
ok: [noodlers-serval.local]
[WARNING]: Platform linux on host noodlers-rpi2.local is using the discovered Python interpreter at /usr/bin/python, but future installation of another Python interpreter could change this. See
https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information.

ok: [noodlers-rpi2.local]
fatal: [noodlers-p50.local]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname noodlers-p50.local: Name or service not known", "unreachable": true}

TASK [Update and upgrade apt packages] *******************************
[WARNING]: Updating cache and auto-installing missing dependency: python-apt

changed: [noodlers-kudu1.local]
changed: [noodlers-serval.local]

changed: [noodlers-rpi2.local]

PLAY RECAP **************************************************
noodlers-kudu1.local           : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
noodlers-p50.local             : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0
noodlers-rpi2.local            : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
noodlers-serval.local          : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

https://stackoverflow.com/questions/25582740/missing-sudo-password-in-ansible was a hint for sudo password issue
become entry wasn't bool for me. It was yes/no instead.

Getting slightly comfortable with ansible. Closing.
Now running apt on all of my Ubuntu computers is done by a single command from a single host.
$ ansible-playbook -i inventory_foo apt_distupgrade.yml -k -K

apt_distupgrade.yml
---
- hosts: ubuntu
  tasks:
    - name: Update and upgrade apt packages
      become: yes
      apt:
        update_cache: yes
        upgrade: 'yes'


missing uploaded images

Weights and Biases version: 0.8.13
Python version:  3.6.4
Operating System: Darwin

Description
Testing the image upload functionality I can't understand why I only get 19 uploaded images, for some reason the last image is not on the run's page
What I Did
$ ipython

In [1]: import numpy as np

In [2]: import time

In [3]: import wandb

In [4]: wandb.init()
   ...: for i in range(20):
   ...:     wandb.log({
   ...:     "some_image": [
   ...:         wandb.Image(
   ...:             np.random.randint(0, np.random.randint(100, 256), (256, 256, 3)),
   ...:             caption='image ' + str(i) + " " + str(j))
   ...:         for j in range(3)
   ...:         ]
   ...:     }, step=i)
   ...:     time.sleep(1)

<IPython.core.display.HTML object>
Failed to query for notebook name, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable

In [5]:
Do you really want to exit ([y]/n)? ^D

This happens consistently: last image never uploaded

This is a bug in Jupyter.  We’re working on a fix.  For now you can call wandb.log({}) once outside of your forloop to push the final image.

Hey @vict0rsch
In the past year we've majorly reworked the CLI and UI for Weights & Biases. We're closing issues older than 6 months. Please comment to reopen.


Repeat mode is set to repeat the queue, but stops playing when the queue is finished

What version of Music Assistant has the issue?
2.3.3
What version of the Home Assistant Integration have you got installed?
2024.12.4
Have you tried everything in the Troubleshooting FAQ and reviewed the Open and Closed Issues and Discussions to resolve this yourself?

[X] Yes

The problem
I have a playlist with 30 songs on my local filedisk. I play the playlist on a Sonos device. I set the repeat mode to repeat the entire queue. Yet, when it reaches the final song, it stops playing music. I tried creating a new playlist with only two songs, same issue. I disabled the random order, same issue.
How to reproduce

play a playlist on the Sonos device
set the repeat mode on repeat the queue
wait untill the last song of the playlist is played
the music stops instead of repeating the queue
look at the screenshot: one song remains in the queue and the current played song stops at the last second:

Music Providers
local file disk: mp3 files
Player Providers
SONOS device
Full log output
No response
Additional information
No response
What version of Home Assistant Core are your running
2024.12.4
What type of installation are you running?
Home Assistant OS
On what type of hardware are you running?
Raspberry Pi

You have ticked that you have read the troubleshooting docs but you have not supplied the required info from there nor the template instructions


Up cloudwatch dependency to 1.9.30 and tick version

This change is

Reviewed 1 of 1 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


Feature Request: Fan tile speed selection on long press

The fan tile doesn't increase icon size which means the icon is much smaller compared to other tiles if you use LARGE entity size.
Can the speed be displayed in addition to state in the top right corner? This will remove the speed text and allow icon to be large like other tiles.
And to set speed, long press to see a set of tiles just like alarm tile.

Speed can not be in the state and is unnatural behavior. We have player with the fan tile for almost a week and this is the best and most logical way to have it at the moment.
There is no long-press on the alarm tile.

There was no icon before at all, we've managed to squeeze one inside while keeping the speed selection.  When you say it looks too small, can you show me a screenshot please?

Looks OK to me. You probably need to set group height to 4 to have that automatons on the right to be in line with the rest.

Ah thanks for the tip. Closing.


Setup Find3

ref #153
TODO

[x] rollout changes to all bluetooth scanners
[x] Integrate find3 mqtt with Home Assistant (using bridge)
[x] reset the database to remove non-sense testing data
[ ] configure the passive scanning parameters

updates are being sent out every ~2 seconds from bluetooth scanners, so set scan window to 5 (2.5 times scanner time)
require a minimum of 3 scanner results

[ ] train find3 in living_room
[ ] train find3 in kitchen
[ ] train find3 in hallway
[ ] train find3 in toilet
[ ] train find3 in landing
[ ] train find3 in baby_room
[ ] train find3 in bedroom
[ ] train find3 in office
[ ] tain find3 in bathroom
[ ] Generate password for find3 internal mqtt server
[ ] Secure access to the find3 api?
[ ] Configure http timeout in ble scanners
[ ] Consolidate post/url etc in http client
[ ] what happens if we rollout a change to a moko, will it turn of the power? Because this is not handy for the office-server-moko 😅

Unfortunately the http_request component is not yet available for esp-idf (https://github.com/esphome/esphome/pull/3256) so have to hack something together from https://github.com/espressif/esp-idf/blob/3df87a91a3c876d6ef654e68b66f4939080fa1fc/examples/protocols/esp_http_client/main/esp_http_client_example.c.

Fixed mqtt connectivity from find3 to mqtt using https://github.com/schollz/find3/issues/102#issuecomment-1386019410.

Payload to start training for a location with accurate settings
{
    "family":"home",
    "location": "living_room",
    "device":"bluetooth-00:00:00:00:00:00",
    "minimum_passive": 3,
    "window": 5
}

Rolling out a esphome update to the server Moko doesn’t restart it! 🎉


[Enhancement] Add a runtime config to enable log stream load request/response

Why I'm doing:
many user may not log stream load http request/response, so we need to log it online when we debug or tuning.
What I'm doing:
add enable_stream_load_verbose_log to control it
Fixes #issue
What type of PR is this:

[ ] BugFix
[ ] Feature
[x] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Does this PR entail a change in behavior?

[ ] Yes, this PR will result in a change in behavior.
[x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

[ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
[ ] Parameter changes: default values, similar parameters but with different default values
[ ] Policy changes: use new policy to replace old one, functionality automatically enabled
[ ] Feature removed
[ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

[ ] I have added test cases for my bug fix or my new feature
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[x] I have checked the version labels which the pr will be auto-backported to the target branch

[x] 3.2
[x] 3.1
[x] 3.0
[x] 2.5

better to support runtime modification of sys_log_verbose_modules and sys_log_verbose_level configs?

better to support runtime modification of sys_log_verbose_modules and sys_log_verbose_level configs?

sys_log_verbose_modules only can control namespace level

better to support runtime modification of sys_log_verbose_modules and sys_log_verbose_level configs?

sys_log_verbose_modules only can control namespace level


Bump golang.org/x/sys to fix compilation on M1 macs

Was the compilation broken? I believe tests cover building binaries at each commit for all platforms.

Yes, on go 1.18 for m1 macs.
see https://github.com/google/go-jsonnet/issues/596 for the same issue just on another repository


Implement a http fuzzer for nettacker

Checklist

[x] I have followed the Contributor Guidelines.
[x] I have added the relevant documentation.
[x] My branch is up-to-date with the Upstream master branch.

Changes proposed in this pull request
This pull request implements the http fuzzer library which will be used in all the HTTP fuzzing modules.
Work to do:

Implement all the types of requests (GET, POST, PUT, PATCH, DELETE, HEAD)
Add JSON body POST/PUT/PATCH requests type [1]
Implement multithreading
Implement filters and rules
Real time event system
Write the docstrings for each function
Update the wiki

Your development environment

OS: Ubuntu
OS Version: 16.04
Python Version: 2.7

Hey,
I've updated the todo list, thanks for your consideration.
Regards.

Hello,
thanks for the contribution, it works perfectly now, we need to support upload file request for later, I merged the PR. I will create a new issue to implement this structure to all HTTP-Based modules.
Best Regards.


Add database and schema as options to models at 03_mart and 04_metric

resolves #
This is a:

[ ] documentation update
[ ] bug fix with no breaking changes
[x] new functionality
[ ] a breaking change

All pull requests from community contributors should target the main branch (default).
Description & motivation
This pull request is meant to give option to save tables bi_column_analysis, bi_dq_metrics and metricflow_time_spine in the same database and schema as dq_issue_log. In addition, there are circumstances where the default target.database  is  reserved for other purposes other than data quality checks, for example the the landing (or raw) layer at medallion architecture.
Checklist

[ ] This code is associated with an Issue which has been triaged and accepted for development.
[x] I have verified that these changes work locally on the following warehouses (Note: it's okay if you do not have access to all warehouses, this helps us understand what has been covered)

[ ] BigQuery
[ ] Snowflake
[x] DataBricks

[ ] I have updated the README.md (if applicable)
[ ] I have added tests & descriptions to my models (and macros if applicable)

Hey @rodrigofp-possiblefinance thanks for actively contributing this PR!
It makes sense with the request but I would rather than call this the enhancement with new behavior, that means I can approve it and release it into v1.6.0.
For this change, is it better if we configure it in dbt_project.yml file instead? It gives flexibility to users who want to store them in another schema:
models:
  dq_tools:
    +database: ...
    +schema: ...

You could also do this in your dbt project without requiring this change.
Let me know what you think.
Thanks

The PR has been created - https://github.com/infinitelambda/dq-tools/pull/31


Explicitly set default timeouts for remaining requests calls

Explicitly add the timeout parameter to the remaining requests calls.
A current bump to pylint (which is not currently pinned) added this check which is making CI runs fail.

The change to tests/test_soap.py breaks the test for me when running them locally. Try running: pytest -k test_call ..
I'll start looking at getting the check jobs functioning again.

Hi @jjlawren: can you pull in the changes from the current master and massage the conflicts in soco/music_services/accounts.py and soco/soap.py where I've temporarily added the pylint: disable= comments.
Then, let's see if the check jobs run successfully, because the test_call test is still failing for me locally.