Can't destroy app

I can't seem to destroy any apps, no matter what tweaks I make to the acls -- I'm always denied:

dokku@dokku:~/.dokkurc$ dokku apps:destroy 2222
!     WARNING: Potentially Destructive Action
!          To proceed, type "2222"
2222
Destroying 2222 (including all add-ons)
User  does not have permissions to modify this repository

you’re destroying the app via command line, so I guess the $NAME variable is unset, effectively comparing an empty string to the ACL list.
Could you try if this patch works?
diff --git a/pre-build b/pre-build
index b65dfc6..1c1b15c 100755
--- a/pre-build
+++ b/pre-build
@@ -2,6 +2,7 @@
 set -eo pipefail; [[ $DOKKU_TRACE ]] && set -x

 APP="$1";
+NAME=${NAME:-$USER}
 ACL="$DOKKU_ROOT/$APP/acl"
 DOKKU_SUPER_USER="${DOKKU_SUPER_USER:-}"

sorry for the extreme delay. Unfortunately no, that doesn't appear to have worked.
$ dokku ps:rebuild app.dnsfilter.com
User dokku does not have permissions to modify this repository
$ dokku acl:add app.dnsfilter.com dokku
$ dokku ps:rebuild app.dnsfilter.com
Or perhaps it did?
I had my account added to the dokku group; so I could run dokku commands...

I’m not quite sure what’s happening. There is no error message any more, is there?

Well; once I added 'dokku' then it works; so thats why I was saying maybe it did work... I just wasn't expecting 'dokku' to be the user I had to add; but instead one of the users listed in the acl settings...

I also cannot destroy an app: I see the following error message:
User  does not have permissions to modify this repository

We should probably whitelist the dokku user.

👍 , I can't destroy apps either 😢

Yes, I think adding the dokku user solved the problem

It looks like at least one variant of this issue is caused by not handling commands run locally on the server (instead of through ssh) correctly. The easiest fix would be to modify the pre-* hooks to allow this case, but it's possible that some user depends on the existing behaviour for their security.
Therefore I believe the best fix is to add a configuration variable to enable local commands that check pre-* hooks, and fix the error message shown when a local command is denied.

The original behaviour (before any of my PRs) is weird and probably unintended. Here's what happens in various situations when a command that checks a pre-* hook is called from the command line:

No app ACL, no DOKKU_SUPER_USER: command succeeds
No app ACL, DOKKU_SUPER_USER=testadmin: Only testadmin can modify a repository if the ACL is empty
App ACL exists, no DOKKU_SUPER_USER: command succeeds
App ACL exists, DOKKU_SUPER_USER=testadmin: User  does not have permissions to modify this repository

(These behaviours are the same whether you run as user dokku or user root.)
I'm still concerned that someone might depend on behaviours 2 or 4, so I don't want to change the default behaviour of this plugin.
I think the best way forward is to add a DOKKU_ACL_ALLOW_COMMAND_LINE flag that defaults to false, and recommend in the documentation that users enable this flag. I'll also improve the error messages in cases 2 and 4.
If someone needs more advanced behaviour (like denying command line access, or restricting it to certain users), they can submit a PR...
@josegonzalez Any opinions on this?

Sounds fine with me, though the dokku user should probably be whitelisted be default.

@josegonzalez Unfortunately there doesn't seem to be a way to tell the dokku user apart from a user who's using dokku via sudoers as described in: http://dokku.viewdocs.io/dokku~v0.8.2/deployment/user-management/#granting-other-unix-user-accounts-dokku-access
I don't want to open this up by default, which is why I propose preserving the existing behaviour (but giving a better error message so folks can figure out what's going on).

Now that I'm further into my fix, I'm even more convinced that this is the right thing to do. The weird behaviour only occurs if DOKKU_SUPER_USER is set, and in that case the user will have read the documentation and will know to set DOKKU_ACL_ALLOW_COMMAND_LINE as well.


new_shell

[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: c7f835af-5905-368a-df68-6646782ad807
Version Independent ID: 153ce2d2-dab5-9c8f-c511-ee3bc8ee2dbc
Content: Set-Mailbox (ExchangePowerShell)
Content Source: exchange/exchange-ps/exchange/Set-Mailbox.md
Service: exchange-powershell
GitHub Login: @chrisda
Microsoft Alias: chrisda

@WatchTower001110, is there a question in there somewhere? Are you asking about the vast number of parameter sets (a.k.a. syntax blocks) in Set-Mailbox?


Grafeas integration

We are looking at Grafeas for managing our software supply chain metadata. As vulnerability scanning is a key component in this process I would like to integrate Clair with Grafeas. I'm currently working at doing such an integration by writing a third party client but I was thinking that maybe this could be done better by doing the integration directly between Clair and Grafeas instead.
I don't have a proposal for how to best do this (or if it is even suitable), but if there are others that are interested maybe this could be the start of a discussion.

I've actually spoken and worked with the Grafeas team and I think an integration between Grafeas and Clair is a great idea.
I need to take a deep dive into their codebase, because it might make sense that Clair not only tag Artifact metadata in Grafeas, but also use it as a backing store rather than something like Postgres.
@R2wenD2 A penny for your thoughts?

Yes! Grafeas is a suitable backing store for Clair. I'd be happy to collaborate here.

Excellent!
Regarding Grafeas as a backend for Clair I think that Grafeas would be a great backend for CVE:s (both notes and occurrences in Grafeas lingo). But Clair stores other things as well, like data related to notifications and more. I don't think this is something that Grafeas is suitable for as Grafeas is not a general purpose database. So while I am certainly not sure, I suspect that there still is need for something like Postgres.
Also: I will most likely be able to contribute to this effort if we move forward.

@furuholm @jzelinskie what are your thoughts on creating a new storage implementation that uses Grafeas for this?
We have an issue in Grafeas to have an audit trail (https://github.com/Grafeas/Grafeas/issues/2) which would allow us to insert the notification message in Grafeas.
We'll probably want an actual design doc - and perhaps a joint datastore that allows one to use both existing clair and Grafeas datastores while the Grafeas api is built out to support audit tracking.

@clrprod might have some feedback here as well.

Having clair write directly to grafeas for vulnerabilities would be pretty awesome!
I agree with furuholm@ that notifications likely wouldn't work (there is some work, but I don't think it's live in grafeas yet). I also think a whole other bunch of stuff would break (i.e. it looks like there is a layer storage system and ancestry which may be moveable to grafeas), but we likely don't want to move the arbitrary key value store to grafeas.
I know that we use clair by actually creating our own binary that implements the internal structures on grafeas (although we use clair v0.13.6 so it's unlikely to be directly portable). We directly pass a DataStore to the clair Updater where only the vulnerability writing functionality is implemented.
If this is a pretty common use case, maybe we could make a updater only version of the clair binary that writes to grafeas, or have flags / config that can make the normal clair binary not do all of the other functionality.

@R2wenD2 I see a few strawman integrations:
1. Clair storage interface implemented by Grafeas API.
If it is even possible to store all the metadata Clair in Grafeas, I worry that replacing Postgres with Grafeas directly for storage would cause a significant performance degradation for the Clair gRPC API. There's a fairly relational object model used for storage in Clair that if implemented with Grafeas, I don't believe could respond to user's requests with interactive latency. This model understands container filesystem layers and abuses content addressability to reduce repeated computation. We also just designed this model for Clair v3 and would like to get it in the hands of our users before considering another complete rewrite.
2. Clair implements the Grafeas API using its own storage.
Do you have a storage layer besides the sample memory store? It seems like it makes more sense for Clair to implement the Grafeas API using its own storage engine and expose both APIs. This leaves Clair in a scenario where we could only expose a read-only Grafeas API or allow change the scope of the Clair project to include managing all the metadata Grafeas does.
3. Clair implements a worker to synchronize with a Grafeas API/database
This is what @furuholm has been implementing as a strawman. I think this gets us a lot of bang for our buck, but is probably not the best long-term solution because of the many moving parts.

@jzelinskie: some thoughts
1: Agree.
2: I hadn't thought about this option. Not a bad idea! I worry that the Grafeas API would step on Clairs' toes though. What would it mean to add vulnerability notes from the Grafeas API for example? Would those go straight into the Vulnerability table? How would that affect data integrity (like namespace mapping)? The vulnerability notes could go into its own table, but then the whole premise of the idea is kind of lost.
But maybe a subset would work? I'm thinking exposing a read-only version of vulnerability notes would allow CVE:s to be imported into a Grafeas server by a third party or by the Grafeas server itself. Alternatively the Grafeas server could proxy vulnerability requests to Clair. I haven't thought the occurrences through, but my guess is that a read only API could work for this as well. (If you don't want to make Clair Grafeas compliant it could be enough to expose the data needed through some other API).
3: Agree
Either way I would love for there to be an open source grafeas server that was built using the same architecture/structure as Clair, but I'm not sure they should be the same.

Agree that 3 is a good short term model, and there's no real reason to not do that now since we all agree on this approach as a short term solution.
Option 2 is interesting. I think there might be room for a hybrid of 2. We don't need to expand the scope of clair to handle all metadata types supported by Grafeas in order for it to support a Read/Write Grafeas. Consider a "bring your own Grafeas" solution where when running Clair, you can point it at an existing Grafeas instance and have it create vulnerability notes/occurrences there.
@jzelinskie and @furuholm are you both around next week? Would you be open for a quick meeting? It might be easier for us to brainstorm over a vc.

Vc sounds good to me and I'm around all of next week. You have my mail address.

I'm also open to VC next week.

Some more input to the discussion/the planned vc. This time about occurrences.
It seems reasonable to consider a one-to-one mapping between vulnerabilities in Clair and Grafeas notes. The project to which notes shall be written can be a single config parameter as this project will be shared by all users. Detected vulnerabilites in containers however are user or customer specific. So to what project should a vulnerability occurrence be reported? The mapping can be done for Google's container scanner since Google Cloud project ids are used for both the container registry and the container scanning. But I'm not sure this mapping is possible to do in the general case.

@furuholm I've made a branch experiment-grafeas-sync which we can PR your PoC to for safe keeping. This way we can get some code review through the normal processes.

@jzelinskie PR created.

I've tried to outline a proposal. Thoughts?
Problem statement:
Allow third party systems (like Grafeas) to continuously ingest Clair CVE listings and ancestry data (i.e. container image metadata and their identified vulnerabilities).
Requirements:

Allow access to current state as well as state changes such as added/removed/changed CVE:s and  ancestries.
Support multiple delivery mechanisms.

Proposed solution:

Clair exposes a (golang) extension point for notifications that is triggered when changes occur. The extension is implemented and configured like the existing Clair extensions.
Potential implementations:

message busses like amqp,
pub/sub systems like Kafka or Google Cloud Pub/Sub,
the current Clair notification API?
...

Clair exposes an API that enables users to communicate with the notification extensions to allow e.g. create/delete notification stream. This would

provide a way for external systems to be late joiners.
allow multiple independent consumers to coexist

Not having an API for creating streams would probably require separate mechanisms for accessing the current state and updates as well as some way to synch these two to ensure data consistency.
Open questions:

How should db migrations be handled? From reading the code it looks like clair creates new tables and drops old ones. This will then be followed by populating the db with new data that may or may not match the state of the previous migration. I'm thinking requiring the consumers to be idempotent (accepting that the same CVE or vulnerability to be delivered more than once) could make this a non-issue. Thoughts on this?

Example use case:

Client A uses the API to create a notification stream S.
The notification extension instance starts by writing/publishing the current state to S
The notification extension instance waits for changes and writes/publishes those to S as they occur
C consumes the data through S

If I understand the clair's purpose correctly, multiple sets of input data comes in through drivers. After that, clair has the tooling to inspect an image and cross-reference it with the input data. Why does it have any part in notifying additional tools? (Not being snarky here, just trying to understand all the pieces.:) )
For grafeas I could see a general driver being created that took grafeas api as input. In turn, I could also see having a grafeas api endpoint for querying.
As an example, say there is a tool that produces errata-to-package information (like the existing drivers today). It's api is grafeas and can thus readily be an input driver for clair. Next an image is scanned by clair and associations made to this data. Let's say the tooling that initiated the scan prefers results back in grafeas format. Perfect.
Now later, the errata-to-package info is updated in clair via the periodic driver. Is the proposal that clair should output information to a stream? What would the stream output? If the consumer of the stream is getting the errata-to-package updates, shouldn't they go get the results directly from source instead? If the consumer is expecting to get updated scan results (ie. a new errata shows up for a previously scanned image) how would that scale?

This is not about making grafeas an input source to Clair. It is about making Grafeas a consumer of the data that Clair outputs. The consumer in the use case I laid out above could in the Grafeas case be either Grafeas itself or a tool that writes the reported changes to Grafeas.
Does this make sense or are we talking past each other?

I think I understand. In the experimental grafeas branch, for example, as vulnerabilities are imported they are in turn exported to a grafeas server (via grafeas protobuf api). Your proposal is working towards defining how that would work.
I see that my suggestion of a "grafeas input driver" is a separate idea, though perhaps could benefit from the register notion for late joiners.

I just tried to integrate the fork with the grafeas instance i have and i am not getting anything. Any document on steps need to be followed.

@bansalrajat I would say that the published fork is a dead end. After discussing this further we (me, @R2wenD2 and @jzelinskie) agreed that the way to solve this problem is probably to use a "clair proxy" that forwards scanning requests to clair and stores the result in grafeas as results are returned to the user. This is similar to how Quay works (or at least how it worked at the time of these discussions). Unfortunately the company I work for changed focus about the time we had concluded this, so I never got around to implement it ☹

@furuholm I was trying to integrate grafeas with harbor so the vulnerabilities data could be loaded into grafeas. But seems like it wouldn't be possible for some time. Thanks for the update BTW. Cheers.

We’re declaring bug bankruptcy as part of the release process for a new major version of Clair. Please open a ticket in our issue tracker if you feel this still needs to be addressed, and we'll triage as part of our v4 development process. Thanks!


Friendly user beta

Folks, this is really exciting, carry on! I want to open this ticket to get announced as soon as friendly non-developers can be bugged with using this. (or may this work now? how stable?)

Hi @axel-rutz, thank you for your enthusiasm! Unfortunately the core team that originally created friends has not been maintaining this project for quite some time. If you're interested in trying to get it working again to the point that it can be packaged and released, contributions are very welcome! I would love to see this project get released by someone enthusiastic about finishing what we started 😄

thanks @ungoldman for that information - although it's not what i hoped to hear. so all i can say now is "if someone pushes this further, count me in for some testing and evangelism..".

@axel-rutz there is a light weight alternative https://substack.neocities.org/chatwizard/#lobby that has less features, but works reliably with webRTC enabled browsers.

hey, thanks a lot for that, will definitely check it out!


Fix #506: Yarn issues on Windows

attempting to fix all the Yarn issues on Windows.
--frozen-lock-file is deprecated it is now --immutable

superseded by #532


[9.x] Add price directive for blade

Convert number format to directive blade

Lets call it money or currency

@ankurk91
ok

{{ number_format(5000000) }}
@money(5000000)

I wouldn't do this. This is a pretty domain specific keyword that could be used in lots of apps where the expected implementation would be different (different library like MoneyPHP, decimal handling, etc). This is best defined in userland.

@driesvints
Sometimes can use directive to use php function!

Thanks for your pull request to Laravel!
Unfortunately, I'm going to delay merging this code for now. To preserve our ability to adequately maintain the framework, we need to be very careful regarding the amount of code we include.
If possible, please consider releasing your code as a package so that the community can still take advantage of your contributions!
If you feel absolutely certain that this code corrects a bug in the framework, please "@" mention me in a follow-up comment with further explanation so that GitHub will send me a notification of your response.


Can't discover/connect new peers using webrtc-star on v0.39.5

Peer discovery and connections are not happening with the latest libp2p version (after esm upgrade).
I have created a failing test for both versions (before & after esm), with screenshots - https://github.com/ratik21/p2p-webrtc-star-test. Please check it out. Thanks.
Related discussion thread :: https://discuss.libp2p.io/t/cant-discover-peers-with-webrtc-star-on-js-libp2p-0-37-0/1438
cc: @MarcoPolo

You need to configure the peer discovery component of the WebRTCStar transport for it to work:
For<EMAIL_ADDRESS>and<EMAIL_ADDRESS>import { WebRTCStar } from '@libp2p/webrtc-star'
import { createLibp2p } from 'libp2p'
//... other stuff

const webRTC = new WebRTCStar({ ... })

const node = await createLibp2p( {
  transports: [
    webRTC
  ],
  peerDiscovery: [
     webRTC.discovery
  ],
  //... other stuff
})

For<EMAIL_ADDRESS>and<EMAIL_ADDRESS>import { webRTCStar } from '@libp2p/webrtc-star'
import { createLibp2p } from 'libp2p'
//... other stuff

const webRTC = webRTCStar({ ... })

const node = await createLibp2p( {
  transports: [
    webRTC.transport
  ],
  peerDiscovery: [
     webRTC.discovery
  ],
  //... other stuff
})

The README for v5 is hopefully a bit clearer.

@achingbrain thanks for responding.
With the changes you have suggested the discovery is working (for<EMAIL_ADDRESS>and @libp2p/webrtc-star@4.x.x). But the connection is still not happening (sc attached).

I was wondering if this is the case now (for after-esm versions)? Because connection is being established for same nodes at before-esm version (one at local & one behind NAT -- reference).

@achingbrain one more thing (side note), i start getting this influx of discovery logs and it never stops, after discovery (for any transport/peer discovery mechanism)

This is the event handler code (taken from node1.js from my repo)
  node.addEventListener('peer:discovery', (event) => {
    const peerInfo = event.detail;
    console.log('Discovered:', peerInfo.id.toString());
  });

Discovered nodes are no longer dialled automatically - please see the upgrade guide - https://github.com/libp2p/js-libp2p/blob/master/doc/migrations/v0.37-v0.40.md#autodial

@achingbrain i tested on<EMAIL_ADDRESS>:)

Furthur more, if i try to manually dial after discovery, i get this error
[AggregateError: All promises were rejected] {
  [errors]: [
    AbortError: The operation was aborted
        at file:///Users/ratikjindal/Zero/libp2p-webrtc-star-issue/after-esm/node_modules/@libp2p/webrtc-star/dist/src/index.js:117:26
        at <anonymous> {
      type: 'aborted',
      code: 'ABORT_ERR'
    }
  ]
}

Update to source code:
  node.addEventListener('peer:discovery', async (event) => {
    const peerInfo = event.detail;
    console.log('Discovered:', peerInfo.id.toString());

    console.log("dialing..");
    try {
      const ma = new Multiaddr(`/dns4/vast-escarpment-62759.herokuapp.com/tcp/443/wss/p2p-webrtc-star/p2p/${peerInfo.id.toString()}`);
      await node.dial(ma);
    } catch (error) {
      console.log(error);
    }
  });

@achingbrain I have upgraded the js-libp2p version on this test, but I am still getting the same issue as above.
As an alternative, could you please suggest what protocol to use for connection b/w two peers where one is behind a NAT? Before the esm upgrade webrtc-star servers were working fine. Anyway, can we use circuit relay (v1/v2) now?, i.e <..>/p2p-circuit/p2p/<..>?
Also there is a small bug in the migration guide for v0.40.0, I have created a PR for it.

@achingbrain -- any update on this?

@ratik21 We likely won't pursue this issue since we are actively working to support WebRTC transport https://github.com/little-bear-labs/js-libp2p-webrtc/ and deprecate WebRTC-star (and direct) as a result.

@p-shahi Ok, thanks for responding. few questions :-

Does js-libp2p-webrtc work for server <-> browser, server <-> server as well? Or is it only browser <-> browser?

Assuming browser to browser only, will this transport be able to discover & connect to "new" peers (not known peers but peers who join the network), behind NAT's ? It seems there aren't many ways for NAT traversal in js-libp2p yet :/

@ratik21 That implementation (tracked here https://github.com/libp2p/js-libp2p/issues/1478 now) is for browser to server
How it behaves is specified here: https://github.com/libp2p/specs/tree/master/webrtc
The browser to browser aspect is just beginning to get underway. See https://github.com/libp2p/js-libp2p/issues/1462. It's not specified yet but necessary parts like supporting Circuit Relay v2 are being added.
AutoNat support is also being added, hopefully before the end of this year per our roadmap: https://github.com/libp2p/js-libp2p/blob/master/ROADMAP.md#end-of-q4-december

@p-shahi thanks! I will try js-libp2p-webrtc for server to server, and browser to browser for new discovery and connections

@ratik21 I'll close this issue for now.


Update esphome add-on configuration for Supervisor 2021.2

Proposed Changes
Update esphome add-on configuration for Supervisor 2021.2
See this PR:
https://github.com/esphome/hassio/pull/17
Related Issues
None

Alright, than we should be fine, once the linked PR is merged. Thanks!


Make —order random more deterministic.

When you’re troubleshooting an order dependent
failure, you want to get the repro case down to
a minimal run that loads and runs as few specs
as possible. With the old random ordering implementation,
that was hard to achieve because while rerunning with
a given seed produced the same order when the exact
same set of examples were loaded, the ordering would
be completely different when a subset was loaded.
By ordering by md5(seed + example_id) it ensures
that the ordering of any two examples should stay
consistent regardless of how many other examples are
loaded.
MD5 is significantly slower than shuffle, and it
loads an additional part of our stdlib (which we
want to minimize) but I think the tradeoff is worth it
here.
This will help us be able to implement the --bisect option discussed in #1767.
What do others think?  Is the tradeoff worth it?  Is there some other hashing function we should consider using?  If there was a hashing function that had similar distribution characteristics to MD5 and was faster and easy to use w/o loading part of the stdlib, that would be ideal, but I can't think of one.
/cc @rspec/rspec

@myronmarston was there a reason you went for md5 instead of the #hash of the string?
Using #hash instead of md5 has the advantage of not doing a the require, and might well be faster (the hash algorithm is cheaper because it's not cryptographic). The disadvantage is that we'll likely see more hash value collisions. I'm not sure if that's a problem. Thoughts?

@myronmarston was there a reason you went for md5 instead of the #hash of the string?
Using #hash instead of md5 has the advantage of not doing a the require, and might well be faster (the hash algorithm is cheaper because it's not cryptographic). The disadvantage is that we'll likely see more hash value collisions. I'm not sure if that's a problem. Thoughts?

I tried out hash but the problem is that it is not deterministic across processes.  "foo".hash within a single ruby process returns a deterministic value, but will return a different value in another ruby process:
➜  ~  ruby -v
ruby 2.1.5p273 (2014-11-13 revision 48405) [x86_64-darwin12.0]
➜  ~  ruby -e 'puts "foo".hash'
2464062624698253809
➜  ~  ruby -e 'puts "foo".hash'
-1837013944110794440
➜  ~  ruby -e 'puts "foo".hash'
-1155068896409297005
➜  ~  ruby -e 'puts "foo".hash'
-97344533053661111

Thus, we can't use hash, unfortunately.
Any other ideas for what we can use, @samphippen?  Obviously, "cryptographically secure" doesn't matter here (which MD5 is not, but it was originally intended or that, IIRC), so I imagine there are better hash functions to use than MD5.

@myronmarston in university I implemented this alg: https://en.wikipedia.org/wiki/Fowler–Noll–Vo_hash_function in C, it was pretty legit. Is it worth rolling an impl of some standard hash instead of doing the MD5 require?

@myronmarston in university I implemented this alg: https://en.wikipedia.org/wiki/Fowler–Noll–Vo_hash_function in C, it was pretty legit. Is it worth rolling an impl of some standard hash instead of doing the MD5 require?

Maybe.  I don't really want to get in the business of maintaining a hash function (not my area of expertise or something I care to get involved with) but if there's a simple one that's not much code and has the desired properties, I'm open to it.  I worry that it might be hard to beat MD5 speeds since it's written in C, but maybe that one you linked to is simple enough?

It's worth benchmarking. That algorithm is standard and won't change. MD5 has a bunch of cost associated with startup and hash finalization that cheaper string hashing algorithms don't. I'm just mildly worried that MD5 is a common module that people require and potentially hiding an error for them could be a big issue.

(We should use each_byte to deal with unicode strings tho).

@myronmarston if it helps, here's an implementation of FNV: https://gist.github.com/samphippen/3e8a93757cad9af47f5a

That looks pretty simple. What are the distribution characterstics like?  Can you throw together a benchmark comparing it to MD5?
BTW, is the initial hash value simply a value that's known to work well for the algo?

I'm just mildly worried that MD5 is a common module that people require and potentially hiding an error for them could be a big issue.

We try to minimize stdlib requires as a courtesy to the user to minimize how much RSpec mutates the global namespace but I would never consider it a "big issue" to load a std lib piece.  It's something to weigh against the benefits we gain from using a particular stdlib (and I'm glad we do that), but I think we perhaps worry more than we should.
I like the idea of using this hash function you've proposed, though.

I use a Jenkins hash in one of my projects for work, it's pretty fast

Looking at FNV it's probably faster than Jenkins - there's not many operations in there

@soulcutter -- got a link for jenkins hash?  I've never heard of it.

http://en.wikipedia.org/wiki/Jenkins_hash_function
https://gist.github.com/soulcutter/4c55548bd742ff26a9a0
It looks like the ruby implementation we've got is doing a lot more to try to bound the number. I wonder if that's necessary…

:+1: for using @samphippen's version, he does have a reasonable crypto background too in terms of maintaining it!

Yeah, after reading a bunch of hash wikipedia articles I think FNV's a good choice and the implementation is very reasonable. Benchmarking seems prudent, but I'm relatively certain it's going to beat md5.

Even if it didn't it'd be worth having to avoid the std lib require

Cool.  I'll try pulling in @samphippen's FNV implementation later today or tomorrow.

this is a neat idea

So either @samphippen's implementation of FNV is wrong or FNV's distribution characteristics are awful or I'm doing something wrong I don't see.
require 'digest/md5'

def fowler_noll_vo_hash(string)
  hash = 0xcbf29ce484222325
  string.each_byte do |byte|
    hash = hash * 0x100000001b3
    hash = hash & ((1 << 64)-1)
    hash = hash ^ byte
    hash = hash & ((1 << 64)-1)
  end

  hash
end

def id_from(n)
  "./some_spec.rb[1:#{n}]"
end

def unique_orderings(hash_function)
  list = 10.times.map { |n| id_from(n) }
  1.upto(1000).map do |seed|
    list.sort_by { |id| hash_function.call(seed.to_s + id) }
  end.uniq.count
end

puts "Unique orderings for MD5: #{unique_orderings(Digest::MD5.method(:digest))}"
puts "Unique orderings for FNV: #{unique_orderings(method(:fowler_noll_vo_hash))}"

Produces:
Unique orderings for MD5: 1000
Unique orderings for FNV: 16

Sorting a list of 10 items 1000 times produced 1000 different orderings for MD5 and only 16 for FNV.  Any idea what's up with that result?

OK, I pushed some updates. Notes:

FNV doesn't have a good distribution, as I noted above, but jenkins appears to.  I sorted a list of 10 items using 50K different seeds and got 49,664 unique orderings (which beats MD5 -- it got 49,638!).
While an apples-to-apples comparison of Jenkins and MD5 may show Jenkins is faster (as the benchmark @soulcutter linked to shows), we're comparing Jenkins-in-Ruby vs MD5-in-C, and Digest::MD5 is still significantly faster (about 2x) than Jenkins.
The speed of the hash function looks like it actually doesn't matter much; in my sorting benchmarks, MD5 vs Jenkins are head-to-head and are both dwarfed by shuffle.  It appears that the main difference is sort_by vs shuffle, and the actual thing you sort by matters much less...which makes a lot of sense if you think about it...sorting has to do the O(N log N) work of sorting, regardless of what the sort_by block does.  shuffle, OTOH, doesn't need to do the work of sorting -- I suspect it's linear (I haven't looked at the algorithm, though).
1.8.7 is failing and will continue to do so until rspec/rspec-expectations#758 is merged.

That makes sense regarding C md5 vs ruby jenkins. It's too bad we can't use a C jenkins extension!
I feel as though avoiding the stdlib require may be a good thing if performance is similar. I could go either way, though, because I understand not necessarily wanting to maintain a hash function implementation that isn't really vetted.
I can say that I had this implementation matching the output of a different C implementation, so it's probably mostly ok. The main thing that probably matters is that it's consistent and it has a good distribution, right?
I suppose I lean slightly towards this jenkins implementation. What do you think @samphippen ?

So this is green and it's ready to merge as far as I'm concerned.  @samphippen, do you want to apply your knowledge in this area and review our use of Jenkins before I merge? (Or anyone, really, I just mentioned Sam since I know he has experience with crypto and hash functions that I don't have).

@myronmarston I'll take a look over this PR this evening and let you know what my thoughts are (if any)

@myronmarston this looks great to me.


Support windows logging

Problem:
windows cluster not support logging yet
Solution:
add build windows fluentd image related files

LGTM

Linux is going to upgrade versions, the version will specify in gemfile.lock. Here are the related linux PR https://github.com/rancher/fluentd/pull/14

LGTM

We need to remove all -head stuff

LGTM


Create installer for paymetheus

There are two options for this:

Create an msi that installs all the things and every new release it is re-rolled and the user performs and upgrade or uninstall/reinstall cycle.
Create an msi that downloads some sort of perpetual installer.  So the application will keep itself updated at all times.

2 is obviously what we want but may be more complex.  1. May be the correct short term option until a later release.

Decision has landed on an msi with all the bits and pieces.  Users are responsible for upgrades.

dup of #102


Extend docker-compose & k8s tutorial environmentsto support CTS

Modify the Docker build to support a Text platform to showcase CTS

Proposal

Add new 'platform' to support the CTS driver

Although the CTS could be run in the current environment just as a new server this keeps the cts aspects distinct - alongside use of a seperate cohort (which can be created in a notebook) and hence seperate kafka topics
The notebook itself will be implemented in #1893 whilst #353 will look at automating the CTS as part of the CI/CD pipeline

After a discussion with @mandy-chessell we will continue using the existing dev platform for launching CTS within the coco pharma environment, as this is representative of what they would do as a company.
There is therefore no need for an additional platform, so the notebook can be written with no changes to the infra (compose or k8s)
#353 automation may result in the creation of a more technical/custom environment that does not use coco pharma, but that will be addressed there.


Launching PIPEv4.3.0

Hello everybody!
I have an issue of launching PIPEv4.3.0 needed for my course at master degree for logistics.
Ive extracted the files from the ZIP file, and as soon as I have Win10 OS running on my PC, I know that I have to launch it via "launch.bat" file. Even running the file as admin it just launches in the background, so I cant use the software itself.
P.S. Ive tried one method to launch it via the cmd, but it also didnt work
Please advice,
Thanks in advance

Edit:
Solved it, simply updating java via the official Oracle web-page, guide on the update is here :
https://explainjava.com/install-java-windows/
and deliting a few components in registry from the guide in the following link:
https://www.winhelponline.com/blog/bat-files-do-not-run-when-double-clicked-fix-association/


Expose a LoggingDnsQueryLifeCycleObserverFactory

Motivation:
There is a use case for having logging in the DnsNameResolver, similar to the LoggingHandler.
Previously, one could set traceEnabled on the DnsNameResolverBuilder, but this is not very configurable.
Specifically, the log level cannot be changed.
Modification:
Expose a LoggingDnsQueryLifeCycleObserverFactory, that permit changing the log-level.
Result:
It is now possible to get logging in the DnsNameResolver at a custom log level, without very much effort.
Fixes #10485

@chrisvest Is it possible to have this in 4.1 where traceEnabled is deprecated instead of removed?

@chrisvest I think @violetagg is right and we should do this for 4.1 as well with @deprecation... That said for master this one is fine.

@netty-bot whitelist

@netty-bot add to whitelist

@normanmaurer PR comments addressed, please take a look.

@chrisvest Why only the log level can be specified?
I expected to have the ability as in LoggingHandler.
Let's say you create some application on top of Netty and you want to have a specific logger for that specific functionality.
Typically in applications you have categories for different type of issues and enabling the logging for a particular category you can debug this specific issue.
With the current implementation we still use Netty specific logger.

@violetagg Because Netty does not have any external logging API. The InternalLogger interface is not meant to be accessible outside of Netty itself. LoggingHandler is the same in that regard. Or are you referring to the name of the logger?

Is it possible something like this
    public LoggingDnsQueryLifeCycleObserverFactory(Class<?> clazz, LogLevel level) {
         requireNonNull(clazz, "clazz");
         this.logger = InternalLoggerFactory.getInstance(clazz);
         this.level = requireNonNull(level, "level").toInternalLevel();
     }

@violetagg good call... that said use String name @chrisvest, just as we do in LoggingHandler

@normanmaurer LoggingHandler does both.

lets do both then here as well

Added those constructors as well, and squashed the commits.

@violetagg any concerns ?

LGTM
thanks a lot

@chrisvest feel free to merge once CI completes


Support for Whisper large-v3-turbo model

Hi @MahmoudAshraf97
I tried for whisper-large-v3-turbo model and it gave me this error:
Invalid model size 'large-v3-turbo', expected one of: tiny.en, tiny, base.en, base, small.en, small, medium.en, medium, large-v1, large-v2, large-v3, large, distil-large-v2, distil-medium.en, distil-small.en
I guess it's not supported yet. Can we expect the support for this model soon?

It's working, thanks a lot!


[Trivial] Resolve warnings for boost 1.61 and gcc 6.1

Add boost to system path when compiling with clang.
Remove check for null for a var that can never be null.
@HowardHinnant @miguelportilla

👍 Nice! Boost 1.61 is now ready for prime time.


How to get the data/KoNViD-1kinfo.mat?

@lidq92 Could you share me the script for the data/KoNViD-1kinfo.mat?Thank you very much!!

@ciwei123 Please see VSFA/data/data_info_maker.m, where I have already provided the script.


Refactor Persistence layer in Infrastructure

OK, this ended bigger than expected. Sit down and take a coffee, or two. Yeah two.
[x] Improved naming for async event bus
[x] Renamed Query in favour of ReadModel for better consistency
[x] Update to Doctrine migrations 3
[x] Updated to symfony 5.15 due to security warnings
[x] Moved jwt and doctrine config into packages folder, where it should be living since day one
[x] Upgraded phpunit to 9.3 and refactor configuration file
[x] Improved async messaging and added routing keys capabilities per transport
[x] Added example for async tasks
[x] Added final in many places
[x] Symplified Async event bus
[x] Simplified test suite
[x] Upgraded to Elastic 7.9
[x] Symfony messenger routing functional tests
Fixes #185

Someone knows how to fix this?
  Line   src/Infrastructure/Shared/Persistence/ReadModel/Repository/MysqlRepository.php
 ------ --------------------------------------------------------------------------------
  24     Unable to resolve the template type T in call to method
         Doctrine\Persistence\ObjectManager::getRepository()

Could you send sources about the decided naming strategy?
I don't say it is wrong, just for me and think it is interesting :)

Can't really tell you an article I follow 100% but few good reads:
First one DDD in PHP, it's the closest thing to this IMO, I don't follow some conventions because I try to ve more explicit, and some patterns like the global event dispatcher are things I avoid. Specially since I mostly do teaching others these days. Said that, you don't have to agree with everything just because is Ina book, but still top 3 PHP books I read. A good issue for context here: https://github.com/dddinphp/book-issues/issues/51
The book is a nice read and I really recommend it after the blue one.
For references, this issue in a good summary of how people structure and name https://github.com/webdevilopers/php-ddd/issues/32
I hope you find this helpful

BTW your feedback it's more than welcome if you want to share something ;)

OK, got that phpstan issue sorted by doing a really shitty design but I need to move forward.
I've also found no documentation about how to do functional test for symfony messenger and ended looking at the tests of the library for references. I came with a solution but needs encapsulation.
Another problem I face now is the phpunit suite running very slow, will soft that next.

@Lutacon feedback welcome in case you want to look at that monster

Merging. I promise no more huge Refactors like this


raw payload doesnt work?

[x] This issue is not about OPSEC or bypassing defensive products
[x] I have followed the steps in the Troubleshooting section

OS and version: Ubuntu 23, Win10
Python version: 3.11.4
Nim version: 2.0.2
Using Docker: No

Issue Description
After "NimPlant.py compile all", the exe works, but the .bin doesnt. Tried using shellcode runner https://github.com/hasherezade/pe_to_shellcode/tree/master/runshc compiled as 64bit, and a private one using fibers.

Screenshots
Start shellcode, then exe:

Result: Only exe gets a connection

Debugger doesnt show the actual line:

Hi @dobin, thank you for your report!
Unfortunately, I'm not able to reproduce this issue. On KUbuntu 23.10 (which should not be much different from Ubuntu) with Python 3.11.6 and Nim 2.0.2, I am able to compile the payload without issue:

Subsequently, all of the payloads seem to work:

For testing, I used Nimplant's shinject command to inject into Explorer, as follows:
shinject 1337 "C:\path\to\NimPlant.bin"

Could you try executing with the shinject command to see if that works? If not, please try to provide as much information as you can to reproduce the issue. Thanks!

Starting the exe, and then issuing the following command to inject into a notepad.exe:

Results in no more beacons:

When executing the command, a new notepad editor appeared. It seems the original one (pid 6636) crashed, and a new one was started? (pid 13236). Reproducible.
Trying to load it with my experimental loader: no connection

Re-compiled it with "nim-debug": Same result

Windows version:

Nimplant Config: (probably because of this? as its the only "different" thing)
nimplantconfig.txt

NimPlant.zip
The debug nimplant.bin

Hi @dobin, thanks for the additional debugging steps. I am still not able to reproduce this issue unfortunately. My best guess would be that it relates to the Windows version of the target. Although I did test 22H2 intensively prior, recent testing has been performed on later builds of Windows 11. Do you see any possibility of trying to reproduce your issue on different Windows builds? Potentially that could help us pinpoint where and why this crash is occurring.


Failed to compile grammar containing imports

Hi,
My grammar is now too big to be kept in a single file. I gave a try to the import mechanism but the grammar failed to compile.
So I tested with two simple files and got the same error:
main.pegjs:
import { Digit } from 'lib.js'

Number = Digit+

lib.js:
Digit = [0-9]

$ yarn peggy lib.pegjs
$ yarn peggy main.pegjs
Error parsing grammar
error: Possible infinite loop when parsing (repetition used with an expression that may not consume any input)
 --> main.pegjs:3:10
  |
3 | Number = Digit+
  |          ^^^^^^
$ yarn peggy --version
4.0.2

What am I doing wrong ?

This is a real problem with how we're currently doing imports.  I think I might have marked all imported rules as "might consume nothing", rather than marking each one with the correct information.
I think I know how to fix this, so assigning it to myself.

In the meantime, you should be able to achieve your goals by removing the import statement and compiling with peggy main.pegjs lib.pegjs.

In order to solve this, I think we're going to need to either import or otherwise read in the library grammar when generating the main grammar.  That likely means that we're going to need a new async top-level entrypoint.
This is something that has been contemplated in the past, but is going to require some discussion.  Join us on the Peggy Discord if you want to follow along.


Format warnings better

For the audit command and others, issues found should have better formatting, with the ability to specify the format, such as to a PDF/HTML so that major problems can be more easily high-lighted.

The report command now contains audit findings.


sagemath 8.x does not install on fresh miniconda

Issue: Sagemath does not install on a fresh 'miniconda' distribution because of a package versioning issue
To reproduce:
Install fresh miniconda distribution (python 2.7)
Perform step 1 and 2 of https://doc.sagemath.org/html/en/installation/conda.html#sec-installation-conda (add conda-forge source, update)
Attempt to install the most recent sage by version (conda create -v -n sage sage=8.3[build=py27_3])
Failure:

PackagesNotFoundError: The following packages are not available from current channels:

sage=8.3[build=py27_3] -> sagemath-db-conway-polynomials[version='>=0.5.*']

Specifically, the version number 0.5.p0 for sagemath-db-conway-polynomials does not appear to match the >=0.5.* specifier.
Conda version: conda 4.6.8

This error seems to date back at least to version 8.1 of the sage package for conda.  The version installed by the recommend conda create -v -n sage sage appears to be 7.6!

This may be related to change #7528 of conda, with version 4.6.0:

bump minimum requests version to stable, unbundled release (#7528)

I think that conda interprets version 0.5.p0 of sagemath-db-conway-polynomials as a prerelease version, whereas the 'p' seems to instead stand for 'patchlevel'.

For now, use conda create -v -n sage sage -c conda-forge/label/cf201901. the package here needs to be rebuilt.

Unfortunately, that workaround does not fix the issue:
$ conda create -v -n sage sage=8.3 -c conda-forge/label/cf201901
Collecting package metadata: ...working... done
Solving environment: ...working... failed
[...]
PackagesNotFoundError: The following packages are not available from current channels:

  - sage=8.3 -> sagemath-db-conway-polynomials[version='>=0.5.*']

Leaving out the version seeks to install:
  sage               conda-forge/label/cf201901/linux-64::sage-7.6-py27_2

However, the install does work as expected after downgrading conda to version 4.5.12.

Thanks. Can you send a PR to sagemath-db-conway-polynomials-feedstock to change the version from 0.5.p0 to 0.5.0 ?


Bug: broken preview border when leftcol > 0

RTFM Checklist

[X] I have searched exisiting issues / discussions
[ ] I have read the Wiki including the Advanced section
[X] I have read man fzf / I am well versed in shell fzf

Operating system
Linux
Shell
zsh
Neovim version (nvim --version)
NVIM v0.11.0-dev-226+g7215512100
Fzf version (fzf --version)
0.54.0 (9e92b6f1)
Output of :lua print(os.getenv('FZF_DEFAULT_OPTS'))
--color="16,hl:-1:reverse,hl+:-1:reverse,fg+:15,bg+:0" --bind=ctrl-p:half-page-up --bind=ctrl-n:half-page-down --bind=ctrl-y:preview-up --bind=ctrl-e:preview-down --bind=ctrl-u:preview-ha lf-page-up --bind=ctrl-d:preview-half-page-down --bind=shift-tab:ignore --bind=tab:toggle
Is the problem reproducible with mini.sh?

[ ] My issue is reproducible with mini.sh
[ ] My issue IS NOT reproducible with mini.sh
[X] I have not tested with mini.sh (not relevant, requires LSP, Windows, etc)

Fzf-lua configuration
require('fzf-lua').setup({
  winopts = {
    preview = {
      horizontal = "left:60%",
    },
  },
  fzf_opts = {
    ["--no-hscroll"] = true,
  },
  keymap = {
    builtin = {
      -- ["<F2>"] = "toggle-fullscreen",
      ["<F5>"] = "toggle-preview-ccw",
      ["<F6>"] = "toggle-preview-cw",
      ["<C-y>"] = "preview-up",
      ["<C-e>"] = "preview-down",
      ["<C-u>"] = "preview-page-up",
      ["<C-d>"] = "preview-page-down",
    },
    fzf = {
      ["ctrl-p"] = "half-page-up",
      ["ctrl-n"] = "half-page-down",
      ["shift-tab"] = "ignore",
      ["tab"] = "toggle",
    },
  },
})

Describe the bug / steps to reproduce
When there is 'wrap' option enabled (or if it's not but first screen column is actually the first one row column, winsaveview().leftcol == 0) then everything is fine. But otherwise ('wrap' disabled and the buffer view is shifted to the right, e.g. with 'zl' keymap) then preview border is also shifting no matter what position (left/right/top/bottom) it has. Some images to demonstrate it.
Test buffer with this silly content:

Wrap is enabled, everything if fine:

Wrap is disabled, but winsaveview().leftcol == 0 (also fine):

Wrap is disabled, line content is longer than the window width, some horizontal split was applied. That's where strange things happen:

Bottom preview is also have this "shifty" behavior and the more shift view has the more shift preview border will have :)

The screens were made on default configuration but my config (with left-side preview) also shows previews with this strange shift.

Ty @librolibro, I will check why this happens, I’m a bit confused with your terminology though as when you write “wrap is enabled” the screenshot behavior displays a line with nowrap (one long line).
To clear, all I need to recreate this bug is a very long line (longer than current columns allow) and :set nowrap?

Yeah, my fault, that was a mistyping :)
Yes, to reproduce this bug you need a long line, 'nowrap' - and then you need to shift the buffer view (zh/zl or just jump forward on the line)

I'll play with my config settings later - probably one of them causes this problem.

I'll play with my config settings later - probably one of them causes this problem.

Alright, lmk what you find out, once reproduced this is most likely an easy fix.

Tysm @librolibro! This is indeed reproduced only with :setlocal:

Might be an upstream issue, I’ll do some digging.

https://github.com/ibhagwan/fzf-lua/commit/5565f4bfe304df30c35962a982f14d8de1043336
@librolibro I have no idea why this fixed it, I had an old workaround that set the border wrap to false locally, for some reason it caused this behavior only when the originating window local wrap was also set, most likely an upstream issue but doesn't matter much as I'm going to remove the use of a separate border window anyways soon.

Yeah, this commit fixed the problem, thanks a lot!


Remove obsolete URL links in the testcase README files

Formerly, input dataset files were provided for the internal unit testcases. However, these testcases are only now being provided as LIS config examples for users to reference in their own setup.
Removal of this obsolete information from the README files should be made.

Have pushed a couple of more commits for this issue and pull request that address the obsolete testcase files, which were updated and removal of former "wget" scripts.
Also, removed the older "README_Discover" file in each software component.

These changes look great.  About to approve the pull request.  Thanks.


click header app dropdown after it becomes visible

Signed-off-by: Will Kutler<EMAIL_ADDRESS>https://github.com/open-cluster-management/backlog/issues/16647

/retest


Why 'tf.python_io.TFRecordWriter' is so SLOW and STORAGE-CONSUMING in TensorFlow?

I'm going to write to TFRecord file using this code The problem is that this process is very slow, such that it's not feasible to write a large dataset even in days! It's just a writer that serialize to disk. Why it's so slow?! Another problem is that the size of the output file is 10 times greater than the original file!
does anyone know any way to speed up the process of TFRecordWriter and compress the result?

Regarding compression, see https://github.com/tensorflow/tensorflow/blob/4addf4b5806cd731949c6582a83f5824599cd1ef/tensorflow/python/lib/io/tf_record.py#L26

@fwalch   thanks,  but the most important thing is the speed.  I have to split the file to 100000 slice and using multiprocessing to process each.
is there any optimize method for speed up ?

is there a c++ demo for tfrecord? maybe faster?

I'm also having trouble reading from TFRecord. I only get 25MB/s instead of the 250MB/s I could get out of the SSD.

@ericyue I'm having some succes with FixedLengthRecordReader. It reads at 150-160MB/s from SSD

I'm having a similar problem; TFRecordWriter doesn't appear to be the issue, rather it's the protobuf library. BytesList() is slow, and SerializeToString() is mega-slow. Writing 1 MiB:
%%prun
def quantize(x):
    return x.astype(np.int8).tostring()
feats = {'':tf.train.Feature(bytes_list=tf.train.BytesList(value=quantize(np.ones((2**10, 2**10)))))}
example = tf.train.Example(features=tf.train.Features(feature=feats))
writer = tf.python_io.TFRecordWriter('/dev/null')
writer.write(example.SerializeToString())

         15729132 function calls (15729090 primitive calls) in 6.355 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        2    1.679    0.839    2.488    1.244 encoder.py:260(RepeatedFieldSize)
        1    1.292    1.292    2.881    2.881 encoder.py:711(EncodeRepeatedField)
  1048580    0.701    0.000    1.013    0.000 encoder.py:372(EncodeVarint)
  3145731    0.535    0.000    0.535    0.000 containers.py:202(__getitem__)
  3145748    0.440    0.000    0.440    0.000 {method 'write' of '_io.BytesIO' objects}
        1    0.366    0.366    0.841    0.841 containers.py:261(extend)
  1048576    0.359    0.000    0.471    0.000 type_checkers.py:100(CheckValue)
3145753/3145751    0.289    0.000    0.289    0.000 {len}
  2097157    0.283    0.000    0.283    0.000 encoder.py:82(_VarintSize)
  1048592    0.150    0.000    0.150    0.000 {chr}
  1048621    0.113    0.000    0.113    0.000 {isinstance}
        1    0.109    0.109    6.355    6.355 <string>:2(<module>)
        6    0.021    0.003    0.021    0.003 {method 'extend' of 'list' objects}
     16/6    0.003    0.000    0.862    0.144 python_message.py:474(init)
      9/4    0.003    0.000    2.495    0.624 python_message.py:1035(ByteSize)
      5/1    0.003    0.001    5.383    5.383 python_message.py:1077(InternalSerialize)
        1    0.003    0.003    0.003    0.003 {_pywrap_tensorflow.PyRecordWriter_WriteRecord}
        1    0.003    0.003    0.003    0.003 {method 'astype' of 'numpy.ndarray' objects}
        1    0.002    0.002    0.002    0.002 {numpy.core.multiarray.copyto}
        1    0.001    0.001    0.001    0.001 {method 'getvalue' of '_io.BytesIO' objects}
        1    0.000    0.000    0.000    0.000 {method 'tostring' of 'numpy.ndarray' objects}
     10/5    0.000    0.000    0.018    0.004 python_message.py:1233(MergeFrom)
        1    0.000    0.000    0.000    0.000 {_pywrap_tensorflow.PyRecordWriter_New}
       21    0.000    0.000    0.000    0.000 python_message.py:1362(__init__)
       12    0.000    0.000    0.000    0.000 python_message.py:790(ListFields)
        8    0.000    0.000    0.000    0.000 python_message.py:429(MakeSubMessageDefault)
       12    0.000    0.000    0.000    0.000 python_message.py:775(_IsPresent)
       12    0.000    0.000    0.000    0.000 {method 'sort' of 'list' objects}
        5    0.000    0.000    0.017    0.003 containers.py:280(MergeFrom)
        5    0.000    0.000    0.000    0.000 containers.py:541(__getitem__)
       18    0.000    0.000    0.000    0.000 python_message.py:1381(Modified)
      4/1    0.000    0.000    5.382    5.382 encoder.py:760(EncodeField)
       40    0.000    0.000    0.000    0.000 {method 'items' of 'dict' objects}
    21/12    0.000    0.000    0.000    0.000 python_message.py:1317(Modified)
        5    0.000    0.000    0.000    0.000 python_message.py:1397(__init__)
      2/1    0.000    0.000    0.000    0.000 python_message.py:1141(IsInitialized)
        6    0.000    0.000    0.000    0.000 python_message.py:421(MakeRepeatedScalarDefault)
        1    0.000    0.000    4.130    4.130 encoder.py:818(EncodeField)
        4    0.000    0.000    0.000    0.000 type_checkers.py:162(CheckValue)
        1    0.000    0.000    5.384    5.384 python_message.py:1071(SerializePartialToString)
        1    0.000    0.000    0.003    0.003 <string>:2(quantize)
        1    0.000    0.000    1.246    1.246 encoder.py:351(FieldSize)
        1    0.000    0.000    0.002    0.002 numeric.py:136(ones)
        5    0.000    0.000    0.000    0.000 python_message.py:1406(Modified)
        2    0.000    0.000    0.000    0.000 python_message.py:660(field_setter)
        6    0.000    0.000    0.000    0.000 containers.py:237(__init__)
       21    0.000    0.000    0.000    0.000 {_weakref.proxy}
      5/2    0.000    0.000    2.488    1.244 encoder.py:307(FieldSize)
        1    0.000    0.000    0.004    0.004 containers.py:595(MergeFrom)
        1    0.000    0.000    0.000    0.000 tf_record.py:57(__init__)
        8    0.000    0.000    0.000    0.000 python_message.py:537(_GetFieldByName)
        5    0.000    0.000    0.000    0.000 python_message.py:1332(_UpdateOneofState)
        6    0.000    0.000    0.000    0.000 containers.py:192(__init__)
       10    0.000    0.000    0.000    0.000 python_message.py:1004(SetListener)
        2    0.000    0.000    0.000    0.000 {setattr}
        1    0.000    0.000    0.000    0.000 python_message.py:271(_IsMapField)
        5    0.000    0.000    0.000    0.000 containers.py:206(__len__)
        1    0.000    0.000    0.000    0.000 {numpy.core.multiarray.empty}
        1    0.000    0.000    0.003    0.003 tf_record.py:78(write)
        2    0.000    0.000    0.000    0.000 python_message.py:379(MakeMessageMapDefault)
        1    0.000    0.000    0.000    0.000 {_codecs.utf_8_decode}
        1    0.000    0.000    5.384    5.384 python_message.py:1057(SerializeToString)
        2    0.000    0.000    0.000    0.000 python_message.py:651(getter)
        1    0.000    0.000    0.003    0.003 pywrap_tensorflow.py:183(WriteRecord)
        2    0.000    0.000    0.000    0.000 containers.py:525(__init__)
        2    0.000    0.000    0.000    0.000 descriptor.py:118(GetOptions)
        1    0.000    0.000    0.000    0.000 {method 'decode' of 'str' objects}
        2    0.000    0.000    0.000    0.000 containers.py:586(__len__)
        5    0.000    0.000    0.000    0.000 {method 'setdefault' of 'dict' objects}
       12    0.000    0.000    0.000    0.000 {method 'get' of 'dict' objects}
       12    0.000    0.000    0.000    0.000 python_message.py:792(<lambda>)
        1    0.000    0.000    0.000    0.000 compat.py:27(as_bytes)
        1    0.000    0.000    0.000    0.000 utf_8.py:15(decode)
        3    0.000    0.000    0.000    0.000 containers.py:589(__iter__)
        4    0.000    0.000    0.000    0.000 {iter}
        1    0.000    0.000    0.000    0.000 python_message.py:277(_IsMessageMapField)
        1    0.000    0.000    0.000    0.000 pywrap_tensorflow.py:182(<lambda>)
        2    0.000    0.000    0.000    0.000 {method 'pop' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

def quantize(x):
    return x.astype(np.int8).tostring()
writer = tf.python_io.TFRecordWriter('/dev/null')
%time q = quantize(np.ones((2**10, 2**10))) #fast
%time bl = tf.train.BytesList(value=q) #slow
%time example = tf.train.Example(features=tf.train.Features(feature={'':tf.train.Feature(bytes_list=bl)})) #fast
%time s = example.SerializeToString() #very slow
%time writer.write(s) #fast

CPU times: user 8 ms, sys: 0 ns, total: 8 ms
Wall time: 5.11 ms
CPU times: user 440 ms, sys: 4 ms, total: 444 ms
Wall time: 440 ms
CPU times: user 20 ms, sys: 0 ns, total: 20 ms
Wall time: 20.4 ms
CPU times: user 2.21 s, sys: 16 ms, total: 2.23 s
Wall time: 2.22 s
CPU times: user 4 ms, sys: 0 ns, total: 4 ms
Wall time: 2.81 ms

i'm using the docker image tensorflow/tensorflow:latest-gpu with nvidia-docker on ubuntu 16.04

Try installing a protobuf-python wheel that we have built that uses optimized C++ code internally: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/g3doc/get_started/os_setup.md#protobuf-library-related-issues

@vrv thanks! I missed that.
the pip command succeeded, except for some SSL warnings, but now:
root@58fb67668f0f:/notebooks# cat test.py
import tensorflow as tf
root@58fb67668f0f:/notebooks# gdb python
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
...
Reading symbols from python...Reading symbols from /usr/lib/debug//usr/bin/python2.7...done.
done.
(gdb) run test.py
Starting program: /usr/bin/python test.py
warning: Error disabling address space randomization: Operation not permitted
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:99] Couldn't open CUDA library libcudnn.so. LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64:
I tensorflow/stream_executor/cuda/cuda_dnn.cc:1562] Unable to load cuDNN DSO
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcurand.so locally
During startup program terminated with signal SIGSEGV, Segmentation fault.
(gdb) bt
No stack.
(gdb)

looks like I have a cuDNN problem too, but that was present already and I assume it is unrelated. If I pip uninstall protobuf, the cuda messages still print before protobuf throws an importerror.

@victor-shepardson: We're not sure why but the binary protobuf wheel we build doesn't work for everyone and results in segfaults for some, but it does work for others.
You might need to figure out how to build the optimized python protobuf package on your own.  Hopefully the github.com/google/protobuf folks can help.

I'm going to close this, since it looks like the problem is related to not using the optimized protobuf/python package, and isn't strictly related to TF.  Thanks for the helpful performance debugging @victor-shepardson !

thanks , I will try the c++ version of protobuf

can you show me a demo code for using FixedLengthRecordReader to read tfrecord files? I think the tfrecord are not fixed length , how to do this ? @titusnicolae

According to google/protobuf#1332 the wheel on pypi for protobuf 3.2.0 now includes the C++ implementation and uses it by default (although maybe not yet on OS X). I was able to verify this on linux x86_64:
$ pip show protobuf
Name: protobuf
Version: 3.2.0
Summary: Protocol Buffers
Home-page: https://developers.google.com/protocol-buffers/
Author<EMAIL_ADDRESS>Author-email<EMAIL_ADDRESS>License: New BSD License
Location: /home/ed/.pyenv/versions/3.6.1/envs/py36/lib/python3.6/site-packages
Requires: six, setuptools
$ python -c "from google.protobuf.internal import api_implementation; print(api_implementation._default_implementation_type)"
cpp

On Windows it seems like the protobuf is still the old version. I'm getting extremely slow speeds writing TFRecords even with multiprocessing, at this point I'd be better off using something like Keras sequence.

TFRecordWriter was incredibly slow for me. The only thing that worked for me was converting the output tensors I wanted to save to numpy arrays, and then saving using pyarrow. Here is some code if it helps anybody:
import pyarrow as pa
import pyarrow.parquet as pq
import shutil, os
import numpy as np

# For faster writing of tensorflow output
class ParquetWriter():
  # context manager stuff
  """ dir_path: directory to put parquet files
    cols: column names for schema
    chunk_size: how many rows per file = chunk_size*buffer_size
    buffer_size: how many rows to buffer before writing (not exact, if using add_rows)
    overwrite: delete existing directory/file if found
  """
  def __init__(self, dir_path, cols, chunk_size=10, buffer_size=5000, overwrite=True):
    self.chunk_size = chunk_size
    self.buffer_size = buffer_size
    self.dir_path = dir_path
    self.cols = cols
    self.chunk = 0
    self.file = 0
    self.writer = None
    self.new_buffer()
  def __enter__(self):
      return self
  def __exit__(self, exc_type, exc_value, exc_traceback):
    if self.writer:
      self.writer.close()
  # clears buffer (does not write)
  def new_buffer(self):
    self.buffer = []
    for i in range(len(self.cols)):
      self.buffer.append([])
    self.buffered = 0
  # actually perform write to parquet file
  def flush(self):
    print("parquet write chunk {}, rows {}".format(self.chunk, self.buffered))
    # convert to chunked pyarrow arrays, instead of list of numpy arrays
    for i in range(len(self.buffer)):
      bi = self.buffer[i]
      # apparently pyarrow can't do > 1d arrays; but doesn't complain if its a list
      if len(bi[0].shape) > 1:
        self.buffer[i] = np.vstack(bi).tolist()
      else:
        self.buffer[i] = pa.chunked_array(bi)
    tbl = pa.Table.from_arrays(self.buffer, names=self.cols)
    # create new file
    if not self.chunk:
      if os.path.exists(self.dir_path):
        if os.path.isfile(self.dir_path):
          os.remove(self.dir_path)
        else:
          shutil.rmtree(self.dir_path)
      os.makedirs(self.dir_path)
      self.writer = pq.ParquetWriter(os.path.join(self.dir_path, "chunk_{}.part".format(self.file)), tbl.schema)
    self.writer.write_table(tbl)
    self.chunk += 1
    self.new_buffer()
    # finished file
    if self.chunk == self.chunk_size:
      print("finished file {}".format(self.file))
      self.chunk = 0
      self.file += 1
      self.writer.close()
      self.writer = None
  # buffer/write multiple rows
  def add_rows(self, rcols):
    assert len(rcols) == len(self.cols), "Doesn't match column schema length"
    self.buffered += len(rcols[0])
    for i in range(len(self.buffer)):
      self.buffer[i].append(rcols[i])
    if self.buffered >= self.buffer_size:
      self.flush()

Use it like so:
with ParquetWriter("some_directory_path", ["col1","col2"]) as f:
  # perform inference here
  f.add_rows(out_col1.numpy(), out_col2.numpy())

I was having the same problem writing a large dataset -- the writing speed would get slower the longer the process ran (starting at 1iteration/sec and going up to 11sec/iteration).
The way I solved it was to spin up sub-processes in a loop to write smaller blocks of the data. Ideally the blocks would be small enough so that the writing would remain at 1iteration/second.
For example my slow code originally was something like:
pool = multiprocessing.Pool(n_processes)
process_blocked_data = split(data, n_processes)
for block in process_blocked_data:
    pool.start(write_tfrecord, args=(block))
pool.join()

And I fixed it by splitting up the data even more:
# split the data into even smaller blocks
factor = 100
blocked_data = split(data, factor)

for block in blocked_data:
   # assign smaller block to each process
   pool = multiprocessing.Pool(n_processes)
   process_blocks = split(block, n_processes)
   for smaller_pblock in process_blocks:
       pool.start(write_tfrecord, args=(smaller_pblock))
   pool.join()

Doing it this way reduced the total write time from 11+hours to 1hour.
My best guess is that the writing slows down the longer the process is run because something is written/added to the graph at each write command which ends up leading to a very large graph which slows write speeds down significantly.


The sound is not transmitting

Hi, Stephen. Thanks for this sdk.
On Chrome, Android to Android devices. The pictures send, but the sound does not transmit through the recipients...

Sometimes you need to use a 3rd party stun/turn server https://github.com/stephenlb/webrtc-sdk#webrtc-adding-custom-stun-and-turn-servers as the network traversal in your case may need a broker.


added jupyter file for ClothingCLI example

** This pr is with respect to issue #96**  It includes Documentation of python scripts into jupyter
notebook.

What is the current behavior? (You can also link to an open issue here)
https://github.com/marqo-ai/marqo/issues/96

Above is an example. Please assign this issue to me. I've already ready with the changes, for other examples  just have to push the pull request.

Thanks @vandit98! Looks good so far, would you be able to show what the output is when you run the notebook main() function is?

yes i will add it

hey
I have added the jupyter file for all examples.

Thanks @vandit98 ! Would you be able to test the notebooks? for example, I am not sure this code should be added to a notebook https://github.com/marqo-ai/marqo/blob/e78ec142dbfc0983a824301f527d59a27a472e2c/examples/ClothingStreamlit/stream_lit_marqo_demo.ipynb since it is an application. I suggest re-writing it so the application 'runs' inside the notebook instead of streamlit. alternatively, exclude this example from the PR

hey @jn2clark
I have excluded the example from the pull request.


【feature request for agenda 3】Adding an button to goto the task block

Calendar view is a great interface to interact with tasks
Sometimes one may want to goto the page of certain task or the block of the certain task
It's great if the plugin provide a button to go to the page of the task and scroll to the block to provide easy way for further editing.

Indeed it is better if we can jump to a block in the calendar view.
There is an icon for jumping to blocks, but only available from the TaskCard modal (see below)

Under the Tasks view we have the same icon on hovering:

In #297, I provided one solution by using shortcut ctrl+click to do this. Try it here if you're interested

Thank you for replying to my request, and providing solutions, the ctrl + click is very intuitive. I just wondered if you could make this feature available in the master version of the plugin.

Thank you for replying to my request, and providing solutions, the ctrl + click is very intuitive. I just wondered if you could make this feature available in the master version of the plugin.

Hi, the shortcut system in that PR might take sometime to be thoroughly tested while the maintainer is a bit busy recently. We will consider merging the shortcut feature (or not) once the maintainer has extra time to work on it. Cheers!

Thank you for replying to my request, and providing solutions, the ctrl + click is very intuitive. I just wondered if you could make this feature available in the master version of the plugin.

Hi the shortcut aforementioned has been available in the latest release. Hope you find it helpful !


seems like new style diff code broke Cell:set_style

If I have rendered some content with say Bold and Blue. And then render more output to that same buffer and overwrite a Blue Bold cell with a a Style::default string, then rather than resetting the style of that cell to default (ie 'plain') it does nothing
    pub fn set_style(&mut self, style: Style) -> &mut Cell {
        if let Some(c) = style.fg {
            self.fg = c;
        }
        if let Some(c) = style.bg {
            self.bg = c;
        }
        self.modifier.insert(style.add_modifier);
        self.modifier.remove(style.sub_modifier);
        self
    }

because the style is basically empty. In prior design (with modifier() fn rather than add / sub modifier) this would have reset the Cell to 'plain'.
Call chain is buffer::set_string, buffer::set_stringn, Cell::set_style.
In order to make this work now I have to create a Cell with a style that subtracts the old values , this means I have to keep track of the old style.

OK I worked out how to get around this but still fells like a serious break
changed
        buf.set_string(
            area.width - small_files_len - 1,
            status_line_y,
            small_files_legend,
            Style::default()
        );

to
        buf.set_string(
            area.width - small_files_len - 1,
            status_line_y,
            small_files_legend,
            Style::default().fg(Color::Reset).bg(Color::Reset).remove_modifier(Modifier::all())
        );

This code is overwriting part of another string that was loaded into the buffer earlier

That's sounds like a bug in the diffing logic when we write to the terminal. Could you create a proper bug report following the template and provide the code of a minimal example triggering this behavior ?

this code shows it
use std::io;
//use termion::raw::IntoRawMode;
use tui::Terminal;
use tui::backend::CrosstermBackend;
use tui::widgets::{Widget, Block, Borders};
use tui::layout::{Layout, Constraint, Direction};
use tui::style::{Color,Style};

fn main() -> Result<(), io::Error> {
    let stdout = io::stdout();//.into_raw_mode()?;
    let backend = CrosstermBackend::new(stdout);
    let mut terminal = Terminal::new(backend)?;
    terminal.clear();
    let buff = terminal.current_buffer_mut();
    buff.set_string(25, 25, "red", Style::default().fg(Color::Red));
// second string that over laps with the first and tries to draw 'plain' un styled text
    buff.set_string(27, 25, "plain", Style::default());
    terminal.draw(|mut f| {
        let size = f.size();
        let block = Block::default()
            .title("Block")
            .borders(Borders::ALL);
        f.render_widget(block, size);

    })?;
    crossterm::event::read();
    Ok(())
}

run with 0.09 the output is

the 'plain' text beats the 'red' where they overlap
using .10

the red wins where they overlap.
This is because Cell::set_style actually means merge_style now, Style::default is empty so the merge does nothing

Ok I misunderstood your first report then. If you are rewriting to a cell in the same draw call then what you are experiencing is the expected behavior. Previously if the style of a cell was successively set with styles S1, S2, S3, the resulting style of the cell was S3 (i.e the last one called). A direct implication of this behavior was that every time you had some sort of nested element and you wanted it to inherit the style of its parent you had to provide the style of the parent element again. This made the declaration of some Widget unnecessarily verbose and not very user-friendly. With the new system, the resulting style is the left to right merge of (S1, S2, S3).
One quality of life improvement in your case could be to provide a Style::reset() constructor that gives you the Style object you are currently building by hand. WDYT ?

I totally got what you changed. In fact in diskonaut (where the bug arose) it was a bug in its code that caused the overlap. That is now fixed. So I dont need a fix.
Yes somehow you need to differentiate between Style= none and Style = dontcare. So having reset() meaning none and default meaning 'dont care' is good. Doc it tho.
Although I wonder about the other way round. In 9 default = None, maybe thats how it should stay and have ::new = dont care. Tough call. Either way people will be unhappy. Depends on how far people have got with migrating 9->10. I know other project still stuck on 9 (gitui for example).
The main thing  tho is to doc it


proxies in session not taken into account

When creating a session object, and setting proxies information, the proxy information is not used in subsequent requests:
SESS = requests.Session()
SESS.auth = (USER, PWD)
SESS.verify = False
SESS.proxies = {'https': None}
Expected Result
SESS.get(URL + '/' + ENDPOINT)
should bypass the proxy and be able to resolve the internal URL.
Actual Result
502: notresolvable for url
But passing the same proxy information to the get works:
SESS.get(URL + '/' + ENDPOINT, proxies={'https': None})
works as expected
Reproduction Steps
import requests
SESS = requests.Session()
SESS.auth = (USER, PWD)
SESS.verify = False
SESS.proxies = {'https': None}

System Information
requests 2.13.0
python 3.5.2
Fedora 20

Hey again @navidof, I'm unable to reproduce this with 2.18.4 and since this hasn't seen an update in the last month, I'm going to close it out.

Hi Nate,
Thanks for your time, I didn't know how to test with another request package, anyway it's not possible for my use case to use another version.
The workaround is simple, so no worries if it's not fixed in my version.
Regards,
Navid
----- Mail original -----
De: "Nate Prewitt"<EMAIL_ADDRESS>À: "requests/requests"<EMAIL_ADDRESS>Cc: "navidof"<EMAIL_ADDRESS>"Mention"<EMAIL_ADDRESS>Envoyé: Dimanche 20 Mai 2018 21:17:00
Objet: Re: [requests/requests] proxies in session not taken into account (#4599)
Hey again @navidof , I'm unable to reproduce this with 2.18.4 and since this hasn't seen an update in the last month, I'm going to close it out.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub , or mute the thread .


Updated index.html

Thanks! We'll take the charset update, appreciate it. Not including the description part, since this is a redirect page.


Deserialization not working on character select screen in MMO mode

Although it is working in single-player, deserialization doesn't seem to be working on the character select screen in MMO. None of the customization options I select are appearing on my character in the character selection screen. It does appear to be working in game, however.
I tested this on a fresh install of MMORPG kit 1.62e and UMA addon version 1.16g with a newly created MySQL database.

Sorry for slowly responding, I just see this today, I've fixed it in 1.16h


Weird usecase where pydantic model has field that represents code but gets invalid json characters, failing model_validate_json

Is your feature request related to a problem? Please describe.
I have a weirdish use case, where one of the fields of the pydantic model represents code.
The code is often returned with a bunch of invalid json characters in it, like control characters (\u0000-\u001F).
This makes instructor fail on errors like this:
  File "/opt/homebrew/lib/python3.11/site-packages/pydantic/main.py", line 530, in model_validate_json     return cls.__pydantic_validator__.validate_json(json_data, strict=strict, context=context)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pydantic_core._pydantic_core.ValidationError: 1 validation error for RustCode   Invalid JSON: control character (\u0000-\u001F) found while parsing a string at line 4 column 0 [type=json_invalid, input_value='\n{\n"generated_code": "...xample_output": "37"\n}', input_type=str]     For further information visit https://errors.pydantic.dev/2.4/v/json_invalid joelkronander@MacBook-Pro-5 swissknife %
Describe the solution you'd like
Maybe one could handle cases like this with some form of "pre-validators" that could for example run byte64 encoding on those non-json compatible strings? Not sure how it would fit in exactly.
Additional context
Instructor is nice.

interesting this might be more of a pydantic issue than instructor.  maybe it should come from a different strict mode?
I'll see if i can pass the modes across json and pydantic

@JoelKronander Let see if this work, can you verify that control characters work with strict mode is turned off?

@jxnl I was not able to get this working without removing the newlines manually first.

this is super dumb but sometimes you gotta remind it to escape new lines in the prompt, acuse the control characters are an issue unless you use the latest models.

Indeed. My best fix was the following prompt with gpt-4-turbo-preview YMMV.


Build failure: python3Packages.sqlalchemy-migrate

Steps To Reproduce
Steps to reproduce the behavior:

build python with sqlalchemy-migrate

Build log
[user@system:~]$ nix shell nixpkgs#python3Packages.sqlalchemy-migrate --impure
error: builder for '/nix/store/840597qwjbdlm343631l2icbfhpkq6r9-python3.11-sqlalchemy-migrate-0.13.0.drv' failed with exit code 1;
       last 10 log lines:
       >     from migrate.changeset.schema import *
       >   File "/build/sqlalchemy-migrate-0.13.0/migrate/changeset/schema.py", line 20, in <module>
       >     from migrate.changeset.databases.visitor import (get_engine_visitor,
       >   File "/build/sqlalchemy-migrate-0.13.0/migrate/changeset/databases/visitor.py", line 7, in <module>
       >     from migrate.changeset.databases import (sqlite,
       >   File "/build/sqlalchemy-migrate-0.13.0/migrate/changeset/databases/sqlite.py", line 13, in <module>
       >     from sqlalchemy.databases import sqlite as sa_base
       > ModuleNotFoundError: No module named 'sqlalchemy.databases'
       > Non-zero exit code (2) from test listing.
       > error: testr failed (3)
       For full logs, run 'nix log /nix/store/840597qwjbdlm343631l2icbfhpkq6r9-python3.11-sqlalchemy-migrate-0.13.0.drv'.

Additional context
The sqlalchemy-migrate library is built using a legacy version of sqlalchemy, maintained as sqlalchemy_1_4. The build input needs to be changed to this, at a minimum, for the library to build.
Notify maintainers

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
[user@system:~]$ nix shell nixpkgs#nix-info --command nix-info -m
 - system: `"x86_64-linux"`
 - host os: `Linux 6.7.6, NixOS, 24.05 (Uakari), 24.05.20240314.d691274`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.1`
 - channels(root): `"nixos-23.05, nixpkgs-unstable"`
 - nixpkgs: `/nix/store/9mf8fkjw3vw1gm4s6j5fqlwrv5wf2knf-source`

Add a :+1: reaction to issues you find important.

Notify maintainers
@SuperSandro2000
@makefu

How did you construct the overlay that downgrades sqlalchemy? Did you properly overwrite src?

How did you construct the overlay that downgrades sqlalchemy? Did you properly overwrite src?

I can't remember exactly, I was able to get it to build using .override, but I think I was having trouble getting any overlay to actually apply to my nix flake. I'm in exam season at Uni at the moment, so it's probably not something I'll be able to take a decent look at again until summer. There might be something in my NixOS-Config repo to give you an idea what I was doing. No guarantees though, it's a bit of a mess, and cleaning it up is also a summer job at this point.

Try using pythonPackagesExtensions https://github.com/NixOS/nixpkgs/blob/3eaeaeb6b1e08a016380c279f8846e0bd8808916/nixos/modules/config/no-x-libs.nix#L74-L79 and that is an example of an overlay https://github.com/NixOS/nixpkgs/blob/3eaeaeb6b1e08a016380c279f8846e0bd8808916/pkgs/applications/office/paperless-ngx/default.nix#L40-L53


Fix permission denied when reading from interfaces on Android

This is related to https://github.com/Dev-Environments/VScode/issues/63; for my Android 12 (Samsung Tab S6) tablet it is not possible to run tailscaled from the ubuntu-based userspace provided by VScode for Android. It does however work on Termux (download from f-droid). The issue seems to be related to how proc/net is queried to get interface information.
$ ./tailscale_1.74.1_arm64/tailscaled --tun=userspace-networking
2024/09/28 09:57:09 netmon.New: route ip+net: netlinkrib: permission denied

Might be possible to use: https://github.com/wlynxg/anet

https://github.com/golang/go/issues/40569#issuecomment-1050381441

This fails: https://github.com/spotsnel/tailscaled-android/blob/c62b0732d23ec5f19c2006c14497dcaa86b92dd8/net/netmon/state.go#L454

https://github.com/tailscale/tailscale/pull/8946

It need android ndk, you can see:

https://github.com/Asutorufa/tailscaled-socks5-android/blob/main/app/src/main/jniLibs/build.sh
https://github.com/Asutorufa/tailscaled-socks5-android/blob/4000ea33088eaedfb8320b9087d59e46d51c792f/.github/workflows/android.yml#L21C1-L42C1
https://github.com/Asutorufa/tailscaled-socks5-android/tree/main
https://github.com/Asutorufa/tailscale

You can also get it from my android apk, just decompress it, github actions https://github.com/Asutorufa/tailscaled-socks5-android/actions/runs/10776956015

Thanks! Seems we have similar 'interests'. I'll have a look as for some reason my NDK did not function as expected; android/log.h missing... but in disk under sysroot. What base platform did you use?

My pc linux system is archlinux, it also no problem.

@Asutorufa compiling your https://github.com/Asutorufa/tailscale/commit/787d121bfab06133dcc0452098405dc2fdfb9198 worked as expected with make buildlinuxarm for use with VScode/deVStudio and Userland. :+1


Support celluloid as player

https://celluloid-player.github.io/
Based on mpv.

As a general rule we don't add support for media players that we don't use, because it is only from using Syncplay with that media player that issues can be identified and fixed and if we supported everything then it would create an increasing burden on Syncplay development.
As such, I'd be happy to provide to support to anyone wishing to add support for a new media player, but they would need to use that media player on a regular basis and be willing to maintain that support in the long term.
Depending on how they are coded some mpv-players can be easily supported, but others would need to be modified to be significantly modified to support Syncplay.

I'm closing this issue as nobody has volunteered to implement the requested feature. If someone does decide to work on this idea then feel free to ask for this issue to be re-opened.

I've been watching movies on a weekly base for over a year with a bunch of friends (who would also use celluloid). I'm not the most proficient developer, but I'm competent enough with Python, and given how Celluloid is just a frontend to MPV (which i think accepts the same flags from the terminal), I volunteer to support. With me and my friends using it weekly, I'd iron out any bugs presented by newer versions of the player.

Thanks for volunteering to help out @CaptainJawZ - if you want to make a PR for Celluloid support then that would be welcome.
A starting point for how to do it would be to look at mpv.net at https://github.com/Syncplay/syncplay/blob/master/syncplay/players/mpvnet.py
As you can see from https://github.com/Syncplay/syncplay/commit/187c8d87fc369faa9b0a71715f771b94ea9810fc we also add player paths and an icon path to syncplay/constants.py and import the player in syncplay/players/__init__.py
Some mpv frontends allow for full pass-through of command line switches and messages to mpv as if it were mpv and some do not. As such, just because something is a frontend to mpv doesn't always mean it will simply work with Syncplay. Sometimes what we've had to do is work with developers on the player-side of the equation so it support Syncplay.

Hello! so I've got myself a bit stuck, I hope it is ok to ask for help in here.
So I modified constants.py and init.py to add the variables that would be required for celluloid.
Now, celluloid can parse any MPV flag so long as it begins with --mpv-
so for instance --force-window=yes would become --mpv-force-window=yes, and so on
In order to achieve this, I added a few placeholder lines (probably there is a better way to do it) on celluloid.py (which is a modified copy of mpvnet.py
    def run(client, playerPath, filePath, args):
        constants.MPV_NEW_VERSION = True
        constants.MPV_OSC_VISIBILITY_CHANGE_VERSION = True
        for mpv_arg in MpvPlayer.getStartupArgs(args):
            celluloid_arg = "mpv-" + mpv_arg
            MpvPlayer.getStartupArgs(args)[celluloid_arg] = MpvPlayer.getStartupArgs(
                args
            ).pop(mpv_arg)
        return CelluloidPlayer(
            client, CelluloidPlayer.getExpandedPath(playerPath), filePath, args
        )

The way this works on my head, is it would simply append the suffix mpv- to every key on the list of arguments that get passed to MPV.
While this suppresses any errors from celluloid, it still fails to open, and gives some errors which relate to clientManager.py client.py and internet/base.py
the errors are as follow
Traceback (most recent call last):
  File "/home/jawz/Development/Git/syncplay/syncplay/clientManager.py", line 16, in run
    syncplayClient.start(config['host'], config['port'])
  File "/home/jawz/Development/Git/syncplay/syncplay/client.py", line 878, in start
    reactor.run()
  File "/usr/lib/python3.10/site-packages/twisted/internet/base.py", line 1318, in run
    self.mainLoop()
  File "/usr/lib/python3.10/site-packages/twisted/internet/base.py", line 1328, in mainLoop
    reactorBaseSelf.runUntilCurrent()
--- <exception caught here> ---
  File "/usr/lib/python3.10/site-packages/twisted/internet/base.py", line 994, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "/home/jawz/Development/Git/syncplay/syncplay/players/celluloid.py", line 29, in run
    for mpv_arg in MpvPlayer.getStartupArgs(args):

as I am getting familiar with the code still, I was wondering if you could orient me as to what scripts to look at for when the player is called, and what other parts of the script add arguments to the player, since i'm assuming the error happens because of arguments which the .lua script requires to run, perhaps there is a better part where I can place the suffix function, I'll try to keep it away from mpv.py as much as possible.

It's been a while since I've had to work with the mpv arg code so I might have got some things wrong but my understanding is as follows:
The Syncplay-modified version of the MPV JSON IPC takes an args argument from mpv (or mpvnet/celluloid) which it then passes through to our custom _get_arglist function. This begins by setting idle, input_ipc_server, input_terminal and terminal values if they are not otherwise set and then extends it with the args from Syncplay. See: https://github.com/Syncplay/syncplay/blob/master/syncplay/vendor/python_mpv_jsonipc/python_mpv_jsonipc.py#L216 and https://github.com/Syncplay/syncplay/blob/master/syncplay/vendor/python_mpv_jsonipc/python_mpv_jsonipc.py#L245
As such, to prepend all of the args you would need to modify the function of self._get_arglist in python_mpv_jsonipc so that when you are using celluloid it prepends the names of all the arguments (default and user-specified) with "mpv-".

@CaptainJawZ if you can also support https://invent.kde.org/multimedia/haruna it would be really nice.

Apologies, I could've written this month's ago, since this is something I noticed the day I began trying to implement Celluloid into syncplay, but I didn't want to give up immediately, so I let the project on idle while I became more proficient (+ there were periods of time where life kept me very busy).
But now after spending some more time looking into this, doesn't seem like implementing this into celluloid is possible, while Celluloid pretty much is compatible with every MPV command, it cannot load scripts the way that syncplay would require, I tried doing some hacky stuff trying to setup a IPC socket, similar to how jellyfin-mpv-shim does it, but didn't had any luck getting Celluloid to play along. So I think for the moment it is not possible, perhaps in future builds of Celluloid, I do intend to look into this again in the future, but for the time being my hands are tied.

@CaptainJawZ haruna cannot be supported too?

@soredake same boat, can't receive MPV commands through command line, and through the gui, you can' attach the script file required for syncplay, only basic MPV functions like controlling volume, shortcuts, etc.

@CaptainJawZ thanks for trying! I'll close it for now.
It would be nice if you can please try adapting syncplay code to flatpak (with the help of flatpak-spawn), https://github.com/Syncplay/syncplay/issues/199#issuecomment-1135659633


Workflow Controller Promotheus Metric for CWF Execution Error

Summary
Workflow Controller should be able to emit a Prometheus metric when an error occurs when trying to execute a CronWorkflow.
Motivation
Currently, when a workflow controller encounters an error when resolving parameters for executing a cron workflow, the error will be logged to stdout but otherwise there is no way to be made aware of this failure.
If the workflow controller was able to emit a Prometheus event when this occurs, users could better handle this case (such as creating an alert).
Proposal
How do you think this should be implemented?
Assuming metrics can be emitted from the controller, then emit an additional metric in the error case when executing CWF (not sure of more specific implementation details)

Message from the maintainers:
If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

There are two issues within this issue.

There is currently no way to see the status of a CronWorkflow except looking at the logs. This should be fixed independently of any metrics being emitted. I have opened https://github.com/argoproj/argo/issues/3036 to track this issue.

We should emit metrics when the controller has certain errors, including but not limited to an error when operating on a CronWorkflow. I have edited this issue to track this problem.

Implementation-wise, how do you think the controller should add these new metrics? Should it upsert a custom metric to Metrics Collector, or should the Metrics type be expanded to hold an additional Counter for workflow-controller errors?

Hey @Syps, sorry for the delay. Opened https://github.com/argoproj/argo/pull/3144 to fix

Awesome, thanks @simster7 !


fix OnUpdate() in smController in 2.6

Signed-off-by: marcolan018<EMAIL_ADDRESS>https://github.com/stolostron/backlog/issues/25564

/retest


Build docker images on Travis CI

This pull request will switch from pulling Docker images from DockerHub to building them within the Travis CI build.  That will make it possible to have parallel pull requests changing SINGE features and eliminate one manual step (building the image locally and pushing it to DockerHub) from the testing process.

The updated version now runs the tests in the new image that Travis CI builds.  The tests have also been condensed into a test script, but it is preferable to activate the conda test environment outside the script.  When pull requests are merged and Travis CI builds the Docker image on the master branch, it will deploy the updated image to DockerHub with the tmp tag.  Versioned Docker images are still built manually.

The deploy appears to have worked.  DockerHub shows a new tmp image was pushed.


User registration form within slack

Let users register and edit their info from slack. User query from slack?

HTML forms can actually be displayed within slack!

just read thru the slack command docs and it looks like we wont be able to do this with a full form inside of slack.
The other idea I came up with after reading thru the docs is this:

User calls slack command /progbot me
This will show some slack buttons, one of which will be the edit button.
User clicks on edit button and is taken to their personal info page on the progbot heroku app. We will use their slack information to only show them their info.
User updates their info there and saves everything.
Pamela buys us beers since she doesnt have to fill in all that info anymore 😄

On top of that, the staff can provide a url (as part of the onboarding process) that directs users to do the same thing.
e.g., they hit a landing page that prompts them for their signup info, then they get routed to their info.

testing slack integration

test slack integration


Remove SFBHTTPInputSource

The way that HTTP input is currently implemented is broken and is not easily fixable without some amount of redesign.
See #258

Also see #206


Release with package_info_plus: ^8.0.0

Hello,
Thanks for that useful package :)
I see you recently upgraded dependencies to use package_info_plus: ^8.0.0.
Do you have other changes planned before the next release ?
For now, I'll point on your latest commit ;)
Thanks for your help.

@kauemurakami we are still having to use another fork because this one depends on package_info_plus ^6.0.0
Are you planning another release?

@kauemurakami @pom11 yo

@Macacoazul01 @Chralu not that I recommend this way of doing things, but you can use it as such, until (or if) they release it.
app_version_update:
  git:
    url: https://github.com/kauemurakami/app_version_update
    ref: 499047d

im already using my fork with the pub update, tks @romatallinn!
@kauemurakami when are you planning to release?

Sorry, I've been very busy these last few months, last time I couldn't reproduce the example on my machine even though I was running all my other projects normally, I'll redo some things this week, if you have any ideas you can contribute a PR, I'll pay attention to the package this week!

Hello @Macacoazul01 @Chralu @romatallinn I just published version 5.0.0, I was having difficulty updating the example, and I discovered that the country specification was causing this error, I tested it with an app of mine in the store with a low version like "4.5.0" and with Facebook which is predefined in the example.
My Mac no longer supports xcode for this latest version of Flutter 😢, if anyone identifies an error on iOS, please let me know.
Test it and don't forget to open an issue if there is a problem, I will work on automatic closure after updating a mandatory update.
I will close this issue, but can reopen in error case, tkhs for patience.

I test with facebook version "4<IP_ADDRESS>.78", and example 1.0.0, so bad news ... don't wor with "normal" version example 4.17.0, i working in this NOW. sorry

Test with 5.0.1