Validate conflicts_with

 Issue by HalfVoxel
Friday Jan 22, 2021 at 15:06 GMT
Originally opened as https://github.com/clap-rs/clap/issues/2307

Make sure you completed the following tasks

[x] Searched the discussions
[x] Searched the closed issues

Describe your use case
Currently, it can sometimes be tricky to know which name to give to conflicts_with, in particular in combination with structopt.
Take a look at this issue for context: https://github.com/TeXitoi/structopt/issues/459
Clap currently doesn't seem to care if conflicts_with is given a name which isn't even a valid field.
Describe the solution you'd like
Clap validates that the field conflicts_with is given actually exists.

 Comment by pksunkara
Friday Jan 22, 2021 at 18:46 GMT

Have you checked this with master? And also please provide a minimal sample code so it's easier for everyone.

 Comment by HalfVoxel
Friday Jan 22, 2021 at 18:57 GMT

I have not yet, my testing was done with clap v2.33.3 which is the latest stable version afaik (though I see that there's a 3.0 beta now).
I use structopt so that generates most of the code for me. You can find an example of that in the linked issue.
Here is the code that structopt generates which corresponds to this issue:
let app = app.about("Runs a module");
let app = app.arg(
    ::structopt::clap::Arg::with_name("module-name")
        .takes_value(true)
        .multiple(false)
        .validator(|s| {
            ::std::str::FromStr::from_str(s.as_str())
                .map(|_: String| ())
                .map_err(|e| e.to_string())
        })
        .help("Name of the cmdlet module to run")
        .short("n")
        .long("name"),
);
let app = app.arg(
    ::structopt::clap::Arg::with_name("module-cid")
        .takes_value(true)
        .multiple(false)
        .validator(|s| {
            ::std::str::FromStr::from_str(s.as_str())
                .map(|_: Cid| ())
                .map_err(|e| e.to_string())
        })
        .help("CID of the cmdlet module to run")
        .short("c")
        .long("cid")
        .conflicts_with("module_name"),
);
app.version("0.1.0")

Note that conflicts_with is given the name module_name while the generated name is actually module-name.

 Comment by pksunkara
Friday Jan 22, 2021 at 19:17 GMT

Can you please test it on clap master and report back?

 Comment by HalfVoxel
Monday Jan 25, 2021 at 15:58 GMT

Hi
I can confirm that this still happens with master.
[dependencies]
clap = { git = "https://github.com/clap-rs/clap.git", rev= "d7f6748" }

use clap::{App, Arg};

fn main() {
    let app = App::new("My Super Program")
        .about("Runs a module")
        .arg(
            Arg::new("module-name")
                .takes_value(true)
                .multiple(false)
                .short('n')
                .long("name"),
        )
        .arg(
            Arg::new("module-cid")
                .takes_value(true)
                .multiple(false)
                .short('c')
                .long("cid")
                .conflicts_with("module_name"),
        )
        .version("0.1.0");

    let matches = app.get_matches();
    println!("{:#?}", matches);
}

 ~/p/e/t/claptest (master) [2]> cargo run --release -- --name=NAME --cid=CID                                   (base)
   Compiling claptest v0.1.0 (/home/arong/projects/embark/test/claptest)
    Finished release [optimized] target(s) in 0.40s
     Running `target/release/claptest --name=NAME --cid=CID`
ArgMatches {
    args: {
        [hash: 49BCF4FC57DA0A4]: MatchedArg {
            occurs: 1,
            ty: CommandLine,
            indices: [
                1,
            ],
            vals: [],
        },
        [hash: 110F162FD0016129]: MatchedArg {
            occurs: 1,
            ty: CommandLine,
            indices: [
                2,
            ],
            vals: [],
        },
    },
    subcommand: None,
}


Expose Rendering Primitives via NodeJS

Hi,
Great to see the Node JS support landing in dev.  Currently the module only exposes the generateOutput and compile.
I've used https://www.npmjs.com/package/jscad to support programmatically generating 3D models.    I've used this to generate programatic models based on external output.
Is there intent to expose the primitives (cube, cylinder, union, etc) in addition to the higher level functions?

Hi ! that is a good question !
I know I would love to be able to use it like that !
I am not 100% sure if it should be part of the main openjscad module:

Some of those features are part of CSG.js which is very likely going to be split out to a seperated module.
perhaps different packages 'specialized' (2d primitives, 3d primitives, boolean operations) is also a good approach , so that one can 'pick & choose' ?

My main concern is programmatic modeling with the ability to generate the resultant model (primary want x3d or WRL).  CSG might be sufficient for a lot of what I want to achieve.  I'd suggest the simplest path and considering specialization later.
If openjscad (which is the closest I've seen to what I want) is not a suitable direction, I'm fine with pushing forward in a different direction.
From https://github.com/Spiritdude/OpenJSCAD.org/issues/74, it seems like exposing CSG.js and formats.js as a node module would probably be the best model for my purpose.  Although jscad exported a lot of the OpenJSCAD interfaces directly.
As stated, my intent is around generating server-side geometry based on user specified data, not just parametric construction.

@mtippett Honestly my points were mostly me 'thinking aloud', I am not entirely sure what the scope of the package should be (not my decision to make either :)
Exposing all the modeling helpers would be trivial, and I also have been using Openjscad like that for a while.

Also , I have just finished a HUGE restructuring PR #205  that makes the different pieces of this package more clear, and more modular/exposable :  the src/modeling folder in that PR (soon to be merged) already contains these features seperated by 'category'
Personally I would love something like this (es6 code)
import { cylinder, cube } from 'jscad-primitives3d'
import { union } from 'jscad-booleanops'

export default function tubeCap (options) {
  const defaults = {length: 12, tubeId: 5, tubeOd: 10}
  const {length, tubeId, tubeOd} = Object.assign({}, defaults, options)

  const capHole = cylinder({d: tubeId, h: length, fn: 64})
  const capTube = cylinder({d: tubeOd, h: length, fn: 64})

  return difference(capTube, capHole)
}

The problem is... Where do you stop? Do you want circles? Polygons? Vertexes? Points?
Remember, this started as a replacement for OpenSCAD, which has a scripting language for composing geometries from simple primitives.
Maybe there's something in between that can make the exposing of functions easier.

@z3dev do you mean as far as granularity goes ?
I honestly kinda like grouping things together by 'topic' (ie 2d primitives, 3d primitives, hull operations etc) so any finer grained thing would be to much (but that is subjective).
HOW things are published 'module' wise for node.js is a seperate concern I think : this repo could also have lots of smaller modules within it that are published seperataly on node.js yet still maintained together (it is actually rather trivial to do).

Those that make JSCAD scripts also need compatibility. Designs created in the GUI (there are two now) have to be usable by the NODE version of OpenJSCAD. There are people investing a lot of time making designs, and even JSCAD libraries. So, if the NODEJS exports don't disable those JSCAD scripts then sure. These environments can co-exist.
In other words, those who create JSCAD scripts should not have to learn how to use NODEJS.
It would also be cool if the geometric primitives (core) could be exposed as well.
Are we talking the same thing?
Anyway, one step at a time. Let's work on getting the new structure reviewed, tested, and released.

Compatibility with existing Jscad scripts is already the case (the current 2 functions exposed in the node module already ensure that), but when you want to use the "prilitives" like this , you are not aiming for the same kind of use : you want to be able to explicitly import/require/use certain primitives.

those who create JSCAD scripts should not have to learn how to use NODEJS
That is not quite true in the case of using Openjscad as a node module: you are using it in the context of Node.js , so I would argue that you do need to know about Node.js , when using it with Node.js :)
This does not impact the UI or CLI use in any way, but 'apples & oranges ' etc :)
There is a lot more power sticking in the belly of OpenJSCAD then what is possible for now in the UI I believe :)

I'm going to try https://github.com/kaosat-dev/OpenJSCAD.org/tree/modulify and see if that will work with my use case.   Is there any particular gotcha's that I should be aware of?

Hi @mtippett sorry , missed your replies :
The modulify PR is now merged into DEV which should make things easier
As you might have seen the geometric primitive are not yet exposed (just to be sure, you meant in the node module right ?) , but yes, help is more then welcome for this :)

NP, noticed that it had been merged a few hours ago.  Switching back to dev :).
Yes, I'm looking at purely node.
About 18 months ago, there was https://github.com/Spiritdude/OpenJSCAD.org/pull/93 which was pushed into NPM independently.  The small sets of patches exposed the functionality that I have been using.
What would be good next steps to achieve the intent of this issue (#208)?

I like those changes, but not a fan of the implict globals : I think we can get rid of that aspect, and still expose things
how about exporting things to openjscad/modeling ie just add an extra entry to the current node module with all moduling functions something like this (simplified)
const modeling = require('jscad').modeling
const {cube, cylinder, difference} = modeling

module.exports = function board(){
  return difference(cube(), cylinder())
}

I am still not exactly a fan of exporting all modeling helpers in a big 'grab bag' so perhaps something like this :
const modeling = require('jscad').modeling
const {cube, cylinder} = modeling.primitives3d
const {difference}     = modeling.booleanOps

module.exports = function board(){
  return difference(cube(), cylinder())
}

There is also the issue with CSG ie CSG/CAG itself: best would be to have it be a different module that is just a dependency , but for now we could just export it like it is done here : https://github.com/Spiritdude/OpenJSCAD.org/blob/dev/src/modeling/index.js

I'm only looking at using the CSG primitives.  The example in that PR were just.  My interest at this stage is just to have the openjscad primitives exposed.  Being more explicit, what I am looking for is the ability to create adhoc constructive geometries and then render them to a file (x3d is preferred).
The gaps that I see currently are

exposed primitives
render output (STL is currently fine)

Finally from what I have seen, x3d currently needs the browser.
I'm new to node (and this my first meaningful node project), so I'm still parsing some of the guidance.

I'm only looking at using the CSG primitives.  The example in that PR were just.  My interest at this stage is just to have the openjscad primitives exposed.  Being more explicit, what I am looking for is the ability to create adhoc constructive geometries and then render them to a file (x3d is preferred).
The gaps that I see currently are

exposed primitives
render output (STL is currently fine)

Finally from what I have seen, x3d currently needs the browser.
I'm new to node (and this my first meaningful node project), so I'm still parsing some of the guidance.

@mtippett ahh ok, so only the primitives in the CSG/CAG 'scope' then ?
X3D will indeed need to be rewritten with a more generic XML support (trivial)
I will give that one a go in the near future

My intent is to leverage existing APIs and interfaces as much as possible, so nothing fancy initially.
As mentioned, I'm happy to give it a go myself, I just would need a nudge in the right direction.  Likely I have a couple of small conceptual humps to get over, and I'll be able to contribute.

I came to this repo with the same need as @mtippett: generate server side geometry with a nodejs script and export it as STL. There's a lot of code that is not strictly speaking needed if your purpose is server side (worker, viewer, ie compat, site interface).
The CSG and the Scad/Stl import/export should both exist as separate libraries, in the case some people want to run their OpenJSCAD in JSBin or Atom (See @bebbi PR which is a remarkable attempt to merge the various works).
As for the library of components, I thought that OpenJSCAD could expose a few basic objects (cube, sphere, teapot), and that we could spawn npm modules for other more advanced things (like a mix between thingiverse and npm). If you're design needs a gear, just import npm-jscad-gear and configure it. I believe someone already mentionned this idea in one of the issues.

@kaosat-dev I'm fine to be involved in that, my personal priority is to expose via the npm interfaces the csg primitives.  Can that align with you?  Feel free to reach out tippettm at gmail.com via hangouts or mail to discuss further.

I came to this repo with the same need as @mtippett: generate server side geometry with a nodejs script and export it as STL. There's a lot of code that is not strictly speaking needed if your purpose is server side (worker, viewer, ie compat, site interface).

I don't see any problem here:
import csg.js, formats.js and Blob.js in your script (I'd need to figure out the exact syntax in node.js)
then just do
var cube = CSG.cube({...}); var sphere = CSG.sphere({...}); var mysolid = cube.union(sphere); return mysolid.toStlBinary();
csg.js and formats.js do not contain any gui/viewer/worker stuff so you should be able to use them server side.

I like those changes, but not a fan of the implict globals : I think we can get rid of that aspect, and still expose things in a very similar way

Just  import the global CSG and CAG objects from CSG.js and use the object oriented interface (see the code above). This gives you the solids and all operations in a single namespace.
The global functions were added to make it easier to convert existing OpenSCAD scripts to OpenJSCad. But again encapsulating those globals would be somewhat like reinventing the wheel
:-)

I guess that's my stopping point.  I'm trying hard (possibly too hard) to avoid forking the openscad to suite my needs.  As you suggested, CSG, formats and Blobs are really all that I need.  I'd prefer being completely node oriented (ie: require), and not js oriented (ie: import).
I'm fine either exposing via
require('openjscad');

or
require('openjscad-csg');
require('openjscad-formats');
require('openjscad-blob');

And keeping within the namespace of openjscad.     I'd assume with that model, the require('openjscad'); would import the other modules as needed.  Of course, if there is a convenient way of exposing csg and formats via require('openjscad'); that might be a preferred model.  I'm assuming that I've gone down a rabbit hole and am probably not seeing the simple answer in front of me.

@joostn
we still need to have an up to date place on npm to import csg.js from though :)
Also @z3dev  was working on adding tests & improvements to csg.js but he can chime in on that.
Side notes:

I did some very minor changes on dev to the csg.js we have in this repo in order to make it es6 module compatible
formats.js was split up into more specialised modules (on dev too)

@mtippett   if you only need the csg /cag namespaces you only need the csg.js lib, exposed as node module,  there is no need to fork anything
What THIS module could/should/might export is the openscad like set of wrappers that USE csg.js

node oriented (ie: require), and not js oriented (ie: import)
you can use both transparently if you are speaking about this module at least :)

@joostn  I forgot the most important part : with the utilies exported by this module (dev again) you should not need to use formats & blob anymore , whatever their shape :)

I propose we join https://gitter.im/OpenJSCAD-org/Lobby to discuss further the matter

@mtippett & @dav-m85 the new csg package is now available on npm here : https://www.npmjs.com/package/@jscad/csg
so is the openscad like api https://www.npmjs.com/package/@jscad/scad-api
this repo will also be but on npm whenever possible,
and all the basic modelling functions are thus exposed : can we close this issue ?

Sure.
If further discussion is required then let's start that via the jscad organization.

Yup.  Confirmed that we can close and track issues in jscad


Fixes broken session

See https://github.com/yyuu/jetty-nosql-memcached/issues/29

Hi, thanks for your reply.
You are right: It may happen in theory that during execution of #idInUse and #storeSession the same session id will be acquired by another user. But this is very unlikely to happen: The timeframe is very short and the chance to get the same session id at all is very very low. Thus I would say that simply checking this would be safe.
Note that Jetty's in-built Mongo implementation also doesn't reserve the session id but simply checks the existence. I think this is also exactly as intended by the method's "contract".
Anyway: Another approach for solving the mentioned bug would be to use a well-defined value (e.g. an empty byte array) when reserving the session and add a special handling for this value when reading in the value again (#loadSession)

Yes, the race condition would very unlikely to happen, but there's possibility to happen and I think it would be better to add some workaround for it if possible.
I think using sensible default value would be better for this. If there's some difficulty to implement it, I think it's okay to merge your current patch if Jetty's mongo impl isn't using strict manner on reserving session id ;)

Putting sensible dummy session in KeyValueSessionIdManager#idInUse is not simply possible since we cannot determine the serializer at that place.
And, I confirmed that Jetty's MongoSessionIdManager using findOne to check if the session ID is in use or not. I think it is acceptable to merge your changes to not store invalid session data on memcached. Thanks for PR!
https://github.com/eclipse/jetty.project/blob/master/jetty-nosql/src/main/java/org/eclipse/jetty/nosql/mongodb/MongoSessionIdManager.java#L579-L598


Added minemytraffic

Website looks down for me. Do you have an example of a mining script running from that domain?

The script is located here https://minemytraffic.com/lib/xminer.min.js

Thanks for the contribution, merging in dev and preparing for an update (0.4.1).


Get the observables into the ensemble results

Right now the ensemble stuff works for getting the underlying dynamics (i.e., the u) but doesn't provide the z from the solutions.  Might be nice to have a way to get that in the ensemble simulations as well.  For example, a setting to concatenate them/etc.

Yeah, I noticed this. We need some way to make the ensembles a bit more general, for here and the optimization results.


C#: Add MustBeVariant attribute and analyzer

MustBeVariant attribute can be used to enforce that generic types must be a marshable from/to Variant.
This attribute allows us to specify that the generic types in Array<T> and Dictionary<TKey, TValue> must be Variant otherwise they will throw in runtime, with the attribute an analyzer will show an error to users on compile time. The analyzer will also check that if the type used is generic it must also be annotated with the [MustBeVariant] attribute.
This attribute is also exposed to users that may find it useful to have the restriction applied to generic types in their own code or when they need it to work with GodotSharp's code annotated with the attribute in a generic context.
TODO:

[x] Not sure what diagnostic Id should be used.
[x] I don't know if the analyzer should be implemented in the Godot.SourceGenerators project or in a different project, but in a different project we'd need a common project to share the MarshalUtils code.
[x] Should we remove the TypeCache(GeneratorExecutionContext context) constructor in favor of the TypeCache(Compilation compilation) constructor since we only seem to need the Compilation from the context? I kept it to avoid changing its usage in all source generators in this PR. Alternatively, I could remove the TypeCache(Compilation compilation) constructor in favor of a new TypeCache(SyntaxNodeAnalysisContext context) which parallels the constructor that takes a GeneratorExecutionContext.

Not sure what diagnostic Id should be used.

It should follow the format GODOT-GDXXXX.
I try to keep related diagnostics together, and for that I reserve some space. Hence, why there are sudden jumps from 0100 to 0200. But don't worry too much about this. Maybe we should have one slot reserved for misc diagnostics.

* [ ]  I don't know if the analyzer should be implemented in the `Godot.SourceGenerators` project or in a different project, but in a different project we'd need a common project to share the `MarshalUtils` code.

I think it's fine here, at least for now.

Thanks!


cdc tidb changefeed checkpoint cannot advance

What did you do?

tidb changefeed running
11/30 create storage changefeed
after creating storage changefeed , ticdc meeted panic
upgrade ticdc , ticdc won't panic
storage changefeed remove

What did you expect to see?
tidb changefeed lag can be decrease after upgrade
What did you see instead?

changefeed lag is increasing before upgrade

changefeed lag is increasing after upgrade

Versions of the cluster
Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):
Release Version: v6.5.0-alpha
Edition: Community
Git Commit Hash: cca78e37e249c235c9664fc6b5f6c2ac4b4a3b79
Git Branch: heads/refs/tags/v6.5.0-alpha
UTC Build Time: 2022-11-30 00:24:59
GoVersion: go1.19.3
Race Enabled: false
TiKV Min Version: 6.2.0-alpha
Check Table Before Drop: false
Store: unistore
(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):
(paste TiKV version here)

TiKV
Release Version:   6.5.0-alpha
Edition:           Community
Git Commit Hash:   bfe29655a2a1634c268a7fb3986892be7dc1f305
Git Commit Branch: heads/refs/tags/v6.5.0-alpha
UTC Build Time:    2022-11-28 11:07:57
Rust Version:      rustc 1.67.0-nightly (96ddd32c4 2022-11-14)
Enable Features:   pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure
Profile:           dist_release
TiCDC version (execute cdc version):
before update:
Release Version: v6.5.0-alpha
Git Commit Hash: 07657c5fa7035f4dc02f82473c50d516ff7b58ff
Git Branch: heads/refs/tags/v6.5.0-alpha
UTC Build Time: 2022-11-28 11:06:34
Go Version: go version go1.19.3 linux/amd64
Failpoint Build: false
now:
Release Version: v6.5.0-alpha
Git Commit Hash: 519c6f08dc5ca000d8c92101e2987daf071d716e
Git Branch: heads/refs/tags/v6.5.0-alpha
UTC Build Time: 2022-11-30 11:07:02
Go Version: go version go1.19.3 linux/amd64
Failpoint Build: false

/severity Critical

/found Automation

This issue is also seen when create storage sink with NFS storage target.
[2022/12/02 15:45:52.108 +08:00] [WARN] [replication_manager.go:499] ["schedulerv3: cannot advance checkpoint since missing table"] [namespace=default] [changefeed=test2] [tableID=78]
[2022/12/02 15:45:52.157 +08:00] [WARN] [replication_manager.go:499] ["schedulerv3: cannot advance checkpoint since missing table"] [namespace=default] [changefeed=test2] [tableID=78]
[2022/12/02 15:45:52.168 +08:00] [WARN] [client.go:259] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=59m48.001033165s] [role=processor]
[2022/12/02 15:45:52.207 +08:00] [WARN] [replication_manager.go:499] ["schedulerv3: cannot advance checkpoint since missing table"] [namespace=default] [changefeed=test2] [tableID=78]

Maybe fixed by #7777.

This issue is also seen when create storage sink with NFS storage target.
[2022/12/02 15:45:52.108 +08:00] [WARN] [replication_manager.go:499] ["schedulerv3: cannot advance checkpoint since missing table"] [namespace=default] [changefeed=test2] [tableID=78]
[2022/12/02 15:45:52.157 +08:00] [WARN] [replication_manager.go:499] ["schedulerv3: cannot advance checkpoint since missing table"] [namespace=default] [changefeed=test2] [tableID=78]
[2022/12/02 15:45:52.168 +08:00] [WARN] [client.go:259] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=59m48.001033165s] [role=processor]
[2022/12/02 15:45:52.207 +08:00] [WARN] [replication_manager.go:499] ["schedulerv3: cannot advance checkpoint since missing table"] [namespace=default] [changefeed=test2] [tableID=78]

Not see this issue with latest build

https://github.com/pingcap/tiflow/issues/7780

/remove-label affects-6.5

This issue is also seen when create storage sink with NFS storage target.
[2022/12/02 15:45:52.108 +08:00] [WARN] [replication_manager.go:499] ["schedulerv3: cannot advance checkpoint since missing table"] [namespace=default] [changefeed=test2] [tableID=78]
[2022/12/02 15:45:52.157 +08:00] [WARN] [replication_manager.go:499] ["schedulerv3: cannot advance checkpoint since missing table"] [namespace=default] [changefeed=test2] [tableID=78]
[2022/12/02 15:45:52.168 +08:00] [WARN] [client.go:259] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=59m48.001033165s] [role=processor]
[2022/12/02 15:45:52.207 +08:00] [WARN] [replication_manager.go:499] ["schedulerv3: cannot advance checkpoint since missing table"] [namespace=default] [changefeed=test2] [tableID=78]

We have recently see this issue again on v6.5.2 with an extremely faulty NFS drive (returning "invalid argument" error when closing files). The "missing table" error goes on for 4 hours since the changefeed inititialized until CDC restarted. This seems probabilistic (sometimes works sometimes stuck) and irrelevant to closing the sink manager.
Should we file a new issue for this?

@kennytm Could you please provide the log file of ticdc?

(sent privately)

@kennytm

When changefeed is restarted or created, the owner scheduler sends heartbeat information to all the capture nodes and determines that the schduler initialization is successful only after receiving replies from all nodes.
The logs show that one of the processors in the cluster has been stuck, and some logs indicate that the p2p system is dropping messages due to congestion:

[2023/07/03 14:24:20.210 +08:00] [WARN] [client.go:171] ["peer message client detected error, restarting"] [error="[CDC:ErrPeerMessageServerClosed]peer-to-peer message server has closed connection: [CDC:ErrPeerMessageTopicCongested]peer-to-peer message topic has congested, aborting all connections."] [errorVerbose="[CDC:ErrPeerMessageServerClosed]peer-to-peer message server has closed connection: [CDC:ErrPeerMessageTopicCongested]peer-to-peer message topic has congested, aborting all connections.\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageClient).runRx\n\tgithub.com/pingcap/tiflow/pkg/p2p/client.go:360\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageClient).run.func2\n\tgithub.com/pingcap/tiflow/pkg/p2p/client.go:208\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.1.0/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1594"]
[2023/07/03 14:24:20.910 +08:00] [WARN] [client.go:259] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=47h37m5.499571149s] [role=processor]

In addition, between 2023/07/03 10:24:42.557 and 2023/07/03 14:23:26.207, no logs were found indicating that all capture initializations were successful, which confirms the analysis of the second point above.
Eventually the scheduler gets stuck in an uninitialized state, and never performed a scheduling task, which caused the missing table error.

In conclusion, to solve this problem, we need to figure out why the processor is stuck.

The issue above has been identified as #9370.


Export bug on H2020 template due to invalid html

Plan written with the H2020 template won't export due to html in the question area. This is particularly on phase 1. See discussion and notes in ticket #217

The current gem for exporting to docx does not support strong tag as ancestor of li tag when converting from html to docx (see https://github.com/karnov/htmltoword/blob/master/lib/htmltoword/xslt/base.xslt#L184).
This fix takes into account this limitation, however any question text exported to .docx no longer has its text bold.

Exports fine now. Closing out


Crowdsec agent cannot parse Nginx ingress logs unless "container_runtime: nginx"

I installed crowdsec on kubernetes similarly to https://github.com/crowdsecurity/helm-charts/issues/46 and i got a similar results, all logs where unparsed by the nginx-logs parser.
I got it working by setting container_runtime: nginx in the values file for the crowdsec helm-chart. Is this the exptected behaviour or have i missconfigured something else? I couldn't find this in the documentation

Hi @william-elastisys,
This option can be set on ly to docker or cri. It's about the log format your containers engine produce.
In normal clusters it will mainly be json format, so docker. But sometimes with kind or k3s, the containers output logs are in CRI format, that why we use cri.
So I don't know if it really worked on your side, you need to fix the value for this parameter.

Hi, I get the following results with different values of container_runtime
With container_runtime: nginx

With container_runtime: cri

With container_runtime: docker

With container_runtime: containerd

The same issue here. I confirm it works with the workaround from @william-elastisys.

I've the same problem with traefik.
As the docs for acquisition and file datasource describe:

The labels part is here to tag the incoming logs with a type. labels.type are used by the parsers to know which logs to process.

So the acquis-configmap.yaml is probably wrong and should look like this:
# ...
    labels:
      type: {{ .program }}

Though I'd rename the param to type, like this:
# ...
    labels:
      type: {{ .type }}

So you have to write this in your helm values.yaml:
agent:
  acquisition:
    - namespace: traefik
      podName: traefik-*
      type: traefik

Fixed for my test-setup (with local IPs) by adding this to values.yaml:
container_runtime: containerd
agent:
  env:
    - name: PARSERS
      value: "crowdsecurity/cri-logs"
    - name: DISABLE_PARSERS
      value: "crowdsecurity/whitelists"

I've the same problem with traefik.
As the docs for acquisition and file datasource describe:

The labels part is here to tag the incoming logs with a type. labels.type are used by the parsers to know which logs to process.

So the acquis-configmap.yaml is probably wrong and should look like this:
# ...
    labels:
      type: {{ .program }}

Though I'd rename the param to type, like this:
# ...
    labels:
      type: {{ .type }}

So you have to write this in your helm values.yaml:
agent:
  acquisition:
    - namespace: traefik
      podName: traefik-*
      type: traefik

This is not advised as you need to have a s00 parser which is normally syslog, docker or cri. Which prepares the log for the next stage which the application parser will then correct parse just the message.

Hi, I get the following results with different values of container_runtime
With container_runtime: nginx
With container_runtime: cri
With container_runtime: docker
With container_runtime: containerd

Setting the container runtime to nginx would work technically as the log is embedded within the log line, however, for multiple container types SQL / nginx this will not work in the long run.
From my experience you most likely needed to set the runtime to cri but you didnt add the cri collection to your values.yaml so it wasnt installed for crowdsec to use.

The same issue here. I confirm it works with the workaround from @william-elastisys.

This is not advised please see my reply above

I'm having the same problem.
I followed your advice @LaurenceJJones. Activated the cri parser, and set it as the type
crowdsec-agent-f4b9x:/# cscli parsers list

PARSERS
───────────────────────────────────────────────────────────────────────────────────────────────────────────────
 Name                             📦 Status   Version   Local Path
───────────────────────────────────────────────────────────────────────────────────────────────────────────────
 crowdsecurity/cri-logs           ✔ enabled   0.1       /etc/crowdsec/parsers/s00-raw/cri-logs.yaml
 crowdsecurity/dateparse-enrich   ✔ enabled   0.2       /etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml
 crowdsecurity/docker-logs        ✔ enabled   0.1       /etc/crowdsec/parsers/s00-raw/docker-logs.yaml
 crowdsecurity/geoip-enrich       ✔ enabled   0.2       /etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml
 crowdsecurity/http-logs          ✔ enabled   1.1       /etc/crowdsec/parsers/s02-enrich/http-logs.yaml
 crowdsecurity/nginx-logs         ✔ enabled   1.3       /etc/crowdsec/parsers/s01-parse/nginx-logs.yaml
 crowdsecurity/sshd-logs          ✔ enabled   2.0       /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml
 crowdsecurity/syslog-logs        ✔ enabled   0.8       /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml
 crowdsecurity/whitelists         ✔ enabled   0.2       /etc/crowdsec/parsers/s02-enrich/whitelists.yaml

Content of /etc/crowdsec/acquis.yaml
---
filenames:
  - /var/log/containers/*-nginx-ingress-*_*_*.log
force_inotify: true
labels:
  type: cri
  program: nginx

Cscli explain with type "nginx"
line: 2023-01-26T12:56:15.04394063Z stdout F <IP_ADDRESS> - - [26/Jan/2023:12:56:15 +0000] "GET /api/live/ws HTTP/1.1" 401 27 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36" "-"
        ├ s00-raw
        |       ├ 🔴 crowdsecurity/cri-logs
        |       ├ 🔴 crowdsecurity/docker-logs
        |       ├ 🟢 crowdsecurity/non-syslog (first_parser)
        |       └ 🔴 crowdsecurity/syslog-logs
        ├ s01-parse
        |       └ 🟢 crowdsecurity/nginx-logs (+23 ~2)
        ├ s02-enrich
        |       ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
        |       ├ 🟢 crowdsecurity/geoip-enrich (+13)
        |       ├ 🟢 crowdsecurity/http-logs (+7)
        |       └ 🟢 crowdsecurity/whitelists (unchanged)
        ├-------- parser success 🟢
        ├ Scenarios
                └ 🟢 crowdsecurity/http-crawl-non_statics

As you can see cri-logs fail. What can it be?

Nevermind. It's working.

Hi team,
I seem to have the same problem as described before where none of my logs get parsed. I am deploying everything within a k8s cluster. The output of my cscli metrics
╭──────────────────────────────────────────────────────────────┬────────────┬──────────────┬────────────────┬────────────────────────┬───────────────────╮
│                            Source                            │ Lines read │ Lines parsed │ Lines unparsed │ Lines poured to bucket │ Lines whitelisted │
├──────────────────────────────────────────────────────────────┼────────────┼──────────────┼────────────────┼────────────────────────┼───────────────────┤
│ file:/var/log/containers/nginx-ingress-microk8s-controller-l │ 1          │ -            │ 1              │ -                      │ -                 │
│ dlp5_ingress_nginx-ingress-microk8s-ace27de928e788d77337c3c2 │            │              │                │                        │                   │
│ a8a60698f9b3a42109a834fb0d88a891eb3f04aa.log                 │            │              │                │                        │                   │
╰──────────────────────────────────────────────────────────────┴────────────┴──────────────┴────────────────┴────────────────────────┴───────────────────╯

Parser Metrics:
╭────────────────────────────────┬──────┬────────┬──────────╮
│             Parsers            │ Hits │ Parsed │ Unparsed │
├────────────────────────────────┼──────┼────────┼──────────┤
│ child-crowdsecurity/nginx-logs │ 3.00k    │ -      │ 3.00k        │
│ crowdsecurity/nginx-logs       │ 3.00k    │ -      │ 3.00k        │
│ crowdsecurity/non-syslog       │ 3.00k    │ 3.00k      │ -        │
╰────────────────────────────────┴──────┴────────┴──────────╯

when my nginx logs look like this:
<IP_ADDRESS> - - [28/Jun/2024:14:15:08 +0000] "GET /selfTestPci586.d HTTP/2.0" 200 53 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36" 32 0.204 [es-dev-es-dev-443] [] <IP_ADDRESS>:8443 53 0.203 200 d77633989c1e405f9321cfa84f545197
<IP_ADDRESS> - - [28/Jun/2024:14:15:08 +0000] "GET /selfTestPci586.d HTTP/2.0" 200 53 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36" 32 0.183 [es-dev-es-dev-443] [] <IP_ADDRESS>:8443 53 0.182 200 733fed93485463aa28918026e93ced5b

and my values.yaml is:
agent:
  acquisition:
    - namespace: ingress
      # The pod name
      podName: nginx-ingress-microk8s-controller-*
      program: nginx

  # Those are ENV variables
  env:
  - name: DISABLE_ONLINE_API
    value: "true"
  - name: COLLECTIONS
    value: "crowdsecurity/nginx"
lapi:
  env:
    - name: DISABLE_ONLINE_API
      value: "true"

I tried setting container_runtime to cri, docker, container and ngnix with no avail. I also tried to add:
- name: PARSERS
    value: "crowdsecurity/cri-logs"

but it did not help


fix tags for 'intentional' errors : (debugging.rst)

[x] ./rst_files/debugging.rst: :class: no-execute

Let's consider a simple (and rather contrived) example

.. code-block:: python3
    :class: no-execute

    import numpy as np
    import matplotlib.pyplot as plt

    def plot_log():
        fig, ax = plt.subplots(2, 1)
        x = np.linspace(1, 2, 10)
        ax.plot(x, np.log(x))
        plt.show()

    plot_log()  # Call the function, generate plot

This code is intended to plot the ``log`` function over the interval

[x] ./rst_files/debugging.rst: :class: no-execute

[x] ./rst_files/debugging.rst: :class: no-execute
[x] ./rst_files/debugging.rst: :class: no-execute
[x] ./rst_files/debugging.rst: :class: no-execute
[x] ./rst_files/debugging.rst: :class: no-execute

[x] ./rst_files/debugging.rst: :class: no-execute
[x] ./rst_files/debugging.rst: :class: no-execute

so I add     :class: no-execute:
Here's what happens when we run the code:

.. code-block:: none
    :class: no-execute

    ---------------------------------------------------------------------------
    AttributeError                            Traceback (most recent call last)
    <ipython-input-1-ef5c75a58138> in <module>()
          8     plt.show()
          9
    ---> 10 plot_log()  # Call the function, generate plot

    <ipython-input-1-ef5c75a58138> in plot_log()
          5     fig, ax = plt.subplots(2, 1)
          6     x = np.linspace(1, 2, 10)
    ----> 7     ax.plot(x, np.log(x))
          8     plt.show()
          9

    AttributeError: 'numpy.ndarray' object has no attribute 'plot'

and so on.
Issue #85

@fkazemian it looks like you've matched these PRs to the current live site. Because we have added an execution layer to the compilation chain - we are wanting to generate errors (during execution) to reduce the amount of output we are tracking. This will require careful consideration of the context -- and not just matched to the current live lecture.


For R, suggest using fread from data.table

Seeing how big this file is, I think data.table::fread('https://storage.googleapis.com/covid19-open-data/v2/main.csv', na.strings = c("", "NA", "#N/A")) may be more appropriate than read.csv for the R-documentation!

@demetrios1 sorry for the late reply. This sounds like a great suggestion! Admittedly I'm not an expert in R. If you are open to it, we'd absolute accept a pull request to improve the documentation.
Do you know if the data.table::fread() function is able to handle gzipped files?

hi @owahltinez .  Thanks for getting back!  I'm not sure if it can to be honest.

@a27cheung will be looking into this as part of a big documentation update coming soon :-)

Hi @demetrios1 - will get to this as soon! Apologies for the delay - we’re working on a presentation to a large group this week :)


Add the Arabic.ahk file

Basically just the opposite of the Franco.ahk

I'd love it if you could take a look at my Qamous project (https://github.com/anthonyyoussef01/Qamous)


PostgreSQL: auto-recover after "current transaction is aborted" error

In PostgreSQL in transactional mode any error leads to consequent errors "current transaction is aborted, commands ignored until end of transaction block", User must manually perform commit or rollback operation.
While in some enterprise applications it makes sense to perform a commit after error (e.g. save changes before last good savepoint) in DBeaver it doesn't make any sense.
So we can freely rollback current transaction.
We can make it automatically without even warning user. It will work only if "Connection auto-recovery option is enabled".

verified


Pin docplex version in CI

Summary
The latest docplex version which was released recently breaks the aqua
quadratic_program module. This commit pins the docplex version to
unblock CI, while this is reported upstream and fixed.
Details and comments

Perhaps pin to the previous version < 2.16

Yeah, that's a better idea. I typically like to pin in constraints to an exact version to make it more deterministic, but in this case that makes sense.

https://github.com/Qiskit/qiskit-aqua/pull/1420 is merged.


Is there any demos for multimodal editing using KE and FT Methods？

Thanks for the outstanding work! I am now reproducing your results, but I encountered a little problem when trying to edit multi-modal LLM with FT and KE. Is there a demo for the KE and FT methods?

Thank you for your interest in our work!
In our original code, we implemented FT and KE (referred to as EFK in MEND) based on the MEND repository. We plan to integrate both FT and KE into our project in the future.

Hi, do you have any further questions?

No, Thanks a lot!


Fix download links for ASP.NET Core 2.1.1

Made some fixes to the download links.

Update name in releases.json for the linux-arm (i.e. arm32v7) aspnetcore-runtime download
Add link for ASP.NET Core runtime binaries for linux-x64
Clarify the meaning of glibc/musl/ARM in the columns of the download table.

Also, how do I go about getting this to say arm32, not arm64? https://www.microsoft.com/net/download/linux

cref https://github.com/dotnet/core-setup/issues/3122#issuecomment-400957801

Ping @leecow

arm32 links are only on the download archive pages at the moment (https://github.com/dotnet/core/blob/master/release-notes/download-archives/2.1.1-download.md). If it's time to promote them to the marketing pages, we can make that addition.

The key problem is the missing download link for arm32 aspnet+dotnet runtime. In
https://github.com/dotnet/core-setup/issues/3122#issuecomment-401087300
@natemcmaster clarified that the 64bit download link is actually 32bit and I verified this on real arm32 hardware. It would be nice, if there is an official dotnet+aspnet arm32 download link, so we can integrate this link in our buildroot workflow.

:bell:

@natemcmaster Sorry for asking stupid questions! The PR is merged and the website looks different now... nevertheless I can't find the download link for

dotnet runtime
arm32
including ASP.Net runtime store

Side note: it seems the ASP.Net runtime is hidden even deeper now.
https://www.microsoft.com/net/download/dotnet-core/2.1

@nzain it looks like the website refactoring still has an error in it. The "arm64" link on the downloads page is incorrectly labeled. The link it downloads is actually arm32. I'll send an email to our website team and hopefully they will fix it. In the meantime, this is the link you're looking for. It contains the 2.1.2 ASP.NET Core runtime nad .NET Core runtime for arm32.

Note: in 2.1, ASP.NET is "shared runtime", not a runtime store anymore (subtle, but important distinction). So, you won't find links in 2.1 that are labeled "package runtime store."


Epic: HTTP API Testing Tools

As the investments we make into building out HTTP APIs for use in the Power Platform take shape, an interesting by-product is that we’re hearing from more developers that they need a way to call their APIs from clients hosted somewhere else. Consider the following scenarios in which developers are building HTTP APIs for use by either Azure or Power Platform clients:

A developer needs to create an Azure Function that, given an ID, gets an order from a database and sends an adaptive card to a Teams channel representing the products in the order. The Function will be called within a Logic App.
A citizen developer is building a mobile app to replace an internal supply chain implemented using an intranet site. The product database page will be replicated as an HTTP API that will be called from a PowerApp.

In both scenarios a developer needs to build a “client” for an API on which they’re also developing. In the Logic App scenario, a developer needs to be able to debug an HTTP API running on localhost but call that HTTP API from a Logic App running in Azure. In the PowerApp scenario, a citizen dev using the PowerApps maker experience – a web-based IDE – will be building and testing a app in a web browser.
In both scenarios, the “client” will need to call the API, and the API developer will need a way to receive those calls and to step through their code using the rich debugging tools present in the Visual Studio family of products. But in both scenarios, the “client” will need a tunnel to localhost, as both are running on the open internet.
Developers need an “F5 to Localhost Tunnel” experience to answer this problem.
User Stories under this Experience:

Closed as this was moved to another planning forum


Allow users to specify the spacing around block imports

A new preference option is added to the Scala > Editor preference page,
which allows users to configure if they want to see spaces around a
block import. The default configuration is that the spaces are added,
because this is the behavior of scalariform (whose behavior is hardcoded
and can't be changed).
Fix #1001692

@wpopielarski ?

LGTM


Replace homepage links with garden home pages

Okay this one's not as well thought out but bear with me.
A lot of times, I mention a person, technology or website, and I just link to thier homepage. It's nice to have some link or context, but it would be even nicer if I could eventually replace this with a page that describes my opinions/experiences/etc with that entity.
So for example, I started changing DistroKid links from being to https://distrokid.com/ and instead going to a page that I just called [[DistroKid]]. But then I didn't really immediately have anything to write about the service, so I created the page (distrokid.md) and just dumped the homepage link in there. A simple one hop, but probably annoying for visitors.
What would be great is if I could specify a "fallback" link for a [[topic]], so that if the page exists in the garden, it goes there first, but if it doesn't it falls back to the homepage or other link I provided.
Not sure what the syntax for this should be. Maybe something like [[DistroKid<https://distrokid.com>|on DistroKid]], where angle brackets in the wiki link indicate the fallback URL (and are stripped from the generated HTML).

I should also do this for links in old blog posts that are just Wikipedia links. Like, I'm dying to tell you more about this topic, dear reader, but the best I could think to link you to is the resource you would most easily find yourself. 😦


stop, start and rm

would be nice to have keys to stop, start and remove container (this could bring some modal 'are you sure' window)

Wouldn't that be hard to make intuitively? Start, Stop and Sort all compete for the "s" key...
As for the implementation, the dockerclient Client has StartContainer, StopContainer and RemoveContainer functions, so it should be possible...

Maybe to have some options dialog, like press 'o' and you get dropdown where you navigate with arrows, or press esc to exit from that menu

Container "actions" dialog would certainly be preferable here -- https://github.com/bcicen/ctop/issues/28 mentions this as well.

cool, sorry didn't see that one, you can close mine as duplicate then. Would you mind if I start looking at implementing this?

@pete911 that would be great.
closing this duplicate now.


remove ruby-prof testing

removes the testing since we're not shipping this any more.

Dupe of https://github.com/chef/chef/pull/11493

i could swear i checked

(i was filtering PRs by author to only mine)


Feature Request: Treat Slots as Sets only internally

Currently, slots in the WebUI directly reflect the behavior of Python Sets which is shown by the fact that the order and duplicates are discarded. This makes slot management quite inconvenient since alphabetical or other sortings the user might want to keep, just disappear.
I don't know how much effort this would take and if this small improvement is worth the time, but if so treating the Slots only internally as Sets and propagating them as cached List to the user might be a reasonable solution.

This sounds good to me. I'll see what I can do.


Analysing the performance of different methods to get windows

I've started looking at the performance of various ways of getting windows:
1-  MNE: epochs.get_data(ind)[0] with lazy loading (preload=False)
2- MNE: epochs.get_data(ind)[0] with eager loading (preload=True)
3- MNE: direct access to the internal numpy array with epochs._data[index] (requires eager loading)
4- HDF5: using h5py (lazy loading)
The script that I used to run the comparison is here:
https://github.com/hubertjb/braindecode/blob/profiling-mne-epochs/test/others/profiling_mne_epochs.py
Also, I ran the comparison on a single CPU using:
>>> taskset -c 0 python profiling_mne_epochs.py
Here's the resulting figure, where the x-axis is the number of time samples in the continuous recording:

For the moment, it looks like:
1- ._data[index] is unsurprisingly the fastest, however it requires to load the entire data into memory.
2- hdf5 is very close, with around 0.5 ms per loop, which is great knowing it's able to only load one window at a time.
3- get_data(index) is much slower, but this is expected as we know it creates a new mne.Epochs object every time it's called. Also, the gap between preload=True and preload=False is about 1.5 ms, which might be OK. The main issue though seems to be the linear increase of execution time as the continuous data gets bigger and bigger.
Next steps
Considering the benefits of using MNE for handling the EEG data inside the Dataset classes, I think it would be important to dive deeper into the inner workings of get_data() to see whether simple changes could make this more efficient. I can do some actual profiling on that. What do you think @agramfort @robintibor @gemeinl ?
Note: I haven't included the extraction of labels in this test.

Thanks @larsoner, very elegant solution! I think this should satisfy our requirements. I have one question:
I understand that you need to call epochs.drop_bad() before calling epochs.get_data(item=idx). However, if I understand correctly, drop_bad starts by loading the entire data in memory. If that's the case, then this defeats our purpose here as we want to avoid eager loading. If drop_bad() only drops epochs based on signal quality criteria, could we just "accept" any epoch, and in doing so bypass the use of drop_bad()?

drop_bad starts by loading the entire data in memory

Not quite. It loads each epoch one at a time and then discards it from memory (does not keep it around). Is that okay?

If drop_bad() only drops epochs based on signal quality criteria, could we just "accept" any epoch, and in doing so bypass the use of drop_bad()?

It's also based on things like whether or not the entire epoch is actually extractable from raw (e.g., a 10 second epoch from an event occurring one second from the end of the raw instance would be discarded as TOO_SHORT or so.
If it's desirable to just check these limits without actually loading the data we might be able to, but it again would be a larger refactoring.

drop_bad starts by loading the entire data in memory

Not quite. It loads each epoch one at a time and then discards it from memory (does not keep it around). Is that okay?

Ok, got it. So in our case, I guess we would run drop_bad() only once, before training. This would load all data so might take some time on very large datasets, however since it's done sequentially memory would not be an issue.
I think this should be fine for now. If calling drop_bad() once before training ends up taking too much time we might want to look into other options though.

yes I agree with @hubertjb that case (loading into and dropping from memory once at start of training) may be fine for now for us, we will see if it causes any practically relevant problems once we have more implemented full examples.

Okay great, let us know of it does indeed end up being a bottleneck

I started working on an example that shows how to do lazy vs eager loading on real data, and that compares the running times of both approaches (https://github.com/hubertjb/braindecode/blob/lazy-loading-example/examples/plot_lazy_vs_eager_loading.py). It's still not a perfectly realistic scenario - for instance, there is no preprocessing/transforms applied - but it should be better than my previous tests. Here are some preliminary results (they are subject to change)!
So far, using the first 10 subjects of the TUH Abnormal dataset, and training a shallow net for 5 epochs on a normal/abnormal classification task, the breakdown is (average of 15 runs, in seconds):
              data_preparation  training_setup  model_training
loading_type
eager             6.539236        0.003837       18.746278
lazy              9.040427        0.136949       59.257529

Observations:

Data preparation is longer for lazy loading, most likely because of drop_data which loads each window one at a time. This is OK for now but we might want to optimize eventually.
I'm not sure why training setup time is larger for lazy loading, but in the end this is pretty short anyways. To investigate.
Model training time is >3x longer with lazy loading.

Next steps:

Look into the num_workers argument of Dataloader to see whether that can help mitigate the overhead of lazy loading
Once we agree on an API for on-the-fly transforms (#72), we can add e.g. a filtering step and see what impact it has on running times

Great @hubertjb . Seems we are getting to a reasonable training time range. Would also be interesting how big the difference is for Deep4. And as you said, maybe num_workers would already close the gap enough to consider it finished. I would say a gap of 1.5x for deep4 to me is acceptable.

Some good news, using num_workers makes lazy loading a lot more competitive (https://github.com/hubertjb/braindecode/blob/lazy-loading-example/examples/plot_lazy_vs_eager_loading.py).
Without (num_workers=0):
              data_preparation training_setup  model_training
loading_type
eager             8.866640        0.003532       25.106193
lazy             12.615696        0.418142       72.939527

With (num_workers=8):
	      data_preparation  training_setup  model_training
loading_type
eager              8.82214        0.004220       28.750993
lazy              23.60280        0.422488       30.715090

The two types of loading are much closer now. Also I noticed GPU usage stayed at 100% during training, which means the workers did what they are supposed to do.
Also, @robintibor, I tried with Deep4, keeping all other parameters equal:
              data_preparation  training_setup  model_training
loading_type
eager             8.902757        0.022913       11.306941
lazy             22.113035        0.456256       17.751904

Surprisingly, it seems more efficient than the Shallow net... Is that expected? I haven't played with the arguments much, maybe I did something weird.

No the behavior is very unexpected, time spent on GPU forward/backward pass should be longer for deep, therefore difference should be smaller. It is also very strange that for you deep is faster than shallow, that should not be.
Before investigating further, please add following line somewhere before your main loop:
torch.backends.cudnn.benchmark = True

(see https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936 for why for our cases it should always be set to True)
Numbers should improve overall. However it does not explain any of the current behavior, as said deep should be slower than shallow, not ~3 times faster (in eager mode)

Latest results can be found in #75.

@hubertjb I have rerun your perf code after the fix in https://github.com/braindecode/braindecode/pull/89 (which improves preload=True performance quite a lot) for batch_size=64 and cuda=True. This is results:

preload
len
model
n_work
data_prep
setup
training

0
False
2
deep
0
11.9
0.1
21.7

12
True
2
deep
0
7.5
0.1
2.6

1
False
2
deep
8
11
0.1
11.4

13
True
2
deep
8
7.6
0
3.5

4
False
4
deep
0
16.8
0.2
29.5

16
True
4
deep
0
7.8
0.1
3.3

5
False
4
deep
8
15.4
0.1
13.2

17
True
4
deep
8
7.9
0.1
4.6

8
False
15
deep
0
28.2
0.2
57.1

20
True
15
deep
0
10
0.1
8.9

9
False
15
deep
8
31.5
0.2
25.8

21
True
15
deep
8
9.8
0.1
11.4

2
False
2
shallow
0
10.5
0.1
24

14
True
2
shallow
0
7.5
0
2.2

3
False
2
shallow
8
12.8
0.1
10.8

15
True
2
shallow
8
7.5
0.1
3.6

6
False
4
shallow
0
10.5
0.1
29.2

18
True
4
shallow
0
7.8
0
3.3

7
False
4
shallow
8
14.8
0.1
13.3

19
True
4
shallow
8
7.9
0.1
5.3

10
False
15
shallow
0
31.2
0.2
53.9

22
True
15
shallow
0
10
0.1
12.3

11
False
15
shallow
8
27.2
0.2
26.6

23
True
15
shallow
8
9.9
0.1
15.1

As can be seen for eager num_workers=0 is better whereas for lazy num_workers=8 is better.
Therefore I made a comparison of these two settings:

win_len_s
model_kind
train_lazy
train_eager
ratio

0
2
deep
11.4
2.6
4.4

1
4
deep
13.2
3.3
4

2
15
deep
25.8
8.9
2.9

3
2
shallow
10.8
2.2
4.9

4
4
shallow
13.3
3.3
4

5
15
shallow
26.6
12.3
2.2

I don't know for these win_len settings which one is a realistic one? What do they correspond to? Would be great if  we can get all of the ratios to <2 at some point, but good that we have a running implementation in any case

outdated


Error parsing clippings file. Error: File already exists

I decided to re-import my kindle highlights with a new template that I designed with Dataview in mind for the YAML front-matter. But after deleting the previous nodes created (all of them), I'm told that after attempting to import the clippings file again, the 'file already exists'. I can't think of any other local files that may be affecting this.

I'm told that after attempting to import the clippings file again, the 'file already exists'

How are you told that? In a pop-up?
Can you turn on your Developers tools, clear it, and then run the process again. If you share any error messages that you might be seeing it will help me troubleshoot your issue.

This is the pop-up, and the developer console results

path="Books\BookHighlights\.md looks suspiciously dodgy to me.
Did you customised the File Name template field? It is possible that the field does not have any unique field variables there (e.g. book title) and hence all your highlights are being overwritten to the one file.
Can you send me a screenshot/copy of your templates?

Yeah it was the template formatting I had used, I've reverted it to the default and everything works fine now.

Thanks for raising this.
I'll rename and keep this issue open. I think it would be good to add some safeguards to either prevent a bad template name to be used, or to show a more descriptive error message to the user.

@adampick99  I haven't been able to recreate this issue at my end. Do you mind if you share the template you had which created this problem?


"Fix" PG search results design

Initial change implemented:

It is possible to open the playground inline and to open it externally. You can also click on the tokens (tags) to search for other playgrouns with the same tags.

Far better :D


Toltec rm2fb version

How can I tell which rm2fb version is on Toltec? It shows that it has version 1.0.0-7, and I'm just wondering when I can tell that this has been updated to the newest version v0.0.3. I'm looking forward to the new PR that it includes as well as just future reference as newer versions are released. Thank you!

It was packaged before rm2fb had version numbers so it was created with 1.0.0
According to https://github.com/toltec-dev/toltec/blob/stable/package/rm2fb/package#L16 it's being pulled from cd2766864c7985fa50cdf85f4ae0a411d7ca4e56

we will bump this repo's version to catch up to toltec's version (so we will probably turn v0.0.3 -> v1.0.1) and then we will stay aligned from there

Ah ok, thanks so much! Very excited about this development, great work!

Also I'm going to go ahead and close this as my question was resolved, hopefully others can find it if they have similar questions. Thanks again

actually, this is more work than just bumping and matching toltec since there are files with the name '1.0.0' in the build. i am going to delete release v1.0.1 and look into it more later

Was this resolved with #54 ? If so, could we tag a new release and update toltec?

Was this resolved with #54 ? If so, could we tag a new release and update toltec?

yes, will take care of it within a few days hopefully

we are also dealing with the new xochitl 2.6 update, so maybe we will do both at same time. (expect it in next few days, still)

this is new stable now


LESS files not imported correctly.

I just did a Bower install of remarkable-bootstrap-notify#3.1.3. I'm using WebStorm as my IDE and the LESS compiler blew up immediately in WebStorm upon installing the project. I added
@import "mixins/responsive-visibility";
@import "variables";
on line 25 of responsive-utilities.less and that seems to have fixed the errors.

I think I need to add more clarification to this. I know that you don't have the file responsive-utilities.less in your plugin. However, when I Bower installed your plugin it caused compilation errors in my bootstrap/less folder, which I also installed via Bower. What caused this I'm not sure but my statement above is how I fixed it. I was hoping you would know what caused my bootstrap LESS file to suddenly have compilation errors by reporting this.

I am sorry, but I do not use LESS and am unsure what caused the issue. I am busy with 4th of the July weekend coming up, but I can ask some of my friends who are more familiar with LESS and see if they have any idea. Basically, I'll be looking into this next week.


Builder does not support all postgres operators

It is not possible to use the IS or IS NOT operators for example in the whereRelation() function.
The problem is that theIlluminate\Database\Query\Builder does not support these operators.
It would work if these operators were included in this array.
https://github.com/laravel/framework/blob/9.x/src/Illuminate/Database/Query/Builder.php#L199-L206
And the whereRelation() calls the where() function inside the builder, which changes unsupported operators to =.
https://github.com/laravel/framework/blob/9.x/src/Illuminate/Database/Query/Builder.php#L739-L741

Heya, thanks for submitting this.
This seems like a feature request or an improvement so I'm converting this to a discussion instead. It's best to post these in the discussions section in the future to get support for your idea. After that you may send a PR to the framework. Please only use the laravel/framework issue tracker to report bugs and issues with the framework.
Thanks!


Ppts model

This was created via python manage.py startproject ppts, then the model file was modified to mostly match @tristanlmiller's schema.

Several variables are named "type", which is a keyword in python.  Is that a problem?


Wrong destination in kafka bus

using the following cloud bus components
    compile('org.springframework.cloud:spring-cloud-starter-config:2.0.0.BUILD-SNAPSHOT')
    compile('org.springframework.cloud:spring-cloud-starter-bus-kafka:2.0.0.BUILD-SNAPSHOT')

when using the config server we found out that the writes were happening to the springCloudBusOutput topic while the client was reading from springCloudBus by default.
I could only overcome the issue by setting the bus explicitly in the client destination:
spring:
  profiles:
    active: local
  application:
    name: ui-app
  cloud:
    bus:
      enabled: true
      destination: springCloudBusOutput

Is this a bug or just a bad configuration?

@mbogoevici any ideas?

@spencergibb  I suspect that we see a repeat of the issues around https://github.com/spring-cloud/spring-cloud-bus/blob/master/spring-cloud-bus/src/main/java/org/springframework/cloud/bus/BusAutoConfiguration.java#L93, now due to some changes in Spring Boot 2.0.  Wondering if that part should change in Bus as well. Will check if there's something that broke on the Stream side.

👍


AWS SDK for JavaScript v3

(node:54701) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.

Please migrate your code to use AWS SDK for JavaScript (v3).
For more information, check the migration guide at https://a.co/7PzMCcy

https://github.com/aws/aws-sdk-js-v3


server: standardize on naming

Fix https://github.com/kubernetes-incubator/cri-o/issues/617
This patch rewrites the way we generate container/pod names following what upstream kube does for the dockershim. Including the sandbox UID is also fixing potential naming collisions as we trust k8s.
@alexlarsson @mrunalp PTAL
Signed-off-by: Antonio Murdaca<EMAIL_ADDRESS>

LGTM

LGTM, assuming test finish happily.  BTB, any plans to adjust/add tests for this in a follow up?

lint and make failures in build.

lint and make failures in build.

should be fixed now

Looks good.

damn travis...

LGTM

This breaks the seccomp tests. It seems like these hardcode the container name:
        sed -e 's/%VALUE%/,"security\.alpha\.kubernetes\.io\/seccomp\/container\/redhat\.test\.crio-seccomp2-1-testname2-0": "runtime\/default"/g' "$TESTDATA"/sandbox_config_seccomp.json > "$TESTDIR"/seccomp2.json

So, we probably have to fix the test

This breaks the seccomp tests. It seems like these hardcode the container name:

yeah, that's awful, fixed now

all good here now

@runcom What happened to tests on Fedora in the last run?

@runcom What happened to tests on Fedora in the last run?

just a failure in vm provision, not related to this pr

LGTM


Receive Form Data on Email

Working Web Application where Form data filled is sent to our email address.

Please assign me the issue under Hacktoberfest 2022. It is a very useful functionality.

the issue has been assigned to @Gaurika-Gupta. Don't forget to add a readme.md in your project's folder.


Topic subscription keepalive

Been noticing dropouts in mirror subscriptions if no transactions are sent for a few minutes, restarting the client fixes the issue.
The error handler in the JS client wasn't triggered when the connection was lost (mirror logs show the connection was cancelled suggesting it was closed by the client).
The Java SDK has a keep-alive, should/is the JS sdk doing the same ?

It's not. It should be.

This is fixed on master (6b1150c395a259378247313a5a4a5c08398a5e15)


trailing missing value unrecognized

First thanks for this wonderful library. It's very fast comparing to other csv parsers.
The issue is when the reader parses the following line (note that this is the last line of the file), column size will be 4. The parser failed to find the trailing missing column:
ABC,123,123,123,\0
If there's an additional line, column size will be 5 which is correct:
ABC,123,123,123,\n\0

Nice finding. I am working on this bug.


No "author" tab on exhibit. Fix #225.

@afred?

rebased. :+1: from me. merge after Travis build passes.


Create a feature/body branch

Create a new feature/body branch (based on the develop branch)

Go to the develop branch and pull the changes - git pull

git checkout -b feature/body - create and switch to the feature/body branch


[Feature Request] Setting for disabling "hover to reveal"

Can you perhaps add a setting to disable "hover to reveal"?

Hello,
I am no longer working on this project and am unlikely to return to it; it is pretty much abandoned. But the code is open-source and distributed under the MIT license, so anyone who wants to fix or improve anything is free to fork the project and do pretty much whatever they'd like.

This feature is now available in Version 0.7.0 thanks to the work of @0xorial .


Expose 'Reimport' on right-click context menu in the FileSystem panel

Implement proposal: #6320
Bugsquad edit:

Closes https://github.com/godotengine/godot-proposals/issues/6320

_reimport_attempt() for multiple files will reimport all files with the same type as the first selected file. You can e.g. select texture and audio file and clicking Reimport will trigger the warning about changing import type. If you accept, it will corrupt the imported files.
Either the function needs to be changed or _reimport() shouldn't appear if there is more than 1 type of resources selected.

I understand. I tested the import tab in the current version and found that when I try to import two different file types, the editor doesn't let me reimport them. To maintain consistency, I'll try to implement the same functionality as the editor.

The implementation can be much simpler:
bool resource_valid = true;
String main_extension;

for (int i = 0; i != p_paths.size(); ++i) {
	String extension = p_paths[i].get_extension();
	if (extension_list.has(extension)) {
		if (main_extension.is_empty()) {
			main_extension = extension
		} else if (extension != main_extension) {
			resource_valid = false;
			break;
		}
	} else {
		resource_valid = false;
		break;
	}
}

Hey, thanks for implementing this. just tried this in the 4.2 dev3 snapshot. It seems like this doesn't work for all types?  For instance, when using the https://github.com/Kiamo2/YATI plugin, .tmj files  don't show Reimport

This was actually the use case that I created the proposal for, to reimport my tilemaps easily. Another plugin I'm using, that I wrote myself, does have the Reimport option

Any idea why it wouldn't show up for some resources?

Maybe there is something wrong with the YATI implementation, like missing function or something. Doesn't it print any errors when used? Can you show the code of your plugin?

I don't think I have errors from YATI but I can check later. This is my importer code which does show Reimport:
https://github.com/dogboydog/YarnDonut/blob/godot4/addons/YarnDonut/Editor/YarnImporter.cs

YATI returns "Node2D" in GetResourceType(), while the method expects Resource. I think it should be PackedScene in this case.

Thanks, I'll let the developer know


Is it possible to determine what the user's input address is?

Thanks!
Is it possible to determine what the user's input address is?
such as IPv4, IPv6, network,  interface...

Each class has a tryParse method.
If parsing fails, it returns null, allowing you to determine the result by calling tryParse in each class.


Made the Website Responsive in Desktop Version

Title and Issue number

Title : Made the Website Responsive in Desktop Version
Issue No. : #178
Code Stack :  HTML , CSS , JS
Close #178

Video/Screenshots (mandatory)

https://github.com/apu52/Travel_Website/assets/124901633/000b274c-e5bd-44c8-a4fa-de45a8969475
Checklist:

[X] I have mentioned the issue number in my Pull Request.
[X] I have commented my code, particularly in hard-to-understand areas
[X] I have gone through the  contributing.md file before contributing

Additional context (Mandatory )
Are you contributing under any Open-source programme?

Yes , in JWOC 2024.

@apu52 I have resolved the Conflicts. Please review it now

@SohanRC But I am giving you 2 more days to make better contributions on this issue. Hope to see your valuable contribution.

@SohanRC This maybe a great contribution. But I can't really want this type of bg changes in my website . So , sorry this PR can't me merged.

But you said that you wanted that animation

@apu52 You said to resolve the merge contributions so that you can merge it
I am still fixing the About us section

@apu52 You said to resolve the merge contributions so that you can merge it I am still fixing the About us section

ok look @SohanRC I have mentioned that is was a demo. you can make more farther good UI changes in the repo on your preference. the last UI change was not up to the mark so that I have closed the PR


Using intersection observer instead of throttling resize/scroll events

I used this package in a project, liked it and thought it could be made even simpler by removing lodash.throttle dependency and using IntersectionObserver api.
From my testing I don't think I broke it functionally. One downside is that IntersectionObserver is not implemented on IE, is this a deal-breaker for this PR?
Also, could you provide some more information about the "edge case" in componentDidUpdate?
Wouldn't it make sense to avoid loading more information if there is no scroll at all?
I would like to write a test for this situation to make sure how it's supposed to work.

Very interested in this getting merged.


Idzorek's method for Black-Litterman

Idzorek's method is a way of translating percentage confidence into the uncertainty matrix required by Black-Litterman. This can be seen in section 3.2 (pg 23) of Idzorek's Step-by-Step Guide to Black-Litterman.
This has been requested in #70, and selfishly, I now think it would be useful for my own investing :)
That being said, Idzorek's method is really quite involved, and I would thus like to open the floor to any suggestions regarding implementation/architecture.
A brief overview of the steps are:

Construct a BL returns vector for each view separately (assume 100% confidence)
Convert these return vectors to implied weights for each view, w_k
Determine the deviation between each view's weight vector and the market weights
Multiply this deviation by the user's confidence to get the tilt_k, and compute target weights for each view using w_k%= w_mkt + tilt_k
The kth uncertainty is the uncertainty that minimises the sum of square difference between w_k and w_k%.
Put these k uncertainties on the diagonal of a matrix.

I think I've figured out how to do it on the backend. Have a static method within the BlackLitterman class (it must be static so that we can instantiate BlackLitterman objects) that computes the Idzorek matrix.
I'm not quite sure what the API should be. Some ideas:
# 1. Recognise omega parameter as a string then automatically construct
bl = BlackLittermanModel(S, absolute_views=viewdict, omega="idzorek")
ret = bl.bl_returns()

# 2. Method to replace current omega with Idzorek omega
bl = BlackLittermanModel(S, absolute_views=viewdict)
bl.omega = bl.idzorek_omega()
# or
bl.set_idzorek_omega()
ret = bl.bl_returns()

# 3. Put function in module rather than class
idzorek_omega = black_litterman.idzorek_uncertainty(absolute_views=viewdict)
bl = BlackLittermanModel(S, absolute_views=viewdict, omega=idzorek_omega)

I think option 1 is the nicest API, but I don't know if it's too confusing to allow the omega parameter in the constructor to accept a string, in addition to the pd.DataFrame and None options currently provided.
@schneiderfelipe any thoughts?

@schneiderfelipe
I've made decent progress so far, and have written the code for steps 1-4. However, I'm having a bit of trouble with step 5:

I think w_k represents the Black-Litterman weights for a single view of uncertainty omega_k (this uncertainty is what we are trying to solve for. Do you have any thoughts on how I should solve this optimisation problem?

@robertmartin8 w_k is a number, right? As such, the equality after "where" in the figure looks strange. Correct me if I'm wrong, but aren't numbers being summed to matrices in there?
With that being said, everything is linear there, so we could transform this optimization problem in a linear one. On the other hand, this transformation is hard. It basically means doing the derivatives, and equating to zero. The problem is the large inverse in the middle of the expression, but this can be rewritten by using this lemma. This and explicitly stating all matrix multiplications will probably do the trick (the good thing is that a lot of stuff will not contribute to the derivative; Pi, for example, obviously won't appear in the derivative or w_k). I tried to sketch something, but it was no good. I'll try it again when I have time. But it might not be worth it.
For the time being, I would say to use scipy.optimize, especially for getting tests right, just to be sure. What do you think, is it worth it?

@schneiderfelipe w_k is a vector of weights, omega_k is a number. In this case, p_k is the 1xN row vector (a single row of the picking matrix). So we are actually adding NxN to NxN matrices in the middle bracket. Then the three brackets together become w_k = (N x N)(N x N)(N x 1) so w_k is (Nx1) as needed.
I'll have a go at the maths – I'm happy to let cvxpy solve it if possible (and it should be possible since the problem is convex?). scipy.optimize is the worst case.

@schneiderfelipe I've worked on this a bit, and I'm not certain it's linear. The sum of squares is quadratic in w_k, and w_k is quadratic in 1/omega_k (because even after using the lemma, there is a 1/omega_k in the middle bracket and another in the last bracket). So actually we end up having to minimise a quartic function. I've attached my working – let me know if you spot any errors (there are bound to be a few)
IdzorekMethod.pdf

This is an implementation using scipy.optimize
def sum_square_deviations(s, w_target, delta, sigma, p, q, pi):
    tau_sigma_inv = np.linalg.inv(tau * sigma)
    middle_bracket = tau_sigma_inv + s * p.T @ p
    last_bracket = tau_sigma_inv @ pi + s * p.T @ q

    w_k = (
        np.linalg.inv(delta * sigma)
        @ np.linalg.inv(middle_bracket)
        @ last_bracket
    )
    return np.sum((w_target - w_k) ** 2)

result = sco.minimize(
    sum_square_deviations,
    50,
    args=(w_target, risk_aversion, cov_matrix, P_view, Q_view, pi),
    bounds=[(0, None)],
)

s_k = result["x"]
omega_k = 1 / s_k

@schneiderfelipe w_k is a vector of weights, omega_k is a number. In this case, p_k is the 1xN row vector (a single row of the picking matrix). So we are adding NxN to NxN matrices in the middle bracket. Then the three brackets together become w_k = (N x N)(N x N)(N x 1) so w_k is (Nx1) as needed.

Right, p_k is a row vector :smile:.

@schneiderfelipe I've worked on this a bit, and I'm not certain it's linear. The sum of squares is quadratic in w_k, and w_k is quadratic in 1/omega_k (because even after using the lemma, there is a 1/omega_k in the middle bracket and another in the last bracket). So actually we end up having to minimise a quartic function. I've attached my working – let me know if you spot any errors (there are bound to be a few)
IdzorekMethod.pdf

Excellent, I haven't found any error. I thought the final system could the linear, but due to the inverses, that was asking too much. It isn't convex either since the derivative is unbound, right?

They may not be zero but are admittedly small. It certainly has to do with Pi and Sigma. Can you run a toy example with fictitious values for Pi and Sigma (like constant prior and identity)? And what happens with zero confidences?

@schneiderfelipe Good idea:
# Identity covariance
S = pd.DataFrame(np.diag(np.ones((5,))), index=range(5), columns=range(5))
# Constant view of 0.3 return
views = {k: 0.3 for k in range(5)}
# Prior
pi = pd.Series(0.1, index=range(5))

# Perfect confidence - should equal views
bl = BlackLittermanModel(
    S, pi=pi, absolute_views=views, omega=np.diag(np.zeros(5))  # perfect confidence
)
pd.testing.assert_series_equal(bl.bl_returns(), pd.Series([0.3] * 5))

# No confidence - should equal priors
bl = BlackLittermanModel(S, pi=pi, absolute_views=views, omega=S * 1e6)
pd.testing.assert_series_equal(bl.bl_returns(), pi)

It's reassuring to see that the current BL functions as intended 😅. Now for Idzorek:
    # Idzorek perfect confidence
    bl = BlackLittermanModel(
        S, pi=pi, absolute_views=views, omega="idzorek", view_confidences=[1] * 5
    )
    # All entries are the same so we only need to look at one of them
    print(bl.omega[0,0])   # 0.0454
    print(bl.bl_returns()[0])  #0.205

    # Idzorek zero confidence
    bl = BlackLittermanModel(
        S, pi=pi, absolute_views=views, omega="idzorek", view_confidences=[0] * 5
    )
    print(bl.omega[0,0])  # 0.186
    print(bl.bl_returns()[0])  # 0.142

So we can see that it is scaling the correct way – lower confidence means higher uncertainty and closer to the prior. I verified this by looping over confidence values between 0 and 1, and it agrees with intuition. The only problem is that the extreme values shown above are not equal to the true zero/perfect confidence cases. I'm going to go through the Idzorek paper again to see if this is expected or not

Excellent! I have a gut feeling that Idzorek's method sort of "breaks the ties" with information from priors, what do you think? A test for this would be to change Pi to a constant value but keep the real Sigma.

On the plus side, I've checked that the code does give the right answer (i.e a divide by zero error) when we pass zero confidence. In this case, omega_k should be infinity. I guess I will catch this error and return 1e6 (huge uncertainty)?
But unfortunately I can't seem to figure out whether or not a confidence of 100% is supposed to result in omega=0 using Idzorek's method. I guess I could put in a special handler: if the confidence for a view is 1, set that omega to zero. But then the confidence won't be "linear", because there will be a big drop in omega from 0.99 confidence to 1.00 confidence.
Anyway, I ran some more tests and think there's a bug somewhere in my implementation:
    df = get_data()
    S = risk_models.sample_cov(df[["GOOG", "AAPL", "FB", "BABA"]])
    viewdict = {"GOOG": 0.40, "AAPL": 0.40, "FB": 0.40, "BABA": 0.40}
    bl = BlackLittermanModel(
        S,
        pi=np.array([0.1] * 4),
        absolute_views=viewdict,
        omega="idzorek",
        view_confidences=[1, 1, 1, 1],
    )
    print(bl.omega)
    print(bl.bl_returns())

[[0.0495849  0.         0.         0.        ]
 [0.         0.92239836 0.         0.        ]
 [0.         0.         0.09367556 0.        ]
 [0.         0.         0.         0.05108767]]

GOOG    0.137354
AAPL    0.124426
FB      0.135541
BABA    0.138346

For reference, the covariance matrix is:
          GOOG      AAPL        FB      BABA
GOOG  0.093211  0.046202  0.030801  0.029793
AAPL  0.046202  0.207537  0.021141  0.023839
FB    0.030801  0.021141  0.137201  0.030073
BABA  0.029793  0.023839  0.030073  0.100958

The uncertainty matrix is pretty reasonable except for the value of AAPL. It does have the highest variance of the group, but certainly not enough to justify the 10x higher uncertainty in the view.
The returns seem to have the correct pattern (i.e higher uncertainty means closer to prior), but they are all wayyy too close to the prior (this is 100% confidence!)
I'll try to look it over with fresh eyes this weekend

@schneiderfelipe I've been thinking of a different way to convert percentage confidence into omega. Let me know if it makes any sense.
In addition to providing the view, we get the user to provide a symmetric error bar and a percentage confidence. e.g I expect AAPL to return 50%, and I am 95% sure that this return will be between 20% and 80%. With this, assuming a normal distribution, you can calculate the standard deviation. In this case, 95% corresponds to 2 sigma, so the standard deviation of the view is 0.15 and the diagonal entry of omega is the square of that.
I'm still trying to get Idzorek's method to work, but I personally prefer the method I've just described (and I'd use that over Idzorek for my investments)

@schneiderfelipe
I've just read Walter's Black-Litterman Model in Detail paper and it's basically solved all my problems. He gives a closed-form solution for Idzorek's method on page 30.
Additionally, the method I described above regarding standard deviations is described on page 15 😂 .
Note to self: if you ever have a question about Black-Litterman, the first place to look is Walter's paper.

This is essentially all there is to it 🤦
view_omegas = []
for view_idx in range(len(Q)):
    conf = view_confidences[view_idx]
    P_view = P[view_idx].reshape(1, -1)
    alpha = (1 - conf) / conf  # formula (44)
    omega = tau * alpha * P_view @ cov_matrix @ P_view.T  # formula (41)
    view_omegas.append(omega.item())

return np.diag(view_omegas)

It's got some very nice theoretical features. With an identity covariance, and constant prior/views, the confidence results in a linear scaling between prior returns and view returns as we go from 0 to 1.

@schneiderfelipe I've been thinking of a different way to convert percentage confidence into omega. Let me know if it makes any sense.
In addition to providing the view, we get the user to provide a symmetric error bar and a percentage confidence. e.g I expect AAPL to return 50%, and I am 95% sure that this return will be between 20% and 80%. With this, assuming a normal distribution, you can calculate the standard deviation. In this case, 95% corresponds to 2 sigma, so the standard deviation of the view is 0.15 and the diagonal entry of omega is the square of that.
I'm still trying to get Idzorek's method to work, but I personally prefer the method I've just described (and I'd use that over Idzorek for my investments).

Your idea is nice! I think it is a nice companion to Idzorek.


Set some defaults based on libcurl

Enable SSL_OP_ALL by default which apparently enables a number of useful bug
workarounds (claimed by libcurl)
Frob a few bits here afterwards to disable some insecure options.
Use the cipher list from libcurl to enable/disable ciphers

Ok, @sfackler I've updated where both schannel and security-framework disable SSL v2/v3

Thanks!


Why doesn't VCR throw an exception if requests on cassette are NOT played back

Shouldn't it throw an exception if the component under test does not make the expected HTTP requests?

I'm going to create a PR that allows eject to take a parameter to control this behavior. If specified, eject will throw an exception the the cassette wasn't played back in its entirety.
Is there a reason why this is a bad idea?

#160


fix: respect falsy custom options

Hello!
It seems like since 1.0.0 falsy custom options have stopped being applied. I added a test that fails without suggested changes.
I found it after updating to<EMAIL_ADDRESS>and an investigation has led me here. In my project I set setDisplayName to false, but its not working anymore.

bump?