Fail strategies: [<HWTool.IPMI: 'ipmi'>] on focal on Dell PowerEdge R650xs

On Dell PowerEdge R650xs running on focal:
unit-hardware-observer-0: 15:43:07 WARNING unit.hardware-observer/0.juju-log Strategy <hw_tools.IPMIStrategy object at 0x7f8268f5ad60> install fail: Could not install package(s) [['freeipmi-tools=1.6.9-2~bpo20.04.1']]: None
unit-hardware-observer-0: 15:43:07 INFO unit.hardware-observer/0.juju-log Fail strategies: [<HWTool.IPMI: 'ipmi'>]

Charm is blocked, all IPMI checks are failing.
Revision 22 of hardware-observer.

I ran apt install freeipmi-tools on affected machines manually without any issues, the charm stayed in the blocked state but at least IPMIDCMICommandFailed alerts got resolved.

What is the package version you manually install?

@jneo8 1.6.4-3ubuntu1.1 gets installed.
Trying to manually install the same version as the charm results in:
$ sudo apt install freeipmi-tools=1.6.9-2~bpo20.04.1
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 freeipmi-tools : Depends: freeipmi-common (= 1.6.9-2~bpo20.04.1) but 1.6.4-3ubuntu1.1 is to be installed
E: Unable to correct problems, you have held broken packages.

If it helps, the "bad" package comes from focal-backports:
$ sudo apt-cache policy freeipmi-tools
freeipmi-tools:
  Installed: 1.6.4-3ubuntu1.1
  Candidate: 1.6.4-3ubuntu1.1
  Version table:
     1.6.9-2~bpo20.04.1 100
        100 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages
 *** 1.6.4-3ubuntu1.1 500
        500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1.6.4-3ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages


Structure for selectively giving access to new features

As we roll out new features, we need to be able to selectively enable them at a center/region level. Right now that's being done with explicitly in the code.
Come up with a way to do this in a easier and more consistent way.

We could do this we feature flags attached to centers. New features are given a new feature flag, and users are gated from using the new feature based on their center having the flag. As we add centers/regions, we just give the centers the new flag.

this is complete.


fix(config): handle webpack and rollup config env var

Short description of what this resolves:
If the user has provided both a webpack and a rollup config env variable this will look to the ionic_bundler variable to decide which bundler to use.
Changes proposed in this pull request:

add an extra check for if the user has provided both a webpack and rollup config env variable

Fixes: #268

Thanks @danbucholtz !


Text update for Local PV hostpath

[x] The NodeAffinityLabel need to be removed from Create Storage Class sub-section and needs to be added as an advanced option.
[x] Review and test the page content.

Link- https://docs.openebs.io/docs/next/uglocalpv-hostpath.html

Removed the NodeAffinity parameter from Storage class creation  YAML and added it as an optional section  with a sample YAML with NodeAffinity parameter.


Error: socket hang up don't return 1

Summary
When running a force:source:deploy in a CI context, some error can occured
ERROR:  Error: socket hang up Unhandled rejection Error: socket hang up     at createHangUpError (_http_client.js:323:15)     at TLSSocket.socketOnEnd (_http_client.js:426:23)     at TLSSocket.emit (events.js:194:15)     at endReadableNT (_stream_readable.js:1125:12)     at process._tickCallback (internal/process/next_tick.js:63:19)
Expected result
Return 1 so we can handle it's an error
Actual result
Return 0
Additional information

SFDX CLI Version(to find the version of the CLI engine run sfdx --version):
sfdx-cli/7.49.1-2b6ba86a25 linux-x64 node-v10.15.3
OS and version:
linux-x64

@alanJaouen we are taking a look, thank you for reporting

Got the same Error since some Days ago....
It should be the latest Version of sfdx, because it is part of our CI that the latest Version of sfdx will be downloaded for the Deployment...
[...]
ERROR:  Error: socket hang up
Unhandled rejection Error: socket hang up
at connResetException (internal/errors.js:610:14)
at TLSSocket.socketOnEnd (_http_client.js:453:23)
at TLSSocket.emit (events.js:327:22)
at endReadableNT (_stream_readable.js:1220:12)
at processTicksAndRejections (internal/process/task_queues.js:84:21)
[...]
Because it is unhandled, the CI Process tries to deploy further and further.... Then later at some Point the Timeout stops the Process.

got the same error today
ERROR:  Error: socket hang up
Unhandled rejection Error: socket hang up
at connResetException (internal/errors.js:610:14)
at TLSSocket.socketOnEnd (_http_client.js:453:23)
at TLSSocket.emit (events.js:327:22)
at endReadableNT (_stream_readable.js:1220:12)
at processTicksAndRejections (internal/process/task_queues.js:84:21)

Nobody assigned to this issue?
@clairebianchi as you wrote last year, that somebody will take a look at this issue... Are there any News for this?
Please don't get me wrong, but "sfdx" is quite essential for Salesforce Deployments.... Just wondering

I've had similar errors twice today, and as others have reported, the job doesn't just not exit with failure, but looks like it hangs until some other timeout is hit.
Once with a force:source:push, which failed, but then just sat there until the 10m no output timeout on my circleCI job expired
ERROR:  Error: socket hang up
Unhandled rejection Error: socket hang up
    at connResetException (node:internal/errors:642:14)
    at TLSSocket.socketOnEnd (node:_http_client:486:23)
    at TLSSocket.emit (node:events:390:22)
    at endReadableNT (node:internal/streams/readable:1307:12)
    at processTicksAndRejections (node:internal/process/task_queues:81:21)

Too long with no output (exceeded 10m0s): context deadline exceeded

And once with an force:apex:test:run, which I think hung until the --wait timeout of 60m was hit (it didn't hit the circle CI no output timeout which is 60m).
ERROR:  Error: socket hang up
Unhandled rejection Error: socket hang up
    at connResetException (node:internal/errors:642:14)
    at TLSSocket.socketOnEnd (node:_http_client:486:23)
    at TLSSocket.emit (node:events:390:22)
    at endReadableNT (node:internal/streams/readable:1307:12)
    at processTicksAndRejections (node:internal/process/task_queues:81:21)

ERROR running force:apex:test:run:  Cannot read property 'message' of undefined
Unhandled rejection undefined

Version:
sfdx --version
sfdx plugins --core

sfdx-cli/7.86.3 linux-x64 node-v15.8.0
You acknowledge and agree that the CLI tool may collect usage information, user environment, and crash reports for the purposes of providing services or functions that are relevant to use of the CLI tool and product improvements.

You acknowledge and agree that the CLI tool may collect usage information, user environment, and crash reports for the purposes of providing services or functions that are relevant to use of the CLI tool and product improvements.

@oclif/plugin-autocomplete 0.1.5 (core)
@oclif/plugin-commands 1.3.0 (core)
@oclif/plugin-help 3.2.2 (core)
@oclif/plugin-not-found 1.2.4 (core)
@oclif/plugin-plugins 1.9.5 (core)
@oclif/plugin-update 1.3.10 (core)
@oclif/plugin-warn-if-update-available 1.7.0 (core)
@oclif/plugin-which 1.0.3 (core)
@salesforce/sfdx-trust 3.6.0 (core)
alias 1.1.5 (core)
analytics 1.12.1 (core)
auth 1.4.7 (core)
config 1.2.4 (core)
generator 1.1.5 (core)
salesforcedx 50.16.1 (core)
├─ limits 1.0.3 (core)
├─ schema 1.0.3 (core)
├─ user 1.0.11 (core)
├─ apex 0.1.4 (core)
├─ templates 50.4.0 (core)
├─ @salesforce/sfdx-plugin-lwc-test 0.1.7 (core)
├─ custom-metadata 1.0.11 (core)
└─ salesforce-alm 50.16.1 (core)
sfdx-cli 7.86.3 (core)
telemetry 1.0.1 (core)

Getting the same error on sfdx force:mdapi:deploy:report:
ERROR:  Error: socket hang up
Unhandled rejection Error: socket hang up
    at connResetException (internal/errors.js:610:14)
    at TLSSocket.socketOnEnd (_http_client.js:453:23)
    at TLSSocket.emit (events.js:327:22)
    at endReadableNT (_stream_readable.js:1220:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)

As with others, this is crippling our Salesforce release pipeline. Whilst this had been happening occasionally in the past, since Spring '21 this has become an hourly occurrence.

Encountered the socket hang up errors yesterday after installing the latest SFDX cli package.
This is the error encounter for sfdx force:source:retrieve. Similar errors for other sfdx commands like deploy.
It works sometimes but mostly errors out with below stacktrace.

Error:
socket hang up
at connResetException (internal/errors.js:610:14)
at TLSSocket.socketOnEnd (_http_client.js:453:23)
at TLSSocket.emit (events.js:327:22)
at endReadableNT (_stream_readable.js:1220:12)
at processTicksAndRejections (internal/process/task_queues.js:84:21)
Outer stack:
at Function.wrap (/usr/local/lib/sfdx/node_modules/salesforce-alm/node_modules/@salesforce/core/lib/sfdxError.js:171:27)
at SourceRetrieveCommand.catch (/usr/local/lib/sfdx/node_modules/salesforce-alm/dist/ToolbeltCommand.js:248:46)
at async SourceRetrieveCommand._run (/usr/local/lib/sfdx/node_modules/@salesforce/command/lib/sfdxCommand.js:97:13)
at async Config.runCommand (/usr/local/lib/sfdx/node_modules/@oclif/config/lib/config.js:173:24)
at async Main.run (/usr/local/lib/sfdx/node_modules/@oclif/command/lib/main.js:27:9)
at async Main._run (/usr/local/lib/sfdx/node_modules/@oclif/command/lib/command.js:43:20)
at async Object.run (/usr/local/lib/sfdx/dist/cli.js:32:20)

This is my current SFDX CLI version.

sfdx --version
sfdx-cli/7.81.0-7b953c80d1 darwin-x64 node-v12.18.3

reported similar issue here https://github.com/forcedotcom/cli/issues/529#issuecomment-821868303

Any update on this issue?
We are running force:source:deploy  and  issue is happening  more frequently.
CLI Version: sfdx-cli/7.93.1-762bca056d linux-x64 node-v14.15.4
ERROR:  Error: socket hang up
Unhandled rejection Error: socket hang up
at connResetException (internal/errors.js:607:14)
at TLSSocket.socketOnEnd (_http_client.js:493:23)
at TLSSocket.emit (events.js:327:22)
at endReadableNT (internal/streams/readable.js:1327:12)
at processTicksAndRejections (internal/process/task_queues.js:80:21)

Salesforce regularly has production DNS issues

Wow this really hit home, given Salesforce took down their DNS globally for hours yesterday!

W-6835454 for internal tracking.

Hmm, I now propose this issue be closed. I don't think most people replying to this issue understand that this isn't an sfdx issue. Instead I propose two new tickets:

Make sfdx have a more descriptive error message during network communication issues, so you don't get tickets like this where people think it's an sfdx issue
Fix sfdx to report an error status code on network communication issues

oh, network problems happens, but this is definitely sfdx issue with handling this situation.  'Unhandled rejection" - this shouldn't happen. This should be handled properly by sfdx. And of course the return code should indicate error (i.e. non zero)

Hmm, I now propose this issue be closed. I don't think most people replying to this issue understand that this isn't an sfdx issue. Instead I propose two new tickets:

Make sfdx have a more descriptive error message during network communication issues, so you don't get tickets like this where people think it's an sfdx issue
Fix sfdx to report an error status code on network communication issues

Using the old version of the force:source:deploy command from the salesforce-alm plugin I could reproduce this by starting a deploy, then killing the network connection.  With the new version of the command in the source plugin I could not reproduce this in the same way; the command returned an exit code of 1.
I also tested with a very low command wait time (1 minute) and the command exited with a timeout error and an exit code of 1.

@shetzel I still have the issue with the version 7.119.2, hapened today at

Please reopen this ticket. We're getting this repeatedly. sfdx is notoriously poor at reporting status codes, I suggest an audit of all exits of sfdx.
This is from salesforce/salesforcedx:7.177.0-slim
ERROR:  Error: socket hang up
Unhandled rejection Error: socket hang up
    at connResetException (internal/errors.js:609:14)
    at TLSSocket.socketOnEnd (_http_client.js:453:23)
    at TLSSocket.emit (events.js:327:22)
    at endReadableNT (_stream_readable.js:1221:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)

great, tests pass!
sfdx is a dangerous tool, the output can't be trusted


envoy: update cilium/proxy to latest version

This commit bumps cilium/proxy to the latest version in Helm and Go deps. In addition the following filters are enabled.

network - set_filter_state
http - set_filter_state
http - basic_auth

/test

/test

/test

/test

/test


add Limitations about resources view

Currently, resources view does not support private cluster.  (ref. https://github.com/Azure/AKS/issues/1801)
Could you please add the limitation to this doc? Thank you.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 0b68f2c4-bb6c-11a2-6c61-8af4057a2438
Version Independent ID: e3498bed-1447-6841-8353-9f1b5d3dc8df
Content: Create a private Azure Kubernetes Service cluster - Azure Kubernetes Service
Content Source: articles/aks/private-clusters.md
Service: container-service
GitHub Login: @mlearned
Microsoft Alias: mlearned

Hello @jyama4ta
I have assigned the issue to the content author for further review and update the doc as appropriate.

Hello @jyama4ta
I have assigned the issue to the content author for further review and update the doc as appropriate.

update: private clusters support resources view for now.
https://github.com/Azure/AKS/issues/1801

update: private clusters support resources view for now.
https://github.com/Azure/AKS/issues/1801


Swapping year 2015 for 2000 in plot tutorials

PR #162 isn't passing the CI tests. The failure seems to be stemming from the plotting tutorials, which use a time in 2015. That recent date triggers an IERS update, which raises a warning. I'm going to see if changing all 2015 dates to 2000 will fix this error.

The tests pass, merging.


Map not loading completely.

Only 19 columns print onto my terminal window.

I can actually go outside the screen and float forever, but when I push down a couple of times, I die.

OS: Ubuntu 14.04.2 LTS
Kernel: 3.16.0-43-generic
Steps to reproduce:

Clone from repo.
make CXX=g++-5 and run binaries/gaem.
Start the game, and it's broken.

EDIT: Tried using the precompiled version. Same exact errors.
May be a flaw in my system, but I really don't think it is.

Post yer savefile

@nabijaczleweli  Not to be found.

What does

Map not loading completely

mean, then?

@nabijaczleweli  Exactly what the screenshots show. Only 19 columns are printed. Are more to be expected or not?

The default map size is 15 by 15, did you increase it?
You should not be able to move out of the map

@nabijaczleweli As the second screenshot shows, I can even die outside. Didn't change anything in the source code.

How did you escape the map?
Repeated pressing of "D"?

@nabijaczleweli Are you building/testing on Linux or Windows?

@nabijaczleweli  Combination of D and W. @nmai Ubuntu, stated in the ticket.

@ChemiCalChems Oh man you weren't kidding about your name being difficult to pronounce haha

@nmai It's quite straight forward actually. That's another spelling. But the pronunciation is exactly the same as this. Just hit the speak button.

Could you post the key combo that lets you leave?

@nabijaczleweli I'll record footage.

@nabijaczleweli [Footage] (https://www.youtube.com/watch?v=2fszORhzpVM) As you can see, first time I got a float exception, then I exited the screen, and also broke the gravity by pushing S a couple of times. I died offscreen too.

You're running an old-ass version, get yerself 58 from dropbox

Experiencing that as well now, though the name didn't make sense

Closed via https://github.com/Skorezore/Gaem/commit/cdfb8fc88db87a3bc042521d7e36a7606a29d548


Release spark-extension using GitHub workflow

What type of PR is this?
This is an improvement PR
What this PR does / why we need it:
The goal of this task is to move the release process to github workflow from current scripts that are run manually.
The current release process is documented in [RELEASE.md](https://github.com/G-Research/spark-extension/blob/master/RELEASE.md/.
Due to its complexity, we reached out to one of the maintainers Enrico who approved this idea of moving release.sh to a github workflow for it's transparent and better usage.
Acceptance criteria / Notes for the reviewer:

[ ] Comment on the issues from the maintainer

[ ] Note: @pavlovic-ivan this is the most detailed AC for this PR together with next point

[ ] Documentation is updated with final proposed solution
[ ] Keep the current process as is (not introducing breaking changes), so the maintainers should always be able to do it the old way - but this PR should introduce a workflow that will copy that behavior

For internal reviewer:

Per instruction for publishing snapshot example workflow
Per  instruction for publishing release (non SNAPSHOOT versions) example workflow

NOTE: I'm not sure if it's possible to mock the nexus repo within CI/GitHub actions but due to lack of access I'm not able to push the release. Hopefully, we can alter current nexus with something similar to local-snapshoot-dummy repo
NOTE: Changes in CHANGELOG.md and pom.xml are required to simulate publishing artifacts, for review please scope it to publish-release.yml and publish-snapshoot.yml workflows

Part of maturity matrix: https://github.com/G-Research/oss-portfolio-maturity/issues/32

Fail-fast should be false for both workflows.


How to fine-tune Llama?

Hello everybody,
I am trying to fine-tune the Llama model, but cannot load the safetensors file. I have modified the training loop for debugging and development:
pub fn run(args: &crate::TrainingCmd, common_args: &crate::Args) -> Result<()> {
    let config_path = match &args.config {
        Some(config) => std::path::PathBuf::from(config),
        None => {
            let api = hf_hub::api::sync::Api::new().unwrap();
            println!("loading the model weights from {}", args.model_id);
            let api = api.model(args.model_id.clone());
            api.get(&args.which_model).unwrap()
        }
    };

    let device = candle_examples::device(common_args.cpu)?;
    let config = Config::tiny();

    let mut varmap = candle_nn::VarMap::new();
    let vb = candle_nn::VarBuilder::from_varmap(&varmap, DType::F32, &device);
    varmap.load(config_path).unwrap();

    /*let cache = Cache::new(false, &config, vb.pp("rot"))?;
    let model = Llama::load(vb, &cache, config, true)?;

    let params = candle_nn::ParamsAdamW {
        lr: args.learning_rate,
        ..Default::default()
    };
    let mut opt = candle_nn::AdamW::new(varmap.all_vars(), params)?;
    for (batch_index, batch) in batch_iter.enumerate() {
        let (inp, tgt) = batch?;
        let logits = model.forward(&inp, 0)?;
        let loss = candle_nn::loss::cross_entropy(&logits.flatten_to(1)?, &tgt.flatten_to(1)?)?;
        opt.backward_step(&loss)?;

        if batch_index > 0 && batch_index % 1000 == 0 {
            varmap.save("checkpoint.safetensors")?
        }
    }*/
    Ok(())
}

I realize this error is likely because I cannot use VarMap::load to load such a large safetensors file (as described here). However, how can I use VarMap (or something else that allows me to modifiy the tensor map) to load the weights? If there is not such a method, how should I implement this myself?
Thank you!
Eric

Would something like a new VarMap::from_tensors be a viable option?

What is the error message that you got?
Note that you should call load after creating the model with the varmap and not before. Calling load will copy the safetensors data in the tensors that were created during the model initialization (so I would refrain from adding a from_tensors bit here as it wouldn't make much sense). See how it's done in the mnist-example.
I'll try to find some time today to add some documentation around this, most likely by adding comments/tutorials to the llama2-c training, this doesn't support loading the model yet but it should be a couple line change.

I got the SafeTensor HeaderTooLarge error. I was calling load after creating the model with the varmap as in the mnist example. When I looked at the Llama example, I realized that it does not call load and instead uses a special method that loads the VarBuilder from tensors. I need a varmap to be the backend for the VarBuilder (for mutability), so that is why I submitted the PR.

Would you mind making a minimal repro for the error? I feel that we should look at the underlying cause rather than tweaking the var-map. For now one of its invariant is that it only holds variables and not non-trainable tensors so it's easy to plug it in optimizers and the new from_tensors function would break this invariant.

Let's have a look at the original error first and if there is no better workaround we'll go for something like you suggested but again would prefer keeping the api as simple as possible for VarMap as it's already pretty tricky to understand.

@Narsil, thank you for the idea! I tried that multiple times, making sure it was redownloading the file, but the error did not change. Here is the error:
"/home/ericbuehler/.cache/huggingface/hub/models--karpathy--tinyllamas/snapshots/0bd21da7698eaf29a0d7de3992de8a46ef624add/stories15M.bin"
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: WithPath { inner: SafeTensor(HeaderTooLarge), path: "/home/ericbuehler/.cache/huggingface/hub/models--karpathy--tinyllamas/snapshots/0bd21da7698eaf29a0d7de3992de8a46ef624add/stories15M.bin" }', examples/llama/training.rs:49:30
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I am able to run the normal Llama inference example, so memory is not an issue. I noticed the inference example uses TransformerWeights::var_builder to load the varbuilder, so I wanted to add VarMap::from_tensors to allow mutability of the map (I want to test candle-lora).
@LaurentMazare, here is a min-repro repo. To see the error, please cargo run --example llama -- train after a clone.

Thanks for the repro. Looking at the error message you give, it could be that the input file is not in the safetensors format: thes files are likely to end with .safetensors, whereas stories15M.bin is most likely using Karphaty's custom binary format.

Right - this is likely why there is a custom loader in the inference example.
Based on this (and the fact I need mutability of the VarBuilder), what are your thoughts on the necessity of the VarMap::from_tensors method - or is there another way to get this sort of mutability?

The easiest would probably be to convert the .bin file to .safetensors and use a safetensors file, these have other advantages.
Besides this, I think you could just mimick the behavior of load once you've created the model based on the VarMap by doing something like.
let model = build_model(vb);
let data = var_map.data().lock().unwrap();
for (key, var) in data.iter() {
    let init_value = // get the tensor value that you're interested in based on its name
    var.set(&init_value)?
}

The current api should expose the necessary functions. Would that work for you?
We could think about supporting more file formats in VarMap::load but sounds better to provide hooks so that users can do arbitrary manipulations here.

True, that would work - thanks!
Regarding VarMap::load, it would likely be hard to support loading .bin files unless they have some sort of type header.

Regarding VarMap::load, it would likely be hard to support loading .bin files unless they have some sort of type header.

Right, either we would need some optional argument for the format. In tch we try to guess it based on the extension but it ends up being flaky and confusing so probably not the best idea (e.g. everyone uses .bin as an extension).

Right.


refactor: enable load components from git URL

If remote git URLs is configured in globalConfig.json, Hypertrons will clone or update the remote repo in a hidden directory .component-repo. Then component files of all versions will be extract from git system. In the future this way may replace that loading component from local files. But at present they are compatible. It may take more time when a git URL is set first time since git clone is slow.
Signed-off-by: pantang<EMAIL_ADDRESS>

What is execa? I do not see the it in package.json

/approve
You may talk about the git history part in the meeting because I don't really understand all the codes.


determineversion blog post draft.

Please feel free to suggest edits/make comments.

Once people feel good about this, I can convert it from a draft to published and update the time.


How to save left and right channel in separate files?

Is there a way to save the left and right channel of a recording into separate files? I'm currently splitting the wav files after but given how comprehensive the soundfile lib is, I feel it might be there already.
Any help in this matter is really appreciated.

You get a frames x channels numpy array, so just index into that (audio[:, 0] for the first channel).


EventCell needs key props to prevent rendering bug on event component

On month view, when I add some events to my stream, sometimes the calendar is re-rendering an old event with the new props, instead of rendering a totally new event component.
Only saw it in my month view.
And said "sometimes" because in my configuration it's not happening on all the cells, sometimes it works, normally. (Mostly happend on the today cell, and the cell that have an event in the same "month line")
Solution
I add a key props to the EventCell component. That's all.
For now I used my own events idAccessor like this <EventWrapper event={event} key={event.id}>
What way would you do this ?
Adding an idAccessor props to the calendar ?

I guess this could happen if a new event has the same start and end time as one that gets removed. I think the idea of an idAccessor sounds good. What are your thoughts @jquense?

I'm not sure why it's important that event components remount if they're different, there really shouldn't be a lot of mount/unmount logic there I think...
I don't really want to require folks to include id's on events for this, it seems like probably a minor use case and one that can be work arounded fairly easily if you do have this issue

Hm you're right. I can't find any shouldComponentUpdate methods in RBC(?), so not sure how this could be a problem?

Now that hooks are commonplace and they depend on their call order to maintain consistency using keys is now more critical. Can we reopen this issue?


[Feature] Script allow run process in background

Problem Statement
Today if you launch a process in the background from script with the way works, it ends up hanging the test because the process is not put to the background.
Solution Description
Either added a new primitive to start a process in the background, or maybe a better option is to add a flag to script allow forking a process to the background.
Alternatives
No response
Additional Context
No response
Slack discussion
No response
Research

[X] I have searched other issues in this repository and mine is not recorded.

Thanks for opening the issue @lockwobr !
Can you elaborate on the use case ?

For testing I build my operator with cover so I can merge my chainsaw test code coverage with my unit test code coverage. Tell now I just run the operator in background before running the chainsaw tests, but I need to test migration code between versions of the operator, so I want to be able to in middle of the chainsaw test kill the old and start a new version of the operator binary to trigger the migration code, and also capture the code coverage stats. There are ways around this but way more involved.

Also to be more clear. When you try to do something like this in a script it will hang because its not running in the background.
nohup ${LOCALBIN}/manager > ${REPORTING}/int/std.out &
echo "$!" > ${PID}

+1


Parse conversion in for range expr

 for _, v := range []byte("foo")

This does not work yet.

Done 🎉
https://github.com/DQNEO/minigo/commit/1094e04d1337585cb5fd2f41f1efe49ac377588d


Added info on how to find the keyID to the readme file.

This module is very useful but I spent a fair amount of time figuring out how to find the keyID that it needs so I figured I'd add that to the readme file.

@genebean thanks for letting me know it turns out to be useful in your environment. I didn't know about the gpg trick myself ;-)

Fixed spelling... sorry 'bout that :)

@genebean I've merged the README changes. Thank you very much for your contribution, it is much appreciated :-)

Awesome, glad to help!


empty items

hello, I downloaded this and tried to implement the system. I've tried saving at various points in the game, and it does save. However, the items always seem to be empty, so the save is never valid when I reload the game. I'm not sure where the issue is stemming from - any help appreciated!

Ooops nevermind! Forgot to add flowcharts to the gamesaver


Update README.md

Hi, this commit is a bit redundant. Here's why,

We add contributors using Emoji keys, Not authors.
You added yarn for linux only
We support both yarn and npm, So it should not be an issue.


Check that environment exists when setting in config

Super simple, fixes #2086

Changelog

modal config set-environment will now raise if the requested environment does not exist.

@prbot approve


(No. 14 C) New Filter for Man-Made vs. Natural Barriers [ Need more info, Medium? ]

From Anne:
most agencies want to target man-made barriers for treatment instead of natural barriers so it would be good for them to see the potential mitigation options (man-made barriers).

[ ] Jocelyn will look into what she can do to help with this, if there's time.


Switch to Ecto 2.0 beta

The plan is to keep this in master and make sure everything works. Then switch to Ecto 2.0 final (once it's release) before releasing a new version of ExMachina.

@paulcsmith one small comment, otherwise looks good

Thanks @joshuaclayton :D


chore: align commercial support org logo appropriately

Provide a description of what has been changed
Checklist

[x] Commits are signed with Developer Certificate of Origin (DCO)

Relates to #924

cc: @tomkerkhove


fix travis badge

Sorry my comment was referring to the second url. The first url is used as the image src. The second is the href for the link

Thanks for doing this, I have fixed it myself. All good now.


extension activation failed [TypeError: this.vscodeNotebook.onDidOpenNotebookDocument is not a function

Version: 1.57.0-insider (user setup)
Commit: c324c61ad572be43a8696780e1aba3c1a2c46eee
Date: 2021-06-02T17:38:06.137Z
Electron: 12.0.9
Chrome: 89.0.4389.128
Node.js: 14.16.0
V8: <IP_ADDRESS>-electron.0
OS:
Environment data

VS Code version: 1.57.0-insider (user setup)
Jupyter Extension version (available under the Extensions sidebar): v2021.5.842923320
Python Extension version (available under the Extensions sidebar): v2021.6.902307158
OS: Windows 10 x64 and version: 10.0.19042
Python and/or Anaconda version: 3.8.2
Type of virtual environment used venv
Jupyter server running: N/A

Expected behaviour
Jupyter Not causing any issues with python extension and not causing extension host to crash.
Actual behaviour
Extension Activation fails due to TypeError, and extensions fails to activated
Steps to reproduce:

Updated the VSCode -Insider
Extension loads, when loading python extension
Due to Jupyter fails , the whole extension python fails to load

Logs

Here are logs

Error 2021-06-03 16:55:34: extension activation failed [TypeError: this.vscodeNotebook.onDidOpenNotebookDocument is not a function
	at new E (c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:90:287467)
	at Object.t.resolveInstance (c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:52:263462)
	at c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:52:262667
	at Array.map (<anonymous>)
	at Object.t.resolveInstance (c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:52:263232)
	at c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:52:262667
	at Array.map (<anonymous>)
	at Object.t.resolveInstance (c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:52:263232)
	at c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:52:262667
	at Object.t.resolve (c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:52:262930)
	at c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:52:254027
	at e._get (c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:52:253793)
	at e.get (c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:52:252505)
	at e.dynamicValue (c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:52:265671)
	at c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:52:262156
	at c (c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:52:261202)
	at c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:52:262096
	at c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:52:261615
	at Array.map (<anonymous>)
	at c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:52:261587
	at Array.map (<anonymous>)
	at Object.t.resolveInstance (c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:52:263232)
	at c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:52:262667
	at Object.t.resolve (c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:52:262930)
	at c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:52:254027
	at e._get (c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:52:253793)
	at e.get (c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:52:252505)
	at a.get (c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:90:516226)
	at c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:52:246130
	at processTicksAndRejections (internal/process/task_queues.js:93:5)
	at async c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:37:826464
	at async t.activate (c:\Users\Mag\.vscode-insiders\extensions\ms-toolsai.jupyter-2021.6.902307158\out\client\extension.js:37:826151)
	at async Promise.all (index 0)
	at async Promise.all (index 101)
	at async Promise.all (index 1)] {
  vslsStack: [
    CallSite {}, CallSite {}, CallSite {},
    CallSite {}, CallSite {}, CallSite {},
    CallSite {}, CallSite {}, CallSite {},
    CallSite {}, CallSite {}, CallSite {},
    CallSite {}, CallSite {}, CallSite {},
    CallSite {}, CallSite {}, CallSite {},
    CallSite {}, CallSite {}, CallSite {},
    CallSite {}, CallSite {}, CallSite {},
    CallSite {}, CallSite {}, CallSite {},
    CallSite {}, CallSite {}, CallSite {},
    CallSite {}, CallSite {}, CallSite {},
    CallSite {}, CallSite {}
  ]
}

@mage1k99 Hey there. The Jupyter extension in insiders sometimes needs to update to take breaking incoming changes from VS Code. The error that you are seeing looks like you have the new Jupyter, but you don't yet have today's build of VS Code - insiders. The commit should end with ad2a5 for the current insiders. Can you please try updating VSCode - insiders and try again?

Yes the issue is with new version, when I installed newer one there are no issues.
Hey I have Installed v2021.6.902307158,
Vscode Insiders says there are no updates, so manually downloaded and installed to check if the version got updated but its not updated.
Current commit has ended with eee46 (c324c61ad572be43a8696780e1aba3c1a2c46eee)

@mage1k99 There is a new VS Code Insiders build which became available about 10 min ago. Could you please try downloading that, restarting VS Code, and trying again?

Yes, received it. Jupyter issue got fixed.
thanks for the support :)
closing the issue.


Demo breaks with<PHONE_NUMBER>001629

FYI This demo breaks with card numbers such as<PHONE_NUMBER>001629 .
Not sure if it is within the scope of this application but the logic does not handle things like card declined codes thrown with Stripe::CardError

Thank you for flagging. I've added a ticket to support card_declines in this demo.

great thanks @ctrudeau-stripe :) I would have also been satisfied with: "This is not within the scope of this project" lol

It should be! :)


LexiconNER (and HashTrie) needs to be smaller, faster

Mihai wrote this once upon a time:
Also, on a different topic: processors has gotten pretty fat. Some of it is out of our control (CoreNLP models), but some of it is our stuff. In particular, we use a lot the LexiconNER class, which stores its internal data (basically multi-word dictionaries) as a HashTrie:
https://github.com/clulab/processors/blob/master/main/src/main/scala/org/clulab/struct/HashTrie.scala
This is a class that I implemented a while ago, and I am fairly sure it’s sub-optimal. At some point I’d like to brainstorm with you about optimizing it…

Here is some of the discussion that followed, simply pasted from the email.
Hi Keith,
Sorry about the delay on this! Comments inline.
On Tue, Dec 11, 2018 at 1:09 PM Keith Alcock<EMAIL_ADDRESS>wrote:
So I've made lots of changes to HashTrie and LexiconNER which should not alter any results and should probably not have impact on other files or projects (only probably, because some public defs were moved).  I still have to do the LexiconNER object and verify with space and time measurements and do regression tests.  However, there are many other possible changes that may be worthwhile but will have consequences outside this environment (including serialization) and someone else should decide their fate.

uniqueStrings.  These are kept in HashTrie for statistics but probably at a significant price.  "Improvements" might be

at least do a clear() after statistics are complete
include a debug argument in constructor to disable statistics
use a subclass to keep statistics only when necessary (e.g., in TestHashTrie)
manage an Option[HashSet] from outside the class and usually use None as an argument to add(tokens) or clear as necessary

If they are not used in a significant way outside of the class (can't check right now, I don't have a laptop with IntelliJ here) then I agree they should at least be made optional. I think the simplest option would be the second one: include a debug argument in constructor, and have it set by default to false. And uniqueStrings should be an Option, as you say.

The labelling in HashTrie.findNormalized and LexiconNER.findLongestMatch.

This is done with string addition which may result in many instances of strings that are the same.  I've used some lazy vals and calculate only once per call in HashTrie or once per hit in LexiconNER, but this is still inefficient.  Here is one work around:

Set the labels in the constructor to HashTrie (or special subclass) and create the B- and I- versions just once per KB.
Let LexiconNer access these values so that this "val matchers: Array[(String, HashTrie)]" is not necessary.  The HashTrie will have this string/label.

You mean in places like this:
https://github.com/clulab/processors/blob/master/main/src/main/scala/org/clulab/sequences/LexiconNER.scala#L97
Yeah, you're right... I think the simplest change would your first: compute the "B_..." and "I-..." labels just once in the c'tor.
A more invasive change would be to completely replace this string accounting with integers in a C style and resort to strings only for output.  It could look something like this:
val LOC = 0x01
val TIME = 0x02
val GEO = 0x03
val OUTSIDE = 0x00
val BEGIN = 0x10
val INSIDE = 0x20
val B_LOC = BEGIN | LOC
val I_LOC = INSIDE | LOC
def printTag(tag: Int) = if (tag & BEGIN) ... else if (tag & INSIDE) ...
etc.

Interning strings.  Processor.internString(casedS) is being called optionally.  It may be that it should always be called and with the interned strings tracked on a per HashTrie basis.  Either way, if == strings are at some point mapped to eq() objects, the entries HashMap can be changed into a Java IdentityHashMap which is probably more efficient, possibly much more.  (It's not just the eq, but identityHashCode that saves time.)  If entries, which already tracks the strings is used, there would also be no double bookkeeping of the interned strings (except where different KBs have the same strings).

If interning is always performed, it should be possible to replace the token of the TrieNode with an integer, either from the order it was seen as it was added to entries or the order after all the words are alphabetized.  It would be very quick to look for an integer within an integer array, whether values are sorted or not.
This is a good idea. At some point, all Strings in processors were interned. But this causes another problem: interned Strings are stored in a different heap memory in the JVM (I can't recall what they call it...), which is typically much smaller than the regular heap. When I was doing this at scale, we used to run a lot into weird OOM errors because of this. We can increase the size of this heap in the JVM, but I think it is always much smaller than the regular heap. So, long story short: it is probably a good idea to intern Strings that live for a long time (e.g., labels such as the examples in (2)) but probably not all Strings. Do you have a different take here?

Ordering strings in the ListBuffer.  Since the strings are ordered, a binary search might be better, assuming that there is a reasonable number of siblings.  This may not be efficient on a ListBuffer, but that could be changed to an ArrayBuffer or an Array.  It may be more efficient to convert the buffers to normal Lists or Arrays after the structure has been built so that lookup is faster.

Which ListBuffers are you referring to?
In any case, I think this is a good idea!

Integers instead of references/pointers.  The collection of TrieNodes can eventually be flattened into a single array of integers, which would use quite a bit less memory and serialize more quickly.  entries turns into a IdentityHashMap[String, Int] with the Int providing the offset into an array of integers formatted like

completePathFlag
childCount
child0 string id
child1 string id
child0 offset
child1 offset
To do this, the HashTrie would be built close to normal with add(tokens), but then converted into a compressed version optimized for the find functions.
Another good idea!
That's the list right now.  It may be difficult to discuss this in writing; we might want to talk about it.
Keith
On Mon, Dec 10, 2018 at 6:16 PM Mihai Surdeanu<EMAIL_ADDRESS>wrote:
Yes,
The project you’re looking for is this one:
https://github.com/clulab/bioresources
Just FYI, not relevant at this point, these resources are created from other dictionaries by this script:
https://github.com/clulab/bioresources/blob/master/ner_kb.sh
(but I don’t think we’re changing the format of the files at this point, so not important)
Thanks!

On Dec 10, 2018, at 5:26 PM, Keith Alcock<EMAIL_ADDRESS>wrote:
Mihai,
I've been working on the optimization of HashTrie and LexiconNER in processors and will soon write or say a small report.  I'd like to test on some data and it seems like I should be looking for files like these which I haven't been able to track down in GitHub.  Can you provide another hint?  Thanks.
Keith
"org/clulab/reach/kb/ner/Gene_or_gene_product.tsv.gz",
"org/clulab/reach/kb/ner/Family.tsv.gz",
"org/clulab/reach/kb/ner/Cellular_component.tsv.gz",
"org/clulab/reach/kb/ner/Simple_chemical.tsv.gz",
"org/clulab/reach/kb/ner/Site.tsv.gz",
"org/clulab/reach/kb/ner/BioProcess.tsv.gz",
"org/clulab/reach/kb/ner/Species.tsv.gz",
"org/clulab/reach/kb/ner/CellLine.tsv.gz",
"org/clulab/reach/kb/ner/TissueType.tsv.gz",
"org/clulab/reach/kb/ner/CellType.tsv.gz",
"org/clulab/reach/kb/ner/Organ.tsv.gz"

On Wed, Nov 28, 2018 at 8:47 AM Mihai Surdeanu<EMAIL_ADDRESS>wrote:

On Nov 27, 2018, at 4:48 PM, Keith Alcock<EMAIL_ADDRESS>wrote:
Mihai,
The rule of thumb about performance is to measure, of course, so I did so with Eidos.  Apparently we only use two small files for NER: Quantifier.tsv and Property.tsv.  These load in around 24ms total for the pair.  Searching for named entities took 19ms for a fairly large, 200KB text file.  For Eidos, it doesn't seem like this is a bottleneck.  That's probably different with other projects, perhaps particularly with Reach.  Do you know of a especially problematic KB file that I can target for tests?

yes, the ruleNer in BioNLPProcessor uses massive lexicons:
https://github.com/clulab/processors/blob/master/corenlp/src/main/scala/org/clulab/processors/bionlp/ner/HybridNER.scala#L23
The HashTries loaded by it are huge. Any memory optimization would help there, I think.
For runtime, this loop should be optimized:
https://github.com/clulab/processors/blob/master/corenlp/src/main/scala/org/clulab/processors/bionlp/ner/HybridNER.scala#L32

There are always small improvements to make, but I don't knew whether they're worth in.  In this code below from HashTrie, it looks like an array of known size, the sentence length, is being constructed with list-like operations.  There is a similar case in LexiconNER.  (FWIW the array doesn't especially have to be created at all because it will be merged with an existing array.  Memory usage is a problem, sometimes more than speed.)  Things like that might halve the time for NER, but probably won't cut overall time by much.  Sometimes it's just nice to have them done, though.

Sorry, what exactly are you suggesting here?
Thanks!

private def findNormalized(sequence:Array[String], label:String, outsideLabel:String):Array[String] = {
var offset = 0
val labels = new ArrayBufferString
while(offset < sequence.length) {
val span = findAt(sequence, offset)
if(span > 0) {
labels += "B-" + label
for(i <- 1 until span) {
labels += "I-" + label
}
offset += span
} else {
labels += outsideLabel
offset += 1
}
}
labels.toArray
}
Keith
On Mon, Nov 26, 2018 at 6:30 PM Mihai Surdeanu<EMAIL_ADDRESS>wrote:
Hi Keith,

On Nov 26, 2018, at 4:52 PM, Keith Alcock<EMAIL_ADDRESS>wrote:
Mihai,
Regarding the HashTrie, as far as I can tell, neither Scala nor Java has a plain old tree in the standard library to borrow from and org.clulab.struct.Tree looks like it has a special purpose.  I don't know if the optimization of the top level HashMap[String,TrieNode] is paying off.

It is certainly paying off at runtime. These lexicons store phrases that are, by and large, single words, with a long tail for multi-word phrases. So, the first level in the trie tends to be extremely wide, while levels beyond the first tend to be very narrow with very few siblings. This is why I chose a HashMap for the first layer, so the search in the first layer is efficient. However, as I am sure you know, HashMaps are expensive, because the array that stores the keys is often much larger than the number of keys actually stored. I wonder if there is a good way to compress the key array?

What worries me a little is var children:Option[ListBuffer[TrieNode]] in the TrieNode in combination with this code:
for(i <- children.indices) {
val child = children(i)
val compare = newChild.token.compareTo(child.token)
if(compare < 0) {
children.insert(i, newChild)
If the ListBuffer is just a List which can count and insert things, this looks like an O(n^2) operation when it could be O(n) (or less) with a(n unfortunately deprecated) LinkedList or something like that, possibly homemade.  Even though these children are sorted, the search for a matching child is forced to be linear because of the list.  If they were stored in an array, then a binary search could speed things up.  For that matter, arrays can be nicely sorted in bulk after all insertions, but whether a quicksort would make up for having to look through the whole list is unknown.  If the KBs were preprocessed to be alphabetical in the first place, then insertion wouldn't be necessary.  Each of these lists could be a HashTrie itself and then the ordering wouldn't matter, but then there would be some unnecessary hashing being done and overhead if a tree is larger than a list.

I think you’re correct that beyond level 1, it is much better to store siblings in a sorted array. As I mentioned above, these levels tend to be very narrow, so creating new arrays when inserting longer phrases should be fairly cheap. And the memory savings compared to those ListBuffers should add up.
I think this is worth doing!

It looks like the HashTrie is used only by LexiconNER and that LexonNER might be serialized.  I don't know how long that's taking, but sometimes the native format is slow.
Those are some initial observations anyway, probably all obvious.
Keith

So as far as I understand it, there is a program, KBGenerator, in the bioresources project that takes a configuration file that specifies a mapping of various knowledge base (KB) contents onto labels.  For example, biopax-simple_chemical	onto Simple_chemical.  Multiple KBs can map onto the same label, so the program combines them.  The result is several text files, one for each label.
There is another program, KBLoader, that can take all these text files, put them into a NER, and serialize it, usually into model.ser.gz.  It is my impression from debugging and also from cluprocessorbio.conf that this serialized version is not normally used.  Rather, the text files generated by KBGenerator are read in again.  This might be due to performance issues.  In my test, it took 57 seconds to read in a serialized version of a LexiconNER while reading it in from text files only required 4 seconds.

There is a reimplementation of HashTrie that addresses some of the issues itemized above, but see below for one more important one that looks like it can speed up runtime performance by 5x.

I moved the tracking of unique strings to a debugging subclass which is normally not used.  This saves a little time and memory.
Labels are calculated only once and no longer added together constantly.  They are not converted into integers, however, because that would require changes far and wide.
I decided not to use an IdentityHashMap for the strings.  This would require two lookups: one to map the string to an identity and another to map the identity to a TrieNode.  However, now the entries are indeed a HashMap<String, Int> where the Int is an offset into an array of what used to be TrieNodes.  Those no longer exist.  Their information is kept in other Arrays of Ints which are offsets of their children and records of the strings involved.
The child TrieNodes are no longer stored in a Option[ListBuffer[TrieNode]] but in a slice of an Array[Int].  They have been sorted and are found using binarySearch.  The original lists could also be sorted if necessary.  The sorting isn't especially advantageous in an average case (I used org/clulab/processors/eng.testa) because there aren't many TrieNodes with a large number of children.  However, they do exist and sorting helps in pathalogical cases (e.g., processing of the KB itself as text).  Some performance numbers will follow.
As mentioned above, there is no longer a TrieNode.  There are just Ints in three Arrays.  This is accomplished by allowing the normal HashTrie to be built and then converting it into a CompactHashTrie.  The HashTrie is quite conventient for adding new tokens, and it now satisfies an interface HashTrieAdder, while the compact version is produced after the adding is finished and is a HashTrieFinder, optimized for looking things up and low memory use.

@kwalcock: you are correct with your reading of KBGenerator and KBLoader.

@kwalcock: your implementation sounds interesting. Where is it?
Also, a good sanity check is to rerun all the Reach unit tests. See this script: https://github.com/clulab/reach/blob/master/runAllTests.sh
Of course, for this, you have to publish processors locally, and adjust the Reach build file to use the locally-published version.
If the unit tests pass, and our review of the code is fine, I am all for pushing this change.

To be continued...

On average text, which basically has no bio NEs, the new version is approximately 10% faster.  One test shows 5490ms vs. 4922ms.  On pathological texts which have more than the expected number of NEs, such as the KB itself, the new version looks to be twice as fast.  One test shows 7001ms vs. 3029ms.  The binary search might not be responsible, since that seemed to make negligible difference: 3029ms vs. 2991ms.  It might instead be the slower string comparisons in the old version.  However, this pathalogical case is probably not the worst case.  There are TrieNodes with up to 1854 children and I think they will slow down the linear search substantially.  There are also NEs up to 350 tokens in length FWIW.  If need be I can time the absolute worst case scenarios.

It seems to be difficult to compare memory usage in Java.  If the size of the serialized LexiconNER can be used as an approximation, the old version comes in at around 40MB and the new one at 18MB.  The majority is for the string map.  Some 45% of NEs are one token only and 49% are two tokens.

Up to this point, the LexiconNER does not have to be changed much.  I added some interfaces and one function call (finishAdding) to convert the TrieNode from one used for adding new entries to one for finding the NEs.  The next optimization does require some modification, though.
It is possible to combine all the separate HashTries into one large hash trie.  Instead of each TrieNode storing a boolean for completePath, it stores an integer which specifies which matcher it came from.  LexiconNER goes through its list of matchers as it gets built and passes the index that the matcher would have (if it really existed anymore).  That gets put into the TrieNode.  When a NE is eventually found, that index gets returned and it is used to create (or look up) the label, which is really the only part of the individual matchers that is left.
The rough draft of this shows it being around 5.3 times faster (3029ms vs. 574ms) on the pathological data.  Right now it's hacked into the existing code.  The final version would require modifying more of the code (which I am reluctant to do without this report) or making a separate class to use instead.

Modifications made one at a time, making it difficult to see the overall picture, especially with all the test code, are in the branch kwalcock-HashTrie.  A squashed version which skips intermediate steps is soon to be kwalcock-CombinedCompactHashTrie.  Remember that it is not complete.

Chiming in to just say that I'm excited to see efficiency improvements to LexiconNER.  As an anecdote, JFR shows that, after parsing, LexiconNER operations are where we're taking the biggest hit in our project.

@kwalcock: any updates on this? Thanks!
(Remember that the key test here is to make sure that unit tests in Reach pass)

No.  The changes to date have mostly preserved the current functionality and added optional relatively minor speedups or compression.  The next change, combining trees into one, which is in rough draft here, requires some internal changes to LexiconNER that would more so replace current functionality.  So does LexiconNER need to become OldLexiconNER and everything gets the updated LexiconNER?  Or is there a NewLexiconNER that can optionally be used?  There are related choices to be made and it might help to discuss them.

Maybe we should:

Move core functionality into a LexiconNER trait.
Implement the above trait in the current LexiconNER class, and rename it TrieLexiconNER.
Implement the above trait in your new class, and call it ArrayLexiconNER, or something like that.

1 sounds good
2 & 3 implies keep both versions, OK
It's more LexiconNER => SeparatedLexiconNER and a new CombinedLexiconNER.  Either one or both can have regular HashTrie storage behind it and/or CompactHashTrie.  The tries have to be slightly different for the Separated and Combined versions.  The separated one keeps a Boolean for completePath and the combined one keeps an Int for which Lexicon completes the path.  That's a slightly troubling number of combinations, but OK.

Do you have a better idea in mind?

@MihaiSurdeanu, FWIW I worked on this again last week.  It's in the testing stage now.  It's contained in a branch kwalcock-DuoHashTrie now, mostly for backup purposes and for the curious.  Quite a bit had to be revised for proper integration, so I changed branches.

Thank you!

Just FYI, if you're still using Reach: it is very dependent on being able to read and use Bioresources through Processors and will probably have to be modified/updated for these changes, too.

Thanks @hickst.  It has been tested quite a bit with the BioCluProcessor and is being tested with reach right now before eventually merging.

One recent sets of runs on pathological data gave the times below.  Data was the first 5MB of the very knowledgebase files that go into the LexiconNER in the first place.  They are from the bioresources project in the src/main/resources/org/clulab/reach/kb directory.  They are pathological because the density of NERs is much higher than would be in normal text.  This in turn makes these numbers probably look better than they would otherwise be.
Original: 13,761ms
Separated: 10,057ms (this is same design as original, just streamlined a bit)
Combined: 5,177ms (a single trie instead of multiple tries)
Compact: 2,450ms (no longer really a trie, just arrays)
That's around 5.6 times as fast as the original.

Very nice!

From: Keith Alcock<EMAIL_ADDRESS>Sent: Tuesday, February 26, 2019 7:47:43 PM
To: clulab/processors
Cc: Mihai Surdeanu; Mention
Subject: Re: [clulab/processors] LexiconNER (and HashTrie) needs to be smaller, faster (#302)
One recent sets of runs on pathological data gave the times below. Data was the first 5MB of the very knowledgebase files that go into the LexiconNER in the first place. They are from the bioresources project in the src/main/resources/org/clulab/reach/kb directory. They are pathological because the density of NERs is much higher than would be in normal text. This in turn makes these numbers probably look better than they would otherwise be.
Original: 13,761ms
Separated: 10,057ms (this is same design as original, just streamlined a bit)
Combined: 5,177ms (a single trie instead of multiple tries)
Compact: 2,450ms (no longer really a trie, just arrays)
That's around 5.6 times as fast as the original.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/clulab/processors/issues/302#issuecomment-467702197, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABH-zjlCejEpPKLX6AOJN3Wl5sA79lVrks5vRfHPgaJpZM4aiN3N.

@kwalcock , can this be closed?

Yes.  Perhaps some other part is the slowpoke now.  Thanks.


Remove unnecessary data from JSON responses

Some responses are returning unnecessary data like "_id"  and  "__v"
Let's cleanup the response JSON and return only what is needed
In the README we have the API documentation, which should be updated too

Hi, may I be assigned to this please? I have some experience with mongoose.

Hi, may I be assigned to this please? I have some experience with mongoose.

cool! assigned! any questions I'm available =)


calls to npmjs.com while server is running

I've noticed that sqlpad makes repeated calls to npm while it's running:

I verified that the node process is indeed the sqlpad process.
Initially I thought it's for some lazy package installation but it happens every time the server is running and several times as well.
Can you please shed some light on this?

@kessler I suspect this the update-check that sqlpad does periodically.

Yeah it's for the update check. Would it be preferred to be able to disable this?

Making it an opt out configuration sounds like a good idea.
It might be helpful to give some indication of that in the readme.
Also, just want to use this opportunity to say that this is a wonderful project!

Thanks @kessler! An opt out will be easy enough to add. I've been meaning to update documentation so I can expand on some of the security/behavior aspects of sqlpad there too

This was added in #186, but has not made it to a release yet. Will leave this issue open until documentation is added.

thanks for the update!
On Sun, Aug 27, 2017 at 7:14 AM, Rick Bergfalk<EMAIL_ADDRESS>wrote:

This was added in #186 https://github.com/rickbergfalk/sqlpad/pull/186,
but has not made it to a release yet. Will leave this issue open until
documentation is added.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rickbergfalk/sqlpad/issues/179#issuecomment-325176145,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAI_p-V2EK0Ds8Z0cDka1WBxEQ2u0tXJks5scO0vgaJpZM4NQgGW
.

This feature is in release 2.3.0, and I've put a blurb on the installation/administration page about it in the configuration section.


bug: dates are typed wrong

Dates are typed as string, whereas the real type seems to be { value: string }.

Which dates on which entity types? We're quite extensively working with dates at the moment and haven't seen that issue.

The start_date and end_date of a Task, it seems.

Ah, right.  I forgot that all our queries are done using the option { decodeDatesAsIso: true }, which uses the upcoming default behaviour of being strings. The types are, as you say, incorrect unless you use that option. They currently come back as Moment object otherwise. We looked a bit at setting the type depending on the options used, but it felt messy (And is one of the reasons we're moving to a single return type as we're discussing in https://github.com/ftrackhq/ftrack-javascript/issues/71).

Aha. I think it's OK for us to use that option as well. Thank you for clearing that up!
Closing in favor of #71 then.

Wait, we just tried this option. It doesn't work. I suppose it means that the option is not yet out until #71 is done? Will re-open this.
What alternatives to we have for now?

That option has been available since quite a while.
The option is on a query/mutation level, the second parameter in any query or mutation takes an options object..

Ah interesting.
I think with that, we'd be afraid to forget to specify it. Can we make it support setting query-level default options in the constructor of the session?

There's already a way to globally add additionalHeaders, the other big thing to use the options for (that makes sense to have globally). Hence, I don't see an issue except that it's planned for removal in 2.0, and might just be best to get to a 2.0 release instead. Create an issue in the JavaScript API repo, and we'll see if we can decide on how to handle it. Meanwhile, I'll try to get a finalization of 2.0 approved for the short-term backlog.

Great. Thank you for that update.
I agree. Maybe this issue can serve as that?

We actually took a decision about the 2.0 release earlier today, so your PR had great timing. We tried getting the 2.0 out this week or the next, but we found internal dependencies that would break with the planned 2.0 changes. They were already scheduled to be removed in the coming month, so it made no sense to rebuild them. We also don't want to release the JavaScript API to a state where we wouldn't be using the latest version. We'll get your PR reviewed and released ASAP.

Awesome stuff! Great news, and thanks for keeping us posted!

Can we reopen this @gismya? We just tried setting that option, and it doesn't work. The strong-typing types it as string but the real value seems to be { value: string } still.

Looks like it should work. Make sure you're running the latest version (1.8.3) and initializing the constructor settings correctly
e.g:
    const decodeDatesAsIsoSession = new Session(
      credentials.serverUrl,
      credentials.apiUser,
      credentials.apiKey,
      {
        decodeDatesAsIso: true,
      },
    );

If you still have issues, send a code sample I can tests that has the issue and I'll try to find any bugs.

It's so weird. That's exactly what we're doing. And we have that latest version.
I'll see if I can make a better repro.

@gismya it is indeed still happening. The type of (for instance) end_date of a Task is just string, but the real runtime type is { value: string }.
Repro:
const session = new Session(
  ftrackServerUrl,
  ftrackApiUser,
  ftrackApiKey,
  { decodeDatesAsIso: true },
);
await session.initializing;

const project = await session.ensure<Project>('Project', {
  end_date: '2023-12-31T00:00:00',
  name: "some-random-project-name",
});
//here, project.end_date is of type string.
expect(project.end_date).toBe('2023-12-31T00:00:00');

Error:

Error: expect(received).toBe(expected) // Object.is equality
Expected: "2023-12-31T00:00:00"
Received: {"type": "datetime", "value": "2023-12-31T00:00:00"}
at toBe (/workspaces/WATS/src/apps/ftrack-event-lambda/src/domain/actions/UpdateMilestoneAndLinkedTasks.test.ts:168:32)
at processTicksAndRejections (node:internal/process/task_queues:95:5)

The above error is for the Project type, but it doesn't matter what type we use.

Based on the above, can we re-open as well?

I think the cause is actually that config set via the constructor (which I attempted to solve) is not working.

Thanks. Also, I'm not sure if this is a bug in the typings or the issue should be moved to the API client project.

Sounds like an API client error, but I'll look into it.

That's not a working reproduction example. Are you using mocking in your tests, or is that running towards an instance? Because the returned value looks incorrect, there should be no "type" key in the object, it should be "type".

No mocking involved - it's an integration test running directly towards a real Ftrack instance, running version 4.12.
However, the types were generated waaaaaay back. Just right after the TS generator was published, and back then, we were using Ftrack 4.10 (so almost 2 year old version). Could this have been the cause?
But then, what I don't get is that the type is wrong at runtime.
When I did the repro, it was just that example I gave you. It fails, towards a real Ftrack instance.
Very strange that you're not getting the same results!

It seems like the result you get from the server is incorrect, which is super weird. This is an on prem instance?
Also, I saw that the GitHub formatting incorrectly picked up the keys I mentioned in the last post: the issue is that the __type__ key is getting to you as type.
But the issue isn't with the type or the API client from what I can tell.

Interesting. Yes, it's an on-prem instance.
Hmmm is that what causes the schema type generator to generate wrong types then, or?

It would cause the API client to not convert it to a string (That the schema says it is). Can you try to verify the data coming from the server somehow, it could also be the test runner, the pasting to github or something else that removes the __? Try running it through the browser console on the actual instance (In the projects, reports or my task tabs) using
const session = ftrackSpark.getSession();
await ftrackSpark.getSession().query('select start_date from Project limit 1')

and inspect the response in the network tab. It should look like
[
    {
        "action": "query",
        "data": [
            {
                "__entity_type__": "Project",
                "id": "17496542-b5b5-4107-9624-0c8180df9969",
                "context_type": "show",
                "start_date": {
                    "__type__": "datetime",
                    "value": "2023-08-18T07:06:33"
                }
            }
        ],
        "metadata": {
            "next": {
                "offset": 1
            }
        }
    }
]

It was indeed GitHub ruining the formatting. I changed from a quote style block into a code block now. I could also see when I edited the comment that the code was indeed __type__.
I saved the modification to the comment now, so it reflects the latest.
Does that change anything?
I'll try the given line soon.

Running the command on my trial cloud account gives me this result:
{"action":"query","data":[{"__entity_type__":"Project","id":"33b1e354-8b3d-11eb-8e99-c2ffbce28b68","context_type":"show","start_date":"2023-10-02T00:00:00.000Z"}],"metadata":{"next":{"offset":1}}}

The command on our on-prem cluster yields a similar result:
{"action":"query","data":[{"context_type":"show","__entity_type__":"Project","id":"000877aa-6e5e-11ee-a4cd-a260f9cca335","start_date":"2020-01-01T00:00:00.000Z"}],"metadata":{"next":{"offset":1}}}

Does it matter if it is run from Node or web perhaps? We are running it from Node.

Still can't find anything wrong either through the browser or node ,on any instance I have or create. I'll close this until I get a reproducible example. From what I see now, my best guess is that the test runner is doing something weird with the value before comparing. Make sure the value you get from a query really is not a string.

Still can't find anything wrong either through the browser or node ,on any instance I have or create. I'll close this until I get a reproducible example. From what I see now, my best guess is that the test runner is doing something weird with the value before comparing. Make sure the value you get from a query really is not a string.

That's the weird part. Our error also occurs in production.
I will try to see if one of our other devs can repro, and then get back to you.

I found the cause @gismya! But not sure how to solve it.
I think it has to do with differences in the value of  serverInformation.is_timezone_support_enabled on some of our environments!
But not sure how that is triggered. Is it a setting somewhere in Ftrack?

I don't know how the server settings are set up for on prem. Send an email to support for further help with that.

Alright, will ping them.
However, actually, some further thoughts... Should that setting even affect the decoding? Why is that check actually there? In my naive viewpoint, decoding of times as strings instead of dates should be unrelated to whether timezone support is enabled. It should be supported either way, right?

It should not break, and I don't know why it does, but since it's a deprecated setting we probably won't make any investigations on it or changes regarding how they are handled in the API client.

Yeah makes sense. Any plans on dropping support for older Ftrack servers anytime soon?

Oh, and decodeDatesAsIso was indeed true at the time.

Also, I did some tests:
Object === {}.constructor //evaluates to true
Object === Object.create(null).constructor //evaluates to false
Object === Object.create({}).constructor //evaluates to true

Is this intentional? Not sure how the Ftrack API creates its objects. But may it proves that this check is not the best, and should be excluded altogether or changed?

And one more thing... I feel like we have enough information now to re-open it.

Good find. You can open this in the ftrack-javascript repository, with a short summary and then you can link to this issue for the details. With this we definitely have enough information to solve this.

There's a move button in github's UI though, and that'd move the full discussion too. I think that makes most sense, but only you can do that.
If you still think I should make a new issue, I will 🙏


unable to run / crash on startup on mod version 6.0.0 with Fabric 1.17.1

im failing to load the game with version 6.0.0 on fabric version 0.11.6 for mc1.17.1, saying that version 0.29.1 or higher, but there's no version like that... right?
https://pastebin.com/JbJNFdWs
after fiddling with files a bit, I managed to find a line in fabric.mod.json file that indicated the required fabric version. i deleted it, and got a different crash
https://pastebin.com/RHpnWWdc
https://pastebin.com/D0j42AFQ
https://pastebin.com/zfuCPNRP
rest of the needed mods are in the pastes

you are not supposed to delete that line, also “there is no versions higher than that”:

yes, i know im not really supposed to do that, I was doing that for debugging purposes
I didn't know there were newer versions than that, MultiMC version picker didn't have anything higher than 0.11.6
appreciate the fast response tho


SAML2 authorization issue with AD FS Microsoft

Attempted Debugging

[X] I have read the debugging page

Searched GitHub Issues

[X] I have searched GitHub for the issue.

Describe the Scenario
Hello,
I try to configure SAML with AD FS.
Tell me how to solve the problem
I set everything up, but the error keeps popping up
My config .env:
AUTH_METHOD=saml2
AUTH_AUTO_INITIATE=false
SAML2_NAME=BOOKSTACK
SAML2_EMAIL_ATTRIBUTE=email
SAML2_EXTERNAL_ID_ATTRIBUTE=id
SAML2_DISPLAY_NAME_ATTRIBUTES=firstName|lastName
SAML2_IDP_ENTITYID=https://data-centre.siecom.pro/FederationMetadata/2007-06/Fe$
SAML2_AUTOLOAD_METADATA=true
SAML2_IDP_SSO=https://data-centre.siecom.pro/adfs/ls/
#SAML2_IDP_SLO=https://data-centre.siecom.pro/adfs/ls/

SAML2_IDP_x509="-----BEGIN CERTIFICATE-----
MIIFZTCCBE2gAwIBAgITFgAAAA/znQbmfG8ocwABAAAADzANBgkqhkiG9w0BAQsF
ADA8MRMwEQYKCZImiZPyLGQBGRYDcHJvMRgwFgYKCZImiZPyLGQBGRYIdXNlcmdh
dGUxCzAJBgNVBAMTAkNBMB4XDTIyMDkxNjA2Mzg1N1oXDTIzMDkxNjA2Mzg1N1ow
ADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKrSC+w1KKqIjKRqqB+O
..........................................................."
SAML2_IDP_AUTHNCONTEXT=true

On the AD FS:
The file with metadata https://wiki.siecom.pro/saml2/metadata in AD FS was installed well, there were no problems with this
Set up the attributes as follows:
LDAP | Party trust wizard
E-Mail-Addresses | email
Given-Name | lastName
User-Principal-Name | id
Surname | firstName
Gives error on debug:
Invalid ACS Response: invalid_response
Tell me what I'm doing wrong?
Exact BookStack Version
v22.09
Log Content
Debug:

#0 /var/www/bookstack/app/Http/Controllers/Auth/Saml2Controller.php(119): BookStack\Auth\Access\Saml2Service->processAcsResponse()
https://github.com/BookStackApp/BookStack/issues/1 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Routing/Controller.php(54): BookStack\Http\Controllers\Auth\Saml2Controller->processAcs()
https://github.com/BookStackApp/BookStack/issues/2 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php(45): Illuminate\Routing\Controller->callAction()
https://github.com/BookStackApp/BookStack/issues/3 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Routing/Route.php(262): Illuminate\Routing\ControllerDispatcher->dispatch()
https://github.com/BookStackApp/BookStack/issues/4 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Routing/Route.php(205): Illuminate\Routing\Route->runController()
https://github.com/BookStackApp/BookStack/issues/5 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Routing/Router.php(721): Illuminate\Routing\Route->run()
https://github.com/BookStackApp/BookStack/issues/6 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(128): Illuminate\Routing\Router->Illuminate\Routing{closure}()
https://github.com/BookStackApp/BookStack/issues/7 /var/www/bookstack/app/Http/Middleware/CheckGuard.php(27): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}()
https://github.com/BookStackApp/BookStack/issues/8 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): BookStack\Http\Middleware\CheckGuard->handle()
https://github.com/BookStackApp/BookStack/issues/9 /var/www/bookstack/app/Http/Middleware/Localization.php(45): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}()
https://github.com/BookStackApp/BookStack/issues/10 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): BookStack\Http\Middleware\Localization->handle()
https://github.com/BookStackApp/BookStack/issues/11 /var/www/bookstack/app/Http/Middleware/RunThemeActions.php(26): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}()
https://github.com/BookStackApp/BookStack/issues/12 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): BookStack\Http\Middleware\RunThemeActions->handle()
https://github.com/BookStackApp/BookStack/issues/13 /var/www/bookstack/app/Http/Middleware/CheckEmailConfirmed.php(47): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}()
https://github.com/BookStackApp/BookStack/issues/14 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): BookStack\Http\Middleware\CheckEmailConfirmed->handle()
https://github.com/BookStackApp/BookStack/issues/15 /var/www/bookstack/app/Http/Middleware/PreventAuthenticatedResponseCaching.php(21): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}()
https://github.com/BookStackApp/BookStack/issues/16 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): BookStack\Http\Middleware\PreventAuthenticatedResponseCaching->handle()
https://github.com/BookStackApp/BookStack/issues/17 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/VerifyCsrfToken.php(78): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}()
https://github.com/BookStackApp/BookStack/issues/18 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Foundation\Http\Middleware\VerifyCsrfToken->handle()
https://github.com/BookStackApp/BookStack/issues/19 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/View/Middleware/ShareErrorsFromSession.php(49): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}()
https://github.com/BookStackApp/BookStack/issues/20 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\View\Middleware\ShareErrorsFromSession->handle()
https://github.com/BookStackApp/BookStack/issues/21 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php(121): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}()
https://github.com/BookStackApp/BookStack/issues/22 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php(64): Illuminate\Session\Middleware\StartSession->handleStatefulRequest()
https://github.com/BookStackApp/BookStack/issues/23 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Session\Middleware\StartSession->handle()
https://github.com/BookStackApp/BookStack/issues/24 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/AddQueuedCookiesToResponse.php(37): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}()
https://github.com/BookStackApp/BookStack/issues/25 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Cookie\Middleware\AddQueuedCookiesToResponse->handle()
https://github.com/BookStackApp/BookStack/issues/26 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/EncryptCookies.php(67): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}()
https://github.com/BookStackApp/BookStack/issues/27 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Cookie\Middleware\EncryptCookies->handle()
https://github.com/BookStackApp/BookStack/issues/28 /var/www/bookstack/app/Http/Middleware/ApplyCspRules.php(33): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}()
https://github.com/BookStackApp/BookStack/issues/29 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): BookStack\Http\Middleware\ApplyCspRules->handle()
https://github.com/BookStackApp/BookStack/issues/30 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(103): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}()
https://github.com/BookStackApp/BookStack/issues/31 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Routing/Router.php(723): Illuminate\Pipeline\Pipeline->then()
https://github.com/BookStackApp/BookStack/issues/32 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Routing/Router.php(698): Illuminate\Routing\Router->runRouteWithinStack()
https://github.com/BookStackApp/BookStack/issues/33 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Routing/Router.php(662): Illuminate\Routing\Router->runRoute()
https://github.com/BookStackApp/BookStack/issues/34 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Routing/Router.php(651): Illuminate\Routing\Router->dispatchToRoute()
https://github.com/BookStackApp/BookStack/issues/35 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(167): Illuminate\Routing\Router->dispatch()
https://github.com/BookStackApp/BookStack/issues/36 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(128): Illuminate\Foundation\Http\Kernel->Illuminate\Foundation\Http{closure}()
https://github.com/BookStackApp/BookStack/issues/37 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Http/Middleware/TrustProxies.php(39): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}()
https://github.com/BookStackApp/BookStack/issues/38 /var/www/bookstack/app/Http/Middleware/TrustProxies.php(41): Illuminate\Http\Middleware\TrustProxies->handle()
https://github.com/BookStackApp/BookStack/issues/39 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): BookStack\Http\Middleware\TrustProxies->handle()
https://github.com/BookStackApp/BookStack/issues/40 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(21): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}()
https://github.com/BookStackApp/BookStack/issues/41 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TrimStrings.php(40): Illuminate\Foundation\Http\Middleware\TransformsRequest->handle()
https://github.com/BookStackApp/BookStack/issues/42 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Foundation\Http\Middleware\TrimStrings->handle()
https://github.com/BookStackApp/BookStack/issues/43 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ValidatePostSize.php(27): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}()
https://github.com/BookStackApp/BookStack/issues/44 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Foundation\Http\Middleware\ValidatePostSize->handle()
https://github.com/BookStackApp/BookStack/issues/45 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/PreventRequestsDuringMaintenance.php(86): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}()
https://github.com/BookStackApp/BookStack/issues/46 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Foundation\Http\Middleware\PreventRequestsDuringMaintenance->handle()
https://github.com/BookStackApp/BookStack/issues/47 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(103): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline{closure}()
https://github.com/BookStackApp/BookStack/issues/48 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(142): Illuminate\Pipeline\Pipeline->then()
https://github.com/BookStackApp/BookStack/issues/49 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(111): Illuminate\Foundation\Http\Kernel->sendRequestThroughRouter()
https://github.com/BookStackApp/BookStack/issues/50 /var/www/bookstack/public/index.php(53): Illuminate\Foundation\Http\Kernel->handle()
https://github.com/BookStackApp/BookStack/issues/51 {main}

PHP Version
No response
Hosting Environment
Ubuntu 18.04

PHP Version: 7.4.30
BookStack Version: v22.09

Hi @Kol8bok,
I've just released BookStack v22.09.1, which within I added additional debug detail intended for this specific scenario.
Please could you update to v22.09.1 then re-attempt the scenario and share the new error message that's shown/logged? It should now provide much better guidance into the reason that the SAML response is found to be invalid.

Since there's been no follow-up I'm going to close this. If the issue remains and is something you still require to be fixed respond to my previous comment and this can then be re-opened.

Hello, I'm experiencing the same issue. Could you help please?

@RidaLeFou Please open a new support issue to focus on your exact error message and system configuration.

Привет @Kol8bok, я только что выпустил BookStack v22.09.1, в который я добавил дополнительные детали отладки, предназначенные для этого конкретного сценария.Не могли бы вы обновить версию до версии 22.09.1, затем повторить попытку сценария и поделиться новым сообщением об ошибке, которое отображается / регистрируется? Теперь он должен предоставить гораздо лучшее руководство по причине того, что ответ SAML признан недействительным.

Привет @Kol8bok, я только что выпустил BookStack v22.09.1, в который я добавил дополнительные детали отладки, предназначенные для этого конкретного сценария.Не могли бы вы обновить версию до версии 22.09.1, затем повторить попытку сценария и поделиться новым сообщением об ошибке, которое отображается / регистрируется? Теперь он должен предоставить гораздо лучшее руководство по причине того, что ответ SAML признан недействительным.

Hi, @ssddanbrown
Dan, I apologize for not answering for a long time, I don’t have time to deal with this problem a lot, so I finally found the time :)
Updated Bookstack l to version v22.09.1, the error also persists, files with metadata are transferred well, without any obvious problems. But when authorizing on the AD FS server, the following debug appears. It seems that everything is set up correctly, AD FS works correctly.
The debug is the following:
Invalid ACS Response; Errors: invalid_response; Reason: The status code of the Response was not Success, was Requester -> urn:oasis:names:tc:SAML:2.0:status:InvalidNameIDPolicy
PHP Version: 7.4.30
BookStack Version: v22.09.1
Theme Configured:
#0 /var/www/bookstack/app/Http/Controllers/Auth/Saml2Controller.php(119): BookStack\Auth\Access\Saml2Service->processAcsResponse() #1 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Routing/Controller.php(54): BookStack\Http\Controllers\Auth\Saml2Controller->processAcs() #2 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php(45): Illuminate\Routing\Controller->callAction() #3 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Routing/Route.php(262): Illuminate\Routing\ControllerDispatcher->dispatch() #4 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Routing/Route.php(205): Illuminate\Routing\Route->runController() #5 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Routing/Router.php(721): Illuminate\Routing\Route->run() #6 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(128): Illuminate\Routing\Router->Illuminate\Routing\{closure}() #7 /var/www/bookstack/app/Http/Middleware/CheckGuard.php(27): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #8 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): BookStack\Http\Middleware\CheckGuard->handle() #9 /var/www/bookstack/app/Http/Middleware/Localization.php(45): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #10 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): BookStack\Http\Middleware\Localization->handle() #11 /var/www/bookstack/app/Http/Middleware/RunThemeActions.php(26): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #12 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): BookStack\Http\Middleware\RunThemeActions->handle() #13 /var/www/bookstack/app/Http/Middleware/CheckEmailConfirmed.php(47): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #14 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): BookStack\Http\Middleware\CheckEmailConfirmed->handle() #15 /var/www/bookstack/app/Http/Middleware/PreventAuthenticatedResponseCaching.php(21): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #16 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): BookStack\Http\Middleware\PreventAuthenticatedResponseCaching->handle() #17 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/VerifyCsrfToken.php(78): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #18 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Foundation\Http\Middleware\VerifyCsrfToken->handle() #19 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/View/Middleware/ShareErrorsFromSession.php(49): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #20 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\View\Middleware\ShareErrorsFromSession->handle() #21 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php(121): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #22 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php(64): Illuminate\Session\Middleware\StartSession->handleStatefulRequest() #23 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Session\Middleware\StartSession->handle() #24 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/AddQueuedCookiesToResponse.php(37): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #25 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Cookie\Middleware\AddQueuedCookiesToResponse->handle() #26 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/EncryptCookies.php(67): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #27 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Cookie\Middleware\EncryptCookies->handle() #28 /var/www/bookstack/app/Http/Middleware/ApplyCspRules.php(33): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #29 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): BookStack\Http\Middleware\ApplyCspRules->handle() #30 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(103): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #31 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Routing/Router.php(723): Illuminate\Pipeline\Pipeline->then() #32 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Routing/Router.php(698): Illuminate\Routing\Router->runRouteWithinStack() #33 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Routing/Router.php(662): Illuminate\Routing\Router->runRoute() #34 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Routing/Router.php(651): Illuminate\Routing\Router->dispatchToRoute() #35 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(167): Illuminate\Routing\Router->dispatch() #36 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(128): Illuminate\Foundation\Http\Kernel->Illuminate\Foundation\Http\{closure}() #37 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Http/Middleware/TrustProxies.php(39): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #38 /var/www/bookstack/app/Http/Middleware/TrustProxies.php(41): Illuminate\Http\Middleware\TrustProxies->handle() #39 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): BookStack\Http\Middleware\TrustProxies->handle() #40 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(21): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #41 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TrimStrings.php(40): Illuminate\Foundation\Http\Middleware\TransformsRequest->handle() #42 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Foundation\Http\Middleware\TrimStrings->handle() #43 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ValidatePostSize.php(27): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #44 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Foundation\Http\Middleware\ValidatePostSize->handle() #45 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/PreventRequestsDuringMaintenance.php(86): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #46 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Foundation\Http\Middleware\PreventRequestsDuringMaintenance->handle() #47 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(103): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #48 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(142): Illuminate\Pipeline\Pipeline->then() #49 /var/www/bookstack/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(111): Illuminate\Foundation\Http\Kernel->sendRequestThroughRouter() #50 /var/www/bookstack/public/index.php(53): Illuminate\Foundation\Http\Kernel->handle() #51 {main}

Hi @Kol8bok,
Based upon that, It seems like your ADFS system is not accepting the  urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress NameIDFormat that BookStack wants to use.
I'm not too familiar with ADFS, and it's been a while since I last tested on an ADFS setup, so can't provide much more guidance myself, although I vaguely remember playing about with transform rules to get things working in the right format.

More precisely, you need to look towards AD FS? Is the email attribute being passed incorrectly?

@Kol8bok I'm afraid I'm not familiar enough with ADFS to make an advisory,
I'm not sure if the email attribute is controlled independently from the nameid-format.
I can only advise a search along these lines.

@ssddanbrown  ok, got it. Thank you!


Correction of alter strategy binary type bug + test

When activating the "Alter" mode, waterline retrieve data, delete tables, recreate them and then reinsert data, if data contains buffers, executing cloneDeep() de lodash causes buffer to be converted to another type of object and then data loss. In this PR, correction of the bug and a test of the mode "alter".

fix https://github.com/balderdashy/waterline/issues/996

First impressions: the indentation is a bit off, most (all?) of waterline files use 2 spaces for tab size. Can you please fix it?

done

Thanks @khalilTN, I think this is good! :white_check_mark:
@devinivy, what do you think?

@tjwebb, can you please take a look?


How is this code licensed?

Right now the code is unlicensed, to use it in some cases the license needs to be known, if you could include one that would be great!

@TyRud to watch issue.

Looks like it's MIT licensed


cachequeryqueue.py: support querying sources when artifacts are cached

Fixes #1749

The changes to match the serial cache query logic make sense to me. A test case would be good but I don't know how tricky this is to add. Build with storage-service and source push enabled with a target that is already built and cached?

This seems to fix the immediate problem based on our testing integration testing in freedesktop-sdk.

@abderrahim can we remove the Draft status from this or something else is still needed?

I've removed the draft status. I think it's ready, but I'm not sure how to test it.
Maybe we should simply make this the default path, and find a way to simplify the output (as it's currently very verbose).

@juergbi note we have done end-to-end testing a lot with this in freedesktop-sdk and it seems to fix the problems. Using storage service works reliably with this change.


Aladdin generated examples for Vm

Sort commands alphabetically in help files for easy diff.
Add Aladdin crafted examples.

This checklist is used to make sure that common guidelines for a pull request are followed.

[x] The PR has modified HISTORY.rst describing any customer-facing, functional changes. Note that this does not include changes only to help content. (see Modifying change log).

[x] I adhere to the Command Guidelines.

Closed per @glGarg request

Thanks! @tjprescott


Upgrade requirements

Codecov Report

Merging #111 (298c0cf) into main (3cf8762) will decrease coverage by 3.49%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##              main     #111      +/-   ##
===========================================
- Coverage   100.00%   96.50%   -3.50%
===========================================
  Files            6        6
  Lines          286      286
  Branches        51       51
===========================================
- Hits           286      276      -10
- Misses           0        8       +8
- Partials         0        2       +2

Impacted Files
Coverage Δ

src/django_linear_migrations/compat.py
69.69% <0.00%> (-30.31%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3cf8762...298c0cf. Read the comment docs.


Update ScenarioController.php

Remove code from scenario as it does not exist in the database migration

Merged, thank you for contributing