fix(qdrant): Support custom payload also when using fromExistingCollection

Fixes #2756

Sounds reasonable :)
Jacob Lee @.***> schrieb am Di., 29. Okt. 2024, 22:04:

@.**** commented on this pull request.
In libs/langchain-qdrant/src/vectorstores.ts
https://github.com/langchain-ai/langchainjs/pull/7069#discussion_r1821530556
:

* @returns Promise that resolves when the documents have been added to the database.

*/

async addDocuments(
documents: Document[],

documentOptions?: QdrantAddDocumentOptions

customPayloads?: QdrantCustomPayload[],

Yeah it's a fine line - I'd err towards keeping it for now with a note to
change it in the next minor bump.
—
Reply to this email directly, view it on GitHub
https://github.com/langchain-ai/langchainjs/pull/7069#discussion_r1821530556,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACAADHUQ2AZQD4XSVO27MXLZ57Z5JAVCNFSM6AAAAABQUBNZVKVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDIMBTGA3TQMBRGY
.
You are receiving this because you authored the thread.Message ID:
@.***>


Feature request: yii migrate/mark all

Similar to Rails migrations, I have a structure.sql (always the latest structure) and incremental migration files.
When I create a new database and import my schemadump, I still need to mark existing migrations as applied. Maybe if I could run something like..
yii migrate/mark all

..this would update migration history, without having to look for the latest migration timestamp myself?

mysqldump -u USER -pPASSWORD DATABASE > /path/to/file/dump.sql
yii migrate/up all
mysql -u USER -pPASSWORD DATABASE < /path/to/dump.sql

Hmm... migrations info is stored in the database so when its dump imported migration table should be imported as well, shouldn't it?

@marcovtwout as @samdark said, when you dump the schema, just dump the migration table too.


Significant performance penalty w.r.t. whisper.cpp

First of all, thank you for your work creating a safe wrapper around whisper.cpp.
As mentioned in #73, the performance of whisper-rs is quite poor compared to the reference implementation. I'll attempt to demonstrate below.
Setup
I'm using an M2 Max Macbook Pro with 64GB of (shared) memory. My goal is to run a web server with CoreML enabled, but if necessary I can run the tests with CPU only later. I'll attach output generated by flamegraph.
Rust script

Click me
// src/main.rs
mod utils;

use clap::Parser;
use anyhow::Result;
use log::info;

use whisper_rs::{FullParams, SamplingStrategy, WhisperContext};
use crate::utils::{decode, forward_pass, load_audio};

#[derive(Parser, Debug)]
#[command(author, version, about, long_about = None)]
struct WhisperCli {
    #[arg(short, long, default_value = "sample_data/jfk.wav")]
    input_file: String,
    #[arg(short, long, default_value = "models/ggml-medium.bin")]
    model_path: String,
}

fn main() -> Result<()> {
    env_logger::init();
    let args = WhisperCli::parse();

    let mut params = FullParams::new(
        SamplingStrategy::BeamSearch { beam_size: 12, patience: 0.1}
    );

    params.set_n_threads(8);
    params.set_translate(false);

    info!("Loading audio file {}", args.input_file);
    let raw_audio = load_audio(&args.input_file)?;

    info!("Decoding audio file");
    let decoded_audio = decode(raw_audio)?;

    info!("Loading model from {}", args.model_path);
    let whisper_context = WhisperContext::new(&args.model_path)?;
    let mut whisper_state = whisper_context.create_state()?;

    info!("Running forward pass");
    let result = forward_pass(&decoded_audio, params, &mut whisper_state);

    println!("{}", result);
    Ok(())
}

// src/utils.rs
use anyhow::Result;
use std::fs::File;
use std::io::Read;

use std::io::Cursor;
use log::{debug, trace};
use rodio::{Decoder, Source};
use rodio::source::UniformSourceIterator;
use whisper_rs::{FullParams, WhisperState};

const SAMPLE_RATE: u32 = 16000;
const CHANNELS: u16 = 1;

const LOW_PASS: u32 = 3000;
const HIGH_PASS: u32 = 200;

/// Load an audio file from the given path
/// and return as a vector of u8
///
/// # Arguments
/// * `path` - The path to the audio file
pub fn load_audio(path: &str) -> Result<Vec<u8>> {
    let mut file = File::open(path)?;
    let mut buffer = Vec::new();

    file.read_to_end(&mut buffer)?;

    Ok(buffer)
}

/// Decode the audio file and return as a vector of f32
pub fn decode(bytes: Vec<u8>) -> Result<Vec<f32>> {
    // Decode the audio file
    let input = Cursor::new(bytes);
    let source = Decoder::new(input)?;

    // Resample to output sample rate and channels
    let resample = UniformSourceIterator::new(
        source, CHANNELS, SAMPLE_RATE,
    );
    // High and low pass filters to enhance the audio
    let pass_filter = resample
        .low_pass(LOW_PASS)
        .high_pass(HIGH_PASS)
        .convert_samples();

    Ok(whisper_rs::convert_integer_to_float_audio(&pass_filter.collect::<Vec<i16>>()))
}

pub fn forward_pass(decoded_audio: &[f32], params: FullParams, whisper_state: &mut WhisperState) -> String {
    debug!("Starting forward pass");
    let _ = whisper_state
        .full(params, decoded_audio)
        .expect("Failed to run model");
    let mut result = String::new();

    debug!("Decoding results");
    // fetch the results
    let num_segments = whisper_state
        .full_n_segments()
        .expect("failed to get number of segments");

    for i in 0..num_segments {
        let segment = whisper_state
            .full_get_segment_text(i)
            .expect("failed to get segment");
        let start_timestamp = whisper_state
            .full_get_segment_t0(i)
            .expect("failed to get segment start timestamp");
        let end_timestamp = whisper_state
            .full_get_segment_t1(i)
            .expect("failed to get segment end timestamp");
        trace!("[{} - {}]: {}", start_timestamp, end_timestamp, segment);
        result.push_str(&format!("[{} - {}]: {}\n", start_timestamp, end_timestamp, segment));
    }
    result
}

# Cargo.toml
[package]
name = "whisper-rs-cli"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
anyhow = "1.0.71"
whisper-rs = { version = "0.8.0", features = ["coreml"] }
clap = { version = "4.3.12", features = ["derive"]}
env_logger = "0.10.0"
log = "0.4.19"
rodio = "0.17.1"

[[bin]]
    name = "whisper-rs-cli"
    path = "src/main.rs"

Data
For testing, I've converted a short JFK speech to WAV. See this link. Converting to WAV is done using ffmpeg:
ffmpeg -i main.mp4 -acodec pcm_s16le -ac 1 -ar 16000 jfk.wav

Results
I won't do averages of iterations since the differences are quite clear. Furthermore I've run both scripts before to ensure that the CoreML model is properly compiled for my architecture. I can confirm that the model loading step is not the issue. The chosen model is ggml-medium.bin. The commands used are, given that you have compiled whisper.cpp & whisper-rs and are in the root of each repository, as follows:
sudo time flamegraph -- ./main -m models/ggml-medium.bin -t 8  -f jfk.wav

sudo time flamegraph -- target/release/whisper-rs-cli

CoreML enabled
whisper-rs
85.27 real       537.47 user         6.41 sys

whisper.cpp
12.30 real        42.14 user        15.36 sys

Please let me know what you think and where I can help out. Admittedly I'm a bit inexperienced with Rust but I'd love to learn, especially solving such an issue.

I was able to reproduce this, and am working on it with @tazz4843

At this point the only thing I have that could be the difference is that by default whisper-rs is using v1.4.2 of whisper.cpp, while you're likely using git master upstream for these tests. Seems that upgrading whisper-rs's version to git master speeds it up somewhat, but I don't have Apple Silicon myself to test on so I can't do much myself in terms of poking around.

My test isn't the same but runs 2 seconds slower after updating whisper-rs to use whisper.cpp master

I've also pulled main for whisper.cpp. I had to edit 2 function signatures in whisper-rs to get it to work.
Only slight improvements:
77.39 real       354.37 user       118.68 sys

@wdoppenberg do you have any guesses as to why there's such a big performance discrepancy?

Given I don't have Apple Silicon myself to test on, I can't do much to help with this besides suggest x86 instead. Hopefully someone else can figure it out.

I have M1 and M2 and can try it out if someone provides a repo with a test case than can be run.
right now, for the canonical "your country" jfk test case, whisper-rs and whisper.cpp they are the same.
I put a timer around whisper.cpp whisper_full_params - it took 1612.61 ms
I put a timer around state.full -> state.get_segment_text - it took 1648 ms
C code changes:
     struct whisper_full_params   params,
                    const float * samples,
                            int   n_samples) {
-    // clear old results
+
+   struct timeval start_time, end_time;
+    gettimeofday(&start_time, NULL);
+  // clear old results
     auto & result_all = state->result_all;

     result_all.clear();
@@ -4761,7 +4768,16 @@ int whisper_full_with_state(
         }
     }

-    return 0;
+      gettimeofday(&end_time, NULL);
+
+    // Calculate the elapsed time in milliseconds
+    double elapsed_ms = (end_time.tv_sec - start_time.tv_sec) * 1000.0 +
+                       (end_time.tv_usec - start_time.tv_usec) / 1000.0;
+
+    // Print the result to stdout
+    printf("Elapsed time: %.2f ms\n", elapsed_ms);
+
+     return 0;
 }

❯ time ./main -m models/ggml-medium.en.bin -t 8  -f jfk.wav
whisper_init_from_file_no_state: loading model from 'models/ggml-medium.en.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51864
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 1024
whisper_model_load: n_audio_head  = 16
whisper_model_load: n_audio_layer = 24
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 1024
whisper_model_load: n_text_head   = 16
whisper_model_load: n_text_layer  = 24
whisper_model_load: n_mels        = 80
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 4
whisper_model_load: mem required  = 1899.00 MB (+   43.00 MB per decoder)
whisper_model_load: adding 1607 extra tokens
whisper_model_load: model ctx     = 1462.58 MB
whisper_model_load: model size    = 1462.12 MB
whisper_init_state: kv self size  =   42.00 MB
whisper_init_state: kv cross size =  140.62 MB
whisper_init_state: loading Core ML model from 'models/ggml-medium.en-encoder.mlmodelc'
whisper_init_state: first run on a device may take a while ...
whisper_init_state: Core ML model loaded

system_info: n_threads = 8 / 8 | AVX = 0 | AVX2 = 0 | AVX512 = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | VSX = 0 | COREML = 1 | OPENVINO = 0 |

main: processing 'jfk.wav' (176000 samples, 11.0 sec), 8 threads, 1 processors, lang = en, task = transcribe, timestamps = 1 ...

[00:00:00.000 --> 00:00:11.000]   And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country.
Elapsed time: 1612.61 ms

whisper-rs:
whisper_init_state: first run on a device may take a while ...
whisper_init_state: Core ML model loaded
[src/main.rs:66] idx = 1041
1041 [00:00:10.410]  And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country.
[src/main.rs:98] elapsed = 1648

 65             if let Some(frames) = buf.add(idx, mel) {
 66                 dbg!(idx);
 67                 let path = format!("{}/frame_{}.tga", mel_path, idx);
 68                 let _ = save_tga_8bit(&frames, n_mels, &path);
 69
 70                 let ms = duration_ms_for_n_frames(hop_size, sampling_rate, idx);
 71                 let time = format_milliseconds(ms as u64);
 72
 73                 let start = std::time::Instant::now();
 74                 let mut params = FullParams::new(SamplingStrategy::Greedy { best_of: 0 });
 75                 params.set_n_threads(6);
 76                 params.set_single_segment(true);
 77                 params.set_language(Some("en"));
 78                 params.set_print_special(false);
 79                 params.set_print_progress(false);
 80                 params.set_print_realtime(false);
 81                 params.set_print_timestamps(false);
 82                 state.set_mel(&frames).unwrap();
 83
 84                 let empty = vec![];
 85                 state.full(params, &empty[..]).unwrap();
 86
 87                 let num_segments = state.full_n_segments().unwrap();
 88                 if num_segments > 0 {
 89                     if let Ok(text) = state.full_get_segment_text(0) {
 90                         let msg = format!("{} [{}] {}", idx, time, text);
 91                         println!("{}", msg);
 92                     } else {
 93                         println!("Error retrieving text for segment.");
 94                     }
 95                 }
 96
 97                 let elapsed = start.elapsed().as_millis();
 98                 dbg!(elapsed);
 99             }

I'me using the set_mel api but there's no reason to think pcm audio will be slower rust vs c. If someone can provide a test case that can be run easily I can try them out

Still haven't been able to find the issue. When examining the call tree I find that, as one might expect, almost all calls are part of the ggml_compute_forward_mul_mat.

It almost feels like the whisper.cpp lib is not compiled with optimization flags enabled.

think it's your machine?

I have found the issue:
In my script, I used SamplingStrategy::BeamSearch { beam_size: 12, patience: 0.1}. This is quite heavy...
Using SamplingStrategy::Greedy { best_of: 0 } it runs in roughly 10.11 seconds (with a sync'd whisper.cpp submodule).


Failed to create solution with default user identify, ie no roles defined in AppID

If you try and create a solution as if you are a default user , ie logged into Ascent with no defined roles. Get an error saying I need Editor identity. All users should be able to create solutions using the wizard
The default user should not be forced to have Editor role in App ID

Fixed


Restructured zoho icons

Current dependencies on/for this PR:

main

PR #4   👈

This stack of pull requests is managed by Graphite.


Retry map task subtasks

TL;DR
Currently, k8s array tasks (map tasks) retry the entire collection of subtasks when one of them fails. This PR enables retries over individual subtasks.
Type

[x] Bug Fix
[ ] Feature
[ ] Plugin

Are all requirements met?

[x] Code completed
[x] Smoke tested
[x] Unit tests added
[x] Code documentation added
[x] Any pending items have an associated Issue

Complete description
A previous PR added tracking of retry attempts for each subtask. As the current implementation retries all subtasks if one fails, this wasn't particularly useful because they were all the same. With this PR we enable subtasks to be retried individually, using this retry attempts array to inform unique pod names and logs for each subtask retry.
When a subtask fails, the current implementation reports the failure (as retryable) to the parent map task. Now we attempt to retry by incrementing the retry attempt value and transitioning the subtask to the "Undefined" phase as if it had never been executed. This is recognized during the next evaluation and another attempt is executed.
We use the max attempts defined on the parent map task to determine the number of times a subtask may be executed. As we track subtask retries internally (rather than within the parent map task), if a subtask exceeds the maximum number of retries we report a permanent failure to the parent map task. This ensure that the entire collection of subtasks is not retried again (with each subtask potentially retried multiple more times). Other retryable failures reported to the map task will still fallthrough.
Tracking Issue
fixes https://github.com/flyteorg/flyte/issues/1276
Follow-up issue

https://github.com/flyteorg/flyte/issues/1533: This solution does not fix any issues with interruptible nodes within k8s array tasks. Will have to be addressed separately. Discerning what is an interrupted failure is very difficult, the current approach handles all "system" failures as interrupted and enables a different configurable number of retries. We might need to track the failure type as well?

Can we add some more unit tests to up the coverage for this patch?

Can we add some more unit tests to up the coverage for this patch?

Done.


encryption: Limit encryption keys to 2 bits

By the way, what has changed since https://github.com/cilium/cilium/pull/8710 which purported to resolve the conflict with k8s?

With the first patch collapsing the key mask to 3 we can just drop the 2nd patch. I don't really have a preference for how that logic is done and I don't see any reason for the change if its not needed. Assuming we will not support keys >3 afte this.

By the way, what has changed since #8710 which purported to resolve the conflict with k8s?

I'm not 100% sure we know at this point but we are seeing this:
[353:77784] -A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
[0:0] -A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000

So it's clear that we hit the drop rule without marking it.

test-me-please

test-me-please

test-me-please

Legit failure:

... value *errors.errorString = &errors.errorString{s:"encryption Key space exhausted, id must be nonzero and less than 3. Attempted "6""} ("encryption Key space exhausted, id must be nonzero and less than 3. Attempted "6"")

test-me-please

test-me-please

Removed v1.8 and v1.9 backporting labels until we figure out https://github.com/cilium/cilium/pull/15487.


Search results massively pushing reports and datasets to top

Coverage increased (+0.02%) to 93.16% when pulling 99018f02a35356b9698f1dbe9053998c8fb473e6 on search-results into b83178e953b85f0d9248f9c46031b33bccc25504 on dev.

Coverage increased (+0.02%) to 93.16% when pulling 46ffb94641b32880e92f48f07dbfae26d9fd976c on search-results into b83178e953b85f0d9248f9c46031b33bccc25504 on dev.


⚠ ClamAV has degraded performance

In edb41e0, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 249 ms

Resolved: ClamAV performance has improved in 251e6e9.


Fix the retrieval of the latest compiler version tag

See https://discourse.purescript.org/t/new-404-ci-failures-in-core-libraries/1225.

Thank you so much!


upgrade dwn-sdk-js, dwn-sql-store and release dwn-server

upgrade dwn-sdk-js to v0.4.3
upgrade dwn-sql-store to v0.6.3
release dwn-server v0.4.2

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 95.08%. Comparing base (5ba2637) to head (f106866).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #150   +/-   ##
=======================================
  Coverage   95.08%   95.08%
=======================================
  Files          28       28
  Lines        2948     2948
  Branches      395      395
=======================================
  Hits         2803     2803
  Misses        144      144
  Partials        1        1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


docs: technical advisory 1002 - MDC components

Definition of Ready

[ ] I am happy with the code
[ ] Short description of the feature/issue is added in the pr description
[ ] PR is linked to the corresponding user story
[ ] Acceptance criteria are met
[ ] All open todos and follow ups are defined in a new ticket and justified
[ ] Deviations from the acceptance criteria and design are agreed with the PO and documented.
[ ] No debug or dead code
[ ] My code has no repetitions
[ ] Critical parts are tested automatically
[ ] Where possible E2E tests are implemented
[ ] Documentation/examples are up-to-date
[ ] All non-functional requirements are met
[ ] Functionality of the acceptance criteria is checked manually on the dev system.

@peintnermax we should maybe explicitly state if login components are affected or not. Might not be clear to all users.


chore(main): release 4.0.5

:robot: I have created a release beep boop
4.0.5 (2024-10-12)
Bug Fixes

src: pick SB/BB player (4f931f7)

This PR was generated with Release Please. See documentation.

:robot: Created releases:

v4.0.5
:sunflower:


Fix warnings

Part of https://github.com/gtk-rs/release/issues/77
cc @GuillaumeGomez

Waiting for CI and then I merge. I'll cherry pick the commits once done.

I already have branch for master too https://github.com/gtk-rs/cairo/compare/master...EPashkin:warnings?expand=1

Oh. No need to worry about it, I'll just cherry-pick the commits like I said.

The release can resume now.


LICENSE.md and license.md collide on a case-insensitive filesystem

Describe the bug
If clone the repository on a Mac, I receive the following message:
warning: the following paths have collided (e.g. case-sensitive paths
on a case-insensitive filesystem) and only one from the same
colliding group is in the working tree:

  'LICENSE.md'
  'license.md'

As a result, one of the files "wins" is automatically removed or modified.
To Reproduce
Steps to reproduce the behavior:

Use a Mac or other operating system that is case-insensitive.
Run git clone https://github.com/intuitmike/information.git from a command line.
See the error above
Run git status and verify that LICENSE.md is modified from git's perspective.

Expected behavior
A clean repo clone, with no errors or changes.
Desktop (please complete the following information):

OS: MacOs 10.14.6 (Mojave)

@Mitzi-Laszlo That's an issue with the information repo which should be fixed over there; I suggest moving this issue back.

Information repo is now superseded by solidproject.org


String for mocks looks weird

I use Jest to mock out some child components, with the react-test-renderer I got
<mocked someProps="someString" />

using enzyme and shallowToJson I got:
<function (props) {return (/* istanbul ignore next */_react2.default.createElement( /* istanbul ignore next */'mocked', /* istanbul ignore next */_extends({ originalComponent: componentName }, props), props.children));} someProps="someString" />

Do you think its possible to recognise mocks and render they just like the react-test-renderer.

Hi, thanks for raising the bug, I will have a look!

@eskimoblood Can you give me more details about your test? How exactly are you mocking the component?

In one of my project, I'm using enzyme-to-json and Jest mocking like that:
jest.mock('../Field', () => {
    return function Field(props) {
        return props.children;
    };
});

and it works fine with Field as a display name.
But I'm interested in your issue, can you provide more details?

I'm using the approach from thus gist

But let me try your approach, maybe this fix the problem.

Cool, thanks for the link, I'm looking at it!

Ok so I've just tried that (basically just copying from your link):
function mockComponent(componentName) {
    return props => {
        return (
            <mocked originalComponent={componentName} {...props}>{props.children}</mocked>
        );
    };
}

jest.mock('./fixtures/mocked-component', () => { return mockComponent('Icon'); });

import MockedComponent from './fixtures/mocked-component';

it('converts a mocked component', () => {
    const shallowed = shallow(
        <MockedComponent className="class">Hello!</MockedComponent>
    );
    expect(shallowToJson(shallowed)).toMatchSnapshot();
});

and the snapshot output is:
exports[`test converts a mocked component 1`] = `
<mocked
  className="class"
  originalComponent="Icon">

    Hello!

</mocked>
`;

which seems ok 🤔

I too had transpiled code in my snapshot even with react-test-renderer https://github.com/facebook/jest/issues/1756
though it is closed, @cpojer acknowledge it was reported by others too.

I just move to
jest.mock('../../components', () => {
  return {
    TextField: () => 'TextField',
    DatePicker: () => 'DatePicker'
  }
});

as suggested in the comments of the gist and it works. So just close the issue. Thanks for your help.

Ok thanks for the insight @bsr203, that's strange 🤔
@eskimoblood I'm glad it works with you 😊 I'm still curious of what was causing that.


connect react with redux to perform operations

Hello Microverse👋 in this PR the following is done:

Wrap  with the  component from react-redux
Pass your configured store into the
Create an initialState variable for booksSlice.
Display your books, received from the slice, in a reusable component
Dispatch actions using useDispatch
Add a  component, which includes:

An event handler that adds a book to the books array (with attributes id, title and author)

Add a  component, which includes:

An event handler that removes a book from the books array (by id)

@amejid Thank you!


Compile error with Injector class as beta.16

I've just synced to latest seed, beta.16, and I'm getting compile errors in the Injector class, it says:
Injector.resolve does not exist on type 'typeof Injector'.
I went back to beta.15 and everything works as before (of course need to backtrack APP_BASE_REF reference).
Is the Injector class broken in beta.16?

There are quite a few breaking changes that need updating, do you have an example of where it breaks?
Does the seed work for you when you do a clean clone and run that?

Yes, the seed seems to work. It is some of my code where I use the Injector class that breaks with beta.16.
Up until beta.15 the Injector class had a bunch of static methods, Injector.resolve() being one of them. They all seem to be gone now. They do not show up in code hinting and now cause a syntax error.
Maybe that class has changed in beta.16 and I need to update my code. Angula2 docs does not cover that class.

@fourctv thanks for reporting the issue!
I think this is not related to the seed. Would you check the Angular issue tracker for related issues?

If you look at the breaking changes there was some changes to the injector, i'd start there https://github.com/angular/angular/blob/master/CHANGELOG.md

Injector was renamed into ReflectiveInjector, as Injector is only an abstract class with one method on it
Injector.getOptional() was changed into Injector.get(token, notFoundValue) to make implementing injectors simpler


Support for more Providers

Feature Description
There aren't many Providers available, having the possibility to download from more providers like FMD2 would be greatly appreciated
Solution you would like
No response
Alternatives you have considered
No response
Additional context
No response

You can use mangal-scrapers. Or create one yourself too. I am not planning to add more to kaizoku.


Trying to establish cloud connection fails

When trying to establish the cloud connection with mail adress and SHA1 password the following message appears:

But login with email and normal password at https://cloud.crownstone.rocks/login works fine and i see my access token there

Thanks for the message. I tried to reproduce, but didn't succeed.
Under the hood, the Node uses the /users/login API call.
As credentials, you can use:
{"email": "your e-mail", "password": "your SHA-1 encoded password"}
(content type: application/json, include: empty)
Does this call work in your case?


Quartz chokes on assets with UTF characters

Describe the bug
Add an image with the following filename to Obsidian: DALL·E 2024-01-17.png. Quartz exits with the following error:
TypeError: Invalid character in header content ["Content-Disposition"]
    at storeHeader (node:_http_outgoing:580:5)
    at processHeader (node:_http_outgoing:575:3)
    at ServerResponse._storeHeader (node:_http_outgoing:451:11)
    at ServerResponse.writeHead (node:_http_server:422:8)
    at module.exports (/usr/src/app/node_modules/serve-handler/src/index.js:754:11)
    at serve (file:///usr/src/app/quartz/cli/handlers.js:345:9)

To reproduce
Add an image with the following filename to Obsidian: DALL·E 2024-01-17.png by dropping it into an Obsidian note.
Note the bullet point in "DALL·E".
The workaround is to remove the bullet point from the filename, and I've tried it with different files.
Expected behavior
HTML page renders with image embedded.
Desktop

Quartz Version: HEAD of v4 (b22bcd17b4301c6ef398124eec11f6dff0c2d37d)
node Version: 20.11.0 (Dockerfile: FROM node:20-slim)
npm version: 10.2.4
OS: MacOs 12.7.2
Browser: Firefox 121.0.1
Obsidian version: 1.5.3
Obsidian theme: Minimal

Probably worth underlining that Obsidian is saving the file to MacOS, and Quartz is reading it from an Alpine Linux-based Docker image. Slim chance there's filesystem mapping weirdness.
Additional context
I think https://www.npmjs.com/package/content-disposition might be the apppropriate solution.

Confirm : I have the same issue sometimes.

Still broken in af3a4ff. Same file, same error.

I poked at it a bit, and I think the problem's in serve-handler. I raised a ticket there: https://github.com/vercel/serve-handler/issues/205

can you try as of b014d060f3f16b2460da08cdf88bef25d05d5d3b ? af3a4ff is before

(but yes, thank you for the upstream report that would help us here too!)

Hah. I pulled the v4 branch from origin, not upstream, when testing. Sorry about that.
Now testing ebf429a, which is the current HEAD of v4. I definitely see your fix in the git log.
Confirmed working :)


Problems with deepl

Deepl Translator does not work

And is it possible to use Yandex Translator? I can't select it from the menu.

config file
[Service]
Endpoint=YandexTranslate
FallbackEndpoint=GoogleTranslateV2


How can configure the src of blueshift with self hosted blueshift.js

https://github.com/segment-integrations/analytics.js-integration-blueshift/blob/bcb42b25ed092c4c250d3be5a3783396087b0344/lib/index.js#L19

Hi @metrue, as part of the monorepo migration, this issue has been moved to new issue. Our engineers have been notified and will prioritize and work on it ASAP. Thank you!
For more information, see README.md.


vm-monitor: Add 'metrics' request/response messages

Problem
The prometheus output provided by node_exporter / vector is a lot more than we need, and parsing it in the autoscaler-agent (because that's currently how we get metrics) is a little hacky. We already maintain communication with the vm-monitor, so let's just fetch the metrics there too — this also means we can more easily customize the information in the future, like providing stats relevant to the neon-postgres cgroup.
This should also help somewhat with providing per-endpoint metrics.
Summary of changes
This repo's half of neondatabase/autoscaling#521, allowing the autoscaler-agent to communicate directly with the vm-monitor to fetch metrics rather than pulling all of the prometheus output from vector/node_exporter.
Checklist before requesting a review

[ ] I have performed a self-review of my code.
[ ] If it is a core feature, I have added thorough tests.
[ ] Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
[ ] If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

[ ] Do not forget to reformat commit message to not include the above checklist

Meant to comment: Closing because I no longer think this is necessary / worthwhile. All the metrics are already available from vector running inside the VM, and it's better to pipe new metrics through there anyways.


How to identify the one-to-one correspondence between data and labels in custom pipeline

I want to write a custom pipeline in mmdet.
so I print all the keys from the input parameter 'input_dict'
but I can't figure out how to determine which labels belong to which image
Let's say the shape of input_dict['gt_bboxes'] is (7, 4), and the shape of input_dict['img'] is (375, 1242, 3)

More details, please.

Sorry for the unclear expression.
if I want to write a custom pipeline function, I would use the parameter 'input_dict' of the function 'call'. My question is: Does each input_dict contain only information from one sample? (such as input_dict['img'], input_dict['gt_boxes'])
在自定义的pipeline函数中，输入参数input_dict只是一个样本的信息吗？还是整个batch的信息？
因为在我设置batch_size等于2的时候，打印input_dict['img']的shape显示的是两个(375, 1242, 3) 而不是 (2, 375, 1242, 3)

input_dict is just the information of one sample, each sample will pass through the pipeline, then form the batch.

thank you so much for your kind answer.


fle-dump: a fast leader election algorithm sniffer

See README for details on usage.
Signed-off-by: Raul Gutierrez S<EMAIL_ADDRESS>

Coverage decreased (-1.46%) when pulling 3dad73ab4c0c1f222f3c10664199f68e979e7124 on rgs1:fle-dump into 01ec4110fb7089d334aaf86525ce183dd61b754e on twitter:master.

Coverage decreased (-1.46%) when pulling 3dad73ab4c0c1f222f3c10664199f68e979e7124 on rgs1:fle-dump into 01ec4110fb7089d334aaf86525ce183dd61b754e on twitter:master.

most comments addressed, thx @chchen!

Coverage decreased (-1.4%) when pulling 8d8392a4195f6ae1bb77dc066afa86b52d0a311f on rgs1:fle-dump into 01ec4110fb7089d334aaf86525ce183dd61b754e on twitter:master.

Coverage increased (+1.75%) when pulling 62b6b7fc77b39715f01b602ab8af738b8a28842d on rgs1:fle-dump into 01ec4110fb7089d334aaf86525ce183dd61b754e on twitter:master.

Coverage decreased (-1.09%) when pulling 966787ba2eb2f436add56ef0cfd6552ef4244551 on rgs1:fle-dump into 01ec4110fb7089d334aaf86525ce183dd61b754e on twitter:master.

Coverage decreased (-2.36%) when pulling 92ba0913fc4aa6b4a917fbdc9b15307b1ae3214c on rgs1:fle-dump into 01ec4110fb7089d334aaf86525ce183dd61b754e on twitter:master.

Coverage decreased (-2.49%) when pulling d4bb17234d7f06d6375a90fd72862b48369bc603 on rgs1:fle-dump into 01ec4110fb7089d334aaf86525ce183dd61b754e on twitter:master.


resolve problem: new page can't handle click event!

Using FlipHorizontalTransformer with viewPager.
The next fragment can't handle touch event after switch page, because the last fragment is above the next fragment!

Same issue with FlipVerticalTransorfmer

Sorry I completely forgot to merge this.
In the future please try to only commit actual changes. You reformatted lines unrelated to the commit in this instance.
It is also worth noting your comment in onPostTransform only makes sense in the context of this PR and does not provide context for any future readers. It would be best to open issues and state the conditional resolves the related issue number so full context of history is retained.

@ToxicBakery Got it.


Add "multiple endpoint implementation" for registry integration

See issue refs #272
Updated the configuration so that we use a list right now.
Still backward compatible.

conflicts are resolved now


fix(plugins/plugin-client-common): grammy command should enforce UI

Fixes #7277

Description of what you did:

My PR is a:

[ ] 💥 Breaking change
[x] 🐛 Bug fix
[ ] 💅 Enhancement
[ ] 🚀 New feature

Please confirm that your PR fulfills these requirements

[x] Multiple commits are squashed into one commit.
[x] The commit message follows Conventional Commits, which allows us to autogenerate release notes; e.g. fix(plugins/plugin-k8s): fixed annoying bugs
[x] All npm dependencies are pinned.

/lgtm


AttributeError: 'NoneType' object has no attribute 'endpoint'

I use the docker command provided in the Readme. I keep getting the following error:
Traceback (most recent call last):
  File "dynamodb-copy-table.py", line 22, in <module>
    ddbc = DynamoDBConnection()
  File "/usr/local/lib/python3.6/site-packages/boto/dynamodb2/layer1.py", line 173, in __init__
    kwargs['host'] = region.endpoint
AttributeError: 'NoneType' object has no attribute 'endpoint'

I have initialized all the required environment variables.

Hello,
Changed this ( line 19 - 23 ) if it can help someone else !
host = 'dynamodb.%s.amazonaws.com' % region
ddbc = DynamoDBConnection(is_secure=False, region=region, host=host)
Setting the default region name for the connection.
DynamoDBConnection.DefaultRegionName = region
ddbc = DynamoDBConnection()
Don't forget to set AWS_DEFAULT_REGION in ENV


Reopen "Background color does not seems to be working on monaco.editor.defineTheme rules section."

Reproducible in vscode.dev or in VS Code Desktop?

[X] Not reproducible in vscode.dev or VS Code Desktop

Reproducible in the monaco editor playground?

[ ] Not reproducible in the monaco editor playground

Monaco Editor Playground Link
https://microsoft.github.io/monaco-editor/playground.html?source=v0.47.0#XQAAAAJyAgAAAAAAAABBqQkHQ5NjdMjwa-jY7SIQ9S7DNlzs5W-mwj0fe1ZCDRFc9ws9XQE0SJE1jc2VKxhaLFIw9vEWSxW3yscw14OcO_1uhs0-3YGTBOsVJ4S4FwfCDQb5-2uUHXTVAefbVSd9IzaZ_vv9VR9KugsKimARHSWlrTPOQHROITHxmAetfeRrm3VHdSe7_Etv6WpgzlHEUT3FrRpJfU0nW28tDtD5OgHyVk3rnhWlhuIaNhLo010ZK-WajoxXyf8yld6yB_zV61-qrJJVvX2fOZZoh_-3dcSQHKIXTuZgxdHu8cns1UZ0GMNHQA-O2TFz4_SMRIVm30405EwLqyMZpjvQXGx7egWwlX_HGHT1Y4NU5BLFtcHYDGPWCIpXFTQav22jRH50VfvJkO0X9Pz_ogXrRq3doHhh4i_vuhdErrQ4gMSS2Hu-vtmRpHDjLUKv2h-gIwifxcdA8Zld6wbgPQtfE5a8ql7gJW6mkCZoONZVBc8ji9DG6_8GAfKNa-HnvXcQZdYTQ7Vgyxe5cdI-YPPUJZNkSqkAUvJeXike5yebu8SZtmYKlFNIoLNihdfAJ_hKmaY4TZkof__3MUYf
Monaco Editor Playground Code
const value = /* set from `myEditor.getModel()`: */ `function hello() {
	alert('Hello world!');
	// this comment should have a background
}`;

const mytheme = {
    base: 'vs',
    rules: [
        { token: 'comment', foreground: "ff0000", background: "ff00ff" },
    ],
    inherit: false,
    colors: {},
}

monaco.editor.defineTheme("myTheme", mytheme)

// Hover on each property to see its docs!
const myEditor = monaco.editor.create(document.getElementById("container"), {
	value,
	language: "javascript",
	automaticLayout: true,
	theme: "myTheme",
});

Reproduction Steps
Background color does not seems to be working on monaco.editor.defineTheme rules array section. I am being able to set the foreground color and fontStyle, but for some reason the background color does not work. Any insight would be greatly appreciated.
Actual (Problematic) Behavior
No response
Expected Behavior
No response
Additional Context
This issue is a reopening of the following issue, that was closed without solution:
https://github.com/microsoft/monaco-editor/issues/586

linking https://github.com/microsoft/monaco-editor/issues/4405

linking https://github.com/microsoft/monaco-editor/issues/207

We have the same issue: the background property is in the API for theme rules, but doesn't work.

Joining the team of people that need this feature.
I can temporarily try to replace it with decorations, but it's a mess.


Native parallel batch indexing with shuffle

Motivation
General motivation for native batch indexing is described in https://github.com/apache/incubator-druid/issues/5543.
We now have the parallel index task, but it doesn't support perfect rollup yet because of lack of the shuffle system.
Proposed changes
I would propose to add a new mode for parallel index task which supports perfect rollup with two-phase shuffle.
Two phase partitioning with shuffle

Phase 1: each task partitions data by segmentGranularity and then by hash or range key of some dimensions.

Phase 2: each task reads a set of partitions created by the tasks of Phase 1 and creates a segment per partition.
PartitionsSpec support for IndexTask and ParallelIndexTask
PartitionsSpec is the way to define the secondary partitioning and is currently being used by HadoopIndexTask. This interface should be adjusted to be more general as below.
public interface PartitionsSpec
{
  @Nullable
  Integer getNumShards();

  @Nullable
  Integer getMaxRowsPerSegment(); // or getTargetRowsPerSegment()

  @Nullable
  List<String> getPartitionDimensions();
}

Hadoop tasks can use an extended interface which is more specialized for Hadoop.
public interface HadoopPartitionsSpec extends PartitionsSpec
{
  Jobby getPartitionJob(HadoopDruidIndexerConfig config);
  boolean isAssumeGrouped();
  boolean isDeterminingPartitions();
}

IndexTask currently provides duplicate configurations for partitioning in its tuningConfig such as maxRowsPerSegment, maxTotalRows, numShards, and partitionDimensions. These configurations will be deprecated and the indexTask will support PartitionsSpec instead.
To support maxRowsPerSegment and maxTotalRows, a new partitionsSpec could be introduced.
/**
 * PartitionsSpec for best-effort rollup
 */
public class DynamicPartitionsSpec implements PartitionsSpec
{
  private final int maxRowsPerSegment;
  private final int maxTotalRows;
}

This partitionsSpec will be supported as a new configuration in the tuningConfig of IndexTask and ParallelIndexTask.
New parallel index task runner to support secondary partitioning
ParallelIndexSupervisorTask is the supervisor task which orchestrates the parallel ingestion. It's responsible for spawning and monitoring sub tasks, and publishing created segments at the end of ingestion.
It uses ParallelIndexTaskRunner to run single-phase parallel ingestion without shuffle. To support two-phase ingestion, we can add a new implementation of ParallelIndexTaskRunner, TwoPhaseParallelIndexTaskRunner. ParallelIndexSupervisorTask will choose the new runner if partitionsSpec in tuningConfig is HashedPartitionsSpec or RangePartitionsSpec.
This new taskRunner does the followings:

Add TwoPhasesParallelIndexTaskRunner as a new runner for the supervisor task

Spawns tasks for determining partitions (if numShards is missing in tuningConfig)
Spawns tasks for building partial segments (phase 1)
When all tasks of the phase 1 finish, spawns new tasks for building the complete segments (phase 2)
Each Phase 2 task is assigned one or multiple partitions

The assigned partition is represented as an HTTP URL

Publish the segments reported by phase 2 tasks.
Triggers intermediary data cleanup when the supervisor task is finished regardless of its last status.

The supervisor task provides an additional configuration in its tuningConfig, i.e., numSecondPhaseTasks or  inputRowsPerSecondPhaseTask, to support control of parallelism of the phase 2. This will be improved to automatically determine the optimal parallelism in the future.
New sub task types
Partition determine task

Similar to what indexTask or HadoopIndexTask do.
Scan the whole input data and collect HyperLogLog per interval to compute approximate cardinality.
numShards could be computed as below:

        numShards = (int) Math.ceil(
            (double) numRows / Preconditions.checkNotNull(maxRowsPerSegment, "maxRowsPerSegment")
        );

Phase 1 task

Read data via the given firehose
Partition data by segmentGranularity by hash or range (and aggregates if rollup)
Should be able to access by (supervisorTaskId, timeChunk, partitionId)
Write partitioned segments in local disk. Multiple disks can be configured, and each task would write partitions in a round-robin manner to utilize disk bandwidth efficiently

Phase 2 task

Download all partial segments from middleManagers where phase 1 tasks ran.
Merge all fetched segments into a single segment per partitionId.
Push the merged segments and report them to the supervisor task.

MiddleManager as Intermediary data server
MiddleManager (and new Indexer) should be responsible for serving intermediary data during shuffle.
Each phase 1 task partitions input data and generates partitioned segments. These partitioned segments are stored in local disk of middleManager (or indexer proposed in https://github.com/apache/incubator-druid/issues/7900). The partitioned segment location would be /configured/prefix/supervisorTaskId/ directory. The same configurations with StorageLocationConfig would be provided for intermediary segment location.
MiddleManagers and indexers would clean up intermediary segments using the below mechanism.

MM will keep expiration time in memory. This expiration time is initialized with current time + configured timeout.
MM periodically checks there are any new partitions created for new supervisorTasks and initializes the expiration time if it finds any.
When a subtask accesses a partition, the expiration time for the supervisorTask is initialized or updated if it's already there.
MM periodically checks those expiration times for supervisorTasks. If it finds any expired supervisorTask, then it will ask the overlord if the task is still running. If not, MM will remove all partitions for the supervisorTask.
The overlord will also send a cleanup request to MM when the supervisorTask is finished. This will clean up the expiration time.

New API lists of MiddleManager

GET /druid/worker/v1/shuffle/tasks/{supervisorTaskId}/partition?start={startTimeOfSegment}&end={endTimeOfSegment}&partitionId={partitionId}

Return all partial segments generated by sub tasks of the given supervisor task, falling in the given interval, and having the given partitionId.

DELETE /druid/worker/v1/shuffle/tasks/{supervisorTaskId}

Removes all partial segments generated by sub tasks of the given supervisor task.
New metrics & task reports

ingest/task/time: how long each task took
ingest/task/bytes/processed: how large data each task processed
ingest/shuffle/bytes: how large data each middleManager served
ingest/shuffle/requests: how many requests each middleManager served

Task failure handling
Task failure handling is same with the current behavior.

If the supervisorTask process is killed normally, stopGracefully method is called which kills all running subtasks. If it's killed abnormally, then parallel index task doesn't handle this case for now.
SupervisorTask monitors subtask statuses and counts how many subtasks have failed to process the same input. If it notices more failures than configured maxRetry, it regards that input can't be processed and exists with an error. Otherwise, it respawns a new task which processes the same input.

Rationale
There could be two alternate designs for the shuffle system, especially for intermediary data server.
MiddleManager (or indexer) as intermediary data server is the simplest design. In an alternative design, phase 1 tasks could serve intermediary data for shuffle. In this alternate, phase 1 tasks should be guaranteed to run until the phase 2 is finished, which means task 1 resources should be held until the phase 2 is finished. This is rejected for better resource utilization.
Another alternate is a single set of tasks would process both phase 1 and phase 2. This design is rejected because it's not very flexible to use cluster resource efficiently.
Operational impact
maxRowsPerSegment, numShards, partitionDimensions, and maxTotalRows in tuningConfig will be deprecated for indexTask. partitionsSpec will be provided instead. The deprecated values will be removed in the next major release after the upcoming one.
Test plan
Unit tests and integration tests will be implemented. I will also test this with our internal cluster once it's ready.
Future work

The optimal parallelism for the phase 2 should be able to be determined automatically by collecting statistics during the phase 1.
To avoid "too many open files" problem, middleManager should be able to smoosh the intermediary segments into several large files.
If rollup is set, it could be better to combine intermediate data in middleManager before sending them. It would be similar to Hadoop's combiner.

This could be implemented to support seamless incremental segment merge in middleManager.

In Phase 1, tasks might skip index generation for faster shuffle. In this case, Phase 2 tasks should be able to generate the complete indexes.

SGTM in general

When the supervisor task is finished (either succeeded or failed), the overlord sends cleanup requests with supervisorTaskId to all middleManagers (and indexers)

does overlord treat "supervisor" task as special task to be able to initiate cleanup requests? what if the MM is down temporarily or is cleanup fails for some reason ? In addition to overlord cleanup requests, It might be good for middleManagers to periodically check whether "supervisor" task is running or not and do the self cleanup.
also maybe have some MM level configuration around maximum disk space that can be utilized for intermediary data.

The supervisor task provides an additional configuration in its tuningConfig, i.e., numSecondPhaseTasks or inputRowsPerSecondPhaseTask, to support control of parallelism of the phase 2. This will be improved to automatically determine the optimal parallelism in the future.

I think a user defined upper limit could always exist in all "supervisor" tasks that spawn extra tasks so that user can plan worker capacity knowing how many tasks at a maximum would be running via parallel [shuffle] task.

@himanshug thanks for taking a look!

does overlord treat "supervisor" task as special task to be able to initiate cleanup requests? what if the MM is down temporarily or is cleanup fails for some reason ? In addition to overlord cleanup requests, It might be good for middleManagers to periodically check whether "supervisor" task is running or not and do the self cleanup.

Ah this is a good point. To handle middleManager failure, a sort of self-cleanup can be triggered when some amount of time is elapsed since the last access to any partition for a supervisorTask. Does this sound good?

also maybe have some MM level configuration around maximum disk space that can be utilized for intermediary data.

Thanks for reminding me of this. Forgot to add it to the proposal. I'm thinking to use the existing StorageLocationConfig for this. To fully utilize the disk bandwidth, the partitions of the same supervisorTaskId will be assigned in a round-robin fashion. Will update the proposal shortly.

I think a user defined upper limit could always exist in all "supervisor" tasks that spawn extra tasks so that user can plan worker capacity knowing how many tasks at a maximum would be running via parallel [shuffle] task.

This is already supported with maxNumSubTasks (https://druid.apache.org/docs/latest/ingestion/native_tasks.html#tuningconfig). maxNumSubTasks is to limit the total number of subtasks at any time while a parallel index task is running. numSecondPhaseTasks is somewhat different. It's the total number of phase 2 tasks and the supervisor task will regard the phase 2 is succeeded once numSecondPhaseTasks phase 2 tasks are succeeded.

To handle middleManager failure, a sort of self-cleanup can be triggered when some amount of time is elapsed since the last access to any partition for a supervisorTask. Does this sound good?

anything is good as long as data is not left behind forever in error scenarios :) , that said it might be more work to track access time of the partition...

are you planning on using the OS managed file access time ? note that many FS are configured to not update access time on reads due to associated IO overhead
are you planning to track access time in MM memory ? what happens if MM process gets restarted for some reason and then it will lose track of that.
I was imagining a simpler world where MM just periodically scans directories where partition files could be stored, accumulates a list of supervisor taskIds(for which some data is stored), makes a call to overlord to get state of all those tasks and then based on state(failed,completed etc) returned deletes data for those. that said, anything is fine really.

it would be extra nice to document the failure cases and expected behavior e.g.

supervisor task process (or MM running it) crashed while phase1/2 tasks were running
one or more of phase1 tasks crashed
one or more of phase2 tasks crashed
will we try to recover in some of those cases? for example if one or more phase1/2 tasks crashed, will supervisor task retry them ?are any/all of these tasks restorable i.e. return true for canRestore() ?

it is not necessary for things to be recoverable as most of this would be iterated upon later but we can just document what to expect in such failure scenarios.

does overlord treat "supervisor" task as special task to be able to initiate cleanup requests?

Oh, BTW, there's nothing special with supervisorTasks. I'm thinking to add a callback system to indexing service, like some callback functions can be executed at some predefined stage, e.g., after segments are published or some tasks are finished.

are you planning to track access time in MM memory ? what happens if MM process gets restarted for some reason and then it will lose track of that.

Yes, this is what I'm thinking. What I'm thinking is pretty similar to yours, but a bit additional stuffs.

MM will keep expiration time in memory.
This expiration time is initialized with current time + configured timeout.
MM periodically checks there are any new partitions created for new supervisorTasks and initializes the expiration time if it finds any.
When a subtask accesses a partition, the expiration time for the supervisorTask is initialized or updated if it's already there.
MM periodically checks those expiration times for supervisorTasks. If it finds any expired supervisorTask, then it will ask the overlord if the task is still running. If not, MM will remove all partitions for the supervisorTask.
The overlord will also send a cleanup request to MM when the supervisorTask is finished. This will clean up the expiration time.

I think it's not very complex, but will reduce the number of calls to overlord API, so it would be good.

it would be extra nice to document the failure cases and expected behavior e.g.

Ah, I didn't document them since it would be same with the existing parallel index task behavior.
Here are descriptions on how it handles failures currently.

supervisor task process (or MM running it) crashed while phase1/2 tasks were running

If the supervisorTask process is killed normally, stopGracefully method is called which kills all running subtasks. If it's killed abnormally, then parallel index task doesn't handle this case for now.

one or more of phase1/2 tasks crashed

SupervisorTask monitors subtask statuses and counts how many subtasks have failed to process the same input. If it notices more failures than configured maxRetry, it regards that input can't be processed and exists with an error. Otherwise, it respawns a new task which processes the same input.
I'll add these to the proposal.

are any/all of these tasks restorable i.e. return true for canRestore() ?

Good point. Any task is not restorable now, but I think it might be useful to support rolling update in the future. Parallel index tasks are supposed to run for a long time and so it would be nice if it can be stopped/restored during rolling update.

thanks for the explanations.

I think it's not very complex, but will reduce the number of calls to overlord API, so it would be good.

that is fine and I am guessing it will batch task status call i.e. ask status of multiple supervisor tasks in one overlord api request.

If supervisorTask killed abnormally, then parallel index task doesn't handle this case for now.

will the worker tasks complete and exit eventually or they will be left running till manual intervention ?
we might consider making supervisor real "supervisor" like Kakfa instead of "task"  so that they get special powers to manage things better ? but I think I am remembering some comment that they are made tasks because tasks are better equipped to work with locking framework available. one option could be to let spawned worker task do the locking , since all worker tasks would be in same group so multiple worker tasks trying to obtain same lock would still work..... this is unverified wishful thinking :)

that is fine and I am guessing it will batch task status call i.e. ask status of multiple supervisor tasks in one overlord api request.

Correct.

will the worker tasks complete and exit eventually or they will be left running till manual intervention ?

All subtasks report the pushed segments to the supervisor task at the end of indexing. So, if the supervisor task is not running, then they would end up being failed at this stage. However, this is still quite annoying since they will occupy middleManager resources unnecessarily. I guess we could add some health check to subtasks for the supervisor task.

we might consider making supervisor real "supervisor" like Kafka instead of "task" so that they get special powers to manage things better ? but I think I am remembering some comment that they are made tasks because tasks are better equipped to work with locking framework available. one option could be to let spawned worker task do the locking , since all worker tasks would be in same group so multiple worker tasks trying to obtain same lock would still work..... this is unverified wishful thinking :)

Yeah, my original intention was to add a "supervisor" like Kafka rather than a new task type, but I changed my mind to use the existing task lock type system. And now I'm inclined to keep the current design of supervisor task because

"supervisor" is designed for more like stream ingestion for each dataSource. It runs forever once it's submitted and the history of its spec changes is recorded in metadata store.
"supervisor" design is less scalable since the overlord handles all supervisors and requests/responses for their tasks. One of our customers already had some issue with this. They were running more than 1000 tasks for each Kafka supervisor. Kafka ingestion got stuck because their overlord couldn't handle too many HTTP requests from tasks in time.

For the second reason, I would upvote to even demote the Kafka/Kinesis supervisor to the supervisor task.

thanks , that reasoning about supervisors makes sense.

I guess we could add some health check to subtasks for the supervisor task.

for their early termination, yes .

Thanks! I updated the proposal to include things discussed.

@jihoonson : Any thoughts about considering making the supervisor task responsible for serving intermediary data during shuffle instead of the Middlemanager process ?
I see MMs as very lightweight process responsible for spawning, monitoring and cleaning up intermediate stuff and not handling the data itself.

Went through the proposal, I see one issue with this which needs more thought,
Above approach may break with autoscaling -
Consider the case when phase1 task ran on a specific middlemanager and finished. (this was the only task running on it), the autoscaler decided to kill this MM as there is no task running on it.
Phase 2 tasks running will be able to locate the data from phase 1 task.
Am i understanding it correctly, or it is supposed to work with current design ?

@nishantmonu51 good point! We currently have two provisioning strategies for auto scaling, i.e., simple and pendingTaskBased, and both of them look to terminate middleManagers if it has been a long time since they completed the last task. I think there could be two options available to avoid this issue.

Improve the provisioning strategy to consider intermediary data in middleManagers. If they are still serving intermediary data for parallel batch tasks, then the auto scaler shouldn't terminate them.
As you mentioned, store intermediary data on the deep storage.

I'm inclined to the first approach because 1) it's more efficient to read data from middleManagers than from deep storage and 2) intermediary data cleanup for deep storage could be more complex than that for middleManagers (it's still doable though). What do you think?
BTW, no matter what way we go, I think this issue could be fixed in a follow-up PR. Does this make sense?

Regarding the autoscaling, there are users who have developed autoscaling strategies outside druid e.g kubernetes users, use simple Horizontal Pod Autoscaler rules and keeping the autoscaling strategies simple help them write the rules easily.
Thinking more on the lines of deep storage,

data cleanup could be easy if we follow a hierarchy, e.g the baseTaskDir/supervisor-task-id of the supervisor task can serve as the base path for the intermediary location and MM can just ensure that the base path and any underlying sub-dirs are cleaned up when the supervisor task fails.
I see MM as very lightweight processes that have the responsiblity of orchestration of peons, monitoring and cleaning any leftover files/data. It would be great if we can keep it that way.
If we use deep storage as handoff for intermediate segments, we probably do not need any additional complexity in the MiddleManager to serve as an Intermediary Data Server.
Keeping MM simple would optionally keep us open to expanding our TaskRunners to directly leverage modern container orchestration platforms like kubernetes, marathon, Yarn, Yunicorn etc, where TaskRunners schedule peons directly in e.g. a kubernetes cluster. See Below Issues, although marked as stale, but i still believe there is value in those proposals -

Autoscaler for Marathon - https://github.com/apache/incubator-druid/issues/3451
Druid on YARN - https://github.com/apache/incubator-druid/issues/4400

Efficiency wise I agree that reading data from deep storage (especially when using s3) would be slower than reading it from another MM, but till now I have seen creation of Druid indexes and final segments to be the major bottleneck instead of data transfer.
What do you think ?
Agreed, it can be done in a follow-up PR.

I think two big changes are happening in Druid's indexing service recently. One would be the native parallel indexing and another could be the new Indexer module (https://github.com/apache/incubator-druid/issues/7900). I think this Indexer module would be better than the middleManager in terms of memory management, monitoring, resource scheduling and sharing and hope it would be able to replace the middleManager in the future. It would be nice if we can keep the current behavior for users even with these changes, but I think it wouldn't be easy to keep the current behavior especially with the Indexer.

data cleanup could be easy if we follow a hierarchy, e.g the baseTaskDir/supervisor-task-id of the supervisor task can serve as the base path for the intermediary location and MM can just ensure that the base path and any underlying sub-dirs are cleaned up when the supervisor task fails.

Yeah, it would be pretty similar to when using MM as the intermediary data server with respect to intermediary data structure and management. But I think it could be more complex when it comes to the operational stuffs like permission or failure handling. Not only middleManager but the overlord should also be able to remove stale intermediary data because the middleManager could miss the supervisor task failure. The attempt to delete intermediary data could also fail and it would be easier if it's a local disk. I mean, it's still doable with deep storage but it would be a bit more complex.

I see MM as very lightweight processes that have the responsiblity of orchestration of peons, monitoring and cleaning any leftover files/data. It would be great if we can keep it that way.

Hmm, the additional functionality would be just serving intermediary data and cleaning up them periodically. Maybe the middleManager would need a bit more memory but it shouldn't be really big. Would you elaborate more about your concern?
Looking at the code around AutoScaler and Provisioner, if the overlord can collect the remaining intermediary data on each middleManager, this information could be used for auto scaling. I guess this information could be easily added to existing code base since the overlord is already collecting some data from middleManagers (like host name and port, etc). And since the AutoScaler and Provisioner are executed in the overlord process, I guess we can easily provide this information to the Provisioner. Looks like the implementation wouldn't be not that difficult?

Efficiency wise I agree that reading data from deep storage (especially when using s3) would be slower than reading it from another MM, but till now I have seen creation of Druid indexes and final segments to be the major bottleneck instead of data transfer.

That sounds interesting. I haven't used Hadoop ingestion much, and I don't know how much data transfer would contribute to the total ingestion performance. Do you have any rough number on this?

https://github.com/apache/incubator-druid/issues/6533 looks another issue in auto scaling. Looks like it needs more information about middleManager state anyway.

Keeping MM simple would optionally keep us open to expanding our TaskRunners to directly leverage modern container orchestration platforms like kubernetes, marathon, Yarn, Yunicorn etc, where TaskRunners schedule peons directly in e.g. a kubernetes cluster.

Thinking more about this, it could be easier and more useful for these users if intermediary data is stored in deep storage. I think we can make this pluggable, like using MM or deep storage for intermediary data depending on the configuration. Does this make sense?

Could I use index_parallel task to reindex my segments for multitenancy case? Just like single-dimension partitioning by tenant_id.  If then, how should I config with turningConfig option?
I had a try but it does not work.

Hi @quenlang, you should be able to do it. An example tuningConfig could be
    "tuningConfig" : {
      "type" : "index_parallel",
      "maxNumConcurrentSubTasks": 2, // max number of sub queries that can be run at the same time
      "forceGuaranteedRollup": true,
      "partitionsSpec": {
        "type": "hashed",
        "numShards": 3 // number of segments per time chunk after compaction
      }
    }

@jihoonson Thanks a lot!

@jihoonson is there any protection against getting inside a deadlock for parallel task implementation. For example, and simplicity, consider case of single phase parallel task ... say user submitted 10 parallel tasks .. each of which are supposed to create 2 sub tasks ... Now say you have only 8 task slots (all of which are occupied by supervisor tasks) ... they will submit subtasks which will never run .... is this situation possible or is there any protection to ensure this deadlock doesn't happen. I am noticing something like that on a cluster, not entirely sure.

@himanshug good point. I don't think we have any protection against this kind of scenario yet. One possible short term workaround could be checking timeout for pending sub tasks.  In the future, we may need to improve task scheduling to consider task type to avoid this kind of deadlock.

@jihoonson hmmm, that solution will fail the indexing ... if there is a lot of parallel indexing tasks submitted (relative to total available task slots) at same time ... indexing progress would be very very slow with many of them failing.
One workaround could be to make sure that all supervisor tasks goto a separate set of middleManagers dedicated to running only supervisor tasks (and no other tasks are sent to these) ... we could probably use https://github.com/apache/incubator-druid/pull/7066#issuecomment-536298861 to achieve that.
that said, above workaround isn't great for usability in general as simple druid cluster setup should just work.
In the long run, I think we need to do something to give special treatment to supervisor tasks in general and run them differently without occupying the task slots.

In the long run, I think we need to do something to give special treatment to supervisor tasks in general and run them differently without occupying the task slots.

Interesting. Another possible workaround for now is you can explicitly specify taskResource to have 0 requiredCapacity for supervisor tasks.
For the long run idea, I'm not sure if it's fine to not reserve any resource, or it should reserve a minimum amount of resource. Maybe we need to distinguish different resource types (e.g., CPU, memory, disk, etc) to schedule tasks better.

with 0 requiredCapacity overlord would try and send them to MM but MM would reject them saying its full, no ?

For the long run idea, I'm not sure if it's fine to not reserve any resource, or it should reserve a minimum amount of resource. Maybe we need to distinguish different resource types (e.g., CPU, memory, disk, etc) to schedule tasks better.
yes, I meant reserve something else and not task slots i.e. maybe a "supervisor slot" concept.
For other reliability reasons, maybe also a first class "Supervisor" class like "Task" ... handling of "Supervisor" could be different than that of a Task e.g. all Supervisors are always restartable , unlike Tasks they are not failed if a MM running them dies and instead they are re-run etc.

with 0 requiredCapacity overlord would try and send them to MM but MM would reject them saying its full, no ?

Hmm, I just tested with RemoteTaskRunner and it worked. Is MM supposed to reject it?

yes, I meant reserve something else and not task slots i.e. maybe a "supervisor slot" concept.

Sounds good. I was thinking a task scheduler respecting task priority. Like, it can schedule high priority tasks more often, but also should be able to avoid starvation for low priority tasks. Maybe the supervisor slot concept can go along with this idea together.

For other reliability reasons, maybe also a first class "Supervisor" class like "Task" ... handling of "Supervisor" could be different than that of a Task e.g. all Supervisors are always restartable , unlike Tasks they are not failed if a MM running them dies and instead they are re-run etc.

Also sounds good.

with 0 requiredCapacity overlord would try and send them to MM but MM would reject them saying its full, no ?

Hmm, I just tested with RemoteTaskRunner and it worked. Is MM supposed to reject it?

looked at the code and workers just do whatever overlord tells them, so behavior you saw is expected and required to fulfill the contract of requiredCapacity = 0. however, that means overlord would just keep assigning supervisor tasks to MMs as they come . Even if requiredCapacity = 0, in reality they do take resources on the MM and would eventually kill it if enough number of those show up.

@jihoonson I am thinking of adding a tuning config in ParallelIndexTuningConfig , minSplitsForParallelMode . ParallelIndexSupervisorTask would run parallel only if firehoseFactory.getNumSplits() >= minSplitsForParallelMode or else it will run sequentially.
default value of minSplitsForParallelMode=0 to retain existing behavior .
in my case, I would set minSplitsForParallelMode=2. most tasks have numSplits = 1 so they will just run sequentially and only few of them would really run parallel with subtasks , that would reduce the likelihood of deadlock. also it is probably more efficient to not go parallel if there is just 1 split.
what do you think ?

nevermind, I tried above and it didn't help my case as contrary to my thinking, most tasks had 2 splits.
running the supervisors on separate set of workers using #7066 is the only decent thing to do for now.

Hi @quenlang, you should be able to do it. An example tuningConfig could be
    "tuningConfig" : {
      "type" : "index_parallel",
      "maxNumConcurrentSubTasks": 2, // max number of sub queries that can be run at the same time
      "forceGuaranteedRollup": true,
      "partitionsSpec": {
        "type": "hashed",
        "numShards": 3 // number of segments per time chunk after compaction
      }
    }

Hi @jihoonson, I split data into different segments by single dimension tenant_id for the multitenancy scene. In this way, I can get higher query performance that filters on the tenant_id dimension. But tenant data was skew, so the segment size was not perfect. For example, the max segment size was nearly 18GB but the min segment size was 5MB. Then perform queries on the 18GB segment ware more slower than that partitioned by all dimensions.
Assume there is a way(eg. maxRowsPerSegment=10000000) can split the max segment into some more smaller segments, the queries will be quick. But maxRowsPerSegment can not exist when specify numShards option.
How do you think about this scene?
Thanks a lot!

Hi @quenlang, I assume you're using hadoop indexing task since the parallel indexing task doesn't support single-dimension range partitioning yet. I think you can set targetPartitionSize and maxPartitionSize to something more optimal. Note that the single-dimension range partitioning doesn't support numShards.

@jihoonson Sorry for I did not describe clearly. I used the parallel indexing task to reindex segments for single-dimension range partitioning.
                "tuningConfig": {
                        "type": "index_parallel",
                        "maxNumConcurrentSubTasks": 30,
                        "partitionsSpec": {
                                "type": "hashed",
                                "numShards": 10,
                                "partitionDimensions": ["mobile_app_id"]
                        },
                        "forceGuaranteedRollup": true
                }

I got a correct result. It seems perfect except the data skew.  I'm confused with the parallel indexing task doesn't support single-dimension range partitioning yet?
Thanks!

Ah, the partitionsSpec you used is the hash-based partitioning. To use the range partitioning, the type of the partitionsSpec should be single_dim instead of hashed. This single-dimension range partitioning is supported only by the hadoop task for now and I believe the native parallel indexing task will support it in the next release.

Hi @jihoonson, I split data into different segments by single dimension tenant_id for the multitenancy scene. In this way, I can get higher query performance that filters on the tenant_id dimension.

I'm pretty surprised by this and wondering how big the performance gain was in your case. Sadly, Druid doesn't support segment pruning in brokers for hash-based partitioning for now (this is supported only for single-dimension range partitioning). That means, even though your segments are partitioned based on the hash value of tenant_id, the broker will send queries to all historicals having any segments overlapping with the query interval no matter what their hash value is. I guess, perhaps you could see some performance improvement when you filter on tenant_id maybe because of less branch misprediction. Can you share your performance benchmark result if you can?

But tenant data was skew, so the segment size was not perfect. For example, the max segment size was nearly 18GB but the min segment size was 5MB. Then perform queries on the 18GB segment ware more slower than that partitioned by all dimensions.

One popular way to mitigate the data skewness is adding other columns to the partition key, so that segment partitioning can be more well balanced. This will corrupt the locality of data so I guess you may need to find a good combination of columns for partition key.

@jihoonson  Thanks for the quick reply!

I'm pretty surprised by this and wondering how big the performance gain was in your case. Sadly, Druid doesn't support segment pruning in brokers for hash-based partitioning for now (this is supported only for single-dimension range partitioning). That means, even though your segments are partitioned based on the hash value of tenant_id, the broker will send queries to all historicals having any segments overlapping with the query interval no matter what their hash value is. I guess, perhaps you could see some performance improvement when you filter on tenant_id maybe because of less branch misprediction. Can you share your performance benchmark result if you can?

I did not get big the performance gain than expected. for the small tenant, the query latency only reduces 50ms-100ms but for the big tenant, the latency increases 10s-30s. I think it caused by the data skewness with the hashed partition of tenant_id. The biggest tenant in an 18GB segment.

That means, even though your segments are partitioned based on the hash value of tenant_id, the broker will send queries to all historicals having any segments overlapping with the query interval no matter what their hash value is.

even though the broker sends queries to all historicals, but only one historical node has the tenant data. So I think the data skewness is the root cause of large latency.

Ah, the partitionsSpec you used is the hash-based partitioning. To use the range partitioning, the type of the partitionsSpec should be single_dim instead of hashed. This single-dimension range partitioning is supported only by the hadoop task for now and I believe the native parallel indexing task will support it in the next release.

Do you mean that druid 0.17.0 will support single-dimension range partitioning in native parallel indexing tasks?
Also, if there is a big tenant in a range set of tenant_id, how to avoid segment size skewness by single-dimension range partitioning in the future native parallel indexing task?
Thanks a lot!

I did not get big the performance gain than expected. for the small tenant, the query latency only reduces 50ms-100ms but for the big tenant, the latency increases 10s-30s. I think it caused by the data skewness with the hashed partition of tenant_id. The biggest tenant in an 18GB segment.

Thank you for sharing! The performance gain does look small but still interesting.

Do you mean that druid 0.17.0 will support single-dimension range partitioning in native parallel indexing tasks?

I hope so. You can check the proposal in https://github.com/apache/incubator-druid/issues/8769.

Also, if there is a big tenant in a range set of tenant_id, how to avoid segment size skewness by single-dimension range partitioning in the future native parallel indexing task?

Hmm, sorry I didn't mean single-dimension range partitioning helps with the skewed partitioning. I'm not sure if there's a better way except adding other columns to the partition key.

@jihoonson  Thank you so much.

This is done.


Blazor Startup class does not find _Host or pages when mocked from different project

Description
If the Startup class is inherited by a TestStartup class in an integration test project, and used to launch the server, the following error is given, despite the content root being set to the main project:
System.InvalidOperationException: Cannot find the fallback endpoint specified by route values: { page: /_Host, area:  }.
   at Microsoft.AspNetCore.Mvc.RazorPages.Infrastructure.DynamicPageEndpointMatcherPolicy.ApplyAsync(HttpContext httpContext, CandidateSet candidates)
   at Microsoft.AspNetCore.Routing.Matching.DfaMatcher.SelectEndpointWithPoliciesAsync(HttpContext httpContext, IEndpointSelectorPolicy[] policies, CandidateSet candidateSet)
   at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.<Invoke>g__AwaitMatch|6_1(EndpointRoutingMiddleware middleware, HttpContext httpContext, Task matchTask)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\3.0.0-preview6.19307.2\Microsoft.AspNetCore.Http.Extensions.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware: Error: An exception was thrown attempting to execute the error handler.

It appears that the page routing is still performed relative to the TestStartup class, instead of to the content root set.
To Reproduce
Steps to reproduce the behaviour:

Using ASP.NET Core 3.0.0-preview6-27804-01 with Blazor Template Project
Create another project ("the test project") in the solution
Create a class TestStartup that inherits from Startup in the test project
Run from the test project

Host.CreateDefaultBuilder(args)
                .ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.UseStartup<TestStartup>();
                }).UseContentRoot("MainProjectPath").Build().Run();

(replace MainProjectPath with the path to the root directory of the main project)
4. Observe that a 500 internal server error is returned on any page that should be mapped, and the error given in this issue's description is found in the console
5. Observe that if TestStartup is changed to Startup the above code runs correctly without errors.
Expected behavior
The location of the Startup class should not affect the behaviour of the server, when UseContentRoot is set correctly.
Additional context
Being able to substitute the Startup class from a test project is important as it allows to change the dependency injection configured in ConfigureServices, e.g. to use mock services.
Include the output of dotnet --info
dotnet --info output:
.NET Core SDK (reflecting any global.json):
 Version:   3.0.100-preview6-012264
 Commit:    be3f0c1a03

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.17134
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\3.0.100-preview6-012264\

Host (useful for support):
  Version: 3.0.0-preview6-27804-01
  Commit:  fdf81c6faf

.NET Core SDKs installed:
  2.1.202 [C:\Program Files\dotnet\sdk]
  2.1.504 [C:\Program Files\dotnet\sdk]
  2.1.800-preview-009696 [C:\Program Files\dotnet\sdk]
  3.0.100-preview6-012264 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0-preview6.19307.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0-preview6-27804-01 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.0.0-preview6-27804-01 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Thanks for contacting us, @TheMightyMat.
We'll investigate this and get back to you as soon as we can. This may take some time, though.

@TheMightyMat I am unable to reproduce this issue. I was able to successfully access referenced project's pages and static content while running from another test project.

Can you try your scenario in the latest SDK and see if the issue persists?
if it does, can you please share a repro project so I can make sure I didn't miss anything?

@ajaybhargavb Here's a project that reproduces the issue. Change the contentRoot variable in the test project to point to the location of the main project after you clone. Run the unit test in debug mode to view the console output - you should see the same error stated above. This was run on 3.0.100-preview6-012264, will try on the latest SDK preview when I get a chance.

@TheMightyMat thanks for that. I am now able to reproduce your issue. It happens even on the latest SDK. I'll look into it.

@TheMightyMat it looks like this happens because of a mismatch in the HostingEnviroment.ApplicationName. Basically in addition to setting content root, you'll also need to set the application name to be the correct one instead of the test project.
You can do something like this. In your case, you'll pass in "MainProject" for appName
public static IHostBuilder CreateHostBuilderWithStartup<TStartup>(string[] args, string appName) where TStartup : class
{
    return Host.CreateDefaultBuilder(args)
        .ConfigureWebHostDefaults(webBuilder =>
        {
            webBuilder.UseStartup<TStartup>();

            webBuilder.ConfigureAppConfiguration((b, c) =>
            {
                b.HostingEnvironment.ApplicationName = appName;
            });
        });
}

I've verified that this fix works.
You might also want to look into https://docs.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-3.0#aspnet-core-integration-tests which is our recommended way of writing integration tests using the provided test server.


[BEAM-7095] Upgrade to RabbitMQ amqp-client 4.9.3 in RabbitMqIO

R: @aromanenko-dev
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[X] Choose reviewer(s) and mention them in a comment (R: @username).
[X] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[X] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Run Java PreCommit


Missing items: Rage of Mugamba

Links to WOTLK Wowhead for each item:
https://www.wowhead.com/classic/item=231350/rage-of-mugamba

Hey @shaugen I'm seeing this item in the warrior sim.

Double check your settings and if that doesn't work, try clicking the ⚙ in the top right then "Restore Defaults"

Found it now, thanks!

Awesome glad you were able to get it resolved!


Bump tinymqttbroker to 0.0.8

0.0.8 created 09.10.2023
6 user (7%)
OK

RE-CHECK!

0.0.8 created 09.10.2023
9 user (10%)
OK


fix: hide fields with admin.hidden attribute

Description
Make sure fields that have admin.hidden = true are not shown in the Admin UI.

[x] I have read and understand the CONTRIBUTING.md document in this repository

Type of change

[x] Bug fix (non-breaking change which fixes an issue)

Checklist:

[ ] I have added tests that prove my fix is effective or that my feature works
[ ] Existing test suite passes locally with my changes
[ ] I have made corresponding changes to the documentation

I tested it in my application and in payload directly (using yarn dev) and it's working for me!

I also tried to add e2e tests, but when I added
    test('should hide hidden fields in list view', async () => {
      await page.goto(url.list);
      await expect(page.locator('.row-1 .cell-hidden')).toHaveCount(0);
    });

it still ran successfully (I tried without my changes), even though in the UI I could see the cell is there... 🤷

Hey @pascalgn this is great! I am going to merge in advance of the test because I think the logic / changes are simple enough.
Good stuff. THANK YOU!

Hmm, I spoke too soon. I ran tests and then realized that we use hidden fields in list views often - for example, in upload-enabled collections, certain hidden fields like filename are used in the table view. The input is hidden, but the data is still there.
You may be looking for admin.disabled, which completely disables the field and omits it entirely from the admin UI. In contrast, the admin.hidden property simply sets the field to be rendered as an <input type="hidden">, which keeps the field in form state and submits the value normally as any other field would.
Does that make sense? I merged the PR but had to roll it back!

I see! Thanks for the explanation! 👍


Dev

This solution is to avoid the Query Throttle exception when retrieving
List items more than Threshold Limit. As contentIterator class is there
in server side which will take care of returning items batch by batch.
So i have implemented the client side Contentiterator class which will
be used to get the items batch by batch in CSOM and also CamlQuery
extension calss which will be used set the required properties for
Contentiterator.

Hi @Anil-Lakhagoudar, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!

In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement. It's all electronic and will take just minutes. I promise there's no faxing. https://cla.microsoft.com.

TTYL, MSBOT;

Hi @Anil-Lakhagoudar, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!

In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement. It's all electronic and will take just minutes. I promise there's no faxing. https://cla.microsoft.com.

TTYL, MSBOT;


Poor vdom benchmark results?

Not necessarily an issue, Mithril's been plenty fast for us in production! More surprised at the results and curious to get @lhorie's thoughts.
http://vdom-benchmark.github.io/vdom-benchmark/
Mithril scores at or very near the bottom in every single benchmark for me in both FF & Chrome.

Yeah, I was wondering about it too.

I haven't been overly enthused with vdom-benchmark ever since I spent a day trying to install it and run my own tests. The rationale behind sandboxing the execution environment makes sense – Dart usage less so. Ultimately there's so much black magic involved I'm not sure how much to trust its results.

Isn't that link what http://lhorie.github.io/mithril-blog/thoughts-on-performance.html was about?

Looks like it probably is, the different URL for it threw off my dupe searching! 😞


HTML

HTML
User Story
As a Bootcamp student
I want the prework notes to be structured on a webpage
so that I can easily find and read information
Acceptance Criteria
GIVEN a Prework Study Guide Website
WHEN I visit the website in my browser
THEN I see four boxes titled HTML, CSS, Git and Javascript with associated notes listed

PR merged 8/13/2023 12:30pm


update contribution guide

motivation: clearer instructions on how to "summon" the various CI jobs
changes: correct reference to "quick" tests, add reference to full toolchain tests

@swift-ci please smoke test self hosted


Release Google.Cloud.SecretManager.V1 version 1.9.0

Changes in this release:
New features

Added support for accessing secret versions by alias (commit 5102b66)

Created release for Google.Cloud.SecretManager.V1-1.9.0


Setting text of unselected inputs

Another use case where I'm putting x and y values in text inputs for a in-game map editor.

Phaser Input expects the text field to be selected and have a place holder. The following changing let me do what I want but not the best code. More of a workaround.
InputField.prototype.setText = function (text) {
    if (text === void 0) { text = ''; }
        if (this.placeHolder) {
            if (text.length > 0) {
                this.placeHolder.visible = false;
            }
            else {
                this.placeHolder.visible = true;
               }
            }
            this.value = text;
            this.domElement.value = this.value;
            this.updateText();
                try {
                    this.updateCursor();
                } catch(e) {}
            this.endFocus();
        };

This is a little trick I use to update a sprite via the inputs without events. This is outside the scope of this issue but interesting nonetheless.
this.update = function() {
	if (this.xInput.focus) {
		this.selectedItem.x = this.xInput.value;
	}
	if (this.yInput.focus) {
		this.selectedItem.y = this.yInput.value;
	}
}

Fixed the crashing, and nice to see your trick there!

Awesome thanks!


removed unused Task status

Pull Request Test Coverage Report for Build 2849

2 of 2 (100.0%)  changed or added relevant lines in 2 files are covered.
2 unchanged lines in 1 file lost coverage.
Overall coverage increased (+0.006%) to 69.575%

Files with Coverage Reduction
New Missed Lines
%

redis-persistence/src/main/java/com/netflix/conductor/jedis/JedisMock.java
2
11.44%

Totals

Change from base Build 2837:
0.006%

Covered Lines:
9634

Relevant Lines:
13847

💛  - Coveralls


Fix bump-formula-pr for taps with ssh remote url.

[x ] Have you followed the guidelines in our Contributing document?
[ x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x ] Have you added an explanation of what your changes do and why you'd like us to include them?
[x ] Have you written new tests for your changes? Here's an example.

[x ] Have you successfully run brew typecheck with your changes locally?
[x ] Have you successfully run brew tests with your changes locally?

#11229

Review period will end on 2021-04-26 at 02:16:26 UTC.

Review period skipped due to critical label.

Thanks @zachauten !


Logger-specific context

I recently stumbled upon your library and figured I give it a try and not roll my own logging infrastructure this time in my new project :)
The library seems very well designed from what I can see, one small feature is however missing for me (which I hand-rolled usually), namely the ability to create child loggers that derive the context configuration of the global / parent logger.
Now, configuration is detached and globally available as it seems (the global "baseContext"), so I don't know how feasible to implement it in the current architecture, but this is something that I actually miss.
Use cases would include the enrichment of logging context for worker instances of any sort (async processing, HTTP requests, ...), the scoping of UI-specific loggers (to always know "on which screen" a particular logging action resulted from) and so on and so forth. The logger's class name is the only way to group / tag things a little, but I think it's ugly to misuse that field for contextual purposes.
If I skip the idea of "deriving" context, maybe adding log-specific (non-derived) context could also already help, by creating an overload for the logger(...) and noCoLogger(...) methods that would accept then a vararg context: Pair<String, Any?>.

Hi @realdadfish, thanks for trying Klogging and for creating this issue.
Logger context
I am interested that you want loggers to have their own context. In Klogging log events have context items (Pair<String, Any?>) at the time they occur. Context comprises:

Base context: items present for the lifetime of the application. They are included in events created by Klogger and NoCoLogger instances.
Coroutine log context: items put into LogContext in the current coroutine context. They are included in events by Klogger instances using suspend logging functions (but not NoCoLogger instances). They are copied into child coroutine contexts.
Other coroutine context: items can be copied from other coroutine contexts, like Project Reactor.
Items copied from non-coroutine contexts, like MDC.
Fields in log message templates.
Items sent in logging function calls.

(I am improving the documentation to make this clearer – suggestions welcome.)
Could that cover what you need?
Parent–child loggers
Loggers in Klogging have no relationship to each other. Log4j and Logback are the only logging systems I have seen where names connote relationships (e.g. a logger called com.example.service is a parent of one called com.example.service.prices). I am not sure I like that kind of implied and perhaps surprising relationship.
(In Klogging, selecting loggers by name to decide whether to emit log events can imply relationships between loggers.)
An explicit relationship between loggers could be created something like this:
class Parent {
    val logger = logger<Parent>()
    // etc.
}
class Child {
    val logger = logger<Child>(parent = Parent.logger)
    // etc.
}

As I understand your question, the need for these relationships is when loggers have some items assigned to them.

Hi @mjstrasser, thanks for your reply. So, to answer your questions

Logger-specific contexts

The thing for me (at least) is that while the coroutine context case is a very smart way to log things running in such a specific context, this kind of mixes two different concepts, the technical concept of having a "context" for a coroutine that carries technical information about coroutine execution, with a logical "context" of having an application component that has it's own logging needs, regardless how it actually executes it's code.
To make things more clear, I wrote a small abstraction and decided to use the NoCoLogger implementation, just because I do not want to enforce the usage of coroutines in any part of my application. That the underlying logging system uses coroutines itself for me is an implementation detail and I want to use the same logging instance in and outside of a coroutine context. Which, in return, makes the proposed use of a "coroutine context" for additional logging data not viable.
And all the other context options you provided are totally fine for me and useful as-is, but not for my "give component-specific context without enforcing the execution context" type of use case.

Parent–child loggers

I agree that package-level relationships are a bad idea, because their implicit magic instead of explicit configuration. My main use case for a relational logger is really context-copying. I understand that logger<Child>(parent = Parent.logger) is not possible yet (can't see it in code), but yes, this would go into the right direction. Ideally, a child logger would be created with a function from the parent logger and also immediately accept logging context itself (see above), and to make this explicit, one could do something like this:
class Child {
    val logger = logger<Child>(context = parent.context + mapOf(/* new context */))
}

if the context (and this is my main concern) should be the only thing that "defines" such a relationship.

Thanks for the clarification. I’ll have a look at it.

First changes are in (I forgot to add #274 to the commit messages). You can add context items to a logger and they will be included in log events.
Copying from one logger to another coming soon.

First pass of documentation is here: https://klogging.io/docs/context/logger-context-items

Many thanks for the quick implementation!


DateTimePicker: adjusting maximum/minimum date

Category
[x ] Enhancement
[ ] Bug
[ ] Question
Version
1.16.0
Expected / Desired Behavior / Question
When using the datetimePicker I would like to have an opportunity to set maximum/minimum date like in Office UI Fabric
Thanks!

@AJIXuMuK  - I will pick this

Great! Thank you @PrasadKasireddy!

The PR has been merged and will be available in the next drop.
@rememberDL - you can check the functionality in the beta version


Points per rum no longer matches the live game

As per 4 september, each bottle of rum deposited will now be worth 10 point, as seen in this blog post.
This should be an easy change in the code. as it is essentially changing a 2 to a 10

Let me know if you'd like to be added as an author for this, as I don't really play anymore