Possible misuse of CAPI causing wal files to remain intact and mek DB inaccessible

Here is the context (have a look at the reply on this too):
https://github.com/duckdb/duckdb/issues/10002#issuecomment-2563013881

This was resolved in the linked thread.

@jraymakers Just wanted to say... the new .stream() api is inanely fast compared to previous versions'  .run() api... i saw like 10x improvement... my use case was to run query on entire data set, but only read first 2048 rows from resultset.
Awesome work! Thanks..
May I sugest to add the usage of .stream() API in npm docs too? Currently it doesn;t list it's usage.. but, only the call... all other api's show using resultset too. I had to refer to this PR to learn how to use it: https://github.com/duckdb/duckdb-node-neo/pull/81/files

Glad to hear it!
I'll think about how I can make the different ways of running results more clear. There are a lot of combinations, so it's tricky to document them all clearly. I did add a bit about streamAndReadUntil, which is my recommended way of doing streaming, but I probably didn't make this clear enough.

@jraymakers yep.. i think so many very similar looking API's are adding a bit cognitive load when deciding which one to use. Maybe grouping them by use-cases might help.

@jraymakers Was trying new streamAndReadUntil api... getting this crash:
No tables to drop.
Tables dropped. Loading data and getting count...

<--- Last few GCs --->

[5009:0x110008000]    32393 ms: Scavenge (reduce) 2047.7 (2082.4) -> 2047.2 (2082.9) MB, 22.29 / 0.00 ms  (average mu = 0.386, current mu = 0.387) allocation failure;
[5009:0x110008000]    32568 ms: Mark-Compact (reduce) 2048.0 (2082.9) -> 2047.3 (2083.1) MB, 112.96 / 0.00 ms  (+ 183.2 ms in 39 steps since start of marking, biggest step 6.4 ms, walltime since start of marking 355 ms) (average mu = 0.374, current mu = 0

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0x102a3959c node::Abort() [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
 2: 0x102a3979c node::ModifyCodeGenerationFromStrings(v8::Local<v8::Context>, v8::Local<v8::Value>, bool) [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
 3: 0x102bbe95c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
 4: 0x102d93050 v8::internal::Heap::GarbageCollectionReasonToString(v8::internal::GarbageCollectionReason) [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
 5: 0x102d96f04 v8::internal::Heap::CollectGarbageShared(v8::internal::LocalHeap*, v8::internal::GarbageCollectionReason) [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
 6: 0x102d93968 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
 7: 0x102d916f0 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
 8: 0x102d88344 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
 9: 0x102d88ba4 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
10: 0x102d6d800 v8::internal::Factory::AllocateRawWithAllocationSite(v8::internal::Handle<v8::internal::Map>, v8::internal::AllocationType, v8::internal::Handle<v8::internal::AllocationSite>) [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
11: 0x102d7bd50 v8::internal::Factory::NewJSArrayBufferView(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::FixedArrayBase>, v8::internal::Handle<v8::internal::JSArrayBuffer>, unsigned long, unsigned long) [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
12: 0x102d7c214 v8::internal::Factory::NewJSTypedArray(v8::internal::ExternalArrayType, v8::internal::Handle<v8::internal::JSArrayBuffer>, unsigned long, unsigned long, bool) [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
13: 0x102be7088 v8::Uint8Array::New(v8::Local<v8::ArrayBuffer>, unsigned long, unsigned long) [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
14: 0x102a10814 node::Buffer::Copy(node::Environment*, char const*, unsigned long) [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
15: 0x102a1065c node::Buffer::Copy(v8::Isolate*, char const*, unsigned long) [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
16: 0x102a070d4 napi_create_buffer_copy [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
17: 0x10861d6b0 DuckDBNodeAddon::get_data_from_pointer(Napi::CallbackInfo const&<EMAIL_ADDRESS>18: 0x10860c74c Napi::InstanceWrap<DuckDBNodeAddon>::InstanceMethodCallbackWrapper(napi_env__*, napi_callback_info__*<EMAIL_ADDRESS>19: 0x1029f797c v8impl::(anonymous namespace)::FunctionCallbackWrapper::Invoke(v8::FunctionCallbackInfo<v8::Value> const&) [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
20: 0x10342def0 Builtins_CallApiCallback [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
21: 0x1380751f4
22: 0x138075e74
23: 0x103463210 Builtins_AsyncFunctionAwaitResolveClosure [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
24: 0x103510fb8 Builtins_PromiseFulfillReactionJob [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
25: 0x103452b94 Builtins_RunMicrotasks [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
26: 0x10342a3f4 Builtins_JSRunMicrotasksEntry [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
27: 0x102d004d0 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
28: 0x102d009bc v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
29: 0x102d00b98 v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*) [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
30: 0x102d27d64 v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
31: 0x102d28500 v8::internal::MicrotaskQueue::PerformCheckpoint(v8::Isolate*) [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
32: 0x102968c64 node::InternalCallbackScope::Close() [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
33: 0x10296877c node::CallbackScope::~CallbackScope() [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
34: 0x102a0885c (anonymous namespace)::uvimpl::Work::AfterThreadPoolWork(int) [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
35: 0x102a08c50 node::ThreadPoolWork::ScheduleWork()::'lambda'(uv_work_s*, int)::operator()(uv_work_s*, int) const [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
36: 0x10340639c uv__work_done [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
37: 0x103409dec uv__async_io [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
38: 0x10341bec4 uv__io_poll [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
39: 0x10340a3b0 uv_run [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
40: 0x102969754 node::SpinEventLoopInternal(node::Environment*) [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
41: 0x102a79c6c node::NodeMainInstance::Run(node::ExitCode*, node::Environment*) [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
42: 0x102a79a08 node::NodeMainInstance::Run() [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
43: 0x102a03718 node::Start(int, char**) [/Users/nitinbansal/.nvm/versions/node/v20.11.1/bin/node]
44: 0x1831da0e0 start [/usr/lib/dyld]

Test code
async function run7(connection) {
    const dataFilePaths = [
        '/Users/nitinbansal/Downloads/mix data/generated.csv',
        '/Users/nitinbansal/Downloads/hackernews_10k_dec 2024.csv'
    ];
    const tableNames = ['table_1', 'table_2'];

    await useConnection(DB_FILE_PATH, async (connection) => {
        await dropTables(connection, true);

        console.log(`Tables dropped. Loading data and getting count...`);

        await loadFile(connection, dataFilePaths[0], tableNames[0], 3);
        await loadFile(connection, dataFilePaths[1], tableNames[1], 3);

        let result = null;
        for(let k = 0; k < 200; k++) {
            console.log(`iteration: ${k+1}`);

            result = await runUsingStreamAndReadUntil(connection, `select * from table_1`, 2048);
            console.log(`selected row count ${result?.rows?.length}`)

            result = await runUsingStreamAndReadUntil(connection, `select * from table_2`, 2048);
            console.log(`selected row count ${result?.rows?.length}`)
        }
    });
}

async function useConnection(dbFilePath, callback) {
    let connection = null;
    try {
        connection = await initConnection(dbFilePath);
        return await callback(connection); // Return the result of the callback
    } catch (error) {
        console.error("Error while using connection:", error);
        throw error;
    } finally {
        if(connection) {
            connection.disconnect();
            // connection = null; // todo: prod: use close() if available
        }
    }
}

async function initConnection(dbFilePath) {
    const instance = await DuckDBInstance.create(dbFilePath);
    return await instance.connect();
}

async function runUsingStreamAndReadUntil(connection, query, maxRowsToFetch = 2048) {
    const result = await connection.streamAndReadUntil(query, maxRowsToFetch);

    const collectedRows = [];

    while(true) {
        const rows = result.getRows();
        if (!rows || rows.length === 0) {
            break;
        }

        console.log(`got ${rows.length} rows`);

        rows.forEach(row => collectedRows.push(row));
    }

    return collectedRows;
}

Source file details

hackernews_10k_dec 2024.csv: 10K rows, 4.1MB CSV
generated.csv: 9M rows, 1.03GB CSV

Also, from docs and PR both, it's not clear how to proceed and use result returned from streamAndReadUntil call. So, I tried chunk based extraction too (on top of rows based on shown above).. net result is same for both... crash

while(true) {
        const chunk = await result.fetchChunk();
        if (!chunk || chunk.rowCount === 0) {
            break;
        }

        for (const row of chunk.getRows()) {
            collectedRows.push(row);
        }
    }

@jraymakers  continuing with above...
I know it's heap related error, but, the same code using .stream() API works without throwing any error.

It looks like you're running into a 2 GB heap limit in your Node environment. Since one of your files is 1 GB and you're creating multiple DuckDB instances in a loop, it's not very surprising you'd hit this limit.
A few options/recommendations:

Don't create multiple DuckDB instances in a loop, especially if you're reading in large files. You'll likely end up with multiple copies of those large files in memory.
If you do want to create multiple instances (though I'm not sure why you would), then explicitly DETACH large databases/files when you're done. This, I think, is likely to free up the memory they're using. Note that disconnecting does not perform this cleanup.
Increase the memory limit of your Node process, using the --max-old-space-size command-line flag.

I agree the documentation of the various ways to run SQL and get results could be improved, but I suspect that isn't what's causing your problem here. If you think it is, please send the exact differences. The streamAndReadUntil helper is just a wrapper around calls to stream and fetchChunk, so it's not clear why it would be any different.

@jraymakers Thanks for the info. A few points:

I believe disconnect/close should immediately free used memory (or mark it as freed for OS to reclaim) since the programmer is making it explicitly clear that they are not going to use it again. This is pretty standard expectation of connections and associated resources.
The difference between stream and streamAndReadUntil is indeed there. Since switching over to stream api yesterday, not a single time heap crash has occured when testing for same files.
The problem with maintaining a single connection is that it forces global state in an application. I could wrap it in a global function, but yet another problem of that long lives connection is that it might have become stale (for whatever reason). This can be handled by reconnecting. But then it becomes a burden on some other part of the code. Another way would be issue a simple select 1 statement, but then this will have to be issued before every lease of the connection. This is where connection pooling helps since it can in background from time to time issue this test query.. But I implement that either here because of restriction of having just one connection. So, overall, im stuck.

I'm mostly from java background, so, im a bit biased to think that way, but, the current behaviour does seem odd where API is not doing fully what it's saying.
Attaching GIF recording of same showing how readUntil is crashing whereas stream is working fine while keeping everything else as-is.

Attaching video recording link instead. This shows without doubt the observed difference
https://drive.google.com/file/d/1HM3J0DKNvY8FhN3EaBiML_DvWZltrtCI/view?usp=sharing
Also, here's the code (I have packed all relevant methods in a single file for easier testing)
import {DuckDBInstance} from "@duckdb/node-api";
import fs from 'node:fs';

async function initConnection(dbFilePath) {
    const instance = await DuckDBInstance.create(dbFilePath);
    return await instance.connect();
}

async function runUsingStreamAndReadUntil(connection, query, maxRowsToFetch = 2048) {
    const result = await connection.streamAndReadUntil(query, maxRowsToFetch);

    const collectedRows = [];

    while(true) {
        const rows = result.getRows();
        if (!rows || rows.length === 0) {
            break;
        }

        for (const row of rows) {
            collectedRows.push(row);
        }
    }

    return collectedRows;
}

async function runUsingStream(connection, query, maxRowsToFetch = 2048) {
    const result = await connection.stream(query);

    const collectedRows = [];
    let rowsFetched = 0;

    while (maxRowsToFetch < 0 || (maxRowsToFetch > -1 && rowsFetched < maxRowsToFetch)) {
        const chunk = await result.fetchChunk();
        if (!chunk || chunk.rowCount === 0) {
            break;
        }

        for (const row of chunk.getRows()) {
            if (maxRowsToFetch > -1 && rowsFetched >= maxRowsToFetch) break;
            collectedRows.push(row);
            rowsFetched++;
        }
    }

    return collectedRows;
}

async function loadFile(connection, filepath, tableName) {
    let query = `CREATE TABLE ${tableName} AS SELECT * FROM read_csv_auto('${filepath}')`;
    await connection.run(query);
}

async function deleteFile(filePath) {
    try {
        await fs.unlinkSync(filePath);
    } catch (err) {
        if (err.code !== 'ENOENT') {
            throw err;
        } else {
            // ignore
        }
    }
}

async function main() {
    const DB_FILE_PATH = './db.bin';
    await deleteFile(DB_FILE_PATH);
    await new Promise(resolve => setTimeout(resolve, 1000));

    const dataFilePaths = [
        '/Users/nitinbansal/Downloads/mix data/generated.csv',
        '/Users/nitinbansal/Downloads/hackernews_10k_dec 2024.csv'
    ];
    const tableNames = ['table_1', 'table_2'];

    const connection = await initConnection();

    await loadFile(connection, dataFilePaths[0], tableNames[0]);
    await loadFile(connection, dataFilePaths[1], tableNames[1]);

    let rows = null;
    for(let k = 0; k < 200; k++) {
        console.log(`iteration: ${k+1}`);

        // runUsingStreamAndReadUntil
        // runUsingStream
        rows = await runUsingStreamAndReadUntil(connection, `select * from table_1`, 2048);
        console.log(`selected row count ${rows.length}`)

        rows = await runUsingStreamAndReadUntil(connection, `select * from table_2`, 2048);
        console.log(`selected row count ${rows.length}`)
    }
}

main().catch(console.error);

And here are the data files links (please uncompress larger file)

Hackernews: https://drive.google.com/file/d/1I-f5eDWVIOYonBXmSh-PdqgqJchijOOJ/view?usp=sharing
Generated: https://drive.google.com/file/d/1kGwu62p8YAjKmEdAxCLSsp4HMl9q1Z9o/view?usp=sharing

Let me know if any other details are needed.
Thanks..

Thanks for the detailed reproduction! I think I see what's going on now.
The problem is a misunderstanding about how the getRows method of DuckDBResultReader (returned by streamAndReadUntil) behaves. It returns all the rows that have been read so far. This is different from the fetchChunk method of DuckDBResult (returned by stream), which tries to fetch the next chunk, and will return null or a zero-length chunk if there are none left to fetch.
So, the while(true) loop in runUsingStreamAndReadUntil will never terminate; it will continue to get and append the same set of rows until memory runs out.
(Note that DuckDBResult also has a getRows method, which fully materializes the result and returns all rows, even if it's a streaming result.)
I apologize for not making this distinction clearer. Obviously the various methods could use more documentation. I'll move that up in my priority list.

Regarding your other comments about connections:
The way DuckDB connections behave is different from other databases. It may take some getting used to. While the lifetime of some state is tied to a connection, such as temporary objects, most in-memory state has a lifetime tied to the instance, not the connection. An instance can have more than one connection. Closing a connection does not affect the instance-level state.
Specifically, if you create a (non-temporary) table using one connection to an instance, and then close that connection and create a new connection to the same instance, that table will still be around. The way to remove a table from an instance is either to DROP it or DETACH its containing database.
Additionally, connections in DuckDB are not network connections. They're just objects in memory. They cannot go stale. There's no need for, or benefit from, issuing occasional queries (e.g. select 1) on a connection.
In my understanding, these are some recommended patterns of use for DuckDB:

Create a single instance per process. (Using multiple instances in the same process might be useful in some advanced scenarios; if this is done, special care must be taken if the same data files are attached to both, or corruption can occur.)
Create multiple connections to an instance if you want to run multiple queries in parallel, or if you want different scopes for temporary objects (or other per-connection state such as prepared statements, settings, and variables).
Keep connections and instances around as long as you'd like. No special action is needed to keep them usable.

(Also, some context: While I communicate frequently with the DuckDB team as part of my work, both as a member of the MotherDuck team and as an author of Node Neo, I didn't build DuckDB nor do I control its design. The above is what I've learned, not something I have any control over.)

@jraymakers  Thanks.. verified working without crashing with this:
const result = await connection.streamAndReadUntil(query, maxRowsToFetch);
return result.getRows().slice(0, maxRowsToFetch);

Regarding single connection, if I can rely on it not going stale (since it's just like any other variable), that makes things much simple.
And yes, docs do need an update.
Thanks for your time..


Clarification needed: Can't Set Type of NPM Package

When creating the NPM package for the midterm, I am never given the prompt to set the "type" field. But when looking over everything to confirm it is correct, the "type" field has git instead of module. I'm not sure how to set this field, any help would be appreciated.

A TA said that she would contact the professor about this since they can't figure it out, but I haven't heard anything yet.

Has this issue been resolved? It looks like the default config for init only lets you set the repository type.

Problem still not solved, but professor said that we will get around to figuring it out.

I also have the same problem.

I am having this issue as well!

I ended up manually adding the field to the appropriate file and that seemed to do the trick!

^^^ I think this is just the solution. Sorry for the delay in response from the TAs but I think we misinterpreted the problem. You should be able to just manually add it to package.json and there shouldn't be any issues.

The directions of the midterm exam state that you must have type: 'module' in package.json, whether that is entered by prompt or manually you must have it in the file. Be sure to follow the directions, as they are stated clearly. Let me know if you have any other issues or questions moving forward!

I already tried adding it in manually, it was one of the first things I tried to do to fix the issue. However it still did not work. :(


Remove Array.from references from Core

A new function was introduced - SliceToArray that will force the return of an array instead of a typed array.
Adressing #9122

Great, LGTM!

(Replaces #9119)


Create second_file.txt

Creat second file for new branch testing!
It is used for GitHub flow testing.

Does it works?

let's see!

It's just a comment!


[Tooltip] Alignment option in interactive tooltip

The problem
Carbon core is adding alignment option in interactive tooltip.
https://github.com/carbon-design-system/carbon/pull/7309
The solution
Implement it.

Assuming this is the alignment of where the caret is on the tooltip, would be great to get this in, I think I've seen our team suggesting the alignment on our own components that would utilize tooltip.
https://react.carbondesignsystem.com/?path=/story/components-tooltip--default-bottom

we may also need to update with some follow up features:
https://github.com/carbon-design-system/carbon/pull/9556
https://github.com/carbon-design-system/carbon/pull/9606


npm package is broken

Hi!
I am just letting you know that the npm package doesn't contain lib/ folder. I managed to make the package working by simply running "npm install" in the node_modules/recorder-js folder. Please fix when you will have time.
Best regards,
a community member.

Hey! Thanks for the heads up! I had actually noticed but hadn't gotten around to fixing it yet. Didn't think anyone would be interested in using this so quickly! Glad you got it working for now. Let me know what you think of it

I'll fix the publishing tomorrow

Everything works cool, both the recorder and wave-stream! Thanks for you work!

Fixed with 1.0.7


Reraise tectonic parse errors as ResourceError when computing schema

@wemrysi, should we call out the clubhouse issue number in a comment?

@nicflores It relates to ch2956, but doesn't resolve it completely, so I didn't link it the ususal way.
Anyhow, hold off on merging as I may have a better idea, I'll ping you when ready.

@nicflores Ready for another look, I consolidated things a bit and also handled the parse errors in mimir.
This still needs a minor update over in slamdata/slamdata-backend to close out the issue, I'll submit that PR shortly.


added the footer

Description

I have added the footer components in the components section names as footer.css and footer.jsx
i have added the current year and github icons with your link
Related Issues

#570 PR
Changes Proposed

Checklist

[ yes] I have read and followed the Contribution Guidelines.
[ yes] All new and existing tests passed.
[yes ] I have updated the documentation to reflect the changes I've made.
[yes ] My code follows the code style of this project.
[yes ] The title of my pull request is a short description of the requested changes.

Screenshots

Note to reviewers

hi @Rajneesh2223,
I had a suggestion from my end.

The text in the footer is not quite readable, maybe changing the footer background or the text color might do the job

@Rajneesh2223 We can add this on landing page. Closing for now.


Ray Actor dies during tokenization process

I'm running tokenization on a bunch of processed files and I find that the Ray Actor dies due to all references to it being removed
  File "/home/ubuntu/miniconda3/envs/rapids-22.06/lib/python3.9/site-packages/ray/_private/worker.py", line 2656, in get
    values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
  File "/home/ubuntu/miniconda3/envs/rapids-22.06/lib/python3.9/site-packages/ray/_private/worker.py", line 871, in get_objects
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(UserCodeException): ray::MapBatches(buffer_write)() (pid=345911, ip=<IP_ADDRESS>)
  File "/home/ubuntu/miniconda3/envs/rapids-22.06/lib/python3.9/site-packages/open_lm/datapreprocess/ray/tokenize_shuffle.py", line 544, in buffer_write
    return {x: [y] for x, y in buffer_writer_pool.get_next().items()}
  File "/home/ubuntu/miniconda3/envs/rapids-22.06/lib/python3.9/site-packages/ray/util/actor_pool.py", line 309, in get_next
    return ray.get(future)
ray.exceptions.ActorDiedError: The actor died unexpectedly before finishing this task.
        class_name: BufferedShardWriter
        actor_id: f4dc4fd5a9a6d1d8a085c6e201000000
        pid: 1720701
        name: 367e67b756e8a199e62de7392cd8ab67aef847680ad88bdf13490848_buffer_writer_0
        namespace: 56ced544-c253-4038-8abe-89c64eb43193
        ip: <IP_ADDRESS>
The actor is dead because all references to the actor were removed. Any thoughts on what may be causing this?

attaching some more in depth logs here
ray-data.log

Update I was able to repro the same error running only on a single processed file as opposed to the 2.8K I was working with to start.

Adding some details

It happens in writing steps (https://github.com/mlfoundations/open_lm/blob/9bb92ef1689333534b7057942a20d18a46d1fa52/open_lm/datapreprocess/ray/tokenize_shuffle.py#L693), and files are written until the error happens.
We first tried local save and then tried s3 save later, but we got the same error in both settings
The error is resolved when I gave allow_imbalanced_write flag in https://github.com/mlfoundations/open_lm/blob/9bb92ef1689333534b7057942a20d18a46d1fa52/open_lm/datapreprocess/ray/tokenize_shuffle.py#L578, though it makes training code skip small files. I think parallelism via ActorPool in buffer_write  is failing somehow.

Sorry for the late response.
If this is still an issue can you give some information on the instance or ray cluster you are using? Besides allow_imbalanced_write, have you also tried increasing num_writers_per_node
Could you also provide more details, such as the command used and the input files for the single file input error reproduction?

Closing for now. Feel free to reopen if this remains an issue.

Same issue here.
I am now testing a small scale by processing a small amount of files (~200M tokens in total) on a single node with 32 CPU cores and 96GB memory (a previous run on 5 nodes also failed). Setting allow_imbalanced_write could resolve the issue but increasing num_writers_per_node does not.
Any ideas to fix the issue without setting allow_imbalanced_write?

could you try using this
https://github.com/revbucket/tokshuf-rust

I was looking through the code and the issue seems to be the buffer writer was not available after the function exits. Here is a walkaround by creating a global reference that seems to work:
global_buffer_writers = {}  # Global variable to store buffer writer references

def buffer_write(rows, folder, counter, buffer_size, num_writers_per_node):
    global global_buffer_writers

    node_id = ray.get_runtime_context().get_node_id()
    worker_id = ray.get_runtime_context().get_worker_id()

    # Create buffer writers if they don't exist for this node
    if node_id not in global_buffer_writers:
        global_buffer_writers[node_id] = []
        for k in range(num_writers_per_node):
            buffer_writer_name = f"{node_id}_buffer_writer_{k}"
            logger.info(f"Creating {buffer_writer_name}")
            buffer_writer = BufferedShardWriter.options(
                name=buffer_writer_name,
                get_if_exists=True,
                scheduling_strategy=ray.util.scheduling_strategies.NodeAffinitySchedulingStrategy(
                    node_id=node_id, soft=False
                ),
            ).remote(buffer_size=buffer_size)
            ray.get(counter.add_buffer_writer.remote(buffer_writer_name, buffer_writer))
            global_buffer_writers[node_id].append(buffer_writer)

    buffer_writer_pool = ray.util.ActorPool(global_buffer_writers[node_id])
    buffer_writer_pool.submit(lambda a, rows: a.write.remote(rows, folder, counter, buffer_size), rows)
    result = buffer_writer_pool.get_next()

    return {x: [y] for x, y in result.items()}

I am less familiar with ray so not sure if this will robustly work in more general cases. Would be great to hear your thoughts on this!

I think I've identified and resolved the issue. The problem appears to stem from the BufferedShardWriter somehow exiting before completing its writing task. This occurs in the following code:
https://github.com/mlfoundations/open_lm/blob/9bb92ef1689333534b7057942a20d18a46d1fa52/open_lm/datapreprocess/ray/tokenize_shuffle.py#L532-L538
To address this, I added the argument lifetime="detached" to the BufferedShardWriter options. With this modification, the tokenization process now runs normally.
For those encountering this issue, here's a quick fix:

Locate the open_lm/datapreprocess/ray/tokenize_shuffle.py file in your Python site-packages environment.
Modify the BufferedShardWriter initialization (lines 532-538) as follows:

buffer_writer = BufferedShardWriter.options(
    name=buffer_writer_name,
    get_if_exists=True,
    scheduling_strategy=ray.util.scheduling_strategies.NodeAffinitySchedulingStrategy(
        node_id=node_id, soft=False
    ),
    lifetime="detached"
).remote(buffer_size=buffer_size)

This modification should resolve the issue and allow the tokenization process to complete successfully.


SnackBar/Notification custom texts?

1.2 version now supports custom texts for Dialog? What about SnackBar and Notification? Would be really nice to have those also customisable.

See Release 2.4.
Thank you 👍


Add Http spec auth policy

Add a policy to simply do basic HTTP auth policy. This can be useful for SDK like EventGrid, that use this auth system with a "SharedAccessKey" auth scheme
See also: https://github.com/Azure/typespec-azure/issues/2706

API change check
APIView has identified API level changes in this PR and created following API reviews.
azure-core


CARTO API IMPORT PROBLEMS

I'm trying to use the import function of the CARTO API, but I had problems trying to import a csv file from the local. I'm not sure what I have to put in the url field in the body, or there is other field different to import files.

Hi @fsalazarcitelum , thanks for your message
The v3 import endpoint does not currently support uploading from local files, only from external URLs. If you're still interested in uploading a local file, I suggest you to use Data Explorer in your CARTO workspace


feat(parser): add syntax error for hyphen in JSXMemberExpression <Foo.bar-baz />

closes #5355

#5440  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @Boshen and the rest of your teammates on  Graphite


Missing icon for "region"

I'm unable to find a suitable icon to represent an Azure region. There is a globe, however there's no indication of a particular location on the globe.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 6624d44c-f9f6-02f8-2c67-0f0637fed62e
Version Independent ID: 9346a8ff-bab5-be2c-9dc1-d546fda1efa9
Content: Azure Icons - Azure Architecture Center
Content Source: docs/icons/index.md
Service: architecture-center
GitHub Login: @doodlemania2
Microsoft Alias: pnp

Thanks for reporting - this issue is under review. This is a Microsoft Internal DevOps Bug ID # 458811

@alexangas
I have assigned the issue to the content author for further review.

Thank you for reporting this, @alexangas.
We've made several updates to the Azure Icons recently, and there are hundreds of new icons added.
The update is available for download here: https://learn.microsoft.com/en-us/azure/architecture/icons/
The region icon is in the general folder with the name 10116-icon-service-Region-Management.svg
Thanks!
#please-close


 I should develop my own implementation.
An alternative approach would be a velocity wide command alias plugins. I think this would be a cooler project idea but it may take up more time.

The idea is to create a network-wide command alias system.
Also some more general stuff I need to do for this project:

[x] Add docker-compose.yml for local testing
[x] #3
[x] Implement Plugin logic
[x] #4

Done


Amazon S3 seems to be unresponsive (blocked?) in China

@BerkeleyTrue's move to CloudFlare may fix this issue.

Cloudflare will not fix this issue since we are serving from amazons servers. It seems like it is temporary and may come back. We will probably have this issue with any CDN we choose unless it is a CDN directly in china


polish notebooks

Codecov Report

Merging #1147 into image_v2.6.0 will not change coverage.
The diff coverage is n/a.

@@              Coverage Diff              @@
##           image_v2.6.0    #1147   +/-   ##
=============================================
  Coverage         47.82%   47.82%
=============================================
  Files                25       25
  Lines              4604     4604
  Branches            762      762
=============================================
  Hits               2202     2202
  Misses             2346     2346
  Partials             56       56

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8a02f20...35d9eb6. Read the comment docs.


[Bug]: "Change theme" property pane persists between the apps

Is there an existing issue for this?

[X] I have searched the existing issues

Description
The Change theme property pane persists between the apps.
Environment  = RELEASE
https://user-images.githubusercontent.com/14138515/168325295-63345dec-6b84-494f-960f-387abd1650ba.mov
Steps To Reproduce

Create a new app.
Click on the change theme in the property pane.
Now click on the appsmith icon in the top left corner to open all the applications.
Now open a new app, and observe that the "Change theme property pane" still persists.

Public Sample App
No response
Version
Cloud

@keyurparalkar Issues has been tested and working fine.


[whales, task 3 output] what is validation/test score?

The validation score (2.0059) is not equal to validation loss (1.01667) or the validation accuracy (0.77713). Similarly, test score is hard to interpret. How are these two scores calculated?

226894.311837 | 2018-04-15 01-09-37 minerva >>> epoch 250 current lr: 0.0003252930814335209
226894.312173 | 2018-04-15 01-09-37 minerva >>> epoch 249 loss:     0.03353
226894.312389 | 2018-04-15 01-09-37 minerva >>> epoch 249 accuracy: 0.99986
226981.769858 | 2018-04-15 01-11-05 minerva >>> epoch 249 validation loss:     1.01667
226981.770167 | 2018-04-15 01-11-05 minerva >>> epoch 249 validation accuracy: 0.77713
227067.955128 | 2018-04-15 01-12-31 minerva >>> training finished...
<...>
227715.884304 | Validation score is 2.0059
227715.884506 | Test score is 2.1295
227715.884696 | That is a solid validation
227715.884888 | Sorry, but this score is not high enough to pass the task

Seems that sometimes it is RMSE and sometimes it is log loss - depends on pipeline stage/subproblem:
See minerva/whales/validation.py
SCORE_FUNCTIONS = {'localization': rmse_multi,
                   'alignment': rmse_multi,
                   'classification': log_loss_whales,
                   'end_to_end': log_loss_whales
                   }

Results for which subproblem you've pasted above?


[uthenticode] Update to 1.0.4

Bumps uthenticode to 1.0.4.

What does your PR fix? Fixes #

None.

Which triplets are supported/not supported? Have you updated the CI baseline?

No changes.

Does your PR follow the maintainer guide?

I believe so!

Thanks @woodruffw :)


Bug: Help Widget not working

Description
Help widget in NCA-> Settings -> Slope selection currently not working
(Currently commented out)
To do:

[ ] Fix help widget so that when clicked, instructions for how to carry out inclusions/exclusions pop up
[ ] Ensure instructions are clear for user

Hi @m-kolomanski the code is currently in ui.R around lines 300-400. @Gero1999 made a very useful box which included written instruction and also gifs showing the user how to click on the plots. Since we converted from an app into a package it stopped working.
If you can get them working again, that would be great. And once I have solved issue #23  the plots will look better and we can just rerecord the gifs to match the new plots.

@js3110
I can work something out.
Currently I cannot see anything to trigger the modal - what was the idea behind it? Should it be displayed when user navigates to the Slope Selector tab? Or should we have info button that pops up the modal?

@m-kolomanski
There was an info button in the ui that we commented out.
I can't seem to find it though so I guess it got deleted instead of commented.
But it just needs to be a little help button (?) that the user can click

Closing with #98


Review and implement better feature phone interaction

Test the game on D-pad/key nav and make it better.

when feature phone, hide full screen btn
show # and * keys guide next to settings icon and mode toggle.

this is after currents PR rands and MVP achieved.


Set up app to render templates

As a user, I want to be able to view the home and dashboard pages in the browser

just completed this today


Update median.md

Fixed the link for PCT
Purpose of this pull request
This pull request (PR) ...
Issue number:

Select the type of change:

[x] Minor Changes - Typos, formatting, slight revisions
[ ] Update Content - Revisions and updating sections
[ ] New Content - New features, sections, pages, tutorials
[ ] Site and Tools - Updates, maintenance, and new packages for the site, Gatsby, React, etc

@stacykor  Same thing here. The PR seems to have been merged. Can I click the "Delete branch" button since this PR is merged and closed.


Extractive Training is not working

Abstractive is working well, but extractive with the following command:
python train.py -task ext -mode train -bert_data_path ../data/bert_data_cnn10k/cnndm -ext_dropout 0.1 -model_path ../models/cnn10_ext -lr 2e-3 -visible_gpus 0 -report_every 50 -save_checkpoint_steps 1000 -batch_size 3000 -train_steps 50000 -accum_count 2 -log_file ../logs/ext_bert_cnndm -use_interval true -warmup_steps 10000 -max_pos 512
This is the output:
Traceback (most recent call last):
File "train.py", line 144, in
train_ext(args, device_id)
File "/home/ubuntu/jose/PreSumm/src/train_extractive.py", line 203, in train_ext
train_single_ext(args, device_id)
File "/home/ubuntu/jose/PreSumm/src/train_extractive.py", line 245, in train_single_ext
trainer.train(train_iter_fct, args.train_steps)
File "/home/ubuntu/jose/PreSumm/src/models/trainer_ext.py", line 137, in train
for i, batch in enumerate(train_iter):
File "/home/ubuntu/jose/PreSumm/src/models/data_loader.py", line 144, in iter
for batch in self.cur_iter:
File "/home/ubuntu/jose/PreSumm/src/models/data_loader.py", line 280, in iter
for idx, minibatch in enumerate(self.batches):
File "/home/ubuntu/jose/PreSumm/src/models/data_loader.py", line 258, in create_batches
for buffer in self.batch_buffer(data, self.batch_size * 300):
File "/home/ubuntu/jose/PreSumm/src/models/data_loader.py", line 230, in batch_buffer
size_so_far = self.batch_size_fn(ex, len(minibatch), self.args.max_ndocs_in_batch)
TypeError: ext_batch_size_fn() takes 2 positional arguments but 3 were given

Can change the code in PreSumm/src/models/data_loader.py line 114
def ext_batch_size_fn(new,count)   to => def ext_batch_size_fn(new,count,max_ndocs_in_batch=None)
The BertSumExt training  will be runnable after that

I created a PR for this:
https://github.com/nlpyang/PreSumm/pull/143

Please use master version for regular training, dev branch should be only used for -mode test_text .


Please add a 'whoami' command

It would be very useful if a 'whoami' command could be added to yotta which returned the username of the user and whether they are currently 'logged in' to the registry.
This could follow the npm whoami format.

Yup, sounds good.
It'll display the email address rather than name though, as there's no separate username to the email address.

Fixed in 0.8.0


qdtrack based on YOLOX detector.

Thanks for your wonderful jobs! Recently, I have tried to make qdtrack implementation based on YOLO-X detector. However, I encountered difficulties. I sincerely expect the authors to release the code or provide help. Best wishes to you！

Hello, when do you plan to release the new models? I have re-produced the version based on YOLO-X detector following the details of the paper published on TPAMI. However, I got bad results(HOTA: only 25.1% on val sets of DanceTrack dataset).  Maybe,  I'm missing some details.  I sincerely look forward to your help. Best wishes to you！

also waitting

I have the same question...

您好，您打算何时发布新型号？我按照 TPAMI 上发表的论文的详细信息重新制作了基于 YOLO-X 检测器的版本。但是，我得到了糟糕的结果（HOTA：在 DanceTrack 数据集的 val 集上只有 25.1%）。也许，我错过了一些细节。我衷心期待您的帮助。祝你好运！

Hello, how can I change the mosaic and mixup in yolox, in case I need to input two frames


Property not found on component

Describe the bug
Installed fresh latest Laravel 8 and filament cms, created model, migration and resource, added title field and tried to save, got this error or sometimes - No property found for validation
Screenshots
If applicable, add screenshots to help explain your problem.

Context

Browser: Chrome
Server OS: Laradock on Windows
PHP version: 7.4
Database: MySQL 8.0

@dmitrijsmihailovs Can you share your form / fields code?
Im guessing you're not using the resource field and using the normal field (not for resource forms).

I tried it this ways
public static function form(Form $form)     {         return $form             ->schema([                 TextInput::make('title')             ]);     }

@dmitrijsmihailovs could you show us where you imported TextInput please? It should be Filament\Resources\Forms\Components\TextInput.


Add estimatefee  to RPC commands

Make estimefee available thru RPC, like bitcoin does:
https://bitcoin.org/en/developer-reference#estimatefee
In my particular case it would be usefull for electrum-server

Assigned to #card-4014795

Why is there a wontfix label?

I removed the label, but I'm not promising a deadline at this moment!

👍  submitting an issue is not committing the maintainers to a deadline, so I think the removal makes sense

Original PR in Bitcoin that adds estimatefee / estimatepriority RPC methods:
https://github.com/bitcoin/bitcoin/pull/3959

True, and besides, nothing is stopping anybody from submitting a PR themselves before I get to it ;)
Thanks for the suggestion, good one!

wontfix: from perspective of issue creator, I only assumed it is not fixable until we provide it, then it will become fixable

deadline is set by accorging milestone


Pair Generators for Split Metacyclic Groups in GL(n,p)

An implementation of pair generators within GL(n,p) for split metacyclic groups has been successfully tested for orders up to 256. It would be beneficial to add in the project references a proven generalization of this result from existing literature.

The generators in $\textbf{GL}(n,p)$ for the group $\textbf{M} \rtimes \textbf{C}_n$, where $\textbf{M}$ is Abelian and $\textbf{C}_n$ is cyclic, have been determined up to order 63 through exhaustive search. This search utilized diagonal matrix generators for $\textbf{M}$ and a permutation matrix generator for $\textbf{C}_n$.
Before proceeding with groups of order 64 and higher, reducing the space of possibilities would be interesting.

Once an effective heuristic is found for generators in $\textbf{GL}(n,p)$ for the group $\textbf{M} \rtimes \textbf{C}_n$, there will be no further need to name subgroups. Calling the backend methods used for naming subgroups, at
https://github.com/aidevnn/FastGoat/blob/3488668b0c263d5f17ac485a68568feed3c3b996/FastGoat/Structures/Subgroups/AllSubgroups.cs#L123-L132
will be more efficient. Subsequently, this code can be moved from the Program.cs file to the Examples namespace.
Instead of improving the current implementation at this moment, it may be more beneficial to address the environment issues while upgrading to Ubuntu 24.04 or tackle other group theory-related problems.

The subgroup of $\text{GL}(n,p)$ generated by the diagonal and permutation matrices is interesting to study. This subgroup is significant for matrix form of abelians, metacyclics and many others groups.

The matrix representation for groups of order up to 63 is nearly complete, except for one (C2 . S4 with GAP ID (48,28)). However, after considering metacyclic groups and the semidirect product of abelian and cyclic groups, finding a practical approach for solvable groups of order more than 64, appears more challenging.
According to Cayley's theorem, a group of order $n$ is always a subgroup of $\mathbf{S}_n$ and also of $\text{GL}(n,2)$. However, using the subgroup of $\text{GL}(n_1, p)$ generated by the diagonal and permutation matrices is interesting when $n_1 < n$, and, characters table in $\text{GL}(n_1,\mathbb{C})$ gives always useful information on $n_1$ and $p$.

The matrix forms of the groups SL(2,3) and GL(2,3) are not currently generated by diagonal or permutation matrices. In particular, A5 poses a significant challenge.

Groups generated by diagonal or permutation matrices in $\text{GL}(n,p)$ up to order 63 have been successfully completed. Additionally, they provide an easy and direct way to obtain representation in $\text{GL}(\mathbb{C})$, and the correctness can be verified using character tables.
Conversely, starting with the character table of a finite group and then programmatically finding an isomorphic group generated by diagonal or permutation matrices in $\text{GL}(n,p)$ with a better control over $n, p$ and the space of all possibilities, is an interesting goal to achieve.


Create log file during test run

Update runtests.py to generate a log file of all the tests run and the
failures encountered.
Log files are by default placed in test/logs, and are named
testrun.{arch}{flavor}.log.
The filename/location can be overridden with the -l commandline switch.

LGTM

Thanks @Jianchun. Yup, took all feedback except for the mkdir one- the assumption is that if the user explicitly passed in the path, we assume it to be valid, we don't do anything to make it valid.


Confirm UVE working on production and bootstrap

This may have caused a memory issue, not sure.

typed bootstrap will enforce this rule.


CI against Ruby 3.3

This PR adds Ruby 3.3 to the CI matrix to confirm lograge working with Ruby 3.3.

CI is passed now.


Hooked checkbox into redux & added simple malcolm put

connect to #92

Codecov Report

Merging #99 into version1 will decrease coverage by 0.48%.
The diff coverage is 84.61%.

@@             Coverage Diff              @@
##           version1      #99      +/-   ##
============================================
- Coverage     93.72%   93.23%   -0.49%
============================================
  Files            23       24       +1
  Lines           223      281      +58
  Branches         29       36       +7
============================================
+ Hits            209      262      +53
- Misses           13       18       +5
  Partials          1        1

Impacted Files
Coverage Δ

...ets/attributeDetails/attributeDetails.component.js
100% <ø> (ø)
:arrow_up:

src/malcolm/malcolmHandlers/attributeHandler.js
100% <ø> (ø)
:arrow_up:

src/malcolmWidgets/checkbox/checkbox.component.js
100% <100%> (ø)
:arrow_up:

src/malcolm/malcolm.types.js
100% <100%> (ø)
:arrow_up:

...Details/attributeAlarm/attributeAlarm.component.js
90.9% <100%> (+0.9%)
:arrow_up:

src/malcolm/malcolmActionCreators.js
100% <100%> (ø)
:arrow_up:

...s/attributeSelector/attributeSelector.component.js
78.94% <62.5%> (-12.72%)
:arrow_down:

src/malcolm/malcolmReducer.js
96.07% <92.3%> (-1.29%)
:arrow_down:

src/malcolm/malcolmSocketHandler.js
93.75% <0%> (ø)
:arrow_up:

src/malcolm/malcolmReduxMiddleware.js
100% <0%> (ø)
:arrow_up:

... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e359901...4dca598. Read the comment docs.


New version: PositiveFactorizations v0.2.3

Registering package: PositiveFactorizations
Repository: https://github.com/timholy/PositiveFactorizations.jl
Created by: @timholy
Version: v0.2.3
Commit: dd968ac767fa720685f73af246851dc6a128cc72
Reviewed by: @timholy
Reference: https://github.com/timholy/PositiveFactorizations.jl/commit/dd968ac767fa720685f73af246851dc6a128cc72#commitcomment-35711727

Your new version pull request met all of the guidelines for automatic merging.
I will automatically merge this pull request during the next cron job.
If you want to prevent this pull request from being auto-merged, simply leave a comment.
(If you want to post a comment without blocking auto-merging, you must include the text [noblock] in your comment.)

[noblock]


Added favorites feature for containers.

Hi, tried to resolve the issue #3067 ,
Some screenshots:
favorite container -

favorites containers and hover on container -

I'm not sure if the code containerActions.js file is right,
have a problem that i want to save the favorite names in the localstorage any toggle,
Because if the computer shutdown and the app exiting, i want the favorite to be saved before,
Review it and comment me if something can be improved.
Thank you,

Added favorites feature for containers.
Fixed side-bar width css, added cursor: pointer to containers.

Signed-off-by: Daniel Amram<EMAIL_ADDRESS>

@FrenchBen Do you mean that the name of the container isn't unique? Its preffered to use the id to indicate the favorites?

ID may be better yes, as the container name isn't unique - Kitematic keeps using the name of the image, which doesn't make it unique.

Got it, but i saw that all the app using the name as an unique id - the container reducer, the services and more it seems weird to use the id where whole the app using name...
More than that, there's a function 'generateContainerName' that generate an unique id.

may be ok - didn't get a chance to try it out with a few different containers. If we merge and issues come up, can I assign them to you?

@FrenchBen For sure ,Thank you.
What's preffered to be done? issues or the roadmap?
I want to continue contribute this repo...


Load data from LocalStorage

Hey,
sorry if there is already an answer or solution for this. Im not very experienced with Web development.
Where is the best place to load data from LocalStorage (if some exist) on client-side and dispatch them to the Redux storage?
My first guess would be in the client.jsx file. But it is only a render function?
Thanks for any help!

Perfekt!
Thanks for the Explanation

@psimyn Awesome!

Closing issue but happy to keep the discussion going!


Simplify Usage

Is your feature request related to a problem? Please describe.
Simplify usage codes
Describe the solution you'd like
Customization part is complex to understand, simplifying will help others to undertand

Issue solved with #10 pull


Support blacklisting regions for WGS

Such as common germline cnv

Given that we have a flag for pruning X, Y, and M/MT, I am putting the arbitrary blacklist for WGS to a later milestone.

Issue moved to broadinstitute/gatk #2832 via ZenHub


Result scrolling

No hardwrap, only scrolling if content view is bigger than 500 px at any dimension
500 was hardcoded because i don't know how to get maxHeight from .bubble-output-container
Replacing of escaped \n was added because of:
If I execute language print variable command (print in python, console.log,...) I get multiline answer with any kernel. But when I execute declaration of variable with multiline text returned answer contains escaped newlines which by design is ok, but looks ugly in hydrogen
overflow: scroll adds additional padding in all result views with/without scrollbars, maybe change it dynamically? But I didn't see advantages

All of statements can be discussed

Otherwise this looks good to go.

I occasionally added "watch" to ticket name. It should work on all result views. Would this be merged on next release?

You've tested that this does work in practice? Previously marker
decorations were simply not scrollable in Atom.
On Fri, Nov 20, 2015 at 4:06 PM Slava<EMAIL_ADDRESS>wrote:

I occasionally added "watch" to ticket name. It should work on all result
views. Would this be merged on next release?
—
Reply to this email directly or view it on GitHub
https://github.com/willwhitney/hydrogen/pull/136#issuecomment-158523334.

If answer can't be presented in a marker smaller than 500px (default bubble lenght) presenter view changed to multiline answer box:

Cool. If it works, merge it!
On Fri, Nov 20, 2015 at 5:43 PM Slava<EMAIL_ADDRESS>wrote:

If answer can't be presented in a marker smaller than 500px (default
bubble lenght) presenter view changed to multiline answer box:
[image: labels]
https://cloud.githubusercontent.com/assets/1011721/11313828/ce526a74-8fe8-11e5-8155-2096482f7fb3.png
—
Reply to this email directly or view it on GitHub
https://github.com/willwhitney/hydrogen/pull/136#issuecomment-158546544.


[RESOLVED] Interface eth0(): High error rate (>2 for 5m)

Problem started at 10:54:29 on 2023.07.21
Problem name: Interface eth0(): High error rate (>2 for 5m)
Host: S100-Primario
Severity: Warning
Operational data: errors in: 1, errors out: 0
Original problem ID: 32903
Event details in Zabbix: http://<IP_ADDRESS>/tr_events.php?triggerid=23483&eventid=32903

Problem has been resolved in 9m 0s at 11:03:29 on 2023.07.21
Problem name: Interface eth0(): High error rate (>2 for 5m)
Host: S100-Primario
Severity: Warning
Original problem ID: 32903


Check for /-/ in GitLab url (closes #1073)

Make sure the /-/ string exists in path before splitting the path to avoid exception
Exception occurred:
File "c:\dev\gitlab\aero-software\internal\docs\venv\lib\site-packages\pydata_sphinx_theme_init_.py", line 998, in parse_url
group_and_subgroups, parts, *_ = path.split("/-/")
ValueError: not enough values to unpack (expected at least 2, got 1)

This URL is valid https://gitlab.com/gitlab-org/gitlab-ce/issues/23251, it might just be an older format but it still works and does expand to https://gitlab.com/gitlab-org/gitlab-foss/-/issues/23251 when opened.
Will investigate the tests.


Add Hetzner provider (single node)

This PR adds Hetzner to deploy the Fermyon Platform in a single node.
Using Hetzner can be very convenient to experiment with the Fermyon Platform as, at the time of writing, it's very cheap compared to other cloud providers: 8.22 EUR/month

3.98 EUR/month for the cheapest server (1vCPU, 2GB RAM)
0.61 EUR/month for the server IP v4 address
3.63 EUR/month for the floating IP address

Hi @vdice, thank you for the prompt reply and for the code review!
I don't think they offer a free trial, but there's a referral program - you'll receive 20 EUR in cloud credits as soon as you sign up from my referral link.
Let me know if there's anything more I can do to help.


looses theme on iOS if orientation changes

Hi JP,
thx for StatusBar - works great on Android and iOS.
found a small problel on iOS
have set StatusBar theme to StatusBar.Dark --> text and icons in StatusBar are white
if orientation changes, then StatusBar looses the theme and text and icons are black
this is my workaround:
if orientation changes on iOS, toggle the Theme, then set it back via Timer
if(Qt.platform.os === "ios") {             toggleStatusBarTheme()         } function toggleStatusBarTheme() {         if(theStatusBar.theme === StatusBar.Light) {             theStatusBar.theme = StatusBar.Dark         } else {             theStatusBar.theme = StatusBar.Light         }         statusBarTimer.start()     }     Timer {         id: statusBarTimer         interval: 200         repeat: false         onTriggered: {             theStatusBar.theme = StatusBar.Dark         }     }

Hi ekke, I don't have a development environment for iOS these days, but you could try adding screen orientation change handling to StatusBarPrivate::setTheme_sys() in src/statusbar_ios.mm. Something similar to the already existing application state change handling, which is there as a workaround for fighting against Qt's internal statusbar updates, if I remember correctly. I could imagine the reason for losing the theme on orientation changes is similar to losing it on application state changes. I'd try connecting to QGuiApp::primaryScreen()'s orientationChanged() signal and call updatePreferredStatusBarStyle(), similarly.

The problem is basically that updateProperties() in Qt's qiosviewcontroller.mm overrides preferredStatusBarStyle.

Hi JP,
thx
will try out to connect to orientationChanged()signal

Hi JP, tried to connect to orientationChanged() SIGNAL, but couldn't figure out the correct connect syntax for
QGuiApplication::primaryScreen::orientationChanged

Would something like this work?
QScreen *screen = qApp->primaryScreen();
screen->setOrientationUpdateMask(Qt::PortraitOrientation | Qt::LandscapeOrientation | Qt::InvertedPortraitOrientation | Qt::InvertedLandscapeOrientation);
QObject::connect(screen, &QScreen::orientationChanged, qApp, [](Qt::ScreenOrientation) {
    updatePreferredStatusBarStyle();
}, Qt::UniqueConnection);

aaah - the trick was
QScreen *screen = qApp->primaryScreen();
have thought I could do it in one step from
QGuiApplication::primaryScreen::orientationChanged
it gets the signal and updatePreferredStatusBarStyle was called, but the theme wasn't set.
So it seems I should stay with my QML Timer solution: at orientation changed setting the opposite theme, start Timer and reset default theme

Fixed by #5.


install expect package in dev/test images

What this PR does / why we need it:

Installs the expect package as part of the dev/test images.
Which issue(s) this PR fixes:

SC-113140
Does this PR require a test?

NONE
Does this PR require a release note?

NONE

Does this PR require documentation?

NONE

Merging as this does not affect tests yet.


DecalBatch should explicitly bind textures to unit 0.

Please ensure you have given all the following requested information in your report.
Issue details
When using a DecalBatch along with custom texture binding, Material.set() called in DecalBatch.render() is not guaranteed to bind to texture unit 0 anymore. This results in garbage results while rendering. An easy fix would be to add this line before calling Material.set():
Gdx.gl.glActiveTexture(GL20.GL_TEXTURE0);
Reproduction steps/code
Manually binding any texture to texture in prior to call DecalBatch.flush() will trigger this. The reason this is happening is because Material.set() does not explicitly bind the texture to a fixed unit. Instead, the currently active texture unit state is used. The shader program used for decal rendering on the other hand requires u_texture to be bound at texture unit 0.
Version of LibGDX and/or relevant dependencies
1.7.2
Please select the affected platforms

[x] Android
[x] iOS
[x] HTML/GWT
[x] Windows
[x] Linux
[x] MacOS

https://github.com/libgdx/libgdx/wiki/More-Info


tickets/DM-34540: Generate telegraf/ds configuration via helm rather than externally

A little commentary to aid in understanding this change:
I removed all the individual environment config files from each of telegraf and telegraf-ds.
Then I created a new template, for a configmap, which is then used as the telegraf configuration file.  That template loops over enabled applications at a given instance (and in the telegraf case, over all Prometheus-enabled apps, selecting only the enabled ones) and builds input and output configurations for each app.
The list of enabled applications is provided as a global variable, as is (now) the instance hostname and the vault-secrets path.
Currently Prometheus endpoints are provided in the telegraf values.yaml file and so would need to be manually updated as we added more prometheus-enabled services.  An operator-based approach (perhaps telegraf-operator itself) that relied on pod or deployment annotations to know to harvest metrics would be more elegant.


When an update is made to a connector, it takes too long to reflect the update

Submitted By
@Vitaliisorokin - GitHub profile name
Your Role

What is your role in this learning context?

Are you a student
career changer
professional developer?

For Product

What do you have feedback on?

A Microsoft Service
A Microsoft Tool
A Technology Topic
Specify a name and share a link to the learning resource you were using

Description
(1-2 lines that describe the topic or focus of your feedback)
One thing that was a little annoying, after updating my connector It wasn't updated immediately, so I was confused while testing it with PA :)

Details
(use this section to provide more details on the feedback if possible)
I had to wait 5 - 10 minutes for the connector to be updated.

To reproduce the error, take these steps: describe steps you took
(Optional) Take and share a screenshot if possible
Improving this experience will have these benefits: describe how it improves developer experience

Hi Julia,
Back in the summer, I tried to make a custom connector and at that time I experienced this delay. But now it seems like this problem is solved and when I am updating the connector from CLI I see that updates are reflected in Power Automate almost instantly.
During the 30 days of fusion, I haven't noticed any issues.


Mocking module imports when loading the app

What does this PR do?

Fixes https://github.com/Lightning-AI/lightning/issues/15428
Does your PR introduce any breaking changes? If yes, please list them.

Before submitting

[ ] Was this discussed/approved via a GitHub issue? (not for typos and docs)
[ ] Did you read the contributor guideline, Pull Request section?
[ ] Did you make sure your PR does only one thing, instead of bundling different changes together?
[ ] Did you make sure to update the documentation with your changes? (if necessary)
[ ] Did you write any new necessary tests? (not for typos and docs)
[ ] Did you verify new and existing tests pass locally with your changes?
[ ] Did you list all the breaking changes introduced by this pull request?
[ ] Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review
Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

[ ] Is this pull request ready for review? (if not, please submit in draft mode)
[ ] Check that all items from Before submitting are resolved
[ ] Make sure the title is self-explanatory and the description concisely explains the PR
[ ] Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?
Make sure you had fun coding 🙃

Closing for now. @tchaton feel free to reopen, otherwise let's keep discussion in the issue.


Classrooms: Add restrictions on when users can change school and grade

Who is this PR for?
K-5 teaching teams, part of https://github.com/studentinsights/studentinsights/issues/1696.
What problem does this PR fix?
When this was first made, the sequence was linear and users could move back and forth freely.  Changing that made the state transitions more complicated, and a scenario that wasn't handled was if a user changed the school or grade after either the list of students or the existing class list had been loaded (or requested).  @alexsoble ran into this right away in helpful testing things out yesterday.
What does this PR do?
This restricts when users can change those values.  An assumption is that folks will proceed linearly, but if not and they do want to go back to change this, there's a message telling them to just go create a new class list instead.
Screenshot (if adding a client-side feature)

Checklists

[x] Author checked latest in IE - Class lists
[x] Author improved specs for code in need of better test coverage
[x] Author included specs for new code

selfie


Tracking and aligning ROIs and tracks

We're close, but still aligning is elusive. Seems to work fine for CSD tracks, but still no cigar for DTI tracks. What's that all about?

Seems like have it now, with CSA-based tracking:
https://github.com/arokem/AFQ-notebooks/blob/31ebaa9ae318190b71353c3cf674aa6d0c740f83/CSA-tracking-white-matter.ipynb

The AFQ registration notebook looks good. But there seems to be an error in
the CSA-tracking, at least in how it renders on github
Jason D. Yeatman, PhD
Assistant Professor, Institute for Learning & Brain Sciences (I-LABS)
Department of Speech & Hearing Sciences
University of Washington
http://depts.washington.edu/bdelab/
On Wed, Sep 2, 2015 at 5:02 PM, Ariel Rokem<EMAIL_ADDRESS>wrote:

Seems like have it now, with CSA-based tracking:
https://github.com/arokem/AFQ-notebooks/blob/31ebaa9ae318190b71353c3cf674aa6d0c740f83/CSA-tracking-white-matter.ipynb
—
Reply to this email directly or view it on GitHub
https://github.com/arokem/AFQ-notebooks/issues/2#issuecomment-137276962.

You mean the "invalid value in divide"? I think that's a benign warning,
but I will investigate!
On Wed, Sep 2, 2015 at 6:56 PM, Jason D. Yeatman<EMAIL_ADDRESS>wrote:

The AFQ registration notebook looks good. But there seems to be an error in
the CSA-tracking, at least in how it renders on github
Jason D. Yeatman, PhD
Assistant Professor, Institute for Learning & Brain Sciences (I-LABS)
Department of Speech & Hearing Sciences
University of Washington
http://depts.washington.edu/bdelab/
On Wed, Sep 2, 2015 at 5:02 PM, Ariel Rokem<EMAIL_ADDRESS>wrote:

Seems like have it now, with CSA-based tracking:

https://github.com/arokem/AFQ-notebooks/blob/31ebaa9ae318190b71353c3cf674aa6d0c740f83/CSA-tracking-white-matter.ipynb

—
Reply to this email directly or view it on GitHub
<https://github.com/arokem/AFQ-notebooks/issues/2#issuecomment-137276962
.

—
Reply to this email directly or view it on GitHub
https://github.com/arokem/AFQ-notebooks/issues/2#issuecomment-137300410.


Small fix for tophittersoperator.md

@kendaleiv : Thanks for your contribution! The author(s) have been notified to review your proposed change.

Looks good. Thanks @kendaleiv.


chore(CI): add a retry mechanism for Axon node's health_check

What this PR does / why we need it?
This PR resolves #1465.
What is the impact of this PR?
No Breaking Change
CI Settings

CI Usage
Tip: Check the CI you want to run below, and then comment /run-ci.
CI Switch

[x] Web3 Compatible Tests
[ ] OCT 1-5 And 12-15
[ ] OCT 6-10
[ ] OCT 11
[ ] OCT 16-19
[ ] v3 Core Tests

CI Description

CI Name
Description

Web3 Compatible Test
Test the Web3 compatibility of Axon

v3 Core Test
Run the compatibility tests provided by Uniswap V3

OCT 1-5 | 6-10 | 11 | 12-15 | 16-19
Run the compatibility tests provided by OpenZeppelin

/run-ci


API does not properly reject bad API request when assigning product categories

Description
Sending the following “bad” PATCH-Request to /api/v3/product/<productID> will result in an error 500.
{"categories":[{"2":{"id":"<cat1id>"},"3":{"id":"<cat2id>"}}]}

Possible Solution
The API should properly validate the request and respond with status 400 if it doesn't match the API's expectations.
Additional context
Argument 2 passed to Shopware\Core\Framework\Api\Converter\ApiVersionConverter::isFromFuture() must be of the type string or null, int given, called in /var/www/shopware/vendor/shopware/core/Framework/Api/Converter/ApiVersionConverter.php on line 170

Kinda similar to #1307

Hello,
Within our company roadmap and work capacity, we try to address each bug or improvement request but admit that not each one will be resolved. To continue our culture of honesty and openness, we are closing this issue to focus on our roadmap on behalf of all Shopware users.
If you feel like this issue is still impacting you, please create a new issue and let us know.
Thank you again for your valuable feedback.


Bug Fix Popover

📌 issue
There was a bug that need double click to open popover due to focus and click event added.
In order to fix it, i remove focus and blur event.

Thank you @eddie0329 !  Would you mind comparing the new behavior with the Bootstrap documentation?  https://getbootstrap.com/docs/5.0/components/popovers/#dismiss-on-next-click
I had added focus/blur to cover this trigger, but my implementation was bad, sorry.

Thank you @eddie0329 !  Would you mind comparing the new behavior with the Bootstrap documentation?  https://getbootstrap.com/docs/5.0/components/popovers/#dismiss-on-next-click
I had added focus/blur to cover this trigger, but my implementation was bad, sorry.

Thank you @eddie0329 ! Would you mind comparing the new behavior with the Bootstrap documentation? https://getbootstrap.com/docs/5.0/components/popovers/#dismiss-on-next-click
I had added focus/blur to cover this trigger, but my implementation was bad, sorry.

Hello @bestguy . Do i have to implement dissmisable popover all the time? Or should user can control it?

Thank you @eddie0329 ! Would you mind comparing the new behavior with the Bootstrap documentation? https://getbootstrap.com/docs/5.0/components/popovers/#dismiss-on-next-click
I had added focus/blur to cover this trigger, but my implementation was bad, sorry.

Hello @bestguy . Do i have to implement dissmisable popover all the time? Or should user can control it?

Good question @eddie0329 , let me see what the best default is here.  It seems like tag choice matters, but not sure if that helps us.  Will look closer this weekend.

Good question @eddie0329 , let me see what the best default is here.  It seems like tag choice matters, but not sure if that helps us.  Will look closer this weekend.

Hello @eddie0329 , I think we might want to change this even further.  Can you please review #223 and let me know if would correct this issue?

Hello @eddie0329 , I think we might want to change this even further.  Can you please review #223 and let me know if would correct this issue?

Corrected in #223, and published in<EMAIL_ADDRESS>Thank you @eddie0329 !

Corrected in #223, and published in<EMAIL_ADDRESS>Thank you @eddie0329 !


New features

I created a factory to facilitate the creation of new methods, as well as I also created a new method that allows mentioning other users.
Unfortunately, you can not keep the name of the robot as the first parameter, as it is optional (since our webhook has a name when registering in the discord itself), as it is an optional parameter, the same must be at the end.
I'll use it that way, feel free to accept or reject it.

Due to the fact that this is purely an improvement to the custom message send function, which is already deprecated I've decided against accepting this request. This strays too much from the current API which I'm happy with and this PR reduces code quality.
I will, however, be implementing a selection of preset colours in the coming weeks for MessageBuilder.
Thank you for your contribution.


Fix helm template error

Fix template #5518

Pull Request Test Coverage Report for Build 8018

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.02%) to 63.914%

Totals

Change from base Build 8012:
0.02%

Covered Lines:
11998

Relevant Lines:
18772

💛  - Coveralls


openzfs 1.5.2

Needs the PR template filled out, next time.

@adidalal For simple bumps like this, you may want to adopt a cask-ified version of https://github.com/Homebrew/brew/blob/master/Library/Homebrew/dev-cmd/bump-formula-pr.rb :)

Cask-repair already does that, afaik

Yep. cask-repair also only needs the version (it calculates the sha256 for you), takes care of appcast as well, and fills out the template automatically.

@vitorgalvao does it work if no appcast is available?

Yes.


ShaderMask needs to play nicely with compositing

This patch adds ShaderLayer and makes ShaderMask use it.
Fixes #1155

@Hixie gave me a verbal LGTM


chore(master): release butler 11.1.2

:robot: I have created a release beep boop
11.1.2 (2024-04-08)
Miscellaneous

Debug Docker image build code in GitHub Actions (db36afa)
Debug Docker image bulild code (c9748e1)
deps: Update dependencies (cc7bf2c)
Docker build debug (d4dc1b8)
Fix broken Docker image build and update versioning to 1.2.3 format (7e96da5)
master: release butler 11.1.1 (46707ff)
master: release butler 11.1.2 (242c347)
master: release butler 11.1.2 (049a938)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/ptarmiganlabs/butler/releases/tag/untagged-5da5e41714416a220451 :sunflower:


Expose Defterm info to ComboBoxItem properly

Summary of the Pull Request
Makes Model::DefaultTerminal an IStringable, which presents it as a string, when possible. This enables text search for defterm's setting. This also enables screen readers to identify the combo box items by their text content (app name, author, and version) as opposed to being treated as an item containing more text. As a part of that, I cleaned up the UIA tree to treat the item's name as "<name>, <author>, <version>". This is consistent with how the Settings App presents installed apps in Apps > Installed apps.
#11251 will be resolved upon verification by the accessibility team.
Validation Steps Performed
Verified using Narrator and Accessibility Insights.

@msftbot merge this in 5 minutes


Cross-model relations fail after requiring pod changes

Bug Description
I have 2 Juju models attached to a single controller. One is a k8s model and the other is a machine model.
In the machine model Vault is deployed.
In the k8s model an app that relates to Vault is deployed.
When I initially relate the two via cross-model relations, there is no issue and juju show-unit <my-app>/0 shows the data bag is populated as expected. Importantly the credentials field is as follows:
credentials: '{"05445a7c9caedbff38042e8cf56611bd": "secret://77577413-d57f-481b-854f-a1231e017724/cqcmqbc6qn7vjti34kg0"}'

After running juju attach-resource <my-app> resource=... to debug some issues on my app, the pod is restarted and I start to get an error secret not found. Running juju show-unit <my-app>/0 shows that the credentials have changed.
credentials: '{"05445a7c9caedbff38042e8cf56611bd": "secret:cqcmqbc6qn7vjti34kg0"}'

Notice that the model reference is gone so the charm expects to find the secret in the model and fails. I'm not certain whether this is a charm bug or a Juju bug.
To Reproduce

Bootstrap a controller on lxd
Add microk8s as a cloud to the controller (this may require some tweaks to microk8s client config depending on your setup as it took me some trial and error to figure out how to get working - this Matrix discussion may be helpful)
juju add-model source localhost
juju deploy vault --channel=1.15/stable
juju offer vault
juju add-model destination microk8s
juju deploy juju-jimm-k8s jimm --channel=latest/edg (or use any charm that relates to Vault)
Relate vault to app - juju relate jimm <offer-url>
Wait until juju show-unit jimm/0 shows the data bag is populated.
Either run what I did juju attach-resource jimm jimm-image=foo or try kubectl delete pod on the workload.
Run juju show-unit jimm/0 to see if the data bag is updated as expected.

Environment
This was consistently replicated locally as described above.
$ microk8s version
MicroK8s v1.28.10 revision 6829

snap info microk8s
tracking:     1.28-strict/stable

juju version
3.5.2-genericlinux-amd64

App         Version  Status  Scale  Charm       Channel      Rev  Exposed  Message
vault                active      1  vault       1.15/stable  353  yes

Relevant log output
unit-jimm-0: 21:37:08 ERROR unit.jimm/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-jimm-0/charm/venv/ops/model.py", line 3049, in _run
    result = subprocess.run(args, **kwargs)  # type: ignore
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-jimm-0/secret-get', 'secret:cqcmqbc6qn7vjti34kg0', '--format=json')' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-jimm-0/charm/venv/ops/model.py", line 3397, in secret_get
    result = self._run('secret-get', *args, return_output=True, use_json=True)
  File "/var/lib/juju/agents/unit-jimm-0/charm/venv/ops/model.py", line 3051, in _run
    raise ModelError(e.stderr) from e
ops.model.ModelError: ERROR secret "secret:cqcmqbc6qn7vjti34kg0" not found

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-jimm-0/charm/./src/charm.py", line 838, in <module>
    main(JimmOperatorCharm)
  File "/var/lib/juju/agents/unit-jimm-0/charm/venv/ops/main.py", line 544, in main
    manager.run()
  File "/var/lib/juju/agents/unit-jimm-0/charm/venv/ops/main.py", line 520, in run
    self._emit()
  File "/var/lib/juju/agents/unit-jimm-0/charm/venv/ops/main.py", line 509, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-jimm-0/charm/venv/ops/main.py", line 143, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-jimm-0/charm/venv/ops/framework.py", line 350, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-jimm-0/charm/venv/ops/framework.py", line 849, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-jimm-0/charm/venv/ops/framework.py", line 939, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-jimm-0/charm/./src/charm.py", line 227, in _on_jimm_pebble_ready
    self._update_workload(event)
  File "/var/lib/juju/agents/unit-jimm-0/charm/src/state.py", line 25, in wrapper
    return func(self, event)
  File "/var/lib/juju/agents/unit-jimm-0/charm/./src/charm.py", line 355, in _update_workload
    vault_config = self._vault_config()
  File "/var/lib/juju/agents/unit-jimm-0/charm/./src/charm.py", line 269, in _vault_config
    secret = self.model.get_secret(id=unit_credentials)
  File "/var/lib/juju/agents/unit-jimm-0/charm/venv/ops/model.py", line 284, in get_secret
    content = self._backend.secret_get(id=id, label=label)
  File "/var/lib/juju/agents/unit-jimm-0/charm/venv/ops/model.py", line 3400, in secret_get
    raise SecretNotFoundError() from e
ops.model.SecretNotFoundError

Additional context
No response

Hello @kian99 thank you for opening this issue. We should be looking at it in the next couple of days. Thank you.

Issue is being addressed in #254 and a bug report has been created in Juju https://bugs.launchpad.net/juju/+bug/2075153


Error in pooling layers, incorrect number of outputs

The output_space.shape of convolution layers does not seem to reflect the pool_shape parameter when printed. It's possible the logic is still correct and a different value should be fetched. Check this works as expected and add a test!

This is definitely a bug, and a tough one to fix. I don't understand PyLearn2's code yet.
Trying to setup a pooling for 32x32 -> 16x16 one element gets lost each time, despite exposing the kernel_stride additional parameter.
Debug output says this:
Input shape: (32, 32)
Detector space: (15, 15)
Output space: (14, 14)

I pushed some code to help investigate this, but the issue is still open.

The issue here is that while border_mode='valid' is intuitive, on the other hand border_mode='full' is different to border_mode='same' (which is what you'd intuitively expect).  The same mode is not implemented in PyLearn2 yet.
Pooling doesn't make sense except with same because from 32x32 you may end up with arbitrary sizes like 14, 15, 17 depending on the kernel_size, etc.  It's a pain to use...

It's possible border_mode='half' from Theano may be the answer:
https://github.com/Theano/Theano/issues/2118
https://github.com/Theano/Theano/pull/2126
This has been merged already...

It's only available on the CUDA version, and not the CPU version.
https://groups.google.com/forum/#!topic/theano-users/TqS3gAhyGUM
This is work in progress for v2 apparently.

Now supported by the Lasagne backend.


Do we have a document that have details on how we can install and proceed with velero

Hi team,
I had been using ark and now after the change we have from today I am a bit concerned with the stability of the same.
I could not find much on velero, as I tried using brew install velero to install the client but was not possible and then I found issues while trying to install the velero server on my cluster.
As the resources are in failure states:
(⎈ |devops3-vlocity:default)➜  vof_xom_deployment_develop_branch git:(feature/XOM-4370-cluster-ark-backup) ✗ kubectl get pods,deployments -n backup
NAME                         READY     STATUS              RESTARTS   AGE
po/restic-7mpdc              0/1       ContainerCreating   0          8s
po/restic-d9rkl              0/1       ImagePullBackOff    0          8s
po/restic-jbx4c              0/1       ContainerCreating   0          8s
po/velero-54d5cd5cc5-lkx4l   0/1       ContainerCreating   0          8s
NAME            DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/velero   1         1         1            0           8s
(⎈ |devops3-vlocity:default)➜  vof_xom_deployment_develop_branch git:(feature/XOM-4370-cluster-ark-backup) ✗ kubectl describe po/velero-54d5cd5cc5-lkx4l -n backup
Name:           velero-54d5cd5cc5-lkx4l
Namespace:      backup
Node:           ip-172-18-5-8.us-east-2.compute.internal/<IP_ADDRESS>
Start Time:     Wed, 06 Feb 2019 22:06:52 +1100
Labels:         component=velero
pod-template-hash=1081781771
Annotations:    prometheus.io/path=/metrics
prometheus.io/port=8085
prometheus.io/scrape=true
Status:         Pending
IP:
Controlled By:  ReplicaSet/velero-54d5cd5cc5
Containers:
velero:
Container ID:
Image:         gcr.io/heptio-images/velero:latest
Image ID:
Port:
Command:
/velero
Args:
server
State:          Waiting
Reason:       ContainerCreating
Ready:          False
Restart Count:  0
Environment:
VELERO_SCRATCH_DIR:  /scratch
AWS_CLUSTER_NAME:    vlocity
Mounts:
/plugins from plugins (rw)
/scratch from scratch (rw)
/var/run/secrets/kubernetes.io/serviceaccount from velero-token-w4h86 (ro)
Conditions:
Type           Status
Initialized    True
Ready          False
PodScheduled   True
Volumes:
plugins:
Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
scratch:
Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
velero-token-w4h86:
Type:        Secret (a volume populated by a Secret)
SecretName:  velero-token-w4h86
Optional:    false
QoS Class:       BestEffort
Node-Selectors:
Tolerations:
Events:
Type     Reason                  Age   From                                               Message

Normal   Scheduled               20s   default-scheduler                                  Successfully assigned velero-54d5cd5cc5-lkx4l to ip-172-18-5-8.us-east-2.compute.internal
Normal   SuccessfulMountVolume   19s   kubelet, ip-172-18-5-8.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "scratch"
Normal   SuccessfulMountVolume   19s   kubelet, ip-172-18-5-8.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "plugins"
Normal   SuccessfulMountVolume   19s   kubelet, ip-172-18-5-8.us-east-2.compute.internal  MountVolume.SetUp succeeded for volume "velero-token-w4h86"
Warning  FailedCreatePodSandBox  10s   kubelet, ip-172-18-5-8.us-east-2.compute.internal  Failed create pod sandbox: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "velero-54d5cd5cc5-lkx4l_backup" network: open /run/flannel/subnet.env: no such file or directory
Normal   SandboxChanged          7s    kubelet, ip-172-18-5-8.us-east-2.compute.internal  Pod sandbox changed, it will be killed and re-created.
Any docs that let us know how to proceed and also we need to  stable version and a stable version of the helm chart for the same

Hi @Sam123ben, we apologize if there has been any confusion. Until we release Velero v0.11.0, please continue to use Ark v0.10.1. The documentation for that release is at https://heptio.github.io/velero/v0.10.0/. As far as I know, the Helm chart should continue to function, since it still points to the ark v0.10.1 image:
https://github.com/helm/charts/blob/08c82581a13ddc1409fffdd22dbc2c90db738c95/stable/ark/values.yaml#L2-L3.
If you continue to have issues, would it be ok with you if we moved the conversation to our Google Group - https://groups.google.com/forum/#!forum/projectvelero?

Closing this as we'll have full migration documentation along with the release of v0.11 in the next few weeks.


Converter Node/Pixel mixup

https://github.com/carla-simulator/carla/blob/860dbe908e7a81232bdba6853943707f6b0102c2/PythonClient/carla/planner/converter.py#L148-L149
The above lines output an int list. It should be a float list. For example, this causes the following error:
converter._check_input_type(map.convert_to_pixel([10.,10.,10.])) == NODE  # returns True, but should return False
Possibly due to a python2/python3 mixup where math.floor returns float/int respectively

Very good ! I will try to update that now.
Thanks.


I let Trintiy run over night, trying to sync mainnet, came back to just getting tons of errors about Too many open files:
OSError: [Errno 24] Too many open files
   ERROR  05-30 10:58:52  base_events  socket.accept() out of system resource
socket: <socket.socket fd=15, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('<IP_ADDRESS>', 30304)>
Traceback (most recent call last):
  File "/usr/lib/python3.6/asyncio/selector_events.py", line 179, in _accept_connection
    conn, addr = sock.accept()
  File "/usr/lib/python3.6/socket.py", line 205, in accept
OSError: [Errno 24] Too many open files
   ERROR  05-30 10:58:52  base_events  socket.accept() out of system resource
socket: <socket.socket fd=15, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('<IP_ADDRESS>', 30304)>
Traceback (most recent call last):
  File "/usr/lib/python3.6/asyncio/selector_events.py", line 179, in _accept_connection
    conn, addr = sock.accept()
  File "/usr/lib/python3.6/socket.py", line 205, in accept
OSError: [Errno 24] Too many open files

How can it be fixed
I don't know...

FYI I did update to 0.1.0a5 and start fresh, it's been syncing mainnet all night

I'm seeing the same thing.  Will re-open if this re-surfaces.

I'm seeing the same thing.  Will re-open if this re-surfaces.


Quartz job instrumentation - distributed tracing pass through JobDataMap

Is your feature request related to a problem?
We have some Quartz jobs immediately triggered by user requests. Both (job and user request) create transaction in APM, with no link between them: user request appears in "request" transactions, Quartz job appears in "scheduled" transaction, as per design I guess.
We would like to keep trace between user request and the triggered job (when jobs are immediately triggered, maybe not when they're scheduled).
Describe the solution you'd like
I think Quartz Job APM plugin could retrieve headers from JobDetailMap if any, thus keeping trace between  parent and child transactions.
It could be done there:
https://github.com/elastic/apm-agent-java/blob/9c4e9351dd39b4413b2fb4990a73d91674201838/apm-agent-plugins/apm-quartz-job-plugin/quartz-common/src/main/java/co/elastic/apm/agent/quartzjob/AbstractJobTransactionNameInstrumentation.java#L117
We could start a child transaction with few modifications:
AbstractJobTransactionNameInstrumentation line 84:
if (transactionName != null) {
    transaction = createAndActivateTransaction(clazz, transactionName, jobExecutionContext, helper);
} else {
    logger.warn("Cannot correctly name transaction for method {} because JobExecutionContext is null or lacking job details", signature);
    transaction = createAndActivateTransaction(clazz, signature, jobExecutionContext, helper);
}

AbstractJobTransactionNameInstrumentation.java line 115:
@Nullable
private static <T> Transaction createAndActivateTransaction(Class<?> originClass, String name, T jobExecutionContext, JobExecutionContextHandler<T> helper) {
    Transaction transaction = GlobalTracer.get().startChildTransaction(helper.getJobDataMap(jobExecutionContext), key -> dataMap.getString(key), PrivilegedActionUtils.getClassLoader(originClass));
// ....

Add getJobDataMap to https://github.com/elastic/apm-agent-java/blob/main/apm-agent-plugins/apm-quartz-job-plugin/quartz-common/src/main/java/co/elastic/apm/agent/quartzjob/JobExecutionContextHandler.java:
@Nullable
JobDataMap getJobDataMap(T jobExecutionContext);

Describe alternatives you've considered
I've considered implementing it with APM Agent API like this:
Job scheduling:
Span span = ElasticApm.currentSpan();
if (span != null) {
  span.injectTraceHeaders((name, value) -> jobDetail.getJobDataMap().put(name, value));
}
/*
 this.scheduler.scheduleJob(jobDetail, trigger);
*/

Job execution:
Transaction transaction = ElasticApm.startTransactionWithRemoteParent((key) -> jobExecutionContext.getJobDetail().getJobDataMap().getString(key));
try (final Scope scope = transaction.activate()) {
  customRunJob(jobExecutionContext);
} catch (Exception e) {
  transaction.captureException(e);
  throw e;
} finally {
  transaction.end();
}

It does create a transaction, but interrupt transaction created by apm-quartz-job-plugin. I lose some spans (executed before I start the transaction), which appear in the plugin transaction. I think implementing distributed tracing between executor and job execution at plugin level would be more accurate.
Additional context

Hi Sylvain,
Sorry for the late response. I'm working with @ssssteffff . I can confirm that this approach would totally work for us. For your second question, those async jobs are trigger immediately in our codebases.
Have a good day !

Thanks for your feedback, we plan to attempt doing something about this in our next milestone, so hopefully we might have something to test in a not-so-distant future.


hcf completed

hcf completed :zany_face:

Code is already there!

Please work on different issue

Hey I did see it. :sob:
But my code was make it in javaScript  and the code in the Folder Math is cpp
Can we push code in any language?

i am sorry, i mistook.

Thank you.
Im appreciate it
:smiley_cat:


Ensure brackets are closed in PoolProperties.toString

Fixes minor annoyance when analyzing log files

Thanks for the PR.


Module 3 exercises [Celestina Deku]

Please "drag & drop" exercise 3 file below.

Please UPDATE THE TITLE of this "Issue" in the title bar above with your name.

If you haven't done so, please go back to Module 3 and make sure to take
the post-training survey.

WS2_M3E1_Template.docx_Celestina Deku.docx

Well done, Celestina. You clearly grasped the material outlined in Module 3 and successfully applied it to these exercises. Thank you for your participation! Please be sure to fill out the post-Module survey before you begin the next Module.


Docs: internal links are broken when viewing via a URI fragment (hash) URL

Describe the bug
When viewing a page from a URL that includes a #topic fragment pointer thingy, internal links are broken.
(Firefox 86.0.1, Ubuntu)
To Reproduce
Steps to reproduce the behavior:

Go to https://docs.statechannels.org/docs/protocol-tutorial/off-chain-funding/#indirect-funding
Click on earlier in the tutorial

Diagnostics
The broken earlier in the tutorial link points to
https://docs.statechannels.org/docs/protocol-tutorial/off-chain-funding/deposit-assets, instead of
https://docs.statechannels.org/docs/protocol-tutorial/deposit-assets.
The issue stems from a window.location redirect of a sort from /route#heading to route/#heading. Internal relative links are truncating only /#-heading when they should truncate the route as well.
Additional context
The search function produces URLs with # fragments and this issue comes into effect.

Probably this caused by a netlify setting Pretty URLs:
https://github.com/facebook/docusaurus/issues/2394#issuecomment-599039072
Netlify may be rewriting route#heading as route/#heading, which breaks docusaurus' method for making internal relative links.