SFTP is way too slow when using Read and Write methods

Hi there,
This issue is not new to this project actually, but I don't know why it has been closed, since the issue hasn't been solved at all.
The regarding issue is #166
The difference here is that this issue happens also with write and read.
Basically I have a method which opens the remote file for write:
fn send_file(
        &mut self,
        local: &FsFile,
        file_name: &Path,
    ) -> Result<Box<dyn Write>, FileTransferError> {
        match self.sftp.as_ref() {
            None => Err(FileTransferError::new(
                FileTransferErrorType::UninitializedSession,
            )),
            Some(sftp) => {
                let remote_path: PathBuf = self.get_abs_path(file_name);
                // Calculate file mode
                let mode: i32 = match local.unix_pex {
                    None => 0o644,
                    Some((u, g, o)) => ((u as i32) << 6) + ((g as i32) << 3) + (o as i32),
                };
                match sftp.open_mode(
                    remote_path.as_path(),
                    OpenFlags::WRITE | OpenFlags::CREATE | OpenFlags::APPEND | OpenFlags::TRUNCATE,
                    mode,
                    OpenType::File,
                ) {
                    Ok(file) => Ok(Box::new(BufWriter::with_capacity(8192, file))),
                    Err(err) => Err(FileTransferError::new_ex(
                        FileTransferErrorType::FileCreateDenied,
                        format!("{}", err),
                    )),
                }
            }
        }
    }

Then the calling method receives the Write trait in a Box and just writes data from a local file using buffers with size 8192.
loop {
    // Read till you can
    let mut buffer: [u8; 8192] = [0; 8192];
    match fhnd.read(&mut buffer) {
        Ok(bytes_read) => {
            total_bytes_written += bytes_read;
            if bytes_read == 0 {
                break;
            } else {
                // Write bytes
                if let Err(err) = rhnd.write(&buffer[0..bytes_read]) {
                    /* ... */
                }
            }
        }
        Err(err) => {  /* ... */      }
    }
}

The code which writes the stream, is the same which, based on the user configuration, works with an FTP stream or with an SCP file returned by the library function scp_send.
Theorically, the performance of SCP and SFTP should be almost the same, but not in this case.
I'm sending a file with size of 64MB to a machine in my local network (ethernet, speed 100mbps).
When using SCP it takes 9.50 seconds to write the file to the remote machine, which is pretty good. With scp command line tool it takes 7 seconds for me, so it's good, especially considering that my implementation performs other operations at each write cycle.
On the other hand, when using SFTP, the same file (and I repeat: also the same code), takes 44.81 seconds to be written to the same remote machine.
I also tried to wrap the SFTP file handler into a BufWriter, but nothing has really changed (maybe improved the performance by 5%).
The same phenomenon happens when reading files from remote: SCP is fast as it should be, while SFTP is way slower (I must admit, when reading is anyway much faster than when writing, but anyway, still slow).
I'm then asking you, if this is a problem of my code, or if there is actually something blocking the I/O when using SFTP.
Could you please check?
Best regards,
Christian

Regarding BufWriter: if the file handle you are reading from gives you lots of small fragmented reads then it makes sense to use BufWriter.  If the source file handle is a regular file then the structure of your copy loop means that it won't give a huge benefit.
However: pay attention to short writes in your copy loop.  In general, any write can succeed but write less than you requested, leaving a remnant for you to write as a follow up.  If you don't handle that case then you'll end up effectively corrupting the destination.
Regarding performance: you've selected 8192 bytes for your buffer size, but the discussion on the issue that you linked to says that a larger buffer size is better.
Why is bigger better?  Consider that each time you call write the ssh session needs to do some book-keeping and produce an encrypted packet to send to the other side, each of these has a certain amount of overhead for both the sender and the receiver.   If you have 64MB of data and split it into 8KB chunks there will be more of those to send to the remote than if you used a larger buffer size.   The reality is a bit more complicated as there is a windowing/scaling system in the ssh protocol that I confess to not fully understanding, but the upshot of it is that bigger is better, up until a point, after which making the buffer larger will have diminishing returns.
What should you do?  My general recommendation would be something like this so that you can control the effective buffer size; have a BufWriter with a larger internal buffer than the default used by std::io::copy will result in fewer calls to the underlying write in the ssh protocol.
/// Use large buffers for the copy operation, leaving it to std::io::copy to handle the slightly
/// fiddly short read/write cases.
/// As-written, this will consume the input handles, but you could decompose this and just
/// wrap them up elsewhere in the program
fn copy_with_large_buffer<R, W>(
  source: R,
  destination: W,
  buf_size: usize
) -> std::io::Result<u64>
where
  R: Read,
  W: Write
{
   let source = BufReader::with_capacity(buf_size, source);
   let destination = BufWriter::with_capacity(buf_size, destination);
   std::io::copy(source, destination)
}

copy_with_large_buffer(fhnd, rhnd, 32*1024)?;

I managed to increase the buffer size to 65536 (and to solve problems related to my write process, which didn't check the write return value). The process takes now half the time it used to take before. Now indeed to write 64MB it takes up to 20 seconds, which is decent, but still, double of scp.
In my opinion there is something which is wrong with the Sftp IO. I'd like to know why the same code, using a file handler retrieved from scp_send takes half of the time, while the file handler returned by open_mode doesn't.
Using copy doesn't make any difference:
// with fhnd as File
let mut fhnd = BufReader::with_capacity(65536, fhnd);
while total_bytes_written < file_size {
    match std::io::copy(&mut fhnd, &mut rhnd) {
        Ok(sz) => total_bytes_written += sz as usize,
        Err(err) =>
        {
            self.log(
            LogLevel::Error,
            format!("Could not read local file: {}", err).as_ref(),
            );
            self.input_mode = InputMode::Popup(PopupType::Alert(
                Color::Red,
                format!("Could not read local file: {}", err),
            ));
        }
    }
}

It still takes the same time it usually takes using write.

After further investigations, I was able to solve the issue using a buffer of size 65536.
And yes, my bad of course, even if there is a strange behaviour of SFTP when using some buffer sizes actually.
I've written a basic program to make the speedtests and I've tested different combinations.
extern crate ssh2;
extern crate tempfile;

use ssh2::{Session, Sftp};
use std::env;
use std::io::{BufReader, Read, Write};
use std::net::TcpStream;
use std::path::PathBuf;
use std::process::exit;
use std::time::Instant;

const BUFSIZE: usize = 65536;

fn write_stream(
    src: &mut BufReader<Box<dyn Read>>,
    dst: &mut tempfile::SpooledTempFile,
    file_size: usize,
) -> Result<(), std::io::Error> {
    let mut total_bytes_written: usize = 0;
    while total_bytes_written < file_size {
        // Read till you can
        let mut buffer: [u8; BUFSIZE] = [0; BUFSIZE];
        match src.read(&mut buffer) {
            Ok(bytes_read) => {
                total_bytes_written += bytes_read;
                if bytes_read == 0 {
                    continue;
                } else {
                    let mut buf_start: usize = 0;
                    while buf_start < bytes_read {
                        // Write bytes
                        match dst.write(&buffer[buf_start..bytes_read]) {
                            Ok(bytes) => buf_start += bytes,
                            Err(err) => {
                                return Err(err);
                            }
                        }
                    }
                }
            }
            Err(err) => return Err(err),
        }
    }
    Ok(())
}

fn main() {
    let args: Vec<String> = env::args().collect();
    if args.len() < 4 {
        eprintln!(
            "Usage: {} <address> <username> <password> <file_path>",
            args.get(0).unwrap()
        );
        exit(255);
    }
    let address: String = args.get(1).unwrap().clone();
    let file_path: PathBuf = PathBuf::from(args.get(4).unwrap().as_str());
    const SSH_PORT: u16 = 22;
    // Create session
    // Create session
    println!("Connecting to {}:{}", address, SSH_PORT);
    let tcp = TcpStream::connect(format!("{}:{}", address, SSH_PORT)).unwrap();
    // Create session
    let mut session = Session::new().unwrap();
    session.set_tcp_stream(tcp);
    session.handshake().unwrap();
    if let Err(err) =
        session.userauth_password(args.get(2).unwrap().as_str(), args.get(3).unwrap().as_str())
    {
        eprintln!("Could not connect to remote: {}", err);
        exit(1);
    }
    println!("Authentication succeeded");
    // Set blocking to true
    session.set_blocking(true);
    // Get SFTP
    let sftp: Sftp = session.sftp().ok().unwrap();
    // Create tempfile
    let mut sftp_dest: tempfile::SpooledTempFile = tempfile::SpooledTempFile::new(64 * 1024 * 1024); // 64MB
    let mut scp_dest: tempfile::SpooledTempFile = tempfile::SpooledTempFile::new(64 * 1024 * 1024); // 64MB
    // Get file size
    let file_size: usize = match sftp.stat(file_path.as_path()) {
        Ok(metadata) => metadata.size.unwrap() as usize,
        Err(err) => {
            eprintln!("Could not stat remote file: {}", err);
            exit(1);
        }
    };
    println!("File has size {}", file_size);
    let mut sftp_stream: BufReader<Box<dyn Read>> = match sftp.open(file_path.as_path()) {
        Ok(f) => BufReader::with_capacity(BUFSIZE, Box::new(f)),
        Err(err) => {
            eprintln!("Could not open remote file: {}", err);
            exit(1);
        }
    };
    println!("Downloading \"{}\" using SFTP...", file_path.display());
    let t_start: Instant = Instant::now();
    // Write to tempfile
    if let Err(err) = write_stream(&mut sftp_stream, &mut sftp_dest, file_size) {
        eprintln!("Could not write file: {}", err);
        exit(1);
    }
    println!("SFTP write took {}ms", t_start.elapsed().as_millis());
    // Now do the same thing using scp
    println!("Downloading \"{}\" using SCP...", file_path.display());
    let mut scp_stream: BufReader<Box<dyn Read>> = match session.scp_recv(file_path.as_path()) {
        Ok(f) => BufReader::with_capacity(BUFSIZE, Box::new(f.0)),
        Err(err) => {
            eprintln!("Could not open remote file: {}", err);
            exit(1);
        }
    };
    let t_start: Instant = Instant::now();
    // Write to tempfile
    if let Err(err) = write_stream(&mut scp_stream, &mut scp_dest, file_size) {
        eprintln!("Could not write file: {}", err);
        exit(1);
    }
    println!("SCP write took {}ms", t_start.elapsed().as_millis());
    session.disconnect(None, "Mandi!", None).unwrap();
    println!("Disconnected from remote");
}

I've noticed that using 65536 as bufsize, the speed difference between downloading file using SFTP and SCP is minimum:
32MB
Authentication succeeded
File has size 33554432
Downloading "/tmp/32m.dat" using SFTP...
SFTP write took 2874ms
Downloading "/tmp/32m.dat" usingn SCP...
SCP write took 2866ms
Disconnected from remote

256MB
Authentication succeeded
File has size 268435456
Downloading "/tmp/256m.dat" using SFTP...
SFTP write took 22967ms
Downloading "/tmp/256m.dat" usingn SCP...
SCP write took 22895ms
Disconnected from remote

While has a bigger offset when using for example 8192 as buffer size
Authentication succeeded
File has size 67108864
Downloading "/tmp/64m.dat" using SFTP...
SFTP write took 8043ms
Downloading "/tmp/64m.dat" using SCP...
SCP write took 5723ms
Disconnected from remote

Pretty much the same when using 2048 as buffer size
Authentication succeeded
File has size 67108864
Downloading "/tmp/64m.dat" using SFTP...
SFTP write took 6347ms
Downloading "/tmp/64m.dat" using SCP...
SCP write took 5729ms
Disconnected from remote

while a total disaster when using 4096
Authentication succeeded
File has size 67108864
Downloading "/tmp/64m.dat" using SFTP...
SFTP write took 19288ms
Downloading "/tmp/64m.dat" using SCP...
SCP write took 5753ms
Disconnected from remote

I think that's very curious actually, and I think it would be interesting discovering why this happens, but yeah, with 65536 works fine for me, even if there is also little offset in performance, but nothing I should be worried for.

Size (MB)
SFTP (seconds)
SCP (seconds)

32
3.159
3.039

64
6.320
5.742

128
12.856
11.472

256
24.107
23.065

512
47.435
46.092

1024
93.589
91.991

I believe this shall be added to the tests, or maybe in the readme or as example at least for any new user. I've been using (naively) write_all from the tests and was getting some timeouts on larger files (~20+ Kb)

If you would like to submit a PR, we would appreciate it.


config-drive: use hostname of VM instance of internal VM id

This fixes config drive to use VM's user provided host-name instead of
the internal VM instance ID for hostname related config in both
cloudstack and openstack metadata bundled in the ISO.
Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[x] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

GitHub Issue/PRs

How Has This Been Tested
I created a L2 network and deployed a VM, and checked contents in the mounted iso:
The openstack metadata folder:

The cloudstack metadata folder:

As confirmed via the unit test and above, the internal i-x-y is not longer seen in the iso metadata.
Checklist:

[x] I have read the CONTRIBUTING document.
[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
Testing
[x] I have added tests to cover my changes.
[x] All relevant new and existing integration tests have passed.
 JID-2054

@blueorangutan test

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-2676)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 44827 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2645-t2676-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_templates.py
Intermitten failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermitten failure detected: /marvin/tests/smoke/test_volumes.py
Intermitten failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 63 look OK, 4 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

test_02_vpc_privategw_static_routes
Failure
274.51
test_privategw_acl.py

test_03_vpc_privategw_restart_vpc_cleanup
Failure
272.73
test_privategw_acl.py

test_04_rvpc_privategw_static_routes
Failure
444.40
test_privategw_acl.py

test_04_extract_template
Failure
128.36
test_templates.py

test_02_not_secured_vm_migration
Error
1.09
test_vm_life_cycle.py

test_hostha_enable_ha_when_host_in_maintenance
Error
3.44
test_hostha_kvm.py

Merging this based on reviews and test results. I'll try to address intermittent issues in tests in a separate PR.

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✖debian. JID-2057


compressor confusion

I noticed that the recipe for creating a compressor requires a bronze plate.
Unless i am missing something, to create a bronze plate you need a compressor.
So i need a compressor to make a bronze plate, and i need a bronze plate to create a compressor?
confused?

I see two recipes for the piston...one which requires a bronze plate, and one which has three copper on the row where the bronze plate is. Do you.have that second recipe available?

there are two recipes: the first one with ingots is meant for you first compressor so you can make plates to craft more compressors for a cheaper price than with ingots.

I no longer play minecraft due to MS.


web: create a widget that exceeds the window size

I need to build a web app that is larger than the browser window. I can't dynamically generate the content because then the browser print mechanism only picks up the visible portions.
I've tried playing with OverflowBox and inheritedMediaQuery and can (almost) get this to work with the html renderer (but there are painting glitches). No luck with canvaskit: Only the content visible in the window is rendered.

I need to build a web app that is larger than the browser window.

Can you elaborate by providing a use case with code sample and what exactly are you trying to achieve ?


Fixed Subcircuit overflowing vertically

Fixes  https://github.com/CircuitVerse/CircuitVerse/issues/815
Screenshots of the changes (If any) -
Before:

After:

Coverage remained the same at 84.266% when pulling 4b6dffa56155bc1f3a1562cdefa8c19054827a9a on yash15601:subcircuit-overflows-vertically into f652ecafe7e9822670c01d402e043af7538480c7 on CircuitVerse:master.

@tachyons   I tryed not use !important but  Please Review this

@Shivansh2407  can you provide a screenshot please will do

Here you can see the space is quite more than as in the website which makes it look a bit weird.

@Shivansh2407  will fix this

@yash15601 still working on this?

Yess @satu0king


How to use onnx model

Thank you for your nice project.It helps my project a lot.Can you share the code to use onnx model?

You can refer to this tutorial: https://github.com/Tencent/ncnn/wiki/use-ncnn-with-pytorch-or-onnx


Request to review Wrong Serial no's in Issues.md after 74

Serial no after 74 should be 75 and 76 but misspelled as 55.

@AbhigyanSrivastav can you please raise the PR to fix this?


updated README.md

Please resolve conflicts. and the routes are now /v1/files and not /api/v1/files


Mouse Movement Stuttering

So, this seems like it could easily be a Wine-related problem, but I thought I'd report it anyway in case someone can reproduce it on Windows.
It seems like mouse movement sometimes causes the rendering to lag. I think it is more severe with faster mouse movement. If you move the mouse slowly, the animation will be smooth, but if you move the mouse quickly, the whole scene will stop rendering briefly, like everything is blocked on some kind of mouse position update.
However, I've also noticed that this does not occur if something like the inventory window is popped up. So whatever event/etc. is blocked by mouse movement runs more consistently while a window is open. I think this even happens on the title screen. if you wave the mouse around it can lag the temple flame animation, but getting to the character creation window alleviates the problem.
This also doesn't seem to be a problem with the original exe, but it affects T+ 1.0.84.
Sorry this is so vague. Graphics isn't really my forte.

Does the logfile provide any hints?
I'd have guessed it pathfinding related, but then you said it happens on the title screen too.

Actually, I think I'm going to close this, because I think my system may have been in a weird state. I can't reproduce it after a(n unrelated) restart. Possibly some weirdness caused by libraries being updated without the running kernel matching or something. If it starts happening again I'll look in the logs.
Sorry about the noise.

Hmmmmmm. Dunno about the logfiles.
I can look out for this on OT as well, there are certainly some areas where our DX11 renderer certainly isn't really optimal, especially when it comes to text rendering.
Do you have the ability to run "perf record" against a wine binary? I have no idea, tbh.


Conservative preciptation for RNNs.

Direct cloud conservative precip model trained:
https://wandb.ai/ai2cm/microphysics-emulation/runs/499ztpfp?workspace=user-aperkins


Is HorizonCalendar a good option for presenting dates? [Question]

Hi, thank you so much for this useful library, I've been playing with it for the past two days and I like it so far!
As I see HorizonCalendar is very suitable to be used in forms for date input, is it that good for presenting dates?
I need to display past events on a calendar like:

If I have a data source like Event:
//Simplified:
struct Event {
    let id: UUID
    let name: String
    let date: Date
}

And I will have thousands of days that have events.
One way to detect that a particular Day has an Event is in the .days { day in code block. But in my opinion, this is inefficient as I have to loop through each Event to see if the dates match to mark that Day in the calendar. So something like:
//HorizonCalendar's SwiftUI modifier
.days { day in
    getDayView(day: day)
}

//Later in View code:
private func getDayView(day: Day) -> some View {
    //Look through all Events
    for event in events {
        //Convert Day into Date
        if let date = calendar.date(from: day.components),
           //Check if that date has an Event
            calendar.isDate(event.date, inSameDayAs: date) {

            return SwiftUIDayView(dayNumber: day.day, event: event)
        }
    }

    return SwiftUIDayView(dayNumber: day.day, event: nil)
}

So I have to look through all the Events in each Day which is an O(n) difficulty and a loop within a loop.
Which brings me to my question of whether HorizonCalendar is suitable for presenting events in a calendar view?
Thanks again for your continued support!

@BredBurr you should restructure your data model (and cache it once you do so) so that you can do O(1) lookups. You would have this same issue using a UICollectionViewDataSource in cellForItemAt:, for example.


Can I install this without supervisor?

Hi, I am new to HA. The latest synology docker is not supporting supervisor, so I am not sure how to install zigber2mqtt? Is there a way to install this to HA?
And I have the cc2531 along with my synology
Thank you

I think you need to use the zigbee2mqtt docker option.
It doesn't install the HA addon though and works through the MQTT integration. I'd love to know how to install the add-on with HA and zigbee2mqtt both in docker.


File randomly not available right after upload via signed URL

We're uploading files from the browser using a signed URL. The files are not very big. Less than 20MB.
Once the upload has completed from the client, a request is made to our server to notify the upload has been completed. The server then checks if the file exists in the storage.
The problem we're having is that the exists() function returns [false] randomly even though we know the upload has completed 100% and we can actually see the file in the bucket a bit later.
This is the function we're using in the server to verify if the file exists in the bucket:
async function checkFileExistsInStorage ({storageKey, bucketName}) {
	const bucket = storageClient.bucket(bucketName);
	const file = bucket.file(storageKey);
	const result = await file.exists();
	return result[0];
}

It would appear there is some random lag/latency between the completion of the upload and exists() returning [true].
To test this hypothesis, we've modified our function to retry for up to 5 times:
export async function checkFileExistsInStorage ({storageKey, bucketName}) {
	const bucket = storageClient.bucket(bucketName);
	const file = bucket.file(storageKey);

	let retries = 0;
	let exists = false;

	do {
		console.log(retries);
		const result = await file.exists();
		console.log(result);
		exists = result[0];
		retries ++;
	} while (exists === false && retries < 5);

	return exists;
}

Typically we can see this in the logs:
0
[ true ]

This indicates the file was available on the first check. But maybe 30% of the times we see this instead:
0
[ false ]
1
[ false ]
2
[ true ]

Which confirms the uploaded files are not immediately available after the upload has completed.
This behavior seems extremely weird. One would assume the file should be immediately available after a successful upload response considering Google Cloud Storage is supposedly strongly consistent.
Is this expected behavior?
If yes, how can we handle this behavior?
The Google Cloud docs don't seem to mention any of this.

Thanks for raising this issue @PierBover. On the browser side how are you verifying the upload has completed? What are you getting as the response? Also, what is delay between success on the browser and the query on the server?

Hey @ddelgrosso1
I'm uploading via XHR to be able to monitor the progress. The upload is assumed as completed when the browser fires the loadend event.
https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/loadend_event
After the loadend event is fired, an empty 200 response is received, and the client makes a request to the server.
Before checking if the file exists, the server does some stuff with the database. I'm guessing maybe 500ms or so have passed after the upload has completed.

I think I've found the problem.
There seems to be a small delay between the loadend and the last readystatechange events.
If I wait for the last readystatechange to be triggered in the client, the issue seems to disappear.
I'll close this issue for now and reopen if this doesn't solve it.


Salsanext

The link for Salsanext predtrained model doesn't work? Any advice?

It still works for me. You may try it again or use wget, such as:
wget https://www.ipb.uni-bonn.de/html/projects/LiDAR-MOS/model_salsanext_residual_8.zip

Got it now,
Thanks！
On Sun, Feb 19, 2023 at 3:59 PM Xieyuanli Chen @.***>
wrote:

It still works for me. You may try it again or use wget, such as:
wget
https://www.ipb.uni-bonn.de/html/projects/LiDAR-MOS/model_salsanext_residual_8.zip
—
Reply to this email directly, view it on GitHub
https://github.com/PRBonn/LiDAR-MOS/issues/68#issuecomment-1435918796,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABRJ52QIJV6BWGOWCMCX52LWYHHFNANCNFSM6AAAAAAVADA4TM
.
You are receiving this because you authored the thread.Message ID:
@.***>

Hi, guys
I using my own lidar data(16 lines） and lidar pose too，but the link for Salsanext predtrained model(residual_8 and residual_1) doesn't work。
# range image parameters range_image:   height: 16   width: 1024   fov_up: 15.0   fov_down: -15.0   max_range: 50.0   min_range: 0.5
Almost all raw points is predicted moving points.
why? Looking forward to your reply

Hey @benemer, you may say something about the generalization abilities of point-based and range-image-based methods and advertise your 4D-MOS here ;-)

Got it now, Thanks！
…
On Sun, Feb 19, 2023 at 3:59 PM Xieyuanli Chen @.> wrote: It still works for me. You may try it again or use wget, such as: wget https://www.ipb.uni-bonn.de/html/projects/LiDAR-MOS/model_salsanext_residual_8.zip — Reply to this email directly, view it on GitHub <#68 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRJ52QIJV6BWGOWCMCX52LWYHHFNANCNFSM6AAAAAAVADA4TM . You are receiving this because you authored the thread.Message ID: @.>

Hey, does it work now?

Hey, Does it work now?

Hey ,Does it work now? @ChiyuanFeng

wget https://www.ipb.uni-bonn.de/html/projects/LiDAR-MOS/model_salsanext_residual_8.zip

Yes.I can download and use this model.

I using my own lidar data(16 lines） and lidar pose too，but the link for Salsanext predtrained model(residual_8 and residual_1) doesn't wor

But I have not use it on my own data.

I using my own lidar data(16 lines） and lidar pose too，but the link for Salsanext predtrained model(residual_8 and residual_1) doesn't wor

But I have not use it on my own data.

Thanks. Do you have a plan using it on your own data?  It doesn't work on my own data, I'm looking for why

Hey @11janson,
we experienced that range image-based models do not generalize well if the calibration (sensor location, intensity) changes. Since you also use a different sensor with fewer beams, I highly doubt that this will give good results.
As @Chen-Xieyuanli, we developed 4DMOS for moving object segmentation which is not based on projection but infers moving objects from a  sparse voxelized 4D point cloud (x,y,z,t). This method generalizes quite well on other datasets, even with other sensors. So far I tested it on Velodyne HDL-64E, Ouster OS1 64, and Velodyne HDL32E from various datasets using the same model trained on SemanticKITTI.
Here is the link to the repository. You can also send me your own data and I can give it a first try.
Best
Benedikt

Hey @11janson,
we experienced that range image-based models do not generalize well if the calibration (sensor location, intensity) changes. Since you also use a different sensor with fewer beams, I highly doubt that this will give good results.
As @Chen-Xieyuanli, we developed 4DMOS for moving object segmentation which is not based on projection but infers moving objects from a sparse voxelized 4D point cloud (x,y,z,t). This method generalizes quite well on other datasets, even with other sensors. So far I tested it on Velodyne HDL-64E, Ouster OS1 64, and Velodyne HDL32E from various datasets using the same model trained on SemanticKITTI.
Here is the link to the repository. You can also send me your own data and I can give it a first try.
Best Benedikt

Hey benemer, this is my own point clous data with some moving people.

Hey @11janson,
we experienced that range image-based models do not generalize well if the calibration (sensor location, intensity) changes. Since you also use a different sensor with fewer beams, I highly doubt that this will give good results.
As @Chen-Xieyuanli, we developed 4DMOS for moving object segmentation which is not based on projection but infers moving objects from a sparse voxelized 4D point cloud (x,y,z,t). This method generalizes quite well on other datasets, even with other sensors. So far I tested it on Velodyne HDL-64E, Ouster OS1 64, and Velodyne HDL32E from various datasets using the same model trained on SemanticKITTI.
Here is the link to the repository. You can also send me your own data and I can give it a first try.
Best Benedikt

Thank you @benemer , I have sended my own data to your e-mail<EMAIL_ADDRESS>is that right?

Yes, that's right.
I just checked your data! So 4DMOS is also not working correctly. It finds some moving points but misses most of them and predicts false positives on the ground.
The problem is the way the sensor is mounted. Since you use a vertical 16-beam LiDAR and rotate it, you only observe an object within your (narrow) sensor FOV if the sensor itself is at the correct rotation angle. Remember that 4DMOS needs to see the object in multiple subsequent scans. If your sensor is mounted vertically and the FOV is small, you often only partially observe the moving objects only in a few scans.
Is there a specific reason you mounted the sensor vertically? Mounting it horizontally would at least help in segmenting the moving objects.

Hey benemer,
Thank you for your reply!
This video shows our handheld scanner porduct, may be that is the reason the sensor is mounted vertically, I am not sure.
I do not understand you say only partially observe the moving objects only in a few scans. All the moving objects can be obtained by our handheld scanner.
And do you have any advices working on the our datas ?
------------------ 原始邮件 ------------------
发件人:                                                                                "PRBonn/LiDAR-MOS"                                                                                @.>;
发送时间: 2023年2月27日(星期一) 下午4:20
@.>;
<EMAIL_ADDRESS>主题: Re: [PRBonn/LiDAR-MOS] Salsanext (Issue #68)
Yes, that's right.
I just checked your data! So 4DMOS is also not working correctly. It finds some moving points but misses most of them and predicts false positives on the ground.
The problem is the way the sensor is mounted. Since you use a vertical 16-beam LiDAR and rotate it, you only observe an object within your (narrow) sensor FOV if the sensor itself is at the correct rotation angle. Remember that 4DMOS needs to see the object in multiple subsequent scans. If your sensor is mounted vertically and the FOV is small, you often only partially observe the moving objects only in a few scans.
Is there a specific reason you mounted the sensor vertically? Mounting it horizontally would at least help in segmenting the moving objects.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>
从QQ邮箱发来的超大附件
FJD Trion S1 Intro Video.mp4 (234.37M, 2023年03月29日 16:45 到期)进入下载页面：https://mail.qq.com/cgi-bin/ftnExs_download?k=673036641f3463c2a068287044390b1c4e440306065d58031b0450555b140d0100011b5c540c001e0e545250530e580253055301622539757c7416301050565d166307442b574d415910600d065c561d5b4002645f&t=exs_ftn_download&code=606db993

Okay, I see.

I do not understand you say only partially observe the moving objects only in a few scans. All the moving objects can be obtained by our handheld scanner.

Your handheld scanner uses a LiDAR sensor with 16 beams which should give you around 32k points per 0.1s. Since the sensor itself is rotating by 180 degrees every second, you change the field of view of the sensor and therefore can measure a larger portion of your environment within one second.
What I mean by partially observing is that you might see a moving object in one scan but not in the next one.  This makes it hard for 4DMOS to identify moving objects in a sequence of scans since we assume to see the moving object for multiple, subsequent scans.

A LiDAR sensor with 64 beams maybe better, right?
This should help, especially for having more measurements on moving objects which are not close to the sensor, since point clouds from a 16-beam LiDAR become quite sparse in the distance. However, the main problem I see is the rotation of the sensor on the handheld itself.

And nother question, how about merging subsequent 4(or may be 5) scans in my datas as a scan in your datas? so that I can see a moving object in current 4 scans and in the next 4 scans.
This is hard to say. In 4DMOS we assume that all points in a scan are measured at the same point in time. Strictly speaking, this does not hold for a rotating LiDAR sensor, but the framerate of 10 Hz is quite large and we also compensate for the ego-motion within a single scan revolution. In your case, you would aggregate points from 1s into one virtual timestamp which is in my view not negligible anymore. Also, if there is a moving object in subsequent frames, you would lose this information by merging.

A LiDAR sensor with 64 beams maybe better, right?

This should help, especially for having more measurements on moving objects which are not close to the sensor, since point clouds from a 16-beam LiDAR become quite sparse in the distance. However, the main problem I see is the rotation of the sensor on the handheld itself.

And nother question, how about merging subsequent 4(or may be 5) scans in my datas as a scan in your datas? so that I can see a moving object in current 4 scans and in the next 4 scans.

This is hard to say. In 4DMOS we assume that all points in a scan are measured at the same point in time. Strictly speaking, this does not hold for a rotating LiDAR sensor, but the framerate of 10 Hz is quite large and we also compensate for the ego-motion within a single scan revolution. In your case, you would aggregate points from 1s into one virtual timestamp which is in my view not negligible anymore. Also, if there is a moving object in subsequent frames, you would lose this information by merging.

Thank you Benemer. I have already understood what you said.  A moving object should be appear in some subsequent frames at very short intevals.  So the 4DMOS is not suitable for our own datas., right?
Thanks again.

At least using 4DMOS out of the box does not work. Do you need online MOS? I think in your case it would be easier to loot at some offline map cleaning approaches.

At least using 4DMOS out of the box does not work. Do you need online MOS? I think in your case it would be easier to loot at some offline map cleaning approaches.

Yeah. I am looking at some offline map cleaning approaches such as "removert", and have got some results.
And I am trying another online approaches called "RF-LIO",  maybe it works.

We are currently working on another MOS approach and might release it in a couple of months. We can also try this on your data. I will let you know as soon as it's available!


Emerald Carrot Horse Problem

I'm playing the most current version of FTB Hermitpack.  I was looking through JEI and noticed the emerald carrot from cyclic.  I had a tamed horse with a saddle and diamond armor that I got last week, so I figured I'd try it on her.  She screamed, turned completely invisible, and then turned into a zombie horse.  (In retrospect, I'm guessing the zombie horse part was what the emerald carrot does?)  She still had a shadow on the ground.  I shift+clicked, removed the armor and saddle, and then killed her.  My screen turned all red and stayed that way for a while, before eventually returning to normal.  There were no potion effects nor text to give me a clue as to what happened, though.
Is that what's supposed to happen?

Really weird.
Emerald carrot should swap the horse around between zombie, skeleton and regular, in a rotation like that.  When i have just cyclic installed and no other mods this works for me.
Also i tried it in hermit pack, looks like this works too.
Were you on a server?  Do you have mods installed that added modded horse types.
Does this still happen to you after restarting the game?
I cant replicate this on my end, thats why im asking so many questions

LOL no worries...I wish I had more information, but I haven't tried it again since that happened.  We've stuck to just the straight Hermitpack installation from Curse with no other mods added to the server.  On my end, I did add WAILA and it's associated mods (WAILA, -harvestability, and WAWLA).  As far as I can tell, we have only vanilla horses.  EnderZoo is part of the pack, but it doesn't add horses.  (I didn't even know there was a mod that added horses...I've seen the cow/pig mod, but that's it.)
I'll see if I can tame a horse I'm not attached to like I was to Bullseye.  She was an awesome horse, and I've setup a small memorial to her in my village that I've taken over.
BTW I love your mod...I seem to find some new toy every day to play with.

Thanks, glad you like it.
Yeah, i tried it with cyclic standalone, in single player, in a server, survival, creative, not seeing it happen.  very weird.  Sorry about the horse.  Closing issue for now, if it happens again or something new comes up feel free to comment and reopen


"Out of bounds", "No media Found" and "No image at time" errors

@samplue Sam Pluemacher reports -

I get an error message: "No media found for "Blood and Gold - Picture Lock 2 - 2022-12-12”."
I’ve named media and xml identical and also adjusted all paths in the sh files. The timeline I’ve been testing is fairly large (+90 minutes and tons of markers).

Another test timeline was a compound clip that I opened in its own timeline. This resulted in "No markers found."
If I “Break apart” the compound clip and export the timeline again the tool will give me results, but fails to generate stills with error “No image at time…"

This could be due to #6 & #7.

Here's the corresponding fcpxml that gave me the error message.
BRS_LOCKED6_20210521 - 01 Opening Scene.fcpxmld.zip

Just ran this through the extractor after I rewrote the parser. I don't have the media file so the thumbnail image output stage was skipped.
I haven't looked at the output carefully but @samplue @IAmVigneswaran do you want to check out the CSV and console log and see if you notice anything wrong or missing?
Once I get 0.2.4 released you can give this a go with the media file, if you still have it.
Here's the output CSV:
BRS_LOCKED6_20210521 - 01 Opening Scene.csv
And here's the console log:
Console Log
Using Notion (csv2notion) export profile.
Extracting markers from "/Users/stef/Desktop/FailingTest.fcpxmld/Info.fcpxml"...
NOTICE: Marker "Taxi Establisher" at 01:09:17:01.00 is out of bounds of its parent clip "Property of Netflix" (00:59:50:00.00 - 01:05:27:11.00) and will be omitted.
NOTICE: Marker "FFA_044_0010" at 01:16:05:24.00 is out of bounds of its parent clip "Property of Netflix" (00:59:50:00.00 - 01:05:27:11.00) and will be omitted.
NOTICE: Marker "FFA_044_0020" at 01:16:27:06.00 is out of bounds of its parent clip "Property of Netflix" (00:59:50:00.00 - 01:05:27:11.00) and will be omitted.
NOTICE: Marker "FFA_044_0025" at 01:16:35:09.00 is out of bounds of its parent clip "Property of Netflix" (00:59:50:00.00 - 01:05:27:11.00) and will be omitted.
NOTICE: Marker "Additional Shot Elias" at 01:16:41:06.00 is out of bounds of its parent clip "Property of Netflix" (00:59:50:00.00 - 01:05:27:11.00) and will be omitted.
NOTICE: Marker "FFA_044_0040" at 01:16:56:02.00 is out of bounds of its parent clip "Property of Netflix" (00:59:50:00.00 - 01:05:27:11.00) and will be omitted.
NOTICE: Marker "Lens Flare 01" at 01:18:36:05.00 is out of bounds of its parent clip "Property of Netflix" (00:59:50:00.00 - 01:05:27:11.00) and will be omitted.
NOTICE: Marker "Lens Flare 02" at 01:31:46:16.00 is out of bounds of its parent clip "Property of Netflix" (00:59:50:00.00 - 01:05:27:11.00) and will be omitted.
NOTICE: Marker "Lens Flare 03" at 01:35:27:12.00 is out of bounds of its parent clip "Property of Netflix" (00:59:50:00.00 - 01:05:27:11.00) and will be omitted.
NOTICE: Marker "Clip Extended" at 01:42:07:07.00 is out of bounds of its parent clip "Property of Netflix" (00:59:50:00.00 - 01:05:27:11.00) and will be omitted.
NOTICE: Marker "Nadja bites Berg - Trimmed X frames" at 01:54:39:10.00 is out of bounds of its parent clip "Property of Netflix" (00:59:50:00.00 - 01:05:27:11.00) and will be omitted.
NOTICE: Marker "Lens Flare 04" at 01:59:33:10.00 is out of bounds of its parent clip "Property of Netflix" (00:59:50:00.00 - 01:05:27:11.00) and will be omitted.
NOTICE: Marker "Lens Flare 05" at 02:02:50:00.00 is out of bounds of its parent clip "Property of Netflix" (00:59:50:00.00 - 01:05:27:11.00) and will be omitted.
NOTICE: Marker "Plane dive - Trimmed X frames" at 02:09:51:18.00 is out of bounds of its parent clip "Property of Netflix" (00:59:50:00.00 - 01:05:27:11.00) and will be omitted.
NOTICE: Marker "Lens Flare 06" at 02:12:55:06.00 is out of bounds of its parent clip "Property of Netflix" (00:59:50:00.00 - 01:05:27:11.00) and will be omitted.
NOTICE: Marker "PET_169_0020 - Clip is 87 frames longer" at 02:15:35:08.00 is out of bounds of its parent clip "Property of Netflix" (00:59:50:00.00 - 01:05:27:11.00) and will be omitted.
NOTICE: Marker "PPL_216_0010 - Clip is 389 frames longer" at 02:28:01:16.00 is out of bounds of its parent clip "Property of Netflix" (00:59:50:00.00 - 01:05:27:11.00) and will be omitted.
NOTICE: Marker "Same clip, but trimmed by X frames" at 02:49:01:03.00 is out of bounds of its parent clip "Property of Netflix" (00:59:50:00.00 - 01:05:27:11.00) and will be omitted.
NOTICE: Marker "Kamera im Bild" at 01:01:04:01.00 is out of bounds of its parent clip "1C-3C-1 MOS" (01:00:52:02.00 - 01:00:52:24.00) and will be omitted.
NOTICE: Marker "ILA_002_0020 - Insert helicopter" at 01:01:07:24.00 is out of bounds of its parent clip "2-1-5" (01:01:09:07.00 - 01:01:12:15.00) and will be omitted.
NOTICE: Marker "ILA_002_0030 - Background - view out of helicopter - set extension - retouch background" at 01:01:22:06.00 is out of bounds of its parent clip "2Bpt 2of2-5-4" (01:01:24:16.00 - 01:01:29:07.00) and will be omitted.
NOTICE: Marker "ILA_002_0030 - Background - insert view out of helicopter - set extension - retouch background" at 01:01:23:15.00 is out of bounds of its parent clip "2Bpt 2of2-5-4" (01:01:24:16.00 - 01:01:29:07.00) and will be omitted.
NOTICE: Marker "Take 1" at 01:00:52:10.79 is out of bounds of its parent clip "NT=5B Nachsprecher Welcome to our air base..." (01:01:39:13.09 - 01:01:41:01.26) and will be omitted.
NOTICE: Marker "Take 2" at 01:01:37:03.69 is out of bounds of its parent clip "NT=5B Nachsprecher Welcome to our air base..." (01:01:39:13.09 - 01:01:41:01.26) and will be omitted.
NOTICE: Marker "Take 1" at 01:00:52:08.79 is out of bounds of its parent clip "NT=5B Nachsprecher Welcome to our air base..." (01:01:40:24.27 - 01:01:42:08.13) and will be omitted.
NOTICE: Marker "Take 2" at 01:01:37:01.69 is out of bounds of its parent clip "NT=5B Nachsprecher Welcome to our air base..." (01:01:40:24.27 - 01:01:42:08.13) and will be omitted.
NOTICE: Marker "Take 1" at 01:00:52:13.01 is out of bounds of its parent clip "NT=5B Nachsprecher Welcome to our air base..." (01:01:42:04.00 - 01:01:45:23.00) and will be omitted.
NOTICE: Marker "Take 2" at 01:01:37:05.71 is out of bounds of its parent clip "NT=5B Nachsprecher Welcome to our air base..." (01:01:42:04.00 - 01:01:45:23.00) and will be omitted.
NOTICE: Marker "Take 1" at 01:01:26:08.00 is out of bounds of its parent clip "6-1-2" (01:02:21:07.00 - 01:02:22:17.00) and will be omitted.
NOTICE: Marker "Take 2" at 01:01:34:15.00 is out of bounds of its parent clip "6-1-2" (01:02:21:07.00 - 01:02:22:17.00) and will be omitted.
NOTICE: Marker "Take 3" at 01:01:47:21.00 is out of bounds of its parent clip "6-1-2" (01:02:21:07.00 - 01:02:22:17.00) and will be omitted.
NOTICE: Marker "Take 4" at 01:01:56:02.00 is out of bounds of its parent clip "6-1-2" (01:02:21:07.00 - 01:02:22:17.00) and will be omitted.
NOTICE: Marker "Take 5" at 01:02:09:15.00 is out of bounds of its parent clip "6-1-2" (01:02:21:07.00 - 01:02:22:17.00) and will be omitted.
NOTICE: Marker "Take 6" at 01:02:20:03.00 is out of bounds of its parent clip "6-1-2" (01:02:21:07.00 - 01:02:22:17.00) and will be omitted.
NOTICE: Marker "Take 7" at 01:02:37:18.00 is out of bounds of its parent clip "6-1-2" (01:02:21:07.00 - 01:02:22:17.00) and will be omitted.
NOTICE: Marker "Take 8" at 01:02:45:05.00 is out of bounds of its parent clip "6-1-2" (01:02:21:07.00 - 01:02:22:17.00) and will be omitted.
NOTICE: Marker "Take 9" at 01:02:55:15.00 is out of bounds of its parent clip "6-1-2" (01:02:21:07.00 - 01:02:22:17.00) and will be omitted.
NOTICE: Marker "Take 10" at 01:03:07:15.00 is out of bounds of its parent clip "6-1-2" (01:02:21:07.00 - 01:02:22:17.00) and will be omitted.
NOTICE: Marker "Take 11" at 01:03:15:20.00 is out of bounds of its parent clip "6-1-2" (01:02:21:07.00 - 01:02:22:17.00) and will be omitted.
NOTICE: Marker "Take 12" at 01:03:24:13.00 is out of bounds of its parent clip "6-1-2" (01:02:21:07.00 - 01:02:22:17.00) and will be omitted.
NOTICE: Marker "Take 13" at 01:03:33:02.00 is out of bounds of its parent clip "6-1-2" (01:02:21:07.00 - 01:02:22:17.00) and will be omitted.
NOTICE: Marker "00422 - ILA_006_0030 - add CG Plane in the BG (Set etension needed for baselayout ?)" at 01:02:35:19.00 is out of bounds of its parent clip "6-Y-4 MOS" (01:02:35:22.00 - 01:02:38:12.00) and will be omitted.
NOTICE: Marker "ILA_008_0020 - add CG plane" at 01:02:38:17.00 is out of bounds of its parent clip "8-2-2" (01:02:40:04.00 - 01:02:47:16.00) and will be omitted.
NOTICE: Marker "Take 1" at 01:03:11:09.00 is out of bounds of its parent clip "10-1-3" (01:02:59:13.00 - 01:03:01:08.00) and will be omitted.
NOTICE: Marker "Marker 1" at 01:03:11:13.00 is out of bounds of its parent clip "10-1-3" (01:02:59:13.00 - 01:03:01:08.00) and will be omitted.
NOTICE: Marker "Take 1" at 01:03:11:13.00 is out of bounds of its parent clip "10-1-3" (01:03:02:24.00 - 01:03:04:13.00) and will be omitted.
NOTICE: Marker "Marker 1" at 01:03:11:17.00 is out of bounds of its parent clip "10-1-3" (01:03:02:24.00 - 01:03:04:13.00) and will be omitted.
NOTICE: Marker "Trigger 3" at 01:03:25:08.00 is out of bounds of its parent clip "20-2-1" (01:03:59:02.00 - 01:04:00:17.00) and will be omitted.
NOTICE: Marker "Trigger 2" at 01:03:43:16.01 is out of bounds of its parent clip "20-2-1" (01:03:59:02.00 - 01:04:00:17.00) and will be omitted.
NOTICE: Marker "00037 - ILA_017_ - Add plane wheel" at 01:04:06:17.00 is out of bounds of its parent clip "17-4-2 W" (01:04:24:09.00 - 01:04:26:21.00) and will be omitted.
NOTICE: Marker "Take 1" at 01:04:42:01.00 is out of bounds of its parent clip "10-1-3" (01:04:37:08.00 - 01:04:39:06.00) and will be omitted.
NOTICE: Marker "Marker 1" at 01:04:42:05.00 is out of bounds of its parent clip "10-1-3" (01:04:37:08.00 - 01:04:39:06.00) and will be omitted.
NOTICE: Marker "Take 1" at 01:04:45:03.00 is out of bounds of its parent clip "10-1-3" (01:04:41:24.00 - 01:04:43:01.00) and will be omitted.
NOTICE: Marker "Marker 1" at 01:04:45:07.00 is out of bounds of its parent clip "10-1-3" (01:04:41:24.00 - 01:04:43:01.00) and will be omitted.
NOTICE: Marker "Marker 1" at 01:04:32:23.00 is out of bounds of its parent clip "10-1-3 Cam A" (01:04:43:01.00 - 01:04:52:10.00) and will be omitted.
NOTICE: Marker "Take 1" at 01:04:43:20.00 is out of bounds of its parent clip "10-1-3" (01:04:49:19.00 - 01:04:56:10.00) and will be omitted.
NOTICE: Marker "Marker 1" at 01:04:43:24.00 is out of bounds of its parent clip "10-1-3" (01:04:49:19.00 - 01:04:56:10.00) and will be omitted.
NOTICE: Marker "Take 1" at 01:04:35:14.00 is out of bounds of its parent clip "10-1-3" (01:04:58:03.00 - 01:05:04:14.00) and will be omitted.
NOTICE: Marker "Marker 1" at 01:04:35:18.00 is out of bounds of its parent clip "10-1-3" (01:04:58:03.00 - 01:05:04:14.00) and will be omitted.
NOTICE: Marker "Take 1" at 01:04:34:17.00 is out of bounds of its parent clip "10-1-3" (01:05:06:17.00 - 01:05:15:10.00) and will be omitted.
NOTICE: Marker "Marker 1" at 01:04:34:21.00 is out of bounds of its parent clip "10-1-3" (01:05:06:17.00 - 01:05:15:10.00) and will be omitted.
NOTICE: Marker "Take 1" at 01:04:39:02.00 is out of bounds of its parent clip "10-1-3" (01:05:19:10.00 - 01:05:21:12.00) and will be omitted.
NOTICE: Marker "Marker 1" at 01:04:39:06.00 is out of bounds of its parent clip "10-1-3" (01:05:19:10.00 - 01:05:21:12.00) and will be omitted.
Duplicate marker IDs found which will be uniqued: ["BRS_LOCKED6_20210521 - 01 Opening Scene_01:04:00:17"]
Project start timecode: 00:59:50:00 @ 25 fps.
No media present. Skipping thumbnail generation.
Generating metadata file(s) into "/Users/stef/Desktop/me-cli-output/BRS_LOCKED6_20210521 - 01 Opening Scene 2023-11-21 10-35-24 [Notion (csv2notion)]".
Exporting marker icons...
Creating "done.json" done file at "/Users/stef/Desktop/me-cli-output/BRS_LOCKED6_20210521 - 01 Opening Scene 2023-11-21 10-35-24 [Notion (csv2notion)]".
Done

Just ran this through the extractor after I rewrote the parser. I don't have the media file so the thumbnail image output stage was skipped.

I have sent you the library with media files.

Ok just got the fcpbundle and exported fcpxmld and ran it through the parser.
It's actually getting all the markers and they appear to be correct timecodes and names etc. (63 markers + 1 to-do marker). The CSV has 65 markers - 1 extra, because upon examination of the FCPXML there is a ghost marker that shares the same timecode as another marker, but FCP is only showing one of them.
So any of the "NOTICE: Marker ... is out of bounds" console messages are actually correct and can be ignored.
As far as thumbnail generation and the "No image at time" errors, I have not touched any of the image generation code in a long time. So whatever bugs may have been present a while back will probably still be present. I've been focusing on FCPXML parser reliability and accuracy which needed to come first. The next stage will be looking at these types of image errors.

Ok I exported a temporary copy of the project as a video and there was definitely a bug in MarkersExtractor when project timelines started at a time later than 00:00:00:00. I've now fixed the bug and all thumbnail images are being generated as expected.

Will close this for now.


Does not work with web player

Any ideas about what changes can be made to get this to work with web player builds?

+1
seems like a WebsocketSharp compatibility problem.

Some ideas here  https://github.com/sta/websocket-sharp/issues/52 I had a go at hacking it all up so it at least compiles for web player but it fails in the web player with no error so I'm also stuck.  gl.


glib: Consider changing GStrPtr

@jf2048 wrote on Matrix

CC @pbor
(ridiculous that you can't copy&paste text properly from matrix)

I agree on the rename to GStringPtr... do we really need GStrPtr as well right now? I do not const char ** is very common in API.
I am also ok with dropping AsRef an Deref.
I am not sure I understand the suggestion about StrVBorrowed

do we really need GStrPtr as well right now? I do not const char ** is very common in API.

I don't think that's needed, yes.

https://github.com/gtk-rs/gtk-rs-core/pull/950 is the simple rename
Thinking more about it I do not think we should change from as_str to to_str because the guidelines say that it is for expensive conversions, which is true for CStr since it needs to check utf-8, but it is not true here.

After some more coffee I realized that the conversion requires walking the string to find the length, so it is expensive.
I added a second commit the the PR to remove all the AsRef implementations and use to_gstr explicitely

There are a couple that return const char **:

https://docs.gtk.org/gio/method.AppInfo.get_supported_types.html
https://docs.gtk.org/glib/method.Variant.get_strv.html

For those there could just be a separate borrowed type.


AKPanner crash when numberOfChannels = 1

I'm doing some CPU optimisation and as part of that I have set AKSettings.numberOfChannels = 1 - audioKit promptly crashed at next run when starting up the panners.
yes I understand having a panner in a mono signal path is completely meaningless.
I have traced the crash to the process(AUAudioFrameCount frameCount, AUAudioFrameCount bufferOffset) function in AKPannerDSPKernel.hpp The problem seems to be the sp_panst_compute() function gets passed uninitialised pointers as the for(;;) loop didn't go through the expected number or loops.
  | float *tmpin[2];
  | float *tmpout[2];
  | for (int channel = 0; channel < channels; ++channel) {
  |     float *in  = (float *)inBufferListPtr->mBuffers[channel].mData  + frameOffset;
  |     float *out = (float *)outBufferListPtr->mBuffers[channel].mData + frameOffset;
  |     if (channel < 2) {
  |          tmpin[channel] = in;
  |          tmpout[channel] = out;
  |      }
  | }
  | sp_panst_compute(sp, panst, tmpin[0], tmpin[1], tmpout[0], tmpout[1]);

tmpin[1] & tmpout[1] never get set if channels = 1
I suppose you could probably just  set outbuffer[0] = inbuffer[0] as if the panner was bypassed when its operating in a mono signal path but I'll leave the solution up to those who know audiokit better

okay, I think your solution makes sense


Passing params outside the Url

Hey,
We are using the react-router in our application and in some cases we are trying to pass non string params, like objects and functions between scenes. Is there a recommended way of doing this with react-router? is this something that you guys recommend in general?

You can attach state to the location descriptor.
<Link to={{ pathname: '/other-place', state={ /* pass extra data here */ }}>Other Place</Link>

You would then access that data through the location prop passed to the component rendered by a matched <Route>. One caveat is that a location that is navigated to directly will not have any state associated with it, so you will need to take that into account.
<Route path='/other-place' component={OtherPlace} />

const OtherPlace = ({ location }) => (
  const { state } = location
  return null
)


bfloat16 arrays are incompatible with pickle

import pickle
import jax.numpy as jnp
pickle.dumps(jnp.bfloat16(0))

---------------------------------------------------------------------------
PicklingError                             Traceback (most recent call last)
<ipython-input-13-9fa7e9015eb7> in <module>()
      1 import pickle
      2 import jax.numpy as jnp
----> 3 pickle.dumps(jnp.bfloat16(0))

PicklingError: Can't pickle <class 'bfloat16'>: attribute lookup bfloat16 on builtins failed

I think this is due to the fact that picle the bfloat16 custom type does not correctly define its full class name. Compare these::
from jax._src.lib import xla_client
print(type(xla_client.bfloat16(0.0)))
# <class 'bfloat16'>

print(type(jnp.array(0)))
# <class 'jaxlib.xla_extension.DeviceArray'>

print(type(np.float16(0))
# <class 'numpy.float16'>

I think for this to work correctly, we'd need the tp_name attribute here to correspond to its actual import location within JAX, i.e. jaxlib.xla_client.bfloat16.
That's not straightforward, though, because this object is shared with tensorflow and exported in a different path within that library.

Indeed the tensorflow dtype causes this issue as well:
import tensorflow as tf
import pickle
import numpy as np
pickle.dumps(np.ones(10, dtype=tf.bfloat16.as_numpy_dtype))

---------------------------------------------------------------------------
PicklingError                             Traceback (most recent call last)
<ipython-input-24-272a81341603> in <module>()
      2 import pickle
      3 import numpy as np
----> 4 pickle.dumps(np.ones(10, dtype=tf.bfloat16.as_numpy_dtype))

PicklingError: Can't pickle <class 'bfloat16'>: attribute lookup bfloat16 on builtins failed

I think once this path issue is resolved in pickle, numpy's pickle serialization/deserialization should work. For example:
import jax.numpy as jnp
x = jnp.arange(4, dtype='bfloat16')
constructor, cargs, state = x.__reduce__()
x2 = constructor(*cargs)
x2.__setstate__(state)
print(repr(x2))
# array([0, 1, 2, 3], dtype=bfloat16)

Here's a hack that can be used to work around this:
import builtins
import pickle
import jax.numpy as jnp

# Hack: this is the module reported by this object.
builtins.bfloat16 = jnp.dtype('bfloat16').type

s = pickle.dumps(jnp.arange(10, dtype='bfloat16'))
y = pickle.loads(s)
print(repr(y))
# array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=bfloat16)

One thing we could do to fix this (though it's kind of terrible) is to set this attribute of builtins when jax is imported.

A marginally simpler hack that we've been using is
builtins.bfloat16 = jnp.dtype('bfloat16').type
instead of
builtins.bfloat16 = xla_client.bfloat16,
as it avoids the private from._src.lib import.

Setting builtins.bfloat16 = jnp.bfloat16 does not seem to work, even though xla_client.bfloat16 == jnp.bfloat16 evaluates to True.


How to load images in an async manner ?

This library needs images from the very beginning to work properly, which make the task of retrieving data from the outside without blocking the user interface somewhat difficult.
I'm sure because of THIS LINE
assert(rowOrColoumnCount != 0, "Row or Coloumn Count must be more than 0")
assert(itemCount >= rowOrColoumnCount, "Image count can not be more than row Count")

am I wrong? if so, how can I load the collage asynchronously?

you can fetch async images and set them in:
func collageView(_ collageView: CollageView, configure itemView: CollageItemView, at index: Int)
should look like:
func collageView(_ collageView: CollageView, configure itemView: CollageItemView, at index: Int) {

    //
    // MAGIC is in this code block
    // You can prepare your item here also,
    // You can fetch Images from Remote here!,
    // Customize UI for item, and etc..

    imageFromServerURL(urlString: "imageUrlString") { (image, error) in

        itemView.image = image
    }

    itemView.layer.borderWidth = 3

    if index == shownImagesCount - 1 {

        addBlackViewAndLabel(to: itemView)
    }
}

You can use Kingfisher library to fetch async images.

you can fetch async images and set them in:
But firstly you should know just the count of total images.

That's escentially the problem.
I don't know how many images I will be loading until the request is done. However, I think it wouldn't be wise to stop the execution of the application waiting the server's response to know the image count.

OK, I fixed this issue and pushed to master thanks for your feedback!! Appreciate it

Many thanks. I test the new changes tonight.

Please Check the new release at https://github.com/ahmetkgunay/CollageView/releases/tag/1.0.4
And also I updated the Cocoapods as the new version is 1.04


网络实现

我看原论文中模型里所有的最大池化层大小都是33，在这个实现里都是22的，请问有没有试验过两种哪个性能好呢？

I don't get your point. Could you please clarify it?

I don't get your point. Could you please clarify it?

About the network implementation.
In the paper, the kernel size of all the Maxpooling layer are 33, and in your implementation, they are 22. Have you ever tested two of them, which has better performance?

I don't get your point. Could you please clarify it?

About the network implementation.
In the paper, the kernel size of all the Maxpooling layer are 3x3, and in your implementation, they are 2x2. Have you ever tested two of them, which has better performance?

Thank you for pointing it out.
I haven't tested them with the kernel_size=(3,3) yet that you could change the parameters to check it. I just set it to 2x2 because I think it is a "default" configuration for images.

Thank you for pointing it out.
I haven't tested them with the kernel_size=(3,3) yet that you could change the parameters to check it. I just set it to 2x2 because I think it is a "default" configuration for images.

Thank you， your implementation has helped me a lot.

Glad to hear that. ;)


Make per-user global_configs use per-user home_dir

write /home/username/.composer.json as user instead of /usr/local/bin/composer.json

Re: https://github.com/escapestudios-cookbooks/composer/issues/47

Thanks for your contribution @john-coleman-w!
Kind regards,
David

@djoos the version needs to be bumped and this change published to the supermarket.  Please!

Sorry for that guys - I remember having issues getting it 'thor release'd and forgot about it afterwards once fixed...
I've just published 2.2.0 on the supermarket, thanks for the heads up!


[Portal] Throw error if the children is undefined or null

[ ] The issue is present in the latest release.
[ ] I have searched the issues of this repository and believe that this is not a duplicate.

Current Behavior 😯

Expected Behavior 🤔

Steps to Reproduce 🕹

Steps:

Context 🔦

Your Environment 🌎

Tech
Version

Material-UI
v4.?.?

React

Browser

TypeScript

etc.

Thanks for the feedback, you are right, the implementation doesn't match the prop-types. We don't flag the children prop as required.
What do you think of this diff?
diff --git a/packages/material-ui/src/Portal/Portal.js b/packages/material-ui/src/Portal/Portal.js
index 88dceb978..44a062c30 100644
--- a/packages/material-ui/src/Portal/Portal.js
+++ b/packages/material-ui/src/Portal/Portal.js
@@ -20,7 +20,7 @@ const useEnhancedEffect = typeof window !== 'undefined' ? React.useLayoutEffect
 const Portal = React.forwardRef(function Portal(props, ref) {
   const { children, container, disablePortal = false, onRendered } = props;
   const [mountNode, setMountNode] = React.useState(null);
-  const handleRef = useForkRef(children.ref, ref);
+  const handleRef = useForkRef(children ? children.ref : null, ref);

   useEnhancedEffect(() => {
     if (!disablePortal) {
@@ -46,7 +46,14 @@ const Portal = React.forwardRef(function Portal(props, ref) {
   }, [onRendered, mountNode, disablePortal]);

   if (disablePortal) {
-    React.Children.only(children);
+    if (!children) {
+      return null;
+    }
+
+    if (process.env.NODE_ENV !== 'production') {
+      React.Children.only(children);
+    }
+
     return React.cloneElement(children, {
       ref: handleRef,
     });

Do you want to submit a pull request? :)

@oliviertassinari It seems good, I can submit a PR later.

We don't flag the children prop as required, and yet, if we don't provide it, it breaks.

This was changed when switching to automatically generating them from typescript. I think the typescript types were too relaxed.
What is the actual use case for undefined children?

@eps1lon I agree, I think that we should encourage people not to render the Portal when the children are empty. So basically,
{branch ? <Portal><Foo /></Portal> : null}

Rather than:
<Portal>{branch ? <Foo /> : null}</Portal>

I think that we should rather focus on supporting any type of children:
<Portal>
  <Foo />
  <Bar />
</Portal>


Task step is failing with no stdout link, but top of page shows end of stdout log

Linux_samsung_a02 openpay_benchmarks__scroll_perf is consistently failing.
https://ci.chromium.org/ui/p/flutter/builders/staging/Linux_samsung_a02 openpay_benchmarks__scroll_perf/307/overview
The end of the log output is in the red box at the top, but the failing step only has execution details, not a link to stdout to see the full log.

This is WAI: b/231460791


Relax Uniqueness Constraint for Write-only Variables

Description
I am hoping to be able to define cells like this, which is a common pattern in existing notebooks.
cell1:
fig, ax = plt.subplots()
ax.plot(...)

cell2:
fig, ax = plt.subplots()
ax.plot(...)

This is a pretty common pattern and the multiple definitions are not a problem since the variables are never read before they are assigned to in any cell.
Suggested solution
Allow multiple definitions of a variable if it is never read before being assigned to in any cell.
Alternative
One alternative is
_fig, _ax = plt.subplots()
_ax.plot(1, 2)

another is (e.g.):
def _():
  fig, ax = plt.subplots()
  ax.plot(1, 2)
_()

But neither is very pretty imo.
Additional context
No response

I was thinking about this earlier when I did for i in range(x) for what felt like the millionth time.
More suggestions:
Using a with
e.g.
with mo.local_scope():
    ax, fig

Use type hints (which can be statically parsed)
ax # mo.local

catching the error and prompting to rename with the _ prefix (that would be my preference, var defaults are just muscle memory)

I was thinking about this earlier when I did for i in range(x) for what felt like the millionth time.

Yes, exactly! Currently, you can not have for i in range(n): in two cells, even if ì` is never read outside the for loop scope. Perhaps I should have mentioned this as well since it is very common!

I need to think through if there are any edge cases or gotchas. Are there any you can think of?

I don't know any. I have been meaning to use Marimo more seriously but I keep running into this when converting existing notebooks. It seems like it would be possible to relax the constraints a bit and still be safe

More on this;
Maybe have a conversion wizard for Jupyter? Prompt the user for variables that are used like ax or fig and automatically replace during the conversion process.
marimo could even suggest variables based on def frequency.
Doing this in conjunction with the "rename variable" option on error would go a long way.
I can only speak for myself, but I do like the fact that I know that all variables are global and not previously defined. I think relaxation starts to introduce the same issues - but I absolutely get the frustration. I think the biggest sticking for me is the actual conversion though, and aside from that- marimo forces me to write better code

I don't know any. I have been meaning to use Marimo more seriously but I keep running into this when converting existing notebooks. It seems like it would be possible to relax the constraints a bit and still be safe

Yes, that makes sense. I thought about it and don't foresee any correctness issues either. Relaxation does make multiple definition errors more confusing when they do happen, since the relaxed rule is more complicated to explain.
In the following sequence,

Create a cell defining x.
Read x in another cell.
Create another cell defining x.

Cell 3 would not be run, and instead would display the error:
MultipleDefinitionError: Variables defined in multiple cells cannot be read before assignment within a cell.
This error message would link to the cells from steps 1 and 2.
Whereas in the following sequence:

Create a cell defining x.
Create another cell defining x.
Read x in another cell.

The cell in step 3 cannot be run. I suppose it would show the same error, and also link to the cells from steps 1 and 2.
It may be confusing for a read to trigger a MultipleDefinitionError. marimo already has a learning curve, so I am hesitant to do things that make that learning curve steeper.

Maybe have a conversion wizard for Jupyter? ...

If it's just for marimo convert — marimo could uniquify variable names that are never read before being defined in a cell. It wouldn't be pretty, but it would at least make the conversion work in more cases without user intervention. So perhaps this could be a compromise we settle on for now?

marimo already has a learning curve, so I am hesitant to do things that make that learning curve steeper.

Fair enough, this is important but the advantages are also pretty big in this case since a quite common pattern would "just work" instead of raising errors

MultipleDefinitionError: Variables defined in multiple cells cannot be read before assignment within a cell.

The error could be more verbose than this.

[!CAUTION]
InconsistentUsageError: Variables may be declared in one cell and referenced in multiple; or declared in multiple cells and used as a referenced nowhere. Cells x, y, z declare variable, and Cells a, b, c reference variable. Remove all references to variable or reduce declarations to one.

But because execution is dynamic I think this is not super straightforward. Back to Akshay's example:

Create a cell defining x.
Create another cell defining x.
Read x in another cell.

Receive error
4 . Delete the definition in cell 2
Cell 1 gets promoted to definition

Change cell 3 to definition

Cell 1 gets demoted.

I think there's a lot more bookkeeping than currently takes place. All repeated cell-level variables will have to be kept in memory and then swapped out from hidden context to global context.

Assigned to myself — to either relax the constraint or improve marimo convert.

MultipleDefinitionError: Variables defined in multiple cells cannot be read before assignment within a cell.

The error could be more verbose than this.
Caution
InconsistentUsageError: Variables may be declared in one cell and referenced in multiple; or declared in multiple cells and used as a referenced nowhere. Cells x, y, z declare variable, and Cells a, b, c reference variable. Remove all references to variable or reduce declarations to one.

Okay, yes, that's much better than my concise error :)

All repeated cell-level variables will have to be kept in memory and then swapped out from hidden context to global context.

@dmadisetti Or we could re-run ancestors as needed. If a cell that has been requested for execution has a reference whose definition isn't in globals, run the cell that defines it (as well as its descendants) before running the requested cell. Thoughts?

Certainly a solution, but could also lead to a number of unintentional reruns.
Another suggestion: app = App(local=("ax", "fig", "i")) might be the easiest from an engineering perspective. Also a couple of user benefits:

outside of runtime context, which is important for initial graph construction
explicit behavior. You might trigger one of the modes and not even know it otherwise until you try to use a 3rd cell.
no runtime penalty
can be tuned for jupyter conversion (cells may utilize "axis" or "figure" instead of "ax" or "fig", and they might be a keyword worth marking as local)

As for implementation:

propagate information to visitor.is_local and just use the current local mechanism
Raise custom error if one of the special locals is a ref
Ensure exporter saves the information
(maybe) UI means of editing the values

I think runtime implementation could put notebooks in an ambiguous state, which is not a great user experience. This is a compromise with the downside of slightly hiding the information that some values will treated as private (but partially mitigated with a custom error).

How would one specify the locals from inside a notebook? Or is the idea that locals would only be specified by editing the notebook file?
I think with some careful bookkeeping, one could minimize unintentional re-runs. Your solution of keeping variables in memory would also work, and would have no unintentional reruns, at the cost of memory consumption.

The variables panel is a great place to mark variables as being private:

Maybe a checkbox? I was only mentioning saving inside the notebook for persistence (same way layout file reference is saved behind the scenes, or show_code state on the cell level).
The book-keeping approach would have to account for additional kernels (thinking the strict execute pr)- but still could work

I started experimenting with relaxing the constraint. There are more edge cases than I originally anticipated. As well as some usability gotchas: imagine redefining a variable (without knowing it) in a very expensive cell. Only to find that you can't read it because it was defined elsewhere. That would make me very sad.

Mock up example:

I think runtime implementation could put notebooks in an ambiguous state, which is not a great user experience.

Yeah, you're totally right. That's what I concluded too after experimentation

... This is a compromise with the downside of slightly hiding the information that some values will treated as private (but partially mitigated with a custom error).

The hiding could be confusing ... but otherwise I agree with your pros, especially with the explicitness.

Mock up example:

Nice mockup! If we wanted to use the existing mechanism for local variables, "change to private read-only variable" could just prepend an underscore to the variable name.
Another solution that would be explict, without having to use underscores, would be to define a custom type annotation that marimo understood. Something like:
fig: "local"
ax: "local"

fig, ax = plt.subplots()
...

That's something I would be open to. But maybe underscores is fine for now ...

Closing as not planned for now — in favor of the marimo convert improvement — given how this can get the runtime in an ambiguous state. Still happy to revist relaxing the constraint later, if anyone has further thoughts, and still happy to riff on other ways we can improve this experience

Thanks for seriously considering it! Maybe it's possible to improve this somehow....


Write documentation for vertex removal, interceptorNode and Lattice node

vertex removal in setPositions
Lattice node and associated contextuals
InterceptorNode

New optimize options
Aggregator
Clean geometry


Label studio on external terminal

I am trying to run label-studio on my external server with the following command:
label-studio start --host "http://<IP_ADDRESS>" --port 9090
I don't see any errors:
=> Database and media directory: /home/maaz/.local/share/label-studio
=> Static URL is set to: /static/
=> Hostname correctly is set to: http://<IP_ADDRESS>
=> Database and media directory: /home/maaz/.local/share/label-studio
=> Static URL is set to: /static/
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /pypi/label-studio/json HTTP/1.1" 200 56156
Performing system checks...

[2023-03-24 17:04:19,634] [django::register_actions_from_dir::97] [INFO] No module named 'data_manager.actions.__pycache_'
[2023-03-24 17:04:19,634] [django::register_actions_from_dir::97] [INFO] No module named 'data_manager.actions.__pycache_'
System check identified no issues (1 silenced).
March 24, 2023 - 17:04:19
Django version 3.2.16, using settings 'label_studio.core.settings.label_studio'
Starting development server at http://<IP_ADDRESS>:9090/
Quit the server with CONTROL-C.

But I am unable to access label-studio using the public IP of my server
I created a conda env with python 3.8 and had to install psycopg2-binary to make label-studio to run but then I am unable to reach the GUI externally.

I was able to do it using NGROK


bluetooth: host: conn: Add const attribute to addr in bt_le_set_auto_…

…conn
This commit adds missed const modifier for addr pointer for
bt_le_set_auto_conn function
Signed-off-by: Radoslaw Koppel<EMAIL_ADDRESS>

Codecov Report

Merging #10337 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #10337   +/-   ##
=======================================
  Coverage   53.09%   53.09%
=======================================
  Files         214      214
  Lines       26229    26229
  Branches     5652     5652
=======================================
  Hits        13925    13925
  Misses      10029    10029
  Partials     2275     2275

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 61ca8c1...cdbfa85. Read the comment docs.

Btw, the commit message is rather long, we use to have a check for 80 columns max so I wonder why CI/gitlint didn't pick this up.

@Vudentz The limit is 72. And the commit message was updated. Currently longest line has 64 characters.


Update Detox to use react native 0.67

Description
Update Detox to use RN67. This fixes issue #3197

For features/enhancements:

[X] I have added/updated the relevant references in the documentation files.

Green builds for ios/android 64/66 test jobs -> https://jenkins-oss.wixpress.com/job/multi-detox-pr/4219/
Green builds for ios 67 test job -> https://jenkins-oss.wixpress.com/view/all/job/detox-ios-67-branch/11/
Green build for android 67 test job -> https://jenkins-oss.wixpress.com/view/all/job/detox-android-67-x-branch/8/

@jonathanmos Touch-base; I see the current gaps are:

Reinstating the CI-script for the iOS example app (as in my previous comment)
Making the necessary adjustments in the buildkite scripts, in order to effectively upgrade the CI jobs to RN .67

I removed detox-instruments phase from the iOS script, although I tried to make it work (I followed the instruments doc, didn't helped). I can investigate its issues separately, regardless of this PR.

@jonathanmos Touch-base; I see the current gaps are:
Reinstating the CI-script for the iOS example app (as in my previous comment)
Making the necessary adjustments in the buildkite scripts, in order to effectively upgrade the CI jobs to RN .67

Done. Is there anything left that blocks us from merging this PR? @d4vidi @jonathanmos
Jenkins fails on the Android jobs with unrelated EACCES error (in other PRs also), however Buildkite pass so I don't care much..

!!!!!!


LKAS fault: Restart car to engage | Toyota hybrid 2022

Describe the bug
I installed camera 3 with the open pilot master branch it's connecting to the car
but after a minute I got this message "LKAS fault: Restart car to engage"
(after the comma 3 is turned on, I can't change the lane assistance settings on the car) any help!
image000000 (2)
Which car does this affect?
Toyota sienna hybrid 2022
Provide a route where the issue occurs
11/12/2024
openpilot version
0.9.7
Additional info
No response

Did you take the necessary steps to recover the CAN security key? That's the error you'll see if you haven't done it. That's why the car isn't on the officially supported plug-and-play list.
Please join the comma Discord server and look at the pinned messages in #toyota-security for everything you need to know.
Side note: it's a bug that we don't communicate this well, you're supposed to be seeing a more informative message, but I haven't had a chance to investigate why the event isn't doing what I intended. You still need to do the same thing, we just suck at communicating it right now.
https://github.com/commaai/openpilot/blob/96f6e186c4d07f5dff65b71122699bc8138af49b/selfdrive/selfdrived/events.py#L385-L389

The original end user can fix themselves with a bit of RTFM, but I'm leaving this issue open to track the Event bug.

@jyoung8607 can you migrate issues? should we move to opendbc or close?

I don't have the mojo to migrate anymore, but we shouldn't need to. The events are handled in controls or car_specific here on the openpilot side. You could retitle it to "SecOC missing key event is broken".

Need to do something like this:
https://github.com/commaai/openpilot/blob/87bc80d35145686903ccb499cd4fe4092054a45e/selfdrive/selfdrived/selfdrived.py#L131-L132


Add the README file to the binary archives in releases

First of all thank you for this tool. I just love it.
I am exploring and testing lotz of plotting in the terminal little tools and this one is really nice and performant.
I'm using the https://aur.archlinux.org/packages/lowcharts-bin package.
Very nice that you have remembered to include extensive help for all the subcommands with the --help flag.
One thing I'm missing is more examples. The ones you have are on the README and are all related to PC stuff.
Maybe in the future if you have lotz of them you could even have them in an extra Examples.md ?
Like, some example test data with "echo" "printf" "seq" "awk" "paste" "shuf" "random etc.
seq 5 | awk '{print 2*$1, $1*$1}'

seq 2.324 0.3 5.234  | shuf | paste - -

(Inspiration from https://www.gnu.org/software/datamash/manual/html_node/index.html and https://github.com/dkogan/feedgnuplot)
I am always forgetting the lowcharts examples  and coming back to the Internet to read the README.
It would be nice to have it offline on my PC if  you ship both those simple markdown files in the tar.gz released.
Just like, https://github.com/juan-leon/lowcharts/issues/4 asked for the license.
Thanks in advance.

Thanks for the suggestion.
I like the "Examples.md" thing, as the README has some github things (badges, refs to png pictures) not suitable for a local markdown file.
I need to think on some good examples and write them (then it is trivial to write that file into release files)


Failed to sync client configuration on the server

This is just a warning, but getting this fairly frequently. Not sure if this is expected, but wanted to call it out.
[Warn  - 10:20:26 PM] OmniSharp.Extensions.LanguageServer.Server.Configuration.DidChangeConfigurationProvider: No ConfigurationItems have been defined, configuration won't surface any configuration from the client! |
[Warn  - 10:20:29 PM] Microsoft.AspNetCore.Razor.LanguageServer.DefaultRazorConfigurationService: Failed to sync client configuration on the server: System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at OmniSharp.Extensions.JsonRpc.ResponseRouter.ResponseRouterReturnsImpl.Returning[TResponse](CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Razor.LanguageServer.DefaultRazorConfigurationService.GetLatestOptionsAsync(CancellationToken cancellationToken) in /_/src/Razor/src/Microsoft.AspNetCore.Razor.LanguageServer/DefaultRazorConfigurationService.cs:line 57 |

This is happening to me in Visual Studio 2022 GA as well as Preview.


I believe this sample should include an expression to check if the location field is present

Team, based on experience this policy is used many times as the base for geo-restricting policies. However, not all resource types require the location field. Resources that don't use the location field will not send them, causing this policy to block all of those resources. Could we add to the policy an extra condition to first check if the location exists before enforcing its value? This should be fine if users do not specify location on resources that do require it as it would return a validation error anyway.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: cc7205c9-e142-3019-39cb-6cde0ffe8baf
Version Independent ID: 471f9317-cb57-7565-c593-3c30413fdf99
Content: Pattern: Field properties in a policy definition - Azure Policy
Content Source: articles/governance/policy/samples/pattern-fields.md
Service: azure-policy
GitHub Login: @DCtheGeek
Microsoft Alias: dacoulte

Thanks for your comment. The feedback has been shared with the content owner for further review. Thanks for your patience.

@milope Thanks for the question! I'm validating with the team that the resource types that don't require a location field are covered by the inclusion of 'global' as a check. As a note, this is a pattern explanation, not a sample. For samples, we recommend looking at the Azure Policy built-ins repo and the Community Policy Repo.
@MicrosoftDocs/azure-cxp-triage #in-progress

@milope Thanks again for asking on this. The pattern is good as-is and only evaluates resource types that have a location by setting "mode": "Indexed". Please make sure that if you are doing a location based policy, that you are setting mode to indexed and it's not set to all.  This is likely the cause of the issue you are seeing. For more information on mode, see Policy definition - mode.  Thanks again!
@MicrosoftDocs/azure-cxp-triage #please-close


No errors from create_server()

Bug
Creating a server in GDScript returns errors, creating it with Rust bindings does not.
Setup
Open a Godot server on port 5000 (just run the snipped below once).
Test
Run the following snippet. It should error because the port is already used by the first server. Prints "worked".
        let mut peer = NetworkedMultiplayerENet::new();
        match peer.create_server(5000, 200, 0, 0) {
            Ok(_) => {
                godot_print!("worked");
            }
            Err(e) =>  {
                godot_print!("failed");
            },
        }

The same code in GDScript returns 20.
var peer = NetworkedMultiplayerENet.new()
var error = peer.create_server(5000, 200)
assert(error == ERR_CANT_CREATE)    # true because error is 20

Could you explain what is the expected behavior here?

@toasteater Sorry for the low effort issue, I was in a hurry. Added more description.

Thanks. This might be a bug in the generator.

Sorry, but I seem to be unable to reproduce this issue using the provided code:

The generated code is fine. It did capture the return value from Godot correctly, which is GODOT_OK.
The GDScript code also returns 0 on my machine.

I have noticed that NetworkedMultiplayerENet is a reference counted type. Maybe the port is closed when the last reference is dropped? That could explain why subsequent attempts to create servers on that port could succeed.
In any case, if the behavior is consistent with GDScript, then this would not be considered a bug in the bindings. If the code still doesn't behave as expected on your machine, providing more information about your environment and a minimal reproduction project could help, if possible.
Otherwise, if you think that this is not the correct behavior, please consider reporting the issue to Godot's issue tracker.
Thanks!

It's reference counted, I had not thought about that!
That was it, the server automatically closes after my ready() function.
If I keep the variable around, it returns an error when I create a second one.
Thank you so much and sorry for the issue.


Blocking issue with ASP.NET Core integration

          > > Seems to be working now 🤷

Glad to know. Did you make any changes from your side to make it work? Would you mind sharing? Thanks!

I had to remove the FunctionContext. Which I think is related to this locking thing here.

Looks like there's some sort of blocking issue

Anyway FunctionContext was only used to access the logger instance, which I am now injecting into the constructor.
Originally posted by @thushanperera95 in https://github.com/Azure/azure-functions-dotnet-worker/issues/1387#issuecomment-1570525309

@thushanperera95 would you be able to share details on the code you've changed so we can have a repro to take a closer look at the issue?
Thank you!

I made a couple of changes:

Removed the FunctionContext from the HTTP trigger function.
Removed the addition of user secrets in the app builder.

using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Hosting;

var builder = new HostBuilder()
    .ConfigureFunctionsWorkerDefaults()
    .ConfigureAppConfiguration(configuration => { configuration.AddUserSecrets<Program>(true); })
    .Build();

builder.Run();

However it seems to all work now even with those changes added back in!
We haven't made any changes to the ARM template that's deployed to Azure, so not sure what happened.

@thushanperera95 would you be able to share a small repro so we can investigate the original issue.


Problem resolving mixed generic/non-generic overload taking a closure as an argument

x as int
System.Threading.Tasks.Task.Factory.StartNew({ x = 42 })

Expected: This should compile.
Observed: Overload resolution fails for argument of type error.
This appears to be happening because ProcessMethodBodies.OnBlockExpression defers processing of closures being passed to an overloaded method in which any of the overloads are generic methods, so that generic type references can be inferred properly later on.  But in this case, there are no generic type references; we're trying to reach the non-generic overload that takes a single argument of type Action, and there is no code in the compiler that handles deferred processing in that case.

Fixed this as part of the async/await merge.


Use a smarter method with docker-compose to determine when the MySQL database is ready and remove hehe from log message

Right now run_jobs.py just waits 30 seconds for the MySQL container to finish initializing, but really it only needs to do that on the first startup. Otherwise, it takes three seconds. MyLA has a better strategy for this. See Matt's comment below.
"I'm doing this in the MyLA start script with the nc command which should be installed by default.
echo "Waiting for DB"
while ! nc -z ${MYSQL_HOST} ${MYSQL_PORT}; do
  sleep 1 # wait 1 second before check again
done

I believe the we could have a shell script before this python or could just do this similar check in python with nc. (I didn't test this code below)
while not os.system(f"nc -z {os.getenv("MYSQL_HOST")} {os.getenv("MYSQL_PORT")}):
    time.sleep (1)
```"

_Originally posted by @jonespm in https://github.com/tl-its-umich-edu/course-inventory/pull/62_

This is pretty excellent.  I made a FIXME comment in my own copy of the code that this needs to be done.  I'm glad others are thinking the same way.
90% of the time I run the app with docker-compose, the DB service is running and ready for use, but the program waits the 30 seconds anyway.

This was a problem I had identified before submitting the run_jobs.py PR, but @jonespm said we could resolve if it after we merged. Thus, PR #81.


ADR for the proposal of terms to be used inside the API

Checklist

[ ] You added tests to cover the change
[ ] mvn clean install build and test completes
[ ] You changed or added to the documentation

Maybe a little late but I'd like to discuss another term.
License.
We will be using the Adoptium api and will be producing 2 different binaries. These will be the same Vendor, same Implementation, possibly a different Distribution (TBD) but otherwise almost identical except for the License. Ideally we won't have to change much when we inevitably fork the api, so we'd appreciate having our use cases considered in the overall design.
Thanks.

^^ @johnoliver


Work Package wp-eg.csv

I would like to work on this please!

Result of Phase 1
wp-eg-results.csv

I completed this work package

Results of Phase I:
wp-eg-results.csv

Results for phase 2
wp-eg-results.csv

Results for phase 2
wp-eg-results.csv


Add low-level documentation.

This PR begins to add some low-level documentation, discussing implementation details. It also pulls some code in from @glyph's automat to render out the FSMs, and puts them into the documentation.

This is cool!
But, is there anything I could do in order to get you to depend on (or vendor in) automat, rather than pulling out bits of its code?

At this point, very little of automat's code is being used here. automat has a fundamentally very different view of its state machines than hyper-h2 does, which necessitated a fairly substantial rewrite of almost all of the code in automat's visualizer.

Fair enough! Thanks for the shout-out and license acknowledgement, at any rate :)

That's genuinely my pleasure: y'all did the hard bit of actually learning how to graphviz.


Avoid adding parent dir in find

Fixes #394

@jorisvandenbossche thanks for the concrete test in https://github.com/dask/s3fs/issues/394#issue-740077082

@jorisvandenbossche thanks for the concrete test in https://github.com/dask/s3fs/issues/394#issue-740077082

The title of the PR is suggesting that it is fixing https://github.com/dask/s3fs/issues/393, but that's another issue?

The title of the PR is suggesting that it is fixing https://github.com/dask/s3fs/issues/393, but that's another issue?

Sorry No, this does indeed fix 394 - the move was failing due to the extra entry in find. Maybe it also fixes 393, I need to check.

Sorry No, this does indeed fix 394 - the move was failing due to the extra entry in find. Maybe it also fixes 393, I need to check.