Allow csv reader builder to do schema inference even when reading csv from stdin

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I would like to stream 100+GB of SARS-CoV-2 sequence data into .parquet with zstd compression (works really well on these sequences)
I would like to do this without having to hard-code the schema, for example through a CLI like https://github.com/domoritz/csv2parquet/blob/main/src/main.rs
However, that CLI requires me to provide a file and does not allow me to read from stdin. Why? Because the reader builder requires input to be seekable which stdin is not. I
Describe the solution you'd like
It'd be good if the reader builder could be more flexible and infer schema from the first say 100 lines that can still be kept in memory.
Describe alternatives you've considered
I could add a schema option to the CLI tool, but that's annoying and unnecessary because I just want a very simple schema: str/str. I could also do schema inference myself but again this is quite difficult and would be good to be provided from arrow-rs directly.

Is this related to https://github.com/apache/arrow-rs/issues/189?

I think that this may be achieved with the current code with a generic struct over R: Read that wraps a reader: R that

implements Read and buffers the result.
implements Seek to positions smaller than the buffer (and panics otherwise)
when the read happens over the buffered result, it reads first from the buffer up to the length of the buffered result, and then from the inner reader

@jorgecarleitao I agree that implementing a custom seekable Reader would be necessary to solve this without dropping the seekable requirement. But I'm not sure your suggestion is the way forward since we shouldn't read all of stdin into memory. Memory is even scarcer than storage.
If you have 1TB uncompressed that compresses to 1GB. I can uncompress the 1GB to storage, then read it in with a normal file reader (which is seekable). Problem: I need 1TB of space and write-time.
What doesn't work is read the 1TB into memory. No way.
Alternative: read 1GB into memory, infer schema, then stream.
Your suggestions seems to read it all into memory, doesn't it? When would you be allowed to drop early parts of the buffer?
Couldn't one drop all back-buffer once one has gone beyond max_read_records in builder = builder.infer_schema(opts.max_read_records);? Once the max has been read, there's no need for seeking anymore. Seeking shouldn't happen anymore so one can drop whatever is behind you.

Sorry that I did not express myself very well. I meant something like this:
use std::io::Read;

struct ReaderA<R: Read> {
    pub reader: R,
    position: Option<u64>,
    pub buffer: Vec<u8>,
}

impl<R: Read> Read for ReaderA<R> {
    #[inline]
    fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
        let length = buf.len();
        if let Some(position) = &mut self.position {
            let start = *position as usize;
            // we have seeked to somewhere in our buffer. Read from buffer first
            if start + length < self.buffer.len() {
                // our buffer fills `buf` => memcopy all
                buf.copy_from_slice(&self.buffer[start..start + length]);
                *position += length;
                Ok(length)
            } else if start <= self.buffer.len() {
                // edge case where the read covers `self.buffer` and `self.reader`:
                // read from both accordingly
                let buffer_remaining = self.buffer.len() - start;
                (&mut buf[..buffer_remaining])
                    .copy_from_slice(&self.buffer[start..start + buffer_remaining]);

                // read the remaining from the reader
                let read = self.reader.read(&mut buf[buffer_remaining..])?;
                *position += (buffer_remaining + read) as u64;

                if *position > self.buffer.len() {
                    // release memory
                    std::mem::swap(&mut self.buffer, &mut vec![]);
                }

                Ok(buffer_remaining + read)
            } else {
                // we are past `self.buffer`,
                self.reader.read(buf)
            }
        } else {
            // no seek was done so far, read to the buffer
            let start = self.buffer.len();
            self.buffer.extend(std::iter::repeat(0).take(length));
            let read = self.reader.read(&mut self.buffer[start..start + length])?;
            (&mut buf).copy_from_slice(&self.buffer[start..start + length]);
            Ok(read)
        }
    }
}

impl<R: Read> std::io::Seek for ReaderA<R> {
    #[inline]
    fn seek(&mut self, pos: std::io::SeekFrom) -> std::io::Result<u64> {
        match pos {
            std::io::SeekFrom::Start(position) => {
                self.position = position;
                Ok(self.position)
            }
            std::io::SeekFrom::End(_) => panic!("This reader does not support seeking from end"),
            std::io::SeekFrom::Current(position) => {
                self.position += position;
                Ok(self.position)
            }
        }
    }
}

fn main() {}

I.e. when we do not have seek and need to use data from a Read more than once, we can store it in a buffer and use it whenever it is requested (via seek to the back + read). This idiom uses as little memory as we need to store in a non-seek environment, namely the data that needs to be used twice (once for inference, one for reading).
I hope this is a bit understandable.

Could we use the buffered reader from the standard library if it had a way to set the buffer size?

Thanks @jorgecarleitao. I'm a beginner at Rust so please bear with me.  Will all of the input be read into this buffer?
Does your code allow to release the buffer once schema inference has been completed? That's how it would be best: read into memory until schema inference is done, then release. Schema inference options would determine how much is read into memory. This could be set by the user to say first 100 lines to keep buffer memory usage down.

I'm a beginner at Rust so please bear with me.

No worries, we are all learning :)

How do you decide how big the buffer will get?

I think we can't: a single CSV line can be megabytes long. Part of parsing CSV is deciding how much bytes are needed per "cell" (row, column).
The code above allocates for as long as the inference requires them (so, driven by the number of lines for inference + size of first lines in the specific CSV).

Does your code allow to release the buffer once schema inference has been completed? That's how it would be best: read into memory until schema inference is done, then release.

Not sure I understood: isn't it the idea that we want to both infer and parse those lines into arrow, but we can't seek to repeat the operation? If that is the case, I think that we can't release the buffer once the inference is done: we can only release it once the data has been infered and parsed into columns. The code above works under this hypothesis: once we move the position past the buffer size, we release (see // release memory comment on the code).

This issue seems good to close as current infer schema logic in arrow-csv takes in a Read generic param and also reads only a specified amount of rows for inference:
https://github.com/apache/arrow-rs/blob/31b5724332666d68fe94a5b3572a13a51022ea81/arrow-csv/src/reader/mod.rs#L268-L278
cc @tustvold


Update README.markdown

Scan submitted to Checkmarx

Checkmarx scan completed
Full Scan Details
Violation Summary

Severity
Count

High
77

Details

Lines
Severity
Category
File
Link

18
High
Command_Injection
website/pictures/conflictview.php
Checkmarx

18
High
Command_Injection
website/pictures/high_quality.php
Checkmarx

9
High
Command_Injection
website/passcheck.php
Checkmarx

8
High
Command_Injection
website/users/check_pass.php
Checkmarx

3
High
File_Inclusion
website/admin/index.php
Checkmarx

27 29
High
File_Manipulation
website/pictures/upload.php
Checkmarx

5
High
Reflected_XSS_All_Clients
website/error.php
Checkmarx

35
High
Reflected_XSS_All_Clients
website/pictures/conflict.php
Checkmarx

49
High
Reflected_XSS_All_Clients
website/users/register.php
Checkmarx

22 35
High
Reflected_XSS_All_Clients
website/comments/add_comment.php
Checkmarx

2 3 4 5
High
Reflected_XSS_All_Clients
website/test.php
Checkmarx

12
High
Reflected_XSS_All_Clients
website/piccheck.php
Checkmarx

9 36
High
Reflected_XSS_All_Clients
website/passcheck.php
Checkmarx

144
High
Reflected_XSS_All_Clients
website/include/html_functions.php
Checkmarx

44
High
Reflected_XSS_All_Clients
website/comments/preview_comment.php
Checkmarx

19 22
High
Reflected_XSS_All_Clients
website/pictures/search.php
Checkmarx

16
High
Reflected_XSS_All_Clients
website/comments/delete_preview_comment.php
Checkmarx

19
High
Reflected_XSS_All_Clients
website/submitname.php
Checkmarx

16 39
High
Reflected_XSS_All_Clients
website/users/login.php
Checkmarx

83
High
Reflected_XSS_All_Clients
website/pictures/upload.php
Checkmarx

3
High
Remote_File_Inclusion
website/admin/index.php
Checkmarx

18
High
SQL_Injection
website/pictures/high_quality.php
Checkmarx

16
High
SQL_Injection
website/pictures/view.php
Checkmarx

18
High
SQL_Injection
website/users/register.php
Checkmarx

18 24
High
SQL_Injection
website/comments/preview_comment.php
Checkmarx

18
High
SQL_Injection
website/pictures/conflictview.php
Checkmarx

11
High
SQL_Injection
website/users/login.php
Checkmarx

15
High
SQL_Injection
website/pictures/search.php
Checkmarx

24 30
High
SQL_Injection
website/users/view.php
Checkmarx

16
High
SQL_Injection
website/pictures/view_flymake.php
Checkmarx

14
High
SQL_Injection
website/guestbook.php
Checkmarx

10
High
SQL_Injection
website/admin/login.php
Checkmarx

102
High
SQL_Injection
website/include/admins.php
Checkmarx

16
High
SQL_Injection
website/comments/add_comment.php
Checkmarx

31 71 104
High
SQL_Injection
website/cart/action.php
Checkmarx

29 30 39 47
High
SQL_Injection
website/pictures/upload.php
Checkmarx

14
High
SQL_Injection
website/comments/delete_preview_comment.php
Checkmarx

16 25
High
SQL_Injection
website/pictures/conflict.php
Checkmarx

37 97
High
Second_Order_SQL_Injection
website/include/cart.php
Checkmarx

60
High
Second_Order_SQL_Injection
website/include/comments.php
Checkmarx

87
High
Second_Order_SQL_Injection
website/include/admins.php
Checkmarx

22 102 157
High
Second_Order_SQL_Injection
website/include/users.php
Checkmarx

117 197
High
Second_Order_SQL_Injection
website/include/pictures.php
Checkmarx

28
High
Stored_XSS
website/include/comments.php
Checkmarx

97 136
High
Stored_XSS
website/include/cart.php
Checkmarx

22 132
High
Stored_XSS
website/include/users.php
Checkmarx

19 39 60 79 98 117 132 221
High
Stored_XSS
website/include/pictures.php
Checkmarx

15
High
Stored_XSS
website/include/guestbook.php
Checkmarx


DOC Document image lazy-loading

Issue https://github.com/silverstripe/silverstripe-assets/issues/452

@maxime-rainville have split it off and made a new changelog issue https://github.com/silverstripe/silverstripe-assets/issues/456

Closed in favor of https://github.com/silverstripe/silverstripe-framework/pull/10029


[BUG][Preview Version] Browse->Extensions plugin page drop-down refresh failed

Steps to reproduce
After pulling down and releasing, nothing happens.
Expected behavior
The dropdown should refresh the extension plugin page
Actual behavior
https://user-images.githubusercontent.com/40033067/171992590-ebe07a1e-3524-40ff-9021-1df5d917a1ea.mp4
Crash logs
N/A
Tachiyomi version
r456
Android version
Android 11
Device
Xiaomi 9
Other details
No response
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open issue.
[X] I have written a short but informative title.
[X] If this is an issue with an extension, I should be opening an issue in the extensions repository.
[X] I have tried the troubleshooting guide.
[X] I have updated the app to version 1.8.3.
[X] I have updated all installed extensions.
[X] I will fill out all of the requested information in this form.

Its most likely refreshing really fast. Are you having issues seeing the avalible extensions?

The available extensions appear to load normally.
But this dropdown refresh is a bit odd, should add logging to see if the refresh is being triggered.

Thats on main Tachi then. I dont mess with the extension loading so UX like that should be handeled by Main Tachi

Confirm that this behavior comes from upstream.
Tachiyomi Preview r4430 does not have this behavior, but r4445 has it.
Maybe it is by design. to prevent multiple refreshes in a short period of time?


[91porn] fix extraction

Please follow the guide below

You will be asked some questions, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your pull request (like that [x])
Use Preview tab to see how your pull request will actually look like

Before submitting a pull request make sure you have:

[x] Searched the bugtracker for similar pull requests
[x] Read adding new extractor tutorial
[x] Read youtube-dl coding conventions and adjusted the code to meet them
[x] Covered the code with tests (note that PRs without tests will be REJECTED)
[x] Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

[ ] I am the original author of this code and I am willing to release it under Unlicense
[x] I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

[x] Bug fix
[x] Improvement
[ ] New extractor
[ ] New feature

Description of your pull request and other information
Explanation of your pull request in arbitrary form goes here. Please make sure the description explains the purpose and effect of your pull request and is worded well enough to be understood. Provide as much context and examples as possible.

Can this be fixed?? THX

See https://github.com/ytdl-org/youtube-dl/pull/30845

TY!

when will this be merged??


Add Google and Office365 / MS Graph API wrappers

Links
https://github.com/PlaceOS/google
https://github.com/PlaceOS/office365
Checklist

[x] - Shard is at least 30 days old
[x] - Shard has CI implemented.
[x] - Shard has daily/weekly periodic builds (ideally with Crystal latest and nightly).

@stakach Now just need to do one for Exchange EWS :laughing:.


No module named 'discord.bot'

After installing everything from the requirements.txt via pip install and running the main.py I get the following error:

Hi!
Can you show us what the command pip list return in the terminal?

OUTPUT.TXT
This is the result of pip3 list

Your py-cord version is outdated, it must be >= 2.0 as it's showed in this image

You can try sudo pip install pycord --upgrade as it's used in this link.
Let me know if you got any progress!

Never mind, the problem was with my installation of Python. Fixed that and works fine now!


Instructions for installing on Windows 7?

when starting a steam shortcut created with GloSC, a small cmd box opens for a split second, then nothing.   There are no controllers installed in 'devices and printers'.  Not exactly sure where to go from here.  Is Windows 7 compatibility still an issue?

Currently, broken.

any ETA?

Can you try out the 1.4.2 pre-release?
You minimally need windows powershell v5

Can you try this https://github.com/Alia5/GloSC/releases/tag/<IP_ADDRESS> ?
In addition to this https://github.com/Alia5/GloSC/issues/42#issuecomment-373951907
Also the necessary preps for Win7 from ViGEm:
https://github.com/ViGEm/ViGEmBus

will do this in around a month, a bit after this didn't work, I went to trying to use a ps3 controller, this worked, but potentially has been throwing bluescreens, I isolated everything around it and want to keep it this way for another 3 weeks to see if this was the cause, then after completely reverting with another week to make sure I got everything.
Will gladly do this the moment the problem is dealt with.

Works perfectly fine on Windows 7 if you manually install ViGEmBus.
Who in their right mind thought installing device drivers over the internet with Powershell was a good idea?
Here is what I did instead (no need for powershell at all):

Go here: https://downloads.vigem.org/stable/ (This is the official site for stable binaries, linked here: https://github.com/nefarius/ViGEm )
I grabbed: ViGEmBus_signed_Win7-10_x86_x64_v<IP_ADDRESS>.zip
Install the correct version. I did it via: Device Manager -> Action -> Add legacy hardware -> Have Disk...
Fire Up GloSC and enjoy

I do have the tools to create the classic "next-next-finish" setup package but they unfortunately don't integrate well into the build chain and I'd rather sacrifice end-user comfort than my automation. devcon also does the job and you are allowed to redistribute and bundle it with your solution.


Mitigate OOM errors on Android

We are seeing OOM crashes on the oldest and smallest (DRAM) devices.
If we OOM in the database or upload code, we should simply do nothing. We
may succeed on a subsequent attempt, but in any event we do not want to
crash the application because telemetry failed to upload.

My apologies for the enormous whitespace diffs; I ran the formatter on this code to make it comply with the project standard, and, well, it really didn't comply before, apparently.

@larvacea please help in getting this merged to the repo post which we can use the updated SDK to resolve the crash issue.

@maxgolov @sid-dahiya @mkoscumb please help in reviewing the PR and getting the it merged to master.


Satisfactory not Showing Player Count

Bug Report
System Information
Linux Ubuntu 20.04.3 LTS
<IP_ADDRESS> Build Date | 227/10/2021 16:08 Mainline
I confirm:

[x] that I have searched for an existing bug report for this issue.
[x] that I am using the latest available version of AMP.
[x] that my operating system is up-to-date.

Symptoms

What are you trying to do?
Try to see the Player Amount of a Satisfactory Server in AMP Panel
What are you expecting to happen?
That AMP shows me how many Users are on my Server
What is actually happening? ('Nothing' is not an acceptable answer!)
AMP Just Shows 0/20.

Reproduction

Create new Satisfactory Server Instance
Setup the Server
Join the Server
Player Amount in AMP Panel gets not Updates

Not a bug - the Satisfactory server does not properly and reliably log player join/leaves at this time - so AMP can't measure the player count.
We have raised the issue with the developers. As soon as the game has proper logging for player events we can implement this.
Closing for now as no action is possible at this time.


⚠ Samsung has degraded performance

In ac2cc82, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2269 ms

Resolved: Samsung performance has improved in 88d882d.


[TUBEMQ-206] Delete residual files after executed unit tests

There are some residual files after executed unit tests by 'tubemq-server' module, create temporary files to avoid this problem.
Jira link with: https://issues.apache.org/jira/browse/TUBEMQ-206

Nice contribution. LGTM


[MM-48079] Fix race condition related to registry loading vs. the welcome screen popup

Summary
On some system, the registry has a tendency to slow a little slowly and may not have its data be fully available until after we've displayed the welcome screen. This can cause the uncloseable popup to be stuck there on systems with only GPO configured servers.
This PR ensures that the app on Windows waits for the registry data to be populated before trying to show the welcome screen.
Ticket Link
https://mattermost.atlassian.net/browse/MM-48079
Closes https://github.com/mattermost/desktop/issues/2335
Fixed a potential race condition where the welcome screen would get stuck on Windows systems with only GPO-configured servers

Building app in separate branch.

Successfully building:
https://circleci.com/gh/mattermost/desktop/25379
https://circleci.com/gh/mattermost/desktop/25378
https://circleci.com/gh/mattermost/desktop/25377

Artifact links:
https://output.circle-artifacts.com/output/job/e0292105-dff0-4cab-bea7-3553ee55d17d/artifacts/0/tmp/artifacts/mattermost-desktop-5.3.0-develop.1-mac-m1.dmg
https://output.circle-artifacts.com/output/job/e0292105-dff0-4cab-bea7-3553ee55d17d/artifacts/0/tmp/artifacts/mattermost-desktop-5.3.0-develop.1-mac-x64.dmg
https://output.circle-artifacts.com/output/job/b0903d25-c9bd-4752-a723-a53be6366cf3/artifacts/0/tmp/artifacts/mattermost-desktop-5.3.0-develop.1-linux-x64.tar.gz
https://output.circle-artifacts.com/output/job/04a31dbd-38c1-47d7-8970-7d0a09b7703b/artifacts/0/tmp/artifacts/mattermost-desktop-5.3.0-develop.1-win-ia32.zip
https://output.circle-artifacts.com/output/job/04a31dbd-38c1-47d7-8970-7d0a09b7703b/artifacts/0/tmp/artifacts/mattermost-desktop-5.3.0-develop.1-win-x64.zip

Cherry pick is scheduled.


Fix mutually exclusive validation for zero as unanswered

What is the context of this PR?
This adds a fix to the mutually exclusive validation when one of the answer values is zero. Checks to make sure that answer value is not in the list of empty-answer-values. Previously '0' was evaluated as false hence total-answer was less than zero and raising a validation error.
How to review
New test scenario added to verify that the the fix works
Checklist

[ ] New static content marked up for translation
[ ] Newly defined schema content included in eq-translations repo

 Bug: Mutually exclusive validation treating zero as unanswered


OCPP Server becomes unresponsive after a while on v0.16.0

Description:
Hi @lorenzodonini :) We use your library at production @daheimladen
We have been having issues with stations connected over WiFi, where at times the stations either would not get or receive ocpp messages from the server, and we would continue to get no response from the stations unless until we restart the station of the csms server itself.
I realised that chaning the defaultMessageTimeout from 30 seconds down to 8 seconds solved our issue.
So doing something like this,
const DEFAULT_MESSAGE_TIMEOUT_SEC time.Duration = time.Second * 8

func setupChargingStationManagementSystem() ocpp16.CentralSystem {
	dispatcher := ocppj.NewDefaultServerDispatcher(ocppj.NewFIFOQueueMap(0))
	dispatcher.SetTimeout(DEFAULT_MESSAGE_TIMEOUT_SEC)
	s := ws.NewServer()
	endpoint := ocppj.NewServer(s, dispatcher, nil, core.Profile, localauth.Profile, remotetrigger.Profile, reservation.Profile, firmware.Profile, smartcharging.Profile)
	return ocpp16.NewCentralSystem(endpoint, s)
}

While this solved most of our troubles, and this worked perfectly in a beta environment with a handful of users, we ran into issues where the status of the stations failed to update, and all station -> server messages (like OnAuthorize, StatusNotification) failed to work just after 24 hours after promoting the changes to production with about 2000 stations connected with us.
Is there something wrong with my approach above or is the decision of 8 second timeout is to blame. I would really appreciate if you can give some feedback :)
Thanks

Hey,
the initialization snippet looks good to me, although 8s seems a bit low.
The timeout you are setting simply makes sure an outgoing request is canceled, if no response is received from the charging station within those 8 seconds. This allows you to overcome the issue where charging stations don't reply quickly enough (e.g. because of network issues), but it also means that message is lost. If the charging station eventually replies, your CSMS will never process that response.
The only case, where automatic retransmission is currently supported, is if the websocket layer recognizes that the other endpoint disconnected. This is typically achieved with the ping-pong mechanism. The ocpp-j layer currently has no other way of knowing, whether the other endpoint is having network issues, which is why the timeout is meant to be a last resort.
It's very hard to just guess the cause of the issue, since you're apparently dealing with intermittent network connections.
Some information that would help:

Are you using websocket ping-pong in the first place? If so, what's the ping-pong timeout?
Are you manually retransmitting messages, when they're canceled due to the timeout?
Can you provide a log dump of both affected endpoints (the part where the issue arises)? If you have debug logs enabled on the CSMS, that should also provide more information about the message flow

Hi @lorenzodonini, I switched back to the previous release where we did have some issues with the stations with network issues, but we didn't face a complete downtime for all charging stations. But with the new release of v0.16.0, even the 30 second timeout is giving us this trouble, at this point my belief is the issue is not with the timeout but rather us switching from v0.16.0 from v0.15.0 because we switched to the custom dispatcher the same time we upgraded to the new release.
1.> No we're not using websocket ping-pong, we just rely on the heartbeats to keep the connection alive.
2.> No we do not retransmit the messages.
3.> I do have the logs but unfortunately the logs are not on the debug level, so they are not of much use except the fact that the quantity of logs drastically reduces over a time period of 24 hrs after which it seems like it continues to run but doesn't accept any new requests.
I have built on top of your examples for the charge-point and I tried connecting to the production server when I started seeing this issue, pasting the logs of the charge-point-emulator below: (the server here is using the 30second default)
tsavanand@Utsavs-MacBook-Pro charge-point-simulator % ./charge-point-simulator
INFO[2022-08-21T16:41:55+05:30] connected to central system at ws://csms.daheimladen.com
INFO[2022-08-21T16:42:25+05:30] <nil> ocpp message<PHONE_NUMBER>): GenericError - request timed out, no response received from server
INFO[0040] sending status notification
INFO[0050] sending status notification
ERRO[2022-08-21T16:42:55+05:30] disconnected from serverread tcp <IP_ADDRESS>:53483-><IP_ADDRESS>:80: i/o timeout
INFO[0060] sending status notification
ERRO[2022-08-21T16:43:00+05:30] disconnected from serverwebsocket: close 1008 (policy violation): a connection with this ID already exists
INFO[0070] sending status notification
ERRO[2022-08-21T16:43:06+05:30] disconnected from serverwebsocket: close 1008 (policy violation): a connection with this ID already exists
ERRO[2022-08-21T16:43:11+05:30] disconnected from serverwebsocket: close 1008 (policy violation): a connection with this ID already exists
INFO[0080] sending status notification
ERRO[2022-08-21T16:43:17+05:30] disconnected from serverwebsocket: close 1008 (policy violation): a connection with this ID already exists
ERRO[2022-08-21T16:43:22+05:30] disconnected from serverwebsocket: close 1008 (policy violation): a connection with this ID already exists
INFO[0090] sending status notification
INFO[0100] sending status notification
INFO[0110] sending status notification
INFO[0120] sending status notification
INFO[0130] sending status notification
INFO[2022-08-21T16:44:08+05:30] ocpp message<PHONE_NUMBER>): GenericError - request timed out, no response received from server

I'll switch our csms service back to v0.15.0, I believe this should fix our problems until we can find out and resolve the issue as it was the last working release for us. If I find anything of interest, I would share it with you here.
I also want to know if you have any plans for opening up your repository to Github Sponsors, as I use this project in production, and being an open source contributor in the past myself for some cloud native projects, I know for a fact that maintaining open source projects is hard, companies benefit from it with no intent of giving back (atleast most of the times). So I would love to pitch in personally as a Github Sponsor for this project ❤

INFO[2022-08-21T16:42:25+05:30] <nil> ocpp message<PHONE_NUMBER>): GenericError - request timed out, no response received from server

if I understand this right, the charge point sent a request to the CSMS, which never replied, triggering the timeout? Can you verify, whether the CSMS ever received the request?
Btw to enable debug logs you can do this:
ws.SetLogger(logger)
ocppj.SetLogger(logger)
// the logger needs to be set to debug level

Any logger that adheres to the interface defined in logging/log.go will do.
If you could provide a detailed message flow between the two endpoints with debug logs enabled, I may be able to identify where the issue is arising at least.
I will definitely have another look at the changes introduced in v0.16 and double-check if something broke. I hope v0.15 serves you well in the meantime.
I'm very happy to know you're using this in production. Yes, maintaning OSS can be tough and I do plan on opening the repo up to Github Sponsors. Before I do so, I just wanted to polish a few things (i.e. publish a documentation site, make sure nothing major is broken). Stay tuned.

Issue #133 looks similar. Also is there a relation with the issue #141 ?

The issue described in #141 is completely separate and not linked to this at all.
#133 might indeed be related though. I'll look into this further and hopefully find a root cause soon.

Hi @lorenzodonini Upon reading the logs from here and conducting some experiments with the library and my dummy ocpp client, I noticed that we get this error when the charging station client abruptly crashes and disconnects from the server:
ERRO[2022-10-01T01:09:46+05:30] disconnected from serverwebsocket: close 1008 (policy violation): a connection with this ID already exists
Could this be the reason for the issue we're facing? I see that this check was in v0.15.0 as well and is not new in v0.16.0, but could be that for some other change in v0.16.0 we're hitting this policy violation more easily.
In #133 the author of the issue also says "after getted BootNotification request more than 10 times from chargepoint ,ths ocpp server is stucked". I'm carefully reviewing the changes in the code myself, but I would really appreciate your thoughts on this as well :))

@utsavanand2 The error you're seeing occurs when a charging station connects to the server, while the server already holds an open connection with the same charging station. The error occurs on websocket level btw.
Normally, this shouldn't happen, but we decided to handle this edge case, since duplicate IDs can lead to other issues instead.
If there's network issues and the websocket close message never gets through to the other side, but for some reason the new connect message does, then:

the connection gets accepted by the server
the server kills the connection immediately afterwards, leading to disconnected from serverwebsocket: close 1008 (policy violation): a connection with this ID already exists

I don't remember why it was decided to send a 1008 error instead of denying a websocket upgrade in the first place, but there was probably a reason. I'm looking through documentation and RFCs to get wiser.

@lorenzodonini Yes I saw the check on the websocket file where a connection to the same ID was ressponded with the aforementioned code. But again I'm a bit puzzled because this check was there in v0.15.0 and around 3000 connected stations are working fine with it.
In the last months, I've tried to rollout v0.16.0 about 2 to 3 times and after 24-48 hrs the stations would not respond to RFID cards, or RemoteStart and Stops.
The last time the ocpp server became unresponsive with v0.16.0, I ran the ocpp-go client library against my production server and the logs I mentioned above is what I received.

Hi @lorenzodonini :)) Good afternoon, I'll be publishing the beta version of the csms server to production, which means we will again go from v0.15.0 to v0.16.0, this time with debug logs enabled. The motive is to get some debug logs which will help us find the issue to the problem.
I've also forked your repo and added some error logs that might help if we encounter errors like last time (for e.g., disconnected from serverwebsocket: close 1008 (policy violation): a connection with this ID already exists)
https://github.com/utsavanand2/ocpp-go/commit/8b82ac15fa7c5563c942b112ddf3b16e18bc0aca
I've decided to do this promotion on Friday evening and monitor it through the weekend.
Is there any other section of the code or file which you would like me to pay attention to and add more debug or error level logs which might aid you as well?

@lorenzodonini I ran some tests yesterday by keeping the library at v0.16.0 but replacing the dispatcher file with the version used in v0.15.0 which doesn't use the context timeout, and it worked in production :)

@utsavanand2 Thanks for collecting some debug logs. Feel free to upload them to any cloud and either send me a link via email or paste it here (maybe obfuscate sensitive data, if any).
I didn't do an extensive load testing, but scaling a sample setup to 100 clients seems to work just fine in my tests. I expect there is some race condition or incorrect state handling that leads to the issue you're experiencing.
Narrowing it down to the new context timeouts definitely helps. I'll have a deeper look asap and hope to have a fix soon.

Hi @lorenzodonini thanks a lot! Can you please drop me an email at<EMAIL_ADDRESS>I would love to discuss this more in detail, and drop a link to the logs via email :)

@lorenzodonini I can be wrong but I was wondering if we should handle the case for when the context is cancelled in waitForTimeout https://github.com/lorenzodonini/ocpp-go/blob/d4afed542168676368053156ef60740a0fd204bc/ocppj/dispatcher.go#L575 and delete the pending request as we do it for the case when we get an elapsed timeout?


Camunda BPM 7.12 compatibility

Tried to figure out mostly minimal changes required for compatibility with the recent releases.
Tests pass.
graphiql is disable, because it seems to need additional work, but I don't need it myself.
mvn-clean-test.log

 All committers have signed the CLA.


Adjust the operator kubebuilder mcad roles for quota trees PR

@asm582 has a PR 475 that fixes the roles for quota management, but it was pointed out by @sutaakar that the roles also needed to be adjusted in the CodeFlare Operator here:
Just a note - roles need to be also adjusted in operator kubebuilder - https://github.com/project-codeflare/codeflare-operator/blob/9c269d0881a0a37166c120963d747b543413df4e/controllers/mcad_controller.go#L106 (and regenerate config files).

Will create a PR that addresses this.

Created PR 240 to address this

Closed PR 240 an have re-done to catch all the latest CRD changes in PR 283


crash on running

LanternExtractor neriaka

EQ Client: RoF2
Unhandled Exception: System.ArgumentException: Parameter is not valid.
at System.Drawing.Bitmap..ctor(Stream stream)
at LanternExtractor.Infrastructure.ImageWriter.WriteImage(Stream bytes, String filePath, String fileName, ShaderType type)
at LanternExtractor.EQ.Pfs.PfsArchive.WriteImage(Int32 index, ShaderType type, String folderName)
at LanternExtractor.EQ.Pfs.PfsArchive.WriteAllFiles(Dictionary`2 textureTypes, String folderName, Boolean onlyTextures)
at LanternExtractor.LanternExtractor.ExtractZoneFile(String shortName)
at LanternExtractor.LanternExtractor.ExtractZone(String shortName)
at LanternExtractor.LanternExtractor.Main(String[] args)

log.zip

Thanks for the report. Are you able to extract any zone or is this error thrown every time?

Sorry, I didn't see this reply. Data did indeed extract, obj files and such, I didn't try opening them just yet (maya isnt' installed). This error happens on every zone I picked (random classic zones like neriaka, ecommons, freportw, etc). I can't fully tell if it was a false alarm or failure since I don't have the product working to compare. if nobody else reports though not a huge deal for now

Are you using a newer install of EQ? Also getting the issue on files from Live a live EQ install. I do not have any previous versions to test on yet. When I turn wld off it works. The error is happening on var image = new Bitmap(bytes);
From MSDN: Exceptions
ArgumentException
stream does not contain image data or is null.
-or-
stream contains a PNG image file with a single dimension greater than 65,535 pixels.
I'm guessing the format is different in newer versions.

@danwilkins I encountered the same issue and have done some experimenting.
ROF2 and P99 clients are both throwing errors when trying to create bitmaps of certain streams. The streams that throw the error all have the same header (68, 68, 83, 32, 124, 0, 0, 0, 7, 16, 10, 0) which I believe from some quick googling might be DDS (DirectDraw Surface) files. The images successfully written are 66, 77, ... headers (BM...) which I believe is the header for varying bitmap files.
Of note as mentioned above, it only has the errors when ExtractWld is flagged as true. When set to false in settings there are no errors.

@MickWPM Hey Mick. Yes, this is a problem on my end. I wrote this extractor for the old EQ zone versions. I'll add experimental extraction support for the post Luclin/live client in the next release.

@MickWPM Hey Mick. Yes, this is a problem on my end. I wrote this extractor for the old EQ zone versions. I'll add experimental extraction support for the post Luclin/live client in the next release.

@danwilkins Oops my bad, forgot the P99 client runs on a much later build (dont have original CDs anymore). Thanks for the reply.
Do you mind if I flick you a brief email using your github email?

Had time to take a look at this. Modern zones use the DDS format for textures. I added support for the extracting of some newer zones and their textures and models in the development/0.2.0 branch. I will continue to work on this. At some point, they switched from S3D archives to EQG and I haven't taken a look into what's involved in extracting this file type.
@MickWPM Please, feel free to send me an e-mail.

@xackery Try again if you would with neriaka using the development/0.2.0 branch. This is an older style zone format so the only difference should be that they use .dds textures.


Free educational licenses

I applying for a free licenses for IntelliJ IDEA Ultimate

There are several files in the request, along with file.zip. It must not be there. Please remove file.zip and make pull request again.  I have to close this request without merging. Thank you for your understanding.


disable docstring test -- numpy-2.0 failures

This PR addresses #768 as a temporary fix for #767

Can we maybe try to fix it rather than disabling failing tests?

@martinfleis
Yes, this was done in haste yesterday evening. I'm on the fence as to:

update the types in docstrings (X.Y vs. np.float64(X.Y)); or
make actual changes in the code base

Very worth noting that none of the actual functional tests fail, only the docstring examples.

make actual changes in the code base

Not worth it imho, just creates unnecessary copies of data.

need to try skipping doctest only in the 310-oldest.yaml environment


test: [Automatic] Constraints upgrades: dask, fsspec, hypothesis, imageio, ipython, napari-plugin-manager, napari-svg, psygnal, pydantic, pyqt6, pytest, rich, scikit-image

This PR is automatically created and updated by this napari GitHub Action:
https://github.com/napari/napari/tree/main/.github/workflows/upgrade_test_contraints.yml
It ensures that dependencies are tested with the most recent version.
The updated packages are:

cmake
contourpy
coverage
dask
filelock
fonttools
fsspec
hypothesis
imageio
ipykernel
ipython
lit
markdown-it-py
markupsafe
napari
napari-plugin-manager
napari-svg
pip
platformdirs
psygnal
pydantic
pyqt6
pytest
rich
scikit-image
typing-extensions
urllib3

Did we have any workflow using Ubuntu 20.04 and PyQt6? The latest release (6.5.1) is broken on ubuntu 20.04

Very confused by:
The conflict is caused by:
    The user requested napari 0.1.dev1+g701d919 (from /tmp/.tox/.tmp/package/1/napari-0.1.dev1+g701d919.tar.gz)
    The user requested (constraint) napari==0.4.15

Why is the constraint 0.4.15?
The dev version issue is similar to what we saw in docs repo.
Maybe can solve by cloning with:
		  # ensure version metadata is proper
          fetch-depth: 0

But why is this happening now?

Depends on #5919 and https://github.com/napari/napari-svg/pull/34. After merging both (or merging alternatives) it needs to be regenerated.

We again hit the wall of circular dependencies...


pytorch version

Hello, which version of pytorch is used in this work? Thanks

Hi, please check the versions of dependencies here.


RSD can be both in cyrilic and latin script

Cool library! It can be really useful. I'll definitely bookmark it for my future projects. But one note:
RSD can be both in cyrilic and latin script
In Serbia, we use RSD, Din and Дин
Where Дин and Din are exactly the same just written in different script. I'm not sure how it would be made to choose which one to use, but maybe other currencies can have multiple scripts.

Hello @eboye ! Nice to hear that! I didn't know it.
I've added a new function (v1.0.2) called setSymbol, so now you can modify and add a custom Symbol to your currency format:
Currency::currency("RSD")->setSymbol("Din")->format(123456789)  // 123.456.789,00 Din

Great, thanks!


docs(testing-karma): Fix incorrect code snippet in README

The code snippet under "Create a karma.conf.js" in the testing-karma README contained syntax error. The esm option was not correctly nested in the second parameter given to the merge function. This PR fixes it by moving the esm option to the correct place.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


String Substring optimization

Default Atom Representation should be (char* offset_start offset_end) instead of just char*.

tail-string is pretty good. Good enough for compiler internals.


Azure DevOps  Dependency Check task needs java

Hi,
I installed  OWASP Dependency checker and works fine with  host agent.  But I found issue on self build server and agent.
I thought last vesion is typescript/nodejs but it still needs java JRE installation?  Please confirm.
==============================================================================
Task         : OWASP Dependency Check
Description  : Dependency Check is a Software Composition Analysis (SCA) tool that attempts to detect publicly disclosed vulnerabilities contained within a project's dependencies.
Version      : 6.1.1
Author       : Dependency Check
Help         : More Information
Starting Dependency Check...
Setting report directory to D:\w2\63\TestResults\dependency-check
Creating report directory at D:\w2\63\TestResults\dependency-check
Downloading Dependency Check latest installer from GitHub..
Downloading ZIP from "https://github.com/jeremylong/DependencyCheck/releases/download/v8.1.0/dependency-check-8.1.0-release.zip"...
Dependency Check script set to D:\w2_tasks\dependency-check-build-task_47ea1f4a-57ba-414a-b12e-c44f42765e72\6.1.1\dependency-check\bin\dependency-check.bat
Invoking Dependency Check...
Path: D:\w2_tasks\dependency-check-build-task_47ea1f4a-57ba-414a-b12e-c44f42765e72\6.1.1\dependency-check\bin\dependency-check.bat
Arguments: --project "VTSOnline" --out "D:\w2\63\TestResults\dependency-check" --scan "D:" --format HTML --format JUNIT --failOnCVSS 1
C:\Windows\system32\cmd.exe /D /S /C "D:\w2_tasks\dependency-check-build-task_47ea1f4a-57ba-414a-b12e-c44f42765e72\6.1.1\dependency-check\bin\dependency-check.bat --version"
'java' is not recognized as an internal or external command,
operable program or batch file.

Hi @mmatonok,
Yes, the Java JRE is needed, because the Azure DevOps extension just utilizes the OWASP Dependency Check CLI.
The extension itself is written in TypeScript.
Br, Chris


why param_receptive_field_center_start is [3, 3, 7, 7, ...] in model v1?

I am studying your code of LFFD and here comes some question. Why in the configuration of model v1, the param receptive field center start is written in list [3, 3, 7...]. just as the paper said, branch1 has a RF size of 55, so the center could not be at (3,3), can you give me an explanation?
Thanks a lot.

@vealocia In the paper, we assumed that the readers know how RF related parameters are calculated. Thus, we do not present formulations. Here, I give you an example of how RF center starts are obtained. The example is presented in 1-D case.


defaultLocale not working

hi
I have used this package in the next 13.4 app router project
config for i18n with persian and english language and defaulLocale equal fa
const I18nMiddleware = createI18nMiddleware({
locales: ['en', 'fa'],
defaultLocale: 'fa',
urlMappingStrategy: 'rewriteDefault',
})
why after the implementation of the project
localhost:4200/en it opens?

What does your Accept-Language header looks like in the initial request? You might want to usethe resolveLocaleFromRequest option in the middleware to manually override the logic.

thank you
My problem is solved


.net Installer Issue

I am getting an error downloading .net.

Can you let me know which file I need so I can install manually?

Have you retried it? Maybe it was just a random Issue (e.g. because some other App in the Background making a ton of http Requests)
Or it can also be an Issue that some AV/Firewall Software is blocking the Request.
In any way: you need the .NET 8 Desktop Runtime (Win x64): https://download.visualstudio.microsoft.com/download/pr/907765b0-2bf8-494e-93aa-5ef9553c5d68/a9308dc010617e6716c0e6abd53b05ce/windowsdesktop-runtime-8.0.8-win-x64.exe
But still run the Installer after that. And when Installing a major .NET Runtime Release for the first Time, it is good practice to reboot the PC.

No further Feedback => Closed


supports Symfony 6

Hello, you have made a great work with this bundle.
This little contribution allows your bundle to work with Symfony 6
Thanks

No problem spoted on my side

Looks great and works for me, as well 🙂 We'll release this in an upcoming update.
Thank you for your contribution!


Giant crawls produce weird errors

A11yWatch version:
a11ywatch_cli 0.8.30
What happened:
 a11ywatch crawl --url va.gov --save -t -S produces different failures. Sometimes I get a broken pipe failure. Othertimes the crawl seems to just crawl the homepage and produces the result:
{"code":0,"data":[{"cdn_connected":false,"domain":"www.va.gov","insight":null,"issues":[],"issues_info":{"ada_score":100,"error_count":0,"issue_meta":{"skip_content_included":true},"issues_fixed_by_cdn":0,"notice_count":0,"possible_issues_fixed_by_cdn":0,"total_issues":0,"warning_count":0},"last_scan_date":"2023-01-17T23:24:03.059Z","page_load_time":{"color":"#A5D6A7","duration":306,"duration_formated":"Lightning Fast"},"script":null,"url":"https://www.va.gov/disability/file-disability-claim-form-21-526ez","user_id":0}],"message":"Crawled 1 page in 2.580977003s","success":true}

Expected behavior:
Completed a crawl with all crawled page errors.

@bbertucc can you upgrade and try only adding the -S for subdomains and not the tld flag. Also if you can add the https:// before the website. When the crawl outputs one link it’s when the protocol was not specific for the url.

@bbertucc the majority of va.gov is not static. They most likely do not have a system that can handle dynamic content at this rate showing the flakiness in results from 960-1400 pages etc. I need to see what the status code output is when the request start to drop.

Deploying a fix now. Sitemap option was missing a critical yield point.

@bbertucc the issue was actually due to the gRPC connection dropping when the page is invalid. If you upgrade the docker images you should be back in action.

https://github.com/a11ywatch/pagemind/releases/tag/v0.7.28 is the service for the release.


f/accelerator-admin-support merge to incubation

set https://github.com/opendatahub-io/odh-dashboard/issues/1255 to incubating
Description

alright... all tests pass but model serving seemed to have the most trouble getting to pass. I am using the tests from main not incubation bc incubation was missing structure.
the other big conflicts had to do with performance metrics, service mesh, and bias metrics. those i had to swithc over to using the new feature flag area stuff. this was not big, just had to replace where usefeatureflag was used
other than that, it was mostly ok and merge combined the rest of the conflicts
How Has This Been Tested?

go over core ares and make sure they still work
Test Impact

all tests pass
Request review criteria:

Self checklist (all need to be checked):

[x] The developer has manually tested the changes and verified that the changes work
[x] Commits have been squashed into descriptive, self-contained units of work (e.g. 'WIP' and 'Implements feedback' style messages have been removed)
[x] Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
[x] The developer has added tests or explained why testing cannot be added (unit tests & storybook for related changes)

If you have UI changes:

[ ] Included any necessary screenshots or gifs if it was a UI change.
[ ] Included tags to the UX team if it was a UI/UX change (find relevant UX in the SMEs section).

After the PR is posted & before it merges:

[ ] The developer has tested their solution on a cluster by using the image produced by the PR to main

Is this a problem?

@alexcreasy

ran through manually testing of the feature with an image in the cluster. all good. adding labels


TextKey - Unit issue

Text key has many distance units. But many of them might be more than PointValue only. Can be pixel value and others.

I just tested some properties (font size, leading, baseline shift) and I don't think that they can have any different unit than pointsUnit. It is possible to enter values in different units such as px and mm, but that doesn't even get reflected in the Action Descriptor. The conversion is done beforehand by Photoshop, so the ActionDescriptor already has the according points value.


Icons show strange

I tried kuf:fts_window_1w_locked, kuf:fts_window_1w_tilt and kuf:fts_window_1w_open, but all show very strange, maybe they overlap or something?

I have the same issue with the kuf:fts_window_2w_open_lr.
 I didn't find a reproducible way to convert the icons with InkScape yet. For HomeAssistant, the icons has to be converted to a single SVG path. This is what I'm doing right now:

EditSelectAll
10*"SelectionUnGroup
StrokeToPath
SelectionCombine
FileSave

This sequence works for most icons. I assume there is a problem in Inkscape. Any help is appreciated.

Sorry, I am not familiar with that, if i could I would help you. I would thankful if you could convert these Icons, i don't need more :D

kuf:fts_window_1w_unlocked looks like  should look like
kuf:fts_window_1w_tilt looks correct
kuf:fts_window_1w_open looks like  should look like
kuf:fts_window_1w_locked looks like  should look like
kuf:fts_window_1w looks like  should look like

Having the same issue. Any fix yet?

kuf:fts_window_1w_unlocked looks like  should look like
kuf:fts_window_1w_tilt looks correct
kuf:fts_window_1w_open looks like  should look like
kuf:fts_window_1w_locked looks like  should look like
kuf:fts_window_1w looks like  should look like

Having the same issue. Any fix yet?

No response, I switched to this "https://github.com/elax46/custom-brand-icons/issues/142#event-6271222355"

This issue still applies. Is there a fix out there or a workaround?

Using version 1.1.0 of KNX User Forum Icons set loaded into Home Assistant through HACS.
Several icons have spurious artifacts/errors
scene_freezer
scene_dishwasher
scene_washing_machine
All have the same error, what looks like a descending bar graph in the lower right hand corner of the icon.
This error is not present in the SVG or PNG files. There may be other icons with this same error. I'm using some of the other "scene" series and they do not have the error.
Since it is an identical error, it is likely stray code in the .js file.
I've tried to delete/ reinstall the repository with the same result.
I tried downloading version 1.0.0 and 1.0.1 and both have the same errors.

Also
I just discovered the measure_battery icons all have the same image
measure_battery_0
measure_battery_100
measure_battery_75
measure_battery_50
measure_battery_25
All display the measure_battery_100 icon

Here's the path for the "scene_freezer" icon
I put breaks in to make it more readable - as I undersatnd "M" and "m" are moving the cursor, "c" is a bezier curve path and "z" closes that path.
I don't know how to read the coordinates to find the lower right hand portion where the anomaly is occurring.
  I deleted entire sections of that and I didn't see the icon change.

The icons for "scene_robo_lawnmower_active" and "scene_robo_lawnmower_paused" are both using the paused icon.


add ElementHandle.parentElement

Is it expected to add such apis, I use them frequently like this
ElementHandle.parentElement.children[1] for example

puppetter also doesn't have this, seems the way to do this is with https://pptr.dev/api/puppeteer.page.evaluatehandle

ElementHandle.parentElement.children[1]

The API as written is not technically feasible. At best (with the current API), it could be
await(await ElementHandle.parentElement()).children()[1]

we will be moving to locators soon, which solve this problem. With locators you could just do
Locator.parentElement.children[1]

and it could just work. TLDR; This is dependent on #2

This trick works in puppeteer https://stackoverflow.com/a/66435477/6900891


New profile method

For each feature:

Orange line: median
Box plot: First-third quantile (25%,75%)
Black caps: 2.5% and 97.5% percentiles
Gray caps: Full range

New style:

Thanks for this, these plots look like a very useful improvement. I haven't reviewed the code yet, but I'm noticing in the Gains plot that the Leaves stats are cropped?

Thanks for this, these plots look like a very useful improvement. I haven't reviewed the code yet, but I'm noticing in the Gains plot that the Leaves stats are cropped?

Sorry, I was just not handling zero values. Now it is fixed

Just a comment: right now I added a dictionary like {feature_name:feature_id} to the ensebleDict created by the xgboost converter. This should be added to all the converters.
I have not done it yet but as a workaround, if this dictionary is not present in the ensembleDict, in the plot you would see just feat_0, feat_1 ,... instead of the actual feature names

I have applied all the suggestions.
This is how the "both" option looks like


Model Builder Error

Model Builder Version : <IP_ADDRESS>.27905
Visual Studio Version : 16.11.18
Bug description
With freshly-updated [as of today, 2022-08-19] VS2019 and model builder, I tried to use the new ML builder wizard "Value prediction". It failed with an exception: Method not found: 'System.Collections.Generic.IEnumerable1<System.ValueTuple2<Microsoft.ML.AutoML.RunDetail`1<!!0>,Int32>> Microsoft.ML.AutoML.BestResultUtil.GetTopNRunResults
Steps to Reproduce
Installed latest VS 2019 and updated Model Builder to latest version
Create new project. Right click on project -> Add -> "Machine Learning"
Choose image classification, "Local ML", Next.
As input, choose an appropriate folder, observe "data preview" looks like I expected, with images being shown under various classifications
Train, Start Training.
Expected Experience
Pretty much what I'd expect from the tutorial, and of a wizard.
Actual Experience
Shortly after click "start training", I get this:
"GPU Service not found. Falling back to CPU AutoML Service."
Moments later, this exception:
Method not found: 'System.Collections.Generic.IEnumerable1<System.ValueTuple2<Microsoft.ML.AutoML.RunDetail1<!!0>,Int32>> Microsoft.ML.AutoML.BestResultUtil.GetTopNRunResults(System.Collections.Generic.IEnumerable1<Microsoft.ML.AutoML.RunDetail1<!!0>>, Microsoft.ML.AutoML.IMetricsAgent1<!!0>, Int32, Boolean)'.
at Microsoft.ML.ModelBuilder.AutoMLService.Experiments.AutoMLExperiment3.d__21.MoveNext() at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.ML.ModelBuilder.AutoMLService.Experiments.AutoMLExperiment3.ExecuteAsync(IDataView trainData, IDataView validateData, ColumnInformation columnInformation, Nullable1 userCancellationToken, Nullable`1 timeout)
at Microsoft.ML.ModelBuilder.AutoMLEngine.d__30.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 147

Dupe of https://github.com/dotnet/machinelearning-modelbuilder/issues/2221


Badge is not updating

I recently achieved 100% for my project, but the badge continues to say 99%.  I've tried clearing cache and cookies, so I think it's something server side?  Perhaps it's run on a schedule?

Also see https://www.bestpractices.dev/en/projects/4282

Gah, I just noticed https://github.com/coreinfrastructure/best-practices-badge/issues/2070, so might be a duplicate of that one.

I'm also having the same issue:
https://www.bestpractices.dev/projects/8358

Drat. We made a number of changes that should have completely eliminated this problem. I notice that https://www.bestpractices.dev/en/projects/4282 is at 100%, so it eventually updates, but it takes its time.
We specifically tell our CDN (Fastly) to throw away the badge image. My best current hypothesis is that there's a race between where we tell the CDN to drop it and the retrieval of the image. We update the data before we tell the CDN to drop it, and take various steps to make sure it works 100% when tested locally. However, on the deployed system, if the two actions are not handled in the order we supply, then this could be the result. That would explain a number of problems.
If this hypothesis is correct, then I think there are two things we can do:

In the /projects list, if the change was made recently, force a specific image to be shown. We already do this on the specific badge entries, for the same reason.
After a badge level change, resend a "throw away the cache of this image" request after a period of time. That way, if the CDN receives the commands out-of-order, it'll eventually clear the cache and get the correct answer this time.

Other ideas welcome.

BTW, congrats on earning the badge!

We've made changes (as I said), but I've left this issue open because it's notoriously hard to be sure that you fixed a race condition - it's not a bug that reliably reproduces.

I'm fine with closing this one, especially if #2070 is the same issue.

Still causing problems. https://www.bestpractices.dev/projects/9469/badge shows 88%
The following workaround works: https://www.bestpractices.dev/projects/9469/badge?foo

Sadly, https://www.bestpractices.dev/projects/9469/badge shows "passing" to me. It's hard to fix it when I can't reproduce :-(.
I have a request: Can you do a force-reload on your web browser? Basically go to the page https://www.bestpractices.dev/projects/9469/badge , hold down the "shift" key, and while holding down the "shift" key press the reload button:
Source: https://shaminospage.blogspot.com/2023/01/quick-tip-use-shift-refresh-in-your-web.html
If that fixes it, GREAT. If not, I could purge all caches, but since it shows correctly for me I don't think that would solve anything.

Ok, I think I've got it. It was really puzzling to figure out. This is on the same computer, first with Windows' curl:
curl https://www.bestpractices.dev/projects/9469/badge
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="184" height="20" role="img" aria-label="openssf best practices: passing"><title>openssf best practices: passing</title><linearGradient id="s" x2="0" y2="100%"><stop offset="0" stop-color="#bbb" stop-opacity=".1"/><stop offset="1" stop-opacity=".1"/></linearGradient><clipPath id="r"><rect width="184" height="20" rx="3" fill="#fff"/></clipPath><g clip-path="url(#r)"><rect width="133" height="20" fill="#555"/><rect x="133" width="51" height="20" fill="#4c1"/><rect width="184" height="20" fill="url(#s)"/></g><g fill="#fff" text-anchor="middle" font-family="Verdana,Geneva,DejaVu Sans,sans-serif" text-rendering="geometricPrecision" font-size="110"><text aria-hidden="true" x="675" y="150" fill="#010101" fill-opacity=".3" transform="scale(.1)" textLength="1230">openssf best practices</text><text x="675" y="140" transform="scale(.1)" fill="#fff" textLength="1230">openssf best practices</text><text aria-hidden="true" x="1575" y="150" fill="#010101" fill-opacity=".3" transform="scale(.1)" textLength="410">passing</text><text x="1575" y="140" transform="scale(.1)" fill="#fff" textLength="410">passing</text></g></svg>

and now with using Brave (Chrome, really)'s copy network request as Powershell request, which is supposed to mimic the original request using Powershell:
Invoke-WebRequest -UseBasicParsing -Uri "https://www.bestpractices.dev/projects/9469/badge" `
>> -WebSession $session `
>> -Headers @{
>> "authority"="www.bestpractices.dev"
>>   "method"="GET"
>>   "path"="/projects/9469/badge"
>>   "scheme"="https"
>>   "accept"="text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"
>>   "accept-encoding"="gzip, deflate, br, zstd"
>>   "accept-language"="en-CH,en;q=0.9,fr;q=0.8,es;q=0.7"
>>   "cache-control"="no-cache"
>>   "pragma"="no-cache"
>>   "priority"="u=0, i"
>>   "sec-ch-ua"="`"Brave`";v=`"129`", `"Not=A?Brand`";v=`"8`", `"Chromium`";v=`"129`""
>>   "sec-ch-ua-mobile"="?0"
>>   "sec-ch-ua-platform"="`"Windows`""
>>   "sec-fetch-dest"="document"
>>   "sec-fetch-mode"="navigate"
>>   "sec-fetch-site"="none"
>>   "sec-fetch-user"="?1"
>>   "sec-gpc"="1"
>>   "upgrade-insecure-requests"="1"
>> }

StatusCode        : 200
StatusDescription : OK
Content           : <svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="234"
                    height="20" role="img" aria-label="openssf best practices: in progress 88%"><title>openssf best
                    practices…
RawContent        : HTTP/1.1 200 OK
                    Connection: keep-alive
                    Server: Cowboy
                    Report-To: {"group":"heroku-nel","max_age":3600,"endpoints":[{"url":"https://nel.heroku.com/reports
                    ?ts=1726923354&sid=af571f24-03ee-46d1-9f90-a…
Headers           : {[Connection, System.String[]], [Server, System.String[]], [Report-To, System.String[]],
                    [Reporting-Endpoints, System.String[]]…}
Images            : {}
InputFields       : {}
Links             : {}
RawContentLength  : 1248
RelationLink      : {}

Notice how the first request is "passing" and the 2nd one is "88%". I believe Fastly caches by IP address and if some headers match as well. I also get 88% when using Edge or Firefox but "passing" when using plain cURL (also on Linux).
With the original Browser (Brave), using shift + enter to load the URL, it gets 88% (I see the 200 reply, and not 304 in the Network tab in the inspector so it's not local caching). But if I use the Tor feature in the same browser (so it uses a different IP address), I get the passing badge (incognito mode gets 88% so it's not caused by it (Tor mode enforces incognito as well)).
Last test with my phone (Brave):

using wifi (same public IP as my computer): 88%
using 4G network (different public IP): passing

THANK YOU so much for that information! I'm still mystified about exactly what the problem is - and thus how to fix it - but this definitely provides much more specific information to help us track this down. Thank you for providing all that!

Notice how the first request is "passing" and the 2nd one is "88%". I believe Fastly caches by IP address and if some headers match as well. I also get 88% when using Edge or Firefox but "passing" when using plain cURL (also on Linux).

I can confirm the header matching, we even specifically identify the headers to use. It caches my source address (really domain name not just IP address), but of course that makes sense, different web sites serve different data :-). As far as the requester IP address goes, that has an indirect effect as I understand it. Fastly has a large number of different intermediate caching systems; if a client has a different IP address, it's more likely to end up connecting to a different Fastly system for a cache.

With the original Browser (Brave), using shift + enter to load the URL, it gets 88% (I see the 200 reply, and not 304 in the Network tab in the inspector so it's not local caching). But if I use the Tor feature in the same browser (so it uses a different IP address), I get the passing badge (incognito mode gets 88% so it's not caused by it (Tor mode enforces incognito as well)).

It sounds like different systems you're using are deterministically getting different answers. That is surprising, but I have a hypothesis. Those intermediate systems implemented by Fastly may have more than 1 computer, and it's possible that they determinististically determine which one to use based on the client IP address and port number (instead of using a round-robin to determine which one is used). I expect that your systems go through a NAT (e.g., on your router), so Fastly will see different port numbers on different applications. Result, in this hypothesis, is that they'd talk to different Fastly systems. If those different systems have inconsistent data, then we have an explanation.
Maybe we need to send several cache clears, over time, for a given data value, since the result may not "take". Fastly is very fast (that what it's designed to do!), but this may suggest that the speed creates some subtle race conditions we must account for. Anyway, that's my current hypothesis.

Okay, I think I have a set of hypotheses & solution if the hypotheses are correct.
On Sun May 12 18:35:02 2024 -0400 we merged commit 69afb7fb06e298951e4d5b579d795930dcf4e5af to fix a race condition on CDN caching (details there). The problem described here predates that fix, so you could argue that "well, the problem was before the fix".
The problem with that hypothesis is that caches expire. The bad cached data should have long since expired, and now have good data provided. But we now use shorter cache times; it's possible that these specific Fastly systems received data when we used longer cache times, and those haven't expired.
Okay, but when there's an update, we send out a request to Fastly to purge old data... that should have eliminated the old bad data. After reviewing things, though, there's a quirk that might explain this. We purge the Fastly data immediately and resend it on a delay; the delayed resend should reset everything. However, the delayed resend is in a job stored in RAM. The best practices site reboots once a day; it's possible the system was rebooted while the job was pending. If that is the problem, the solution would be to move the job queue to the database, making the database a little busier but persisting the job.
So I think I'll force a "purge all" on the production site. That will clear all caches, and that should fix the problem for all current badges. If that works, I'll look into moving the jobs into the database (that shouldn't be difficult, just take a little time).

I've purged all cached data, so you should only get correct data now.
I'm a little suspicious of my hypothesis. Even in a race condition, once the badge expired it should have gotten a new one. I don't have a better idea for a hypothesis, though.

Still getting the old badge with shift + reload.

Still getting the old badge with shift + reload.

That... shouldn't be possible... {BOGGLE}. Sorry that it's still not working.
I'm running out of explanations. Maybe our API call to Fastly failed silently?! So, I've logged directly into Fastly and invoked a forced "purge all" of all cached values, through directly demanding it of them.
Please do shift + reload again, and let me know if all is okay now. That should DEFINITELY fix it. If that does NOT fix it, I'll be completely perplexed and annoyed at Murphy.

If shift-reload still fails, I need to review the cache-control settings. This should not be happening with the values we send. So if it's happening, my best guess is that the "settings we think we're using aren't the ones we're using'.

I just re-tested now and it finally works! passing badge.


Initial Pass at Interface for MessageBus Abstraction

Checkout interface.go for the heart of the interface.

@jpwhitemn , corect. The user will transfer the configuration data from the services config from consul into the MessageBusConfig. Used this same pattern for the Registry.

Late to the party, I know.
Given Go support for closure and the compiler/optimizer internally represents structures as pointers to minimize copying, I've grown to appreciate the esthetics and simplicity of always working with a struct. By extension, I suggest we return struct rather than returning *struct as well as never create side effects in function/method parameters.

Closing as this was a preview and still WIP


Add Bulk Actions

Currently you have to update posts one at a time. It'd be great to add bulk actions such as image removal or category updates etc.

A nice feature to have, but something I don't immediately plan to incorporate for this simple blog project at the moment. If someone wants to take a crack at it, feel free to!


code coverage with coveralls.io

https://coveralls.io/docs/ruby

We will be adding code coverage reporting down the road soon. Thank you for raising it.


[Feature request] Cancel ajax request

Sometimes when you have a custom filter (let say a search form) maybe you want some kind of google like live refresh while the user type the search, in this cases when i refresh the data it make multiple request to server,
Im thinkin in a (maybe) cancel method to cancel the current request when the params change and you want to update the ajax request.

I don't think that can be done in ES6 at the moment. I also do not plan to do anything more on this repo as I've already shifted to Vuetable-2. Unless someone will submit PR for new feature, no new feature will get added.

Im sorry im thinking about the vuetable-2, i made a mistake when google the package, but as i see in vuetable-2 you use axios, so its possible to cancel the request:
https://github.com/mzabriskie/axios#cancellation
I will try to do do a pull request with this feature, thanks


[Suggestion] It would be better to remove the extra "Container Apps" node

OS: All
Build Version: 20220706.3
Repro Steps:

Make sure the resource groups by Resource Type.
Install the Container app extension -> Expand the subscription.
Check whether there is no extra "Container Apps" node.

Actual:
There is an extra "Container Apps" node.

Suggestion:
It would be better to remove the extra "Container Apps" node.
More Info:
The "Container Apps Environment" node contains the container apps.

I plan on adding a setting that will take care of this problem.
By default, the Environment node will be displayed but the user can also display the Container Apps node as well.


error in sign-in at Create Transaction

FatalThrowableError in PrivateKeyList.php line 54:
Type error: Argument 1 passed to BlockCypher\Crypto\PrivateKeyList::addPrivateKey() must be an instance of BitWasp\Bitcoin\Key\PrivateKeyInterface, instance of BitWasp\Bitcoin\Crypto\EcAdapter\Impl\PhpEcc\Key\PrivateKey given, called in /home/pixel/Documents/tests/vendor/blockcypher/php-client/lib/BlockCypher/Crypto/PrivateKeyList.php on line 43
here is code
public function addPrivateKey(PrivateKeyInterface $privateKey)
{
$pubKeyHex = $privateKey->getPublicKey()->getHex();
$this->privateKeys[$pubKeyHex] = $privateKey;
}

Had the same problem when signing a bitcoin transaction. The problem has to do with the interface file its expecting has been moved and renamed inside the bitWasp library.
from:
use BitWasp\Bitcoin\Key\PrivateKeyInterface;
to:
use BitWasp\Bitcoin\Crypto\EcAdapter\Key\PrivateKeyInterface;
Changing this inside the BlockCypher\Crypto\PrivateKeyList.php file will fix this problem as long as you do not update blockcypher/php-client or reinstall with composer. This is not ideal of course. I made a pull request for this but the only solution I can think of is forking the main repo, creating a branch for the change and adding it to your composer.json as a dependancy (a bit complicated but composer explains this in their docs) or just symlink into the vendor folder but you have to ensure this is done correctly too. Good luck

Try using v2.0.0-beta.1 and please report back. Thank you!


does postgresql need to be installed on the elasticsearch server in order for the river to run?

I have both postgresql and elastic search installed on the same machine, and pull data from the a postgres server. The river runs fine.
I have started a new elastic search server with the jdbc and postgres plugins. But the river does not run. I checked the examples given on the github page. It looks like for postgresql the first step is to install postgresql on the same machine of the elasticsearch. However, we don't need to do the same thing for mysql in the example 2.

It turned out the issue is the jdbc plugin 's access permission is not set properly.


Support for CAroot for Windows Firefox

Hello - is there a way to get this working with Windows Firefox? It is working with Chrome as expected. TY!

OK, found the solution in another thread:
On Firefox 75, go to about:config and enable security.osclientcerts.autoload
https://blog.mozilla.org/security/2020/04/14/expanding-client-certificates-in-firefox-75/


Fix issues #51

overflow property has been removed from Stack in latest master, dev channel
and it's will also remove in future.
https://api.flutter.dev/flutter/widgets/Stack/overflow.html
This overrides clipBehavior for now due to a staged roll out without breaking Google. We will remove it and only use clipBehavior soon

thanks for contributing!


Updated the pagination

Issue Number
fixes #195

Describe the changes you've made
I had made pagination arrows disable/hidden along with their text ('prev' and 'next') when the pagination is not needed.
Describe if there is any unusual behavior (Any Warning) of your code(Write NA if there isn't)
N/A
Checklist

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] All new and existing tests passed.
[x] The title of my pull request is a short description of the requested changes.

Provide a Deployed link of route/page that needs to review
Preview: Deploy preview link here with the appropriate route
http://localhost:3000/

Ma'am, I tried to change it altogether (whole DOM element). But it didn't work it out.As the icons is still visible when the pagination is not needed. then I take a dig into Eos icons file and they have several props in which color props is one of them. When I implemented with the color prop. It worked and That's why I used it like this. In the browser, It works perfectly fine.

Changes requested

Ma'am, the main issue is the icon visibility actually. The whole injection has already skipped the DOM altogether. There is the main span class that has the condition for hiding the 'prev' and 'next' buttons when pagination is not needed but it doesn't change the icon's color because EOS icons have their several own props which color prop is one of them. I just changed the icon visibility color which matches the text with the help of the 'color' prop when pagination is not required. It works perfectly fine in the browser. You can see this hosted link http://localhost:3000/. Kindly review it @cyntss.


example file name mismatch

The example says to create a file called request_body.json but the example command lines use test_sentiment_payload.json

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 94f56647-73eb-3d35-49a8-44f4f9143902
Version Independent ID: 03149d1c-3567-6977-4070-79f88779d292
Content: Quickstart: Use the Sentiment Analysis client library and REST API - Azure AI services
Content Source: articles/ai-services/language-service/sentiment-opinion-mining/quickstart.md
Service: cognitive-services
Sub-service: language-service
GitHub Login: @aahill
Microsoft Alias: aahi

@chrbsg
Thanks for your feedback! We will investigate and update as appropriate.

@chrbsg
Thanks for reporting this! We have created a PR for this issue and the changes should go live soon.


chore: release prep

Summary

Updated PowerValidatedSolutions from v13.0.0 to v13.1.0
Updated VMware.PowerCLI from v2.5.0 to v2.6.0.
Updated module versions in docs/snippets/install-module.ps1
Added PowerShell Gallery downloads badge to the docs/index.md

Type

[ ] Bugfix
[ ] Enhancement or Feature
[ ] Code Style or Formatting
[ ] Documentation
[ ] Refactoring
[x] Chore
[ ] Other
Please describe:

Breaking Changes?

[ ] Yes, there are breaking changes.
[x] No, there are no breaking changes.

Test and Documentation

[x] Tests have been completed.
[ ] Documentation has been added or updated.

Issue References

Additional Information

Just a minor change.

Addressed


Is there method available to request facebook email

Hi.
I am storing users information against their Email on Firebase. I am using this library to show Firebase Login Dialog.
However in case of Facebook i am not getting user email. I want to know if there is any helper method available in the SDK that can set permissions on the Facebook Login. If not then will it be available in future?
Thanks.

Not currently, no!
However you should be able to easily modify the FacebookAuthProvider class to request email scope and retrieve it that way while we look for a better solution.

FirebaseUI now gets email from Facebook, closing this issue.  Please re-open if the new solution is not working for you.


Default logLevel DEBUG or INFO?

Inside the PyCharm Console, I always get messages at the DEBUG level:

However, when I look to ixmp/ixmp/utils.py is this for me not clear, because the level is only set there to INFO:
def logger():
    """Access global logger"""
    global _LOGGER
    if _LOGGER is None:
        logging.basicConfig()
        _LOGGER = logging.getLogger()
        _LOGGER.setLevel('INFO')
    return _LOGGER

Is there another possibility to change the logging level?

Sorry, I believe that the output in your screenshot is determined in the compiled ixmp.jar, and this is distinct from the _LOGGER.setLevel(). Not sure why this was changed in the first place before the latest release.
We will rework the Java logging in the next release.

Make sense, because I could not get access via the python logging root handler...

@zikolach and @fonfon would it be possible to access and set the log level in java from python as well?

@gidden you can do it by programmatically by calling e.g.:
org.apache.log4j.Logger.getLogger(Platform.class.getPackage().getName()).setLevel(org.apache.log4j.Level.DEBUG)

Alternatively I can add a (static) method to Platform class for setting it. But we have bundled log4j.properties file with some predefined settings:
log4j.appender.FA=org.apache.log4j.DailyRollingFileAppender
log4j.appender.FA.file=D:/tmp/GasmAgeServlet-app.log
log4j.rootLogger=FA, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d %5p %c:%L - %m%n
log4j.logger.at.ac.iiasa.kolp.cleanage.servlet.DatabaseHelper=info
log4j.logger.at.ac.iiasa.ixmp.servlet.AuthDbFilter=info
log4j.logger.at.ac.iiasa.ixmp.servlet.RestServlet=debug

which I would replace with
log4j.rootLogger=WARN, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d %5p %c:%L - %m%n
log4j.logger.org.flywaydb.core.internal=warn

as default

Hi @tzipperle —going to close this now since the code is substantially changed and I don't suspect this can occur anymore. Now, the top-level ("ixmp") logger is configured here:
https://github.com/iiasa/ixmp/blob/59e99e47082a52203e493a5b0b68afc65506e469/ixmp/__init__.py#L44-L48
The Java backend, at the point when any new Platform is instantiated, uses either a user argument or copies this value:
https://github.com/iiasa/ixmp/blob/59e99e47082a52203e493a5b0b68afc65506e469/ixmp/backend/jdbc.py#L212-L216
After this, the mp.set_log_level() method can be used to control the log messages emitted from Java. At any time or point, the "ixmp" logger can be manipulated in the usual Python way.
If any similar issue is still happening with the newest code, please open a new issue.


Pushing to remote failed

[I'm supposed to describe steps to replicate my issue here and if I don't do that, I understand that my issue will be probably ignored.]

Brackets 1.3.0-16022 (win), Git 1.9.5.msysgit.1 & Brackets-Git 0.14.27
Pushing to remote failed
Error: cmd-spawn-timeout: C:\Program Files (x86)\Git\cmd\git.exe push --porcelain --progress origin master

call: git push --porcelain --progress origin master
Error: cmd-spawn-timeout: C:\Program Files (x86)\Git\cmd\git.exe push --porcelain --progress origin master
    at timeoutPromise (file:///C:/Users/Antonio/AppData/Roaming/Brackets/extensions/user/zaggino.brackets-git/src/Cli.js:219:27)
    at file:///C:/Users/Antonio/AppData/Roaming/Brackets/extensions/user/zaggino.brackets-git/src/Cli.js:262:33

#1031


Zap-cli context commands return connection errors

I am trying to import a saved context into Zap-cli and any command involving 'context' returns the below error. Also, running a regular zap-cli quick scan command works just fine, so I don't think it's an API or daemon issue. any insight would be appreciated
-bash-4.2$ zap-cli context list
Traceback (most recent call last):
File "/usr/bin/zap-cli", line 11, in
load_entry_point('zapcli==0.10.0', 'console_scripts', 'zap-cli')()
File "/usr/lib/python2.7/site-packages/click/core.py", line 664, in call
return self.main(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/click/core.py", line 644, in main
rv = self.invoke(ctx)
File "/usr/lib/python2.7/site-packages/click/core.py", line 991, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python2.7/site-packages/click/core.py", line 991, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python2.7/site-packages/click/core.py", line 837, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python2.7/site-packages/click/core.py", line 464, in invoke
return callback(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/click/decorators.py", line 26, in new_func
return ctx.invoke(f, ctx.obj, *args[1:], **kwargs)
File "/usr/lib/python2.7/site-packages/click/core.py", line 464, in invoke
return callback(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/zapcli/commands/context.py", line 25, in context_list
contexts = zap_helper.zap.context.context_list
File "/usr/lib/python2.7/site-packages/zapv2/context.py", line 35, in context_list
return six.next(six.itervalues(self.zap._request(self.zap.base + 'context/view/contextList/')))
File "/usr/lib/python2.7/site-packages/zapv2/init.py", line 159, in _request
data = self._request_api(url, get)
File "/usr/lib/python2.7/site-packages/zapv2/init.py", line 149, in _request_api
return self.session.get(url, params=query, proxies=self.__proxies, verify=False)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 546, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 510, in send
raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPConnectionPool(host='<IP_ADDRESS>', port=8090): Max retries exceeded with url: http://zap/JSON/context/view/contextList/?apikey= (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7efbfbde60d0>: Failed to establish a new connection: [Errno 111] Connection refused',)))

same error happen to me

Same problem for me, did you guys find a solution?

Same thing happens for me as well...

This is probably caused due to the fact that you did not specifiy the API_KEY. As you can see in the URL, the API KEY parameter is empty. This can be solved by setting the ZAP_API_KEY env variable (I advise to also set the ZAP_PORT and ZAP_PATH env variables). You can find the API KEY in ZAP desktop via Tools - Options - API


doc(operator) The certificates are files and need to be referenced as encryptedFile.

The current examples try to load the certificate files in as environment variables which causes the pods to crashloop. Since these secrets are files we need to reference them as files so that they get mounted into the container as a file.

Thanks for opening this PR @damian-myerscough!


Initial SQLite3 XC7 routing graph implementation.

This converts the xc7 routing graph import from using channels.json to channels.db, a sqlite3 database.  This was done for several reasons:

Certain operations (e.g. pruning dead nodes) was fairly difficult in the previous data structure and layout.
Memory usage was very high as a side affect of keeping all data in memory
Parse times of channels.json (~800 MB) was high.
It was hard to see what was taking time, because database-like operations were in the code.  Now data lookup is explicit and can be optimized via SQL indicies or denormalized tables if shown to be required.

In addition, by moving to a database to keep relations between tiles, wires, nodes (prjxray node), graph_node (VTR rr_node), and graph_edges (VTR rr_edge) to the database, it should be easier to do optimizations on the graph, like merge nodes or splitting tiles (ala #415).
This also fixes #292 by avoiding recomputing the graph every time a pb_type XML is changed.  Instead the graph is computed only when prjxray DB changes.
This is fairly large rework of the routing import, but hopefully it is more understandable than the previous incarnation.

Looks like this approach could have a lot of benefits. I may be out of the loop on other discussions around this. @litghost What alternatives have been considered?
As it gets closer I think a description of the tables and overall approach should be added before we merge.

@litghost What alternatives have been considered?

Do you mean alternative data stores?  Or do you mean alternative approaches?

@litghost What alternatives have been considered?

Do you mean alternative data stores? Or do you mean alternative approaches?

Both

@litghost What alternatives have been considered?

Do you mean alternative data stores? Or do you mean alternative approaches?

Both

In terms of alternative data stores, SQLite is built into python3 distributions, requires no servers, is file oriented and is fairly battle tested.  Most of the operations that were giving me the feeling that I was re-implementing a poor mans database fit neatly into SQL.  Most of operations are relational oriented except for the final graph build in prjxray_create_edges, so I felt that a SQL db versus say a graph DB or NoSQL.  There is an open question of whether there should be two databases, one the is SQL for the relation queries (e.g. node<->wire<->pip) versus the graph itself.  However there are still some relational type queries done on the constructed graph (e.g. is this graph node alive?) that works very nicely in the SQL format.
If you have other concrete data store suggestions, I'm open to examining them.
In terms of alternative approaches, the initial xc7 implementation is an alternative in of itself.  The problems of that implementation (repeated work, expensive data passing, poor data querying) led me to the conclusion that a fully bespoke approach had reached its limit.  In particular, two tasks that were fairly challenging in the previous implement (dead node pruning and tile split) were signalling that another approach was required.  In terms of a third way, I don't have any bright ideas besides some external data store.  As I mentioned above, there is potentially value in fragmenting the data stores, but I don't believe we have good justification to do that at this time.

As it gets closer I think a description of the tables and overall approach should be added before we merge.

Added.

Any initial measurement of runtime/memory improvement?

Right now everything is actually about the same.  However the new structure should be amendable to some easy optimizations.
Prjxray database change:
Before runtime:  20:54.88
After runtime: 22:12.28
Memory usage before: 25845448 kB
Memory usage after: 28814828 kB
Routing import only:
Before runtime: 7:05.24
After runtime: 6:57.06
Memory usage before: 25845448 kB
Memory usage after: 28814828 kB
I've started looking into what is going on with the memory usage during routing import in the new method, and with incremental XML generation, memory usage is down to 5075020 kB and runtime is down a minute.  There are further gains that are easy now that a database is storing the data throughout, so with some further PR's, can probably lower the runtime and memory foot print even further.

With incremental edge import, memory usage on routing import is down to 3632632 kB with a small increase in runtime.  Probably not getting the memory usage down any more without further increases in runtime.


Migration from RAFT to BFT: IT: check mixed orderer

Check what happens to a Raft-based orderering servive that has two channels, A & B.
Channel A is migrated to BFT but B does not.
If the ordering service exits maintenance mode, do both channels continue to function properly?
We do not plan to support this scenario in production, but we would like to experiment and see what happens.

Tested it at the end of this online meetup about deploying Hyperledger Fabric 3.0 using the bevel-operator-fabric

Tested it at the end of this online meetup about deploying Hyperledger Fabric 3.0 using the bevel-operator-fabric

@dviejokfs Thanks! Could you summarize here what exactly you tested and how, and what were the results?
Our decision right now is to not officially support mixed orderers, i.e. an orderer with channels in both etcdraft & BFT. This does not mean it will not work, we just do not want to commit to it working and devote testing and development resource to it. However it would be interesting and important to know what actually happens in this case.


minecraft-server: EULA var needs to be a string (docker-compose.yml)

docker-compose refuses to build a container without the quotation marks.

Thanks for catching and fixing that...it used to work, but apparently broke somewhere along the way.


Normalize the relevance_score returned by the rerank API

Is your enhancement related to a problem? Please describe.
It seems that normalizing the relevance_score returned by the rerank API is a recommended practice:
Finally, if I have to choose the best predictor among these three, I would give it to the reranker model.
For two reasons:
The reranker model's trendline on both impressions and clicks is more well-spanned over the X-axis compared to the embedding model's trendline, giving it more "dynamic range," which makes it closer to the ground truth trendline.
The score is well-distributed between 0 and 1. Note that this is mostly because our latest Reranker v2 model is calibrated, whereas our earlier jina-embeddings-v2-base-en released in Oct. 2023 was not, so you can see its values spread over 0.60 to 0.90. That said, this second reason has nothing to do with its approximation to G(q,d)G(q,d); it is just that a well-calibrated semantic score between 0 and 1 is more intuitive to understand and compare.
https://huggingface.co/BAAI/bge-reranker-v2-m3
You can get a relevance score by inputting query and passage to the reranker. And the score can be mapped to a float value in [0,1] by sigmoid function.
from FlagEmbedding import FlagReranker
reranker = FlagReranker('BAAI/bge-reranker-v2-m3', use_fp16=True) # Setting use_fp16 to True speeds up computation with a slight performance degradation

score = reranker.compute_score(['query', 'passage'])
print(score) # -5.65234375

# You can map the scores into 0-1 by set "normalize=True", which will apply sigmoid function to the score
score = reranker.compute_score(['query', 'passage'], normalize=True)
print(score) # 0.003497010252573502

Describe the solution you'd like
Normalize the relevance_score returned by the rerank API, enabled by default and allowed to be disabled.

https://github.com/gpustack/llama-box/releases/tag/v0.0.80

closed as completed


os: Remove empty directory fails with "process cannot access the file because it is being used by another process" on <= Windows 10

I don't have a simple test case for you, but some data points:

This is a new issue in Go 1.20. It's working OK in Go 1.19.
It's been reported failing on Windows 8 and 10. It always fail for me on my Windows 8 VM.
It's not failing on Windows 11.

The failing code is more or less the snippet below:
_, err = f.Readdirnames(1)
if err == io.EOF {
   // Empty dir.
   err = os.Remove(dirname)
  // Gets "process cannot access the file because it is being used by another process" on Windows <= 10.
}

And dirname has recently been used (but not anymore).
https://github.com/gohugoio/hugo/issues/10781

OK, turns out I had the directory in question open, which now fails on Windows. I guess that's sensible. Closing.


chore: Use normal functions so this.skip has the correct context

Removes fat arrow functions in favour of function functions so this.skip can be used.

Didn't I fix this with https://github.com/ipfs/interface-ipfs-core/commit/2545ddd3dd6cdc46a3aef340cd26cf23460a4708? Why isn't it showing as a merge conflict, interesting.

Weird, maybe it's because the changes are now identical to master?  Anyway, this is no longer required.


[SPARK-29829][SQL] SHOW TABLE EXTENDED should do multi-catalog resolution

What changes were proposed in this pull request?

Add ShowTableStatement and make SHOW TABLE EXTENDED go through the same catalog/table resolution framework of v2 commands.
We don’t have this methods in the catalog to implement an V2 command

catalog.getPartition
catalog.getTempViewOrPermanentTableMetadata

Why are the changes needed?

It's important to make all the commands have the same table resolution behavior, to avoid confusing
USE my_catalog
DESC t // success and describe the table t from my_catalog
SHOW TABLE EXTENDED FROM LIKE 't' // report table not found as there is no table t in the session catalog
Does this PR introduce any user-facing change?

yes. When running SHOW TABLE EXTENDED Spark fails the command if the current catalog is set to a v2 catalog, or the table name specified a v2 catalog.
How was this patch tested?

Unit tests.

Can one of the admins verify this patch?

cc: @cloud-fan @rdblue @viirya @imback82

Can one of the admins verify this patch?

ok to test

Test build #113849 has started for PR 26540 at commit 1110168.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18719/
Test PASSed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113849/
Test FAILed.

Test build #113887 has started for PR 26540 at commit db116d8.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18755/
Test PASSed.

Test build #113891 has started for PR 26540 at commit c973705.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18759/
Test PASSed.

Test build #113887 has finished for PR 26540 at commit db116d8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113887/
Test PASSed.

Test build #113891 has finished for PR 26540 at commit c973705.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113891/
Test PASSed.


Pass roc_dealloc a size argument

We should pass roc_dealloc a size and then alignment (in that order, just like the order we use for roc_alloc).
This way, we can use Rust's alloc::dealloc to implement roc_dealloc, which is useful in wasm (currently we just leak memory there because we can't use alloc::dealloc) as well as in bindgen.
Also, if I'm using an arena allocation strategy in my host, and a request comes in from the application to deallocate something, I could check to see if that happens to be the allocation at the very end of the bump arena; if it does happen to be, then I can actually free the memory by moving the pointer back.

Oh, so this actually has a worse problem for seamless slices. It isn't just about "self-promoting". It is about freeing seamless slices in general:
a = List.repeat 7 7  # allocate A
b = List.dropFirst a # b is a slice of a. It increments the refcount.
# a goes out of scope and decrements the refcount
...
# b goes out of scope, decrements the refcount and needs to free a
# if b needs to know the full capacity of a, we have a problem. It doesn't know.
# no way to free a.

Yeah, I think we should avoid this feature and let someone wanting to use rusts allocator do this: https://shift.click/blog/on-dealloc/ Though with only needing to cache the size, not the layout. Will just add an extra max(8, alignment) bytes to each allocation. Otherwise, I think we would need to do this under the hood in roc. Which would add the cost all the time even when it isn't needed. This will only add the cost in the case it is needed for rust. @rtfeldman thoughts?

Aha, that's a clever solution! Seems like the way to go. 👍


Replication api

I see that Replication API is still missing, any plan to add it later? I need it enough to make a PR later.

Replication is something we are definitely interested in covering, but there is no immediate plan to implement it at this point. If you are interested in taking a stab at it, I'd be happy to accept a PR!