Add Unix socket support to Stolon

Is your feature request related to a problem? Please describe.
In the Stolon design RFC (#1006), transactions spread anonymously through the Zcash network using Tor via a SOCKS port.
Tor also supports Unix sockets, which have a number of advantages:

Unix sockets are the easiest way to use the iOS Tor.framework: https://github.com/iCepa/Tor.framework#usage
Ports can be accessed by any local process, but Unix sockets are protected by filesystem permissions (or are passed between processes as file descriptors)

(Tor.framework can also use host/port connections. But we're not sure if iOS apps can open localhost ports - or if they require extra permissions from the user.)
Describe the solution you'd like
Stolon currently uses tokio-socks::Socks5Stream for its connection to Tor.

Add UnixStream support to tokio-socks::Socks5Stream.

Add a send_transaction_unix_socket method, which takes tor_addr: unix::net::SocketAddr. send_transaction_unix_socket should only be available on Unix platforms -  it should require cfg(unix).

See the existing Stolon::send_transaction, which takes a std::net:SocketAddr containing an IPv4 or IPv6 address.
Describe alternatives you've considered
Make the existing send_transaction method generic over std::net::SocketAddr (TCP) and unix::net::SocketAddr (Unix socket). This design might be confusing, because the method would only be generic on Unix platforms. And is much harder to document that the function arguments are conditional on cfg(unix). Rustdoc is designed for conditional functions.
Do nothing. Stolon would be harder to implement on iOS and similar platforms, and less secure on Unix platforms. (Particularly when other network services are running on the same host.)
Additional context
For more context, see the original discussion :
https://github.com/ZcashFoundation/zebra/pull/1006#discussion_r483348582
See also Tor's SOCKS port and unix socket documentation:
https://2019.www.torproject.org/docs/tor-manual.html.en#SocksPort
And Tor.framework's Unix Socket and Port initialisers:
https://github.com/iCepa/Tor.framework/blob/master/Tor/TORController.h#L28

If I understand correctly, the benefits of this change are that on existing platforms (Unix), we'd get better security, and it would allow us to more easily support a new platform (iOS).
In the Unix case, I'm a little unclear on how much of a security improvement we'd get by allowing file permissions for the SOCKS port.  What kind of risks are there to sharing the SOCKS listener with other applications on the same system?  For something like the Tor control port, it seems much more obvious, but I'm not sure about the SOCKS listener.
If the security benefit on existing platforms (Unix) is relatively marginal, it seems like this would be a good feature to have but probably a low priority until we know we want to support new platforms (iOS) where it's required. Does that seem like a fair assessment?

Another potential complication is that the existing networking code assumes that peer connections are made over TCP, not over Unix sockets.  So to support this change, in addition to adding support to the tokio-socks library and changing the stolon API, I think we'd also need to change an unknown amount of the existing network code to allow UnixStreams.
Probably the way to do this would be to make the existing network code generic over an AsyncRead + AsyncWrite implementation, which could have some other benefits.  But scoping those changes out might not make sense until we know we want to move forward on implementing this.

This isn't a high priority for us right now - and it's a lot of work.
There is a security benefit - we prevent local processes from exfiltrating data over the SOCKSPort.

@teor2345 how does this relate to Android? Is there an Android equivalent to the iOS Tor.framework? Does it similarly rely on Unix Sockets?
Context:
I'm trying to assess the feasibility of using Stolon and/or Tor on mobile, particularly in conjunction with gRPC. It seems like direct use of Tor from a mobile application is still not quite viable.

@gmale using Tor on iOS and Android is a bit different to the desktop, but it works fine.
Here are some example Tor mobile apps:

OnionBrowser on iOS, using Tor.framework: https://onionbrowser.com/
Tor Browser for Android: https://www.torproject.org/download/#android
Orbot on Android, which lets other Android apps use Tor: https://guardianproject.info/apps/orbot/

Tor Browser for Android originally used Orbot, but they now have their own Tor integration:
https://blog.torproject.org/new-alpha-release-tor-browser-android
I am not sure about the Unix socket requirement for iOS. Tor.framework has a host/port method:
https://github.com/iCepa/Tor.framework/blob/master/Tor/TORController.h#L28
But I don't know if iOS apps can open localhost ports - or if that requires extra permissions. I'll edit the ticket to clarify.

Thanks @teor2345 for pointing me to those useful links. I pulled down the source for orbot and there's a lot there. Ideally, a wallet wouldn't have to rely on the existence of another app on the device and I assume that's at least part of the reason why Tor Browser switched approaches. So I dug around all the Tor open source projects and found a Tor service that seems promising but still feels maybe a bit too bulky for most apps.
I wonder if there is anything out there that streamlines this or takes a more modern approach on Android. I'm not sure what it would take to get gRPC to use Tor as a transport layer. No matter how we approach it, I think network privacy is going to be a very heavy lift on mobile.

@gmale have a chat to The Guardian Project about Tor mobile integration.
Maybe @n8fr8 can point you to the right person or channel?
They are the experts here. I mainly worked on the generic Tor binary/library, rather than integrating it in specific environments.

(Tor can act as a SOCKS or HTTP proxy, so if your gRPC library supports proxy settings, it should be pretty easy to set up.)

Tor Onion Proxy Library is an option:
https://github.com/thaliproject/Tor_Onion_Proxy_Library
Otherwise, there is also our Tor-Android library, which is the Tor binary and some very simple code to configure and start it up:
https://github.com/guardianproject/tor-android
Lastly, you can ask the user to just install Orbot, and if it is available, proxy through the Tor socks/http ports it provides using our NetCipher library:
https://github.com/guardianproject/netcipher

Thanks @n8fr8! There seem to be a lot of options for Android. What would be the best options for iOS, particularly if we want to avoid requiring the user to configure a companion app? Is iCepa the best analog to tor-android?

You'd want to integrate directly with Tor.framework: https://github.com/iCepa/Tor.framework
iCepa is our attempt to move this to an iOS Network Extension approach, but Tor isn't quite ready/able to run in that manner yet.

Unix sockets aren't required for reliable Tor integration, because arti supports an in-process DataStream type.


regress-vlg_constfunc11_iv

Created Surelog issue: https://github.com/chipsalliance/Surelog/issues/2342

Issue fixed.


Wood blocks

The WoodMaterial class needs to generate the following blocks:

Crafting table
Stripped bark
Stripped log
Plate
Composter
Trapdoor
Barrel
Button
Ladder
Planks
Stairs
Chest
Fence
Door
Gate
Sign
Slab

Recipes also need to be automatically added

commit b33df38 added most wooden blocks and recipes
missing:

Crafting table
Stripped bark
Stripped log
Composter
Barrel
Sign


Use argparse to get environment variables

The .env file isn't convenient for a docker deployment. Better use argparse to give the application the env values in a dedicated flag.

Done in docker branch.
Running on host  :
export DISCORD_TOKEN="xxx"
export DISCORD_APP_ID="xxx"
python -m pip install -r requirements.txt
python bot_discord &

Running on docker :
docker build -t wildpasta/discord_bot_chaise:1.8.2 .
docker run --env DISCORD_TOKEN="xxx" --env DISCORD_APP_ID="xxx" wildpasta/discord_bot_chaise:1.8.2


[Draft] Fix #578

Here is my first attempt to fix #578 without reintroducing a global context.
Two concerns here disturb me:

The Twig MacroTokenParser class is final so we need to rewrite the full class
Most important: In the #455 PR, @LukeTowers removed the controller property inside the Twig cms Extension class, but without this, I can't get any reference of the currently running controller to retrieve the controller context's variable, so afaik we need to reintroduce it. I created a method setController to bind the controller to the extension.

Replaced by https://github.com/wintercms/winter/commit/5b8d189df4cd1f7fdf8e318f111543b85a0e2613, thanks for your help @RomainMazB!


wrong example code

Hi,
I just want to advice that this code (README.md):
<form action="index.php" method="post">
    <?= \Volnix\CSRF\CSRF::getAsHiddenInput('custom_token_name') ?>
    <input type="text" name="action" placeholder="Enter an action."/>
    <input type="submit" value="Submit" name="sub"/>
</form>

is invalid because the method getAsHiddenInput() doesn't exist.
The real name of the method is getHiddenInputString()

Good catch!  Fixed in 3387e499aa7711be61e8b793005c4723bfa8ae01


Unable to use controllerAs syntax on controls attribute

From @Raptormstr on April 2, 2014 14:1
jQuery considers the attribute "controls" as a reserved word and will rename any string to simple "controls".  Look at the jQuery.js source code lines 623 & 6888 for 2.1.0 and 665 & 7960 for 1.11.0.  I do not believe jqLite has this same issue.
For example:
Before $compile: <leaflet controls="ctrl.myCustomControls">
After $compile: <div class="angular-leaflet-map ng-scope" controls="controls">
NOTE: I removed the other attributes and classes for brevity.
The only workaround if jQuery is required is to use the variable name "controls" and assign it to the $scope.
Copied from original issue: tombatossals/angular-leaflet-directive#348

From @tombatossals on April 2, 2014 17:3
You're right, good point.
What would you suggest to solve this? Rename the attribute, like "lfControls", for example?

From @Raptormstr on April 3, 2014 7:34
I believe renaming the attribute is the only option.  Since Angular will
execute the jQuery code long before this directive, there is no way to
intervene.  Also, "controls" is an HTML5 attribute for audio and video so
it should not be used at all.
As for your namespace suggestion, that would work but it should be applied
to all the attributes for consistency.
Stephen
On Apr 2, 2014 1:03 PM, "David Rubert"<EMAIL_ADDRESS>wrote:

You're right, good point.
What would you suggest to solve this? Rename the attribute, like
"lfControls", for example?
--
Reply to this email directly or view it on GitHubhttps://github.com/tombatossals/angular-leaflet-directive/issues/348#issuecomment-39356110
.

From @jsthivierge on March 5, 2015 16:30
Got a similar problem. I used grunt-contrib-htmlmin task to minimize my html and since controls is a HTML5 attribute and my scope variable is named controls (<leaflet controls="controls"></leaflet>), it got renamed to <leaflet controls></leaflet>.
The htmlmin task option collapseBooleanAttributes was set to true, therefore controls="controls" was collapsed to controls since it's an html5 attribute...
Any idea if the attributes will be renamed soon to prevent this type of conflict?


Last minute changes

Changed policy on project creation (everyone can do it)
Made the custodians field of project mandatory
Add is_superuser flag in /whoami response
Changed test to reflect new policies.
Fixed a weird dependency problem

This change is

Reviewed 3 of 3 files at r1, 1 of 1 files at r2.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


less import inline

Hi
is import inline supported?
it seems that with "postcss-less-engine":"^0.6.1" and "less": "^2.7.1" it's ignored
thanks so much

this is how i use the plugin at the `moment
const postCssPlugins = [];
postCssPlugins.push(stylelint({ configFile: path.join(process.cwd(), '.stylelintrc.json') }));
postCssPlugins.push(less({ strictMath: true, paths: lessPath }));
postCssPlugins.push(reporter({ clearMessages: true }));
postCssPlugins.push(autoprefixer({ browsers: AUTOPREFIXER_BROWSERS }));
// transform image url for CDN
postCssPlugins.push(url({
	url(imageurl) {
		if (minify) {
			if (!imageurl.startsWith('http')) {
				return imageurl.startsWith('/') ? `${cdnDomain}/${projectName}${imageurl}` : imageurl;
			}
		}
		return imageurl;
	},
}));

// CAN I USE
postCssPlugins.push(doiuse({
	browsers: ['ie >= 9', '> 1%'],
	ignore: [], // an optional array of features to ignore 'rem'
	ignoreFiles: [path.join(process.cwd(), 'node_modules', '/**/*.less')], // an optional array of file globs to match against original source file path, to ignore
	onFeatureUsage(usageInfo) {
		logger.warn('CAN I USE', usageInfo.message);
	},
}));
// postcss(plugins)  list of PostCSS plugins to be included as processors.
const cssProcessor = postcss(postCssPlugins);

// process less file
const processedCssObject = await cssProcessor.process(lessInput, { parser: less.parser, from: filepath });

maybe i do something wrong?

All Less features should be supported. But note that postcss-less needs to be the first PostCSS plugin used in your plugin stack.


Prevent install of Pester v5

Add -MaximumVersion 4.99.99 on Install-Module Pester. In yml files and ps1.

@jhoneill , @ili101. Whenever it works, extra eyes for this would be great. Thanks.

Agree.
I think the stuff @ili101  did a while back meant we're aligned with 4's syntax and should find moving to 5 fairly easy.  Suffering a bit of brain fade, but I think the version on Azure DevOps is a bit out of date, I'm sure something I did need me to install 4 or a later version of 4.


HHH-16089 allow @Synchronize for a collection

The h2 failure is unrelated to this change. I'm inclined to merge this. Any objectors?


How to use MaybeUninit on the some functions

Hallo,
as the title says I don't know how to use MaybeUninit in an idiomatic way. It is the first time I need to use it because of some functions in socket2. After some reading I implemented the following code snippet:
fn receive(&self) -> Result<Vec<u8>> {
    let mut buffer: [u8; 4096] = [0; 4096];

    let buffer = unsafe { &mut *(&mut buffer as *mut [u8; 4096] as *mut [MaybeUninit<u8>; 4096]) };

    let received = self.socket.recv( buffer)?;

    let buffer = unsafe { *(buffer as *mut [MaybeUninit<u8>; 4096] as *mut [u8; 4096]) };

    Ok(buffer[..received].to_vec())
}

Is this the way to use it? Is there an approach to convert from MaybeUninit and to it witthout this unsafe blocks and this ugly casting?
Thanks

Hi @eddi0815 working with uninitialised buffer is a bit of a pain on stable Rust at the moment, but improvements are coming. For example using the unstable Vec::spare_capacity_mut function (but you could implement that yourself in a couple of lines).
// Required for https://doc.rust-lang.org/std/vec/struct.Vec.html#method.spare_capacity_mut.
#![feature(vec_spare_capacity)]

use std::io;

use socket2::Socket;

fn receive(socket: &Socket) -> io::Result<Vec<u8>> {
    let mut buffer = Vec::with_capacity(4096);
    let received = socket.recv(buffer.spare_capacity_mut())?;
    // SAFETY: just received into the `buffer`.
    unsafe {
        buffer.set_len(received);
    }
    Ok(buffer)
}

Playground link: https://play.rust-lang.org/?version=nightly&mode=debug&edition=2018&gist=cd9fa7367e8d5162eb43c1ab4b44f108.

Hi @Thomasdezeeuw ,
thanks for the hint. This looks much cleaner.


Windows-Chrome artifact fails to install

Over the past couple days I have been hitting the following error when trying to deploy the Windows Chrome artifact on our Azure VMs::
Validating parameters.
Configuring PowerShell session.
Ensuring latest Chocolatey version is installed.
Preparing to install Chocolatey packages: googlechrome.
Chocolatey v0.10.15
Installing the following packages:
googlechrome
By installing you accept licenses for the packages.
chocolatey-core.extension v<IP_ADDRESS> (forced) [Approved]
chocolatey-core.extension package files install completed. Performing other installation steps.
Installed/updated chocolatey-core extensions.
The install of chocolatey-core.extension was successful.
Software installed to 'C:\ProgramData\chocolatey\extensions\chocolatey-core'
GoogleChrome v81.0.4044.138 (forced) [Approved]
googlechrome package files install completed. Performing other installation steps.
Downloading googlechrome 64 bit
from 'https://dl.google.com/tag/s/dl/chrome/install/googlechromestandaloneenterprise64.msi'
Download of googlechromestandaloneenterprise64.msi (60.46 MB) completed.
Error - hashes do not match. Actual value was '10570B6B3931CCE7F13763FD4F1AC7075E3036A38F13EDC2754C9894D2114B9F'.
ERROR: Checksum for 'C:\windows\TEMP\chocolatey\GoogleChrome\81.0.4044.138\googlechromestandaloneenterprise64.msi' did not meet '2566c802687c014aa7e89bce14de3cb7ed119da8a0374c9a7eb37b4bc3b0dc0a' for checksum type 'sha256'. Consider passing the actual checksums through with --checksum --checksum64 once you validate the checksums are appropriate. A less secure option is to pass --ignore-checksums if necessary.
The install of googlechrome was NOT successful.
Error while running 'C:\ProgramData\chocolatey\lib\GoogleChrome\tools\chocolateyInstall.ps1'.
See log for details.
googlechrome not installed. An error occurred during installation:
Stopping further execution as googlechrome has failed install.
This is try 1/3. Retrying after 300 milliseconds.
Error converted to warning:
Could not find a part of the path 'C:\ProgramData\chocolatey\lib\GoogleChrome.chocolateyPending'.
This is try 2/3. Retrying after 400 milliseconds.
Error converted to warning:
Could not find a part of the path 'C:\ProgramData\chocolatey\lib\GoogleChrome.chocolateyPending'.
Maximum tries of 3 reached. Throwing error.
Chocolatey installed 0/0 packages.
See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).
Did you know the proceeds of Pro (and some proceeds from other
licensed editions) go into bettering the community infrastructure?

ERROR: Installation failed (-1). Please see the Chocolatey logs in %ALLUSERSPROFILE%\chocolatey\logs folder for details.
The artifact failed to apply.
Is this a known issue?
Thanks,
Courtney

@TCourtneyOwen, the issue seems to point to the underlying package coming from the Chocolatey repo (see comments). It looks like we need to wait until it is fixed at the source.
Alternatively, if you trust the installer, you can install the same package using the Install Chocolatey Packages artifact instead.

@leovms That works!  Thanks


Align button styles to the UNDP guidelines

Feature Description
hey! let's change some style properties of buttons to aligned them with the UNDP design system.

Paddings are ’1rem 1.5rem’
Text-transform: uppercase
Font-size: 1rem;
Font-weight: 700;
letter-spacing: .03em; (optional)
I would strive to remove icons since UNDP design system use them rarely and also in different style. But this could be done case by case actually.

Feature Implementation
No response
Feature Alternatives
No response
Feature Impact
No response
Feature Rationale
No response
Feature Considerations
No response
Feature Requirements
No response
┆Issue is synchronized with this Wrike task by Unito

Use is-uppercase is-normal has-text-weight-bold for No.1, No.2 and No.3
Use $button-padding-vertical and $button-padding-horizontal for No.1

Also, just wanted to illustrate the idea about button color choice with this example.  Here, the 'Discard' button, which deletes user information, is colored blue, while the 'Keep changes' button is red. This setup should actually be reversed. The 'Discard changes' button, leading to irreversible actions, should be red to signal caution or alert

No. 1 is fixed by #2882


Upgrade CI and Python dependencies

fix #498
Hi @EnricoMi,
I just discovered this project.
This PR upgrades the checkout action to release v4. I also used the upgrade-pip-packages.sh script to upgrade the pip dependencies.
I hope that this PR might be helpful. I'd be grateful if you would consider sharing your feedback on this below.

Thanks for the contribution.
I tend to upgrade all CI dependencies at once and actions/checkout is not the only one that can be upgraded. If you could also upgrade all other dependencies, that would be grand.
I tend to upgrade the Python dependencies just before a release to avoid keeping upgrading them without releasing, which is some fruitless exercise.

Blocked by flaky download action v4: https://github.com/actions/download-artifact/issues/249

Thanks!


(#472) - Remove checkstyle suppression

This is for #472.
The changes are small, but I verified that:

all the tests were present
there was no more things to extract from the class
that it didn't make sense to transform the build method into a class because we would end up with two classes, one for XmlGraph just delegating to the second, and the second very similar to XmlGraph as it is currently written.

It also gave me the opportunity to understand what the code was doing, and I opened a few issues in cactoos and jpeek.

@paulodamaso 0rat seems to have missed this PR, can you tell him to take it into scope? Thx :)

@0crat in

This pull request #476 is assigned to @vzurauskas/z, here is why; the budget is 15 minutes, see §4; please, read §27 and when you decide to accept the changes, inform @paulodamaso/z (the architect) right in this ticket; if you decide that this PR should not be accepted ever, also inform the architect; this blog post will help you understand what is expected from a code reviewer; there will be a monetary reward for this job

@paulodamaso The points listed in the PR description make sense, nothing to comment. Can be merged.

@rultor merge

@rultor merge

@paulodamaso OK, I'll try to merge now. You can check the progress of the merge here

@rultor merge

@paulodamaso Done! FYI, the full log is here (took me 10min)

@0crat status

@0crat status (here)

@vzurauskas This is what I know about this job in C7JGJ00DP, as in §32:

The job #476 is in scope for 3days
The role is REV
The job is assigned to @vzurauskas/z for 3days
There is a monetary reward attached
@vzurauskas/z will get 25% of their hourly rate for this job
The job doesn't have any impediments
The budget is 15 minutes/points
These users are banned and won't be assigned:

@victornoel/z: This user reported the ticket

Job footprint (restricted area)

@paulodamaso I think this should be assigned to QA now?

@0crat out

@sereshqua/z please review this job completed by @vzurauskas/z, as in §30; the job will be fully closed and all payments will be made when the quality review is completed

@0crat quality bad

Quality is low, no payment, see §31


Feature Request: Import Jupyter notebooks

I've been looking for efficient ways to help users search Jupyter notebooks, and storing them in SQLIte may be one way to support this.
The Jupyter notebook .ipynb document format is a JSON document (specification), but only some of the elements are meaningful to users. For example, Jupyter notebooks include the notion of "typed" cells, such as markdown or code cells. Storing the contents of these cells, along the type of the cell and the cell order in a notebook, could provide one way of supporting search over them in a meaningful way.
More generally, indexing a Jupyter documents could be seen as an instance of only storing a subset of a JSON document in a structured way and as such provide a test case for a "parsed" JSON import?

Thank you for your request.
That's an interesting feature. I will consider the feature for the future release.

@psychemedia
I had implemented Jupyter Notebook support at sqlitebite 0.16.0
For example, https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/recurrent_network.ipynb will converted into the following:
$ sqlitebiter url "https://raw.githubusercontent.com/aymericdamien/TensorFlow-Examples/master/notebooks/3_NeuralNetworks/recurrent_network.ipynb"
[INFO] sqlitebiter url: convert 'raw.githubusercontent.com/aymericdamien/TensorFlow-Examples/master/notebooks/3_NeuralNetworks/recurrent_network.ipynb' to 'cells_source' table
[INFO] sqlitebiter url: convert 'raw.githubusercontent.com/aymericdamien/TensorFlow-Examples/master/notebooks/3_NeuralNetworks/recurrent_network.ipynb' to 'cells_outputs' table
[INFO] sqlitebiter url: convert 'raw.githubusercontent.com/aymericdamien/TensorFlow-Examples/master/notebooks/3_NeuralNetworks/recurrent_network.ipynb' to 'cells_outputs_kv' table
[INFO] sqlitebiter url: convert 'raw.githubusercontent.com/aymericdamien/TensorFlow-Examples/master/notebooks/3_NeuralNetworks/recurrent_network.ipynb' to 'cells_kv' table
[INFO] sqlitebiter url: convert 'raw.githubusercontent.com/aymericdamien/TensorFlow-Examples/master/notebooks/3_NeuralNetworks/recurrent_network.ipynb' to 'metadata_kernelspec' table
[INFO] sqlitebiter url: convert 'raw.githubusercontent.com/aymericdamien/TensorFlow-Examples/master/notebooks/3_NeuralNetworks/recurrent_network.ipynb' to 'metadata_language_info' table
[INFO] sqlitebiter url: convert 'raw.githubusercontent.com/aymericdamien/TensorFlow-Examples/master/notebooks/3_NeuralNetworks/recurrent_network.ipynb' to 'metadata_kv' table
[INFO] sqlitebiter url: convert 'raw.githubusercontent.com/aymericdamien/TensorFlow-Examples/master/notebooks/3_NeuralNetworks/recurrent_network.ipynb' to 'kv' table
[INFO] sqlitebiter url: number of created tables: 8
[INFO] sqlitebiter url: database path: out.sqlite

schema
sqlite> .schema
CREATE TABLE IF NOT EXISTS 'cells_source' (cell_id INTEGER NOT NULL, line_no INTEGER NOT NULL, text TEXT);
CREATE TABLE IF NOT EXISTS 'cells_outputs' (cell_id INTEGER NOT NULL, type TEXT NOT NULL, line_no INTEGER, data BLOB);
CREATE TABLE IF NOT EXISTS 'cells_outputs_kv' (cell_id INTEGER NOT NULL, key TEXT NOT NULL, value TEXT);
CREATE TABLE IF NOT EXISTS 'cells_kv' (cell_id INTEGER NOT NULL, key TEXT NOT NULL, value TEXT);
CREATE TABLE IF NOT EXISTS 'metadata_kernelspec' (key TEXT NOT NULL, value TEXT NOT NULL);
CREATE TABLE IF NOT EXISTS 'metadata_language_info' (key TEXT NOT NULL, value TEXT NOT NULL);
CREATE TABLE IF NOT EXISTS 'metadata_kv' (key TEXT NOT NULL, value TEXT NOT NULL);
CREATE TABLE IF NOT EXISTS 'kv' (key TEXT NOT NULL, value TEXT);

tables
cells_source

cell_id
line_no
text

0
0
# Recurrent Neural Network Example

0
1

0
2
Build a recurrent neural network (LSTM) with TensorFlow.

0
3

0
4
- Author: Aymeric Damien

0
5
- Project: https://github.com/aymericdamien/TensorFlow-Examples/

1
0
## RNN Overview

1
1

1
2

1
3

1
4
References:

1
5
- Long Short Term Memory, Sepp Hochreiter & Jurgen Schmidhuber, Neural Computation 9(8): 1735-1780, 1997.

1
6

1
7
## MNIST Dataset Overview

1
8

1
9
This example is using MNIST handwritten digits. The dataset contains 60,000 examples for training and 10,000 examples for testing. The digits have been size-normalized and centered in a fixed-size image (28x28 pixels) with values from 0 to 1. For simplicity, each image has been flattened and converted to a 1-D numpy array of 784 features (28*28).

1
10

1
11

1
12

1
13
To classify images using a recurrent neural network, we consider every image row as a sequence of pixels. Because MNIST image shape is 28*28px, we will then handle 28 sequences of 28 timesteps for every sample.

1
14

1
15
More info: http://yann.lecun.com/exdb/mnist/

2
0
from future import print_function

2
1

2
2
import tensorflow as tf

2
3
from tensorflow.contrib import rnn

2
4

2
5
# Import MNIST data

2
6
from tensorflow.examples.tutorials.mnist import input_data

2
7
mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)

3
0
# Training Parameters

3
1
learning_rate = 0.001

3
2
training_steps = 10000

3
3
batch_size = 128

3
4
display_step = 200

3
5

3
6
# Network Parameters

3
7
num_input = 28 # MNIST data input (img shape: 28*28)

3
8
timesteps = 28 # timesteps

3
9
num_hidden = 128 # hidden layer num of features

3
10
num_classes = 10 # MNIST total classes (0-9 digits)

3
11

3
12
# tf Graph input

3
13
X = tf.placeholder("float", [None, timesteps, num_input])

3
14
Y = tf.placeholder("float", [None, num_classes])

4
0
# Define weights

4
1
weights = {

4
2
'out': tf.Variable(tf.random_normal([num_hidden, num_classes]))

4
3
}

4
4
biases = {

4
5
'out': tf.Variable(tf.random_normal([num_classes]))

4
6
}

5
0
def RNN(x, weights, biases):

5
1

5
2
# Prepare data shape to match rnn function requirements

5
3
# Current data input shape: (batch_size, timesteps, n_input)

5
4
# Required shape: 'timesteps' tensors list of shape (batch_size, n_input)

5
5

5
6
# Unstack to get a list of 'timesteps' tensors of shape (batch_size, n_input)

5
7
x = tf.unstack(x, timesteps, 1)

5
8

5
9
# Define a lstm cell with tensorflow

5
10
lstm_cell = rnn.BasicLSTMCell(num_hidden, forget_bias=1.0)

5
11

5
12
# Get lstm cell output

5
13
outputs, states = rnn.static_rnn(lstm_cell, x, dtype=tf.float32)

5
14

5
15
# Linear activation, using rnn inner loop last output

5
16
return tf.matmul(outputs[-1], weights['out']) + biases['out']

6
0
logits = RNN(X, weights, biases)

6
1
prediction = tf.nn.softmax(logits)

6
2

6
3
# Define loss and optimizer

6
4
loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(

6
5
logits=logits, labels=Y))

6
6
optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)

6
7
train_op = optimizer.minimize(loss_op)

6
8

6
9
# Evaluate model (with test logits, for dropout to be disabled)

6
10
correct_pred = tf.equal(tf.argmax(prediction, 1), tf.argmax(Y, 1))

6
11
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))

6
12

6
13
# Initialize the variables (i.e. assign their default value)

6
14
init = tf.global_variables_initializer()

7
0
# Start training

7
1
with tf.Session() as sess:

7
2

7
3
# Run the initializer

7
4
sess.run(init)

7
5

7
6
for step in range(1, training_steps+1):

7
7
batch_x, batch_y = mnist.train.next_batch(batch_size)

7
8
# Reshape data to get 28 seq of 28 elements

7
9
batch_x = batch_x.reshape((batch_size, timesteps, num_input))

7
10
# Run optimization op (backprop)

7
11
sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})

7
12
if step % display_step == 0 or step == 1:

7
13
# Calculate batch loss and accuracy

7
14
loss, acc = sess.run([loss_op, accuracy], feed_dict={X: batch_x,

7
15
Y: batch_y})

7
16
print("Step " + str(step) + ", Minibatch Loss= " + \

7
17
"{:.4f}".format(loss) + ", Training Accuracy= " + \

7
18
"{:.3f}".format(acc))

7
19

7
20
print("Optimization Finished!")

7
21

7
22
# Calculate accuracy for 128 mnist test images

7
23
test_len = 128

7
24
test_data = mnist.test.images[:test_len].reshape((-1, timesteps, num_input))

7
25
test_label = mnist.test.labels[:test_len]

7
26
print("Testing Accuracy:", \

7
27
sess.run(accuracy, feed_dict={X: test_data, Y: test_label}))

cells_outputs

cell_id
type
line_no
data

2
text
0
Extracting /tmp/data/train-images-idx3-ubyte.gz

2
text
1
Extracting /tmp/data/train-labels-idx1-ubyte.gz

2
text
2
Extracting /tmp/data/t10k-images-idx3-ubyte.gz

2
text
3
Extracting /tmp/data/t10k-labels-idx1-ubyte.gz

7
text
0
Step 1, Minibatch Loss= 2.6268, Training Accuracy= 0.102

7
text
1
Step 200, Minibatch Loss= 2.0722, Training Accuracy= 0.328

7
text
2
Step 400, Minibatch Loss= 1.9181, Training Accuracy= 0.336

7
text
3
Step 600, Minibatch Loss= 1.8858, Training Accuracy= 0.336

7
text
4
Step 800, Minibatch Loss= 1.7022, Training Accuracy= 0.422

7
text
5
Step 1000, Minibatch Loss= 1.6365, Training Accuracy= 0.477

7
text
6
Step 1200, Minibatch Loss= 1.6691, Training Accuracy= 0.516

7
text
7
Step 1400, Minibatch Loss= 1.4626, Training Accuracy= 0.547

7
text
8
Step 1600, Minibatch Loss= 1.4707, Training Accuracy= 0.539

7
text
9
Step 1800, Minibatch Loss= 1.4087, Training Accuracy= 0.570

7
text
10
Step 2000, Minibatch Loss= 1.3033, Training Accuracy= 0.570

7
text
11
Step 2200, Minibatch Loss= 1.3773, Training Accuracy= 0.508

7
text
12
Step 2400, Minibatch Loss= 1.3092, Training Accuracy= 0.570

7
text
13
Step 2600, Minibatch Loss= 1.2272, Training Accuracy= 0.609

7
text
14
Step 2800, Minibatch Loss= 1.1827, Training Accuracy= 0.633

7
text
15
Step 3000, Minibatch Loss= 1.0453, Training Accuracy= 0.641

7
text
16
Step 3200, Minibatch Loss= 1.0400, Training Accuracy= 0.648

7
text
17
Step 3400, Minibatch Loss= 1.1145, Training Accuracy= 0.656

7
text
18
Step 3600, Minibatch Loss= 0.9884, Training Accuracy= 0.688

7
text
19
Step 3800, Minibatch Loss= 1.0395, Training Accuracy= 0.703

7
text
20
Step 4000, Minibatch Loss= 1.0096, Training Accuracy= 0.664

7
text
21
Step 4200, Minibatch Loss= 0.8806, Training Accuracy= 0.758

7
text
22
Step 4400, Minibatch Loss= 0.9090, Training Accuracy= 0.766

7
text
23
Step 4600, Minibatch Loss= 1.0060, Training Accuracy= 0.703

7
text
24
Step 4800, Minibatch Loss= 0.8954, Training Accuracy= 0.703

7
text
25
Step 5000, Minibatch Loss= 0.8163, Training Accuracy= 0.750

7
text
26
Step 5200, Minibatch Loss= 0.7620, Training Accuracy= 0.773

7
text
27
Step 5400, Minibatch Loss= 0.7388, Training Accuracy= 0.758

7
text
28
Step 5600, Minibatch Loss= 0.7604, Training Accuracy= 0.695

7
text
29
Step 5800, Minibatch Loss= 0.7459, Training Accuracy= 0.734

7
text
30
Step 6000, Minibatch Loss= 0.7448, Training Accuracy= 0.734

7
text
31
Step 6200, Minibatch Loss= 0.7208, Training Accuracy= 0.773

7
text
32
Step 6400, Minibatch Loss= 0.6557, Training Accuracy= 0.773

7
text
33
Step 6600, Minibatch Loss= 0.8616, Training Accuracy= 0.758

7
text
34
Step 6800, Minibatch Loss= 0.6089, Training Accuracy= 0.773

7
text
35
Step 7000, Minibatch Loss= 0.5020, Training Accuracy= 0.844

7
text
36
Step 7200, Minibatch Loss= 0.5980, Training Accuracy= 0.812

7
text
37
Step 7400, Minibatch Loss= 0.6786, Training Accuracy= 0.766

7
text
38
Step 7600, Minibatch Loss= 0.4891, Training Accuracy= 0.859

7
text
39
Step 7800, Minibatch Loss= 0.7042, Training Accuracy= 0.797

7
text
40
Step 8000, Minibatch Loss= 0.4200, Training Accuracy= 0.859

7
text
41
Step 8200, Minibatch Loss= 0.6442, Training Accuracy= 0.742

7
text
42
Step 8400, Minibatch Loss= 0.5569, Training Accuracy= 0.828

7
text
43
Step 8600, Minibatch Loss= 0.5838, Training Accuracy= 0.836

7
text
44
Step 8800, Minibatch Loss= 0.5579, Training Accuracy= 0.812

7
text
45
Step 9000, Minibatch Loss= 0.4337, Training Accuracy= 0.867

7
text
46
Step 9200, Minibatch Loss= 0.4366, Training Accuracy= 0.844

7
text
47
Step 9400, Minibatch Loss= 0.5051, Training Accuracy= 0.844

7
text
48
Step 9600, Minibatch Loss= 0.5244, Training Accuracy= 0.805

7
text
49
Step 9800, Minibatch Loss= 0.4932, Training Accuracy= 0.805

7
text
50
Step 10000, Minibatch Loss= 0.4833, Training Accuracy= 0.852

7
text
51
Optimization Finished!

7
text
52
Testing Accuracy: 0.882812

cells_outputs_kv

cell_id
key
value

2
name
stdout

2
output_type
stream

7
name
stdout

7
output_type
stream

cells_kv

cell_id
key
value

0
cell_type
markdown

0
metadata
{'collapsed': True}

1
cell_type
markdown

1
metadata

2
cell_type
code

2
execution_count
1

2
metadata
{'collapsed': False}

3
cell_type
code

3
execution_count
2

3
metadata
{'collapsed': False}

4
cell_type
code

4
execution_count
3

4
metadata
{'collapsed': True}

5
cell_type
code

5
execution_count
4

5
metadata
{'collapsed': False}

6
cell_type
code

6
execution_count
5

6
metadata
{'collapsed': True}

7
cell_type
code

7
execution_count
6

7
metadata
{'collapsed': False}

8
cell_type
code

8
execution_count

8
metadata
{'collapsed': True}

metadata_kernelspec

key
value

display_name
Python [default]

language
python

name
python2

metadata_language_info

key
value

codemirror_mode_name
ipython

codemirror_mode_version
2

file_extension
.py

mimetype
text/x-python

name
python

nbconvert_exporter
python

pygments_lexer
ipython2

version
2.7.12

metadata_kv
kv

key
value

nbformat
4

nbformat_minor
0

Any feedback or comments would be appreciated.

@thombashi oh, great - looks interesting; I'll try to have a play in next day or two and get comments back to you.
Thanks,
--tony

One of the things I'd like to be able to do is build a simple search tool around notebooks. It's great that if I can extract content from multiple notebooks into the same tables, but it would also be useful if there was another table that listed notebook_id and the notebook filepath/filename (or maybe, directory path and filename as separate columns) and then used notebook_id e.g. as part of a compound primary key in the other tables. Then I could run queries on the content of a single notebook, or pull back the name of a notebook that contained a particular search term.

Thank you for your feedback.
Nootebook ids (and the notebook filepath/filename) are surely useful.
I will consider that for the future release (probably the next release).

@psychemedia

another table that listed notebook_id and the notebook filepath/filename (or maybe, directory path and filename as separate columns)

I had added the above as _source_infor_ table.
Since sqlitebiter 0.19.0, the output SQLite database scheme is as follows when you convert IPython Notebooks (you can specify multiple notebooks)
CREATE TABLE IF NOT EXISTS '_source_info_' (source_id INTEGER NOT NULL, dir_name TEXT, base_name TEXT NOT NULL, format_name TEXT NOT NULL, dst_table TEXT NOT NULL, size INTEGER, mtime INTEGER);
CREATE TABLE IF NOT EXISTS 'cells_source' (source_id INTEGER NOT NULL, cell_id INTEGER NOT NULL, line_no INTEGER NOT NULL, text TEXT);
CREATE TABLE IF NOT EXISTS 'cells_kv' (source_id INTEGER NOT NULL, cell_id INTEGER NOT NULL, key TEXT NOT NULL, value TEXT);
CREATE TABLE IF NOT EXISTS 'cells_outputs' (source_id INTEGER NOT NULL, cell_id INTEGER NOT NULL, type TEXT NOT NULL, line_no INTEGER NOT NULL, data BLOB);
CREATE TABLE IF NOT EXISTS 'cells_outputs_kv' (source_id INTEGER NOT NULL, cell_id INTEGER NOT NULL, key TEXT NOT NULL, value TEXT);
CREATE TABLE IF NOT EXISTS 'metadata_kernelspec' (source_id INTEGER NOT NULL, key TEXT NOT NULL, value TEXT NOT NULL);
CREATE TABLE IF NOT EXISTS 'metadata_language_info' (source_id INTEGER NOT NULL, key TEXT NOT NULL, value TEXT NOT NULL);
CREATE TABLE IF NOT EXISTS 'kv' (source_id INTEGER NOT NULL, key TEXT NOT NULL, value TEXT);

Some of the attributes names differ from your statements (to also use for other file formats other than IPython Notebook):

notebook_id -> source_id
filepath/filename -> dir_name, base_name

Now, you can execute example queries that you provided in the previous post as the followings:
SELECT cell_id, GROUP_CONCAT(text, '\n')
FROM (
   SELECT cell_id, text
   FROM cells_source
   ORDER BY cell_id, line_no
   )
GROUP BY cell_id;

SELECT notebook, cell_id, GROUP_CONCAT(text, '\n')
FROM (
   SELECT base_name AS notebook, cell_id, text
   FROM cells_source, _source_info_
   WHERE _source_info_.source_id = cells_source.source_id
   ORDER BY cell_id, line_no
   )
GROUP BY cell_id;


Allow View Controllers to lazily register their own routes on ARSwitchboard

It's not reasonable to do this work on + initialize - so instead, create a protocol, find every class that conforms to the protocol ( and make sure this works with swift. ) Then when an ARSwitchboad is setup, ask for every class which conforms to register themselves.
see  http://stackoverflow.com/questions/9882984/how-do-i-get-list-of-all-classes-conform-to-certain-protocol-in-xcode-4
Protocol *protocol = @protocol(YourProtocol);

int numberOfClasses = objc_getClassList(NULL, 0);
Class *classList = malloc(numberOfClasses * sizeof(Class));
numberOfClasses = objc_getClassList(classList, numberOfClasses);

for (int idx = 0; idx < numberOfClasses; idx++)
{
    Class class = classList[idx];
    if (class_getClassMethod(class, @selector(conformsToProtocol:)) && [class conformsToProtocol:protocol])
    {
        NSLog(@"%@", NSStringFromClass(class));
    }
}
free(classList);

also, discover if this is slow.


[URLSession] Fix handling of compressed responses.

When the server answers with a compressed response (with
Content-Encoding header), cURL is configured to automatically decompress
the response. However, there was some inconsistencies with the
Foundation implementation in Darwin platforms, and some bugs in the
handling of compressed responses.

In Darwin Foundation, the expected number of bytes is -1 for
compressed responses. This is because Content-Lenght will report the
length of the compressed content, but the total bytes written reports
uncompressed lengths. The code is changed to set the expected number
of bytes to -1 if the response includes a Content-Encoding header
different to "identity".
Since the expected number of bytes is unknown, the data received
callback cannot check for the total bytes received to be equal to
close the file handler and provide the temporary file URL to the upper
level. That responsability has been moved into the complete task
callback, where it was already happening for tasks with completion
blocks.
Added two tests (one for data tasks, one for download tasks) with
gzipped data.
Since gzipped data cannot be represented by UTF-8 strings, the
HTTPServer code has to be modified to allow providing raw data as part
of the HTTP response. There's a lot of changes so the body is raw
data, and the previously provided strings are transformed into data
using UTF-8 instead.
There was a small bug in the HTTPServer code where the setUp will wait
for a flag to be different of -2 to indicate the server is ready.
However, the flag should be checked against -1, which is the initial
state, while -2 is the final state. I found this when the server port
that my test wanted to use was uninitialized, because the server is
started asynchronously in another queue, and the value wasn't valid
yet.

@swift-ci test

Changed an incorrect test assert message.

@swift-ci test

@swift-ci test

Fixed the indentation. I hope I fixed every case, but don’t hesitate to point out if you find any more.

@swift-ci test

@swift-ci test

@drodriguez Thanks! Looks good to me now.

@swift-ci test and merge

@swift-ci test and merge


Fixed a bug where the OauthPopupOpener was access the UI in a backgro…

The UI is no longer available in background threads so we now cache it when the Opener is created.

Its a fairly simple fix. I've tested it in my application and it works nicely.

@bdunn44 Please merge and publish to maven

@bsutton @bdunn44  please merge this fix and release a new version. thank you


[BUG] Cannot grow BufferHolder by size 559976464 because the size after growing exceeds size limitation<PHONE_NUMBER>

CTAS query failed with:
Cannot grow BufferHolder by size 559976464 because the size after growing exceeds size limitation<PHONE_NUMBER>

If we disable the "GPU accelerated row-column transpose" feature in GpuColumnarToRowExec.scala and built a new Spark RAPIDS jar myself, then the query works fine.
I can have a reproduce using NDS data in house. Below are my reproduce steps:
Env:
Spark 3.2.1 standalone cluster(single node)
rapids-4-spark_2.12-22.08.0-20220812.093657-44.jar
1 * Quadro RTX 6000
apache-hive-3.1.2
Sample Data

Firstly use NDS2.0 tool to generate 10GB TPCDS data with decimal and converted it to parquet files.
Saystore_sales parquet files are located in /home/xxx/data/tpcds_parquet_10G/store_sales directory.

Create a Hive external table on top of store_sales

CREATE EXTERNAL TABLE `nds`.`store_sales` (
  `ss_sold_time_sk` INT,
  `ss_item_sk` INT,
  `ss_customer_sk` INT,
  `ss_cdemo_sk` INT,
  `ss_hdemo_sk` INT,
  `ss_addr_sk` INT,
  `ss_store_sk` INT,
  `ss_promo_sk` INT,
  `ss_ticket_number` INT,
  `ss_quantity` INT,
  `ss_wholesale_cost` decimal(7,2),
  `ss_list_price` decimal(7,2),
  `ss_sales_price` decimal(7,2),
  `ss_ext_discount_amt` decimal(7,2),
  `ss_ext_sales_price` decimal(7,2),
  `ss_ext_wholesale_cost` decimal(7,2),
  `ss_ext_list_price` decimal(7,2),
  `ss_ext_tax` decimal(7,2),
  `ss_coupon_amt` decimal(7,2),
  `ss_net_paid` decimal(7,2),
  `ss_net_paid_inc_tax` decimal(7,2),
  `ss_net_profit` decimal(7,2),
  `ss_sold_date_sk` INT)
USING parquet
PARTITIONED BY (ss_sold_date_sk)
LOCATION 'file:/home/xxx/data/tpcds_parquet_10G/store_sales/';

Then run below to detect all Hive partitions automatically for this table.
MSCK REPAIR TABLE `nds`.`store_sales`;

Create another table with a new column with String type.
Here I try to create a task skew situation, so it only has a single value.

create table testbyhao_src as select *, 'a string col' newcol from nds.store_sales limit 5000000;

Run below query to trigger the issue

set spark.rapids.sql.hasNans=false;
set spark.sql.files.maxPartitionBytes=256MB;
set spark.sql.shuffle.partitions=1000;
set spark.rapids.sql.castDecimalToString.enabled=true;
set spark.sql.adaptive.coalescePartitions.enabled=false;

create table testbyhao as
select ss_sold_time_sk,
ss_item_sk,
concat('abcdef ', ss_customer_sk) as ss_customer_sk,
ss_cdemo_sk,
ss_hdemo_sk,
ss_addr_sk,
ss_store_sk,
max(COALESCE(ss_wholesale_cost,"abc")) as ss_wholesale_cost,
max(COALESCE(ss_list_price,'def')) as ss_list_price,
max(ss_sales_price),
case when min(ss_sales_price)=1 then 'abcdefgh'
when min(ss_sales_price)=2 and MAX(COALESCE(ss_sales_price,0)) > 10 then 'abcdefgh' else 'abcdefghxyz' end as ss_sales_price,
max(COALESCE(ss_ext_discount_amt,'SOMEVALUEFORTEST')) as ss_ext_discount_amt,
max(COALESCE(ss_ext_sales_price,'SOMEVALUEFORTEST')) as ss_ext_sales_price,
max(COALESCE(ss_ext_wholesale_cost,'SOMEVALUEFORTEST')) as ss_ext_wholesale_cost,
max(COALESCE(ss_ext_list_price,'SOMEVALUEFORTEST')) as ss_ext_list_price,
max(COALESCE(ss_ext_tax,'SOMEVALUEFORTEST')) as ss_ext_tax,
max(COALESCE(ss_coupon_amt,'SOMEVALUEFORTEST')) as ss_coupon_amt,
max(COALESCE(ss_net_paid,'SOMEVALUEFORTEST')) as ss_net_paid,
max(COALESCE(ss_net_paid_inc_tax,'SOMEVALUEFORTEST')) as ss_net_paid_inc_tax,
max(COALESCE(ss_net_profit,'SOMEVALUEFORTEST')) as ss_net_profit,

cast(max(COALESCE(ss_wholesale_cost*2,'SOMEVALUEFORTEST')) as decimal(18,3)) as ss_wholesale_cost_2,
max(COALESCE(ss_list_price*2,'SOMEVALUEFORTEST')) as ss_list_price_2,
max(COALESCE(ss_sales_price*2,'SOMEVALUEFORTEST')) as ss_sales_price_2,
max(COALESCE(ss_ext_discount_amt*2,'SOMEVALUEFORTEST')) as ss_ext_discount_amt_2,
max(COALESCE(ss_ext_sales_price*2,'SOMEVALUEFORTEST')) as ss_ext_sales_price_2,
max(COALESCE(ss_ext_wholesale_cost*2,'SOMEVALUEFORTEST')) as ss_ext_wholesale_cost_2,
max(COALESCE(ss_ext_list_price*2,'SOMEVALUEFORTEST')) as ss_ext_list_price_2,
max(COALESCE(ss_ext_tax*2,'SOMEVALUEFORTEST')) as ss_ext_tax_2,
max(COALESCE(ss_coupon_amt*2,'SOMEVALUEFORTEST')) as ss_coupon_amt_2,
max(COALESCE(ss_net_paid*2,'SOMEVALUEFORTEST')) as ss_net_paid_2,
max(COALESCE(ss_net_paid_inc_tax*2,'SOMEVALUEFORTEST')) as ss_net_paid_inc_tax_2,
max(COALESCE(ss_net_profit*2,'SOMEVALUEFORTEST')) as ss_net_profit_2,

max(COALESCE(ss_wholesale_cost*3,'SOMEVALUEFORTEST')) as ss_wholesale_cost_3,
max(COALESCE(ss_list_price*3,'SOMEVALUEFORTEST')) as ss_list_price_3,
max(COALESCE(ss_sales_price*3,'SOMEVALUEFORTEST')) as ss_sales_price_3,
max(COALESCE(ss_ext_discount_amt*3,'SOMEVALUEFORTEST')) as ss_ext_discount_amt_3,
max(COALESCE(ss_ext_sales_price*3,'SOMEVALUEFORTEST')) as ss_ext_sales_price_3,
max(COALESCE(ss_ext_wholesale_cost*3,'SOMEVALUEFORTEST')) as ss_ext_wholesale_cost_3,
max(COALESCE(ss_ext_list_price*3,'SOMEVALUEFORTEST')) as ss_ext_list_price_3,
max(COALESCE(ss_ext_tax*3,'SOMEVALUEFORTEST')) as ss_ext_tax_3,
max(COALESCE(ss_coupon_amt*3,'SOMEVALUEFORTEST')) as ss_coupon_amt_3,
max(COALESCE(ss_net_paid*3,'SOMEVALUEFORTEST')) as ss_net_paid_3,
max(COALESCE(ss_net_paid_inc_tax*3,'SOMEVALUEFORTEST')) as ss_net_paid_inc_tax_3,
max(COALESCE(ss_net_profit*3,'SOMEVALUEFORTEST')) as ss_net_profit_3,

PERCENT_RANK() OVER(PARTITION BY max(COALESCE(newcol,'SOMEVALUEFORTEST')) ORDER BY max(COALESCE(newcol, 0))) AS aaaa
from testbyhao_src
group by 1,2,3,4,5,6,7
;

Note: Here I disabled spark.sql.adaptive.coalescePartitions.enabled because if the AQE feature kicks in, it may firstly fail with GPU OOM. So you may not see the expected error.
And also to-be-safe, I set spark.sql.shuffle.partitions=1000 to large enough to avoid OOM of GPU memory as well.
If you read it here, you will get the error:
22/08/19 00:48:47 WARN TaskSetManager: Lost task 705.1 in stage 5.0 (TID 2022) (<IP_ADDRESS> executor 0): org.apache.spark.SparkException: Task failed while writing rows.
	at org.apache.spark.sql.errors.QueryExecutionErrors$.taskFailedWhileWritingRowsError(QueryExecutionErrors.scala:500)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:321)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$16(FileFormatWriter.scala:229)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:131)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.IllegalArgumentException: Cannot grow BufferHolder by size 559976464 because the size after growing exceeds size limitation<PHONE_NUMBER>
	at org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder.grow(BufferHolder.java:71)
	at org.apache.spark.sql.catalyst.expressions.codegen.UnsafeWriter.grow(UnsafeWriter.java:63)
	at org.apache.spark.sql.catalyst.expressions.codegen.UnsafeWriter.writeUnalignedBytes(UnsafeWriter.java:127)
	at org.apache.spark.sql.catalyst.expressions.codegen.UnsafeWriter.write(UnsafeWriter.java:110)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.writeFields_0_2$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithIterator(FileFormatDataWriter.scala:92)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:304)
	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1496)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:311)
	... 9 more

Hope it is not too complex.
I think I love NDS data so I used it as my data source. If you have better way to generate large enough data, that is fine too.

Since the code that caused the issue was reverted, I am moving this to 22.10.

I haven't made any changes to the block between (L920-L936)  https://github.com/amahussein/spark-rapids/blob/333a508700859802c8fe22935212f4a091[…]c/main/scala/com/nvidia/spark/rapids/GpuRowToColumnarExec.scala.
    // We can support upto 2^31 bytes per row. That is ~250M columns of 64-bit fixed-width data.
    // This number includes the 1-bit validity per column, but doesn't include padding.
    // We are being conservative by only allowing 100M columns until we feel the need to
    // increase this number. Spark by default limits codegen to 100 fields
    // "spark.sql.codegen.maxFields".
    if ((1 until 100000000).contains(output.length) &&
        CudfRowTransitions.areAllSupported(output)) {
      val localOutput = output
      rowBased.mapPartitions(rowIter => GeneratedInternalRowToCudfRowIterator(
        rowIter, localOutput.toArray, localGoal, semaphoreWaitTime, streamTime, opTime,
        numInputRows, numOutputRows, numOutputBatches))
    } else {
      val converters = new GpuRowToColumnConverter(localSchema)
      rowBased.mapPartitions(rowIter => new RowToColumnarIterator(rowIter,
        localSchema, localGoal, converters,
        numInputRows, numOutputRows, numOutputBatches, semaphoreWaitTime, streamTime, opTime))
    }

Basically, that block checks if the number of columns in the schema is less than "100000000". If so, it will pick the optimized InternalRowToCudf
This check might lead to the crash because a schema containing strings may exceed the size  2^31 bytes.
However, this is not a simple fix. It is a fundamental design issue because in order to know the row size, we have to process the entire columns to get their sizes.

To clarify, the problem that originally triggered this issue was 2^31 bytes per row was exceeded, not 2^31 bytes per batch (i.e.: the cudf limits).  It is very, very unlikely any valid data is going to exceed the 2GB row limit.  The problem that initially triggered the issue was pretty clearly a data corruption issue, as it was trying to build a single row's value, and one of the string columns had a >100MB value that when added to the current size of the single row caused it to exceed the 2GB limit (implying earlier columns had already pushed it crazy far in size).  I suspect there's a problem with GPU-accelerated string column conversions that corrupted one or more string column values (e.g.: causing the character data across all rows in the string column to appear in one row or some similar type of row value expanding corruption).

To clarify, the problem that originally triggered this issue was 2^31 bytes per row was exceeded, not 2^31 bytes per batch (i.e.: the cudf limits). It is very, very unlikely any valid data is going to exceed the 2GB row limit. The problem that initially triggered the issue was pretty clearly a data corruption issue, as it was trying to build a single row's value, and one of the string columns had a >100MB value that when added to the current size of the single row caused it to exceed the 2GB limit (implying earlier columns had already pushed it crazy far in size). I suspect there's a problem with GPU-accelerated string column conversions that corrupted one or more string column values (e.g.: causing the character data across all rows in the string column to appear in one row or some similar type of row value expanding corruption).

Thanks @jlowe for the clarification.
Still, aren't we supposed to change this condition output.length <= 100000000) as String columns are being supported?
I believe I find a problem in the generated code that would potentially corrupt the data.
When the number of bytes remaining in the databuffer is less than the fixed width size of the row, then column meta could be written past the buffer end address.
The "done" condition in GpuRowToColumnarExec-L818-L820 is set as follows: https://github.com/amahussein/spark-rapids/blob/333a508700859802c8fe22935212f4a091dae38e/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuRowToColumnarExec.scala#L818-L820
         |        done = !(currentRow < numRowsEstimate &&
         |            dataOffset < dataLength &&
         |            input.hasNext());

With variable sized rows,

currentRow might be lower than numRowsEstimate and
dataOffset can be slightly less than dataLength.
Going through the next iteration, the generated code will be written into the buffer.
The bug is that the generated code checks for the variable data exceeding the buffer limit.

The fix is to change how done is being set.
         |        done = !(currentRow < numRowsEstimate &&
         |            (dataLength - dataOffset < estimatedRowSize) &&
         |            input.hasNext());

Still, aren't we supposed to change this condition output.length <= 100000000) as String columns are being supported?

I'm not sure why that check is there.  We don't support more than 2^31 bytes per row, but neither does Spark, as indicated by the BufferHolder error in the description of this issue.  Probably worth commenting in the code, but I don't see the need to explicitly check for some max number of columns, especially when the columns are variable sized (like strings, arrays, etc.)

Closing old bug against code that has since been reverted.


FactBase types of incoming and outgoing data

[problem]
get the raw output of the command. To process data inside a function
[decision]
add the ability to get raw data in str type

[problem]
sometimes it is necessary for a fact to return a string and not a list
[decision]
add the ability to return not only a list or dictionary but also a string

Hi @bronte2k7 - this should already be possible, the process method is passed the stdout lines but can return anything it wants; ie:
class StringFact(FactBase):
    command = 'some-command'
    default = six.text_type  # until py2 is dropped

    def process(self, output):
        return '\n'.join(output)

Hi @Fizzadar
I believed that the function works differently, and this caused me confusion. Thanks

I've now updated the facts API doc to make this more clear + include an example! https://github.com/Fizzadar/pyinfra/commit/e73711ad20a89b956622b8579a2b002463b1de16


Use env to find ssh path

Summary
Changed the call to /usr/bin/ssh in to /usr/bin/env ssh to make capistrano nixos compatible.
Nixos does not keep the binaries in /usr/bin.
Short checklist

[x] Did you run bundle exec rubocop -a to fix linter issues?
[ ] If relevant, did you create a test?  not applicable
[x] Did you confirm that the RSpec tests pass? - Didn't run the  bundle exec rake features ones, I got some problem with virtualbox on my system at the moment that would require rebuilding the kernel
[ ] If you are fixing a bug or introducing a new feature, did you add a CHANGELOG entry?  -- is the changelog still used? It's empty

LGTM @JoeDupuis, thanks :tada:
@mattbrictson any objections? This seems like a no-brainer

I just did some digging to check how ubiquitous env really is, and learned a little bit.
Making a long story short, it's everywhere part of POSIX.1:

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/env.html

I had never made the link between env (to print print env vars), and /usr/bin/env to find a utility on the PATH were the same, but seems like they are. :tada:

Matt approved of the concept in a PR I threw out that does the same thing.

Thanks everyone!

If you are fixing a bug or introducing a new feature, did you add a CHANGELOG entry? -- is the changelog still used? It's empty

Good point. I have opened a PR to fix this: #2054


1040 enhance nunaliit to provide an atlas export

Create a tarball of entire atlas data for download by a user. To use the features

Setup an export user name in install.properties file like export.complete.user=myexportuser
Setup an export password in sensitive.properties file like export.complete.password=myexportpass
To create a new export use HTTP POST to /servlet/export/complete/export with basic auth as configured (example: curl -X POST http://myexportuser:myexportpass@localhost:8080/servlet/export/complete/export
To list available downloads use HTTP GET to /servlet/export/complete (example: curl http://localhost:8080/servlet/export/complete)
To download an export use HTTP GET to /servlet/export/complete/<file_name> with basic auth (example: http://localhost:8080/servlet/export/complete/full_export_2024-10-16-14_33-51.tar.gz)

write/prepare documentation on how to use

https://github.com/GCRC/nunaliit/wiki/Full-Atlas-Export


Useful to have the priority and lastmod for a sitemap

Just had a play with the library, It's awesome and works very well for large sitemaps.
I have an improvment idea, it would be nice to have the priority and lastmod of the sitemap also, as users may want to only crawl pages that have changed or off a certain priority. (High priority = daily crawls, low priority = weekly crawls).
I was hoping to use this to read a sitemap, insert it into a db (url, priority, lastmod) - so I don't have to keep hitting the server. Then I would continue to process it as required by my application. So having it return an array (or an option of returning the full array or url only) would be really useful.
+1 ;)

Awesome! Thank You!

Ah, I just saw, that I only included the priority, not the lastmod. :/ Will also check this.

Ha! was just about to play with it! no probs! thanks for the update!

Now you have both information! Hope it helps you!


Prisma + Turso: getting "LibsqlError: SYNC_NOT_SUPPORTED: sync not supported in http mode"

I've followed the guide to setup turso here https://www.prisma.io/docs/orm/overview/databases/turso, but there seems to be issues with synchronizing my local dev.db-file, specifically this part.
When I run my server, and try to fetch data using the Prisma api, I get the error "LibsqlError: SYNC_NOT_SUPPORTED: sync not supported in http mode". If I remove await libsql.sync(), I dont get the error, and at an earlier point everything seemed to be working (I but I guess without changes stored locally? I'm not 100% sure about what role the local dev.db-file plays).

I moved the issue to https://github.com/tursodatabase/libsql


is libcurl exporting a function to parse the command line params?

It would be useful to have a way to copy a request from the browser as curl and have a way to execute it from elixir.
Or even better to parse it first - have a way to edit the params and then send. Currently I use postman for this but livebook would be much better.

Hey, I am not aware of libcurl itself exposing the ability to parse the command line params. I thought your idea of using curl commands in livebooks was really interesting so I created a new library called kino_curl which adds a new cURL Command cell that allows using curl commands in Livebook by parsing and converting the command into Elixir code.
Here is a gif demonstrating how it works:

You should be able to convert the smart cell into code in order to edit the parameters before sending as well.

Thanks for looking into this. This is awesome.
You can make a demo that copies the request from the browser and then pipe the result to https://github.com/spawnfest/the-arborists or a table then it will show it's full potential.


Process QuickLinks tables on category pages

data/Spain/Spain.html is a category page that has a QuickLinks table at the top of it. One page linked to in the QuickLinks is Spain_Measuring.html which isn't linked from anywhere else. That page is not converted because we don't process QuickLinks tables on category pages.

Fixed in #504


fix: add js for icon property in button group

Description
Adds JS for icon property in the style section of button group
Fixes #11142
Type of change

Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Manual (verified that the icon property within the style panel of button widget has a JS icon and is JS configured)

Checklist:
Dev activity

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[ ] PR is being merged under a feature flag

QA activity:

[x] Test plan has been approved by relevant developers
[x] Test plan has been peer reviewed by QA
[ ] Cypress test cases have been added and approved by either SDET or manual QA
[ ] Organized project review call with relevant stakeholders after Round 1/2 of QA
[ ] Added Test Plan Approved label after reveiwing all Cypress test

/ok-to-test sha=7405c8b

Tested this PR and LGTM

Added JS for icon property in Button Group widget

@iamrkcheers @Tooluloope would this need any kinda doc change, do we should what properties are JS convertible?

https://docs.appsmith.com/reference/widgets/button-group
I didn't see JS convertible on properties in the current doc @somangshu

Adding test coverage for js editor option for Icon property in Button Group widget
https://github.com/appsmithorg/TestSmith/issues/2094


chunkwm & skhd stuck in Privacy settings after uninstalling

I've uninstalled chunkwm and skhd but they both still appears in the Security and Privacy settings. How can I remove these?

If you look at the whole settings window (and not just your zoomed in screenshot), you'll see a big minus button for removing entries from that list. Select the entry you wish to remove and then remove it by pressing said button. This design language using plus and minus labelled buttons for adding and removing entries from list is spread all throughout macOS.

@dominiklohmann thanks! Other uninstalled applications disappeared when toggling the checkbox and these didn't.


Object Mapping Table not shown up for a table that has special char in its column name

Object Mapping Table not shown up for a table that has special char in its name when user expands db node and select table node immediately.

Fix has been merged.


Added Solution to FizzBuzz

Thank you for your PR!
Unfortunately your solution isn't quite right. It's not printing the number if the current number is neither Fizz nor Buzz.
Also, the goal of this repository is to find creative/abstract new solutions to FizzBuzz.
See The top 5 samples here.
Your solution is a rather "standard" approach on solving it.
Thanks anyway!
And feel free to submit a new PR :)


Brand new version

[x] update angular-cli env
[ ] remove icons dependancy
[ ] make tests great again
[ ] implement plugins (https://github.com/stevermeister/ngx-wig/issues/18)

https://github.com/stevermeister/ngx-wig/tree/brand-new


How can I get all properties that apply to a given object?

Given a schema and an object to validate, instead of (or in addition to) validating, I want to get all properties that apply to (or are valid for) the given object (i.e. all fixed properties and all conditional properties).
Can this be done?
For example, given the following data I want to get the following properties:
street_address, country, and the postal_code with the pattern "^\d{5}$|^\d{5}-\d{4}$".
const schema = {
  type: "object",
  properties: {
    street_address: {
      type: "string",
    },
    country: {
      enum: ["United States of America", "Canada", "Netherlands"],
    },
  },
  allOf: [
    {
      if: {
        properties: { country: { const: "United States of America" } },
      },
      then: {
        properties: {
          postal_code: { type: "string", pattern: "^\\d{5}$|^\\d{5}-\\d{4}$" },
        },
      },
    },
    {
      if: {
        properties: { country: { const: "Canada" } },
      },
      then: {
        properties: {
          postal_code: {
            type: "string",
            pattern: "[A-Z]\\d[A-Z] \\d[A-Z]\\d",
          },
        },
      },
    },
    {
      if: {
        properties: { country: { const: "Netherlands" } },
      },
      then: {
        properties: {
          postal_code: { type: "string", pattern: "\\d{4} [A-Z]{2}" },
        },
      },
    },
  ],
};
const data = {
  country: "United States of America",
  postal_code: "12345-999",
};

You can do something like hooks with custom keywords that would be called as the schema traverses through the properties. You cannot in general case define valid properties though - they may depend on each other.

Yes, I tried a custom keyword hook, but that only works for schemas that I control or embellish.

Another option would be to use json-schema-traverse package that calls provided pre/post hooks on each subschema - Ajv uses it internally.
I understand that the idea is to support subschema-level (or even per-keyword) pre/post hooks in ajv, but the situation when you can neither modify the schema nor iterate it independently using json-schema-traverse seems relatively niche - that's the reason I am reluctant to add it.
If it were to be added, it would be some instance-level option(s) that would add validation code to call supplied hooks. It would be good to understand the use case when neither of the alternatives is sufficient and some user support for this feature before it is added.

Another option would be to use json-schema-traverse package that calls provided pre/post hooks on each subschema - Ajv uses it internally.
I understand that the idea is to support subschema-level (or even per-keyword) pre/post hooks in ajv, but the situation when you can neither modify the schema nor iterate it independently using json-schema-traverse seems relatively niche - that's the reason I am reluctant to add it.
If it were to be added, it would be some instance-level option(s) that would add validation code to call supplied hooks. It would be good to understand the use case when neither of the alternatives is sufficient and some user support for this feature before it is added.


Increase load peer timeout

Temporary workaround for a rare issue where many peers (> hundreds) cause a timeout for the GenServer.call function, preventing the wireguard_public_key from being set properly.
This will be removed in 0.8, so only an increased timeout is needed for now.

Pull Request Test Coverage Report for Build 340583dfb36b6b20a5371a6d4e83f718400ee8e2-PR-1396

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 69.358%

Totals

Change from base Build 9b086dc3de9d9cc993c19610f8d6c47b6089d50a:
0.0%

Covered Lines:
1426

Relevant Lines:
2056

💛  - Coveralls


Add a XML sitemap to our app

This will be implemented when we have our app already hosted.

I don't think it is necessary, our website will be rather small.
It might be worth it if it helps with SEO somehow.

We can make XML sitemap, even it is small.


Metainfo2.txt error

Hello
I tried to install with a sdcard but I get metainfo2.txt error.
I read somewhere in a post that the same thing happened to others.
What to do? Did I do it wrong or what?
Thanks for doing a good job. I hope to succeed in installing the tool.

Same case with me.

Same case with me.

duplicate of #54  and #4
It's nothing you did.. just a strange thing happening that I cannot reproduce myself.

duplicate of #54  and #4
It's nothing you did.. just a strange thing happening that I cannot reproduce myself.

What else can I try?

What else can I try?

Same error to me!!...

Same error to me!!...


Analyzing Sample write up issue

Under Emperor Plot, some of the bullet points must be fixed to match the rest of the help doc.

This was fixed and deployed in the release-candidate, if it needs more changes please open a new issue.


Add negative lookahead

#262

Coverage increased (+3.02%) to 84.256% when pulling 15f5ea0f519b566e85d82a264797a5ef6941056e on gtors:not_macros into ec2c5fe5f400810133d352ec7bbeddb8ce1aca87 on Geal:master.

Hello,
could you add some unit tests with this?

Done

Coverage increased (+3.2%) to 84.282% when pulling ab3eb1978e5a7127bbda538ad267745d476c2927 on gtors:not_macros into 6937d8ee5e086b89bccff90a5acb09713f496eca on Geal:master.

Thanks!


feat(intersection)!: disable the exception behavior in the private areas

Description
hot fix of https://github.com/autowarefoundation/autoware.universe/pull/5229
Related links
https://tier4.atlassian.net/browse/RT0-29269
Tests performed
psim test and tier4 internal evaluator has passed for pilot-auto
Also x2 test in psim was performed as the following

Notes for reviewers
BEFORE merging this PR,
https://github.com/tier4/autoware_launch.x2/pull/476
have to be merged.
Interface changes
A new parameter is required
Effects on system behavior
No longer behave exceptionally in private area.
Pre-review checklist for the PR author
The PR author must check the checkboxes below when creating the PR.

[x] I've confirmed the contribution guidelines.
[x] The PR follows the pull request guidelines.

In-review checklist for the PR reviewers
The PR reviewers must check the checkboxes below before approval.

[ ] The PR follows the pull request guidelines.
[ ] The PR has been properly tested.
[ ] The PR has been reviewed by the code owners.

Post-review checklist for the PR author
The PR author must check the checkboxes below before merging.

[ ] There are no open discussions or they are tracked via tickets.
[ ] The PR is ready for merge.

After all checkboxes are checked, anyone who has write access can merge the PR.

別PRで一気に取り込んだのでクローズさせていただきます


Bugs abuse

When setting discounts for certain groups, append the file sh_config so here: if ply:IsUserGroup('superadmin') then return math.Round(item.Price * 0.15) end Poluchaetsya this situation, if the purchase goes off, but when you bought the item that you can sell, then there vystavlyaetsya price higher discount, for example, the price is 1000 for the original. you have a 25% discount you're buying, but sell from the original price 1000, 75% of it. And if the above discount above, we and so you can do endlessly, you can fix it, I would not disable the selling things.

Change the sell price calculation to follow alongside the purchase price.

Yeah, I thought about it, I understand, but I would not change the percentage of the sale.

Just write another return for the rank?

Do you have to do just the sales price, was the price that the person with the discount, for example from 1 person, no discount price is 100, and the discount on items and they will cost 30, but the sale of 70, it turns baws, I wanted that, the system would automatically configure the sales price from the discount in the store, that this was not. And it turns out, it will be necessary to put the sales price for all is very little.

It's made so you can optionally change things for some ranks, and possibly modify the selling price.  Just multiply the default return by the discount percent.

thanks man


KAFKA-15853: Move KRAFT configs out of KafkaConfig

Move all Kraft configs/docs/defaults into KRaftConfigs.
Note: We have already RaftConfig but it seems to contain limited amount of configs that only configure controller raft and shouldn't include configs shared by both broker/controller in KRAFT mode or only for broker in KRAFT mode.

RaftConfig.QUORUM_VOTERS_CONFIG is the only exception to this rule and I am not sure if it should be defined in KRaftConfigs and shared by RaftConfig or if it should be RaftConfig. Originally this config had an alias in KafkaConfig but most places where it referred to KafkaConfig.QuorumVotersProp has access to RaftConfig.QUORUM_VOTERS_CONFIG.
I choice to use  RaftConfig.QUORUM_VOTERS_CONFIG everywhere as the prefix for defining this config is defined in RaftConfig if people prefer that we move this to KRaftConfigs (this might be tricky and we may need to move KRaftConfigs int server-common instead) or revert this commit and keep the original approach from KafkaConfig please let me know.

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@chia7712 & @mimaison can you please have a look into this PR?

Note: We have already RaftConfig but it seems to contain limited amount of configs that only configure controller raft and shouldn't include configs shared by both broker/controller in KRAFT mode or only for broker in KRAFT mode.

What about moving KRaftConfig to raft module? and we can moving all configs from RaftConfig to KRaftConfig.  RaftConfig can be a POJO.

We can move it to raft module but

I am concerned that this will create other circle dependencies later when we move the raft package and KafkaRaftServer out of core
I don't think we should merge them with RaftConfig as RaftConfig should be separate as it is only for the quorum raft and not KRAFT mode which are a bit different in my option.
Most of the configs in KRaftConfigs are used by SharedServer, KafkaRaftServer or BrokerLifeCycle.

I think we either keep the pr as proposed or maybe we can have in server-common module the following

MetadataLogConfigs for anything with prefix metadata
KafkaRaftServerConfigs for the rest anything
Keep RaftServer separated but move it to server-common

Personally I think the first option is easier to navigate as we know where all the config used by Kafka Raft server (maybe renaming this to KafkaRaftServerConfigs instead of KRaftConfigs would be better) are and where all the config used by Quorum Raft. WDYT?

Personally I think the first option (which keep the pr as it is) is easier to navigate as we know where all the config used by Kafka Raft server (maybe renaming this to KafkaRaftServerConfigs instead of KRaftConfigs would be better) are and where all the config used by Quorum Raft. WDYT?

I'm ok with it. BTW, KafkaRaftServerConfigs is a bit verbose to me.

I am concerned that this will create other circle dependencies later when we move the raft package and KafkaRaftServer out of core

just curios. Why moving to raft module can cause circle dependencies? It seems raft module depends on only clients and server-common. I feel KafkaRaftServer won't be in either clients or server-common in the future.

just curios. Why moving to raft module can cause circle dependencies?
Currently raft doesn't depend of core or server but I just fear that mixing quorum raft and KRaft mode related classes in same module might led to situations where we hit this issue when we start to move things more and more out of core. One example I have in mind is KafkaRaftManager

KafkaConfig depends on RaftConfig for raft configs
KafkaRaftManager depend on KafkaConfig and RaftConfig and it is used by BrokerServer and ControllerServer
If we start to move related kraft classes to raft module like KafkaRaftManager this will be tricky as now org.apache.kafka.raft.KafkaRaftManager needs org.apache.kafka.server.config.KafkaConfig,  org.apache.kafka.server.config.KafkaConfig needs org.apache.kafka.raft.RaftConfig and org.apache.kafka.server.BrokerServer needs org.apache.kafka.raft.KafkaRaftManager

Also one other thing to notice is at the moment RaftConfig signature accept AbstractConfig instead of KafkaConfig and it redefine the getters for all raft configs instead of using KafkaConfig.quorum* methods which at the moment aren't used at all except KafkaConfig.quorumRequestTimeoutMs.

If we start to move related kraft classes to raft module like KafkaRaftManager this will be tricky as now org.apache.kafka.raft.KafkaRaftManager needs org.apache.kafka.server.config.KafkaConfig, org.apache.kafka.server.config.KafkaConfig needs org.apache.kafka.raft.RaftConfig and org.apache.kafka.server.BrokerServer needs org.apache.kafka.raft.KafkaRaftManager

org.apache.kafka.server.config.KafkaConfig -> this means KafkaConfig will be moved to server module in the future? If so, is it weird to make raft module depend on server module? Please correct me if I misunderstand anything.

If we start to move related kraft classes to raft module like KafkaRaftManager this will be tricky as now org.apache.kafka.raft.KafkaRaftManager needs org.apache.kafka.server.config.KafkaConfig, org.apache.kafka.server.config.KafkaConfig needs org.apache.kafka.raft.RaftConfig and org.apache.kafka.server.BrokerServer needs org.apache.kafka.raft.KafkaRaftManager

org.apache.kafka.server.config.KafkaConfig -> this means KafkaConfig will be moved to server module in the future? If so, is it weird to make raft module depend on server module? Please correct me if I misunderstand anything.

KafkaConfig will move out of core and into server, however, for the time being I don't plan to make raft depend on server for now as far as I can see the only case I might need to do so is to use the getters from KafkaConfig but they are only 5 getters as far as I can see so don't worth it. I am planning to just remove these getters from KafkaConfig.

KafkaConfig will move out of core and into server, however, for the time being I don't plan to make raft depend on server for now as far as I can see the only case I might need to do so is to use the getters from KafkaConfig but they are only 5 getters as far as I can see so don't worth it. I am planning to just remove these getters from KafkaConfig.

It seems org.apache.kafka.raft.KafkaRaftManager needs org.apache.kafka.server.config.KafkaConfig won't happen if the 5 getters are moved, right?

I don't think we should merge them with RaftConfig as RaftConfig should be separate as it is only for the quorum raft and not KRAFT mode which are a bit different in my option.

Maybe we should rename RaftConfig to QuorumConfig as all configs in QuorumConfig have prefix QUORUM_

It seems org.apache.kafka.raft.KafkaRaftManager needs org.apache.kafka.server.config.KafkaConfig won't happen if the 5 getters are moved, right?

Sorry I maybe wan't clear org.apache.kafka.raft.KafkaRaftManage this was an example for depend circle if we start to move quorum raft and Kraft code into raft module.  KafkaRaftManage at the moment is in core under kafka.raft package and not raft module and it need KafkaConfig to initialise org.apache.kafka.raft.RaftConfig (which is waiting for AbstractConfig and not KafkaConfig and to access the following getters nodeId, ProcessRoles, metadataLogDir, quorumRequestTimeoutMs, controllerListenerNames, saslMechanismControllerProtocol, saslInterBrokerHandshakeRequestEnable.
In the future if we decided to move kafka.raft.KafkaRaftManage it is most likely going to be moved into server and not raft

Maybe we should rename RaftConfig to QuorumConfig as all configs in QuorumConfig have prefix QUORUM_

This is good point, maybe this can be a followup PR

Maybe we should rename RaftConfig to QuorumConfig as all configs in QuorumConfig have prefix QUORUM_

This is good point, maybe this can be a followup PR

After renaming RaftConfig to QuorumConfig,  KRaftConfigs created by this PR is a acceptable naming I think

Maybe we should rename RaftConfig to QuorumConfig as all configs in QuorumConfig have prefix QUORUM_

This is good point, maybe this can be a followup PR

After renaming RaftConfig to QuorumConfig, KRaftConfigs created by this PR is a acceptable naming I think

Raised here https://github.com/apache/kafka/pull/15797


[Bug report]: Error when trying to create a profile with a once used name

Platform
Desktop
Version
1.4.0-beta-1
Description
After creating a profile, renaming it from P2 to P1 and then logging out to create a new profile (only possible after logging out once after creation of P2, see #2643), when attempting to create a new profile P2, in the end of the setup process the Error "A wallet with this name already exists." is shown.
(Also should say "profile" instead of "wallet")

Expected behaviour
Creation of P2 should be possible because existing P2 was renamed to P1 before.
Actual behaviour
Error message
Can the issue reliably be reproduced?
Yes
Steps to reproduce the issue

Create a new profile P2
Log out (necessary because of #2643)
Log into P2 again
Rename P2 to P1
Log out
Create a new profile P2 and go trough the setup process

Operating System
Mac
Diagnostics
Firefly v1.4.0-beta-1
Platform: macOS
Platform Version: 12.3
Platform Architecture: x64
CPU Count: 12
Total Memory: 32768.0 MB
Free Memory: 3446.1 MB

Developer Profile
Yes
Profile Type
Software (Stronghold)
Staking
No response
Crash Status
None of the above
Error Messages
No response
Error Logs
No response
Duplicate declaration

[X] I have searched the issues tracker this issue and there is none

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Investigating: seems that this is the case until you restart FF, and the error doesn't appear until the end screen and not on the profile name screen. we should fix this.


Trivy failed to download ARM architecture node despite running on ARM GHA runner

Behaviour
Steps to reproduce this issue

Build a arm architecture only docker image and push to ECR
Use trivy to download and scan the image

Expected behaviour

Trivy should perform the requested security scan

Actual behaviour

Trivy tried to pull down the amd64 image even tho its running on an ARM runner.

2023-01-11T03:31:03.726Z	FATAL	image scan error: scan error: unable to initialize a scanner: unable to initialize a docker scanner: 4 errors occurred:
	* unable to inspect the image (***.dkr.ecr.***.amazonaws.com/image:tag): Error: No such image: ***.dkr.ecr.***.amazonaws.com/image:tag
	* unable to initialize Podman client: no podman socket found: stat podman/podman.sock: no such file or directory
	* containerd socket not found: /run/containerd/containerd.sock
	* no child with platform {Architecture:amd64 OS:linux OSVersion: OSFeatures:[] Variant: Features:[]} in index ***.dkr.ecr.***.amazonaws.com/image:tag

Configuration

Repository URL (if public):
Build URL (if public):

runs:
  using: "composite"
  steps:
    - uses: actions/checkout@v3
    - uses: docker/setup-buildx-action@v2
    - uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: '${{ inputs.AWS_ACCESS_KEY_ID }}'
        aws-secret-access-key: '${{ inputs.AWS_SECRET_ACCESS_KEY }}'
        aws-region: '${{ inputs.AWS_REGION }}'
    - id: login-ecr
      uses: aws-actions/amazon-ecr-login@v1
    - name: build and push image
      uses: docker/build-push-action@v3
      with:
        context: .
        file: ${{ inputs.dockerfile }}.Dockerfile
        push: true
        platforms: ${{ inputs.platforms }}
        # cache-from: type=gha
        # cache-to: type=gha,mode=max
        tags: ${{ steps.login-ecr.outputs.registry }}/${{ inputs.registry }}:${{ inputs.tag }}
    - uses<EMAIL_ADDRESS>      if: ${{ steps.image_meta.outputs.IMAGE_META == ''}}
      with:
        image-ref: '${{ inputs.AWS_ACCOUNT_ID }}.dkr.ecr.${{ inputs.AWS_REGION }}.amazonaws.com/${{ inputs.registry }}:${{ inputs.tag }}'
        format: 'table'
        exit-code: '0'
        ignore-unfixed: true
        timeout: '10m0s'
        vuln-type: 'os,library'
        severity: 'CRITICAL'

I was able to solve this issue in my workflow by specifying the platform in trivy.yaml file.
Example:

In trivy.yaml:
image:
  platform: linux/arm64

In GitHub Workflow file:
 - uses<EMAIL_ADDRESS>   with:
     image-ref: '${{ inputs.AWS_ACCOUNT_ID }}.dkr.ecr.${{ inputs.AWS_REGION }}.amazonaws.com/${{ inputs.registry }}:${{ inputs.tag }}'
     trivy-config: trivy.yaml

Found a better solution than trivy.yaml config file.
See related comment: https://github.com/aquasecurity/trivy-action/issues/279#issuecomment-1925050674


I want to uninstall this bloatware.

Dev Home version
0.701.323.0
Windows build number
10.0.22631.2715
Other software
OS Build Version: 10.0.22631.2715.amd64fre.ni_release.220506-1250
.NET Version: .NET 6.0.24
Steps to reproduce the bug
I want to uninstall DevHome, but Microsoft has taken away my right to remove this unwanted application on my own home computer. This is outrageous! Windows 11 is not freeware to be able to do such things to customers. You don't respect anyone. There is no law or court on you. You are like a cartel and mafia that first makes the public dependent on your services and then you exploit them as ruthlessly as criminals.
Expected result
I expect to be able to uninstall any application on my home computer.
Actual result
Microsoft rules my home computer and does what it wants without asking my opinion. Microsoft has set itself up as the administrator of home computers and has the final word on everything.
Included System Information
No response
Included Extensions Information
No response

/dup https://github.com/microsoft/devhome/issues/1615


replace outdated link in readme

Thanks, I've noticed it but been too lazy to do it, will merge it now.


Hotbar items drag-n-drop

Hotbar items should be able to be restacked via drag-n-drop.

same as #2490?


Add Either.liftPredicate

Type

[ ] Refactor
[x] Feature
[ ] Bug Fix
[ ] Optimization
[ ] Documentation Update

Description

Add Either.liftPredicate. In my current project we were missing it so we created a custom helper for it, but I found out in the Discord that other people would like it to be implemented (see thread here). So I figured it could be an opportunity to contribute here!
I took inspiration from Option.fromPredicate but I went a bit further and made Either.liftPredicate dual
Related

N/A

Thanks for the review @gcanti! I've pushed some more commits to update what you suggested. I just had a typing issue in my tests, but fixed it by adding a NoInfer where it was needed!


Valid pages should include anything Rails handles

Instead of requiring an extension like .html.haml, allow .haml. Look
up what handlers exist within the application using
ActionView::template.template_handler_extensions.
Resolves https://github.com/thoughtbot/high_voltage/issues/211

@dgalarza I'm not sure that I understand.  What I meant to ask was: did my PR #238 not solve the issue?  It looks like #250 did no more to solve the issue than #238 (and, in fact, did not solve all cases, as I commented there).

The specific change we wanted to avoid losing was the removal of the Page class as it was removed in #238.


Rotativa doesn't work when published to Windows Azure.

I have a web app developed in Asp.net mvc 5 and I used rotativa for the pdf, It seems that rotativa doesn't work when published to Windows Azure but it works on my local machine.
Thanks.

It does work on Azure cloud service or VM. it doesn't work on Azure WebSites due to restrictions on executing external processes.

I see. Thanks.

I developed a service with the same Rotativa usage but you can use without having to worry about installing executables and having special permissions, so it will work with azure websites or shared hosting accounts. It's online at http://rotativahq.com


generator release-v2: Add the function of automatically publishing pull request

@tadelesh have you guys considered reusing some of the existing scripts and pipeline templates for create a pull request? https://github.com/Azure/azure-sdk-for-go/blob/main/eng/common/pipelines/templates/steps/create-pull-request.yml It would be ideal to share this functionality instead of implementing it again.

@tadelesh have you guys considered reusing some of the existing scripts and pipeline templates for create a pull request? https://github.com/Azure/azure-sdk-for-go/blob/main/eng/common/pipelines/templates/steps/create-pull-request.yml It would be ideal to share this functionality instead of implementing it again.

The tool is used locally. Let us try to reuse the pw script.

@tadelesh have you guys considered reusing some of the existing scripts and pipeline templates for create a pull request? https://github.com/Azure/azure-sdk-for-go/blob/main/eng/common/pipelines/templates/steps/create-pull-request.yml It would be ideal to share this functionality instead of implementing it again.

We use this tool to generate code locally, it is not very convenient to use pipeline yml to create pull request, which uses 'git -c user.name="azure-sdk" -c user.email="azuresdk@microsoft .com " commit -m', which failed locally

When you use this tool locally what credentials do you use? Also the pipeline yml is just a wrapper around a PS script which could be used locally as well.

Handle multi-package releasee:

If release request's tag has sub package name like package-xxx-date, we need to release the specific sub package.
If RP has multiple packages like mysql, we need to release both packages.


[Chore] - add wait for admin API in e2e tests.

📈 Kaizen
What
Introduce a check for Admin API availability in e2e integration tests.
Why
Tests fail when we run the integration-tests create-data command before the admin api is fully started.
Checklist
PR Structure

[x] This PR has a reasonably narrow scope (if not, break it down into smaller PRs).
[x] This PR title and description are clear enough for anyone to review it.
[x] This PR does not mix refactoring changes with feature changes (split into two PRs otherwise).

Thoroughness

[x] This PR adds tests for the new functionality or fixes.
[x] This PR contains the link to the Jira ticket it addresses.

Configs and Secrets

[x] No new CONFIG variables are required -OR- the new required ones were added to the helmchart's values.yaml file.
[x] No new CONFIG variables are required -OR- the new required ones were added to the deployments (pr-preview, dev, demo, prd).
[x] No new SECRETS variables are required -OR- the new required ones were mentioned in the helmchart's values.yaml file.
[x] No new SECRETS variables are required -OR- the new required ones were added to the deployments (pr-preview secrets, dev secrets, demo secrets, prd secrets).

Release

[x] This is not a breaking change.
[x] This is ready for production.. If your PR is not ready for production, please consider opening additional complementary PRs using this one as the base. Only merge this into develop or main after it's ready for production!

Deployment

[ ] Does the deployment work after merging?

stellar-disbursement-platform-backend-preview is available here:SDP:      https://sdp-backend-pr263.previews.kube001.services.stellar-ops.com/healthAP:       https://sdp-ap-pr263.previews.kube001.services.stellar-ops.com/healthFrontend: https://sdp-backend-dashboard-pr263.previews.kube001.services.stellar-ops.com

stellar-disbursement-platform-backend-preview is available here:SDP:      https://sdp-backend-pr263.previews.kube001.services.stellar-ops.com/healthAP:       https://sdp-ap-pr263.previews.kube001.services.stellar-ops.com/healthFrontend: https://sdp-backend-dashboard-pr263.previews.kube001.services.stellar-ops.com

stellar-disbursement-platform-backend-preview is available here:SDP:      https://sdp-backend-pr263.previews.kube001.services.stellar-ops.com/healthAP:       https://sdp-ap-pr263.previews.kube001.services.stellar-ops.com/healthFrontend: https://sdp-backend-dashboard-pr263.previews.kube001.services.stellar-ops.com

stellar-disbursement-platform-backend-preview is available here:SDP:      https://sdp-backend-pr263.previews.kube001.services.stellar-ops.com/healthAP:       https://sdp-ap-pr263.previews.kube001.services.stellar-ops.com/healthFrontend: https://sdp-backend-dashboard-pr263.previews.kube001.services.stellar-ops.com

stellar-disbursement-platform-backend-preview is available here:SDP:      https://sdp-backend-pr263.previews.kube001.services.stellar-ops.com/healthAP:       https://sdp-ap-pr263.previews.kube001.services.stellar-ops.com/healthFrontend: https://sdp-backend-dashboard-pr263.previews.kube001.services.stellar-ops.com


fix: enable ↦ in server mode

Apparently we didn't enable the pp.unicode.fun in server mode, so you got inconsistent output depending on whether you run the lean file on the command-line (↦) or in the editor (=>).  This change makes the options uniform.

Maybe we should only enable warningsAsErrors in command-line mode though because it changes the color of the squiggles.

I would be happy to not have warningsAsErrors in server mode. If you're hacking on two files at once it is just annoying to have to keep the first one clean.

@semorrison If you Refresh Dependencies in the second file, the first file will be recompiled "in command-line mode", so it would still fail on a warning

bors merge

CI error was an SSL error in lake cache , so most likely completely unrelated.  I'll take the bors merge as a d+:
bors r+


syntax error on lines 21, 23

I'm getting the following:
./multipart.sh: line 21: *(1-1)/20: syntax error: operand expected (error token is "*(1-1)/20")
./multipart.sh: line 23: (*1/20)-1: syntax error: operand expected (error token is "*1/20)-1")
curl --silent --range - -o temp.part1 http://download.gimp.org/pub/gimp/v2.8/osx/gimp-2.8.14.dmg &

Any help would be appreciated!
Edit: it appears the size is getting logged as an empty string.

This will happen if the url can't be resolved. e.g. size is = not set

I am using aria2 now for multi-part downloads. Hence won't be looking into debugging issues on this project anymore.

@rbalicki2 Try running this command separately and see if it is able to pull out the size correctly
https://github.com/paramaggarwal/multipart/blob/c54cce53c6aace93b0da62ff8e9deeba94c13e11/multipart.sh#L14