adding anchors to make the ensure_resource behaive

Adding anchors to contain the resources within the redis module. I posted on the puppet mailing list about this issue.
Happy to do a different solution if you don't like anchors.
-Shawn

Alright, looks good.
Thanks for contributing!


Part 3 Deliverables 10 : Tool Use on GitHub

Regular and consistent use of GitHub by all team members to share files for the project deliverables, to effectively track issues, and to manage tasks.

github tool use is being done


The precision difference between tensorflow and numpy when using tf.reduce_mean and np.mean

### System information
- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**: Yes
- **TensorFlow installed from (source or binary)**:binary
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:
== cat /etc/issue ===============================================
Linux quad6 4.4.0-83-generic #106-Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
VERSION="16.04.1 LTS (Xenial Xerus)"
VERSION_ID="16.04"

== are we in docker =============================================
No

== compiler =====================================================
c++ (Ubuntu 5.4.1-2ubuntu1~16.04) 5.4.1 20160904
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

== uname -a =====================================================
Linux quad6 4.4.0-83-generic #106-Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

== check pips ===================================================
numpy (1.13.1)
protobuf (3.3.0)
tensorflow (1.2.1)
tensorflow-fold (0.0.1)
tensorflow-gpu (1.2.1)
tensorflow-tensorboard (0.1.4)

== check for virtualenv =========================================
True

== tensorflow import ============================================
tf.VERSION = 1.2.1
tf.GIT_VERSION = v1.2.0-5-g435cdfc
tf.COMPILER_VERSION = v1.2.0-5-g435cdfc
Sanity check: array([1], dtype=int32)

== env ==========================================================
LD_LIBRARY_PATH /usr/local/cuda-8.0/lib64:
DYLD_LIBRARY_PATH is unset

== nvidia-smi ===================================================
Fri Aug 18 16:14:35 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.66                 Driver Version: 375.66                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN Xp            Off  | 0000:05:00.0     Off |                  N/A |
| 28%   51C    P0    67W / 250W |      0MiB / 12188MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  TITAN Xp            Off  | 0000:06:00.0     Off |                  N/A |
| 31%   55C    P0    66W / 250W |      0MiB / 12189MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  TITAN Xp            Off  | 0000:09:00.0     Off |                  N/A |
| 54%   84C    P2   130W / 250W |  11655MiB / 12189MiB |     52%      Default |
+-------------------------------+----------------------+----------------------+
|   3  TITAN Xp            Off  | 0000:0A:00.0     Off |                  N/A |
| 23%   33C    P0    61W / 250W |      0MiB / 12189MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    2      5854    C   python3                                      11651MiB |
+-----------------------------------------------------------------------------+

== cuda libs  ===================================================
/usr/local/cuda-8.0/doc/man/man7/libcudart.so.7
/usr/local/cuda-8.0/doc/man/man7/libcudart.7
/usr/local/cuda-8.0/lib64/libcudart.so.8.0.27
/usr/local/cuda-8.0/lib64/libcudart_static.a

python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
v1.2.0-5-g435cdfc 1.2.1

Describe the problem
When using tf.reduce_mean, i found the behaviour between tensorflow and numpy was different when dtype was float32. The experiment shows that this maybe caused by the precision problem in tensorflow. When I changed the dtype to tf.float64, the problem fixed.
Or if I calculate the mean axis by axis, the problem also disappear.
However, if I use numpy array, the result was right no matter the dtype is  float32 or float64.
Is this a normal behaviour or will be fixed later?
The test code I used as following:
Source code / logs
In [61]: x = tf.constant(0.6931, shape=[32, 100, 79804], dtype=tf.float32)

In [62]: y = tf.reduce_mean(x)

In [63]: sess.run(y)
Out[63]: 0.26278782

In [64]: y = tf.reduce_mean(tf.reduce_mean(tf.reduce_mean(x, axis=2), axis=-1))

In [65]: sess.run(y)
Out[65]: 0.693142

In [66]: x = tf.constant(0.6931, shape=[32, 100, 79804], dtype=tf.float32)

In [67]: y = tf.reduce_mean(x)

In [68]: sess.run(y)
Out[68]: 0.26278782

In [69]: x = tf.constant(0.6931, shape=[32, 100, 79804], dtype=tf.float64)

In [70]: y = tf.reduce_mean(x)

In [71]: sess.run(y)
Out[71]: 0.6931000008030902

In [72]: y = tf.reduce_mean(tf.reduce_mean(tf.reduce_mean(x, axis=2), axis=-1))

In [73]: sess.run(y)
Out[73]: 0.69310000000034644

In [74]: xn = np.full((32, 100, 79084), 0.6931, dtype=np.float64)

In [75]: yn = np.mean(xn)

In [76]: yn
Out[76]: 0.69310000000032723

In [77]: xn = np.full((32, 100, 79084), 0.6931, dtype=np.float32)

In [78]: yn = np.mean(xn)

In [79]: yn = np.mean(xn)

In [80]: yn
Out[80]: 0.69321907

@ekelsen do you know why this issue is occurring?

This is on CPU?
There are likely two issues:

The CPU reductions use a numerically terrible algorithm for doing sums.  Summing into one or few accumulators - this leads to inaccurate sums for large N.  Breaking the sum in 3 parts ameliorates this problem.  This problem has come up multiple times before.

The mean reductions are also done by dividing the inputs by N and then summing which losing some accuracy in the mantissa (likely not the big issue here).

The GPU uses a more numerically stable sum algorithm, so (1) is not a problem there.  And the GPU mean calculation will soon also fix (2).  Using float64 is less of a cost on CPU than GPU, so if accurate reductions are important, the best workaround for now is probably to change the type before the reduction to float64 and after back to float32.

I did a test on GPU, the test code is as following:
import tensorflow as tf

with tf.device('/gpu:0'):
    x = tf.constant(0.6931, shape=[32, 100, 79804], dtype=tf.float32)
    y = tf.reduce_mean(x)
    sess = tf.Session()
    res = sess.run(y)
    print(res)
    x = tf.constant(0.6931, shape=[32, 100, 79804], dtype=tf.float64)
    y = tf.reduce_mean(x)
    res = sess.run(y)
    print(res)

The Result is just the same:
CUDA_VISIBLE_DEVICES=1 python3 test.py
2017-08-19 11:05:45.390445: I tensorflow/core/common_runtime/gpu/gpu_device.cc:940] Found device 0 with properties:
name: TITAN Xp
major: 6 minor: 1 memoryClockRate (GHz) 1.582
pciBusID 0000:09:00.0
Total memory: 11.90GiB
Free memory: 11.75GiB
2017-08-19 11:05:45.390474: I tensorflow/core/common_runtime/gpu/gpu_device.cc:961] DMA: 0
2017-08-19 11:05:45.390479: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0:   Y
2017-08-19 11:05:45.390495: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN Xp, pci bus id: 0000:09:00.0)
0.262788
0.693100000803

Is the test right?

Then my intuition that the problem was due to (1) was wrong - it looks it is due to (2).  This should already be fixed for the GPU internally.

@wenjunpku @ekelsen
I have met the same problems. tf.reduce_mean() gets different results from numpy.mean() on GPU (float32 and float16). But code with tensorflow + keras gets the right answer.
How to get the same results as the numpy.mean() with tensorflow?

you can try do the mean part by part, just like y=tf.reduce_mean(tf.reduce_mean(tf.reduce_mean(x, axis=2), axis=-1))
which fix my problem:)

I can confirm this is fixed internally on the GPU - it now returns .6931 as expected

Any updates on this?

This issue no longer occurs. If you have a different precision problem, file a new issue.


Document packages in list-features

has('packages') works and my :version shows +packages, but the feature is not mentioned in :help feature-list.
The N annotation for the normal build in +feature-list is possibly incorrect, I'm not sure how to determine that—I'm looking at these notes and I'm not sure what table is being referred to in the line about ex_version(). I am not seeing any FEAT_ defined in patch 7.4.1384 (f6fee0e2d4341) or subsequently, so perhaps the build version qualification isn't defined (yet?).
If I'm wrong and someone educates me, I'll update the patch.

Ches Martin wrote:

has('packages') works and my :version shows +packages, but the
feature is not mentioned in :help feature-list.
The N annotation for the normal build in +feature-list is possibly
incorrect, I'm not sure how to determine that—I'm looking at these
notes and I'm not sure what table is being referred to in the line
about ex_version(). I am not seeing any FEAT_ defined in patch
7.4.1384 (f6fee0e2d4341) or subsequently, so perhaps the build version
qualification isn't defined (yet?).
If I'm wrong and someone educates me, I'll update the patch.

That was indeed still missing, thanks.
--
TERRY GILLIAM PLAYED: PATSY (ARTHUR'S TRUSTY STEED), THE GREEN KNIGHT
SOOTHSAYER, BRIDGEKEEPER, SIR GAWAIN (THE FIRST TO BE
KILLED BY THE RABBIT)
"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD
/// Bram Moolenaar --<EMAIL_ADDRESS>-- http://www.Moolenaar.net   \
///        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \
\\  an exciting new programming language -- http://www.Zimbu.org        ///
\\            help me help AIDS victims -- http://ICCF-Holland.org    ///

included.


fix broken url

when the this post link in the help menu is clicked, it redirects to a broken url (the post has been deleted).

i fixed this with no pr, closing thanks


nodejs bigquery client fails for ssl error, if run behind corporate proxy

It seems bigquery client uses axios, which has issues connecting via corporate proxy. Is there a fix for this ?
I am using     "@google-cloud/bigquery": "^1.2.0",
Below is the error log.
[0]  I'm having the same problem

This seems like a possible duplicate of https://github.com/googleapis/nodejs-storage/issues/277, in that proxy problems all lead to an issue in one dependency. Please follow along there for some discussion that might be useful.

The corporate proxy that we use requires an HTTP tunnel. Here's the hack we implemented to get things working:
// this is a nodejs lib, to install do `npm install tunnel`
const tunnel = require('tunnel');

firewallTunnel = tunnel.httpsOverHttp({
  proxy: {
    host: 'some-firewall-url.com',
    port: '4153'
  }
});

module.exports = (httpModule) => {
  const original = httpModule.request;

  httpModule.request = function(options, callback) {
    return original({...options, agent: firewallTunnel}, callback);
  };
};

const axiosPrependConfig = {
  baseURL: 'https://www.googleapis.com',
  httpsAgent: firewallTunnel,
  proxy: false,
}

module.exports = (axios) => {
  const original = axios.post;
  axios.post = (url, data, config) => {
    const newUrl = "https://www.googleapis.com:443/oauth2/v4/token"
    const newConfig = {...config, ...axiosPrependConfig}
    if(process.env.AXIOS_LOGS)
      console.log( "axios-prepend: ", JSON.stringify({newUrl, data, newConfig}))
    return original(newUrl, data, newConfig)
  };
};

@stephenplusplus I know you were doing some work with proxies, did it resolve this issue?

This really should be fixed with the upgrade to teeny-request in common.

Haven't seen any activity here in a while, I'm going to assume that the move to teeny-request has resolved this. If anyone continues to experience similar issues, please let us know and we'll be happy to re-open. 😎


Concatenate datasets together

game_converter outputs a single HDF5 file. Running in parallel is a reasonable use-case to speed it up, but we don't have a way to combine the results yet.
states and actions may be concatenated. file_offsets only need to be slightly smarter. Shouldn't be very difficult

https://github.com/Rochester-NRT/AlphaGo.data/blob/master/scripts/h5_concatenation_snippet.py


regarding the difference between mask image channel and output layer channel

Hi Marko,
Thank you for sharing the code.
I have two questions while reading your code.

The input mask images have channel 1, while the output layer(or the prediction) from U-net is of channel 2. How does your code handle this kind of scenario?
After reading your code, I have not been very clear about the mechanism you use for handling this.

I am not very clear about the usages of this function

def preprocess(imgs):     imgs_p = np.ndarray((imgs.shape[0], imgs.shape[1], img_rows, img_cols), dtype=np.uint8)     for i in range(imgs.shape[0]):         imgs_p[i, 0] = cv2.resize(imgs[i, 0], (img_cols, img_rows), interpolation=cv2.INTER_CUBIC)     return imgs_p
I can see this code is used to transform the mask images, but I do not know why?
Thank you very much for your help.

Hi,

The input mask images have channel 1, while the output layer(or the prediction) from U-net is of channel

In the original U-net they used 2 output channels, with softmax activation and log-loss. Background and foreground are represented as 2 classes, thus 2 channels, 1 for each.

How does your code handle this kind of scenario?

In my solution, I didn't use softmax activation and log-loss, but sigmoid activation and Dice score as loss. Basically, authors of original U-net defined the problem as 2-class segmentation (background class and foreground class), while I defined it as binary segmentation (only foreground).

I am not very clear about the usages of this function

That preprocessing function just resizes the images to manageable size, as original images were to big  and thus memory consuming.

Thanks a lot.


Detect misuse in keys > decode command

Issue To Be Solved
Error messages output when using flow keys decode ... are unclear, leading to misuse of the command
context

(Optional): Suggest A Solution
@turbolent suggested we could detect if a given input doesn't start with what it should. For instance, in the case of a pem, if the input doesn't begin with -----BEGIN, it is malformed and the user can be informed that they are likely not using the correct input.
Personally, I think that detecting when an input looks like a file, but the --from-file flag wasn't given, the cli should ask if the user meant to include --from-file
(Optional): Context
Someone in the flow discord wasn't sure how to use this command, they were missing a flag

I think detecting input is very good, even we can check if it is filename.  Also probably stdin support would be good idea.


Pg 210 Compiler error when using code example

Hello!
I hope I'm being more helpful than annoying by opening these issues everywhere. Using the code on page 210
let html_link = get_link(&body["HtmlBody"].as_str().unwrap());
let text_link = get_link(&body["TextBody"].as_str().unwrap());

I get the error:
error[E0277]: the type str cannot be indexed by &str
--> tests/api/subscriptions.rs:128:31
|
128 |     let text_link = get_link(&body["TextBody"].as_str().unwrap());
|                               ^^^^^^^^^^^^^^^^ string indices are ranges of usize
|
= help: the trait SliceIndex<str> is not implemented for &str
= note: required because of the requirements on the impl of Index<&str> for str
I fixed this error by writing
    let html_body_index = body.find("HtmlBody").unwrap();
    let text_body_index =  body.find("TextBody").unwrap();
    let html_link = get_link(&body[html_body_index..text_body_index]);
    let text_link = get_link(&body[text_body_index..]);

Although I can't guarantee that it gets the intended logic correct, it does get me to the test failure message described on page 11.
Edit: I needed to change the string formats to use '' inside of "" but other than that the test passed.

I think I have the correct answer for this.  The body needs to be parse into a serde_json Value.
//! tests/api/subscriptions.rs
// [...]
use serde_json::Value;

#[actix_rt::test]
async fn subscribe_sends_a_confirmation_email_with_a_link() {
    // [...]

    // Parse the body as a string, starting from raw bytes
    let body: Value =
        serde_json::from_str(std::str::from_utf8(&email_request.body).unwrap()).unwrap();

Yes! I finished the chapter and find that out, although that information is introduced a couple pages later. Are these compile errors in the code examples teaching tools or mistakes? I've definitely learnt a lot by debugging them but I just wanted to put an issue here in case the intention was that they work immediately.

I don't know @LukeMathWalker would have to answer that.  Are you just being a zen master and pretending to make mistakes so we learn ;)

Ahah I'll have to check!
I go through all the issues once a week, so I'll come around to look into this one (and others) over this weekend :)

@emiddleton was indeed correct - there a line missing parsing the bytes to JSON 😅


Add configuration file for Thread

@SeppoTakalo , would you please review?

Commit include in the oob_test_mbed-os-5.4 branch.


ambiguous call between methods

I am trying to build a project with TinyIoC. (Win8, VS2012 sp3)
And get three errors:
Error	561	The call is ambiguous between the following methods or properties:
'TinyIoC.AssemblyExtensions.SafeGetTypes(System.Reflection.Assembly)' and 'TinyIoC.AssemblyExtensions.SafeGetTypes(System.Reflection.Assembly)'
Error	583	The call is ambiguous between the following methods or properties: 'TinyIoC.TypeExtensions.GetGenericMethod(System.Type, System.Reflection.BindingFlags, string, System.Type[], System.Type[])' and 'TinyIoC.TypeExtensions.GetGenericMethod(System.Type, System.Reflection.BindingFlags, string, System.Type[], System.Type[])'
Error	607	The call is ambiguous between the following methods or properties: 'TinyIoC.TypeExtensions.GetGenericMethod(System.Type, System.Reflection.BindingFlags, string, System.Type[], System.Type[])' and 'TinyIoC.TypeExtensions.GetGenericMethod(System.Type, System.Reflection.BindingFlags, string, System.Type[], System.Type[])'

Just to help people that may run in the same issue, it happened to me in a project that was including both TinyIoC and Nancy... Took me quite a while to figure it out!

@ThomasWeiss How did you solve it? Use Nancy.TinyIoC?

@jgillich Yes


Add missing native imports and remove "duplicates"

Currently, ImGui.NET is very close to having full coverage of all cimgui exports for version 1.49. I did a quick diff and identified the following list of functions that are not yet imported. There are also a few functions in ImGuiNative that define helper overloads which call the actual imported function. IMO, there should only be a single import for each cimgui function, and helper overloads should go into the ImGui class, which is supposed to have the safe, friendly wrappers.
@dpethes Did you mention that you were working on TreeNode stuff? I think I am going to try to finish all of the below before I start handling the 1.50 upgrade and breaking changes.
Missing imports (Added in 93f9e60dafe179ec1c584f369db32b522ec5f674)

igCreateContext
igDestroyContext
igGetColorU32
igGetColorU32Vec
igGetCurrentContext
igGetFontTexUvWhitePixel
igGetKeyIndex
igGetTreeNodeToLabelSpacing
igIsItemClicked
igIsRootWindowOrAnyChildHovered
igNewLine
igPlotHistogram2
igPlotLines2
igProgressBar
igSetCurrentContext
igSetItemAllowOverlap
igSetNextWindowSizeConstraints
igTreeAdvanceToLabelPos
igValueColor
igValueColor2
ImFontConfig_DefaultConstructor
ImGuiIO_ClearInputCharacters

Use va_list or ... (problematic)

igBulletTextV
igLabelTextV
igSetTooltipV
igTextColoredV
igTextDisabledV
igTextV
igTextWrappedV
igTreeNodePtrV
igTreeNodeStrV
igTreeNodeExV
igTreeNodeExPtr
igTreeNodeExStr
igTreeNodeExVPtr

I think we should ignore the above. You can just use string.Format() and pass that to the single-string overload.
Duplicate imports

ImFontAtlas_AddFontDefault
igSliderFloat
igBeginPopupModal (x2)

As part of this analysis I identified a few glaring problems and fixed them with https://github.com/mellinoe/ImGui.NET/commit/aef6b98bdb506bd8c2c6222474d24971a978cf46.

I added the missing imports in 93f9e60dafe179ec1c584f369db32b522ec5f674, and published version 0.2.0 onto NuGet.

Agreed about va_list. I added the FontAtlas_GetGlyph... functions in my text improvement branch, that should cover most of the FontAtlas imports as well.

Ah, those must have been added after the cimgui commit I'm using. I'll probably re-do the analysis with version 1.50 this week, and add the new functions. I don't think there is a ton of new stuff, so it shouldn't take too long.

There aren't many, but there are some important changes in IO struct, which can break stuff easily.

I think this can be closed.

We have complete coverage in the current version we are targeting (1.50), although there are some new functions and breaking changes in 1.51.


Fixed typo and added code quote

On line 43, fixed the typo (unless you really meant nom to run her ;) and hot reload :P :P).

hahaha. Dang auto-correct.


Unmet peer dependencies after installing Bootstrap 3.3.7, but it works

Please provide us with the following information:

OS?

Windows 7, 8 or 10. Linux (which distribution). Mac OSX (Yosemite? El Capitan?)

Ubuntu 16.04
Versions.

Please run ng --version. If there's nothing outputted, please run in a Terminal: node --version and paste the result here:

Could not start watchman; falling back to NodeWatcher for file system events.
Visit http://ember-cli.com/user-guide/#watchman for more info.
angular-cli: 1.0.0-beta.16
node: 6.7.0
os: linux x64
Repro steps.

Was this an app that wasn't created using the CLI? What change did you do on your code? etc.

ng new myapp
npm install --save bootstrap
The log given by the failure.

Normally this include a stack trace and some more information.

├── UNMET PEER DEPENDENCY<EMAIL_ADDRESS>├── UNMET PEER DEPENDENCY<EMAIL_ADDRESS>├── UNMET PEER DEPENDENCY<EMAIL_ADDRESS>├── UNMET PEER DEPENDENCY<EMAIL_ADDRESS>└──<EMAIL_ADDRESS>npm WARN optional Skipping failed optional dependency /chokidar/fsevents:
npm WARN notsup Not compatible with your operating system or architecture<EMAIL_ADDRESS>npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none was installed.
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none was installed.
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none was installed.
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none was installed.
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none was installed.
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none was installed.
Mention any other details that might be useful.
Added this to angularcli.json apps[0].styles
"../node_modules/bootstrap/dist/css/bootstrap.min.css"
Bootstrap is available for the app, at the end. But those UNMET PEER DEPENDENCY asking for an upper version of some of the angular components bother me. Is there anything broken?

Thanks! We'll be in touch soon.

Must be something else. Bootstrap does not have any dependencies on angular.
Try clearing your node_modules and run "npm install" again.

With a fresh new install (starting with ng new) keeps happening. The same message.
Versions:
angular-cli: 1.0.0-beta.16
node: 6.7.0
npm: 3.10.3

Fixed in https://github.com/angular/angular-cli/pull/2387.


clean jobs for all queues in dashboard

add clean buttons in the dashboard header. each button clean each type of jobs for all queues

All queue actions are inside 3dots, it should be there.
Lets call it truncate

I guess that it should be located in dashboard page (not inside a specific queue page.)

this is dashboard page

We encountered a situation where a 30+ megabyte file was accidentally put into the data in base64 format. Then there were more than 1,000 history records, and the browser would freeze when opening the page. So we hope there is a function to clear the queue job outside the queue details page, so that the job can be cleared directly without loading any job data. @felixmosh

Hi @Diluka
You can add a "data" formatter on your setup, just to show the board, than remove the job
queueAdapter.setFormatter('data', (data) => redact(data));


refactor(navbar): mutualize common component for mobile & desktop views

Based on our team's decision, we no longer require approval of the refactoring pull request. Instead, we will review and approve the refactoring once all changes related to a specific component have been completed.


Finding popular searches?

How to obtain popular searches from a particular type of object. Specifically say for example the search is supposed to return a list of Car objects. What would the query look like to return the top searches?
x = Car.pure_whoosh('Aud*')

print x
<Top 1 Results for Or([Prefix('display_name', u'au'), Prefix('description', u'au'), Prefix('name', u'au')]) runtime=0.00333499908447>

x

x.collector             x.extend                x.has_exact_length      x.order                 x.searcher
x.copy                  x.facet_names           x.has_matched_terms     x.q                     x.top_n
x.docnum                x.fields                x.highlighter           x.query_terms           x.upgrade
x.docs                  x.filter                x.is_empty              x.runtime               x.upgrade_and_extend
x.docset                x.formatter             x.items                 x.score
x.estimated_length      x.fragmenter            x.key_terms             x.scored_length
x.estimated_min_length  x.groups                x.matched_terms         x.scorer

I cannot find how how to get the Car primary key or the top search results back or the car objects from x variable. Any suggestions would help

+1

I'm looking for an sqlalchemy solution not a Pony solution.


test(mempool_p2p): test tx received from p2p reaches mempool

chore: changing closure Fn requirements on integration test utils
test(mempool_p2p): test received rpc transactions are being broadcasted
test(mempool_p2p): test tx received from p2p reaches mempool

This change is


Speeding Up the Training

Hi,
Just going through the Distributed RL Example. Basically, I have only just started working with AirSim itself, I was wondering if you can possibly train the model without rendering the simulation? Secondly, my rig consists of 2x1080's so would be great if I could utilise both GPU's but as far as I can tell the sample project uses only one. Is there any way to support multi-gpu configuration?
Cheers,

To your first point: even if you could, it'd be unlikely that it would speed things up. Ultimately, the unreal engine would still have to render the image, so it would still have to draw most of the scene. During our training on Azure, we actually did not have the visual window running (due to some details about how Batch starts the training process). We did not observe a speed increase due to not rendering the scene.
Regarding the multiple GPUs: It's unlikely that a single training instance would benefit from multiple GPUs. The model itself is quite small, so it easily fits inside a single GPU. Also, the bottleneck isn't training the model, it's generating the data. So even if you configured the tensorflow session to use both GPUs, you wouldn't see any benefit.
What would potentially be interesting is starting multiple agent sessions on the same machine. In this case, it would be similar to the scenario we had in batch in which multiple agents were running in parallel, and communicating the weight updates to the parameter server. In your scenario, you would have two agents running and one parameter server. You'd have to modify the code to conditionally use the different devices (probably depending on a command line arg) for the agents, but it could lead to a ~2x speedup, if your CPU doesn't become the bottleneck.

Hi,
Great that explains a lot, the multiple-agent is probably an objective to go for right now, having multiple agents collecting different observations should probably do a lot of good.
One more question regarding the code, so I've noticed that the model is saved to .json files, I have been training the model from the scratch for few days now, just really testing this thing and seeing how it works, just to confirm if I stop the training and want to resume it can I just use the .json file to load the model/weights and continue from the point I have stopped?
Cheers,


Request for fix: the ppp rec section echo example

Hi, the following example doesn't work properly for me:
echo 'Hello10010.2True{"id":100,"title":"sample"}'|./ppp rec -l5 -t 2:i,3:f,4:b,5:j "type(f1),type(f2),type(f3),type(f4),type(f5)"

I received an error message:
Traceback (most recent call last):
  File "/home/horw/PycharmProjects/pypipe/./ppp", line 694, in <module>
    args.handler(args)
  File "/home/horw/PycharmProjects/pypipe/./ppp", line 361, in rec_handler
    exec_code(code, args)
  File "/home/horw/PycharmProjects/pypipe/./ppp", line 187, in exec_code
    _exec_code(code, args)
  File "/home/horw/PycharmProjects/pypipe/./ppp", line 182, in _exec_code
    exec(compile(code, '<string>', 'exec'), _globals)
  File "<string>", line 29, in <module>
ValueError: not enough values to unpack (expected 5, got 1)

In my environment, I don't encounter any errors. It is speculated that the tab characters are being converted to spaces when copying examples from the README based on the error message.

For me, I used this command, and it still has an error. I tried to do it in a different environment.

echo 'Hello10010.2True{"id":100,"title":"sample"}'|./ppp rec -l5 -t 2:i,3:f,4:b,5:j "type(f1),type(f2),type(f3),type(f4),type(f5)"

I got it, the problem was with the Fish terminal. You are right; it looks like tab characters in 'Hello 100 10.2 True {"id":100,"title":"sample"}' were replaced with spaces. Thanks for your help

It seems to have been resolved, so I will close this issue.


The order of raw1.info["bads"] should not matter when concatenating with raw0.info["bads"] #11501

Reference issue
Fixes #11501.
What does this implement/fix?
The order of bad channels no longer matters.
Note, that this is already correctly implemented in _ensure_infos_match:
https://github.com/mne-tools/mne-python/blob/603a96da58ddba3c2c05d3a01639abd669cc9543/mne/io/meas_info.py#L2903-L2904
It's only wrong in _check_raw_compatibility(raw):
https://github.com/mne-tools/mne-python/blob/603a96da58ddba3c2c05d3a01639abd669cc9543/mne/io/base.py#L2501-L2513

Shit I got confused by git. I hope I didn't mess everything up. I think some files were moved to the cloud during a commit, that might be why there are so many file changes

Yikes, a ton of files deleted :)
I looked through your commits and saw which one did this (bff61e8c0bb03b9385b7ebad548f66023cb650a9), and then did:
git revert bff61e8c0bb03b9385b7ebad548f66023cb650a9
git checkout bff61e8c0bb03b9385b7ebad548f66023cb650a9 mne/io/base.py
git commit -am 'FIX: Try this'
git push

Looks like the diff is now +24 −13 so probably closer to correct at least. Locally if you git pull origin compare_bads you should get to the same state, then you can fix whatever is still broken/missing!

Looks like the diff is now +24 −13 so probably closer to correct at least. Locally if you git pull origin compare_bads you should get to the same state, then you can fix whatever is still broken/missing!

Thank you for your help! ☺

Dear @larsoner, are the changes good to go now? :)

any chance you can update or add a test for this?

@agramfort: Check ✅

Can you move your testing there to be consistent? Also rename your test test_concatenate_raws_bads_order

@agramfort done :)

@larsoner is there more I should do? :)

@moritz-gerster you need to fix conflicts

@agramfort  done ✅

@larsoner is it good now?

@larsoner good to go now?

@moritz-gerster lets wait for main to be green and lets merge this after its green again

Thanks @moritz-gerster !


Show Honey Icon in DiplomacyDealView

Issue #9 - When initiating a trade, Honey would not show correctly in the DealContentStack (on either player side).

Details:
Fix was to not include a size value when calling the SetIconToSize method.  The size value specified would cause the game to look for a 64x64 icon.   The definitions for the Honey and Maize resources are found in GranColombia_Maya_Icons_Resources.xml.  There does not appear to be a 64x64 icon in MayaGran_Resources.dds.
In Vanilla, this icon size is 50x50.  Before completing this fix, I added special-case code so I could side-by-size compare the size 50x50 icon with the supposed size 64x64...and there is no discernable difference.  That may be due to the fact that this object - IconAndText as defined in DiplomacyDealView.xml, only allows 44x44 for the Icon size.
Either way, the fix is to simply allow the game to use the 50x50 icon instead of trying to get the 64x64 icon, which is what the previous version was doing.

@alimulhuda thank you for testing this. I added you to civfanatics-cqui team, which has "write" permission to this repo. your PR approvals will now count towards the number of approvals required to merge a PR in this repo.
the goal is to have this repo be team-owned - if at least two people with WRITE permission approve a PR, it should be good enough.


Temperature modification link to TAV

Should temperature sensitivity analysis (environmental modification) for TMAX and TMIN include a change to the TAV used in soil temperature calculations? This would be important for soil organic matter, automatic planting, emergence, plant N and soil N processes. We should at least look at the impact.

After discussion with Jeff White, it seems there are cases where you would not want to change TAV with TMAX and TMIN, for example an experiment with field heaters for climate change study. It might be best to add a separate input in Environmental Modifications for TAV so the user can control it.


Adding Responsiveness & Hover Effect.

I have added the responsiveness to the web pages and also the hover effect to the various cards .
#Description
i included  the responsiveness to the web pages and also added the hover effect to the various cards  , align them in proper format so over UI will Enhance.
#Issue Fix #(1)

I have added the responsiveness to the web pages and also the hover effect to the various cards .

@Mohit5Upadhyay since there has been some modification in the code file, this branch is showing conflict can u resolve it.


Toyota: remove RAV4h and Highlander(H) from pedal sensitivity condition

Rav4 hybrid, highlander, and highlander hybrid are NO_STOP_TIMER_CAR.
Although these cars have very sensitive accelerators, they don't need a comma pedal to get out of a full stop, so it doesn't make sense to include them in the condition.

I'm not sure I understand the point of this. Are you saying that a pedal shouldn't be used anyway with these cars since they don't need a resume press? In that case, we should maintain a list of cars where we ignore the pedal if it's installed instead of just excluding them from these lists.

I'm not sure I understand the point of this. Are you saying that a pedal shouldn't be used anyway with these cars since they don't need a resume press? In that case, we should maintain a list of cars where we ignore the pedal if it's installed instead of just excluding them from these lists.

Yea a pedal is unnecessary on these cars, they can SnG without a resume press, so they shouldn't be included here.
The NO_STOP_TIMER_CAR list is pretty much the "ignore standstill/no pedal required" list, these cars are already in that list, yet they're still included in this if condition which doesn't make sense.

In that case, this change simply makes these cars drive worse with a pedal. Unless the pedal performance is way worse for some reason, I also don't see a reason to ignore it on certain cars.


Added useful links to readme and commented out stuff in main.

Weird, didn't know this is what my branch protection stuff did. I might need to adjust these....

Looks like I need to bypass branch protections for now. I should look closer into this so it's setup properly.


[SPARK-28970][SQL] Implement USE CATALOG/NAMESPACE for Data Source V2

What changes were proposed in this pull request?
This PR exposes USE CATALOG/USE SQL commands as described in this SPIP
It also exposes currentCatalog in CatalogManager.
Finally, it changes SHOW NAMESPACES and SHOW TABLES to use the current catalog if no catalog is specified (instead of default catalog).
Why are the changes needed?
There is currently no mechanism to change current catalog/namespace thru SQL commands.
Does this PR introduce any user-facing change?
Yes, you can perform the following:
// Sets the current catalog to 'testcat'
spark.sql("USE CATALOG testcat")

// Sets the current catalog to 'testcat' and current namespace to 'ns1.ns2'.
spark.sql("USE ns1.ns2 IN testcat")

// Now, the following will use 'testcat' as the current catalog and 'ns1.ns2' as the current namespace.
spark.sql("SHOW NAMESPACES")

How was this patch tested?
Added new unit tests.

cc: @cloud-fan @rdblue

@cloud-fan / @rdblue This will probably be refactored after https://github.com/apache/spark/pull/25747 is merged, but I wanted to send this out to get some feedback on the usage of current catalog.

Can one of the admins verify this patch?

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15476/
Test PASSed.

Test build #110501 has started for PR 25771 at commit 0d5246d.

Test build #110501 has finished for PR 25771 at commit 0d5246d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110501/
Test FAILed.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15615/
Test PASSed.

Test build #110708 has started for PR 25771 at commit 12685e0.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15866/
Test PASSed.

Test build #110708 has finished for PR 25771 at commit 12685e0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public abstract class DelegatingCatalogExtension implements BaseSessionCatalog
class CatalogManager(conf: SQLConf, defaultSessionCatalog: BaseSessionCatalog) extends Logging

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110708/
Test FAILed.

Test build #110797 has started for PR 25771 at commit ad08096.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15931/
Test PASSed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110797/
Test FAILed.

test this please

Test build #110806 has started for PR 25771 at commit ad08096.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15938/
Test PASSed.

Test build #110806 has finished for PR 25771 at commit ad08096.

This patch fails Java style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public abstract class DelegatingCatalogExtension implements CatalogExtension

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110806/
Test FAILed.

Test build #110822 has started for PR 25771 at commit 1fb5751.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15953/
Test PASSed.

Test build #110822 has finished for PR 25771 at commit 1fb5751.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110822/
Test FAILed.

Test build #110872 has started for PR 25771 at commit 45d4cf1.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15998/
Test PASSed.

Test build #110872 has finished for PR 25771 at commit 45d4cf1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110872/
Test FAILed.

Test build #110932 has started for PR 25771 at commit 4d2865c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16047/
Test PASSed.

Test build #110932 has finished for PR 25771 at commit 4d2865c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110932/
Test FAILed.

Test build #110948 has started for PR 25771 at commit 467a8fa.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16056/
Test PASSed.

Test build #110948 has finished for PR 25771 at commit 467a8fa.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110948/
Test FAILed.

@cloud-fan I noticed that three hive-thriftserver related tests were failing after this change because of SELECT current_database(). Basically it was getting current database from SessionCatalog directly.
https://github.com/apache/spark/blob/c1bb3316bd7e992897ebb48f7f648194db4d06f1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala#L81-L85
The fix is to use the currentNamespace from CatalogManager, but I wasn't sure what the best way to pass the CatalogManger around.  I can just change Optimizer to take in CatalogManager instead of SessionCatalog. What do you think?

Optimizer takes a SessionCatalog parameter and pass it to GetCurrentDatabase. We can change that parameter to CatalogManager

Test build #111025 has started for PR 25771 at commit 2dbf689.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16120/
Test PASSed.

Test build #111025 has finished for PR 25771 at commit 2dbf689.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class Optimizer(catalogManager: CatalogManager)
case class GetCurrentDatabase(catalogManager: CatalogManager) extends Rule[LogicalPlan]

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111025/
Test FAILed.

@cloud-fan / @rdblue, what should be the behavior USE <namespace> if the <namespace> doesn't exist? Should we follow the same behavior as v1 where database is required to exist?

I would expect it to fail. USE CATALOG non-existing should fail too.

Test build #111124 has started for PR 25771 at commit 86b6c79.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16204/
Test PASSed.

Test build #111124 has finished for PR 25771 at commit 86b6c79.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111124/
Test FAILed.

Test build #111127 has started for PR 25771 at commit 8c5a337.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16207/
Test PASSed.

Test build #111127 has finished for PR 25771 at commit 8c5a337.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class JavaClassifierParams(HasRawPredictionCol, JavaPredictorParams):
class JavaClassifier(JavaPredictor, JavaClassifierParams):
class JavaClassificationModel(JavaPredictionModel, JavaClassifierParams):
class JavaProbabilisticClassifierParams(HasProbabilityCol, HasThresholds, JavaClassifierParams):
class JavaProbabilisticClassifier(JavaClassifier, JavaProbabilisticClassifierParams):
class JavaProbabilisticClassificationModel(JavaClassificationModel,
class LinearSVC(JavaClassifier, HasMaxIter, HasRegParam, HasTol,
class LinearSVCModel(JavaClassificationModel, JavaMLWritable, JavaMLReadable):
class LogisticRegression(JavaProbabilisticClassifier, HasMaxIter, HasRegParam, HasTol,
class LogisticRegressionModel(JavaProbabilisticClassificationModel, JavaMLWritable, JavaMLReadable,
class DecisionTreeClassifier(JavaProbabilisticClassifier, HasWeightCol,
class DecisionTreeClassificationModel(DecisionTreeModel, JavaProbabilisticClassificationModel,
class RandomForestClassifier(JavaProbabilisticClassifier, HasSeed, RandomForestParams,
class RandomForestClassificationModel(TreeEnsembleModel, JavaProbabilisticClassificationModel,
class GBTClassifier(JavaProbabilisticClassifier, GBTClassifierParams, HasCheckpointInterval,
class GBTClassificationModel(TreeEnsembleModel, JavaProbabilisticClassificationModel,
class NaiveBayes(JavaProbabilisticClassifier, HasThresholds, HasWeightCol,
class NaiveBayesModel(JavaProbabilisticClassificationModel, JavaMLWritable, JavaMLReadable):
class MultilayerPerceptronClassifier(JavaProbabilisticClassifier, HasMaxIter, HasTol, HasSeed,
class MultilayerPerceptronClassificationModel(JavaProbabilisticClassificationModel, JavaMLWritable,
class OneVsRestParams(JavaClassifierParams, HasWeightCol):
class QuantileDiscretizer(JavaEstimator, HasInputCol, HasOutputCol, HasInputCols, HasOutputCols,
class LinearRegression(JavaPredictor, HasMaxIter, HasRegParam, HasTol, HasElasticNetParam,
class LinearRegressionModel(JavaPredictionModel, GeneralJavaMLWritable, JavaMLReadable,
class IsotonicRegression(JavaPredictor, HasWeightCol, JavaMLWritable, JavaMLReadable):
class IsotonicRegressionModel(JavaPredictionModel, JavaMLWritable, JavaMLReadable):
class DecisionTreeRegressor(JavaPredictor, HasWeightCol, DecisionTreeParams, TreeRegressorParams,
class DecisionTreeModel(JavaPredictionModel):
class RandomForestRegressor(JavaPredictor, HasSeed, RandomForestParams, TreeRegressorParams,
class GBTRegressor(JavaPredictor, GBTRegressorParams, HasCheckpointInterval, HasSeed,
class AFTSurvivalRegression(JavaPredictor, HasFitIntercept, HasMaxIter, HasTol,
class AFTSurvivalRegressionModel(JavaPredictionModel, JavaMLWritable, JavaMLReadable):
class GeneralizedLinearRegression(JavaPredictor, HasFitIntercept, HasMaxIter, HasTol, HasRegParam,
class GeneralizedLinearRegressionModel(JavaPredictionModel, JavaMLWritable,
class JavaPredictorParams(HasLabelCol, HasFeaturesCol, HasPredictionCol):
class JavaPredictor(JavaEstimator, JavaPredictorParams):
class JavaPredictionModel(JavaModel, JavaPredictorParams):
abstract class PartitionTransformExpression extends Expression with Unevaluable
case class Years(child: Expression) extends PartitionTransformExpression
case class Months(child: Expression) extends PartitionTransformExpression
case class Days(child: Expression) extends PartitionTransformExpression
case class Hours(child: Expression) extends PartitionTransformExpression
case class Bucket(numBuckets: Literal, child: Expression) extends PartitionTransformExpression
 implicit class OptionsHelper(options: Map[String, String])
trait WriteConfigMethods[R]
trait CreateTableWriter[T] extends WriteConfigMethods[CreateTableWriter[T]]

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111127/
Test FAILed.

I would expect it to fail. USE CATALOG non-existing should fail too.

Added these checks.

Test build #111128 has started for PR 25771 at commit e6e71a9.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16209/
Test PASSed.

Test build #111128 has finished for PR 25771 at commit e6e71a9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111128/
Test FAILed.

Test build #111138 has started for PR 25771 at commit 401f275.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16217/
Test PASSed.

Test build #111138 has finished for PR 25771 at commit 401f275.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111138/
Test FAILed.

Test build #111386 has started for PR 25771 at commit 30a0f7a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16428/
Test PASSed.

Test build #111387 has started for PR 25771 at commit 110f6bf.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16429/
Test PASSed.

Test build #111387 has finished for PR 25771 at commit 110f6bf.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111387/
Test FAILed.

Test build #111462 has started for PR 25771 at commit 341d8d2.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16496/
Test PASSed.

Test build #111462 has finished for PR 25771 at commit 341d8d2.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111462/
Test FAILed.

Test build #111463 has started for PR 25771 at commit a1b6d72.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16497/
Test PASSed.

Test build #111463 has finished for PR 25771 at commit a1b6d72.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111463/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16499/
Test PASSed.

Test build #111465 has started for PR 25771 at commit d42d72a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16500/
Test PASSed.

Test build #111466 has started for PR 25771 at commit c192710.

Test build #111465 has finished for PR 25771 at commit d42d72a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111465/
Test FAILed.

Test build #111466 has finished for PR 25771 at commit c192710.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class _GaussianMixtureParams(HasMaxIter, HasFeaturesCol, HasSeed, HasPredictionCol,
class GaussianMixtureModel(JavaModel, _GaussianMixtureParams, JavaMLWritable, JavaMLReadable,
class GaussianMixture(JavaEstimator, _GaussianMixtureParams, JavaMLWritable, JavaMLReadable):
class _KMeansParams(HasMaxIter, HasFeaturesCol, HasSeed, HasPredictionCol, HasTol,
class KMeansModel(JavaModel, _KMeansParams, GeneralJavaMLWritable, JavaMLReadable,
class KMeans(JavaEstimator, _KMeansParams, JavaMLWritable, JavaMLReadable):
class _BisectingKMeansParams(HasMaxIter, HasFeaturesCol, HasSeed, HasPredictionCol,
class BisectingKMeansModel(JavaModel, _BisectingKMeansParams, JavaMLWritable, JavaMLReadable,
class BisectingKMeans(JavaEstimator, _BisectingKMeansParams, JavaMLWritable, JavaMLReadable):
class _LDAParams(HasMaxIter, HasFeaturesCol, HasSeed, HasCheckpointInterval):
class LDAModel(JavaModel, _LDAParams):
class LDA(JavaEstimator, _LDAParams, JavaMLReadable, JavaMLWritable):
class _PowerIterationClusteringParams(HasMaxIter, HasWeightCol):
class PowerIterationClustering(_PowerIterationClusteringParams, JavaParams, JavaMLReadable,
case class PostgreCastStringToBoolean(child: Expression)

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111466/
Test FAILed.

Test build #111474 has started for PR 25771 at commit 1579f97.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16506/
Test PASSed.

Test build #111474 has finished for PR 25771 at commit 1579f97.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111474/
Test FAILed.

Test build #111498 has started for PR 25771 at commit b7c1e79.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16522/
Test PASSed.

Test build #111498 has finished for PR 25771 at commit b7c1e79.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111498/
Test PASSed.

Test build #111519 has started for PR 25771 at commit 2534baf.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16539/
Test PASSed.

Test build #111519 has finished for PR 25771 at commit 2534baf.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class SimpleTestOptimizer extends Optimizer(

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111519/
Test FAILed.

@cloud-fan this is ready for another round of review. Thanks!

Test build #111609 has started for PR 25771 at commit 3f2e1c1.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16607/
Test PASSed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111609/
Test FAILed.

I think it's ready to go. @imback82 can you fix the conflict? thanks!

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16664/
Test PASSed.

Test build #111670 has started for PR 25771 at commit 88872ea.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16665/
Test PASSed.

I just resolved the conflicts. Thanks @cloud-fan!

Test build #111670 has finished for PR 25771 at commit 88872ea.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111670/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16675/
Test PASSed.

Test build #111681 has started for PR 25771 at commit 8485619.

Test build #111681 has finished for PR 25771 at commit 8485619.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111681/
Test PASSed.

thanks, merging to master!


BEFORE WE FORK 1.6.x FROM master

We want https://docs.angularjs.org to reflect the latest full release of Angular.  At the moment the docs shown are the current snapshot of master. This requires two things:

changes to the hosting server to redirect docs.angularjs.org to code.angularjs.org/latest rather than code.angularjs.org/snapshot.
changes to the release process to additionally copy released files into code.angularjs.org/latest.

Resolved by #16093


HLint Parse error

Hello,
I write this Haskell code in Sample.hs file :
{-# LANGUAGE GADTs #-}

-- GADT : schema field

data SchemaField schemaField where
  SchemaField :: (Eq fieldType, Show fieldType) => {
    name :: String,
    fieldType :: fieldType,
    nullable :: Bool
  } -> SchemaField (String,fieldType,Bool)

-- concrete schema field
instance Show (SchemaField schemaField) where
  show (SchemaField name fieldType nullable) =
    "SchemaField " ++ name ++ " " ++ show fieldType ++ " " ++ show nullable
data FieldType = StringType deriving (Show,Eq)
schemaFieldName = SchemaField "name" StringType False

-- run test
main :: IO ()
main = putStrLn (show schemaFieldName)

And hlint raise an error :
gitpod /tmp $ hlint Sample.hs
Sample.hs:6:52: Error: Parse error
Found:
    data SchemaField schemaField where
  >   SchemaField :: (Eq fieldType, Show fieldType) => {
        name :: String,
        fieldType :: fieldType,

1 hint

But GHC compiler has no problem
gitpod /tmp $ ghc Sample.hs
[1 of 1] Compiling Main             ( Sample.hs, Sample.o ) [flags changed]
Linking Sample ...
gitpod /tmp $ ./Sample
SchemaField name StringType False

I used an old Hlint version :
gitpod /tmp $ hlint --version
HLint v2.1.11, (C) Neil Mitchell 2006-2018

But the last one, the issue is gone :
gitpod /tmp $ hlint --version
HLint v3.2.7, (C) Neil Mitchell 2006-2021

I do not know if version 2.1.11 is still maintained

Only the latest version is still maintained - what you wrote was probably not legal when the corresponding GHC was released. I suggest you upgrade.

I upgraded to last HLint version, problem is gone.
Thank you


Sample-Dashboard for Web Application

Please review it and mark it as accepted.

Thanks for contributing. The code looks good.


storage: add bloom filter for immutable memtables

Is your feature request related to a problem? Please describe.
as title
Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
No response

btw, I once thought that we might have one bloom filter for all imm memtables. However, when one of them is replaced in ReadVersion, it will become hard to maintain.

There is a discussion in #7406 to replace bloom filters with RIBBON filters. @Little-Wallace does it make sense to consider RIBBON filters here as well? If the memtable is immutable a XOR-based filter may be feasible, right?


[Polyfills] Replace deprecated babel-polyfills with core-js

Description
@babel/polyfill has been deprecated in favor of core-js and regenerator-runtime.
Checklist

[x] I have added a changelog entry, prefixed by the type of change noted above

@TheMallen - Any idea why the tests fail?

Closing in favor of #1323


Random Error from appendToConsole in Logger

I randomly get an error that bubbles to the user. It is in lib/Logger.js. In appendToConsole, it attempts to write something to the console. I would love to wrap this in a try/catch so that it doesn't bubble to the user. If I PR this, would you accept?

Yes, this definitely shouldn't bubble up. It sounds like the process is closing when the logger is trying to write, I would definitely try catch on this.

PR coming soon. Thanks @justinmchase!

What should I do in the catch? Normally I would log an error or something, but in this case, the console is not available. So... not sure WTF to do. Thoughts?

PR Submitted. Let me know if you want a change.

Fixed bye #38, thanks.


Improve cache in github actions

Which issue does this PR close?
N/A
Rationale for this change
I have seen several PR checks failing recently related to "out of disk space", for example: https://github.com/apache/arrow-datafusion/runs/7430371132?check_suite_focus=true
You can see from that PR that the cargo cache doesn't seem to be very effective (it downloads many crates). It also seems to recompile all the dependencies, despite restoring a cache.
I believe the cache that is being used is not useful (just restoring a bunch of old files that have been outdated for a long time)
What changes are included in this PR?

Change the cache key for target to include the git sha (as the point is to restore files from the linux-build-lib job earlier)
TODO change the cargo cache key likewise so the build job caches the data

Are there any user-facing changes?
No

🤔  it did not appear to help: https://github.com/apache/arrow-datafusion/runs/7432229787?check_suite_focus=true

Took a different approach instead: https://github.com/apache/arrow-datafusion/pull/2948


Error on starting, followed directions in readme

discord-overseer-info-1  |     baseUrl: Config.overseer.baseUrl,
discord-overseer-info-1  | [2024-09-08 19:30:52.690] ERROR: An unhandled promise rejection occurred.
discord-overseer-info-1  |     err: {
discord-overseer-info-1  |       "type": "DiscordjsError",
discord-overseer-info-1  |       "message": "Shard 0's process exited before its Client became ready.",
discord-overseer-info-1  |       "stack":
discord-overseer-info-1  |           Error [ShardingReadyDied]: Shard 0's process exited before its Client became ready.
discord-overseer-info-1  |               at Shard.onDeath (/app/node_modules/discord.js/src/sharding/Shard.js:178:16)
discord-overseer-info-1  |               at Object.onceWrapper (node:events:632:26)
discord-overseer-info-1  |               at Shard.emit (node:events:517:28)
discord-overseer-info-1  |               at Shard._handleExit (/app/node_modules/discord.js/src/sharding/Shard.js:439:10)
discord-overseer-info-1  |               at ChildProcess.emit (node:events:517:28)
discord-overseer-info-1  |               at ChildProcess._handle.onexit (node:internal/child_process:292:12)
discord-overseer-info-1  |       "code": "ShardingReadyDied"
discord-overseer-info-1  |     }
discord-overseer-info-1  |                              ^
discord-overseer-info-1  |
discord-overseer-info-1  | TypeError: Cannot read properties of undefined (reading 'baseUrl')
discord-overseer-info-1  |     at file:///app/dist/commands/chat/movie-command.js:11:30
discord-overseer-info-1  |     at ModuleJob.run (node:internal/modules/esm/module_job:195:25)
discord-overseer-info-1  |     at async ModuleLoader.import (node:internal/modules/esm/loader:337:24)
discord-overseer-info-1  |     at async loadESM (node:internal/process/esm_loader:34:7)
discord-overseer-info-1  |     at async handleMainPromise (node:internal/modules/run_main:106:12)
discord-overseer-info-1  |
discord-overseer-info-1  | Node.js v18.20.4
discord-overseer-info-1  | file:///app/dist/commands/chat/movie-command.js:11
discord-overseer-info-1  |     baseUrl: Config.overseer.baseUrl,
discord-overseer-info-1  |                              ^
discord-overseer-info-1  |
discord-overseer-info-1  | TypeError: Cannot read properties of undefined (reading 'baseUrl')
discord-overseer-info-1  |     at file:///app/dist/commands/chat/movie-command.js:11:30
discord-overseer-info-1  |     at ModuleJob.run (node:internal/modules/esm/module_job:195:25)
discord-overseer-info-1  |     at async ModuleLoader.import (node:internal/modules/esm/loader:337:24)
discord-overseer-info-1  |     at async loadESM (node:internal/process/esm_loader:34:7)
discord-overseer-info-1  |     at async handleMainPromise (node:internal/modules/run_main:106:12)

Can you please try this...
Get API Key from Overseer:

Open your Overseer instance
Go to settings -> general

Copy API Key

Copy Overseer Settings to config.json
In the config.json files created in the instructions change:

Update overseer.apiKey with the API Key copied earlier
Update baseUrl to YOUR_INSTANCE/api/v1

Example -> you access overseer at http://<IP_ADDRESS>:5055 then baseUrl is http://<IP_ADDRESS>:5055/api/v1


golangci-lint as a linter?

I prefer to use golangci-lint as a linter and was wondering if you considered using it for this project. I did a quick run on on the current codebase (42cb803c) using default settings and it took only a few seconds. It reveals a bunch of potential issues and some code improvements (see below), but it can also be noisy / annoying sometimes and it's quite strict about error checks (although this check can be disabled at your own peril 😅 ). The unused check emits false positives if there are build flags in the codebase, but I think that can be fixed. I also checked if enabling the extra bodyclose, gofmt and goimports checks would work and they didn't cause slowness or noise.

golangci-lint output

> golangci-lint run

lib/service/run.go:205:20: Error return value of `cli.ShowAppHelp` is not checked (errcheck)
				cli.ShowAppHelp(c)
				               ^
lib/service/run.go:341:19: Error return value of `cli.ShowAppHelp` is not checked (errcheck)
			cli.ShowAppHelp(context)
			               ^
lib/service/run.go:353:19: Error return value of `cli.ShowAppHelp` is not checked (errcheck)
			cli.ShowAppHelp(context)
			               ^
lib/service/run.go:356:14: Error return value of `flags.Parse` is not checked (errcheck)
		flags.Parse(os.Args[1:])
		           ^
lib/service/run.go:366:9: Error return value of `app.Run` is not checked (errcheck)
	app.Run(os.Args)
	       ^
lib/service/service.go:327:23: Error return value of `httpServer.Shutdown` is not checked (errcheck)
			httpServer.Shutdown(context.Background())
			                   ^
lib/service/service.go:341:37: Error return value of `(*runtime/pprof.Profile).WriteTo` is not checked (errcheck)
			pprof.Lookup("goroutine").WriteTo(os.Stderr, 1)
			                                 ^
lib/service/service.go:355:37: Error return value of `(*runtime/pprof.Profile).WriteTo` is not checked (errcheck)
			pprof.Lookup("goroutine").WriteTo(os.Stderr, 1)
			                                 ^
lib/service/deprecated.go:62:4: S1038: should use fmt.Fprintf instead of fmt.Fprintln(fmt.Sprintf(...)) (but don't forget the newline) (gosimple)
			fmt.Fprintln(os.Stderr, fmt.Sprintf("Configuration sanitise error: %v", err))
			^
lib/service/deprecated.go:79:4: S1038: should use fmt.Fprintf instead of fmt.Fprintln(fmt.Sprintf(...)) (but don't forget the newline) (gosimple)
			fmt.Fprintln(os.Stderr, fmt.Sprintf("Configuration marshal error: %v", err))
			^
lib/service/deprecated.go:85:4: S1038: should use fmt.Fprintf instead of fmt.Fprintln(fmt.Sprintf(...)) (but don't forget the newline) (gosimple)
			fmt.Fprintln(os.Stderr, fmt.Sprintf("Configuration marshal error: %v", err))
			^
internal/batch/error.go:11:6: `partError` is unused (deadcode)
type partError struct {
     ^
internal/batch/tracker.go:39:6: `hasTag` is unused (deadcode)
func hasTag(p types.Part, tag tagType) bool {
     ^
internal/batch/tracker.go:50:6: `withTag` is unused (deadcode)
func withTag(tag tagType, p types.Part) types.Part {
     ^
internal/batch/count.go:30:8: Error return value of `m.Iter` is not checked (errcheck)
	m.Iter(func(i int, p types.Part) error {
	      ^
internal/batch/error.go:62:15: Error return value of `e.source.Iter` is not checked (errcheck)
	e.source.Iter(func(i int, p types.Part) error {
	             ^
internal/batch/tracker.go:11:2: `index` is unused (structcheck)
	index int
	^
internal/batch/error.go:12:2: `i` is unused (structcheck)
	i   int
	^
internal/batch/error.go:13:2: `err` is unused (structcheck)
	err error
	^
lib/buffer/parallel_batcher.go:99:22: Error return value of `tran.Payload.Iter` is not checked (errcheck)
				tran.Payload.Iter(func(i int, p types.Part) error {
				                 ^
lib/buffer/parallel_wrapper.go:125:20: Error return value of `mWriteCount.Incr` is not checked (errcheck)
			mWriteCount.Incr(1)
			                ^
lib/buffer/parallel_wrapper.go:126:21: Error return value of `mWriteBacklog.Set` is not checked (errcheck)
			mWriteBacklog.Set(int64(backlog))
			                 ^
lib/buffer/parallel_wrapper.go:128:18: Error return value of `mWriteErr.Incr` is not checked (errcheck)
			mWriteErr.Incr(1)
			              ^
lib/buffer/parallel_wrapper.go:184:20: Error return value of `mWriteCount.Incr` is not checked (errcheck)
			mWriteCount.Incr(int64(len(transactions)))
			                ^
lib/buffer/parallel_wrapper.go:185:21: Error return value of `mWriteBacklog.Set` is not checked (errcheck)
			mWriteBacklog.Set(int64(backlog))
			                 ^
lib/buffer/parallel_wrapper.go:187:18: Error return value of `mWriteErr.Incr` is not checked (errcheck)
			mWriteErr.Incr(1)
			              ^
lib/buffer/parallel_wrapper.go:221:18: Error return value of `mReadErr.Incr` is not checked (errcheck)
				mReadErr.Incr(1)
				             ^
lib/buffer/parallel_wrapper.go:234:18: Error return value of `mReadCount.Incr` is not checked (errcheck)
		mReadCount.Incr(1)
		               ^
lib/buffer/parallel_wrapper.go:248:22: Error return value of `mSendSuccess.Incr` is not checked (errcheck)
				mSendSuccess.Incr(1)
				                 ^
lib/buffer/parallel_wrapper.go:249:20: Error return value of `mLatency.Timing` is not checked (errcheck)
				mLatency.Timing(time.Since(msg.CreatedAt()).Nanoseconds())
				               ^
lib/buffer/parallel_wrapper.go:253:18: Error return value of `mSendErr.Incr` is not checked (errcheck)
				mSendErr.Incr(1)
				             ^
lib/buffer/parallel_wrapper.go:257:17: Error return value of `mAckErr.Incr` is not checked (errcheck)
				mAckErr.Incr(1)
				            ^
lib/buffer/parallel_wrapper.go:262:17: Error return value of `mBacklog.Set` is not checked (errcheck)
				mBacklog.Set(int64(blog))
				            ^
lib/buffer/single_wrapper.go:122:20: Error return value of `mWriteCount.Incr` is not checked (errcheck)
			mWriteCount.Incr(1)
			                ^
lib/buffer/single_wrapper.go:123:21: Error return value of `mWriteBacklog.Set` is not checked (errcheck)
			mWriteBacklog.Set(int64(backlog))
			                 ^
lib/buffer/single_wrapper.go:125:18: Error return value of `mWriteErr.Incr` is not checked (errcheck)
			mWriteErr.Incr(1)
			              ^
lib/buffer/single_wrapper.go:158:19: Error return value of `mReadErr.Incr` is not checked (errcheck)
					mReadErr.Incr(1)
					             ^
lib/buffer/single_wrapper.go:167:27: Error return value of `m.buffer.ShiftMessage` is not checked (errcheck)
					m.buffer.ShiftMessage()
					                     ^
lib/buffer/single_wrapper.go:173:20: Error return value of `mReadCount.Incr` is not checked (errcheck)
				mReadCount.Incr(1)
				               ^
lib/buffer/single_wrapper.go:192:20: Error return value of `mLatency.Timing` is not checked (errcheck)
				mLatency.Timing(time.Since(msg.CreatedAt()).Nanoseconds())
				               ^
lib/buffer/single_wrapper.go:196:17: Error return value of `mBacklog.Set` is not checked (errcheck)
				mBacklog.Set(int64(backlog))
				            ^
lib/buffer/single_wrapper.go:197:22: Error return value of `mSendSuccess.Incr` is not checked (errcheck)
				mSendSuccess.Incr(1)
				                 ^
lib/buffer/single_wrapper.go:199:18: Error return value of `mSendErr.Incr` is not checked (errcheck)
				mSendErr.Incr(1)
				             ^
lib/buffer/parallel_batcher.go:19:2: `conf` is unused (structcheck)
	conf  Config
	^
lib/test/integration/cache/integration_test_helpers.go:78:6: `testOptTimeout` is unused (deadcode)
func testOptTimeout(timeout time.Duration) testOptFunc {
     ^
lib/test/integration/cache/integration_test_helpers.go:84:6: `testOptLogging` is unused (deadcode)
func testOptLogging(level string) testOptFunc {
     ^
lib/test/integration/cache/dynamodb_test.go:45:20: Error return value of `client.CreateTable` is not checked (errcheck)
	client.CreateTable(&dynamodb.CreateTableInput{
	                  ^
lib/test/integration/cache/dynamodb_test.go:88:17: Error return value of `resource.Expire` is not checked (errcheck)
	resource.Expire(900)
	               ^
lib/test/integration/cache/memcached_test.go:30:17: Error return value of `resource.Expire` is not checked (errcheck)
	resource.Expire(900)
	               ^
lib/test/integration/cache/redis_test.go:30:17: Error return value of `resource.Expire` is not checked (errcheck)
	resource.Expire(900)
	               ^
lib/stream/example_base64_test.go:28:13: Error return value of `result.Iter` is not checked (errcheck)
	result.Iter(func(i int, part types.Part) error {
	           ^
lib/stream/example_base64_test.go:32:10: Error return value of `e.Write` is not checked (errcheck)
		e.Write(part.Get())
		       ^
lib/stream/example_base64_test.go:77:14: Error return value of `s.Stop` is not checked (errcheck)
	defer s.Stop(time.Second)
	            ^
lib/stream/example_config_test.go:54:14: Error return value of `s.Stop` is not checked (errcheck)
	defer s.Stop(time.Second)
	            ^
lib/stream/example_split_to_batch_test.go:26:8: Error return value of `m.Iter` is not checked (errcheck)
	m.Iter(func(i int, b types.Part) error {
	      ^
lib/stream/example_split_to_batch_test.go:70:14: Error return value of `s.Stop` is not checked (errcheck)
	defer s.Stop(time.Second)
	            ^
lib/stream/example_split_to_messages_test.go:26:8: Error return value of `m.Iter` is not checked (errcheck)
	m.Iter(func(i int, b types.Part) error {
	      ^
lib/stream/type.go:59:11: Error return value of `w.Write` is not checked (errcheck)
			w.Write([]byte("input not connected\n"))
			       ^
lib/stream/type.go:64:11: Error return value of `w.Write` is not checked (errcheck)
			w.Write([]byte("output not connected\n"))
			       ^
lib/stream/type.go:67:11: Error return value of `w.Write` is not checked (errcheck)
			w.Write([]byte("OK"))
			       ^
lib/stream/type.go:373:36: Error return value of `(*runtime/pprof.Profile).WriteTo` is not checked (errcheck)
		pprof.Lookup("goroutine").WriteTo(dumpBuf, 1)
		                                 ^
lib/stream/example_base64_test.go:83:2: S1000: should use a simple channel send/receive instead of `select` with a single case (gosimple)
	select {
	^
lib/stream/example_config_test.go:60:2: S1000: should use a simple channel send/receive instead of `select` with a single case (gosimple)
	select {
	^
lib/stream/example_split_to_batch_test.go:76:2: S1000: should use a simple channel send/receive instead of `select` with a single case (gosimple)
	select {
	^
cmd/tools/benthos_config_gen/main.go:160:4: S1017: should replace this if statement with an unconditional strings.Replace (gosimple)
			if strings.Contains(path, alias) {
			^
lib/util/aws/lambda/client/type.go:146:20: Error return value of `l.mLimitErr.Incr` is not checked (errcheck)
			l.mLimitErr.Incr(1)
			                ^
lib/api/dynamic_crud_test.go:224:9: S1030: should use response.Body.String() instead of string(response.Body.Bytes()) (gosimple)
	res := string(response.Body.Bytes())
	       ^
lib/output/cassandra.go:186:2: `boff` is unused (structcheck)
	boff          backoff.ExponentialBackOff
	^
lib/output/writer.go:35:2: `closeOnce` is unused (structcheck)
	closeOnce      sync.Once
	^
lib/output/batcher.go:22:2: `conf` is unused (structcheck)
	conf  Config
	^
lib/output/drop_on_test.go:23:3: S1023: redundant `return` statement (gosimple)
		return
		^
lib/output/drop_on_test.go:75:3: S1023: redundant `return` statement (gosimple)
		return
		^
lib/output/wrap_with_pipeline_test.go:82:2: SA4006: this value of `newOutput` is never used (staticcheck)
	newOutput, err := WrapWithPipeline(&procs, mockOut, func(i *int) (types.Pipeline, error) {
	^
lib/output/async_writer_test.go:182:2: SA2002: the goroutine calls T.Fatal, which must be called in the same goroutine as the test (staticcheck)
	go func() {
	^
lib/output/async_writer_test.go:233:2: SA2002: the goroutine calls T.Fatal, which must be called in the same goroutine as the test (staticcheck)
	go func() {
	^
lib/output/async_writer_test.go:291:2: SA2002: the goroutine calls T.Fatal, which must be called in the same goroutine as the test (staticcheck)
	go func() {
	^
lib/processor/encode_test.go:211:8: ineffectual assignment to `res` (ineffassign)
	msgs, res = proc.ProcessMessage(message.New(input))
	      ^
lib/processor/grok_test.go:60:3: `name` is unused (structcheck)
		name        string
		^
lib/processor/jq_test.go:75:3: `name` is unused (structcheck)
		name   string
		^
lib/processor/parse_log_test.go:78:3: `format` is unused (structcheck)
		format  string
		^
lib/processor/parse_log_test.go:79:3: `codec` is unused (structcheck)
		codec   string
		^
lib/processor/parse_log_test.go:80:3: `bestEff` is unused (structcheck)
		bestEff bool
		^
lib/processor/http_old_test.go:22:3: S1023: redundant `return` statement (gosimple)
		return
		^
lib/processor/workflow.go:434:2: S1033: unnecessary guard around call to delete (gosimple)
	if _, exists := r.succeeded[k]; exists {
	^
lib/processor/workflow.go:443:2: S1033: unnecessary guard around call to delete (gosimple)
	if _, exists := r.succeeded[k]; exists {
	^
lib/processor/workflow.go:446:2: S1033: unnecessary guard around call to delete (gosimple)
	if _, exists := r.skipped[k]; exists {
	^
lib/processor/process_dag.go:335:5: S1009: should omit nil check; len() for nil maps is defined as zero (gosimple)
	if procs == nil || len(procs) == 0 {
	   ^
lib/processor/workflow_branch_map.go:326:5: S1009: should omit nil check; len() for nil maps is defined as zero (gosimple)
	if branches == nil || len(branches) == 0 {
	   ^
lib/processor/avro_test.go:327:25: S1039: unnecessary use of fmt.Sprintf (gosimple)
	conf.Avro.SchemaPath = fmt.Sprintf("file://path_does_not_exist")
	                       ^
lib/processor/jsonschema_test.go:320:31: S1039: unnecessary use of fmt.Sprintf (gosimple)
	conf.JSONSchema.SchemaPath = fmt.Sprintf("file://path_does_not_exist")
	                             ^
lib/processor/awk.go:714:4: SA9003: empty branch (staticcheck)
			if err == nil {
			^
lib/processor/batch_test.go:251:2: SA4006: this value of `res` is never used (staticcheck)
	msgs, res = proc.ProcessMessage(message.New([][]byte{[]byte("baz: end_batch")}))
	^
lib/processor/while.go:215:4: SA1019: s.LogEvent is deprecated: use LogFields or LogKV  (staticcheck)
			s.LogEvent("loop")
			^
lib/processor/protobuf.go:14:2: SA1019: package github.com/golang/protobuf/proto is deprecated: Use the "google.golang.org/protobuf/proto" package instead.  (staticcheck)
	"github.com/golang/protobuf/proto"
	^
internal/codec/reader.go:192:2: `ackOnce` is unused (structcheck)
	ackOnce   sync.Once
	^
internal/codec/reader.go:284:2: S1001: should use copy() instead of a loop (gosimple)
	for i, hdr := range headers {
	^
lib/input/reader/async_preserver_test.go:318:7: ineffectual assignment to `aFn` (ineffassign)
	msg, aFn, err = pres.ReadWithContext(ctx)
	     ^
lib/input/reader/redis_pubsub.go:44:2: `url` is unused (structcheck)
	url  *url.URL
	^
lib/input/reader/redis_list.go:43:2: `url` is unused (structcheck)
	url     *url.URL
	^
lib/input/reader/kinesis_balanced.go:58:2: `session` is unused (structcheck)
	session *session.Session
	^
lib/input/reader/kinesis_balanced.go:61:2: `namespace` is unused (structcheck)
	namespace     string
	^
lib/input/reader/kafka_cg.go:389:38: S1019: should use make(chan asyncMessage) instead (gosimple)
	k.msgChan = make(chan asyncMessage, 0)
	                                    ^
lib/pipeline/processor.go:35:2: `mSndErr` is unused (structcheck)
	mSndErr  metrics.StatCounter
	^
lib/pipeline/processor.go:34:2: `mSndSucc` is unused (structcheck)
	mSndSucc metrics.StatCounter
	^
internal/docs/component.go:289:23: ineffectual assignment to `err` (ineffassign)
			commonConfigBytes, err = config.MarshalYAML(tmp)
			                   ^
lib/broker/dynamic_fan_in.go:40:2: `internalTsChan` is unused (structcheck)
	internalTsChan  chan types.Transaction
	^
lib/broker/fan_in_test.go:60:4: SA2002: the goroutine calls T.Fatalf, which must be called in the same goroutine as the test (staticcheck)
			go func() {
			^
lib/buffer/single/mmap_cache_test.go:233:6: S1009: should omit nil check; len() for nil slices is defined as zero (gosimple)
		if bytes == nil || len(bytes) == 0 {
		   ^
lib/input/tests/http_server_test.go:161:9: ineffectual assignment to `err` (ineffassign)
			req, err := http.NewRequest(
			     ^
lib/input/tests/http_server_test.go:646:5: ineffectual assignment to `err` (ineffassign)
				err = nil
				^
lib/util/text/function_vars.go:276:19: SA6001: m[string(key)] would be more efficient than k := string(key); m[k] (staticcheck)
				targetFunc := string(content[3 : len(content)-1])
				              ^
lib/util/text/function_vars.go:284:19: SA6001: m[string(key)] would be more efficient than k := string(key); m[k] (staticcheck)
				targetFunc := string(content[3:colonIndex])
				              ^
internal/bloblang/query/functions.go:57:6: `fieldFunctionCtor` is unused (deadcode)
func fieldFunctionCtor(args ...interface{}) (Function, error) {
     ^
internal/bloblang/query/methods.go:1541:4: S1011: should replace loop with `values = append(values, m...)` (gosimple)
			for _, e := range m {
			^
lib/condition/bounds_check.go:54:2: `conf` is unused (structcheck)
	conf  Config
	^
lib/condition/jsonschema_test.go:187:31: S1039: unnecessary use of fmt.Sprintf (gosimple)
	conf.JSONSchema.SchemaPath = fmt.Sprintf("file://path_does_not_exist")
	                             ^
lib/condition/metadata.go:117:10: SA6005: should use strings.EqualFold instead (staticcheck)
		return strings.ToLower(md.Get(key)) == strings.ToLower(argStr)
		       ^
lib/service/test/processors_provider.go:165:2: S1031: unnecessary nil check around range (gosimple)
	if environment != nil {
	^
lib/test/integration/integration_test_helpers.go:151:6: `testOptLogging` is unused (deadcode)
func testOptLogging(level string) testOptFunc {
     ^
internal/bloblang/parser/query_function_parser.go:67:6: `parseLiteralWithTails` is unused (deadcode)
func parseLiteralWithTails(litParser Func) Func {
     ^
internal/bloblang/parser/query_literal_parser_test.go:39:3: `content` is unused (structcheck)
		content string
		^
internal/bloblang/parser/query_literal_parser_test.go:40:3: `meta` is unused (structcheck)
		meta    map[string]string
		^
internal/bloblang/parser/query_arithmetic_parser_test.go:336:3: `content` is unused (structcheck)
		content string
		^
internal/bloblang/parser/query_arithmetic_parser_test.go:337:3: `meta` is unused (structcheck)
		meta    map[string]string
		^
lib/input/http_server.go:219:2: `mSyncCount` is unused (structcheck)
	mSyncCount     metrics.StatCounter
	^
lib/input/http_server.go:220:2: `mSyncErr` is unused (structcheck)
	mSyncErr       metrics.StatCounter
	^
lib/input/http_server.go:221:2: `mSyncSucc` is unused (structcheck)
	mSyncSucc      metrics.StatCounter
	^
lib/input/kafka_cg.go:156:38: S1019: should use make(chan asyncMessage) instead (gosimple)
	k.msgChan = make(chan asyncMessage, 0)
	                                    ^
lib/input/csv.go:306:4: S1011: should replace loop with `headers = append(headers, records...)` (gosimple)
			for _, r := range records {
			^
lib/input/wrap_with_pipeline_test.go:75:2: SA4006: this value of `newInput` is never used (staticcheck)
	newInput, err := WrapWithPipeline(&procs, mockIn, func(i *int) (types.Pipeline, error) {
	^
lib/input/wrap_with_pipeline_test.go:138:2: SA4006: this value of `newInput` is never used (staticcheck)
	newInput, err := WrapWithPipelines(mockIn, func(i *int) (types.Pipeline, error) {
	^
lib/input/sequence_test.go:22:73: SA9002: file mode '777' evaluates to 01411; did you mean '0777'? (staticcheck)
		require.NoError(t, ioutil.WriteFile(filepath.Join(dir, k), []byte(v), 777))
		                                                                      ^
lib/input/aws_sqs.go:57:15: SA4021: x = append(y) is equivalent to x = y (staticcheck)
		FieldSpecs: append(
		            ^
lib/output/writer/kinesis.go:33:2: `kinesisPayloadLimitExceeded` is unused (deadcode)
	kinesisPayloadLimitExceeded = regexp.MustCompile("Member must have length less than or equal to")
	^
lib/output/writer/kinesis_firehose.go:24:2: `kinesisFirehoseMaxRecordsCount` is unused (deadcode)
	kinesisFirehoseMaxRecordsCount = 500
	^
lib/output/writer/kinesis_firehose_test.go:20:2: `firehoseMockStats` is unused (deadcode)
	firehoseMockStats = metrics.Noop()
	^
lib/output/writer/kafka.go:98:2: `mDroppedMaxBytes` is unused (structcheck)
	mDroppedMaxBytes metrics.StatCounter
	^
lib/output/writer/kinesis.go:81:2: `endpoint` is unused (structcheck)
	endpoint     *string
	^
lib/output/writer/kinesis_firehose.go:65:2: `endpoint` is unused (structcheck)
	endpoint    *string
	^
lib/output/writer/amqp_1.go:48:2: `key` is unused (structcheck)
	key     field.Expression
	^
lib/output/writer/amqp_1.go:49:2: `msgType` is unused (structcheck)
	msgType field.Expression
	^
lib/output/writer/elasticsearch.go:103:2: `interpolatedIndex` is unused (structcheck)
	interpolatedIndex bool
	^
lib/output/writer/elasticsearch.go:250:13: SA1019: e.client.Index().
	Index(index).
	Pipeline(e.pipelineStr.String(0, msg)).
	Type is deprecated: Types are in the process of being removed.  (staticcheck)
		_, err := e.client.Index().
		          ^
lib/test/integration/azure_test.go:17:6: type `AzuriteTransport` is unused (unused)
lib/input/http_client.go:244:22: func `(*HTTPClient).parseResponse` is unused (unused)
lib/output/subprocess_test.go:19:6: func `testProgram` is unused (unused)
lib/buffer/parallel_wrapper.go:138:27: func `(*ParallelWrapper).batchedInputLoop` is unused (unused)
lib/buffer/parallel_wrapper.go:42:6: type `parallelBatchedWrite` is unused (unused)
internal/bloblang/parser/query_literal_parser_test.go:38:7: type `easyMsg` is unused (unused)
internal/bloblang/parser/query_arithmetic_parser_test.go:335:7: type `easyMsg` is unused (unused)
lib/message/message_test.go:435:6: type `dummyCond` is unused (unused)
internal/batch/tracker.go:10:6: type `tag` is unused (unused)
internal/batch/tracker.go:14:6: type `tagType` is unused (unused)
internal/batch/tracker.go:20:6: type `tagChecker` is unused (unused)
internal/batch/tracker.go:24:6: type `tagValue` is unused (unused)

Hey @mihaitodor, yeah this is something I'm interested in adding. I want to avoid trying to merge a huge PR to fix all of the lints as I'm worried about regressions and the possibility of breaking backwards compatibility with sweeping changes.
We could try fixing all lints incrementally over time in smallish instalments, starting with (suspected) actual bugs and slowly working our way out to the more opinionated lints. I'm not sure we'll reach a 100% pass as I'd wager there are some lints regarding APIs that we can't change (yet).

Hey @mihaitodor, yeah this is something I'm interested in adding. I want to avoid trying to merge a huge PR to fix all of the lints as I'm worried about regressions and the possibility of breaking backwards compatibility with sweeping changes.
We could try fixing all lints incrementally over time in smallish instalments, starting with (suspected) actual bugs and slowly working our way out to the more opinionated lints. I'm not sure we'll reach a 100% pass as I'd wager there are some lints regarding APIs that we can't change (yet).

@Jeffail I agree. golangci-lint makes it easy to disable linters via config, so I think enabling them one at a time could work. I'll try to take an initial crack at enabling the harmless stuff this week.

@Jeffail I agree. golangci-lint makes it easy to disable linters via config, so I think enabling them one at a time could work. I'll try to take an initial crack at enabling the harmless stuff this week.

I think this is well covered for now. I'll open separate issues for gabs and tunny when I'll get some time.


FISH: not recognize programs.fish.enable when shell = pkgs.fish; is set to a different file

Describe the bug
I'm using NixOS and I deploy my configuration by flakes. I have two different files where I set fish shell.
./home-manager/shells/fish/default.nix
{ home-manager, username, shell, ... }:
{
    home-manager.users.${username} = { pkgs, ...}: {
      home.packages = with pkgs; [
        neofetch
        zoxide
      ];
      xdg.configFile."fish/functions/fish_prompt.fish".source = ./fish_prompt.fish;
      programs.fish = {
        enable = true;
      };
    };
}

and ./users/default.nix
{ pkgs, username, ... }:
{
# Define a user account. Don't forget to change the password by ‘passwd’.
  users.users.${username} = {
    shell = pkgs.fish;
    ignoreShellProgramCheck = true;
    isNormalUser = true;
    initialPassword = "changeme";
    extraGroups = [ "wheel" "input" "video" "render" "networkmanager" ];
    packages = with pkgs; [
      git
      vscodium
      wget
    ];
  };
}

when I run this configuration by flakes, I get the following error:
       error:
       Failed assertions:
       - users.users.myuser.shell is set to fish, but
       programs.fish.enable is not true. This will cause the fish
       shell to lack the basic nix directories in its PATH and might make
       logging in as that user impossible. You can fix it with:
       programs.fish.enable = true;

       If you know what you're doing and you are fine with the behavior,
       set users.users.myuser.ignoreShellProgramCheck = true;
       instead.

Why am I getting this error if I specified fish to be enabled?
Expected behavior
Building process with no error.
Notify maintainers
@adamcstephens @cole-h @winterqt

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.68, NixOS, 24.05 (Uakari), 24.05.20231219.54aac08`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.1`
 - channels(root): `"nixos-23.11"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Add a :+1: reaction to issues you find important.

 You are expected to set programs.fish.enable in your NixOS configuration.

./home-manager/shells/fish/default.nix

Oh. Do you mean outside home-manager.users.${username} = { pkgs, ...}: { block?

Yes, assuming that is a NixOS configuration file.

Thank you.


Migration design and requirements

Closing in favour of https://github.com/iotaledger/iota-sdk/issues/112