autofocus: true is usually applied to the first not :hidden input of the form. Wouldn't corresponding magic option be more reasonable?

Today I remembered Devise forms stuffing my own ones with autofocus: true and, yeah, one wishes autofocus: true for first not :hidden input in most cases:
#
# ... devise/confirmations/new ...
#
= simple_form_for resource, as: resource_name, url: confirmation_path(resource_name), html: { method: :post } do |f|
  = f.input :email, autofocus: true
#
# ... devise/passwords/edit ...
#
= simple_form_for resource, as: resource_name, url: password_path(resource_name), html: { method: :put } do |f|
  = f.input :reset_password_token, as: :hidden
  = f.input :password, label: 'New password', autofocus: true,
    input_html: { minlength: @minimum_password_length, autocomplete: :off }
#
# ... devise/passwords/new ...
#
= simple_form_for resource, as: resource_name, url: password_path(resource_name), html: { method: :post } do |f|
  = f.input :email, autofocus: true
#
# ... devise/registrations/edit ...
#
= simple_form_for resource, as: resource_name, url: registration_path(resource_name), html: { method: :put } do |f|
  = f.input :email, autofocus: true
#
# ... devise/registrations/new ...
#
= simple_form_for resource, as: resource_name, url: registration_path(resource_name) do |f|
  = f.input :email, autofocus: true
#
# ... devise/sessions/new ...
#
= simple_form_for resource, as: resource_name, url: session_path(resource_name) do |f|
  = f.input :email, autofocus: true
#
# ... devise/unlocks/new ...
#
= simple_form_for resource, as: resource_name, url: unlock_path(resource_name), html: { method: :post } do |f|
  = f.input :email, autofocus: true
#
# ...
#

So my question is: is it possible to implement suchlike option for config/initializers/simple_form.rb and will it worth the hassle?
Also I have no idea regarding the order of the autofocus triggering for multiple forms on the page.

I don't know if it's an issue yet, so I'm closing it. Feel free to comment or reopen it if necessary.


Init tests

Contribute to #2

@arichar6 ok if I merge?

So, I have a question about what branching strategy we will adopt for this project.
I like the ideas outlined in this GitHub Flow article. It seems like the most simple strategy, and merges changes back into the master branch once they pass testing.
A lot of projects use additional branches for various reasons. There's something to be said for keeping a separate develop branch, as described here.
I don't have a strong opinion either way, but do lean towards the simpler GitHub Flow strategy. If that sounds good to you, then I'd say go ahead and merge this branch into master.

For now, I like the basic GitHub Flow where we cut branches from master and merge back into master. Either you or I can confirm the branch passes tests, provides documentation, and closes one or more correctly formatted issues (or TODOs on an issue) before merging. At some point we should have a CONTRIBUTING doc that outlines all of this.
If we were ever going to significantly rework the codebase and support updates to the production code (that wouldn't necessarily go into the new codebase) while also developing on a new branch, then I think it would make sense to have a develop branch. But as long as all of the changes we are making can go immediately into production once they pass tests, we should just branch from master.
I really want to reinforce the behavior in the GitHub Flow: 1) create and document TODO(s)/issue(s); 2) branch; 3) iterate on develop, commit often, discuss in issue(s) 4) pull request (with work-in-progress (WIP) in title if not ready for merge); 5) iterate on develop, commit often, discuss; 6) pass tests and satisfy TODO(s)/issue(s); 7) merge and close TODO(s)/issue(s).


Issue 510

List of fixed issues (if they exist):

#510

Looks like these snapshots are failing because meshes are not being rendered. They will also fail due to UI widget differences between OSes captured in the full page screenshots. I will need to just save the image of the canvas (like our other tests).

Yes, your patch looks more comprehensive, but the minimal fix is to change one line from:
//urlParts = layer.url.split("/");

to define this this variable, e.g.
let urlParts = null;

@neurolabusc , thanks for the minimal code, but I do not think that is impacting the tests, because they work locally.
There's a few issues here:

I was attempting to create screenshots of our demos pages for meshes, so that we avoid publishing broken demos in the future.
however, the demos pages will look different based on the OS (standard sliders, etc. in html look different on Mac vs. Win). I will need to just capture the canvas image, rather than the full page screenshot to avoid the UI widgets.
the mesh demos don't seem to render in headless mode. @hanayik can test this more on a windows machine that will do headless tests.


[WFLY-16119] Upgrade google guava 31.1-jre

Fixes https://issues.redhat.com/browse/WFLY-16119

@chengfang @jasondlee @ehsavoie @jamezp @ronsigal @jimma @fl4via I have taken your names from the previous PR as reviewers. Could you please review and approve? Please find the links in the Jira.

@chengfang @jasondlee @ehsavoie @ronsigal @jimma @fl4via Normally I would avoid a reminder, but Tuesday 22nd March is closing for 26.x - it would be great if you can review and approve here, to get it in the beta.

@bstansberry Could you have a look at this PR for main and the 26.x counterpart of it, please? I hope it has enough positive votes to be acceptable for the 26beta. Tests passed.

@boris-unckel  @bstansberry  I upgraded this version in wss4j and everything works, so this change looks good for webservice subystem.


Document source link and run to click debugger options.

@chuckries,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot

@caslan

@Mikejo5000 Thanks Mike! Any idea when the live site will be updated?

Generally it gets updated by EOD Friday, plus once or twice during the week.


Remove pypy bugs tests

As suggested in https://github.com/cython/cython/pull/4471#issuecomment-979489947
I don't think that we're getting any benefit from running the pypy
known bugs test as part of the CI (and it's causing minor problems)
Feel free to reject this if you disagree though.

FWIW, I agree that this is not needed here.


Tip: Using PiHole as DNS server for Home Assistant and other addons

Problem/Motivation
It is useful to be able to have other addons use the PiHole addon for DNS requests, but this won't work with the default config. The fix for this is easy, but not obvious.
Expected behavior
Home Assistant and addons can use the PiHole addon for DNS requests.
Actual behavior
DNS requests timeout.
Proposed changes
Two simple config changes allow for this to work.

Set HassIO's static IP as a dns server in the ResinOS network config (Do not use <IP_ADDRESS> as Docker will ignore that). eg:

[ipv4]
address1=<IP_ADDRESS>/24,<IP_ADDRESS>
dns=<IP_ADDRESS>;<IP_ADDRESS>;<IP_ADDRESS>;
dns-search=lan.example.net
method=manual

In the PiHole Settings -> DNS tab, select "Listen on all interfaces, permit all origins".

I recommend including this information in the documentation.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

I like this idea @TechAsh!
Since it is a nice-to-have, I've given this a low priority.
Feel free to open a PR if you'd like to add it yourself 😉

Hi - I am using HassOS vs resinOS what are the changes?

Second thought, while this works (and is pretty awesome).
I'm not going to recommend "Listen on all interfaces, permit all origins".
Therefore I'm closing this issue/suggestion for now.


Initiate the Babylon.Viewer project

Notes:

Support for plain html, web components, Angular, React, WordPress
Extensions
Support for custom UI (buttons like fullscreen, brandingm webvr)


chore: Skip building sample images to avoid job timeout

Codecov Report

Merging #17 (423a838) into main (f1970ab) will increase coverage by 0.89%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main      #17      +/-   ##
==========================================
+ Coverage   26.66%   27.56%   +0.89%
==========================================
  Files           2        2
  Lines         780      780
==========================================
+ Hits          208      215       +7
+ Misses        527      522       -5
+ Partials       45       43       -2

Impacted Files
Coverage Δ

pkg/controller/vm_controller.go
26.19% <0.00%> (+1.85%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f1970ab...423a838. Read the comment docs.


ability to require multiple approvals

One feature that would be useful for the MPA worklfows is to set to have MIN_APPROVERS=2, where if there's 5 ACTIVATION_REQUEST_MAX_REVIEWERS in the list, 2 would be required, otherwise default to 1 approval only.

I assume the lack of state being the reason 2 is more difficult, yeah honestly the Slack feature might help w/ visibility and awareness of approvals amongst peers. The feedback originally was that some only check their email once a day.
Implementation of 1 could be a slight boost as well, otherwise someone might just try and skirt around the more scrutinizing senior members, and opt to get approval from the intern :)

This feature is now available in release 1.4, see ACTIVATION_REQUEST_MIN_REVIEWERS, ACTIVATION_REQUEST_MAX_REVIEWERS.
Thanks again for the feature suggestion!


File-roller extracts an empty file from apk

Describe the bug
Test file.
EarthViewForMuzei-2.1.zip
When open and extract it with file-roller, an empty file with the same name will be extracted, i.e. EarthViewForMuzei-2.1. This file does not exist in the apk and I can't see that with unzip and bsdtar.
Notify maintainers

@jtojnar
Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
output here

What version are you running? I am unable to reproduce with 43.0, with neither unzip on PATH or not. Do you have some other compression CLI programs installed (_7zz, …)?

Good catch! I guess 7zz is used automatically. Sadly, without 7zz I'll suffer from https://github.com/NixOS/nixpkgs/issues/166241... Thanks for your help!

I cannot reproduce with 7zz on PATH either. Are you sure it works if you uninstall 7zz, or mask the path?
env PATH= "$(type -p file-roller)"

Yes. After I uninstall 7zz the dummy empty file disappeared. I can also reproduce the problem with 7zz l.

Weird, it works for me:
$ result/bin/7zz l /tmp/mozilla_jtojnar0/EarthViewForMuzei-2.1.zip

7-Zip (z) 22.01 (x64) : Copyright (c) 1999-2022 Igor Pavlov : 2022-07-15
 64-bit locale=en_GB.UTF-8 Threads:8, ASM

Scanning the drive for archives:
1 file, 854093 bytes (835 KiB)

Listing archive: /tmp/mozilla_jtojnar0/EarthViewForMuzei-2.1.zip

--
Path = /tmp/mozilla_jtojnar0/EarthViewForMuzei-2.1.zip
Type = zip
Physical Size = 854093

   Date      Time    Attr         Size   Compressed  Name
------------------- ----- ------------ ------------  ------------------------
2022-11-06 13:56:44 .....      1113224       853865  EarthViewForMuzei-2.1.apk
------------------- ----- ------------ ------------  ------------------------
2022-11-06 13:56:44            1113224       853865  1 files

Ah, I see.
Maybe try opening an issue in https://sourceforge.net/p/sevenzip/bugs/

If I have a SF account I have done that...


chore: update travis to only run unit tests

Pull Request Test Coverage Report for Build 178

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 95.313%

Totals

Change from base Build 166:
0.0%

Covered Lines:
93

Relevant Lines:
97

💛  - Coveralls


Exception has occurred: WebDriverException (Windows 10)

Following the quick example:
from handler import Handler

chatgpt = Handler(<<my username>>, <<my password>>)

answer = chatgpt.interact("Hello, how are you today")

print(answer)

I receive the following error:
Exception has occurred: WebDriverException
Message: unknown error: cannot connect to chrome at <IP_ADDRESS>:65421
from unknown error: unable to discover open pages
Stacktrace:
Backtrace:
	GetHandleVerifier [0x007ADCE3+50899]
	(No symbol) [0x0073E111]
	(No symbol) [0x00645588]
	(No symbol) [0x006655A9]
	(No symbol) [0x0065F5B1]
	(No symbol) [0x0065F391]
	(No symbol) [0x00691FFE]
	(No symbol) [0x00691CEC]
	(No symbol) [0x0068B6F6]
	(No symbol) [0x00667708]
	(No symbol) [0x0066886D]
	GetHandleVerifier [0x00A13EAE+2566302]
	GetHandleVerifier [0x00A492B1+2784417]
	GetHandleVerifier [0x00A4327C+2759788]
	GetHandleVerifier [0x00845740+672048]
	(No symbol) [0x00748872]
	(No symbol) [0x007441C8]
	(No symbol) [0x007442AB]
	(No symbol) [0x007371B7]
	BaseThreadInitThunk [0x769F0099+25]
	RtlGetAppContainerNamedObjectPath [0x77AD7B6E+286]
	RtlGetAppContainerNamedObjectPath [0x77AD7B3E+238]
  File "C:\Users\<<user>>\Projects\ChatGPT_Selenium\handler.py", line 94, in __init__
    self.browser = uc.Chrome(options=options, use_subprocess=True)
  File "C:\Users\<<user>>\Projects\ChatGPT_Selenium\chat.py", line 8, in <module>
    chatgpt = Handler(YOUR_USERNAME, YOUR_PASSWORD)
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at <IP_ADDRESS>:65421
from unknown error: unable to discover open pages
Stacktrace:
Backtrace:
	GetHandleVerifier [0x007ADCE3+50899]
	(No symbol) [0x0073E111]
	(No symbol) [0x00645588]
	(No symbol) [0x006655A9]
	(No symbol) [0x0065F5B1]
	(No symbol) [0x0065F391]
	(No symbol) [0x00691FFE]
	(No symbol) [0x00691CEC]
	(No symbol) [0x0068B6F6]
	(No symbol) [0x00667708]
	(No symbol) [0x0066886D]
	GetHandleVerifier [0x00A13EAE+2566302]
	GetHandleVerifier [0x00A492B1+2784417]
	GetHandleVerifier [0x00A4327C+2759788]
	GetHandleVerifier [0x00845740+672048]
	(No symbol) [0x00748872]
	(No symbol) [0x007441C8]
	(No symbol) [0x007442AB]
	(No symbol) [0x007371B7]
	BaseThreadInitThunk [0x769F0099+25]
	RtlGetAppContainerNamedObjectPath [0x77AD7B6E+286]
	RtlGetAppContainerNamedObjectPath [0x77AD7B3E+238]

I have confirmed both WebDriver and undetected_chromedriver function on their own with sample scripts, however the handler doesn't seem to want to connect.
I've tried specifying the chromedriver.exe in the options within handler.py but that hasn't work, and specifying the path wasn't necessary in the quick checks of undetected_chromedriver and WebDriver on their own.
I'm using ChromeDriver 112.0.5615.49 (bd2a7bcb881c11e8cfe3078709382934e3916914-refs/branch-heads/5615@{#936}) attempting to align with Chrome 112.0.5615.138 (Official Build) (64-bit).
Thanks in advance for any thoughts.

This may be because of a mismatch between undetected_chrome and Chrome versions. Check this page: https://github.com/ultrafunkamsterdam/undetected-chromedriver/issues/868 .
You can try removing and reinstalling Selenium and undetected_chrome as a brute-force solution, but I'm not sure if that helps.
But before doing that, can you pass headless=False to the Handler, perhaps it is the headless mode that causes the problem.

Thanks for that!
With headless=False it fails with Raised Exceptions enabled while debugging, but only because it's looking for that element that doesn't exist.  Disabling the checkbox allows it to run successfully in non-headless mode, with the exception handling set to Pass.

Setting it back to headless however fails with a similar error, though not a version mismatch like the linked Issue:
Exception has occurred: WebDriverException       (note: full exception trace is shown but execution is paused at: <module>)
Message: unknown error: cannot connect to chrome at <IP_ADDRESS>:64823
from unknown error: unable to discover open pages

In headless mode while debugging it appears to hang on the line 94 of handler.py when trying to create the chatgpt instance:
...
        self.browser = uc.Chrome(options=options)
        self.browser.set_page_load_timeout(15)
...

After the 15 second timeout the element fails to attach because there's seemingly no browser instance accessible by Python.
I'm not sure how to troubleshoot this, but happy to do so if there's more information I can provide.  For now it's cool that I can play with it in non-headless mode, so thanks again for your quick reply!

Might be related/caused by https://github.com/ugorsahin/ChatGPT_Selenium/issues/4

I fixed the problem with new settings, please pull again and let me know if headless is functional on your end as well.

Looks like it's hanging on the same issue, catching an Exception on line 32 of chatgpt_client.py:
Exception has occurred: WebDriverException
Message: unknown error: cannot connect to chrome at <IP_ADDRESS>:49266
from unknown error: unable to discover open pages

As a note, to make the usage example work with the changes, I had to change the module reference like so: from chatgpt_automation.chatgpt_client import ChatGPT_Client and also update the chatgpt = Handler(... reference to chatgpt = ChatGPT_Client(..., but otherwise the changes looked clean.

Can you upgrade the undetected chrome to the latest version? I guess that may be the reason.

Can you link to how I go about doing that?  I've placed the updated EXE in the project folder but it seems the code is downloading the 112 version directly, though that should be the right one for my version of Chrome:
DEBUG:undetected_chromedriver.patcher:getting release number from /LATEST_RELEASE
DEBUG:undetected_chromedriver.patcher:downloading from https://chromedriver.storage.googleapis.com/112.0.5615.49/chromedriver_win32.zip
DEBUG:undetected_chromedriver.patcher:unzipping C:\Users\dijikul\AppData\Local\Temp\tmpyrwhuokd
INFO:undetected_chromedriver.patcher:patching driver executable C:\Users\dijikul\appdata\roaming\undetected_chromedriver\1dedd9a927d6d106_chromedriver.exe

Tagging these two issues which seem related:

https://github.com/ultrafunkamsterdam/undetected-chromedriver/issues/1135

https://github.com/ultrafunkamsterdam/undetected-chromedriver/issues/1046

Oh, you can try pasting the following line to the command line (cmd or powershell)
python -m pip install undetected-chromedriver==3.4.6

Thank you!
Upgrading undetected-chromedriver to 3.4.6 now allows it to run headlessly.
It looks like the 3.1.7 version might have been re-downloading due to the requirements.txt specification: https://github.com/ugorsahin/ChatGPT_Automation/blob/main/requirements.txt
Thanks again!  This repo is great!


Update Sentinel-2 metadata and rename the file

Changes:

Added extra bands (AOT and WVP)
Renamed bands to be the same as what comes from Element 84

Questions:

Should the AOT and WVP be included in the product, or just added in the dataset (make them optional)?
Should we make the product name different? (Currently it's s2a_msil2a and s2b_msil2a
Should we name the file different?
Are the band aliases and names ok?

I think in other products we prefer human readable names as a canonical name for a band, and original name B01 goes into alias. This does mean that STAC -> yaml code will need to support band renaming.
Difference between aliases and canonical name is that canonical name is used when loading data without asking for a set of specific bands.

I think in other products we prefer human readable names as a canonical name for a band, and original name B01 goes into alias. This does mean that STAC -> yaml code will need to support band renaming.
Difference between aliases and canonical name is that canonical name is used when loading data without asking for a set of specific bands.

I actually think this is fine because the band names B01 - B12 are commonly used for S2 while human readable names are interpretive.

Is this QA band section going to work ok, @Kirill888?
- name: 'SCL'
  aliases: [mask, qa]
  units: '1'
  dtype: uint8
  nodata: 0
  flags_definition:
    qa:
      bits: [0,1,2,3,4,5,6,7]
      description: Sen2Cor Scene Classification
      values:
        1: saturated or defective
        2: dark area pixels
        3: cloud shadows
        4: vegetation
        5: bare soils
        6: water
        7: unclassified
        8: cloud medium probability
        9: cloud high probability
        10: thin cirrus
        11: snow or ice

@alexgleith it will work, it's just names for categories that are shared across products are different. There is value in using names as defined by provider's documentation though. We can always add another flags_definition item later on with "harmonized" names as used in fmask-based products: shadow,water,snow, doesn't really work for things like that since there are several cloud classifications.


Use correct method calls in ProjectAuthorityController.update()

Coverage remained the same at 13.851% when pulling 478da4f16fa51bc0f41289488a9cc6e49a51545a on ProjectAuthorityController-IncorrectMethods into 75a1975361847788f8b044895b5a98831f219920 on master.


BatchNorm does not converge in digits

I was having problems with batchnorm layer not converging when using the nvidia version of caffe. This problem did not occur when using the caffe branch.

Hello, do you have more details to share about this?

Hi,
Most of the details was logged here in the caffe google groups forum:
Batchnorm does not converge
The first page was copied below
My model based on 3 convolutions has no problems converging when using dropout and LRN. After i substituted both layers out with a batch norm layer, it no longer converges. I believe im using it correctly but doesn't work no matter how small the learning rate i set is. Can anyone here shed some light on why?
Original Model:

layer {
name: "conv1"
type: "Convolution"
bottom: "data"
top: "conv1"
param {
lr_mult: 1
}
param {
lr_mult: 2
}
convolution_param {
num_output: 48
pad: 0
kernel_size: 5
stride: 1
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "drop1"
type: "Dropout"
bottom: "conv1"
top: "conv1"
dropout_param {
dropout_ratio: 0.5
}
}
layer {
name: "relu1"
type: "ReLU"
bottom: "conv1"
top: "conv1"
}
layer {
name: "norm1"
type: "LRN"
bottom: "conv1"
top: "norm1"
lrn_param {
local_size: 3
alpha: 0.0001
beta: 0.75
}
}
layer {
name: "pool1"
type: "Pooling"
bottom: "norm1"
top: "pool1"
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
.... Repeat x3
New Model:

layer {
name: "conv1"
type: "Convolution"
bottom: "data"
top: "conv1"
param {
lr_mult: 1

}
param {
lr_mult: 2

}
convolution_param {
num_output: 48

pad: 0

kernel_size: 5

stride: 1

weight_filler {

  type: "xavier"

}

bias_filler {

  type: "constant"

}

}
}
layer {
name: "relu1"
type: "ReLU"
bottom: "conv1"
top: "conv1"
}
layer {
name: "conv1_BN"
type: "BatchNorm" include { phase: TRAIN}
bottom: "conv1"
top: "conv1_BN"
param {
lr_mult: 0

decay_mult: 0

}
param {
lr_mult: 0

decay_mult: 0

}
param {
lr_mult: 0

decay_mult: 0

}
batch_norm_param {
use_global_stats: false

moving_average_fraction: 0.95

}
}
layer {
name: "conv1_BN"
type: "BatchNorm" include { phase: TEST}
bottom: "conv1"
top: "conv1_BN"
param {
lr_mult: 0

decay_mult: 0

}
param {
lr_mult: 0

decay_mult: 0

}
param {
lr_mult: 0

decay_mult: 0

}
batch_norm_param {
use_global_stats: true

moving_average_fraction: 0.95

}
}
layer {
name: "pool1"
type: "Pooling"
bottom: "conv1_BN"
top: "pool1"
pooling_param {
pool: MAX

kernel_size: 2

stride: 2

}
}
.....repeat x3

Try adding the following to the batch_norm_param fields:
scale_filler {
type: "constant"
val: 1
}
bias_filler {
type: "constant"
val: 0
}

Are you using the latest version of CuDNNv4.0.5 and nv-caffe 0.14? In particular, there were changes in CuDNN release which fixed some issues around batch normalization that were present in the release candidates (v4.0.4 and earlier). From the release notes:
UPDATES SINCE RELEASE CANDIDATE

The API of cudnnBatchNormalizationBackward has been changed to include an
additional set of scaling parameters (alphaParamsDiff and betaParamsDiff) applied
to the dBnScaleResult and dBnBiasResult outputs of the function.

The prior restriction of batch size 512 in all Batch Normalization routines has been
removed.

Numerical stability and performance of cudnnBatchNormalizationBackward in
some cases has been improved.

Yes, it was nv-caffe 0.14 that was retrieved around Feb 2016. I only installed the latest caffe after caffe-nv was failing and did not upgrade CuDNN so I am not sure which version of CuDNN I currently have (will report later).
This leads me to believe that the caffe-nv version was the cause. I'll try slayton's advice and will report back.

Actually it looks like we're at 4.0.7 now, sorry:
$ dpkg -s libcudnn4
Package: libcudnn4
...
Version: 4.0.7

@sodeypunk
I've just come across a slightly different batch normalization setup that is working for me in DIGITS.  There is a prototxt file posted in the DIGITS user group: Digits3 GoogLeNet Batch Normalization?
After changing some old names ("BN" to "BatchNorm" and "shift_filler" to "bias_filler"), this is how DIGTIS batch normalization layer followed by its activation looks like:
## BatchNorm
layer {
  bottom: "conv1/7x7_s2"
  name: "conv1/7x7_s2/bn"
  top: "conv1/7x7_s2/bn"
  type: "BatchNorm"
  param {
    lr_mult: 1
    decay_mult: 0
  }
  param {
    lr_mult: 1
    decay_mult: 0
  }
  batch_norm_param {
    scale_filler {
      type: "constant"
      value: 1
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layer {
  bottom: "conv1/7x7_s2/bn"
  top: "conv1/7x7_s2/bn"
  name: "conv1/relu_7x7"
  type: "ReLU"
}

Now, a few observations (hopefully to get my head around it too). First, there is no mention of use_global_stats and no separate TRAIN and TEST definitions. Both train_val and deploy proto have the same layer definition. If I got it right, this is because Caffe automatically infers the correct state for the use_global_stats variable, as mentioned in https://github.com/BVLC/caffe/issues/3347

This means by default (as the following is set in batch_norm_layer.cpp), you don't have to set use_global_stats at all in the prototxt.
use_global_stats_ = this->phase_ == TEST;

Second, this approach is different. There are only two lr_multi parameters here and they are not all set to zero like suggested elsewhere. I'm guessing that these two correspond to gamma and beta parameters mentioned in the paper (Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift). I'm not sure what exactly do the three lr_multi mean, except that they relate to some global parameters which the solver isn't supposed to update in that approach. I got that from here: is Batch Normalization supported by Caffe?

The parameters are the collected batch norm statistics. The parameter learning rates need to be set to zero or else the solver will think these are learnable parameters that need to have updates applied to them (with momentum, weight decay, and so forth).

Finally, there is a difference in the current BVLC and NVIDIA proto. The BVLC's doesn't have scale_filler and bias_filler in message BatchNormParameter while NVIDIA's does. https://github.com/NVIDIA/caffe/blob/caffe-0.14/src/caffe/proto/caffe.proto#L465-L483
https://github.com/BVLC/caffe/blob/master/src/caffe/proto/caffe.proto#L493-L503

@mfernezir you may be interested in https://github.com/BVLC/caffe/pull/3919. It seems like you've got a good grasp of the situation, so please comment on that PR if you have any suggestions!

@lukeyeager I've added a comment about batch normalization layer usage. I'm not sure if there are some remaining issues there, hope this helps!

@mfernezir I'm so confused by a lot of batch norm implementations!
If I use your batch normalization i get bad results on accuracy. Using a standard alexnet train_val with the add of your  batchnorm before every convolution layer:
layer {   name: "bn0"   type: "BatchNorm"   bottom: "data"   top: "bn0"   batch_norm_param {     scale_filler {       type: "constant"       value: 1     }     bias_filler {       type: "constant"       value: 0     }   }    param {     lr_mult: 1     decay_mult: 0   }   param {     lr_mult: 1     decay_mult: 0   } }
With this setup the train loss goes down while the accuracy keeps at zero. If I use classify one to see the batch norm effect, I get wrong(very high) mean and variance on data and weights..
If I use this batch norm param form:
layer {   name: "bn0"   type: "BatchNorm"   bottom: "data"   top: "bn0"   batch_norm_param {     moving_average_fraction: 0.98     use_global_stats: false     scale_filler {       type: "constant"       value: 1      }     bias_filler {       type: "constant"       value: 0      }   }   param {     lr_mult: 1     decay_mult: 1   }   param {     lr_mult: 1     decay_mult: 1   }     param {     lr_mult: 0     decay_mult: 0   }   param {     lr_mult: 0     decay_mult: 0   }   }
I get an accuracy that keeps moving up and down, somewhat converging but not in a uniform way.
If I try to visualize the weights via classify one, I get the mean 0 var 1 effect of batch normalization, but the network seems to always missclassify the sample.
I'm using nvcaffe v0.15: https://github.com/drnikolaev/caffe/tree/caffe-0.15

@engharat I've had some (limited) success using the following notation:
layer {
  name: "bn0"
  type: "BatchNorm"
  bottom: "data"
  top: "bn0"
  batch_norm_param {
    scale_filler {
      type: "constant"
      value: 1
    }
    bias_filler {
      type: "constant"
      value: 0
    }
    moving_average_fraction: 0.95
    engine: CUDNN
  }
}

The use_global_stats parameter is automatically derived during run-time (see batch_norm_layer.cpp). Using engine: CUDNN ensures that the CUDNN implementation is used, since the normal CUDA implementation and the CUDNN implementation are totally incompatible and need different parameters to work.
For me, adding BN with this setup resulted in slightly faster convergence, but also slightly lower accuracy (it's not an AlexNet, but close):

Also, try to start with just one BatchNorm layer, as I had problems when using too many of them, too! And you might need to adjust your learning rate parameter (in my case, lowered by a factor of 4).
Good luck!

@dgschwend I'll try your implementation in the next minutes! Meanwhile, could you please confirm me that you get mean 0 var 1.0 out of the batchnorm layers when visualizing weights with digits classify one?
I suppose your batch normalization is not working properly. While the slightly faster-slightly worse convergence is just suspicious, the main benefit of this technique is to use much higher learning rates. The 2nd version of batch norm I posted, even with all this weird behaviour, enable me to use 0.08 learning rate instead of the usual 0.01 on alexnet!
And, even if the use_global_stats parameter is automatically derived, If I remove use_global_stats: false I get no mean/var normalization on batchnorm output layer when using classify one weight visualization digits option! I'm starting to think that maybe the weight visualization script is not inferring correctly the use_global_stats.

@engharat I get
mean: -0.0359341 / var: 0.332215
mean: 0.127164 / var: 0.502211
mean: 0.092401 / var: 0.413767
and
mean: 0.175899 / var: 1.06823
mean: 0.125708 / var: 0.688624
mean: 0.19241 / var: 0.722601
in the two BN layers, using three arbitrary test images. I'd say that's not too far from the ideal mean 0 / var 1.

@engharat Note that in the setup I found in DIGITS user group BN comes after each convolutional layer and just before activations (ReLU). This may or may not have some importance regarding your issues. I haven't tried using BN directly after Data layers like you are doing here. Also, there might be some differences between your Caffe fork and the current NVIDIA's one. I'm using 0.14.2.
Here are my GoogleNet v3 training and deploy files generated with DIGITS. Some notes: this is modified for a 151 class problem, xavier initialization is changed to msra and most importantly, this is an older format generated with DIGITS 3.2. There is a change in DIGITS 3.3. which requires slightly different syntax to determine test and train layers (I haven't moved to that yet).
This network had severe overfitting issues for the problem at hand, but no convergence issues. In the end I used different setups and smaller images, but also including one net with BN layers with good results (81% accuracy for my problem). However, I can't actually attribute the effect to BN layers since another version without them ended up practically the same (with differences in training procedure and intermediate results).
I didn't specify the engine parameter manually like @dgschwend did, but hopefully Caffe inferred it correctly with engine parameter set to "DEFAULT". I'm using CUDNN4.
v3-inception-151class.zip

@sodeypunk you can put BatchNorm before ReLU, and have a try

@mfernezir Thanks for your provided network! It doesn't seems a v3
inception, it looks more the v1 googlenet with added batchnorm!
2016-04-14 23:02 GMT+02:00 mfernezir<EMAIL_ADDRESS>
@engharat https://github.com/engharat Note that in the setup I found in
DIGITS user group BN comes after each convolutional layer and just before
activations (ReLU). This may or may not have some importance regarding your
issues. I haven't tried using BN directly after Data layers like you are
doing here. Also, there might be some differences between your Caffe fork
and the current NVIDIA's one. I'm using 0.14.2.
Here are my GoogleNet v3 training and deploy files generated with DIGITS.
Some notes: this is modified for a 151 class problem, xavier initialization
is changed to msra and most importantly, this is an older format generated
with DIGITS 3.2. There is a change in DIGITS 3.3. which requires slightly
different syntax to determine test and train layers (I haven't moved to
that yet).
This network had severe overfitting issues for the problem at hand, but no
convergence issues. In the end I used different setups and smaller images,
but also including one net with BN layers with good results (81% accuracy
for my problem). However, I can't actually attribute the effect to BN
layers since another version without them ended up practically the same
(with differences in training procedure and intermediate results).
I didn't specify the engine parameter manually like @dgschwend
https://github.com/dgschwend did, but hopefully Caffe inferred it
correctly with engine parameter set to "DEFAULT". I'm using CUDNN4.
v3-inception-151class.zip
https://github.com/NVIDIA/DIGITS/files/219893/v3-inception-151class.zip
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/NVIDIA/DIGITS/issues/629#issuecomment-210147458

@engharat
Ah yes, it's just v1 with LRN repleaced with BN and added BN after convolutional layers. I haven't really used that net besides one quick run. I just wanted to try BN layers and then I used them on another net.
In any case, I hope you've got your network running.

I have some quite interesting behaviour to show you. I still cannot let batchnorm perform correctly.
Just to remember, I'm training from scratch on a big private and quite easy dataset. easy means that on few epochs any network can reach 95% of accuracy.
Here is an example of the first output(so the first 3-4 inception layers I don't remember) of a GoogleNet V1, when NOT using batch normalization:

Everything goes smoothly.
The same network same first output but with added batch normalization:

The accuracy is jumping wildly until the learning rate goes very small.
And here some other examples on a slightly modified dataset,still pretty easy; lets remember I should get 70-80% accuracy after 2 epochs. I tried different batchnorm hyperparameters:
this is wihe esponential moving average parameter 0.95:

This is with 0.999 (that should be the default value):

And finally with the use_global_stats= false

After I tried every solution I get into my mind in order to let this accuracy not oscillate ( and not being at 6% after 2 epochs!) I tried an interesting experiment. I have extracted the features from the last layer of my network and with those features I have trained a linear SVM on some other data. This is just to check how the output layer is performing as a feature extractor.
The network without batchnorm reached 73% of average accuracy on the SVM. The network with batchnorm that I let converge to the high ninties gave me 75% of accuracy on the SVM. But the most incredible thing(for me) is that when I train the SVM on the networks that after 2 epochs are at 10% accuracy, I get 70% on SVM!
This means that the accuracy values that I get from digits during training are completely false with the batchnorm. Something is not performing as it should while on validation data, while if I use a network snapshot via python code and I extract the layer values for training an SVM everything is working fine.
For mfernezir: this is what happens if I use your batchnorm parameters:

As you can see, after 1 epoch I'm on 0.38 on train loss, that correspond to 85-90% of accuracy from the other tests. But still the accuracy calculated by digits is wrong (24%)
I'm thinking it is something related to the esponential moving average, to the use of accumulated statistics when training/validating and when testing...

I've also been meaning to comment again since I've tried a few more batch normalization networks in the last couple of  days. I have also observed similar strange effects: low training loss and small validation accuracy on high learning rates and then sudden validation accuracy jumps on low learning rates.
There is an interesting comment and code change in the BVLC version which should solve problems with validation accuracy calculations:
https://github.com/BVLC/caffe/pull/3919#issuecomment-215174898

You could try using another method to compute accuracy: on DIGITS 3.3, on the model page after training is finished, select the model corresponding to epoch 2. Then in "Test a list of images", upload the val.txt file from your dataset job folder. Click "Classify many". This will show a new page with Top-1 accuracy, a confusion matrix, etc. (provided it doesn't time out as you have quite a few images in the validation set).

The issue is that batch normalization networks have to calculate global mean and variance statistics from batch statistics and then store these values inside the actual model to be used for inference. If these calculations are wrong in some extreme cases like mentioned in the above BVLC comment, then no matter how we try to calculate validation or test accuracy the results would be wrong.
Still, I've just run some inference tests with DIGITS 3.3.0. I have a couple of different nets trained on the same dataset, some having BN layers and some without. The problem is that I've run into another bug. All of my BN networks use scaling parameter in the Data Layer, for example:
layer {
  name: "data"
  type: "Data"
  top: "data"
  top: "label"
  include {
    phase: TEST
  }
  transform_param {
    scale: 0.0078125
    crop_size: 227
  }
  data_param {
    batch_size: 32
  }
}

However, this parameter is not written in the deploy file. This causes much lower accuracy when classify many is used when compared to validation accuracy reported by graphs. This is also the case for another network that doesn't have BN layers but does have scaling. For that network, on random 50 validation images accuracy reported with classify many is zero percent while it should be around 25%. Models without scaling parameters (and batch normalization) have similar classify many and graph validation accuracies, but even this can't be said decisively since my validation set is around 100k images.
Maybe engharat could confirm that graph accuracies and classify many accuracies coincide, but this is likely not relevant for the underlying Caffe issue.

Indeed the Data layer isn't propagated to the deploy network (since this layer type is for reading from an LMDB). Can you use a Power layer instead (like there)?.
Thanks a lot for your detailed analysis of the BN issue. cc @drnikolaev - can you suggest anything to make progress on this?

Yep, I could use power layer instead and I'll do that for convenience in future networks.

Seems the problem is cudnnBatchNormalizationForwardTraining produce wrong running mean or variance. I've tried replace cudnnBatchNormalizationForwardInference with standard BN code for blobs 3 and 4, result was exactly the same. Another small problem is that epsilon_ is not initialized, but fixing it is not helping.

I was struggling with the convergence issue, but finally the following worked for me. Specifying the engine as CAFFE is important. CUDNN BatchNorm doesn't converge for me.
layer {
name: "bn2"
bottom: "conv2"
top: "conv2"
type: "BatchNorm"
param { #scale
lr_mult: 1
decay_mult: 1
}
param { #shift/bias
lr_mult: 1
decay_mult: 1
}
param { #global mean
lr_mult: 0
decay_mult: 0
}
param { #global var
lr_mult: 0
decay_mult: 0
}
batch_norm_param {
scale_filler {
type: "constant"
value: 1
}
bias_filler {
type: "constant"
value: 0
}
engine: CAFFE
}
}

@mathmanu  I just verified that "engine: CAFFE" is what makes it train vs not train.
@lukeyeager  - any idea why engine: CUDNN does not converge?  Looks like there's almost a 10x slowdown when using engine: CAFFE (but at least it's working)

Top and bottom blobs need to be different for engine:CUDNN BatchNorm. This constraint is not there for engine:CAFFE BatchNorm. This is the reason for non-convergence.
See the following thread for a working prototxt with CUDNN BatchNorm. https://github.com/BVLC/caffe/pull/3919
@borisgin @lukeyeager - Since others are also facing the same issue and wondering about the convergence issue , why not put a check/exit in CUDNN BatchNorm reshape function, if the top and bottom blobs are same - this will save a lot of headache.

Thanks, that cleared things up a lot.  Sad to see such divergence between BVLC/Caffe and NVIDIA/Caffe. I can't keep all the differences straight.  Hope there will be more similarity in the future so that networks can be ported from one to the other without much effort.

And the saddest thing is that you cannot test/use any BVLC pretrained batchnorm networks on Nvcaffe, because the two batchnorm formats are not compatible ;if when working on prototxt you can still translate between the two formats, in order to use pretrained inception/residual networks you have to install bvlc caffe.
Anyway, I think they bvlc caffe should converge towards nvcaffe:if you train on those two forks on classic alexnet you will find very little difference, but if you train a heavily batchnormed network you will find that bvlc is 50% slower in training speed and use 30% more GPU memory(slowing down further because forcing you with a smaller batch size)  so nvcaffe is winning down easily.

After reading the posted link by mathmanu,  I see that bvlc caffe is converging to the CuDNN batchnorm with all the boon and the boost that it brings - still I don't fully understand if with that CuDNN implementation we can seamlessly switch  bvlc caffe and nvcaffe caffemodel networks,or if there still is some different implementation that would rise some mismatch.

My suggestions to fix these issues are the following:

Rename the NVIDIA/caffe's BatchNorm to BatchNormScale, since it now includes Scaling as well.

Put a check/exit in CUDNN BatchNormScale reshape function, if the top and bottom blobs are same - so that the user will get a warning.

Fix the inconsistency in blob shape between engine:CAFFE and engine:CUDNN

In BatchNormScale, If you change the oder of the blobs to: gloabl_mean, and global_variance, scale, bias, global_counter, then I don't have to specify 4 param fields for lr_mult and decay_mult - but only 2.

If the definition of scale and bias fields in BatchNormParameter is changed to:
optional float scale_filler = 5 [default = 1];
optional float bias_filler = 6 [default = 0];
Then I don't have to specify these also in the prototxt.

Keep the original BatchNorm from BVLC/caffe as it is, un-touched - so that compatibility to BVLC/caffe is not affected and old BVLC/caffe models can be used for fine tuning. If possible, provide a CUDNN version of this original BatchNorm without scaling as well, so that it can be accelerated.


Adding  a  WAF feature which implements ML

Todo

[ ] Building a Proxy Server to intercept HTTP/HTTPS traffic.

[ ] Building the Dataset for Kmeans Clustering

[ ] Training the model to create two clusters one for good and bad traffic.

[ ] Feature Extractions from the web traffic

[ ] Implementing Logging and Alerting Mechanism

[ ]  Testing the feature.

The proxy server should be able to intercept concurrent web traffic and analyze the traffic to extra features from it.

Dataset can be built by simulating OWASP top 10 Vulnerabilities, extracting the logs from the burp suite and other sources of web traffic.

The web traffic ie the Request should be parsed to extract features like <>, single quote , double quote, etc to be sent to the model which would predict whether the request is malicious.

More research should be done to understand what features should be extracted so that we can cover major attack scenarios.

@prince-7  Currently these are things that I have planned and will update,if anything comes up and feel free to add your ideas or suggestions.

Submitted an updated version of the same


Verbose API results

This PR is dependent on #74

TSD either returns a null array if there is no test failures or an array of Diagnostic objects. This PR aims to make TSD return with ExtendedDiagnostic object no matter if the tests fail or not.
Merging this PR allows custom test runners like Jest to report the number of assertions it evaluated in each file.
It also means that it introduces a breaking change because TSD now returns...

An object with numTests and null diagnostics array if all tests passed.

{
    numTests: 1024, // Total number of tests it ran through
    diagnostics: [] // No failed test cases
}

An object with numTests and an diagnostics array populated with test fail reports if a test is failed.

{
    numTests: 1024, // Total number of tests it ran through
    diagnostics: [
        {
            fileName: 'add.test-d.ts',
            message: 'A message here',
            severity: 'error',
            line: 10,
            column: 34,
        },
        ...
    ]
}

Unit tests have been added
A more detailed explanation can be found here - Reporting Test Count

This PR is Co-Authored By:

Karan Sanjeev<EMAIL_ADDRESS>SaurabhAgarwala<EMAIL_ADDRESS>

rebase this now that #74 has landed?

Still very interested in seeing this land 😀

I'm fine with this change. Having an object also gives us the opportunity to add additional properties in the future.

Instead of making a breaking change, maybe tsd can just have verbose: boolean option?

I don't like options that change the return value type.
I think a breaking change is ok here. Programmatic use is not that common.


Power Toys download issue

Microsoft PowerToys version
0.68.1
Installation method
GitHub
Running as admin
No
Area(s) with issue?
System tray interaction
Steps to reproduce
Windows 11 app finder cannot find power Toys. The download ocurred but the file is not in the download folder and cannot be found. The Power Toys Setting can be found and that is it. Cannot add Power Toys to the tray because Windows cannot find the app.
✔ Expected Behavior
To see the downloaded169Mb file in my Windows download folder - not there
To be searchable using the Windows 11 pro app search query-box and/or Windows icon - it cannot be found, just invitations to download again from the web!
❌ Actual Behavior
Download was shown as completed and installation also shown as completed from the github web-screen.
None of the expected above functions are available.
Other Software
Windows 11 pro 10.0.22621
DELL XPS 8940 x 64 (i7-10700 cpu at 2.90 GHz, 8 cores - mem 16GB

/reportbug
are you installing under a different user?

Clint Rutkas writes, 'You right click on our system tray icon and just go to report bug. Then drag the zipfile from your desktop onto the GitHub comment box in this issue. Thanks!'
....but I have done the report bug and have no such file appearing on my desktop!!!!
On 4/6/2023 3:50:00 PM, microsoft-github-policy-service[bot] @.***> wrote:
Hi there!
We need a bit more information to really debug this issue. Can you add a "Report Bug" zip file here?  Then drag the zipfile from your desktop onto the GitHub comment box in this issue. Thanks!
[Report Bug] [https://user-images.githubusercontent.com/11349917/133042052-4975be21-4699-4363-83c9-a8e1869d079d.png]
—
Reply to this email directly, view it on GitHub [https://github.com/microsoft/PowerToys/issues/25170#issuecomment-1499538142], or unsubscribe [https://github.com/notifications/unsubscribe-auth/A65OAW5H24SOJZP6YOG7NGLW74M6RANCNFSM6AAAAAAWQO2OJE].
You are receiving this because you authored the thread.Message ID: @.***>
[cb254772-dd40-4996-9880-69231ce2e17b]

as @.***
On 4/6/2023 3:49:54 PM, Clint Rutkas @.> wrote:
/reportbug
are you installing under a different user?
—
Reply to this email directly, view it on GitHub [https://github.com/microsoft/PowerToys/issues/25170#issuecomment-1499538060], or unsubscribe [https://github.com/notifications/unsubscribe-auth/A65OAW2JVTTQCARA72ECLFLW74M6FANCNFSM6AAAAAAWQO2OJE].
You are receiving this because you authored the thread.Message ID: @.>
[6eec8d1c-4988-4c7c-b41d-35455a544f85]

Not able to repro this, going to close. Let us know if you see this again!


Summary - Extending Conductor

Link

https://netflix.github.io/conductor/extend/

Backend

Conductor provides a pluggable backend. The current implementation uses Dynomite.
There are 4 interfaces that needs to be implemented for each backend

//Store for workflow and task definitions
com.netflix.conductor.dao.MetadataDAO

//Store for workflow executions
com.netflix.conductor.dao.ExecutionDAO

//Index for workflow executions
com.netflix.conductor.dao.IndexDAO

//Queue provider for tasks
com.netflix.conductor.dao.QueueDAO

System Tasks

To create system tasks follow the steps below:

Extend com.netflix.conductor.core.execution.tasks.WorkflowSystemTask
Instantiate the new class as part of the startup (eager singleton)

Check it

I should extend conductor backeds
dynomite to redis cluster or other reliable solution
not sure for replacing ES5


UD-1648: update tags to define floating tag in addition to version

Description
update the release workflow to include an additional floating tag based on the version's major/minor
Linked Issues
How has this been tested?
Checklist

[X] I have labeled this PR with the relevant Type labels
[ ] I have documented my code (if applicable)
[ ] My changes are covered by tests

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

I'll need to push up a new commit, the second commit isn't signed by github so we cannot merge

Misread the warning, the commit had to be cryptographically signed.


[CTASection]: passing Onclick to the cta does not work

Detailed description

I'm trying to use the CTA Section component, with CTA as a button. The buttons work properly except for the onClick property, that seems to be ignored. This is how I'm calling the component:

<CTASection {...props.contentBlockProps} cta={{ style: "button", type: "local", buttons: [{ type: "local", copy: "CTA 1", onClick: () => console.log(true) }] }} />
https://www.ibm.com/standards/carbon/react/?path=/story/components-cta-section--default
I used a similar approach in the LeadSpace component and it works fine.
Could you help me figured out this issue, please?
Inspecting the CTASection component code, on the "@carbon/ibmdotcom-react" package, seems that the prop type for "onClick" is not there

General

CTA Section component

The onClick should work as on the cta on leadspace

Google Chrome

looks like the onClick is being overwritten here: https://github.com/carbon-design-system/carbon-for-ibm-dotcom/blob/master/packages/react/src/components/CTA/ButtonCTA.js#L79

Opening this ticket in the main carbon-for-ibm-dotcom repo here: https://github.com/carbon-design-system/carbon-for-ibm-dotcom/issues/6844


Add ability to change shopcart product qty

Hello, the CI build check is not passed. I think you need to correct the code to make it all good.


AJ-517 first pass describeRecordType

Adds API to describe the schema of a recordType.

Do we need to add a count for the number of records in a record type?

Yes, I didn't do it at first because it's not explicitly required in the ticket, but I've added it in the describe-all-types ticket which I'm also working on.  I can port it over.

All the sonarcloud failures were previously existing/getting fixed in a concurrent PR.


Generate separated statements for easy debugging

Description
Currently, the function body is concatenated as one big single statement. It is very hard to debug in browser,
For example, the following FSharp code:
let parse s =
    let input = Input.init s
    let result = Parser.run integer input
    match result with
    | Failure e -> Failure e
    | Success (v, next) -> Success v

Compile to (formatted by hand):
var parse = exports.parse = function (s) {
  var input, result, v, next, e;
  return input = (0, _Input.init)(s),
    result = (0, _Parser.run)(integer, input),
    result.Case === "Success"
      ? (
        v = result.Fields[0][0],
        next = result.Fields[0][1],
        new _DomainTypes.Result("Success", v))
      : (
        e = result.Fields[0],
        new _DomainTypes.Result("Failure", e));
};

This could be much better:
var parse = exports.parse = function (s) {
  var input = (0, _Input.init)(s);
  var result = (0, _Parser.run)(integer, input);
  return result.Case === "Success"
    ? (function () {
      var v = result.Fields[0][0];
      var next = result.Fields[0][1]
      return new _DomainTypes.Result("Success", v))()
    : (function () {
      var e = result.Fields[0];
      return new _DomainTypes.Result("Failure", e))();
};

Then, we can set breakpoint on each var statement to see what is returning.

Yeah, this was one of the caveats as this is the way do expressions compile in Babel. Using do expressions makes the compilation much simpler and also the generated code is more compact and (hopefully) more performant. The problem, as you noted, is debugging is not ideal.
We could consider generating debug (with separated statements) and production code. Unfortunately, we're a small team and we have to focus on other priorities at the moment :( But definitely this will be something to look at in the future.

As it happened other times, finally I yielded to reason and removed do expressions 😉  The debugging experience should be much better now. You can check it with the new alpha version: fable-compiler@next and fable-core@next. Your sample above now compiles as follows:
var parse = exports.parse = function (s) {
  var input = Input.init(s);
  var result = Parser.run(integer, input);

  if (result.Case === "Success") {
    var v = result.Fields[0];
    var next = result.Fields[1];
    return new _DomainTypes.Result("Success", v);
  } else {
    var e = result.Fields[0];
    return new _DomainTypes.Result("Failure", e);
  }
};

Please feel free to reopen the issue if you're not still happy with the results :+1:


Add snippets support

Incorporate snippets support for v3 & v4 scripts

Hey @Leonx254 thanks for your PR. Is event instead of sub for v4? Do you have any code example I can see to verify the intent of this code change?

Thanks for the response, event is in fact the right term, discovered thanks to origins including the script compiler for v4, we had to do a soft-rework to the scripts to allow origins to run s1/s2 scripts.
You can find more examples in the script decomp github, on the cleanup branch, but i'll left one here for a quick check.
Invincibility.txt

Oh, I was not aware of the script decompiler rework. Let me know as soon as you do not intend to make any further change to the PR, so I can merge it and push the new version on the VSCode Marketplace :)

i have a friend doing a checkup because i got some things wrong from the outdated handbook, i'll let you know once i get his PR.
Thanks for your time!

Sorry for taking so long, we've been busy finishing up the cleanup for public testing.
My friend couldn't get time to make the PR, but instead pointed me most of the errors i got so i could fix it myself.
I don't expect to do more updates for now, so this is ready for merge

@Xeeynamo I'm sorry, i was supposed to mention you directly? i'm not really used to doing PR's


SubPageLink - field mapping

In C/AL we were able to add a SubPageLink where a Text field could be linked to an integer field as follows:
SubPageLink = "Table No."=CONST(454), "Key Field 1"=FIELD("Entry No.");
Where "Key Field 1" is a text field and "Entry No." is an integer.
In AL I get the following error:

Hi @nmahate, could you tells us if there is any reason not to use a matching data type?

We have a factbox based on RecordID, however in v1 we couldn't add code to the page, so we couldn't pass the RecordID to the factbox
We made use of FlowFilters on the source table for the Factbox, by passing the table id and the primary key values using the subpage link. This way we were able extract the values to build the RecordID.
As you can see from the screenshot, the table id is set to the flowfilter field "Table No." and the primary key, "Entry No.", to "Key Field 1". I am aware that now with Page Extensions we're able to add code to call a function on the factbox, to set the RecordID.

Does it mean then, that using page extensions you can now refactor the solution a bit and have the same data type in the related table?

It means we can call a function we have on our Factbox to set the RecordID. So they can be linked
Added the following code to our Page Extension
    trigger OnAfterGetCurrRecord()
    begin
        CurrPage."My Factbox".Page.SetDocumentRecordID(RecordId);
    end;

With v1 extensions we couldn't add that line of code. Which made us rethink, and have to use FlowFilters instead.
So, we've found an alternative way to do the linking for our factbox. But wanted to point out that the SubPageLink doesn't work as it did in C/AL

Thanks for the update. We'll see if this is a common problem.

@dennisfrostlander We face the same problem when moving to extensions. We have a 'documents' factbox that is added to hundreds of pages using the 'SubpageLink' hack to pass filters and get the recordref / recordid of the master record via the filters in the factbox. In C/AL, we could use dummy text fields to pass any kind of datatype for the primary key field values, in AL we've to create dummy fields for each data type in the primary key :(
It would help us a lot if we could just pass 'RecordID' as value in the subpagelink.
Tried the solution of @nmahate , but this does NOT work in all cases, so no valid workaround for us.

 trigger OnAfterGetCurrRecord()
    begin
        CurrPage."My Factbox".Page.SetDocumentRecordID(RecordId);
    end;

This code is fine while navigating through existing records. When opening a page and inserting a NEW record, the OnAfterGetCurrRecord trigger is not called, so the RecordID of the linked factboxes is not updated. Only way to force an auto update (after insert of the record) in the linked factboxes is by using the 'Subpagelink'....

As a reference (Feel free to upvote)
https://experience.dynamics.com/ideas/idea/?ideaid=4bcb9af5-aa6a-e911-b047-0003ff689546

I will close this issue. Any future readers should vote on the idea so that it can be considered for a future release.


Camera switching stops working after a while

After about 15 minutes of use (active or inactive), the user can no longer switch cameras using x o etc,

Fixed. Turned out to be a controller issue.


Major refactor of survey simulation

This version is a major refactoring of the code to take advantage of the refactoring in desisurvey 0.11.0 and simplify the simulation classes used to track survey statistics and exposure metadata.
Refer to the updated tutorial for an overview (written assuming that 18.12 is already available at NERSC).

The files owned by surveysim that probably need documented data models are:

Survey per-tile and per-night statistics written by surveysim.stats.SurveyStatistics.save
Simulated exposure metadata written bysurveysim.exposures.ExposureList.save

Something similar to these are required during operations, but might duplicate tracking that is already handled better by ICS.  This should become clearer when we finalize the wrapping of desisurvey afternoon planning and next-tile selection into ICS applications.  For now, these contain everything needed to save/restore simulation state and study simulation outputs.

Note to self:

[ ] Change DESISURVEY_VERSION from refactor to 0.11.0 in .travis.yml after merging and tagging desihub/desisurvey#91

Overall, looks good.  One request that is needed for integration with the rest of the end-to-end simulation chain:  this branch broke surveysim.util.add_calibration_exposures because of the fewer columns in the new exposures_surveysim.fits file compared to the table that used to be generated by Progress.get_exposures().
Closely related: Although these are derivable from the TILEID, MJD, ephemeris, etc they would be really convenient to retain pre-calculated in the exposures file:  RA, DEC, EBMV, NIGHT, PASS, PROGRAM, FLAVOR, MOONALT, MOONFRAC, MOONSEP.  I also don't see an obvious way to tell if an exposure was a twilight exposure without crossmatching the MJD to the ephemeris file (e.g. a SUNALT column).
Traceback:
In [1]: from surveysim.util import add_calibration_exposures
   ...:

In [2]: explist = Table.read('exposures_surveysim.fits', 'EXPOSURES')

In [3]: e2 = add_calibration_exposures(explist)
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-3-45c3483604ff> in <module>()
----> 1 e2 = add_calibration_exposures(explist)

/global/cscratch1/sd/sjbailey/desi/code/surveysim/py/surveysim/util.py in add_calibration_exposures(exposures, flats_per_night, arcs_per_night, darks_per_night, zeroes_per_night, exptime, readout)
     49     expid_in_exposures = 'EXPID' in exposures.colnames
     50     for c in ('RA', 'DEC'):
---> 51         if output[c].unit is None:
     52             output[c].unit = 'deg'
     53     current_night = '19730703'

/global/common/software/desi/cori/desiconda/20180709-1.2.6-spec/conda/lib/python3.6/site-packages/astropy/table/table.py in __getitem__(self, item)
   1220     def __getitem__(self, item):
   1221         if isinstance(item, six.string_types):
-> 1222             return self.columns[item]
   1223         elif isinstance(item, (int, np.integer)):
   1224             return self.Row(self, item)

/global/common/software/desi/cori/desiconda/20180709-1.2.6-spec/conda/lib/python3.6/site-packages/astropy/table/table.py in __getitem__(self, item)
    107         """
    108         if isinstance(item, six.string_types):
--> 109             return OrderedDict.__getitem__(self, item)
    110         elif isinstance(item, (int, np.integer)):
    111             return self.values()[item]

KeyError: 'RA'

I will look at fixing add_calibration_exposures now.
(What was the no-op to include [ci skip] commit for?)

@sbailey What per-exposure fields are currently required by the  end-to-end simulation chain?  I propose to add those now, so this PR does not break anything, then we should separately discuss how exposure tracking should work during ops (and who owns the authoritative exposure list).

Trying to answer my own question: the only caller of add_calibration_exposures I found is wrap_newexp in desisim, when then references the following fields:

existing: MJD, TILEID
need to be added: PROGRAM, NIGHT, EXPID, FLAVOR

@sbailey I have updated add_calibration_exposures (and its unit tests) to be backwards compatible for wrap_newexp:  can you test this easily?

@dkirkby I'm trying to get the minitest working with these new branches.  Can you help debug the files in /global/cscratch1/sd/sjbailey/desi/dev/end2end/survey/ ?

test-tiles.fits
desisurvey-config.yaml
generated ephem*.fits, surveyinit*.fits, exposures*.fits, stats*.fits, planner*.fits, scheduler*.fits

There are 10 test tiles across all layers, which the old surveysim was able to observe within a month.  New surveysim is only getting 1 or 2 of the 3 bright tiles.  Perhaps I need a different fiberassign delay strategy for the bright tiles?
It would be helpful if surveyinit et al would use the start/stop dates from the config file instead of hardcoded ranges in desisurvey/ephem.py .  I know I could copy the ephem file from $DESI_ROOT/datachallenge/surveysim2018/shared/, but I would prefer the minitest to be standalone and include running surveyinit without it taking 50 minutes.  For now I can work around it by loading the module, changing ephem.START_DATE and ephem.STOP_DATE before proceeding with programmatically calling surveysim.main(), but it would be helpful to also be able to use the command line with the limited date range too.
Are the planner and scheduler files documented somewhere, even if they aren't in the formal data model yet?  That would be helpful for me trying to understand this.  e.g. I'm not even certain about the conceptual difference between "planner" and "scheduler".
Thanks for the help.

The moon params are not currently factored into the exposure times, except for an average factor, so the BRIGHT tiles simulated by newexp-mock should not be taken too seriously.  With this in mind, I suggest that we fix MOONFRAC=0.5, MOONALT=-10deg and MOONSEP=ALT+10deg.  This will be revisited soon when I implement the full moon exposure-time model.

Summarizing:
I've verified that I can use these desisurvey and surveysim branches with the end-to-end integration test, though it requires some hacks.  I would prefer the following changes before merging, but if they are unduly complicated let's chat options:
Provide options to surveyinit to override the hardcoded first and last days of the ephemeris calculation, which takes a long time, plus an option to surveysim to control which ephemeris file is loaded.  Or provide both with an option to use the first_day and last_day from the config file or similar.
Add columns MOONFRAC, MOONALT, MOONSEP to exposures_surveysim.fits output.  Even if they aren't used by the survey simulations yet, they have well defined values and can be meaningfully simulated and used downstream.  At minimum let's get these into the data model.
Add columns RA, DEC to exposures_surveysim.fits output.  This is less critical since this is an easy join with the tiles table but would still be nice to have; the moon params are harder.

Provide options to surveyinit to override the hardcoded first and last days of the ephemeris
calculation, which takes a long time, plus an option to surveysim to control which ephemeris file is
loaded.

One of the changes in this PR is to decouple the ephemerides date range from the nominal survey start/stop dates.  This is slightly less convenient for testing and special runs, but more robust for commissioning and operations.  Specifically, the ephemerides are now calculated (by default) to cover all of 2019 - 2025, so one file should be sufficient for all needs.  At NERSC, a copy of this file is in:
$DESI_ROOT/datachallenge/surveysim2018/shared/ephem_2019-01-01_2025-12-31.fits

Note that the time-consuming step of surveyinit is not calculating the ephemerides, but optimizing the design hour angles for some nominal survey start / stop dates, which are taken from the config file, as always.  The ephem date range only needs to cover the survey start / stop dates.  For the default start / stop dates, the design hour angles are precomputed in:
$DESI_ROOT/datachallenge/surveysim2018/shared/surveyinit.fits

For testing in an environment where all files need to be downloaded or created from scratch (e.g., travis), I set the ephem and survey date ranges to just one month (Dec 2019).  This is handled in one place for all unit tests by a unittest.TestCase superclass defined in desisurvey.test.base.
For special runs at NERSC, you can copy / link one or both of the files above to your $DESISURVEY_OUTPUT to avoid regenerating them.

OK, the ephemeris calculation takes 2 minutes, not too bad.
MOONFRAC, MOONALT, MOONSEP are in the datamodel but we'll need to address #61 with the expanded exposures table for them to be useful downstream.  I'll work around this for now in the integration tests.
Proceeding with merging desisurvey and surveysim PRs.


Unused functions in data_gathering

I've noticed that we have some old functions that are no longer used, or have been replaced by other functions. I believe that all the the following can be removed:
from the data_gathering module:

extract_domain - this was replaced by search_filters.make_geometry_filter_from_geojson
save_to_csv - we can let the user decide how they want to save these themselves (e.g. pandas df.to_csv)
show_img - we can let the user decide how they want to visualize images (rasterio, xarray, other packages)

This has been addressed with #36


Wa/mobile/feature/create post page 348

create post page is ready except location picker, enes will work on it

LGTM!


Composes should contain frozen blueprint

Currently, we just copy the blueprint to the new compose. This means they can contain wildcard package versions.
However, Lorax does a different thing: Takes all the extra packages and groups, depsolves them and replaces the wildcard versions with specific ones. This process is known in lorax's codebase as blueprint freezing.
This has visible outcomes in some APIs like /compose/info or /compose/metadata. Generally it's a good feature for reproducibility of composes.

This was closed by #214


[HDFS-15786] - Minor improvement use isEmpty

Use isEmpty instead size() > o.
size() can be O(1) or O(N), depending on the data structure; .isEmpty() is never O(N).

@belugabehr this is right in your area :)
Even without the performance improvement, I tend to prefer isEmpty() as it has a clear semantic.

@belugabehr this is right in your area :)
Even without the performance improvement, I tend to prefer isEmpty() as it has a clear semantic.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
5m 53s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s

No case conflicting files found.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 48s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
34m 14s

trunk passed

+1 :green_heart:
compile
6m 38s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-Ubuntu-0ubuntu1.18.04

+1 :green_heart:
compile
5m 34s

trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

+1 :green_heart:
checkstyle
1m 19s

trunk passed

+1 :green_heart:
mvnsite
2m 44s

trunk passed

-1 :x:
shadedclient
8m 15s

branch has errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
2m 10s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-Ubuntu-0ubuntu1.18.04

+1 :green_heart:
javadoc
2m 16s

trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

+0 :ok:
spotbugs
1m 42s

Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
5m 48s

trunk passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 33s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m 21s

the patch passed

+1 :green_heart:
compile
6m  0s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-Ubuntu-0ubuntu1.18.04

+1 :green_heart:
javac
6m  0s

the patch passed

+1 :green_heart:
compile
5m 49s

the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

+1 :green_heart:
javac
5m 49s

the patch passed

+1 :green_heart:
checkstyle
1m 13s

the patch passed

+1 :green_heart:
mvnsite
2m 26s

the patch passed

+1 :green_heart:
whitespace
0m  0s

The patch has no whitespace issues.

+1 :green_heart:
shadedclient
20m 57s

patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
2m  2s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-Ubuntu-0ubuntu1.18.04

+1 :green_heart:
javadoc
2m  5s

the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

+1 :green_heart:
findbugs
5m 10s

the patch passed

_ Other Tests _

+1 :green_heart:
unit
2m 22s

hadoop-hdfs-client in the patch passed.

-1 :x:
unit
4m 36s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt
hadoop-hdfs-httpfs in the patch passed.

-1 :x:
unit
9m 29s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
hadoop-hdfs-rbf in the patch passed.

+1 :green_heart:
asflicense
0m 32s

The patch does not generate ASF License warnings.

157m  7s

Reason
Tests

Failed junit tests
hadoop.fs.http.server.TestHttpFSServerNoACLs

hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem

hadoop.fs.http.server.TestHttpFSServer

hadoop.fs.http.server.TestHttpFSServerWebServerWithRandomSecret

hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem

hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem

hadoop.fs.http.server.TestHttpFSAccessControlled

hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem

hadoop.fs.http.server.TestHttpFSServerWebServer

hadoop.lib.service.hadoop.TestFileSystemAccessService

hadoop.fs.http.server.TestHttpFSServerNoXAttrs

hadoop.hdfs.server.federation.metrics.TestMetricsBase

hadoop.hdfs.server.federation.router.TestRouterRPCClientRetries

hadoop.hdfs.server.federation.router.TestRouterAllResolver

hadoop.hdfs.server.federation.store.TestStateStoreMembershipState

hadoop.hdfs.server.federation.router.TestRouterFaultTolerant

hadoop.hdfs.server.federation.metrics.TestRBFMetrics

hadoop.hdfs.server.federation.resolver.order.TestAvailableSpaceResolver

hadoop.hdfs.server.federation.store.TestStateStoreRouterState

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2641/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2641

JIRA Issue
HDFS-15786

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 672fd0dba917 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / d09e3c929f2

Default Java
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-<IP_ADDRESS>+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2641/1/testReport/

Max. process+thread count
2919 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2641/1/console

versions
git=2.17.1 maven=3.6.0 findbugs=4.0.6

Powered by
Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
5m 53s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s

No case conflicting files found.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 48s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
34m 14s

trunk passed

+1 :green_heart:
compile
6m 38s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-Ubuntu-0ubuntu1.18.04

+1 :green_heart:
compile
5m 34s

trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

+1 :green_heart:
checkstyle
1m 19s

trunk passed

+1 :green_heart:
mvnsite
2m 44s

trunk passed

-1 :x:
shadedclient
8m 15s

branch has errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
2m 10s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-Ubuntu-0ubuntu1.18.04

+1 :green_heart:
javadoc
2m 16s

trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

+0 :ok:
spotbugs
1m 42s

Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
5m 48s

trunk passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 33s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m 21s

the patch passed

+1 :green_heart:
compile
6m  0s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-Ubuntu-0ubuntu1.18.04

+1 :green_heart:
javac
6m  0s

the patch passed

+1 :green_heart:
compile
5m 49s

the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

+1 :green_heart:
javac
5m 49s

the patch passed

+1 :green_heart:
checkstyle
1m 13s

the patch passed

+1 :green_heart:
mvnsite
2m 26s

the patch passed

+1 :green_heart:
whitespace
0m  0s

The patch has no whitespace issues.

+1 :green_heart:
shadedclient
20m 57s

patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
2m  2s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-Ubuntu-0ubuntu1.18.04

+1 :green_heart:
javadoc
2m  5s

the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

+1 :green_heart:
findbugs
5m 10s

the patch passed

_ Other Tests _

+1 :green_heart:
unit
2m 22s

hadoop-hdfs-client in the patch passed.

-1 :x:
unit
4m 36s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt
hadoop-hdfs-httpfs in the patch passed.

-1 :x:
unit
9m 29s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
hadoop-hdfs-rbf in the patch passed.

+1 :green_heart:
asflicense
0m 32s

The patch does not generate ASF License warnings.

157m  7s

Reason
Tests

Failed junit tests
hadoop.fs.http.server.TestHttpFSServerNoACLs

hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem

hadoop.fs.http.server.TestHttpFSServer

hadoop.fs.http.server.TestHttpFSServerWebServerWithRandomSecret

hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem

hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem

hadoop.fs.http.server.TestHttpFSAccessControlled

hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem

hadoop.fs.http.server.TestHttpFSServerWebServer

hadoop.lib.service.hadoop.TestFileSystemAccessService

hadoop.fs.http.server.TestHttpFSServerNoXAttrs

hadoop.hdfs.server.federation.metrics.TestMetricsBase

hadoop.hdfs.server.federation.router.TestRouterRPCClientRetries

hadoop.hdfs.server.federation.router.TestRouterAllResolver

hadoop.hdfs.server.federation.store.TestStateStoreMembershipState

hadoop.hdfs.server.federation.router.TestRouterFaultTolerant

hadoop.hdfs.server.federation.metrics.TestRBFMetrics

hadoop.hdfs.server.federation.resolver.order.TestAvailableSpaceResolver

hadoop.hdfs.server.federation.store.TestStateStoreRouterState

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2641/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2641

JIRA Issue
HDFS-15786

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 672fd0dba917 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / d09e3c929f2

Default Java
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-<IP_ADDRESS>+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2641/1/testReport/

Max. process+thread count
2919 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2641/1/console

versions
git=2.17.1 maven=3.6.0 findbugs=4.0.6

Powered by
Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

HI, this is my first time pushing in Hadoop.now what? what do I have to change?
Regards

HI, this is my first time pushing in Hadoop.now what? what do I have to change?
Regards

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m 46s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 11s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
29m 33s

trunk passed

+1 :green_heart:
compile
5m 25s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-Ubuntu-0ubuntu1.18.04

+1 :green_heart:
compile
5m 11s

trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

+1 :green_heart:
checkstyle
1m 19s

trunk passed

+1 :green_heart:
mvnsite
2m 27s

trunk passed

+1 :green_heart:
shadedclient
23m 24s

branch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
1m 45s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-Ubuntu-0ubuntu1.18.04

+1 :green_heart:
javadoc
1m 49s

trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

+0 :ok:
spotbugs
1m 35s

Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
4m 58s

trunk passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 27s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 52s

the patch passed

+1 :green_heart:
compile
5m  6s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-Ubuntu-0ubuntu1.18.04

+1 :green_heart:
javac
5m  6s

the patch passed

+1 :green_heart:
compile
4m 11s

the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

+1 :green_heart:
javac
4m 11s

the patch passed

+1 :green_heart:
checkstyle
0m 57s

the patch passed

+1 :green_heart:
mvnsite
1m 48s

the patch passed

+1 :green_heart:
whitespace
0m  0s

The patch has no whitespace issues.

+1 :green_heart:
shadedclient
16m 58s

patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
1m 44s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-Ubuntu-0ubuntu1.18.04

+1 :green_heart:
javadoc
2m  1s

the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

+1 :green_heart:
findbugs
6m 22s

the patch passed

_ Other Tests _

+1 :green_heart:
unit
2m 42s

hadoop-hdfs-client in the patch passed.

-1 :x:
unit
5m 28s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt
hadoop-hdfs-httpfs in the patch passed.

-1 :x:
unit
11m 20s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
hadoop-hdfs-rbf in the patch passed.

+1 :green_heart:
asflicense
0m 39s

The patch does not generate ASF License warnings.

154m 51s

Reason
Tests

Failed junit tests
hadoop.fs.http.server.TestHttpFSServerNoACLs

hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem

hadoop.fs.http.server.TestHttpFSServer

hadoop.fs.http.server.TestHttpFSServerWebServerWithRandomSecret

hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem

hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem

hadoop.fs.http.server.TestHttpFSAccessControlled

hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem

hadoop.fs.http.server.TestHttpFSServerWebServer

hadoop.lib.service.hadoop.TestFileSystemAccessService

hadoop.fs.http.server.TestHttpFSServerNoXAttrs

hadoop.hdfs.server.federation.metrics.TestMetricsBase

hadoop.hdfs.server.federation.router.TestRouterQuota

hadoop.hdfs.server.federation.router.TestRouterRPCClientRetries

hadoop.hdfs.server.federation.router.TestRouterAllResolver

hadoop.hdfs.server.federation.store.TestStateStoreMembershipState

hadoop.hdfs.server.federation.router.TestRouterFaultTolerant

hadoop.hdfs.server.federation.metrics.TestRBFMetrics

hadoop.hdfs.server.federation.resolver.order.TestAvailableSpaceResolver

hadoop.hdfs.server.federation.router.TestRouterMountTableCacheRefreshSecure

hadoop.hdfs.server.federation.store.TestStateStoreRouterState

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2641/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2641

JIRA Issue
HDFS-15786

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 97e6e2ad491c 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 91d4ba57c5b

Default Java
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-<IP_ADDRESS>+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2641/2/testReport/

Max. process+thread count
2916 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2641/2/console

versions
git=2.17.1 maven=3.6.0 findbugs=4.0.6

Powered by
Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m 46s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 11s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
29m 33s

trunk passed

+1 :green_heart:
compile
5m 25s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-Ubuntu-0ubuntu1.18.04

+1 :green_heart:
compile
5m 11s

trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

+1 :green_heart:
checkstyle
1m 19s

trunk passed

+1 :green_heart:
mvnsite
2m 27s

trunk passed

+1 :green_heart:
shadedclient
23m 24s

branch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
1m 45s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-Ubuntu-0ubuntu1.18.04

+1 :green_heart:
javadoc
1m 49s

trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

+0 :ok:
spotbugs
1m 35s

Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
4m 58s

trunk passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 27s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 52s

the patch passed

+1 :green_heart:
compile
5m  6s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-Ubuntu-0ubuntu1.18.04

+1 :green_heart:
javac
5m  6s

the patch passed

+1 :green_heart:
compile
4m 11s

the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

+1 :green_heart:
javac
4m 11s

the patch passed

+1 :green_heart:
checkstyle
0m 57s

the patch passed

+1 :green_heart:
mvnsite
1m 48s

the patch passed

+1 :green_heart:
whitespace
0m  0s

The patch has no whitespace issues.

+1 :green_heart:
shadedclient
16m 58s

patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
1m 44s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-Ubuntu-0ubuntu1.18.04

+1 :green_heart:
javadoc
2m  1s

the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

+1 :green_heart:
findbugs
6m 22s

the patch passed

_ Other Tests _

+1 :green_heart:
unit
2m 42s

hadoop-hdfs-client in the patch passed.

-1 :x:
unit
5m 28s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt
hadoop-hdfs-httpfs in the patch passed.

-1 :x:
unit
11m 20s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
hadoop-hdfs-rbf in the patch passed.

+1 :green_heart:
asflicense
0m 39s

The patch does not generate ASF License warnings.

154m 51s

Reason
Tests

Failed junit tests
hadoop.fs.http.server.TestHttpFSServerNoACLs

hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem

hadoop.fs.http.server.TestHttpFSServer

hadoop.fs.http.server.TestHttpFSServerWebServerWithRandomSecret

hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem

hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem

hadoop.fs.http.server.TestHttpFSAccessControlled

hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem

hadoop.fs.http.server.TestHttpFSServerWebServer

hadoop.lib.service.hadoop.TestFileSystemAccessService

hadoop.fs.http.server.TestHttpFSServerNoXAttrs

hadoop.hdfs.server.federation.metrics.TestMetricsBase

hadoop.hdfs.server.federation.router.TestRouterQuota

hadoop.hdfs.server.federation.router.TestRouterRPCClientRetries

hadoop.hdfs.server.federation.router.TestRouterAllResolver

hadoop.hdfs.server.federation.store.TestStateStoreMembershipState

hadoop.hdfs.server.federation.router.TestRouterFaultTolerant

hadoop.hdfs.server.federation.metrics.TestRBFMetrics

hadoop.hdfs.server.federation.resolver.order.TestAvailableSpaceResolver

hadoop.hdfs.server.federation.router.TestRouterMountTableCacheRefreshSecure

hadoop.hdfs.server.federation.store.TestStateStoreRouterState

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2641/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2641

JIRA Issue
HDFS-15786

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 97e6e2ad491c 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 91d4ba57c5b

Default Java
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-<IP_ADDRESS>+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2641/2/testReport/

Max. process+thread count
2916 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2641/2/console

versions
git=2.17.1 maven=3.6.0 findbugs=4.0.6

Powered by
Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.


MINOR: Disable shader compile error check in production build

Codecov Report

Merging #1424 into master will decrease coverage by 0.1%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1424      +/-   ##
==========================================
- Coverage   55.85%   55.75%   -0.11%
==========================================
  Files         271      270       -1
  Lines       24275    24164     -111
  Branches     5431     5398      -33
==========================================
- Hits        13560    13472      -88
+ Misses      10715    10692      -23

Impacted Files
Coverage Δ

@here/harp-test-utils/lib/WebGLStub.ts
100% <100%> (ø)
:arrow_up:

@here/harp-mapview/lib/MapView.ts
60.46% <100%> (+0.02%)
:arrow_up:

@here/harp-text-canvas/lib/rendering/TextStyle.ts
79.76% <0%> (-3.58%)
:arrow_down:

@here/harp-mapview/lib/text/Placement.ts
88.77% <0%> (-2.18%)
:arrow_down:

...arp-text-canvas/lib/rendering/GlyphTextureCache.ts
63.81% <0%> (-1.01%)
:arrow_down:

...here/harp-text-canvas/lib/rendering/FontCatalog.ts
75% <0%> (-0.79%)
:arrow_down:

@here/harp-mapview/lib/DecodedTileHelpers.ts
57.05% <0%> (-0.14%)
:arrow_down:

...here/harp-mapview/lib/text/TextElementsRenderer.ts
66.42% <0%> (ø)
:arrow_up:

@here/harp-mapview/lib/TextureLoader.ts

...e/harp-mapview/lib/geometry/TileGeometryCreator.ts
40.4% <0%> (+0.09%)
:arrow_up:

... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e7562a9...6acb030. Read the comment docs.


MongoError: file with id ### not opened for writing

Need to store picture with my API using FeatherJS.
I got the error MongoError: file with id ### not opened for writing.
with ### is the name of the current file. Do I have to set up manually the ObjectId of the metadata written inside mongo ?
my code for my /pub service (if you are familiar with feathersJS):

pub.service.js

...
app.use('/pub',
      multipartMiddleware.single('uri'),
      function(req, res, next){
        req.feathers.file = req.file;
        next();
      },
      createService(options)
    );
...

pub.hook.js

...
create: [
      hook => {
        const gfs = hook.app.get('gfs');
        const file = hook.params.file;

        console.log(hook)

        const writestream = gfs.createReadStream({
          filename: file.originalname,
          content_type: file.mimetype, // image/jpeg or image/png
        })
      }
    ],
...

What did I miss ?

the same issue . Did you solve it?

As you want to write to GridFS, you should use Grid.createWriteStream instead of Grid.createReadStream.
Then, using fs to read the content of your file fs.createReadStream(filePath).pipe(writestream) to actually write the content of the file to GridFS.

pub.hook.js

...
create: [
      hook => {
        const gfs = hook.app.get('gfs');
        const file = hook.params.file;

        console.log(hook)

        const writestream = gfs.createWriteStream({
          filename: file.originalname,
          content_type: file.mimetype, // image/jpeg or image/png
        });
        fs.createReadStream(filePath).pipe(writestream);
      }
    ],
...


bug with the travel enchor

hello
i'had a bug, i painted my travel enchor with a block and after i painted my travel enchor with another travel enchor. i place my travel anchor and the texture was not pin up, and my game crashed.My friend was at 500meters thant me and he has crashed too. I must delet your mod and load my map for play . :/ .
Ps: sorry for my bad english i'm french
good luck

Please update the mod, this was fixed


Rename Application to AppHandler

Application is the global-equivalent of WindHandler it should be called AppHandler for a few reasons:

Consistency with WinHandler
Clearer name. To me, finding an "Application" in your code implies you've found the composition root of your app-specific state, when in fact you've found a lightweight handle to platform bindings.

I've bumped into this problem when writing Masonry, and I think @Maan2003 bumped into it as well last year: https://xi.zulipchat.com/#narrow/stream/147926-druid/topic/Remove.20Application.3A.3Aglobal/near/238902785

To add another bikeshed to the mix, when I wrote code, it's basically a 50/50 tossup whether I name the correct WinHandler or WindowHandle on the first try. Is there a reason to abbreviate one and not the other? And are these just the standard names in the world of windowing, or are there less-similar names that are also standard?

are there less-similar names that are also standard?

Perhaps we could adopt macOS's Delegate for the Handler?

Simply renaming Application to AppHandler wouldn't work because there already is an AppHandler.
There has been some related discussion that isn't yet referenced in this specific thread. In writing we have #168 and the Removing WinHandler Zulip thread. The topic was also discussed during office hours but that is ephemeral, so no transcript.
Thus far the plan has shaped up to the following:

Rename Application to Glazier
Remove WinHandler and migrate its functionality to AppHandler
Potentially rename AppHandler to something better, but no consensus on that yet - more name proposals welcome!

Remove WinHandler and migrate its functionality to AppHandler

Wait, isn't WinHandler a bridge for some stuff the platform sends you? Removing it sounds awkward.


Use Big Elements with ot-layout directive

Hi,
I am able to use the 'OT_big' class on ot-publisher elements. However, is there any way to get the double-click resizing (dynamically adding & removing the class) effect seen in the opentok-layout-js demo using these directives? It seems that layout() has to be called to make it take effect, but how is layout() available to us using the directives?
Many thanks for your great work.

The directive is listening for the otLayout event and calling the layout method whenever that happens. So you just need to emit the otLayout event like so:
$scope.$emit('otLayout');

Here is an example of a directive which adds an expand button that does this.

Beautiful, thanks!


Small question. Why picker rotation was changed to false? Seems like it looked beautifully on android

Compose doesn't support z translations so it's hard to make it really "rotate", the existing effect looks bad/unpolished in some configurations like big container width or long texts. I'll provide it as an option if you'd like to see the rotation in your app despite everything

It would be great) I was testing this feature on example app, so that works normally