feat: gwas catalog top-hit + study step

Step to run the top-hits in isolation of everything else.
The GWASCatalogTopHitIngestionStep included here was run in dataproc in 17 minutes.
Subsequent PRs will handle GWAS Catalog + Summary Statistics + PICS road which might result in removing some steps.

I'm having second thoughts. Because the 2 DAGs have interdependencies in the business logic, I want to see both working before we review this.
I'm reverting to draft

I'll summarise the status of this PR, and I'd appreciate some thoughts @DSuveges and @project-defiant
In general it isn't straightforward because it touches four gwas catalog processes:
a) top hits
b) sumstats pics
c) susie finemapping
d) gwas catalog curation table
With the current state of the PR, the DAGs for a) and b) run to completion without considering curation or sumstats QC c) is not yet implemented, but I made a pass to the logic to generate the study index in the context of QC sumstats and/or availability of sumstats and/or curation, but it hasn't been tested.  d) I didn't pay too much attention to it.
In terms of features. This PR contains:

Addition of the gwas_catalog_study_index step, which is a refactor of the previous gwas_catalog_study_inclusion. Now, this step does not aim to filter; just generates a study index as complete as possible, allowing for 3 optional parameters: sum stats QC, OT curation and availability of summary statistics (ideally from our harmonised bucket). This step could be reused in b-d but never in a) which contains its own study index using the study splitting.

A new gwas_catalog_top_hits step, which simplifies the older gwas_catalog_ingestion step. This step produces StudyLocus from GWAS catalog associations and their respective StudyIndex. It no longer tries to consolidate information on summary statistics, significantly simplifying the process.

A new class to parse the OT curation, by just moving the existing code to a separate class.

It needs to be better tested. I might be distracted for a few days, but I'm happy to return and add more unit tests unless this work becomes essential for something else earlier.

Example how to run the step:
poetry run gentropy step=gwas_catalog_study_index \
step.catalog_study_files="[ '/Users/ochoa/Datasets/gwas_catalog_download_studies.tsv' ]" \
step.catalog_ancestry_files="[ '/Users/ochoa/Datasets/gwas_catalog_download_ancestries.tsv' ]" \
step.study_index_path=/Users/ochoa/Datasets/study_index_annotated \
step.gwas_catalog_study_curation_file=/Users/ochoa/Datasets/20241004_output_curation.tsv \
step.sumstats_qc_path=/Users/ochoa/Datasets/20241015_GwasCatQCLogs

Copy of the outputs
gs://ot-team/dochoa/study_index_annotated

Breakdown of GWAS Catalog flags
+----------+-----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----+
|studyType |analysisFlags          |qualityControls                                                                                |count|
+----------+-----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----+
|gwas      |[]                     |[GWAS Catalog study has not been curated by Open Targets., Harmonized summary statistics are not available or empty.]                                                                                |54168|
|gwas      |[]                     |[]                                                                                |16842|
|pQTL      |[]                     |[]                                                                                |16533|
|gwas      |[Metabolite]           |[]                                                                                |12064|
|gwas      |[ExWAS]                |[Harmonized summary statistics are not available or empty.]                                                                                |3695 |
|gwas      |[Metabolite]           |[Harmonized summary statistics are not available or empty.]                                                                                |2849 |
|gwas      |[]                     |[GWAS Catalog study has not been curated by Open Targets.]                                                                                |2145 |
|Microbiome|[]                     |[]                                                                                |1128 |
|gwas      |[Metabolite]           |[The number of SNPs in the study is below the expected threshold.]                                                                                |1033 |
|gwas      |[]                     |[The mean beta QC check value is not within the expected range.]                                                                                |793  |
|gwas      |[]                     |[Harmonized summary statistics are not available or empty.]                                                                                |604  |
|Microbiome|[]                     |[The mean beta QC check value is not within the expected range.]                                                                                |414  |
|gwas      |[ExWAS]                |[The number of SNPs in the study is below the expected threshold.]                                                                                |294  |
|gwas      |[]                     |[The PZ QC check values are not within the expected range.]                                                                                |267  |
|gwas      |[]                     |[The number of SNPs in the study is below the expected threshold.]                                                                                |255  |
|gwas      |[Multivariate analysis]|[Harmonized summary statistics are not available or empty.]                                                                                |248  |
|pQTL      |[]                     |[The number of SNPs in the study is below the expected threshold.]                                                                                |200  |
|Microbiome|[]                     |[Harmonized summary statistics are not available or empty.]                                                                                |138  |
|gwas      |[GxE]                  |[Harmonized summary statistics are not available or empty.]                                                                                |73   |
|gwas      |[Case-case study]      |[]                                                                                |68   |
|gwas      |[GxE]                  |[]                                                                                |51   |
|gwas      |[Non-additive model]   |[]                                                                                |45   |
|gwas      |[]                     |[GWAS Catalog study has not been curated by Open Targets., The number of SNPs in the study is below the expected threshold.]                                                                                |41   |
|gwas      |[GxG]                  |[Harmonized summary statistics are not available or empty.]                                                                                |29   |
|gwas      |[]                     |[The GC lambda value is not within the expected range., The number of SNPs in the study is below the expected threshold.]                                                                                |27   |
|gwas      |[Non-additive model]   |[The mean beta QC check value is not within the expected range., The PZ QC check values are not within the expected range., The GC lambda value is not within the expected range.]                                                                 |25   |
|gwas      |[]                     |[GWAS Catalog study has not been curated by Open Targets., The PZ QC check values are not within the expected range., The number of SNPs in the study is below the expected threshold.]                                                            |21   |
|gwas      |[GxG]                  |[]                                                                                |21   |
|Microbiome|[]                     |[The PZ QC check values are not within the expected range.]                                                                                |20   |
|gwas      |[]                     |[The mean beta QC check value is not within the expected range., The PZ QC check values are not within the expected range.]                                                                                |16   |
|gwas      |[GxE]                  |[The number of SNPs in the study is below the expected threshold.]                                                                                |16   |
|gwas      |[Multivariate analysis]|[]                                                                                |15   |
|gwas      |[]                     |[The mean beta QC check value is not within the expected range., The GC lambda value is not within the expected range., The number of SNPs in the study is below the expected threshold.]                                                          |11   |
|gwas      |[Non-additive model]   |[The PZ QC check values are not within the expected range.]                                                                                |11   |
|gwas      |[]                     |[The mean beta QC check value is not within the expected range., The PZ QC check values are not within the expected range., The GC lambda value is not within the expected range.]                                                                 |10   |
|gwas      |[]                     |[GWAS Catalog study has not been curated by Open Targets., The PZ QC check values are not within the expected range.]                                                                                |9    |
|gwas      |[]                     |[The PZ QC check values are not within the expected range., The number of SNPs in the study is below the expected threshold.]                                                                                |9    |
|gwas      |[]                     |[GWAS Catalog study has not been curated by Open Targets., The mean beta QC check value is not within the expected range., The PZ QC check values are not within the expected range., The GC lambda value is not within the expected range.]       |7    |
|gwas      |[]                     |[The GC lambda value is not within the expected range.]                                                                                |7    |
|gwas      |[Case-case study]      |[Harmonized summary statistics are not available or empty.]                                                                                |7    |
|gwas      |[Non-additive model]   |[The mean beta QC check value is not within the expected range., The PZ QC check values are not within the expected range.]                                                                                |6    |
|gwas      |[]                     |[The mean beta QC check value is not within the expected range., The number of SNPs in the study is below the expected threshold.]                                                                                |6    |
|gwas      |[]                     |[The mean beta QC check value is not within the expected range., The GC lambda value is not within the expected range.]                                                                                |5    |
|pQTL      |[]                     |[The PZ QC check values are not within the expected range.]                                                                                |3    |
|gwas      |[Metabolite]           |[The PZ QC check values are not within the expected range.]                                                                                |3    |
|Microbiome|[]                     |[The PZ QC check values are not within the expected range., The number of SNPs in the study is below the expected threshold.]                                                                                |3    |
|gwas      |[]                     |[GWAS Catalog study has not been curated by Open Targets., The GC lambda value is not within the expected range., The number of SNPs in the study is below the expected threshold.]                                                                |2    |
|gwas      |[ExWAS]                |[]                                                                                |2    |
|gwas      |[]                     |[GWAS Catalog study has not been curated by Open Targets., The mean beta QC check value is not within the expected range., The PZ QC check values are not within the expected range.]                                                              |2    |
|gwas      |[]                     |[GWAS Catalog study has not been curated by Open Targets., The mean beta QC check value is not within the expected range.]                                                                                |2    |
|gwas      |[Case-case study]      |[The PZ QC check values are not within the expected range.]                                                                                |2    |
|gwas      |[]                     |[GWAS Catalog study has not been curated by Open Targets., The mean beta QC check value is not within the expected range., The number of SNPs in the study is below the expected threshold.]                                                       |2    |
|gwas      |[]                     |[GWAS Catalog study has not been curated by Open Targets., The PZ QC check values are not within the expected range., The GC lambda value is not within the expected range.]                                                                       |1    |
|gwas      |[]                     |[The PZ QC check values are not within the expected range., The GC lambda value is not within the expected range.]                                                                                |1    |
|gwas      |[]                     |[The mean beta QC check value is not within the expected range., The PZ QC check values are not within the expected range., The number of SNPs in the study is below the expected threshold.]                                                      |1    |
|gwas      |[Non-additive model]   |[The PZ QC check values are not within the expected range., The GC lambda value is not within the expected range.]                                                                                |1    |
|gwas      |[]                     |[GWAS Catalog study has not been curated by Open Targets., The mean beta QC check value is not within the expected range., The GC lambda value is not within the expected range., The number of SNPs in the study is below the expected threshold.]|1    |
+----------+-----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----+

Added changes in ancestry mapping in gwas_population_2_LD_panel_map.json since we are going to remake the study index. These changes add CSA (central south asians) from UKBB to the list of ancestries.


Fix #23.

The duration_to_string method has a small bug.
The padding for the microseconds was 7; however, given the way the remaining micros are calculated, the remaining micros will only ever be a maximum of 6 digits.  In addition, microseconds in datetime strings should be formatted as exactly 6 digits.
Say you had 999999 microseconds remaining.  We would read this as 999 ms.  However, using a padding of 7 makes this 0999999 microseconds.  I am guessing that app insights ignores the last digit, and consumes this as 099999 microseconds, or 99 ms.
This should fix that issue.

Great find ❤

Can you update the test case at the end of the file as well?

@frigus02, no problem: thanks for building this app insights support. 😄


Version 0.10.12?

Version 0.10.12 is referenced in the readme (python 3.5+, django 2.0) but not available in python repository... Where are we supposed to get this? It's hard to tell if the the code is even ahead of 0.10.0... if it is, I suppose I could just fork it...

@cobyrne09 This repo is no longer maintained. Please report the issue in this fork https://github.com/swappsco/django-qa


User Setup

Right now as it runs local only there's no setup step at all.
At some point when we fully integrate with GitHub and Travis we might want to assist the user in setting everything up, setting environment variables and keys and such.

So we need to ask the user for:

travis config
crates.io token
github credentials

Optionally we need to ask for GIT_COMITTER_NAME and GIT_COMITTER_EMAIL


sidebar stops working when we call window.DocsifyCopyCodePlugin.init(),

Hello.
According to #459 and #461 sidebar stops working when we call window.DocsifyCopyCodePlugin.init(), in code.
take a look at this repository and place this code(as it was before). It stops sidebar.

Just wanted to say that I can confirm this and to mention that it also breaks the coverpage. As far as I know, everything was working a day or two ago.
You can see the effects on this site also: https://adrianbj.github.io/TracyDebugger/

Instructions on how to fix here: https://github.com/jperasmus/docsify-copy-code/issues/7
Making these changes has everything working again for me.

So the bug is from docsify-copy-code?
Even if this solution work (it does) this issue should be open till docsify-copy-code is fixed.

This is an issue with docsify-copy-code because we've updated the plugin to a new major version, meaning it has breaking changes.
You no longer need to manually add the stylesheet or call window.DocsifyCopyCodePlugin.init() to initialize the plugin, it will initialize automatically. These changes were made in preparation for better theming support going forward. The changes also include better support for older browsers.
Apologies for the inconvenience.
In short, for others experiencing the same, you can remove all previous code needed to add the plugin and replace it with the following script tag below the docsify script tag:
<script src="https://unpkg.com/docsify-copy-code"></script>

FYI...
The new URL allows unpkg.com to redirect requests to the appropriate file. This is better than linking directly to a developer may change file names or locations. Using the redirect makes this a non-issue.
As for the trailing slash...

Without slash = redirect to script file: https://unpkg.com/docsify-copy-code
With slash = list files in a package: https://unpkg.com/docsify-copy-code/

Appending a A[number] after the package name let's you specify versions:

v1.xx redirect = https://unpkg.com/docsify-copy-code@1
v1.xx files = https://unpkg.com/docsify-copy-code@1/
v1.xx redirect = https://unpkg.com/docsify-copy-code@2
v1.xx files = https://unpkg.com/docsify-copy-code@2/

😄


Wheelrate for textArea

textArea is automatically inside scroll - correct?
now, textArea_ does not have cfg for wheelrate
how can I reach the wheelrate of the implicit scroll of textArea?

I'm not sure I understand what you want to do. If your objective is being able to change the wheelRate configuration of the scroll that wraps your codeEditor, you can do it here (from your repository).
That scrollCfg list is the regular configuration list you pass to a widget. This means you can add wheelRate to it and scroll will receive it. If you want to make your widget expose this configuration option, you can add a new field to your configuration record and add the CmbWheelRate combinator instance.

Thanks - with your explanation I was able to find the position in the widget && added a config option


[Feature]Send to message queue request after reading data（读取数据后发送到消息队列需求）

Describe the solution you'd like
A clear and concise description of what you want to happen.
Each time you read 100 data in the database, it is sent to the message queue. The message queue is a place where big data is used. You don't have to write a jar package to send it.

Describe the solution you'd like
A clear and concise description of what you want to happen.
每次读取数据库100条数据后发送到消息队列，消息队列是大数据用的比较多的地方，不用自己再写个jar包程序发送

very good, thx!
if anybody else also need this feature please reply some words!

read what data and output what message queue? I think users can do it by themself, not DolphinScheduler's scope


Use UnaligningVertex in LinearPolymer class.

Related Issues:  #156
Requested Reviewers: @lukasturcani
Note for Reviewers: If you accept the review request add a :+1: to this post
This PR makes no changes visible to the user but allows the user to placer molecules like BrCBr on the polymer.linear topology graph using UnaligningVertex. This is handled by the Linear initializer when defining vertices (all vertices with building blocks that only have one placer are converted to a UnaligningVertex - same as in Cof).
Added a new init class method to the UnaligningVertex to allow this as the init class is used elsewhere and did not want to change that - happy to discusss.

src/stk/molecular/topology_graphs/polymer/linear/linear.py:_get_building_block_vertices: Added a check for whether each vertex should be an UnaligningVertex.

src/stk/molecular/topology_graphs/polymer/linear/vertices.py:UnaligningVertex: Added new init class method.

Two new tests added for vertex and molecule fixture.

Closing to be implemented from scratch and cleaned.


Escape file names

Fixes #959. Characters such as " " and "%" in file names would break when installing the package with nuget (and possibly paket).
I couldn't see any tests around NupkgWriter.fs so I have created a batch script that reproduces the issue.

related to https://github.com/fsprojects/Paket/issues/937?

thx. this is now in the latest alpha version.


Btree repair validation design

We need to first design the different methods of validating the btree repair. Based on the design we would have multiple tasks/issues to track on each of the validation.

https://docs.google.com/document/d/1Jk1mgTNWmSobq_6vL6uLD9gwTYK8xCd7u-feK1VSG2o/edit#heading=h.p88gqmaerwds


6c0b6e64-974d-44d9-82cc-bf200617d411

Type: Bug
I was trying to correct a line length warning and it told me to report this ID 6c0b6e64-974d-44d9-82cc-bf200617d411.
 Do you consistently hit this or was it just a blip on your end?

This particular issue happens quite often, but this was the first time I
received this type of message. I live in an area where the internet is
unpredictable at the best of times.
On Wed, Oct 16, 2024, 9:13 AM Logan Ramos @.***> wrote:

This failure appears to be caused by a slow network connection on your
part. Do you consistently hit this or was it just a blip on your end?
—
Reply to this email directly, view it on GitHub
https://github.com/microsoft/vscode-copilot-release/issues/1864#issuecomment-2416970575,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BGREDBLF4RXGRJ2IE2SKSW3Z3ZYB5AVCNFSM6AAAAABP5YGMESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJWHE3TANJXGU
.
You are receiving this because you authored the thread.Message ID:
@.***>

Thank you for your help. This particular message just appeared and I did as
it said. I hope I didn't waste your time.
On Wed, Oct 16, 2024, 10:41 AM Logan Ramos @.***> wrote:

Hmm, I don't think there is really anything actionable we will be able to
do here. The server is closing the connection because it is taking more
than 5 seconds for your machine to finish sending the small JSON payload
—
Reply to this email directly, view it on GitHub
https://github.com/microsoft/vscode-copilot-release/issues/1864#issuecomment-2417214176,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BGREDBOTIXHNTM7JZK25TNLZ32CLNAVCNFSM6AAAAABP5YGMESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJXGIYTIMJXGY
.
You are receiving this because you authored the thread.Message ID:
@.***>


Make audit10_install_one_audit_table_manual.sql robust

[x] Currently it can only be run once for creation
[x] After patching the audit10 script needs to be run to recreate the triggers
[x] The table creation should be bypassed, if the table exists.
[x] The data in audit_table and audit column needs a refresh for this action.

All fixed in V0_0_3.


Get tag values from bam file

Rudimentary support for retrieving tags from BAM files. To use, instantiate a BamReader with a list of tags to be returned:
let reader = BamReader::new_from_path("my/path", vec!["MD", "NM"])
reader.read_from_ipc()

The returned structure contains extra columns for each of the columns specified.
This PR has a number of limitations:

There's no way to say "return all tags", without specifying all known tags. Maybe making the tags parameter optional and returning all tags when that's the case?

It currently converts all tags to strings when they are returned. This is because I couldn't figure out how to select the appropriate arrow:array function to use to accommodate the multitude of types that could be in noodles::sam::record::data::field::Value.

There's likely other shortcomings but these are the immediately obvious ones. Any feedback or help with either of these points would be greatly appreciated.

Looking more at the existing API, I should pass tags into records_to_ipc rather than BamReader. Will do that in the next rev.

Addressed part concern #1 above so that all tags are returned if no tags are explicitly asked for.
Not addressing concern #2 atm. It's not absolutely terrible that we're returning strings for tags. Just pawning the complexity of parsing off on the downstream user.

As discussed, closing this PR and merging #43


Authentication Issue 401

I am having a issue where I keep on get 401 HTTP errors with a user even though I can log into the GUI and SSH.
from f5.bigip import ManagementRoot
import requests
from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

# Connect to the BIG-IP
mgmt = ManagementRoot("<IP_ADDRESS>", "username", "password", verify=False)

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/apps/CSFI/.python/lib/python3.6/site-packages/f5/bigip/__init__.py", line 168, in __init__
    self.post_configuration_setup()
  File "/apps/CSFI/.python/lib/python3.6/site-packages/f5/bigip/__init__.py", line 114, in post_configuration_setup
    self._get_tmos_version()
  File "/apps/CSFI/.python/lib/python3.6/site-packages/f5/bigip/__init__.py", line 122, in _get_tmos_version
    response = connect.get(base_uri)
  File "/apps/CSFI/.python/lib/python3.6/site-packages/icontrol/session.py", line 284, in wrapper
    raise iControlUnexpectedHTTPError(error_message, response=response)
icontrol.exceptions.iControlUnexpectedHTTPError: 401 Unexpected Error: F5 Authorization Required for uri: https://<IP_ADDRESS>:443/mgmt/tm/sys/

Here are the details about the User and the BigIP version that is running
# tmsh list auth user all

 auth user username {
    description "username"
    encrypted-password ""
    partition Common
    partition-access all
    role resource-admin
    shell bash
}

# tmsh show /sys version

Sys::Version
Main Package
  Product  BIG-IP
  Version  11.5.3
  Build    0.0.163
  Edition  Final
  Date     Wed Apr 22 16:32:47 PDT 2015

@jmcgrath207
are you logging in with the username 'admin'? If not, then it won't work on the version of BIGIP that you have. generally speaking the REST API doesn't "work" until 12.0.0 at the earliest


Adding memtable_cleanup_threshold YAML config option

Vinay,
Please create a PR for branch 2.x if this change is applicable to 2.0.x C* version.
Regards


Document ports 839 and 88 as required to be opened in firewall on the host

Description
When running a firewall on the host that automatically blocks all traffic, it's mandatory to open ports 389 and 88 for containers hooked up with gMSA be able to talk to domain controllers and pass Kerberos tokens.
Please list these ports in the gMSA related documentation.

We just published a troubleshooting section for firewall configuration. It should be live on https://aka.ms/containers/identity shortly.
Thanks for the suggestion!


Thumbnail generator error with image cropping

I have received an error for using the script crop.py:
Traceback (most recent call last):
  File "./scripts/crop.py", line 149, in <module>
    white_img_cropped = crop(green_img, white_img)
  File "./scripts/crop.py", line 32, in crop
    mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
ValueError: need more than 2 values to unpack

I fixed it by changing the number of outputs from:
  _ ,  contours, _ = cv2.findContours( mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
to
    contours, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)

thanks for finding that issue, this is probably due to different versions of OpenCV, while the version shipped with ROS prefers the latter version. I'll make the necessary fixes, thanks!

@ngwk1 alternative, you could take a look at this new pipeline, it should work as expected. if you find any issues, feel free to reach out
https://github.com/osrf/traffic_editor/pull/191

Yes, we have a replacement for the existing thumbnail_generator. Can close this issue for now. Thanks for raising this :+1:


feat: generate metrics documentation

Which problem is this PR solving?
This PR implements the logic to parse out all metrics defined in the Refinery code base and generates a markdown table for them.

part 2 of #1152

Short description of the changes

add two new commands to the convert tool to generate the metricsMeta.yaml file and a metrics.md file

I have a style suggestion here -- instead of writing out a .md file, save the data in a []struct, where the struct has fields for Name, Type, Unit, Description. Then write out that entire slice to a .yaml file that we store in the same place as other metadata. Then we can write a quick template and use convert to make a really pretty markdown file, maybe a table instead, and it can have headers and intro text, etc.

I like it!

Awesome! Do you mind to give the part 1 PR another look as well? https://github.com/honeycombio/refinery/pull/1350


StringNode flag or new node type to mark if frozen or not

This would be a lot more convenient for semantic analysis if a StringNode knew if it was a frozen literal or not.
It could also be a different node type, that probably makes sense for this case.
This notably also affects the line events (#1099) and as such seems a blocker to get correct line events:
# frozen_string_literal: true

known_failures = %w[
  seattlerb/heredoc_nested.txt
  seattlerb/pct_w_heredoc_interp_nested.txt
]

$ bin/insns test.rb
== disasm: #<ISeq:<main>@test.rb:1 (1,0)-(6,1)> (catch: false)
local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
[ 1] known_failures@0
0000 duparray                               ["seattlerb/heredoc_nested.txt", "seattlerb/pct_w_heredoc_interp_nested.txt"](   3)[Li]
0002 dup
0003 setlocal_WC_0                          known_failures@0
0005 leave

so only line 3 is an event, because the whole literal is immutable
but
# frozen_string_literal: false

known_failures = %w[
  seattlerb/heredoc_nested.txt
  seattlerb/pct_w_heredoc_interp_nested.txt
]

$ bin/insns test.rb
== disasm: #<ISeq:<main>@test.rb:1 (1,0)-(6,1)> (catch: false)
local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
[ 1] known_failures@0
0000 putstring                              "seattlerb/heredoc_nested.txt"(   4)[Li]
0002 putstring                              "seattlerb/pct_w_heredoc_interp_nested.txt"(   5)
0004 newarray                               2                         (   3)
0006 dup
0007 setlocal_WC_0                          known_failures@0
0009 leave

line 4 is an event here, because of the string allocation.
(unsure why line 5 is not a line event, maybe it's a bit like CallNode arguments where no arguments is a line event unless it's a new call target like a block, anyway that is orthogonal)

It seems StringNode now has a FROZEN flag.


Add ReaderT and Documentation for both Reader and ReaderT

I can fly anywhere!!

This PR does the following:

Adds ReaderT
Fixes up Reader specs to bring them in line with the new 🔥
Adds documentation for both Reader and ReaderT

Coverage remained the same at 100.0% when pulling bbfb449486b6f747be3e74cdfb9e237f68d42eda on reader-t into 68de04589e6d783e917aa954dfb3133abaa9a608 on master.

:eyes:

Coverage remained the same at 100.0% when pulling 4237601d9c6b63fb602fbbab6803c86b0cc54c7b on reader-t into 68de04589e6d783e917aa954dfb3133abaa9a608 on master.


Worked on the swipe to delete function

Added the functionality to swipe the app to reveal a delete button, and an alert to confirm, but currently having trouble on how to fully delete the app from the token storage. #38

Something seems to be off with my package-lock.json file, but the only library I imported was the 'react-native-gesture-handler'


Tab managing

So far the tab manager only helps with filtering/finding tabs and closing them. Are there plans to include more features like manual re arrange of the tabs being able to select multiple tabs to close reload suspend bookmark copy their url move to a new window or existing window etc?

Maybe this belongs on an other issue but this depends on this issue. If you are going to implement those tab managing features it would be nice to have a view that does not group tabs by url and shows the tabs like they appear on each window.

@rayman89 Thank you for the suggestions. You can see plans for upcoming releases here: https://github.com/adamjamesadair/manage-my-tabs/projects/4.
Some of the features you mentioned are open as issues #60 and #61. I've opened issues #131, #132 and #133 to address the other features.

Yeah, I agree with @rayman89. It would be nice if the criteria used to group the tabs could be changed. But actually, if you go to the side bar and change the "Max number of tabs" to 1 and "Sort groups" to "Window order", you should be able to have them like that. A little bit tricky, but it can be done.

@BorFour On the issues you listed you refer to actions that affect 1 tab at a time. Are you willing to implement doing stuff like moving to an other window re arranging bookmarking etc in bulk? (please allow to manually select the tabs not only do them in bulk when they are grouped).

@rayman89 #143 adds the option to reload or send individual tabs to a different window. With #146 we plan to extend that functionality to tab groups and windows. I've opened issue #147 to address selecting manually selecting tabs, tab groups and windows.


The homepage rendering will be stuck in FireFox.

Describe the bug
v0.20.0 has problems in FireFox. When using FireFox to open the homepage of the Memos, the rendering will be stuck, resulting in a white screen. If the size of the browser viewport is changed at this time, the rendering can continue. At the same time, I observed the last request after got stuck was GetAuthStatus, and the response to this request was only 0B.
In addition, this problem can not reproduce in other pages like /explore and /about.

Steps to reproduce
I deployed it directly on Raspberry Pi without using docker. This is my build scripts:
Script for building:
#!/usr/bin/env sh
# script.sh

set -e

TARGET_DIR=build
VERSION=0.20.0
if [ ! -d $TARGET_DIR ]; then
  mkdir -p $TARGET_DIR
fi
rm -rf $TARGET_DIR/*

apk add --no-cache wget go nodejs npm
npm install -g corepack

WORK_DIR=$(pwd)
wget https://github.com/usememos/memos/archive/refs/tags/v$VERSION.tar.gz
tar -xzvf v$VERSION.tar.gz
mv memos-$VERSION memos

cd $WORK_DIR/memos
export CGO_ENABLED=0
export GOOS=linux
export GOARCH=arm64
go build -o memos ./bin/memos/main.go

cd $WORK_DIR/memos/web
corepack enable
pnpm install --frozen-lockfile
pnpm build

cd $WORK_DIR/
cp -r memos/web/dist $TARGET_DIR/dist
cp memos/memos $TARGET_DIR/memos

cd $TARGET_DIR
tar -czvf memos-$VERSION.tar.gz memos dist

Script to start the build process:
#!/usr/bin/env bash
# build.sh
TARGET_DIR=target
docker run --rm -v $(pwd)/script.sh:/memos/script.sh -v $(pwd)/$TARGET_DIR:/memos/build -w /memos alpine:edge /memos/script.sh

The version of Memos you're using
0.20.0
Screenshots or additional context
No response

GetAuthStatus很慢，打开页面会白屏几秒

It is not a temporary white screen, but a white screen until the browser viewport size changes.

This problem has disappeared in version 0.20.1.

This problem has disappeared in version 0.20.1.这个问题在0.20.1版本中已经消失。
并没有，首页加载还是很慢


Sprinkle some more demo on the docs

fixes https://github.com/open-telemetry/opentelemetry.io/issues/2137
There's several ways to skin this cat I guess, but I went with a shortcode shared across a few pages.
Preview: https://deploy-preview-2139--opentelemetry.netlify.app/docs/

Lgtm in general. One nit is to bold the demo title in the TOC like the other items. Ex. K8s operator. Thanks for doing this!

Fixed!

I like it!
Eventually I think we should remove one of the Demo links in the docs (either we have it beyond "Getting Started" or as top level item), but maybe we can track which one is used more and then later drop the other one?

Yep, we can track via GA I believe.

I would hesitate to contribute the full increase to the docs change as the new year really starts but there seems to be a noticeable increase in demo repo traffic from the website which is great to see.


Update README.md

Changes
:broom: Simplify finding the good stuff

add link to Broker README.md
add link to Source README.md

Thanks!
/lgtm
/approve

Thanks!
/lgtm
/approve


Register a user

As a prospective user, I want to be able to create an account access Note App

Closed by  #30


remove unnecessary omp single that cause deadlock (fixes #6273)

fixes #6273
omp single clause has implicit barrier, which waits for all threads, including threads that did not execute omp single block.
Therefore, if omp single clause is used conditionally inside omp parallel, it will cause a deadlock (explanation below).
I think it is safe to remove omp single clause from omp_get_max_threads() (introuduced in #6226).

omp_get_max_threads() is thread-safe https://www.openmp.org/spec-html/5.1/openmpse6.html#x27-260001.6
omp single does not affect the current team (threads participating in the execution of a parallel region) from which omp_get_max_threads() gets its value.

background
I digged into #6273.
import lightgbm as lgb
import numpy as np
import pandas as pd

X = np.random.randint(0, 5000, 10000)
X = pd.DataFrame({"x1": X})
y = np.random.rand(10000)

full_data = lgb.Dataset(X, y, categorical_feature=["x1"])
full_data.construct() # <-- hangs

I suspect a deadlock near OMP_NUM_THREADS() from the backtrace.
  thread #15
    frame #0: 0x000000018820106c libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x000000018823e5fc libsystem_pthread.dylib`_pthread_cond_wait + 1228
    frame #2: 0x000000016bd97ca0 libomp.dylib`void __kmp_suspend_64<false, true>(int, kmp_flag_64<false, true>*) + 264
    frame #3: 0x000000016bd818c0 libomp.dylib`kmp_flag_64<false, true>::wait(kmp_info*, int, void*) + 2012
    frame #4: 0x000000016bd7dcc8 libomp.dylib`__kmp_hyper_barrier_release(barrier_type, kmp_info*, int, int, int, void*) + 152
    frame #5: 0x000000016bd7c8c8 libomp.dylib`__kmp_barrier + 1276
    frame #6: 0x000000016bd52dd0 libomp.dylib`__kmpc_barrier + 340
    frame #7: 0x000000016c3a4db8 lib_lightgbm.so`OMP_NUM_THREADS + 112
    frame #8: 0x000000016c11a8f0 lib_lightgbm.so`LightGBM::ArrayArgs<int>::ArgMaxMT(std::__1::vector<int, std::__1::allocator<int>> const&) + 44
    frame #9: 0x000000016c151cdc lib_lightgbm.so`LightGBM::BinMapper::FindBin(double*, int, unsigned long, int, int, int, bool, LightGBM::BinType, bool, bool, std::__1::vector<double, std::__1::allocator<double>> const&) + 8412
    frame #10: 0x000000016c1cd4cc lib_lightgbm.so`.omp_outlined. + 792
    frame #11: 0x000000016bdaee5c libomp.dylib`__kmp_invoke_microtask + 156
    frame #12: 0x000000016bd6235c libomp.dylib`__kmp_invoke_task_func + 312
    frame #13: 0x000000016bd61500 libomp.dylib`__kmp_launch_thread + 400
    frame #14: 0x000000016bd9669c libomp.dylib`__kmp_launch_worker(void*) + 280
    frame #15: 0x000000018823e034 libsystem_pthread.dylib`_pthread_start + 136

In this case, LightGBM::DatasetLoader::ConstructFromSampleData has omp parallel and OMP_NUM_THREADS() (FindBin => ArgMax => ArgMaxMT => OMP_NUM_THREADS()) has omp single.
deadlock explanation code
#include <omp.h>
#include <stdio.h>

void fn() {
    #pragma omp single
    {
        puts("single");
        printf("omp_get_max_threads: %d\n", omp_get_max_threads());
        printf("omp_get_num_threads: %d\n", omp_get_num_threads());
    }
    // <-- implicit barrier (waits for all threads, including threads that did not execute omp single block)
    // deadlock here because other threads will not reach the barrier
}

int main() {
    printf("omp_get_max_threads: %d\n", omp_get_max_threads());
    printf("omp_get_num_threads: %d\n", omp_get_num_threads());

    #pragma omp parallel for schedule(static)
    for (int i = 0; i < 10; i++) {
        if (i == 1) {
            fn();
        }
    }
    puts("finish");
    return 0;
}

@microsoft-github-policy-service agree

Thanks for the update! We'll review them soon.
In the future when you contribute to LightGBM (and we hope you will!), don't force-push here. It's not necessary, as we squash all commits into 1 when merging to master: https://github.com/microsoft/LightGBM/commits/master/.


Linting and stricter TypeScript rules

Most of these look good. I'm not super excited about the last items in lists needing commas after them, and single parameters in lambda functions needing parenthesis, but 🤷

Most of these look good. I'm not super excited about the last items in lists needing commas after them, and single parameters in lambda functions needing parenthesis, but 🤷

Just to be clear, this kind of stuff can be auto-fixed on save, so it's not a big deal. I actually like the last items having commas, means that you don't need to change two lines when adding an element, so the diffs are cleaner :)


myid settings and zoo.cfg dynamic settings

Hi Guys;
Wanted to say you guys are doing an absolutely marvellous job. I noticed my zk cluster an mesos cluster does not register the slaves properly without the /var/lib/zookeeper/myid reflects the server ID as per the zoo.cfg.
How come that file is not being populated as its so fundamental to the operation of zookeeper and mesos?
or Am I missing something?
Thanks
Ali.

@aktarali generally speaking, it’s up to you as the operator to create that file with the appropriate value inside. There are a few ways to handle that:

In your wrapper cookbook, utilize a mechanism to uniquely provide each instance with its myid at initial bootstrap

This would make sense when using any service mechanism other than exhibitor

Use Exhibitor, which managed the myid for you

This would mean including the exhibitor cookbook, which actually wraps this cookbook, to manage your ZK cluster

We use Exhibitor at EverTrue, and I believe the Simple folks use it too, since they initially wrote the exhibitor cookbook.

Guys;
@jeffbyrnes @asenchi @daveyeu @3n @solarce
The method used by EverTrue and SimpleFinance does not seem truly dynamic. For that reason, I have been playing with using Chef CMDB (which is its actual reason for existing - to allow storage of node attributes) to grab the valued necessary for both zoo.cfg and myid.
My ruby and DSL skills are still limited hence why I have opted to use BASH. Is there anyway you guys would consider adding this functionality to master branch functionality. This method is working so good that I think its worth considering. Here is what I've added from all your help:
Created a wrapper cookbook:
Attributes:
set up zk attributes
default[:zookeeper][:config] = {
clientPort: 2181,
dataDir: '/var/lib/zookeeper',
tickTime: 2000,
initLimit: 11,
syncLimit: 17
}`
recipe: zk.rb
`i = 0
search(
:node,
"roles:zookeeper AND chef_environment:#{node.chef_environment}").sort.map do |n|
i += 1
node.set['zookeeper']['config']["server.#{i}"] = "#{n['fqdn']}:2888:3888"
end
bash 'add_myid' do
code <<-EOH
cat find / \! -type l -name zoo.cfg|grep hostname|cut -d"=" -f1|cut -d"." -f2 > /var/lib/zookeeper/myid
chown -R zookeeper:zookeeper /var/lib/zookeeper
rm -rf /etc/zookeeper/conf/zoo.cfg; ln -s /opt/zookeeper/zookeeper-3.4.6/conf/zoo.cfg /etc/zookeeper/conf/zoo.cfg
zookeeper-server-initialize --myid=cat /var/lib/zookeeper/myid --force
EOH
end
`
I could have added this to the same zk.rb file, but did it in a separate file to restart zk service, could have been handled differently I think, but I'm sure you clever guys can come up with a better method.
zk_service:
bash 'restart_zookeeper' do code <<-EOH   chown -R zookeeper:zookeeper /var/lib/zookeeper   service zookeeper stop   sleep 3   service zookeeper start EOH end
These modifications allow for zoo.cfg values to be identical on all nodes in the cluster, allow for myid to be grabbed from zoo.cfg and service reloaded. This is now working in a fully dynamic way without any external s3 configs or manual process or databag.
Your contributions and thoughts on this are welcome.
Ali.

@aktarali hope you don’t mind, I added some code block formatting to make that a bit easier to read.

Reworked in a more Chef-like fashion:
# attributes
set[:zookeeper][:config][:initLimit] = 11
set[:zookeeper][:config][:syncLimit] = 17

# recipe
i = 0
search(
  :node,
  "roles:zookeeper AND chef_environment:#{node.chef_environment}"
).sort.map do |n|
  i += 1
  node.set['zookeeper']['config']["server.#{i}"] = "#{n['fqdn']}:2888:3888"
end

include_recipe 'zookeeper::default'

file '/var/lib/zookeeper/myid' do
  content node['zookeeper']['config'].key("#{node['fqdn']}:2888:3888").sub 'server.', ''
  owner 'zookeeper'
  group 'zookeeper'
end

include_recipe 'zookeeper::service'

For the attributes, in your wrapper, you’ll want to use set (aka normal) not default precedence level, so you properly override the defaults in this cookbook. Additionally, you only need to set the ones that are different; otherwise, leave the defaults alone.
Then we find our ZK nodes & set the necessary attributes for the ZK config.
Then we install & configure ZK using the supplied zookeeper::default recipe.
Once we’ve got all the necessary files & directories, we create our myid file, searching the attributes to find this particular server’s ID. In this way, we ensure we use the correct ID for this particular server.
Rather than fight with the cookbook on where the config file lives with this symlinking dance, best to just use the supplied zoo.cfg, as it will be configured the way you want it to be, so we can omit all of that.
Finally, we include the service recipe, which actually sets up & starts the service itself, starting ZooKeeper.
There’s no need to restart ZK in your recipe (nor would you want to in live environment, owing to the potential for data loss), since you don’t start the service until after you’ve configured things. If you added a new ZK node, or removed one, you’d want to reconverge the existing nodes, then manually restart them in a rolling fashion to pick up the changes.

I don't see how this would allow nodes to be added in any shorter period of time than your Chef run interval and therefore why it's an improvement over using Exhibitor?
The BASH resource shouldn't be messing with zoo.cfg directly but relying on the "zookeeper_config[zookeeper config]" resource to set up zoo.cfg correctly.
Restarting zookeeper should be done by sending a :restart action to the "service[zookeeper]" resource so it can queued/deduplicated with other restart actions to that resource in the same Chef run (rather than using a BASH block to stop/start zookeeper).

@aktarali I’d be inclined to agree w/ @williamsjj, using Exhibitor is probably the strongest solution; I can say definitively that it’s given us a rock solid ZK infrastructure.

@aktarali @jeffbyrnes It takes Zookeeper a somewhat long time as it is to settle down when nodes join and leave...adding 15-30minutes of latency to that due to Chef runs would make the interval really painful.

@williamsjj oh man, so true. Even with Exhibitor, it can take a little while to calm down after a change.

Hi Guys;
Completely agree with all comments bar one. What I've tried doing is
removing the need to have any dependency on another tool/filestore/file
external to chef.
Some scenarios might explain my rationale:
1: When you first bring up a cluster - 3 nodes say.
Depending on how often you run chef-client (I run it every 15 min). After
the initial bootstrap, I expect my cluster to be fully up and running
within 15min.
2: Any new masters joining will join cluster with the existing configs from
chef-server. After the next run in 15 min, the other 3 will have the new
node configs, after another 15 min, the new node will get its config from
chef-server and complete its zoo.cfg. Since 3 or 5 or 7 is recommended for
a good cluster setup, the quorum will only need changing when your server
requirements reaches 5 or 7.
I know the time it takes for the initial cluster to come up could be a lot
quicker with a filestore. Subsequent addition to the cluster may take
longer. However, if rolling-in the new hosts to join an existing cluster is
not time constrained then I'm happy with it taking its time. I have in this
manner eliminated another dependency.
Really appreciate your help in this regard and all the feedback you guys
have been providing. Really awesome ppl to work with.
Cheers
Ali.
On 1 April 2016 at 20:35, Jeff Byrnes<EMAIL_ADDRESS>wrote:

@williamsjj https://github.com/williamsjj oh man, so true. Even with
Exhibitor, it can take a little while to calm down after a change.
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/SimpleFinance/chef-zookeeper/issues/162#issuecomment-204537665

--
Ali Aktar
Zircon Solutions LTD

Hi Jeff;
I've set the code as per your instructions:
Attributes:
set up mesos attribute
default[:zookeeper][:config] = {
clientPort: 2181,
dataDir: '/var/lib/zookeeper',
tickTime: 2000
}
#default['zookeeper']['config']['server.1'] =
'ip-172-31-40-37.eu-west-1.compute.internal:2888:3888'
set[:zookeeper][:config][:initLimit] = 11
set[:zookeeper][:config][:syncLimit] = 17
Recipe:
i = 0
search(
:node,
"roles:zookeeper AND chef_environment:#{node.chef_environment}").sort.map
do |n|
i += 1
node.set['zookeeper']['config']["server.#{i}"] = "#{n['fqdn']}:2888:3888"
end
include_recipe 'zookeeper::default'
file '/var/lib/zookeeper/myid' do
content node['zookeeper']['config'].key("#{node['fqdn']}:2888:3888").sub '
server.', ''
owner 'zookeeper'
group 'zookeeper'
end
include_recipe 'zookeeper::service'
Error:
When running chef-client, I'm getting an error:
================================================================================
Recipe Compile Error in /var/chef/cache/cookbooks/wrapper/recipes/zk.rb
================================================================================
NoMethodError

undefined method `sub' for nil:NilClass
Cookbook Trace:

/var/chef/cache/cookbooks/wrapper/recipes/zk.rb:38:in `block in

from_file'
/var/chef/cache/cookbooks/wrapper/recipes/zk.rb:37:in `from_file'

Relevant File Content:

/var/chef/cache/cookbooks/wrapper/recipes/zk.rb:
31:    i += 1
32:    node.set['zookeeper']['config']["server.#{i}"] =
"#{n['fqdn']}:2888:3888"
33:  end
34:
35:  include_recipe 'zookeeper::default'
36:
37:  file '/var/lib/zookeeper/myid' do
38>>   content
node.set['zookeeper']['config'].key("#{node['fqdn']}:2888:3888").sub
'server.', ''
39:    owner 'zookeeper'
40:    group 'zookeeper'
41:  end
42:
43:  include_recipe 'zookeeper::service'
44:
Running handlers:
Running handlers complete
Chef Client failed. 0 resources updated in 04 seconds
Can you assist?
Cheers
Ali.

@aktarali this is a straightforward; the value of node['zookeeper']['config'].key("#{node['fqdn']}:2888:3888") is nil, so the sub method doesn’t exist.
In other words, there isn‘t a key in the node['zookeeper']['config'] attribute with the value "#{node['fqdn']}:2888:3888". This might be a chicken & egg problem on first convergence of the first node of the cluster. A node’s Chef attributes aren’t written to the Chef Server until after first convergence completes, and so, the first node won’t find any nodes with that search, and thus, won’t populate node['zookeeper']['config'] with any servers (i.e., node['zookeeper']['config']['server.1'] doesn’t exist, nor any other numbered servers).
To your earlier point, how do you plan to handle when a ZooKeeper node goes down? As in, the underlying hardware (whether that’s bare steel or a virtual machine) fails, and now you need to quickly re-provision & add a new node?

@JeffByrnes I've gone passed that problem by reordering the run order. The
other problem I now have is that after myid file is updated, the zookeeper
process
is not restarting and as a result the node is not picking up its new myid.
Any thoughts?
Ali.
On 4 April 2016 at 14:32, Jeff Byrnes<EMAIL_ADDRESS>wrote:

@aktarali https://github.com/aktarali this is a straightforward; the
value of node['zookeeper']['config'].key("#{node['fqdn']}:2888:3888") is
nil, so the sub method doesn’t exist.
In other words, there isn‘t a key in the node['zookeeper']['config']
attribute with the value "#{node['fqdn']}:2888:3888". This might be a
chicken & egg problem on first convergence of the first node of the
cluster. A node’s Chef attributes aren’t written to the Chef Server until
after first convergence completes, and so, the first node won’t find any
nodes with that search, and thus, won’t populate
node['zookeeper']['config'] with any servers (i.e.,
node['zookeeper']['config']['server.1'] doesn’t exist, nor any other
numbered servers).
To your earlier point, how do you plan to handle when a ZooKeeper node
goes down? As in, the underlying hardware (whether that’s bare steel or a
virtual machine) fails, and now you need to quickly re-provision & add a
new node?
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/SimpleFinance/chef-zookeeper/issues/162#issuecomment-205297193

--
Ali Aktar
Zircon Solutions LTD

@aktarali I'm not sure you should change the myid for an existing node. That sounds bad as I understand it.

@neil-greenwood
For an initial cluster setup, automating the zookeeper install and allocating its correct MYID on the fly is a little cumbersome. Once MYID is set (from the values in zoo.cfg) zookeeper service has to be restarted with a chef resource handler "notifies" for service to restart if file has changed. When zookeeper is first initially provisioned, myid file does not exist and renders the Mesos cluster unusable. Myid has to be set somehow and service reloaded. I have not re-allocated MYID from one node to another, its a fair point, I wonder what would happen? If all nodes are sync'd, would it matter?
I can't wait for Mesos to adopt consul. ZK is a bad choice i believe for mesos.

I think something similar to this in chef would work:
bash 'add_myid' do   user 'zookeeper'   group 'zookeeper' code <<-EOH   host=hostname   file=find / ! -type l -name zoo.cfg   zoovalue=grep $host $file|cut -d"=" -f1|cut -d"." -f2  if [[ $zoovalue -eqcat /var/lib/zookeeper/myid]]   then      echo "Nothing to do as myid file value is correct as per zoo.cfg"   else      catfind / ! -type l -name zoo.cfg|grep hostname|cut -d"=" -f1|cut -d"." -f2 > /var/lib/zookeeper/myid   fi EOH   notifies :restart, 'service[zookeeper]', :immediately end

@aktarali so the myid file should be created by this resource I provided to you:
file '/var/lib/zookeeper/myid' do
  content node['zookeeper']['config'].key("#{node['fqdn']}:2888:3888").sub 'server.', ''
  owner 'zookeeper'
  group 'zookeeper'
end

However, as I said, you’re gonna run into a chicken & egg problem on first convergence. You mentioned you solved that by changing the recipe run order.
I should point out, you do not want to change a node’s myid, ever, once it’s live. So you want to start it up with the correct myid in place already. This is why I placed include_recipe 'zookeeper::service' after the file[/var/lib/zookeeper/myid] resource, so that the service would not be started until the myid was in place.
I’m going to have to bow out of providing any more assistance at this point, I just don’t have any more cycles right now. My recommendation stands to use Exhibitor to take care of all of this for you. We have a system that looks like this:

ZooKeeper, managed by Exhibitor, w/ a shared config living on S3
Mesos
Kafka
Storm
Spark
Cassandra

Everything works brilliantly, I do not wrangle with the myid file at all, and I haven’t touched our ZK cluster in months, maybe even a year. I appreciate the desire to decrease your dependencies, but I’d encourage you to take advantage of what the ecosystem has provided. Netflix already solved this problem for us, and we have a great Exhibitor cookbook to help you do it with Chef.

@aktarali Really have to second @jeffbyrnes here. I hate having extra dependencies, but Exhibitor is actually the way to go:

It's helpful to think of Exhibitor as a process supervisor like daemontools or runit, rather than as a heavy weight dependency. A process supervisor that also handles Zookeeper's temperamental nature when adding and subtracting nodes.
If you deep dive on Zookeeper's docs and design blog articles by its creators, you'll see they say that Zookeeper was designed to be run by a supervisor...they (in typical ZK fashion unfortunately) just didn't provide one. Thankfully Netflix built one in Exhibitor.
The cost of Exhibitor S3 usage is completely negligible. Under $5 a month.

Thats great guys. Really appreciate your help.

@aktarali I believe we’ve answered your questions to the best of our abilities. I’m going to close this out; please feel free to re-open or create a new issue if you have further questions.


Delay between the last "DRAG" and "DRAGEND" events

I got something which looks a lot like the 300ms delay issue: there is a very noticeable latency when I release the card.
I added some logs for all the "dragstart", "drag" and "dragend" events, and there is clearly a pause between the last "drag" event and the "dragend" event.
I observe exactly the same delay here : http://ionicframework.com/demos/swipe-cards/ or here http://codepen.io/ionic/pen/nxEdH
I mainly have tested on the iOS virtual machine, the android virtual machine, my browser (Chrome), and 2 android physical devices through Ionicview.

Another observation:
if I completely drag the card out of screen, I do not need to release the drag. The "dragend" event is immediately fired as soon as my cursor is out of the virtual machine.
Same behavior observed on the android virtual machine.
I do not have an android phone available at the moment (and even less an iPhone) so I cannot test for real.
But can you already tell me if you observe the same latency on the examples I gave previously, and if this latency then disappear on the real device?

i've tested on my android device with ionic view and the performance was terrible, i'm searching if i did something wrong.
Someone have tested it?


JENKINS-30115 Allow downstream jobs to attach GitHub status messages

Added support for GitHub contexts and messages to be added from downstream jobs
This pull request is related to this issue: https://issues.jenkins-ci.org/browse/JENKINS-30115

Thank you for a pull request! Please check this document for how the Jenkins project handles pull requests

Nice! @DavidTanner are you up for a review? :)

When will this be merged? :) So looking forward to this feature.

@kevinsuwala I compiled your branch and updated my jenkins' ghprb plugin. How do I configure downstream jobs to send contexts to upstream jobs?

Please check my pull request against your branch.

Closed for #180


Add lighten/darken blend modes

For Issue #294, lighten and darken blend modes + tests.
Unfortunately I still can't get the xunit test runners working, so I can't check my logic here. Any takers for running the unit tests? It was straightforward enough that I feel confident (famous last words).

I think the xunit/Resharper issue is due to xunit not doing any updates until Corefx hits RC2. I am able, however, to run all my tests via the Test Explorer window in Visual Studio 2015 without issue.

I think the public will want to use them also so yeah we should probably offer both variants.

Ok cool I'll get to fixing the implementation here and only use the 'pre' version for internal stuff. I'm sorry it was rather roundabout to get to the core of the issue but I'm glad we figured it out :+1:

Don't be sorry chum, Discussion about it is great, we're both learning this stuff as we go along. :smile:

Ok, I've been mulling it over. We have a color object but ATM it doesn't track what the current status it is - it just 'is' a color and we have no way of marking it pre-multipied or not. Even though we know our colors are premultiplied, the users need to have an option to 'fix' their color.
Options -

"Right Way" - Add a default-valued bool 'IsPreMultiplied' to the constructor of Color (using telescoping constructor pattern if necessary). We can default it to the scenario most users will require, then the implementations of blend modes and such can just simply use the correct implementation of the algorithms depending on circumstance. It also gives us a really simple pattern for identifying blend mode color errors -
Client - "Hey the color is slightly off."
AUSD (Abused Unpaid Support Dev :)): "Try switching to post-multiplied in the constructor!"
Client -
Ghetto Easy - Provide a PreMultiplyX + PostMultiplyX and let the user decide which their color is. This just seems like an awful experience for the end user, who would think that there method by which we perform lighten wouldn't be intelligent enough to know?
Overkill Painful - Make color itself and abstract and have two derived color classes - PreMultipiedColor + PostMulipliedColor - then provide either a static class on the base color that changes the implementation based on type (oh god no) or just provide static implementations on each derived class. I hate this option don't pick it.

What do you think is the best?

Options -

"Right Way" - Add a default-valued bool 'IsPreMultiplied' to the constructor of Color (using telescoping constructor pattern if necessary). We can default it to the scenario most users will require, then the implementations of blend modes and such can just simply use the correct implementation of the algorithms depending on circumstance. It also gives us a really simple pattern for identifying blend mode color errors -

By default all the (en|de)coders convert to and from premultiplied color to you would end up with color going off on save. Whatever we do, it should be consistent across the board. I'm very wary of adding such a value to the constructor.

Ghetto Easy - Provide a PreMultiplyX + PostMultiplyX and let the user decide which their color is. This just seems like an awful experience for the end user, who would think that there method by which we perform lighten wouldn't be intelligent enough to know?

Color already offers FromNonPremultipliedand ToNonPremultiplied.

Overkill Painful - Make color itself and abstract and have two derived color classes - PreMultipiedColor + PostMulipliedColor - then provide either a static class on the base color that changes the implementation based on type (oh god no) or just provide static implementations on each derived class. I hate this option don't pick it.

I agree, let's not do this.

What do you think is the best?

Ummmmm.... Honestly, I don't know for sure. This is what is done in Lerp. I'm assuming that if the color's alpha component is less that 1 then the color is premultiplied then running two different algorithms. As long as the methods is well documented then I don't see an issue with that.
public static Color Lerp(Color source, Color destination, float amount)
{
    amount = amount.Clamp(0f, 1f);

    if (Math.Abs(source.A - 1) < Epsilon && Math.Abs(destination.A - 1) < Epsilon)
    {
        return source + ((destination - source) * amount);
    }

    // Premultiplied.
    return (source * (1 - amount)) + destination;
}

However we would have to carefully check the output of such methods to ensure consistency.

Isn't the A value completely independent of whether or not you're dealing with pre or post multiplied colors? That's why you can still convert to and from post once you've gone to pre (albeit lossily). Is Lerp accurate for this scenario?
Remember the constructor could have a default value so most people wouldn't notice, and changing implementations of converters would be fairly straightforward (based on which conversion you were doing).
If anyone, including us, plan to work with colors in any way pre or post status is something that belongs inside the color itself so that we can make intelligent and accurate decisions.

As far as I understand it, my Lerp is correct. You can see it in action in the filter tests. Vignette, Blend and Polaroid all use it. I can't remember where I found the correct example but if I use either formula for both scenarios the result is incorrect.
Perhaps a readonly property? IsPremultiplied should be used which is toggled internally when FromNonPremultiplied and ToNonPremultiplied. That way the user can know for sure?

RE: Lerp accuracy, I think we need a scholarly article or something. At least according to http://stackoverflow.com/questions/9147973/how-do-i-know-if-a-image-is-premultiplied-alpha, you can't know if it is except in one specific case, and it doesn't seem like the Lerp impl is checking for the non ambiguous case.
What's I'm proposing is (quick switch here because bool is weird for these two scenarios):
 public enum Multiply {        Pre,        Post, } public Color(float r, float g, float b, float a, Multiply colorMultiply=Multiply.Pre)             : this()         {             this.backingVector = new Vector4(r, g, b, a);             this.ColorMultiply = colorMultiply;         }
Most users would be initializing colors as pre-multiplied because most image formats use pre-multiplied, if I remember from some of the documentation I've read. If they know for a fact it will be Post they can initialize it as such. The implementations of color blending modes can perform a check for IsPreMultiplied and perform their work much easier. For  "ToNonPremultiplied" we can throw an exception ("Already non pre-multiplied!") or ignore calls to it if the color is already set to post-multiplied.

The amount of raw information in the spec _should _really be enough  unfortunately I simply don't understand it enough. I think we should try to set our color struct and all methods to work in exactly the same way as the W3 rgba color.
https://www.w3.org/TR/compositing-1/

The spec is precisely why we're having this conversation - the blend mode needs to know whether the color is pre or post in order to determine how the mixing occurs in relation to backdrop alpha. I wish the spec provided more guidance for that but they seem to be assuming - somehow - you know what the pre or post status of the working color is prior to blending. We need that information somehow as well.

OK then, well let's say this....
Any static method attached to Color expects premultiplied alpha. Anything higher up in the API that takes a color as a Color parameter will first convert using Color.FromNonPremultiplied(color);
I've just realised that we already do this in
public static Image BackgroundColor(this Image source, Color color, ProgressEventHandler progressHandler = null)
{
}

What I need to know now though is why does my premultiplied Lerp algorithm not work with an opaque color?
Would that be sensible?

Initialization is the time at which we would know the most about the color's origin (pre or post).
Blend modes can even perform the backdrop blending in 1 function that all blend modes call, providing just the implementation of the mixing function as the final argument.
Is there a reason this isn't a good idea?

myColor.ToPostMultiply().ToPostMultiply().ToPostMultiply().ToPostMultiply();

You just know someone will do that!
Ok let's go with the enum in the constructor with a default value. Thanks for being so patient!

We both just want IP to be successful and easy to use - I thought it was a thought-provoking discussion! I'll probably do the color struct change in a separate branch and merge that, then fix the implementation in this branch.

Hey @christopherbauer I got rid of premultiplied as the default color format so this should all be a lot easier now. Whoop!

Hey guys, I'm trying to get into implementing a new blend mode but I'm running into some trouble. I'm fairly new to this project so I have a fairly silly question to ask :)
Say I have implemented a new blend mode, how would I go about testing it? I guess I'm confused because it's not (obviously) a part of the API. Hope this makes sense :)


fix(Bootstrap): remove deprecated classes

Describe the PR
This PR remove all the deprecated classes mentioned on https://github.com/bootstrap-vue-next/bootstrap-vue-next/issues/1779#issuecomment-1999436419
PR checklist

What kind of change does this PR introduce? (check at least one)

[x] Bugfix :bug: - fix(...)
[ ] Feature - feat(...)
[ ] ARIA accessibility - fix(...)
[x] Documentation update - docs(...)
[ ] Other (please describe)

The PR fulfills these requirements:

[x] Pull request title and all commits follow the Conventional Commits convention or has an override in this pull request body This is very important, as the CHANGELOG is generated from these messages, and determines the next version type. Pull requests that do not follow conventional commits or do not have an override will be denied

@anrolmar @dwgray @VividLemon  is this pr still in need of changes?

I'll create a new pr based on this.


Declarative services dependency for karaf feature

depend on scr feature to bring up declarative services automatically
this build on previous PR's #130 , #131

Similar to https://github.com/atlassian/commonmark-java/pull/131#issuecomment-405460021, I'm declining this.


fix: code review comments and autosave

:tada: This PR is included in version 0.369.1 :tada:
The release is available on:

npm package (@next dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


[stable/prometheus-operator] Remove dependencies to support the chart being used in an umbrella chart

Is this a request for help?: No

Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST

Prometheus operator is a component that a lot of people consider a core part of setting up a kubernetes cluster. Therefore it should be able to use it in an umbrella chart. To my understanding there is currently issues with enabling/disabling subsubcharts through requirement conditions. This means that the only viable solution would be to not have those dependencies in the chart.
Version of Helm and Kubernetes: Helm 2.11.0, K8s 1.11.2
Which chart: stable/prometheus-operator
What happened:
I tried to use the chart in our organization umbrella chart, and tried to disable grafana by setting
prometheus-operator:
  grafana:
    enabled: false

I suspect this is currently not supported due to this issue: https://github.com/helm/helm/issues/2479
What you expected to happen:
I would expect that non of the resources from the grafana subsubchart would be generated.
How to reproduce it (as minimally and precisely as possible):
Depend on the chart in an umbrella chart and see if you can disable grafana
Anything else we need to know:
Related issue: https://github.com/helm/helm/issues/2479
Umbrella charts: https://github.com/helm/helm/blob/master/docs/charts_tips_and_tricks.md#complex-charts-with-many-dependencies

I'm using prometheus-operator as a subchart of my own chart, and values are propagated properly to it. How is your scenarios is different from mine? Your description (answer to what you did, and what you expect to happen) is not clear at all at current state.

@paskal are you configuring any of the subcharts of prometheus-operator?

Yes, of course, here is my values.yaml:
prometheus-operator:
  # remove parent chart name from this one
  fullnameOverride: prometheus-operator
  fullname: prometheus-operator
  # Kube-scheduler on gke
  kubeScheduler:
    enabled: false

  # Controller Manager on gke
  kubeControllerManager:
    enabled: false

  # kubeDNS instead of coreDns on gke
  kubeDns:
    enabled: true
  coreDns:
    enabled: false

  # etcd on gke
  kubeEtcd:
    enabled: false

  grafana:
    # set label to reuse in prometheus-config
    sidecar:
      dashboards:
        label: grafana_dashboard
    # anonymous access
    grafana.ini:
      auth:
        disable_login_form: true
        disable_signout_menu: true
      auth.basic:
        enabled: false
      auth.anonymous:
        enabled: true
        org_role: Editor

@paskal It might be related to me changing the boolean used as condition in requirements.yaml. I will see if I can reproduce it again on a new chart in a few minutes.

I also tried to improve the issue description a little to make it more clear.

By the way, it's important how do you use that chart: if your version of prometheus-operator is unaltered, you should just run helm dependency update in root of your parent chart, and it will fetch prometheus-operator tar.gz to charts folder. If you have in unpacked, then you need to run same command in charts/prometheus-operator/ directory so it will fetch it's own dependencies.

I think this issue is really a matter of deciding wether or not the prometheus-operator chart should have subcharts which are able to be toggled on and off. That functionality does not work when using it in an umbrella chart.
I would argue that it shouldn't have grafana at least. For people who want an all-in-one installation, it would be better to have a separate chart for that which installed and configured both the prometheus-operator chart along with any related charts eg. grafana.
We don't run grafana along with our prometheus, and we don't want to. Therefore we cannot use prometheus-operator in our umbrella chart. We currently workaround this issue by separately installing our core umbrella chart and prometheus-operator.

FYI there are some issue which is present only in 11.2+ versions of k8s, https://github.com/helm/helm/issues/4925. I'll try to reproduce your issue and understand what is wrong.

I was able to reproduce it.
To fix the issue you need to:

download prometheus-operator to charts directory of your parent chart
issue helm dependency update command in /charts/prometheus-operator
change condition: grafana.enabled to condition: prometheus-operator.grafana.enabled in /charts/prometheus-operator/requirements.yaml

This will fix the problem for you, downside of that is that you need to manually do git pull to update prometheus-operator code.
I'm against excluding grafana from prometheus-operator requirements and I think what way I've mentioned is good enough if you need such setup.

I dislike the idea of changing stuff in the charts folder. Then I would need to commit the hacked prometheus-operator chart to our umbrella chart.
Grafana is used for a lot more than just prometheus-operator, so to me it doesn't make sense that the prometheus-operator chart should claim ownership of configuring grafana. I understand that it is nice to be able to template out the grafana configurations based on how prometheus-operator is configured, but it is tight coupling between the two, which I personally would like to avoid.
At it's current state the prometheus-operator chart is not what it says to be. Currently it is a prometheus-operator + grafana chart.
When you run multiple kubernetes clusters it is nice to be able to access metrics across them. To me that means having 1 grafana outside the clusters and multiple prometheuses inside the clusters. That is the main reason why I dislike this hard coupling.

@blacksails Using your original example can you please try this:
grafana:
  enabled: false
prometheus-operator:
  grafana:
    enabled: false


Poorly written docs - improvement required

I am writing to request that this page is rewritten to provide further clarity around the impact Tamper Protection has on valid configuration mechanisms such as SCCM/Intune/GPO etc.
I recently had to raise a support case as some settings we had enabled in SCCM werent taking effect as expected. After many hours of troubleshooting it transpired these settings were TP protected and therefore will not change from the defaults regardless of the settings in SCCM.
The existing documentation states clearly that  "With tamper protection, malicious apps are prevented from taking actions"
This statement should also clarify that non malicious management solution apps such as SCCM and Intune will also be prevented from making changes! There is a blue highlighted box beneath the overview section that does state the impact TP has on  Intune settings - why does this not include other management mechanisms that are also impacted?
It also states that in order to obtain the list of TP protected settings we should contact support. This should not be the case at all, as a full list of tamper protected settings should be clearly published.
To further add to the lack of cohesion in the existing docs, the section entitled "How Tamper Protection Works" (which is immediately below the blue box that states the impact to Intune pushed settings) states:
Tamper protection essentially locks Microsoft Defender Antivirus to its secure, default values, and prevents your security settings from being changed through apps and methods such as:

Configuring settings in Registry Editor on your Windows device
Changing settings through PowerShell cmdlets on your device

Again, there is no mention of the impact TP has on settings pushed from management platforms such as Intune or SCCM!
Had the docs been better written and more cohesive, the support ticket I had to raise and many hours of lost time could have been avoided.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: f31e7b39-b7c3-188a-0b2a-02ff10a5be6d
Version Independent ID: 44940a13-0a2d-e281-560b-cb4d8a5d1ccf
Content: Protect security settings with tamper protection
Content Source: microsoft-365/security/defender-endpoint/prevent-changes-to-security-settings-with-tamper-protection.md
Service: microsoft-365-security
Sub-service: mde
GitHub Login: @denisebmsft
Microsoft Alias: deniseb

Hello @pjrcdf and thank you for your feedback. This issue appears to be a lot like #11179. I am looking into this and will get the docs updated accordingly. My apologies for the inconvenience. Thank you so much for filing this issue so we can fix it.

Hello @pjrcdf.  I wanted to let you know that the tamper protection docs have been updated and the changes went live this afternoon. Doc updates include changes to provide more clarity around what happens when tamper protection is enabled. Please let me know if you have any other questions or feedback. Thank you again for posting this issue and helping me improve our documentation.


Select - items list to be triggered on stateChange outside Select component

Is your feature request related to a problem? Please describe.
Trying to use SingleSelect  , passing in an array of strings to items.
However, this array is created during render, and when click on the SingleSelect, the pop-down list is empty.
If I add a character (preferably 'a') in the SingleSelect field, the list populates, but not before.
I´ve tried UseEffect in order to get a re-render when the array is populated, but this has no effect on the SingleSelect list. The only way I get the list populated is when the list is hardcoded  as a constant as in the example you show in the StoryBook.
So I think this occurs because SingleSelect (or any of the EDS selects) does not trigger of state changes after initial render.
Describe the solution you'd like
The ability to pass in a list of items in Select which is created during render and stored in a useState variable, and control re-render with useEffect (as possible with material UI select.)
Describe alternatives you've considered
The workaround (not elegant) I need to do now to get the list populated is to delay render of the Select until I know that the items list is ready.
{displayFields.length > 0 &&
     <SingleSelect id={"fields"} label={"Fields"} items={displayFields} handleSelectedItemChange={onChangeFieldFilter} />
}

The solution I would like would be something like this:
useEffect(() => {
  if (fields) {
      const list = fields.map((field) => field.name);
      setDisplayFields(list);
    }
  }, [])};

return (
    <SingleSelect id={"fields"} label={"Fields"} items={displayFields} handleSelectedItemChange={onChangeFieldFilter} />
)

Additional context

Thanks for reporting this!
We'll have a look at soon as possible, question is if this is a bug or not, because you should be able to have dynamic content 🤔  Is the content for select fetch from somewhere asynchronously?
If possible a codesandbox would be perfect for us to test against this specific use-case (instead of us trying to replicate 😅)

Thanks for reporting this!
We'll have a look at soon as possible, question is if this is a bug or not, because you should be able to have dynamic content 🤔  Is the content for select fetch from somewhere asynchronously?
If possible a codesandbox would be perfect for us to test against this specific use-case (instead of us trying to replicate 😅)


[docs-infra] Remove emoji from callouts

[x] I have followed (at least) the PR section of the contributing guide.

This PR removes the now-not-needed emojis in callouts to illustrate their severity! All Markdown changes that aren't related to that were probably because of Prettier. These were the instances I could find through using the global search on my editor. 😬
Closes https://github.com/mui/material-ui/issues/38689

I removed the last one I found with the search
:::info
[^A-Z]

:::warning
[^A-Z]

:::error
[^A-Z]

[^A-Z] means "not a character between A and Z

@alexfauquette any idea why the tests are frozen?


Another attempt at fixing the race between cache and sinks

Fixes #509

Thank you so much @vishh for this PR! I'll take a look into it today.

LGTM from me, but wait for @mwielgus LGTM as well

LGTM

Failing test is flaky, merging.

Apparently some other, this time related test is failing too:
--- FAIL: TestSyncLastUpdated (3.47s)
        Location:       external_test.go:191
	Error:		Not equal: time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)} (expected)
			        != time.Time{sec:63577396626, nsec:894450989, loc:(*time.Location)(0xb90f00)} (actual)
	Messages:	now: 2015-09-09 13:48:06.894450989 +0200 CEST, expected: 0001-01-01 00:00:00 +0000 UTC, actual: 2015-09-09 13:57:06.894450989 +0200 CEST

FAIL


landdetector groundcontact: adjust climbrate if landing speed is low

The threshold for no vertical movement is a constant value of 0.5. If the parameter landspeed is set low (for instance below 0.5), then the landdetector will always detect no vertical movement. This PR is a protection for having low landspeed.
issue:
https://github.com/PX4/Firmware/issues/7811

@PX4TestFlights Please cross test by flying very low and very slow with low land speed in position control mode.

@bkueng, can you please retest with the Intel Aero?

I quickly checked LAND_SPEED and it seems to be 0.5m/s on the aero we tested.


Remote writing NETCDF4 files to Amazon S3

Hi all,
I am trying to use xarray's .to_netcdf() function to write an array remotely to a file in an Amazon S3 bucket in NETCDF4 format, but I have absolutely no idea how to do it. So far I have been able to write locally, but it all completely falls apart when I try and provide something that isn't a local path to the .to_netcdf() function. Example code is provided below.
import boto3
import json
import numpy as np
import xarray as xr
with open('<MY_PATH>/boto_test_credentials', 'r') as f:
secrets = json.load(f)
sn = secrets['service_name']
aaki = secrets['aws_access_key_id']
asak = secrets['aws_secret_access_key']
eu = secrets['endpoint_url']
session = boto3.session.Session()
s3 = session.client(
service_name = sn,
aws_access_key_id = aaki,
aws_secret_access_key = asak,
endpoint_url = eu,
)
x = xr.DataArray(np.random.randn(450, 450))
Then finally something like this?
x.to_netcdf(<URL_TO_MY_AMAZON_S3_BUCKET>)
I've thought about trying to trick the .to_netcdf() function by using the urllib module to convert a URL into a file-like object, but I don't understand thing like the authentication process etc well enough, and I don't even know if I'm heading down a blind alley here. Any advice would be most appreciated!

Hi @Non-Descript-Individual
I've found that the netcdf4-python library really wants to have direct access to a disk/filesystem to work, it also really wants to do its own file access management. I've always attributed this to the python library being a wrapper for the netcdf C library.
My guess would be that the easiest way to do what you want is to separate the writing of the netcdf file step in xarray from the putting the file into S3. Something like this:
x.to_netcdf('temp_file.nc')
s3.upload_file('temp_file.nc', 'bucketname', 'real_name_for_temp_file.nc')

The netcdf4-python library does seem to provide an interface for the "diskless" flags. In this case, from the examples it looks to give you a bunch of bytes in a memoryview object on calling close(). I'm not sure this is accessible from xarray though.
Alternatively, @rabernat is an advocate of using zarr when putting netcdf compatible data into cloud storage, the zarr docs provide an example using s3fs

Forget about zarr for a minute. Let's stick with the original goal of remote access to netcdf4 files in S3. You can use s3fs (or gcsfs) for this.
import xarray as xr
import s3fs
fs_s3 = s3fs.S3FileSystem(anon=True)
s3path = 'era5-pds/2008/01/data/air_temperature_at_2_metres.nc'
remote_file_obj = fs_s3.open(s3path, mode='rb')
ds = xr.open_dataset(remote_file_obj, engine='h5netcdf')

<xarray.Dataset>
Dimensions:                      (lat: 640, lon: 1280, time0: 744)
Coordinates:
  * lon                          (lon) float32 0.0 0.2812494 ... 359.718
  * lat                          (lat) float32 89.784874 89.5062 ... -89.784874
  * time0                        (time0) datetime64[ns] 2008-01-01T07:00:00 ... 2008-02-01T06:00:00
Data variables:
    air_temperature_at_2_metres  (time0, lat, lon) float32 ...
Attributes:
    source:       Reanalysis
    institution:  ECMWF
    title:        "ERA5 forecasts"
    history:      Wed Jul  4 22:08:50 2018: ncatted /data.e1/wrk/s3_out_in/20...

This takes about a minute to open for me. I have not tried writing, but this is perhaps a starting point.
If you are unsatisfied by the performance of netcdf4 on cloud, I would indeed encourage you to investigate zarr.

This takes about a minute to open for me.

It took me much longer earlier this week when I tried :roll_eyes: Is the bottleneck in the parsing of the coordinates?

Thanks @rabernat I had forgotten about the other netcdf storage engines... do you know if h5netcdf stable enough that I should use in "production" outside of xarray for my netcdf4 reading/writing needs?

Is it possible to read mulitple netcdf files on s3 using open_mfdataset?

I ran into this issue, here's a simple workaround that seems to work:
def dataset_to_bytes(ds: xr.Dataset, name: str = "my-dataset") -> bytes:
    """Converts datset to bytes."""

    nc4_ds = netCDF4.Dataset(name, mode="w", diskless=True, memory=ds.nbytes)
    nc4_store = NetCDF4DataStore(nc4_ds)
    dump_to_store(ds, nc4_store)
    res_mem = nc4_ds.close()
    res_bytes = res_mem.tobytes()
    return res_bytes

I tested this using the following:
import BytesIO

fname = "REDACTED.nc"
ds = xr.load_dataset(fname)
ds_bytes = dataset_to_bytes(ds)
ds2 = xr.load_dataset(BytesIO(ds_bytes))

assert ds2.equals(ds) and all(ds2.attrs[k]==ds.attrs[k] for k in set(ds2.attrs).union(ds.attrs))

The assertion holds true, however the file size on disk is different. It's possible they were saved using different netCDF4 versions, I haven't had time to test that.
I tried using just ds.to_netcdf() but get the following error:
`ValueError: NetCDF 3 does not support type |S32`

That's because it falls back to the 'scipy' engine. Would be nice to have a non-hacky way to write netcdf4 files to byte streams. :smiley:

That's because it falls back to the 'scipy' engine. Would be nice to have a non-hacky way to write netcdf4 files to byte streams. 😃

I agree, this would be a welcome improvement!
Currently Dataset.to_netcdf() without a path argument always using the SciPy netCDF writer, which only supports netCDF3. This is mostly because support for bytestreams is a relatively new feature in netCDF4-Python and h5py.

It looks like #23 is related. Do we have a plan about this?

Hi all,
I'd love to have an effective method to save a netcdf4 Dataset to a bytes object (for the S3 purpose specifically). I'm currently using netcdf3 through scipy as described earlier which works fine, but I'm just missing out on some newer netcdf4 options as a consequence.
Thanks!


Get started page: image and text updates for clarity

Short-term:
P1

[ ] find resources to help with images as needed

P2

[ ] Fix images - add drop shadow around them / get new images for whole page.

Long-term:
Feed info from user journey research into this page, considering how people might navigate through this

@ericlindley-g i'm not entirely sure how to break this one out....

Gotcha—it's a little awkward, because there's the larger task of user research.
If there are small things that we can do in the short term to improve the support we offer users, then let's make those changes (text, simple illustrations or screenshots)
But for the bigger changes, I vote to wait for the data from research. WDYT?

oh i just meant i will leave it up to you to break this out into separate tickets to prioritize. :) i think you already have one for the first one.


Allow annotations on enum constants

See error code ANNOTATION_ON_ENUM_CONSTANT
Why the restriction? Hitting this here: https://github.com/dart-lang/json_serializable/issues/38

The spec indeed doesn't allow annotations on enum item declarations.
That's is probably an oversight, so we should just fix it in Dart 2.

Duplicate of #23441


Openni2 ir stream

resolves #4366 (kinda, not tested with ASUS sensor)
What does this PR change?
Add support for IR image retrieval using OpenNI2. Tested with:

Orbbec Astra
Kinect 1

Looks good to me!
Could you please squash all commits into one?

@alalek sure, squashed

:+1:


Artifact Name not being populated when parameterized in a Loop

Working on looping through a list of artifacts and publishing via the PublishPipelineArtfact and having it wrapped in a template. The loop value is being passed in for the targetPath; however the artifact value does not seem to get recognized.
I've tried both w/ and w/o single quotes, also with and with out the () surrounding the loop value. Can see the dacpacName value is used in target path and that piece is working.
task template:
parameters:
    targetPath: ''
    artifactName: ''
steps:
- task: PublishPipelineArtifact@1
  inputs:
    targetPath: ${{parameters.targetPath}}
    artifact:  ${{parameters.artifactName}}

Calling Loop
parameters:
- name: dacpacNames
  type: object
- name: solutionName
  type: string
- name: configurationName
  type: string
  default: 'Debug'

jobs:
- job: Build_Publish__dacpacs
  displayName: Build Publish dacpacs
  steps:
  - template: ../tasks/vs_build_task.yml@YAMLTemplates
  - ${{ each dacpacName in parameters.dacpacNames}}:
    - template: ../tasks/ado_publish_pipeline_artifact.yml@YAMLTemplates
      parameters:
        artifact: '$({{ dacpacName }})'
        targetPath: '$(Pipeline.Workspace)\s\${{parameters.solutionName}}\bin\${{ parameters.configurationName }}\${{ dacpacName }}.dacpac'

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 861ce7a5-4fa4-55f1-2c97-46a9cc6a2f66
Version Independent ID: bdf954b9-9bbe-8b34-b674-71ad8153519d
Content: Publish and download artifacts in your pipeline - Azure Pipelines
Content Source: docs/pipelines/artifacts/pipeline-artifacts.md
Product: devops
Technology: devops-cicd-artifacts
GitHub Login: @ramiMSFT
Microsoft Alias: rabououn

Closing....parameter is artifactName not artifact and default was blank


Fixed a potential stack overflow

At
https://github.com/KoynovStas/wsdd/blob/46ba07be8510bdef6e9213badafb8d71e4f7e60d/src/wsdd.c#L337
We pass the memory address of struct imr_address to function get_addr_of_if, then get_addr_of_if will copy sizeof(struct sockaddr_in) amount memory to this address.
https://github.com/KoynovStas/wsdd/blob/46ba07be8510bdef6e9213badafb8d71e4f7e60d/src/net_utils.c#L181-L193
However, sizeof(struct sockaddr_in) is larger then sizeof(struct imr_address) , so this could cause a potential stack overflow if the compiler doesn't preserve enough memory for stack, or could just mess up with the stack. (For example on a arm machine)
So I fixed it. Tested on both x86 and arm, worked just fine.

Thanks for the pull request!
Sorry, but your code is not correct - it does not compile!.
You forgot to change the function definition in the header file.
You forgot to set the addr_len parameter for get_addr_of_if in the void init_gsoap() function.
The get_addr_of_if function takes the second argument (Address families), which
must be AF_INET or AF_INET6!
If you use the get_addr_of_if function, you yourself must monitor the parameters passed.
For the AF_INET type, you must use struct sockaddr_in
For type AF_INET6, you must use struct sockaddr_in6
In general, you can use struct sockaddr_storage,
but then you will need to use additional static_cast (struct sockaddr_in) or (struct sockaddr_in6)
See the note in http://man7.org/linux/man-pages/man7/ipv6.7.html

The sockaddr_in6 structure is bigger than the generic sockaddr.
Programs that assume you can
struct sockaddr for sockaddr_storage
that instead.

We use wsdd for the address #define WSDD_MULTICAST_IP "<IP_ADDRESS>"
This is IPv4 and we use the AF_INET type and struct sockaddr_in

Sorry, My mistake. I've tested it on an other project that I'm currently working on. I've issued an other pr. Please review.


Quiet protobuf warning in XCode build

Jenkins Build Summary
Built from this commit
Built at 20180315 - 21:33:43
Test Results

Build Type
Log
Result
Status

docs
logfile
1 cases, 0 failed, t: 2s
PASS :white_check_mark:

clang.debug.unity
logfile
1003 cases, 0 failed, t: 245s
PASS :white_check_mark:

msvc.debug
logfile
999 cases, 0 failed, t: 553s
PASS :white_check_mark:

coverage
logfile
1003 cases, 0 failed, t: 599s
PASS :white_check_mark:

clang.debug.unity, PARALLEL_TESTS=false
logfile
1003 cases, 0 failed, t: 470s
PASS :white_check_mark:

clang.debug.nounity
logfile
1001 cases, 0 failed, t: 229s
PASS :white_check_mark:

gcc.debug.unity
logfile
1003 cases, 0 failed, t: 236s
PASS :white_check_mark:

gcc.debug.nounity
logfile
1001 cases, 0 failed, t: 257s
PASS :white_check_mark:

clang.release.unity
logfile
1002 cases, 0 failed, t: 322s
PASS :white_check_mark:

gcc.release.unity
logfile
1002 cases, 0 failed, t: 344s
PASS :white_check_mark:

gcc.debug.unity -Dstatic=true
logfile
1003 cases, 0 failed, t: 237s
PASS :white_check_mark:

msvc.debug, PROJECT_NAME=rippled_classic
logfile
999 cases, 0 failed, t: 1146s
PASS :white_check_mark:

msvc.release
logfile
998 cases, 0 failed, t: 495s
PASS :white_check_mark:

Codecov Report

Merging #2435 into develop will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop    #2435   +/-   ##
========================================
  Coverage    70.37%   70.37%
========================================
  Files          701      701
  Lines        53437    53437
========================================
  Hits         37605    37605
  Misses       15832    15832

Impacted Files
Coverage Δ

src/ripple/protocol/impl/STVar.cpp
85.71% <0%> (-2.6%)
:arrow_down:

src/ripple/app/misc/SHAMapStoreImp.cpp
75.78% <0%> (-0.25%)
:arrow_down:

src/ripple/basics/DecayingSample.h
86.11% <0%> (+8.33%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9af994c...225a484. Read the comment docs.

@scottschurr yes, that linker warning is unfortunate...seems to be a "feature" of CMake's xcode generator:  https://stackoverflow.com/a/34436314/1362. Using Boost IMPORTED targets in our cmake might help, but I think that should only be done with a modernizing of the entire cmake build

@scottschurr on second look, there is a link_directories call that is really not needed (except for windows)...so I've if'd around it. If you can give this is try, I'd appreciate it. I think that quiets the linker warning.