External Data Sinks for OSQuery Results and Status messages

I am about to spend to much time on planes and as PoC I am planning on adding osquery results logging directly into fleet.  This would allow the results to be processed downstream and outside of fleet.
Normally I would just reach for kafka and move on, but after looking at the code base I think NATs https://github.com/nats-io/gnatsd might be a slightly better option.  Here is why:

NATs  installation is easier
using NATs in place of Redis for web sockets is much simpler
code is much simpler

With that being said ignoring kafka is crazy, but for a PoC I think simple is better.
Sent with GitHawk

#1899 was created as Proof of Concept of this.  I open the PR well before it's done to gather feedback early :)

I have a much better understand of how Fleet works, and how the code is put together after doing my POC in #1899.  With this I would like to proposal the following as a method for integration of external data sinks for osquery results/status.
Proposal
Rather then adding more server configuration that needs to be completed on hosts running fleet.  It would be simpler to add the routing of data to sinks using apiVersioned yaml specification.
This would keep the server configuration very simple while also allowing a simple path users that upgrade.  Here is an example of what the config would look like.
apiVersion: v1
kind: output
spec:
  name: windows_kafka_queue
  descriptions: All windows hosts request pro
  type: kafka
  kafka:
     topic: fleet.windows.results
     brokers: <IP_ADDRESS>:9018
  match:
    - type: results
    - platform: windows
---
apiVersion: v1
kind: output
spec:
  name: unix_nats_queue
  descriptions: All
  type: kafka
  kafka:
     topic: fleet.unix.{.PlatForm}
     brokers: <IP_ADDRESS>:9018
  match:
    - type: results
    - platform: !windows
---
apiVersion: v1
kind: output
spec:
  name: local_results
  descriptions: All
  type: file
  file:
     path: /tmp/fleet_results
     log_rotate: false
  match:
    - type: results
---
apiVersion: v1
kind: output
spec:
  name: local_status
  descriptions: All
  type: file
  file:
     path: /tmp/fleet_status
     log_rotate: false
  match:
    - type: status

I think we should go back and forth on the file format here, but I love the idea of defining logging pipeline details with the file format! Great idea, @jrossi.

It's not clear to me that Fleet should be in the business of sophisticated routing to different output streams. @jrossi can you give some examples of other projects that do sophisticated routing like this (vs. simple output and handling complex routing in the logging pipeline)?

@zwass sure.  The one the comes mind is different parts of the same org consuming things in different ways.  Security teams want to know the results and status messages for knowing the state of the world.  While desktop support needs the data in splunk, but the desktop engineering wants to push the data into sql server to build global list of all listening ports. I know of 5-6 different engineering teams in my company that would consume just a sub-set of the data.
While I understand the question is seams best to do the routing where all the state information is stored.  Otherwise we have to expand the results with the host metadata from fleets database into the results and send them on. With the host metadata added  effective filtering can happen outside of fleet. While doable does not seam like the best choice, but I could be missing some or a better way.
Here is my companies use case: Currently I am also working on building pull request model for our internal teams to define scheldued queries and have them role out in groups of hosts using fleetctl and labels.  With routing defined as I proposed I would also be let them consume the data they wanted in a manor they wanted.
In the future if all  this makes sense is how to effective route results results name of schleuded queries so results can be filtered even more.  Think something like the following, but I have not gotten that far into this yet.
‘’’
match:
query-name-startwith: desktop-suppory
Or
match:
query-name:
- secops.*
- ir.*
‘’’

Here is what I would think would be a sane way forward:

Refactor fleets osquery output into a module like system, will required refactor of file output as a module that is enable by default to match current fleet usage
Work with team on the apiSpec for output

routing / matching
connection strings
all the other things

Output services I would plan on adding:

kafka
nats
http
redis streaming (Maybe)

Given this is a rather large number of changes and dependent on each other how would you like me to move forward?

Hi, is any work ongoing with this?
If not, I'd like to implement a very simple Google PubSub publisher (no routing/filtering logic).

Hi @therealmik. I just pushed support for logging to AWS Firehose in #2022. There is no work ongoing to enabling routing/matching for output plugins. You could totally follow the patterns introduced in that PR to add support for GCP pubsub.

That looks great - it should be pretty easy to write with all the hard work done already :)

FYI, the code is written, I'm just doing some testing before creating the PR

Hi @zwass - the filesystem module is changed from previous releases - it no longer writes newlines between messages in the results log.
On this line:
https://github.com/kolide/fleet/pull/2022/files#diff-b016735b4714c1a8b7942556a87e4b68R205
The append must be allocating new buffers almost always, so the append is lost.  I'm not really sure if the newline is needed for firehose (it isn't for pubsub) - if not, maybe move it into the filesystem logger anyway?

@therealmik Great find there. Another user reported the symptoms of that problem as well. That was a mistake on my part. I just put up https://github.com/kolide/fleet/pull/2029 to fix this. The newline is needed in both filesystem and Firehose. These new changes ought to work appropriately with pubsub.

Awesome, that patch looks good to me.

I am going to close this as we now have both AWS Firehose and GCP Pubsub as logging output plugins, and the pattern is established for adding new ones (see https://github.com/kolide/fleet/pull/2022 and https://github.com/kolide/fleet/pull/2049).


THEEDGE-2533

Description
This PR adds a folder structure depending on Images, Repos, Systems, etc.
Also new /repositories/CustomRepositories.cy.js is added, to test main repo table.
Type of change
What is it?

[ ] Bug fix (non-breaking change which fixes an issue)
[X] New feature (non-breaking change which adds functionality)
[ ] Documentation update
[X] Tests update

Checklist:

[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] I run npm run lint:js:fix to check that my code is properly formatted

Codecov Report
Base: 31.07% // Head: 31.07% // No change to project coverage :thumbsup:

Coverage data is based on head (1c34658) compared to base (2917152).
 Consider uploading reports for the commit 28b28ed to get more accurate results

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #531   +/-   ##
=======================================
  Coverage   31.07%   31.07%
=======================================
  Files         124      124
  Lines        2616     2616
  Branches      798      798
=======================================
  Hits          813      813
  Misses       1654     1654
  Partials      149      149

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Parameter attribute with data annotations

IMHO it would be great to allow the blazor components to enforce parameter contracts using the existing data annotations; eg.
[Required,Parameter,StringLength(80)]
string Name { get; set; }

this would avoid a lot of boilerplate code with the components and/or replication of feature that exist in data annotation libraries today.

I think it is a bad idea to mix between the two. The Parameter attribute fulfills a different functionality than that of the data annotations. And the Parameter attribute is specific to Blazor.

IMHO I believe the Blazor Parameter attribute represents the component view state and with that in mind data annotations are in fact the correct attributes to use with Blazor parameters/data bindings to enforce parameter/model validation on both client and server using common/shared logic by way of the existing validations.
@enetstudio yes, the Parameter attribute is specific to Blazor; again IMHO there is nothing stopping Blazor from understanding the existing DataAnnonations and applying/generating checks accordingly.  I'm not suggesting the Blazor ParameterAttribute be moved/changed.

Similar ideas (required parameters) are already planned here https://github.com/aspnet/Blazor/issues/1#issuecomment-363034967
Full validation (not related to parameters) is discussed here #17.

Validation is being tracked in #17.
FYI the syntax [Parameter, Required] means the same thing as [Parameter][Required] so will work, assuming there ends up being a validation system that works with DataAnnotations attributes.


Spike: How can we make the donut chart for the staff dashbord index page?

We want to display some charts on the Staff dashboard index to provide quick glance information. Currently in our Figma we have a donut chart, essentially a pie chart with the center cut out.
This ticket is to investigate the options we have to implement this in our Rails app. No code required.
Off the top of my head, I am thinking we could use Chartkick gem to creat the pie chart, then use some custom CSS to overlay a white circle to create the donut look. But, perhaps there are other options?
This image is from the Figma file > Dashboard section

It sounded like in the meeting we aren't considering this design anymore. Is that correct and we can close this @kasugaijin ?

Yes I think we can go with a simpler design approach for now @mononoken


add a simple workflow

A skeleton of a future CI. It does two jobs:

run test
run lint

If you have any requirements, feel free to ask.
I choose node 18.14 as a default one.

Thank you for the PR!


Require creation metadata during auto release

Fixes #1817
In the following circumstances:

A registry experiences a fault where it fails to enumerate all
tags when asked without signalling an error condition
The tag for an image in an automated workload is not returned
by the registry API
Older image tags are returned

Flux would roll back the image to an older version, because it
couldn't tell the current image is newer. Instead, Flux now refuses
to automate unless it has timestamp information for the current and
new version, as this assures the made comparison is valid.

Nice -- less code, and better safeguard. Can we have a test case, please?


fix: Default trame visualization in Jupyter environments

The default visualization breaks the usage of the plotter in Jupyter environments. When the documentation implementation is ready from PyVista side we should revisit how to generate the interactive plotters, since from now on, we will need "html" for documentation and ideally "trame" for regular Jupyter usage.
This arises from #897

Doc build failing - have a look at it. I haven't read the logs yet.

Why this?
os.environ["PYDEVD_DISABLE_FILE_VALIDATION"] = "1"

Documentation building is getting frozen, and it throws this warning:
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.

I though I might give it a try

Why this?
os.environ["PYDEVD_DISABLE_FILE_VALIDATION"] = "1"

Documentation building is getting frozen, and it throws this warning:
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.

I though I might give it a try

Not remotely... I can't see that log on the workflows.

Documentation looks good now. This should be good to go @RobPasMue


API: image inspection doesn't return tags

I use the Docker API, and when I run this request, the tags doesn't return.
GET /images/{id}/json

This information is given when I list all images.
What's the best practices to get image's tags information ?

try this
http://serverip:port/v1.16/images/json?filter=ubuntu

This solution works, but not sure that response give only one image.
Future, not interesting to add a similar feature ?

#dibs

Hi VictorDenisov,
Please remember to add a closes or fixes keyword once youve submitted your PR to help tracking fixes easier. Thanks!

BTW, I found that in addition to "tags", there are two more fields that are missing from GET /images/{id}/json response. They are "RepoDigests" and "Labels". Would there be an interest to add them as well?


Hyperlink for Sorting companies based on internship date in content page is broken

https://github.com/AY2324S1-CS2103T-T10-4/tp/pull/219


Machine RancherOS Support

@ibuildthecloud  @sheng-liang

@ibuildthecloud  @sheng-liang -- ACPID daemon does not receive events inside a container
The issue has something to do with the mount namespaces, you can see it running though.
I've updated the code.

I propose that we do not containerize acpid for now. It is really not required to get rancheros ready for machine.
We can revisit this, once we have a working machine build.
@ibuildthecloud  @sheng-liang

Also note, this PR requires OS base with acpid support


Repo example code returns nil for highlight method

I'm trying to use the example code provided in the repo. For some reason that is not clear to me, the following line will return nil (and in the example case, crash because it's force unwrapped)
let highlights = try client.highlights(in: NSRange(0..<24), provider: provider, mode: .required)!

I had a look at the code but couldn't find an immediate cause for it. I'm using the main branch to test this, as it's the one used in the example code.
I also tried using the sample project. There, the TextViewHighlighter example seems to work as expected. The other method provided (setUpTreeSitter) has the same issue though.
Am I doing anything wrong here? Or should I stick with 0.6.0 for now?

Thank you for reporting this. Probably a bug - investigating now.

Ok so this is a bug. Even though synchronous access with the .required fill mode is specified, BackgroundingLanguageLayerTree.accessTreeSynchronously is returning false, disallowing the synchronous access.
So far, this is a valid situation. You've asked for synchronous access, and that access is impossible. But, it should not be in this situation. I think this is because the pendingWork calculation is too simplistic.

Thank you for looking into it! I can confirm that it does work if I use the asynchronous methods. Does this mean I should hold off of using the synchronous methods for now? Or is there a case where they work as expected?

Ok, I think this is fixed now. It took a long time because I added a test, but that test didn't work because a whole bunch of stuff went wrong with accessing Swift tree-sitter highlights. https://github.com/alex-pinkus/tree-sitter-swift/issues/289

Yep, all working now. Thank you for the quick fix!


[multistage] Initial (phase 1) Query runtime for  window functions - empty OVER() and OVER(PARTITION BY)

This PR introduces the multi-stage runtime changes to support Phase 1 of Window Functions, specifically targeting empty OVER() and OVER(PARTITION BY). Runtime support has been added for the following classes of window queries:

Empty OVER() - single and multiple empty OVER()s in the same query
OVER(PARTITION BY) - single and multiple OVER(PARTITION BY) using the same PARTITION BY key in the same query

This PR also fixes some bugs in the window function planning, specifically:

Create Empty OVER() and OVER(ORDER BY) as singleton stages which need a single node assignment (similar to global aggregation without group by and global sort)
~Add some logic for OVER(PARTITION BY key1 ORDER BY key1) queries which get marked as OVER(PARTITION BY key1) to ensure that the ORDER BY key1's direction and null direction are the defaults, otherwise treat these as OVER(PARTITION BY key1 ORDER BY key2) type queries during planning.~

The window functions supported as part of Phase 1 are: SUM, AVG, MIN, MAX, COUNT, BOOL_OR, and BOOL_AND.
PostgreSQL supports many aggregation functions inside window functions as mentioned in their documentation:

In addition to these functions, any built-in or user-defined ordinary aggregate (i.e., not ordered-set or hypothetical-set aggregates) can be used as a window function; see Section 9.21 for a list of the built-in aggregates. Aggregate functions act as window functions only when an OVER clause follows the call; otherwise they act as plain aggregates and return a single row for the entire set.

This PR also adds a new JSON file for window function query runtime tests via the output mechanism (H2 doesn't support window functions and got parse errors when trying to use it).
Design document: https://docs.google.com/document/d/13CmFm4djI09JKF_Xty5acoXxJoxC9CLXAsmikgzgtIs/edit?usp=sharing
OSS issue: https://github.com/apache/pinot/issues/7213
Window Function Planner PR: https://github.com/apache/pinot/pull/10228
This PR does not include support for:

Execution engine changes for Phase 1 for OVER(ORDER BY) and OVER (PARTITION BY ORDER BY) (will be the next PR for this feature)
Custom frames
Other window functions related to rank and values
Multiple window groups (basically multiple OVER clauses with different PARTITION BY, ORDER BY and/or FRAME specifications, if these specifications are the same they get grouped into a single window group)

The above will be part of future changes for window function support
cc @siddharthteotia @walterddr @vvivekiyer @Jackie-Jiang

Codecov Report

Merging #10286 (976df16) into master (addb652) will decrease coverage by 56.68%.
The diff coverage is 0.00%.

@@              Coverage Diff              @@
##             master   #10286       +/-   ##
=============================================
- Coverage     70.39%   13.71%   -56.68%
+ Complexity     5777      182     -5595
=============================================
  Files          2015     1962       -53
  Lines        109318   107020     -2298
  Branches      16615    16358      -257
=============================================
- Hits          76954    14678    -62276
- Misses        26961    91186    +64225
+ Partials       5403     1156     -4247

Flag
Coverage Δ

integration1
?

integration2
?

unittests1
?

unittests2
13.71% <0.00%> (-0.02%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

.../org/apache/pinot/query/planner/StageMetadata.java
0.00% <0.00%> (-93.34%)
:arrow_down:

...g/apache/pinot/query/planner/stage/WindowNode.java
0.00% <0.00%> (-55.11%)
:arrow_down:

...inot/query/runtime/operator/AggregateOperator.java
0.00% <0.00%> (-92.47%)
:arrow_down:

...uery/runtime/operator/WindowAggregateOperator.java
0.00% <0.00%> (ø)

.../pinot/query/runtime/plan/PhysicalPlanVisitor.java
0.00% <0.00%> (-92.31%)
:arrow_down:

...src/main/java/org/apache/pinot/sql/FilterKind.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ain/java/org/apache/pinot/core/data/table/Key.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...in/java/org/apache/pinot/spi/utils/BytesUtils.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ain/java/org/apache/pinot/common/CustomObject.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...n/java/org/apache/pinot/core/data/table/Table.java
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 1589 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Will be reviewing this today

lgtm with some more comments. thank you @somandal for the contribution. this is a huge effort and we definitely appreciate the patience and consideration during the review/comment process.

Thanks @walterddr! Appreciate the thorough review and the guidance on this from all involved.
I've addressed all comments. PTAL when you get a chance


polymer.dart 0.10.0-pre.9 triggers a warning in platform.js

it's caused by use_native_dartium_shadowdom.js
  if (flags.shadow && document.querySelectorAll('script').length > 1) {
    console.warn('platform.js is not the first script on the page. ' +
        'See http://www.polymer-project.org/docs/start/platform.html#setup ' +
        'for details.');
  }
the warning is harmless, but misleading. Not sure the best solution, as we must run this code before platform.js. It will be obsolete soon though, if it isn't already. It's worth checking if it's still needed. Otherwise we can wait for Dartium 35 to remove.

This issue has been moved to dart-lang/polymer-dart#168.


Move Alist.Alist 3.27.0 to AList.AList 3.27.0

Microsoft Reviewers: Open in CodeFlow

@wingetbot run


hterm fork inside zeit org?

As we are always monkey patching hterm behavior.
It would be a good option to have a fork and update it's default behavior.

I think this will make sense once we actually can't fix some of its problems with monkey-patching. For now it's ok


Can websockets do video call...

Is it possible to do video conferencing like zoom application.

I dont think so...
Better use WebRTC


discovered unknown device

I get this error:
Please raise an issue in the GitHub repository (https://github.com/lprhodes/homebridge-broadlink-rm/issues) with details of the type of device and its device type code: "4ead" so that we can handle it correctly and prevent this message from appearing.
I have a Broadlink WiFi power socket in my network

Thanks for reporting - can you link me to the product you purchased please? I want to make sure I make note of all of the right details.

It’s an RM Pro bought about 2 years ago from Banggood

It has Firmware V30 and underneath is printed „e-remote RM Pro“

4ead appears to be for the Hysen Heating Controller.
I'll have the plugin ignore it in the next update

Yes, I have 2 Floor Heating Controller in my WiFi. Named Fluoreon.
So I have to wait for the update to use the plugin for my RM Pro?

No - the error is just to make sure you report it here

I see. But the plugin does not work for me.
I don’t get any devices in the home app
Von meinem iPhone gesendet

Am 27.02.2018 um 23:19 schrieb Luke Rhodes<EMAIL_ADDRESS>No - the error is just to make sure you report it here
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

v2.6.11 includes a fix for the error log you're seeing.
As for it not working, please create a separate issue.


Application Insights internal exception

Hi,
we're using Serilog and the Application Insights Sink in an Azure Functions project. We're using the code you have in the Readme about adapting the Telemetry's Context to include a UserId, operation_Id and operation_parentId. In our case, right now we only have the operation_id, but that's not important for this issue, I believe.
We're getting the operation_Id from a header the upstream service sends. This is the code we use to create the logger:
public static ILogger CreateLogger(IHeaderDictionary reqHeaders)
        {
            var instrumentationKey = Environment.GetEnvironmentVariable("APPINSIGHTS_INSTRUMENTATIONKEY");
            var correlationId =
                reqHeaders.ContainsKey(traceIdHeaderName) ? reqHeaders[traceIdHeaderName].ToString() : Guid.NewGuid().ToString();

            return new LoggerConfiguration()
                .Enrich.WithProperty("operation_Id", correlationId)
                .WriteTo.ApplicationInsights(instrumentationKey, ConvertLogEventsToCustomTraceTelemetry)
                .CreateLogger();
        }

When we deploy this and call the Azure Function, we get some internal exceptions from AI that we don't know where they come from:

Are we configuring Serilog in an incorrect way? If we replace
.WriteTo.ApplicationInsights(instrumentationKey, ConvertLogEventsToCustomTraceTelemetry)

with
.WriteTo.ApplicationInsightsTraces(instrumentationKey)

we don't get those exceptions (but we don't have the desired OperationId, of course).
Thanks,

Hi @vgaltes I'm sorry, I can't reproduce this. Exception seems to come from AI itself. I've tried simple console app test and all seems good.

Thanks for your quick response @aloneguid . It's an Azure function. We're calling this at the beginning of the function execution and logging afterwards. It fails before logging anything:
public static ILogger CreateLogger(IHeaderDictionary reqHeaders)
        {
            var instrumentationKey = Environment.GetEnvironmentVariable("APPINSIGHTS_INSTRUMENTATIONKEY");
            var correlationId =
                reqHeaders.ContainsKey("x-company-traceId") ? reqHeaders["x-company-traceId"].ToString() : Guid.NewGuid().ToString();

            return new LoggerConfiguration()
                .Enrich.WithProperty("operation_Id", correlationId)
                // .WriteTo.ApplicationInsightsTraces(instrumentationKey)
                .WriteTo.ApplicationInsights(instrumentationKey, ConvertLogEventsToCustomTraceTelemetry)
                .CreateLogger();
        }

        private static ITelemetry ConvertLogEventsToCustomTraceTelemetry(LogEvent logEvent, IFormatProvider formatProvider)
        {
            // first create a default TraceTelemetry using the sink's default logic
            // .. but without the log level, and (rendered) message (template) included in the Properties
            var telemetry = GetTelemetry(logEvent, formatProvider);

            // then go ahead and post-process the telemetry's context to contain the user id as desired
            if (logEvent.Properties.ContainsKey("UserId"))
            {
                telemetry.Context.User.Id = logEvent.Properties["UserId"].ToString();
            }
            // post-process the telemetry's context to contain the operation id
            if (logEvent.Properties.ContainsKey("operation_Id"))
            {
                telemetry.Context.Operation.Id = logEvent.Properties["operation_Id"].ToString();
            }
            // post-process the telemetry's context to contain the operation parent id
            if (logEvent.Properties.ContainsKey("operation_parentId"))
            {
                telemetry.Context.Operation.ParentId = logEvent.Properties["operation_parentId"].ToString();
            }
            // typecast to ISupportProperties so you can manipulate the properties as desired
            ISupportProperties propTelematry = (ISupportProperties)telemetry;

            // find redundent properties
            var removeProps = new[] { "UserId", "operation_parentId", "operation_Id" };
            removeProps = removeProps.Where(prop => propTelematry.Properties.ContainsKey(prop)).ToArray();

            foreach (var prop in removeProps)
            {
                // remove redundent properties
                propTelematry.Properties.Remove(prop);
            }
            return telemetry;
        }

        private static ITelemetry GetTelemetry(LogEvent logEvent, IFormatProvider formatProvider)
        {
            if (logEvent.Exception != null) {
                // Exception telemetry
                return logEvent.ToDefaultExceptionTelemetry(
                    formatProvider,
                    includeLogLevelAsProperty: false,
                    includeRenderedMessageAsProperty: false,
                    includeMessageTemplateAsProperty: false);
            }
            else {
                // default telemetry
                return logEvent.ToDefaultTraceTelemetry(
                    formatProvider,
                    includeLogLevelAsProperty: false,
                    includeRenderedMessageAsProperty: false,
                    includeMessageTemplateAsProperty: false);
            }
        }

I'll try to prepare an example today.

I did a bit of investigation. Let me explain you the steps. First, I stumbled into this Stack Overflow post. I took a look at the code and you're initializing the client as the persons that answers say that it doesn't work.
So, I changed the way I initialize the logger and, instead of passing the instrumentation key, I passed a TelemetryConfiguration object. That made a lot of the exceptions to disappear, but instead, I got a couple of them saying that the EventSource already existed. Doing some googling, I end up here. It looks like that AzureFunctions is using AppInsights version 2.7.2 and using other versions cause this problem.
So, my third test was, still using the TelemetryConfiguration version, downgrade the version of AppInsights to 2.7.2.
After doing that, everything looks good on AI.

Happy it worked for you, thanks for extensive explanation. Please go ahead with the PR!

@vgaltes I've updated the docs: https://github.com/serilog/serilog-sinks-applicationinsights#using-with-azure-functions, this approach seems to work for me i.e. serilog messages will be correlated with built-in reporting and visible in azure portal.


version number 0.4.6.dev0

Codecov Report

Merging #872 into master will increase coverage by 1.25%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #872      +/-   ##
==========================================
+ Coverage   77.07%   78.32%   +1.25%
==========================================
  Files          52       52
  Lines        5509     5509
==========================================
+ Hits         4246     4315      +69
+ Misses       1263     1194      -69

Impacted Files
Coverage Δ

setup.py
0% <ø> (ø)
:arrow_up:

vent/api/plugins.py
83.07% <0%> (-0.2%)
:arrow_down:

vent/api/menu_helpers.py
93.11% <0%> (+0.4%)
:arrow_up:

vent/menus/main.py
79.21% <0%> (+1.31%)
:arrow_up:

vent/api/actions.py
78.29% <0%> (+1.89%)
:arrow_up:

vent/core/network_tap/ncontrol/rest/nlist.py
100% <0%> (+33.33%)
:arrow_up:

vent/menus/ntap.py
57.6% <0%> (+33.6%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7d174e7...e6d1308. Read the comment docs.


Add a DOI for citation?

Last issue for a bit (most likely). I am probably going to use some calculations performed using this library in a manuscript. Would you be interested in adding a DOI to the repo so it can easily be cited/credited?
I usually use zenodo for this (for github repo's): https://guides.github.com/activities/citable-code/

Done! Here's a link to the official DOI. I've updated the README to reflect this, and added some citation instructions. Please do cite the DOI for the wrapper library, but also be sure to cite either Toon and Ackerman (1981) or Bohren and Huffman (1983) depending on which code you used!
Closed by 9e16c61


Add postgis ci container

Adds a container running postgres 14 with postgis 3.2 (by default) to the CI agents. This is necessary to run the tests for the Mongo->Postgresql conversion for imminence. If we decide to go ahead with that, it'll be necessary to merge this into the main branch.
Related PR:
https://github.com/alphagov/imminence/pull/766

This might not be necessary? I guess things are working in the CI now (both Github and Jenkins), so maybe we don't need to care. Going to close.


Downcasting classes in python

We have a good set of python bindings, where we are able to create classes in python and use them as they are. However, there is one problem with the current bindings, where we are not able to downcast from an Abstract class to the class instance that is being developed.
For example, when creating a class IntegratedActionModelEuler in python, the calc(self, data, x, u) needs access to the object differential in IntegratedActionDataEuler. However, data object is of type ActionDataAbstract, and thus doesn't contain any element of such name. In c++, we get around it by statically casting data to its expected type, and using this type to obtain our element. However, in python, such runtime polymorphism is not yet implemented.
The solution for this would be https://www.boost.org/doc/libs/1_55_0/libs/smart_ptr/pointer_cast.html, which would enable us to downcast pointers in python, in a visitor fashion for pre-defined types.

In c++, I have enable the automatic downcast of Python bings. You could observe this but typing:
actionModel.state # retrieve the derived state implemented in c++

However, it is right that we cannot derive data classes in Python. If you look the code, I didn't define createData as a virtual function for the Python bindings. So, what it happens is we create the base data for any model written in Python.
To be honest, I have tried to defined this function as virtual when I started the project, but I didn't manage to figure out a solution. The main reason is that data structures only define new data objects, not virtual functions.
In short, I don't think the this limitations is related with static casting of shared pointers. Perhaps, you might need to explain it more.

I think I agree with your remark, @cmastalli. Here, the problem is that createData is not virtual, and the re-implementation in the inheriting class is not called. Consequently, a ActionData is created instead of an InheritedActionData.

The main reason is that data structures only define new data objects, not virtual functions
I don't understand why this would prevent createData from being binded as virtual. Could you please explain?

Related to this issue I report a minimal case in which it can be seen that the ShootingProblem is not using the definition of the createData method in the definition of the abstract class.
While in python this is possible to do and set the differential ad an attribute of a custom defined Data class
import crocoddyl, example_robot_data

class IntegratedActionModelBase(crocoddyl.ActionModelAbstract):
    '''
        Same as integrated action model euler
    '''
    def __init__(self, diffModel, timeStep=1e-3, withCostResiduals=True):
            crocoddyl.ActionModelAbstract.__init__(self, crocoddyl.StateVector(diffModel.state.nx), diffModel.nu)
            self.differential = diffModel
            self.timeStep = timeStep
            self.withCostResiduals = withCostResiduals
            if self.timeStep == 0:
                self.enable_integration_ = False
            else:
                self.enable_integration_ = True

    def createData(self):
        '''
            Not called by the Shooting problem
        '''
        data = IntegratedActionDataBase(self)
        print(data, 'But correctly called by python')
        return data

    def calc(self, data, x, u=None):
        print('...')
        return 0, 0

    def calcDiff(self, data, x, u=None):
        print('...')

class IntegratedActionDataBase(crocoddyl.ActionDataAbstract):
    '''
        Ideally would create a data class with differential alternative to the IAMEuler, not working with C++ bindings
    '''
    def __init__(self, am):
        crocoddyl.ActionDataAbstract.__init__(self, am)
        self.differential = am.differential.createData()

robot_model = example_robot_data.loadTalosArm().model
state = crocoddyl.StateMultibody(robot_model)
runningCostModel = crocoddyl.CostModelSum(state)
actuation = crocoddyl.ActuationModelFull(state)
differential=crocoddyl.DifferentialActionModelFreeFwdDynamics(state, actuation, runningCostModel)
IAMBase=IntegratedActionModelBase(differential)
problem=crocoddyl.ShootingProblem(state.rand(), [IAMBase], IAMBase)
myData=IAMBase.createData()
assert hasattr(myData, 'differential'), 'No differential here'
assert hasattr(problem.runningDatas[0], 'differential'), 'No differential here'

@gfadini it seems I found the solution. I am working on it, it requires to do work on each single thing: action, diff-action, cost, contacts, impulses, actuation and activation

solved in #808


Password Reset Endpoint

An endpoint (perhaps /api/v1/users/password_reset?email=<taget email> is needed to request a password reset link for a specific Drupal user account by email address (or perhaps UID).
The password reset endpoint will be used by the mbc-user-importscript. A password reset link is needed to send as part of the message content in the transactional email message for newly created Drupal users.
An example being the daily niche.com user import.
Related:
https://trello.com/c/Qsywpzzo
https://github.com/DoSomething/mbc-user-import/blob/master/MBC_userImport.class.inc#L128
https://github.com/DoSomething/mbc-user-import/issues/27
https://github.com/DoSomething/mb-toolbox/issues/29

There is a Drupal Services password reset to look into before we build anything custom... although I believe it just sends an email (does not return the URL)

Looking at a blog post, it seems to be triggered by the following:
$mail = _user_mail_notify('password_reset', $account, $language); -
http://www.grasmash.com/article/create-passwordreset-resource-services
Looking for alternatives

Found it!
https://api.drupal.org/api/drupal/modules!user!user.module/function/user_pass_reset_url/7

This endpoint will need an access check for administrator roles only. If the links were public, the endpoint could be used to login to any account.

[ ] Create a new targeted action: users/[uid]/password_reset_url which returns the output of user_pass_reset_url for the given uid.  Add this code into the member_resource.inc in the dosomething_api/resources.

To make the endpoint accessible, enable the new password_reset_url resource and click Save. You'll need to clear cache to get it working and test it.  If all looks good, commit the code!

[ ] drush fu dosomething_api will modify the dosomething_api.services.inc file to enable the endpoint. Commit that code too.


KAFKA-4024: Fix metadata update to not apply backoff until it experiences failure

Issue: https://issues.apache.org/jira/browse/KAFKA-4024
Fixes a bug that inappropriately applies backoff as interval between metadata updates even though the current one is outdated.

Think I would follow up more unit tests.

@ijuma Can you review this please?

Thanks for the PR, LGTM. The documentation for retry.backoff.ms says that it only applies to failed requests:
"The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios."
@hachikuji, can you double-check that this change is fine?

One more thing, do we want to make the first Metadata.update call consistent between KafkaProducer and KafkaConsumer?
The producer does:
this.metadata.update(Cluster.bootstrap(addresses), time.milliseconds());

The consumer does:
this.metadata.update(Cluster.bootstrap(addresses), 0);

One more thing, do we want to make the first Metadata.update call consistent between KafkaProducer and KafkaConsumer?

Yeah actually this is how this PR first look like since I thought this is the problem that happens only on the first send.
After this fix it is no longer harmful but let's fix it just to keep usages consistent.

@kawamuray There are a bunch of cases in the clients where we can refetch metadata in a tight loop (e.g. if a partition leader is not available). The metadata fetches are successful, but we are waiting for a condition inside the response to become true. Previously the backoff protected us in this case, but we seem to be losing that, right? Is this not a concern?

@hachikuji That's a good point. If we want to continue using it in those cases, we probably need to update the config documentation.

@ijuma Yeah, agreed. Seems like the problem is that the definition of "failed request" might not be too clear in the documentation. I think what we really mean is any metadata fetch containing errors, which covers topics not existing, partition leaders not available, etc. These cases are considered "successful" updates from the perspective of the Metadata object, but not from that of the producer/consumer. I think there are two problems with the current implementation:

We treat the bootstrapped cluster as a regular metadata update and enforce the backoff, which causes the weird delay on startup. This should be easy to fix by providing a separate path for bootstrapping the metadata.
The current implementation takes the interpretation of the backoff setting to the extreme and prevents any repeat metadata fetches within each backoff duration. Perhaps we can loosen this and override the backoff interval if the set of topics being fetched changes? In fact, if we did this, then the first problem would be solved as a matter of course.

Thoughts?

@hachikuji Thanks for your comment.
"if a partition leader is not available" means a leader broker went down but metadata isn't updated yet so it still points dead broker a a leader for topic-partition right?
Indeed in that case a client can continually send metadata update request in each time it fails to produce to the dead leader, but the interval between each metadata update request can still controllable by reconnect.backoff.ms IIUC.
When producer tries to send next message which will be routed to dead broker, it tries to re-initiate connection to dead broker, fail, and reconnect.backoff.ms applied.
Out of curiosity, but how is it harmful if clients continually send metadata request during broker failover?

@hachikuji

but we bypass the backoff if the set of topics has changed since the last refresh.

Hmm, I think this approach still doesn't work for following case?

Successful metadata update.
Successful metadata update again which is triggerred by metadata expiration. Highly likely the fetched metadata havn't changed since the last time, so refreshBackoffMs isn't overridden.
BrokerA failed immediately after 2.
Client wants to send metadata update request immediately but blocked by refreshBackoffMs before it allowed to do.

@kawamuray I think a longer delay to respond to failures is basically the price you have to pay to avoid overwhelming the servers. It takes a little time on the server side to detect the failure and migrate partitions anyway, so I don't think immediately sending a metadata update after the failure would have much value, though I could be wrong. I'd prefer for now to leave the current backoff behavior in tact except when we know something in the request has changed. Perhaps it would make sense to consider in a follow-up issue whether something like exponential backoff would be a more ideal way to address this problem in practice.

@hachikuji

It takes a little time on the server side to detect the failure and migrate partitions anyway

I think this isn't true for some cases. For example, when we perform rolling restart of brokers, each broker will shutdown gracefully so the controller can detect shutdown broker much faster than waiting zk session timeout.

so I don't think immediately sending a metadata update after the failure would have much value

However I agree with this. At least in case of broker failure, the first immediate metadata request likely results unchanged metadata, which results producer to wait retry.backoff.ms at least to recover anyways.
Actually I'm bit confusing now. The reason I lengthened retry.backoff.ms was to prevent a producer being send retry Produce requests in tight loop and exhaust retry attempts w/o waiting sufficient time for broker recovery, but seems the actual behavior is different from what I expected.
When I shortened retry.backoff.ms, on each Sender#run it tries to re-initiate connection to dead leader, so the NetworkClient#ready doesn't return true for unavailable topic-partition then retry count won't decreased in that attempt. Even on next loop, the Selector already knows that the dead leader has already disconnected so the same thing happens, so producer won't exhaust retry attempts during this loop. Also the WARN logs such as "Got error produce response with correlation id ..." won't flood in logs.
Then the first case(the first producer.send() blocked inappropriately) is the only problem as long as we keep retry.backoff.ms to reasonably smaller value?
In case, is there any reason we still better follow your idea(do not apply backoff when the last result changed)? Or is the single line of change - metadata.update(..., 0) - enough to fix this?
Anyway, I strongly agree with that it's much better to have exponential backoff or just an idea but something like "listen to metadata update" mechanism which sounds like not that hard to implement as we already have purgatory implementation in server side, in order to minimize MTTR. WDYT?

@hachikuji
ugh, I think I didn't understand your idea properly.

Perhaps we can loosen this and override the backoff interval if the set of topics being fetched changes

I read through your comments again and realized that "set of topics" is topics field of Metadata you meant. Then your idea would prevents this:
producer.send(new ProducerRecord("topic-A", ...));
producer.send(new ProducerRecord("topic-B", ...)); // Maybe blocked refreshBackoffMs

Of course it prevents the first problem as well.
Sounds good to try. Let me update this PR to apply your suggestion.

@kawamuray Yeah, sounds good, sorry if my suggestion wasn't clear at first (I've been looking at this code a little too long so I just assume everyone else has as well). Also good to know that we won't exhaust the retry limit for produce requests when dealing with partition leader failover.

@hachikuji Updated PR based on your suggestion.
9c24e2d is a commit which implements your idea but let me explain about this new patch a bit as I had to make an unexpected modification 64ec2ec to accomplish the purpose.
I modified the test code like this: https://gist.github.com/kawamuray/fdfcfec85f179417c3a2d4866f055df5 and ran against 9c24e2d then the result was look like below:
./bin/kafka-run-class.sh -Dlog4j.configuration=file:./log4j.properties -Dretry.backoff.ms=5000 KafkaProducerMetadataUpdateDurationTest
...
[2016-09-13 16:07:53,676] INFO Kafka version : <IP_ADDRESS>-SNAPSHOT (org.apache.kafka.common.utils.AppInfoParser)
[2016-09-13 16:07:53,676] INFO Kafka commitId : 9c24e2d4025e5170 (org.apache.kafka.common.utils.AppInfoParser)
[2016-09-13 16:07:53,679] DEBUG Initialize connection to node -1 for sending metadata request (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:07:53,679] DEBUG Initiating connection to node -1 at HOST-1:9022. (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:07:53,731] DEBUG Initialize connection to node -2 for sending metadata request (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:07:53,732] DEBUG Initiating connection to node -2 at HOST-2:9022. (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:07:53,732] DEBUG Initialize connection to node -3 for sending metadata request (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:07:53,732] DEBUG Initiating connection to node -3 at HOST-3:9022. (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:07:53,737] DEBUG Completed connection to node -1 (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:07:53,738] DEBUG Completed connection to node -2 (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:07:53,738] DEBUG Completed connection to node -3 (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:07:58,751] DEBUG Sending metadata request {topics=[test]} to node -3 (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:07:58,786] DEBUG Updated cluster metadata version 2 to Cluster(nodes = [HOST-3:9022 (id: 3 rack: null), HOST-1:9022 (id: 1 rack: null), HOST-2:9022 (id: 2 rack: null)], partitions = [Partition(topic = test, partition = 1, leader = 1, replicas = [1,2,3,], isr = [2,3,1,]), Partition(topic = test, partition = 0, leader = 3, replicas = [1,2,3,], isr = [3,2,1,]), Partition(topic = test, partition = 2, leader = 2, replicas = [1,2,3,], isr = [3,2,1,])]) (org.apache.kafka.clients.Metadata)
[2016-09-13 16:07:58,799] DEBUG Initiating connection to node 1 at HOST-1:9022. (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:07:58,799] DEBUG Initialize connection to node 3 for sending metadata request (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:07:58,799] DEBUG Initiating connection to node 3 at HOST-3:9022. (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:07:58,799] DEBUG Initiating connection to node 2 at HOST-2:9022. (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:07:58,801] DEBUG Completed connection to node 1 (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:07:58,803] DEBUG Completed connection to node 3 (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:07:58,803] DEBUG Completed connection to node 2 (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:08:03,801] DEBUG Sending metadata request {topics=[test,kawamuray]} to node 1 (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:08:03,807] DEBUG Updated cluster metadata version 3 to Cluster(nodes = [HOST-2:9022 (id: 2 rack: null), HOST-3:9022 (id: 3 rack: null), HOST-1:9022 (id: 1 rack: null)], partitions = [Partition(topic = test, partition = 1, leader = 1, replicas = [1,2,3,], isr = [2,3,1,]), Partition(topic = kawamuray, partition = 1, leader = 2, replicas = [1,2,3,], isr = [3,2,1,]), Partition(topic = test, partition = 0, leader = 3, replicas = [1,2,3,], isr = [3,2,1,]), Partition(topic = kawamuray, partition = 0, leader = 1, replicas = [1,2,3,], isr = [2,3,1,]), Partition(topic = test, partition = 2, leader = 2, replicas = [1,2,3,], isr = [3,2,1,]), Partition(topic = kawamuray, partition = 2, leader = 3, replicas = [1,2,3,], isr = [3,2,1,])]) (org.apache.kafka.clients.Metadata)
tSendTest1 = 5121, tSendTest2 = 0, tSendKawamuray = 5008
[2016-09-13 16:08:03,807] INFO Closing the Kafka producer with timeoutMillis =<PHONE_NUMBER>854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer)

This reuslt has 2 problems:

It still blocks producer.send()
NetworkClient initiates connection to all brokers before sending the first metadata request.

I digged into DefaultMetadataUpdater and found that timeToNextReconnectAttempt is causing this.
The cause of problem 1 is that DefaultMetadataUpdater updates lastNoNodeAvailableMs even when a connection is being established to a broker, which leads retry.backoff.ms interval among each maybeUpdate().
The cause of problem 2 is that DefaultMetadataUpdater is not waiting a connection being established to a broker in the previous loop. By changing maybeUpdate() to wait a connection being established properly, maybeUpdate() is no longar riskly to call in tight loop so we can let each loop of maybeUpdate() to try checking connection availability for sending metadata request.
After appling 64ec2ec, the experiment result look like this:
./bin/kafka-run-class.sh -Dlog4j.configuration=file:./log4j.properties -Dretry.backoff.ms=5000 KafkaProducerMetadataUpdateDurationTest
...
[2016-09-13 16:25:18,526] INFO Kafka version : <IP_ADDRESS>-SNAPSHOT (org.apache.kafka.common.utils.AppInfoParser)
[2016-09-13 16:25:18,526] INFO Kafka commitId : 8cd55d2a465e3b0a (org.apache.kafka.common.utils.AppInfoParser)
[2016-09-13 16:25:18,528] DEBUG Initialize connection to node -1 for sending metadata request (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:25:18,528] DEBUG Initiating connection to node -1 at HOST-1:9022. (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:25:18,593] DEBUG Completed connection to node -1 (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:25:18,612] DEBUG Sending metadata request {topics=[test]} to node -1 (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:25:18,644] DEBUG Updated cluster metadata version 2 to Cluster(nodes = [HOST-1:9022 (id: 1 rack: null), HOST-2:9022 (id: 2 rack: null), HOST-3:9022 (id: 3 rack: null)], partitions = [Partition(topic = test, partition = 1, leader = 1, replicas = [1,2,3,], isr = [2,3,1,]), Partition(topic = test, partition = 0, leader = 3, replicas = [1,2,3,], isr = [3,2,1,]), Partition(topic = test, partition = 2, leader = 2, replicas = [1,2,3,], isr = [3,2,1,])]) (org.apache.kafka.clients.Metadata)
[2016-09-13 16:25:18,657] DEBUG Initiating connection to node 1 at HOST-1:9022. (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:25:18,658] DEBUG Initiating connection to node 3 at HOST-3:9022. (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:25:18,658] DEBUG Initiating connection to node 2 at HOST-2:9022. (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:25:18,661] DEBUG Completed connection to node 1 (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:25:18,662] DEBUG Sending metadata request {topics=[test,kawamuray]} to node 1 (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:25:18,662] DEBUG Completed connection to node 3 (org.apache.kafka.clients.NetworkClient)
[2016-09-13 16:25:18,667] DEBUG Updated cluster metadata version 3 to Cluster(nodes = [HOST-3:9022 (id: 3 rack: null), HOST-2:9022 (id: 2 rack: null), HOST-1:9022 (id: 1 rack: null)], partitions = [Partition(topic = test, partition = 1, leader = 1, replicas = [1,2,3,], isr = [2,3,1,]), Partition(topic = kawamuray, partition = 1, leader = 2, replicas = [1,2,3,], isr = [3,2,1,]), Partition(topic = test, partition = 0, leader = 3, replicas = [1,2,3,], isr = [3,2,1,]), Partition(topic = kawamuray, partition = 0, leader = 1, replicas = [1,2,3,], isr = [2,3,1,]), Partition(topic = test, partition = 2, leader = 2, replicas = [1,2,3,], isr = [3,2,1,]), Partition(topic = kawamuray, partition = 2, leader = 3, replicas = [1,2,3,], isr = [3,2,1,])]) (org.apache.kafka.clients.Metadata)
[2016-09-13 16:25:18,667] DEBUG Completed connection to node 2 (org.apache.kafka.clients.NetworkClient)
tSendTest1 = 130, tSendTest2 = 0, tSendKawamuray = 9
[2016-09-13 16:25:18,667] INFO Closing the Kafka producer with timeoutMillis =<PHONE_NUMBER>854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer)

@hachikuji ping.

@kawamuray Thanks for the reminder. I'll look at this in the morning.

@hachikuji thanks for review. Updated. PTAL again.

@hachikuji  let me ping you again...

@kawamuray Sorry, I'll probably have to get to this after the release (code freeze is on Monday). Unfortunately, I think it's a little too late to get this in for 0.10.1. We've had so many small problems around metadata fetch behavior that we should make sure there is plenty of time for validation.

Yep, I agree with that we better spend much time for validating this change.
I know you're doing really busy as the manager of next release. I always appreciate your help!

@kawamuray The recent changes are looking good. I'll take one more look in the morning, but seems like this is nearly there.

LGTM. Thanks for the patch! I'm going to run some system tests prior to merging to make sure we didn't miss anything obvious, but hopefully we can wrap this up shortly.

Quick update, there were some system test failures: http://testing.confluent.io/confluent-kafka-branch-builder-system-test-results/?prefix=2016-10-31--001.1477945677--hachikuji--KAFKA-4024--944f9f8/. I see a lot of messages like this from the consumer:
[2016-10-31 19:58:09,501] TRACE Skipping fetch for partition test_topic-0 because there is an in-flight request to worker3:9092 (id: 2 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)

Haven't had time yet to dig into it to see if it related to any changes in this patch.

Doh! super good catch @hachikuji .
I ran the system test locally and it reproduced exactly.
Investigated and found that we made another mistake WRT .requestUpdate() 😢
I turned on TRACE logging of VerifiableProducer and found that it stops requesting update of metadata after it received following response from the broker, which is incomplete as the leader of partition 0 is missing, then it never keep producing accumulated records even though broker failovered successfully:
[2016-11-02 07:48:12,455] DEBUG Updated cluster metadata version 68 to Cluster(id = hbf5ulUERN642ViwSrvdxQ, nodes = [worker1:9092 (id: 2 rack: null)], partitions = [Partition(topic = test_topic, partition = 0, leader = none, replicas = [1,2,], isr = [])]) (org.apache.kafka.clients.Metadata)

This is because we made this change: https://github.com/apache/kafka/pull/1707/commits/fdb0984a308001041892e3347ba1919d27ce9e6f#diff-499d1af91cfaf42966a8e2e485a1b4b9L180
Metadata.add() triggers requestUpdate() whenever the added partition is missing in the current list but in this case, the topic-partition was already added previously so no one will call requestUpdate().

Reverted requestUpdate() for unknown leader partitions 48f598f and the system tests turns to passing.
So in the end  	fdb0984 couldn't reduce even single line from production code :(

@kawamuray Good find and thanks for investigating.

Merged to trunk. Thanks again for the very delicate patch. We'll have to keep an eye on the nightly builds to make sure we didn't miss anything. 😅

We had a long run :D Thanks so much for your insightful review @hachikuji!

Oh, forgot to mention. Prior to merging, I added a little extra explanation to one of the comments in the code. Hopefully it's reasonable.


docker builds appear to be failing

I'm trying to rehome my laptop install of gonorth to a server so I can use it less tethered to one machine but am running into issues. I've tried cloning over the gonorth install from my laptop and rebuilding with that i end up with the below but am allowed to login (all data appears to be missing however):
sudo docker-compose up
Creating mongodb ... done
Creating gonorthlaptop_web_1 ... done
Attaching to mongodb, gonorthlaptop_web_1
web_1      | warn: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]
web_1      |       No XML encryptor configured. Key {04d4c90c-d6dc-4b55-bfa0-6c40efb0afff} may be persisted to storage in unencrypted form.
web_1      | Hosting environment: Production
web_1      | Content root path: /app
web_1      | Now listening on: http://[::]:80
web_1      | Application started. Press Ctrl+C to shut down.
web_1      | fail: Microsoft.AspNetCore.Antiforgery.Internal.DefaultAntiforgery[7]
web_1      |       An exception was thrown while deserializing the token.
web_1      | System.InvalidOperationException: The antiforgery token could not be decrypted. ---> System.Security.Cryptography.CryptographicException: The key {bdcb965b-99ac-4e9b-836e-b81d840227bc} was not found in the key ring.
web_1      |    at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, Boolean allowOperationsOnRevokedKeys, UnprotectStatus& status)
web_1      |    at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.DangerousUnprotect(Byte[] protectedData, Boolean ignoreRevocationErrors, Boolean& requiresMigration, Boolean& wasRevoked)
web_1      |    at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Unprotect(Byte[] protectedData)
web_1      |    at Microsoft.AspNetCore.Antiforgery.Internal.DefaultAntiforgeryTokenSerializer.Deserialize(String serializedToken)
web_1      |    --- End of inner exception stack trace ---
web_1      |    at Microsoft.AspNetCore.Antiforgery.Internal.DefaultAntiforgeryTokenSerializer.Deserialize(String serializedToken)
web_1      |    at Microsoft.AspNetCore.Antiforgery.Internal.DefaultAntiforgery.GetCookieTokenDoesNotThrow(HttpContext httpContext)

If I try to start from scratch and just copy over the dbdata i get a much longer series of errors:
GoNorth]$ sudo docker-compose up
Creating mongodb ... done
Creating gonorth_web_1 ... done
Attaching to mongodb, gonorth_web_1
web_1      | warn: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]
web_1      |       No XML encryptor configured. Key {6a1b47d2-7348-42b8-9689-2fb2b1e720c9} may be persisted to storage in unencrypted form.
web_1      | Hosting environment: Production
web_1      | Content root path: /app
web_1      | Now listening on: http://[::]:80
web_1      | Application started. Press Ctrl+C to shut down.
web_1      | fail: Microsoft.AspNetCore.Antiforgery.Internal.DefaultAntiforgery[7]
web_1      |       An exception was thrown while deserializing the token.
web_1      | System.InvalidOperationException: The antiforgery token could not be decrypted. ---> System.Security.Cryptography.CryptographicException: The key {04d4c90c-d6dc-4b55-bfa0-6c40efb0afff} was not found in the key ring.
web_1      |    at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, Boolean allowOperationsOnRevokedKeys, UnprotectStatus& status)
web_1      |    at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.DangerousUnprotect(Byte[] protectedData, Boolean ignoreRevocationErrors, Boolean& requiresMigration, Boolean& wasRevoked)
web_1      |    at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Unprotect(Byte[] protectedData)
web_1      |    at Microsoft.AspNetCore.Antiforgery.Internal.DefaultAntiforgeryTokenSerializer.Deserialize(String serializedToken)
web_1      |    --- End of inner exception stack trace ---
web_1      |    at Microsoft.AspNetCore.Antiforgery.Internal.DefaultAntiforgeryTokenSerializer.Deserialize(String serializedToken)
web_1      |    at Microsoft.AspNetCore.Antiforgery.Internal.DefaultAntiforgery.GetCookieTokenDoesNotThrow(HttpContext httpContext)
web_1      | fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[0]
web_1      |       An unhandled exception has occurred: A timeout occured after 30000ms selecting a server using CompositeServerSelector{ Selectors = MongoDB.Driver.MongoClient+AreSessionsSupportedServerSelector, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 } }. Client view of cluster state is { ClusterId : "1", ConnectionMode : "Automatic", Type : "Unknown", State : "Disconnected", Servers : [{ ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/mongo:27017" }", EndPoint: "Unspecified/mongo:27017", State: "Disconnected", Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: No such device or address
web_1      |          at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6)
web_1      |          at System.Net.Dns.ResolveCallback(Object context)
web_1      |       --- End of stack trace from previous location where exception was thrown ---
web_1      |          at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |          at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult)
web_1      |          at System.Net.Dns.EndGetHostAddresses(IAsyncResult asyncResult)
web_1      |          at System.Net.Dns.<>c.<GetHostAddressesAsync>b__25_1(IAsyncResult asyncResult)
web_1      |          at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
web_1      |       --- End of stack trace from previous location where exception was thrown ---
web_1      |          at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |          at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |          at MongoDB.Driver.Core.Connections.TcpStreamFactory.<ResolveEndPointsAsync>d__10.MoveNext()
web_1      |       --- End of stack trace from previous location where exception was thrown ---
web_1      |          at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |          at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |          at MongoDB.Driver.Core.Connections.TcpStreamFactory.<CreateStreamAsync>d__4.MoveNext()
web_1      |       --- End of stack trace from previous location where exception was thrown ---
web_1      |          at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |          at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |          at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__48.MoveNext()
web_1      |          --- End of inner exception stack trace ---
web_1      |          at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__48.MoveNext()
web_1      |       --- End of stack trace from previous location where exception was thrown ---
web_1      |          at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |          at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |          at MongoDB.Driver.Core.Servers.ServerMonitor.<HeartbeatAsync>d__27.MoveNext()" }] }.
web_1      | System.TimeoutException: A timeout occured after 30000ms selecting a server using CompositeServerSelector{ Selectors = MongoDB.Driver.MongoClient+AreSessionsSupportedServerSelector, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 } }. Client view of cluster state is { ClusterId : "1", ConnectionMode : "Automatic", Type : "Unknown", State : "Disconnected", Servers : [{ ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/mongo:27017" }", EndPoint: "Unspecified/mongo:27017", State: "Disconnected", Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: No such device or address
web_1      |    at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6)
web_1      |    at System.Net.Dns.ResolveCallback(Object context)
web_1      | --- End of stack trace from previous location where exception was thrown ---
web_1      |    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |    at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult)
web_1      |    at System.Net.Dns.EndGetHostAddresses(IAsyncResult asyncResult)
web_1      |    at System.Net.Dns.<>c.<GetHostAddressesAsync>b__25_1(IAsyncResult asyncResult)
web_1      |    at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
web_1      | --- End of stack trace from previous location where exception was thrown ---
web_1      |    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |    at MongoDB.Driver.Core.Connections.TcpStreamFactory.<ResolveEndPointsAsync>d__10.MoveNext()
web_1      | --- End of stack trace from previous location where exception was thrown ---
web_1      |    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |    at MongoDB.Driver.Core.Connections.TcpStreamFactory.<CreateStreamAsync>d__4.MoveNext()
web_1      | --- End of stack trace from previous location where exception was thrown ---
web_1      |    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |    at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__48.MoveNext()
web_1      |    --- End of inner exception stack trace ---
web_1      |    at MongoDB.Driver.Core.Connections.BinaryConnection.<OpenHelperAsync>d__48.MoveNext()
web_1      | --- End of stack trace from previous location where exception was thrown ---
web_1      |    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |    at MongoDB.Driver.Core.Servers.ServerMonitor.<HeartbeatAsync>d__27.MoveNext()" }] }.
web_1      |    at MongoDB.Driver.Core.Clusters.Cluster.ThrowTimeoutException(IServerSelector selector, ClusterDescription description)
web_1      |    at MongoDB.Driver.Core.Clusters.Cluster.WaitForDescriptionChangedHelper.HandleCompletedTask(Task completedTask)
web_1      |    at MongoDB.Driver.Core.Clusters.Cluster.<WaitForDescriptionChangedAsync>d__50.MoveNext()
web_1      | --- End of stack trace from previous location where exception was thrown ---
web_1      |    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |    at MongoDB.Driver.Core.Clusters.Cluster.<SelectServerAsync>d__43.MoveNext()
web_1      | --- End of stack trace from previous location where exception was thrown ---
web_1      |    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |    at MongoDB.Driver.MongoClient.<AreSessionsSupportedAfterSeverSelctionAsync>d__37.MoveNext()
web_1      | --- End of stack trace from previous location where exception was thrown ---
web_1      |    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |    at MongoDB.Driver.MongoClient.<AreSessionsSupportedAsync>d__34.MoveNext()
web_1      | --- End of stack trace from previous location where exception was thrown ---
web_1      |    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |    at MongoDB.Driver.MongoClient.<StartImplicitSessionAsync>d__26.MoveNext()
web_1      | --- End of stack trace from previous location where exception was thrown ---
web_1      |    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |    at MongoDB.Driver.MongoCollectionImpl`1.<UsingImplicitSessionAsync>d__91`1.MoveNext()
web_1      | --- End of stack trace from previous location where exception was thrown ---
web_1      |    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |    at MongoDB.Driver.IAsyncCursorSourceExtensions.<FirstOrDefaultAsync>d__5`1.MoveNext()
web_1      | --- End of stack trace from previous location where exception was thrown ---
web_1      |    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |    at GoNorth.Data.User.UserMongoDbAccess.<GetUserByNormalizedUserName>d__6.MoveNext() in /app/Data/User/UserMongoDbAccess.cs:line 80
web_1      | --- End of stack trace from previous location where exception was thrown ---
web_1      |    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |    at GoNorth.Authentication.GoNorthUserStore.<FindByNameAsync>d__7.MoveNext() in /app/Authentication/GoNorthUserStore.cs:line 96
web_1      | --- End of stack trace from previous location where exception was thrown ---
web_1      |    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |    at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
web_1      |    at Microsoft.AspNetCore.Identity.SignInManager`1.<PasswordSignInAsync>d__34.MoveNext()
web_1      | --- End of stack trace from previous location where exception was thrown ---
web_1      |    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |    at GoNorth.Controllers.AccountController.<Login>d__14.MoveNext() in /app/Controllers/AccountController.cs:line 132
web_1      | --- End of stack trace from previous location where exception was thrown ---
web_1      |    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |    at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
web_1      |    at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeActionMethodAsync>d__12.MoveNext()
web_1      | --- End of stack trace from previous location where exception was thrown ---
web_1      |    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |    at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextActionFilterAsync>d__10.MoveNext()
web_1      | --- End of stack trace from previous location where exception was thrown ---
web_1      |    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |    at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
web_1      |    at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
web_1      |    at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeInnerFilterAsync>d__14.MoveNext()
web_1      | --- End of stack trace from previous location where exception was thrown ---
web_1      |    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |    at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeNextResourceFilter>d__22.MoveNext()
web_1      | --- End of stack trace from previous location where exception was thrown ---
web_1      |    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |    at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
web_1      |    at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
web_1      |    at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeFilterPipelineAsync>d__17.MoveNext()
web_1      | --- End of stack trace from previous location where exception was thrown ---
web_1      |    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |    at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeAsync>d__15.MoveNext()
web_1      | --- End of stack trace from previous location where exception was thrown ---
web_1      |    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |    at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()
web_1      | --- End of stack trace from previous location where exception was thrown ---
web_1      |    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |    at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.<Invoke>d__6.MoveNext()
web_1      | --- End of stack trace from previous location where exception was thrown ---
web_1      |    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |    at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.<Invoke>d__4.MoveNext()
web_1      | --- End of stack trace from previous location where exception was thrown ---
web_1      |    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
web_1      |    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
web_1      |    at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>d__6.MoveNext()
^CGracefully stopping... (press Ctrl+C again to force)
Stopping gonorth_web_1 ... done
Stopping mongodb       ... done

Any thoughts on how to get this working?

The second, longer error message seems like GoNorth can not connect to the MongoDB.
I will try and reproduce the first error message as well and get back to you as soon as I know more regarding this problem.


expose board state

It would be really nice for users to be able to query the board state. Perhaps something like game.getPiece(x,y)?
I will open a PR.

Hey @lino-levan! Sorry for the slow reply - I need to double-check my notification rules...
I added a .getSpace(coord) method in #11 that should let you query the board state directly. It doesn't take 2 numeric parameters like you were asking, but instead the space's coordinates in standard algebraic form (eg: "b2") since that's a wee bit more consistent with the other APIs. Hope this helps!


added default header value

added DEFAULT_HEADERS to Client class and passing composed headers dict to GoTrueApi class.
What kind of change does this PR introduce?
Feature
What is the current behavior?
Default headers constant exists
What is the new behavior?
Default header constant is filled with version and passed into Client class
Related to Issue

Hey @doubledare704, thanks for your contribution!
We are closing this PR since it's based in the current main branch, which is going to be completely changed by #23.
Don't worry, your contribution has already been added to that PR so it makes it to the main branch as soon as we merge it.
Thanks!


TypeError: TextDecoder is not a constructor

Summary
Having the error TypeError: TextDecoder is not a constructor.
Stryker config
{
  "$schema": "https://raw.githubusercontent.com/stryker-mutator/stryker/master/packages/api/schema/stryker-core.json",
  "mutator": "typescript",
  "packageManager": "npm",
  "reporters": [
    "clear-text",
    "progress",
    "dots",
    "html"
  ],
  "testRunner": "jest",
  "transpilers": [
    "typescript"
  ],
  "coverageAnalysis": "off",
  "tsconfigFile": "tsconfig.json",
  "mutate": [
    "src/**/*.ts",
    "!src/**/*.spec.ts",
    "!src/**/__mocks__/*.spec.ts",
    "!src/**/__mocks__/"
  ],
  "allowConsoleColors": true,
  "dashboard": {
    "baseUrl": "https://dashboard.stryker-mutator.io/api/reports",
    "reportType": "full"
  },
  "fileLogLevel": "info",
  "logLevel": "info",
  "symlinkNodeModules": true,
  "tempDirName": "stryker-tmp",
  "timeoutFactor": 2,
  "timeoutMS": 5000,
  "plugins": [
    "@stryker-mutator/*"
  ]
}

Test runner config (jest.config.js)
// For a detailed explanation regarding each configuration property, visit:
// https://jestjs.io/docs/en/configuration.html

module.exports = {
  // All imported modules in your tests should be mocked automatically
  automock: false,

  // Stop running tests after `n` failures
  bail: false,

  // Respect "browser" field in package.json when resolving modules
  // browser: false,

  cache: true,

  // The directory where Jest should store its cached dependency information
  cacheDirectory: `./.cache-jest`,

  // Automatically clear mock calls and instances between every test
  clearMocks: true,

  // Indicates whether the coverage information should be collected while executing the test
  collectCoverage: false,

  // An array of glob patterns indicating a set of files for which coverage information should be collected
  // collectCoverageFrom: undefined,

  // The directory where Jest should output its coverage files
  coverageDirectory: `coverage`,

  // An array of regexp pattern strings used to skip coverage collection
  // coveragePathIgnorePatterns: [
  //   "\\\\node_modules\\\\"
  // ],

  // A list of reporter names that Jest uses when writing coverage reports
  coverageReporters: [`json`, `text-summary`, `lcov`],

  // An object that configures minimum threshold enforcement for coverage results
  // coverageThreshold: undefined,

  // A path to a custom dependency extractor
  // dependencyExtractor: undefined,

  displayName: {
    color: `magenta`,
    name: `Sonia`,
  },

  // Make calling deprecated APIs throw helpful error messages
  errorOnDeprecated: true,

  // Force coverage collection from ignored files using an array of glob patterns
  // forceCoverageMatch: [],

  // A path to a module which exports an async function that is triggered once before all test suites
  // globalSetup: undefined,

  // A path to a module which exports an async function that is triggered once after all test suites
  // globalTeardown: undefined,

  // A set of global variables that need to be available in all test environments
  globals: {
    "ts-jest": {
      compiler: `ttypescript`,
    },
  },

  // The maximum amount of workers used to run your tests. Can be specified as % or a number. E.g. maxWorkers: 10% will use 10% of your CPU amount + 1 as the maximum worker number. maxWorkers: 2 will use a maximum of 2 workers.
  // maxWorkers: "50%",

  maxConcurrency: 5,

  // An array of directory names to be searched recursively up from the requiring module's location
  moduleDirectories: [`./node_modules`],

  // A map from regular expressions to module names that allow to stub out resources with a single module
  // moduleNameMapper: {},

  // An array of regexp pattern strings, matched against all module paths before considered 'visible' to the module loader
  // modulePathIgnorePatterns: [],

  // An array of file extensions your modules use
  moduleFileExtensions: [`js`, `json`, `ts`, `node`],

  // Activates notifications for test results
  notify: true,

  // An enum that specifies notification mode. Requires { notify: true }
  notifyMode: `failure-change`,

  // Run tests from one or more projects
  // projects: undefined,

  // A preset that is used as a base for Jest's configuration
  preset: `ts-jest/presets/js-with-ts`,

  prettierPath: `prettier`,

  // Use this configuration option to add custom reporters to Jest
  reporters: [
    [
      `jest-silent-reporter`,
      {
        showPaths: true,
        showWarnings: true,
        useDots: true,
      },
    ],
  ],

  // Automatically reset mock state between every test
  resetMocks: true,

  // A path to a custom resolver
  // resolver: undefined,

  // Reset the module registry before running each individual test
  resetModules: false,

  // The root directory that Jest should scan for tests and modules within
  // rootDir: undefined,

  // Automatically restore mock state between every test
  restoreMocks: true,

  // Allows you to use a custom runner instead of Jest's default test runner
  // runner: "jest-runner",

  // A list of paths to directories that Jest should use to search for files in
  roots: [`./src`, `./scripts`],

  // The paths to modules that run some code to configure or set up the testing environment before each test
  setupFiles: [`./config.ts`],

  // A list of paths to snapshot serializer modules Jest should use for snapshot testing
  // snapshotSerializers: [],

  // Prevent tests from printing messages through the console
  silent: true,

  // A list of paths to modules that run some code to configure or set up the testing framework before each test
  // setupFilesAfterEnv: [`./__tests__/setup.js`],

  // The test environment that will be used for testing
  testEnvironment: `node`,

  // Options that will be passed to the testEnvironment
  // testEnvironmentOptions: {},

  // Adds a location field to test results
  // testLocationInResults: false,

  // The glob patterns Jest uses to detect test files
  // testMatch: [
  //   "**/__tests__/**/*.[jt]s?(x)",
  //   "**/?(*.)+(spec|test).[tj]s?(x)"
  // ],

  // An array of regexp pattern strings that are matched against all test paths, matched tests are skipped
  // testPathIgnorePatterns: [
  //   "\\\\node_modules\\\\"
  // ],

  // The regexp pattern or array of patterns that Jest uses to detect test files
  // testRegex: [],

  // This option allows the use of a custom results processor
  // testResultsProcessor: undefined,

  // This option allows use of a custom test runner
  // testRunner: "jasmine2",

  // This option sets the URL for the jsdom environment. It is reflected in properties such as location.href
  // testURL: "http://localhost",

  testTimeout: 5000,

  // Setting this value to "fake" allows the use of fake timers for functions such as "setTimeout"
  timers: `fake`,

  // An array of regexp pattern strings that are matched against all source file paths, matched files will skip transformation
  // transformIgnorePatterns: [
  //   "\\\\node_modules\\\\"
  // ],

  // An array of regexp pattern strings that are matched against all modules before the module loader will automatically return a mock for them
  // unmockedModulePathPatterns: undefined,

  // A map from regular expressions to paths to transformers
  transform: {
    ".ts": `ts-jest`,
  },

  // An array of regexp patterns that are matched against all source file paths before re-running tests in watch mode
  // watchPathIgnorePatterns: [],

  // Indicates whether each individual test should be reported during the run
  verbose: true,

  // Whether to use watchman for file crawling
  watchman: true,
};

Stryker environment
+-- "@stryker-mutator/core": "3.2.4"
+-- "@stryker-mutator/jest-runner": "3.2.4"
+-- "@stryker-mutator/typescript": "3.2.4"

+-- "jest": "26.0.1"
+-- "ts-jest": "26.0.0"
+-- "discord.js": "12.2.0"

Test runner environment
stryker run

Your Environment

software
version(s)

node
13.7.0

npm
6.14.4

Operating System
Windows 10 64bits

Add stryker.log
stryker.log
My repo issue:
https://github.com/Sonia-corporation/il-est-midi-discord/issues/459

Does someone can tell me at least if the error is fixable from Stryker or not?
I do not know after two months now if this is the right place for this issue.
Clearly Stryker and discord.js are not compatible but both can eventually add a fix.
Maybe the way discord.js export stuff is not yet handled by Stryker or on the other maybe that discord.js is fucked up at some point in the exports so both can fix this - eventually -.

Fixed. See https://github.com/Sonia-corporation/sonia-discord/issues/459

Fixed. See https://github.com/Sonia-corporation/sonia-discord/issues/459

Great to hear you managed to get Stryker working! Sorry for not responding 😟

Great to hear you managed to get Stryker working! Sorry for not responding 😟

@nicojs no problem I understand completely.
I think it should be hard to maintain and time consuming.
Plus it's open-source so no need to be sorry for this 😄

@nicojs no problem I understand completely.
I think it should be hard to maintain and time consuming.
Plus it's open-source so no need to be sorry for this 😄


Make cli more inclusive

Closes https://github.com/safe-global/safe-cli/issues/319

I have read the CLA Document and I hereby sign the CLA

coverage: 87.033% (-0.1%) from 87.136%
when pulling 8ecd24eed78daa92e3c943c40dceb05ff3b5086b on make_cli_more_inclusive
into 29be0a254631763e4c1ba79818507608eb4beafe on main.

coverage: 87.033% (-0.1%) from 87.136%
when pulling b062113a9095f9bcbd1144f06b8cee0e625b6232 on make_cli_more_inclusive
into 29be0a254631763e4c1ba79818507608eb4beafe on main.

coverage: 87.033% (-0.1%) from 87.136%
when pulling 95a5e076f8eeeee079c724072f146a1de9b7bc72 on make_cli_more_inclusive
into 29be0a254631763e4c1ba79818507608eb4beafe on main.