Addressing the Format challenges in PowerShell

Summary of the new feature/enhancement
Since version 1, the implementation of the formatting layer in PowerShell has included many challenging limitations, some of which include the following:

Format-* cmdlets output custom format data objects that are then rendered in the console once they are passed into Out-Default for processing. e.g. Get-Process -Id $PID | Format-Table | Get-Member.
Since these cmdlets convert PSObjects into format data objects, you cannot pipe their results to other commands to do something meaningful. e.g. Get-Process -Id $PID | Format-Table | Stop-Process -WhatIf
Other than Format-List, Format-* is not respected when you output heterogeneous types that are not compatible with one another. e.g. & {Get-Process -Id $PID; Get-Item C:\} | Format-Table.
To create a command that produces objects that render in a specific format by default, you must define the format using format ps1xml files, and you must ensure that the PSTypeNames of objects returned from your command have a PSTypeName that matches the specific format you want. This would require a much more complicated example involving ps1xml format files, so I have left that out for now.

I have attempted to eliminate many of these limitations in PowerShell 5.1 and earlier with my [FormatPx module](https://github.com/KirkMunro/FormatPx], and while that worked out quite well, I also discovered several issues with the FormatPx approach, namely:

Producing output formatted in a specific way from a command is easy with FormatPx, but capturing that output in such a way that it maintains the desired format when that output data may also be captured as reference and output via other means (e.g. $Error for errors) where you don't want the same format is a difficult problem to solve without some redesign.
Rendering formatted output when formatted output is not needed (e.g. when you are capturing output or piping a command to another command) is a processing expense that must be avoided until it is needed. Otherwise you may be spending CPU cycles formatting data, only to then have that format replaced with other formats.
When outputting some data with truly out of band data (e.g. errors, warnings, etc.) in the middle of the output, it is desirable to show that out of band data and then continue the output that you previously started rather than close off the output and then start output anew with headers, etc. being shown again.

Now I have some users who would like to use FormatPx in PowerShell (Core) on Linux, and even though that only requires some relatively minor changes, there are issues I would like to deal with and I believe it would be much more valuable for the community if I take the knowledge gained through my FormatPx work and apply it natively to PowerShell 7, since it is open source after all.
Proposed technical implementation details
The specific problems I would like to solve in PowerShell 7 are as follows:

Make it much easier for scripters to select the format that they want for data output from any command that they author.
By "much easier", I mean without having to unnecessarily muck around with the PSTypeNames array on non-custom objects that are output, and being able to easily select existing, named format views or identify new, command-specific formats that should be used as the default format for the objects output by the command, all while still just outputting objects that can be captured, output, used in expressions, or piped to other commands.

Fix output processing such that heterogeneous types can be output to Format-Table, Format-Wide, or Format-Custom (but that's hardly used) with PowerShell rendering that output in tabular, wide, or custom formats, respectively, without implicitly treating objects with different types as out of band objects.
This differs from today's behavior, in which the Formatting engine identifies the current "shape" to use when rendering formatted data, and any object whose type does not match the type(s) used in formatting data in the current shape are marked as out of band and shown as if they were output with Format-List if they are non-value, non-string types or shown as their string representation if they are value types or strings.
This is a breaking change, because commands that output multiple heterogeneous types would render differently if they are piped to Format-Table, Format-Wide, or Format-Custom, and scripts that capture that formatted output and do something with it may have issues; however, it is not very likely that there are many scripts out there that produce heterogeneous data types with different formats that then have their data captured in string format and processed. For this reason, I believe it is worth the break, because it corrects a long-standing issue in PowerShell and makes PowerShell Format-* cmdlets function in a WYSIWYG fashion, as they should.

To solve these problems, I would like to make the following changes:

Extend the OutputType attribute such that it includes a new Format property.
For example, consider this command:
function Get-ProcessByStartTime {
    [CmdletBinding()]
    [OutputType([System.Diagnostics.Process], Format='Table', FormatParameters=@{
        View = 'StartTime'
    })]
    Get-Process | Where-Object StartTime | Sort-Object StartTime
}

While that is a contrived example, it demonstrates how a command could be written to apply a specific, named view to the objects that it outputs.
Now consider this example, which would likely be a much more common approach to specifying custom formatting to use:
function Get-Something {
    [CmdletBinding()]
    [OutputType([System.ServiceProcess.ServiceController], Format='Table', FormatParameters=@{
        Property = 'Name','DisplayName'
        GroupBy = 'Status'
    })]
    Get-Service | Sort-Object -Property Status,Name
}

That example shows how you can define a default format for a command without having to create a custom format data entry: you simply specify the desired default format using the OutputType attribute.
Here's one more example to show how it could work with custom objects:
function Get-OperatingSystem {
    [CmdletBinding()]
    [OutputType('MyModule.OperatingSystem', Format='List')]
    $osData = Get-CimInstance -Class Win32_OperatingSystem
    [pscustomobject]@{
        PSTypeName = 'MyModule.OperatingSystem'
        Name = $osData.Caption
        Version = $osData.Version
        Architecture = $osData.OSArchitecture
    }
}

In all cases, what I envision would happen behind the scenes is that the returned objects would be wrapped in a lightweight PSFormatObject type that is derived from PSObject. This type would capture the desired format as specified in the OutputType attribute, but it wouldn't actually render the data in that format -- rendering would be deferred until Out-Default, allowing the command output to be piped to other commands like normal and saving the time required to render the data until it is needed. If you pipe the output from these commands to any Format-* command, the data would immediately be rendered in the desired format. If you show the data output from these commands without using a Format-* command, the data would render using the format information that was attached to the PSFormatObject objects.
Note that commands that return multiple object types could define the desired format for each object type.

Fix the heterogeneous output type problem.
To solve this problem, the formatting engine would be updated such that a new "shape" doesn't result in out of band processing. Instead, if the object was not a scalar nor a string (and perhaps other simple types -- I would need to double-check the code to ensure I'm covering the proper types), and if the object was output on the standard output stream, it would result in the closing of the current group/shape, and a new format would be started with the changed, incompatible, heterogeneous type. Non-standard output data as well as scalars and strings would still result in out of band data rendering.

Alternate proposals and considerations
Alternate proposal: Change how Format-* cmdlets work, such that formatting is always deferred until Out-Default
Instead of extending OutputType, we could update Format-* cmdlets to do the same thing (create a PSFormatObject that contains a shared format object reference for each object processed). This would allow piping beyond Format-*, and Format-* cmdlets would be used to define the format you want when the data is output rather than to convert the data to the desired format output.
On the plus side, this would allow scripters to just use Format-* where they want, even without dealing with custom type names.
There are several downsides to just changing Format-* behavior, including the following:

It would result in functions that return format data in downlevel versions, and object data in current + future versions, which would be confusing.
It would require users to use the same Format-* invocation anywhere that objects are returned from a function or script, which is more difficult than defining the returned format as part of OutputType.
It is less declarative than an OutputType format, and less discoverable (OutputType has the added benefit of being easily parsed and discovered via programmatic inspection).

For these reasons, I would stay away from this approach, and stick with the proposal of extending OutputType.
Alternate proposal: use something like @lzybkr's PSMore module
@lzybkr started a PSMore module a few years ago to address some of PowerShell's formatting issues. That module is a side project that has not seen much movement.
Regardless of how that module proceeds going forward, I believe the proposal documented herein to be worth the investment because it is additive, and it solves some significant formatting issues without changing how formatting is defined in configuration and without changing how formatting is rendered in PowerShell today. I also don't believe that these changes get in the way of what could be done in the PSMore module going forward.
Consideration: Define a Format-Default cmdlet
With support for more easily customizable formatting without breaking the pipeline-ability of commands, I would also advocate for defining a Format-Default cmdlet. This cmdlet would simply replace any PSFormatObject objects that were passed into it with their PSObject counterparts that do not have format data associated with them, allowing objects to have custom formatting stripped from them.
Consideration: Deferring output may result in named formats being unavailable if they are defined in modules that are unloaded
Consider this scenario:

You have a module with a command that returns data configured to use a specific named format.
The named format is loaded as part of the same module.
You invoke that command and capture the results in a variable.
You then unload that module.
You then pass your captured data to Out-Default.

By the time the data reaches Out-Default, the desired format that was specified in the command will not be available in memory anymore because it will have unloaded with the module.
In this case, the data would fallback to the default output for that data, as if it did not have associated PSFormatObject information to work with.
Consideration: Use a new System.Management.Automation.OutputAttribute instead of extending OutputType
The name OutputType indicates that the attribute is used to define the output type. Instead of extending that to include formatting information as well, it may be better to define a new Output attribute (System.Management.Automation.OutputAttribute) that supercedes OutputType, and use that attribute to define the type of objects that are output, the format used to render the objects that are output, which parameter sets return those type/format details, and leave the door open for additional future properties that could be added to the Output attribute (e.g. Contract=$true to indicate that output is contractual in a function, which would result in errors if an object that does not match an output type was returned from that function).
Creating a new attribute in place of extending the OutputType attribute has two benefits:

Downlevel users will not be confused when they see OutputType used with different syntax and get errors. Instead they will see a new attribute that they haven't used before. They'll still get errors, but at least those errors won't be related to an attribute that is supported/documented in the version they are using.
Since this makes output attribution about much more than a type, having a new attribute for that with a different name may bring more clarity to what is being done. The old attribute would still be supported, so backwards compatibility would persist, but the Parser should probably raise an error if both attributes are used in the same command (see additional comments about Parser improvements when it comes to attributes below).

Consideration: New attributes and attribute extensions would result in runtime errors in downlevel versions
It is worth pointing out that it doesn't matter if you extend an attribute with additional properties or if you define a new attribute -- both result in runtime errors, not parse errors. I'm not surprised for the latter since attributes can be defined in modules, but I'm a little surprised for the former. I feel it would be better if attribute extensions via additional properties resulted in parse errors, but there may be a valid reason that they don't that I am not aware of. At any rate, I call this out as a consideration because scripts written with attribute changes will result in runtime errors, not parse errors, in downlevel versions of PowerShell.
I think it is also worth updating the PowerShell parser to return parse errors if attributes are used that do not exist in a version of PowerShell, or if properties within those attributes that are used do not exist in a version of PowerShell. That is something that can be determined at parse time, and having a parser error would help guide users towards finding a version of PowerShell where those attributes and their properties are defined/supported. This work should be done as a separate PR (and a separate issue will be logged to track this need if we have consensus on this point).

We are tracking PSMore here #7857. I believe it could be preferred way because it is not only enhance formatting capabilities but also for native command wrappers (encoding to/from, parsing output, ...), completers and so on. (Jason noted that it can not be addition.)

I think PSMore is a longer term project that will take much longer to bring into use. Of course a complete revamp of the format system is desirable; however, I prefer to solve this problem now than make the community wait for a new formatting system to be implemented.
The changes proposed here are easier to implement now without mucking around with the complexities of the XML format system -- the format definitions themselves, and how they are rendered into the text we see in the console all stays the way it is now. These changes would be implemented natively in PowerShell from the start, and I don't think they get in the way of what PSMore is trying to accomplish, especially if the way they work is kept internal.
As for downlevel support, If necessary, I believe, but would need to confirm, the changes proposed here could be supported downlevel via an update to FormatPx. I strongly prefer not having FormatPx be the solution for current+future PowerShell though, because it needs to proxy certain core cmdlets for it to work, which means it can only be imported with -Force, and current solutions shouldn't have to take the extra dependency when this solution can just be implemented in-place in PowerShell today.

what I envision would happen behind the scenes is that the returned objects would be wrapped in a lightweight PSFormatObject type that is derived from PSObject

When I was mulling over PSMore a bit, I was considering adding new PSObject property for the purpose.

I was considering adding new PSObject property for the purpose

A new type (PSFormatObject) is a better design. Using that approach, it allows automatically stored objects (i.e. errors stored in $error) to be stored without formatting information, as they must be.

PSObject already by design is proxy/wrapper object. In particular, there is InstanceMembers that is ETS although one could do new type for ETS is derived from PSObject.

I know, it's not about that. This is about the need to separate formatted data from stored data. It's something I learned while building FormatPx.

bump, this is still a problem.


Setting up GAM is to hard!

Installing GAM and creating the client secrets, service account and authorizing is to complicated. Let's make it simpler. Everything we can automate should be automated. Everything that requires manual user work should be well documented and we should confirm they did it right.

Happy to help out - are there some APIs or libraries we can use to make
this easier? Seems a bit of a chicken and egg problem.

@daethnir It's definitely a chicken/egg issue but there's a few things we can do. See my latest commit:
https://github.com/jay0lee/GAM/commit/968c096a99f78ed02080e204297f6aef78eae59d
Google now has APIs that can:

Create an API project
Create a service account and private key for the project.
Enable APIs for a project (still need to figure out accepting ToS for some projects)

In terms of pure project setup. That leaves the admin with:

Create Client ID / Secret for the client_secrets.json file.
Enable Domain-Wide-Delegation for the service account.

The current rough draft patch automates the first three and walks user through the other two which should hopefully make life easier. It needs a lot of cleanup.
The embedded client id/secret is my project and only has the three used APIs enabled. API docs are at:
Resource Manager (create project):
https://cloud.google.com/resource-manager/reference/rest/v1/projects
Service Management (turn on services like Drive, Admin SDK, Calendar, etc):
https://cloud.google.com/service-management/reference/rest/v1/services/enable
IAM API (create service account and private key):
https://cloud.google.com/iam/reference/rest/
Jay

Besides cleaning "gam create project" up I'd also like to do a few more things:

jump right into admin authentication (create oauth2.txt authorization). Use hint= query parameter to make sure user is authorizing with email address they entered into GAM.
Walk user through admin console authorization of the service account.
Test and confirm service account is authorized for those scopes. Make API calls to confirm and explain what user needs to do if authorization is failing.

Just checked in a gam-install.sh script for MacOS and Linux. Goal would be that new GAM user can run:
bash <(curl -s -S -L https://raw.githubusercontent.com/jay0lee/GAM/master/src/gam-install.sh)
and GAM is downloaded, installed and setup kicked off.


Fix in the Bitmap documentation

Fixes a small error in the documentation (probably the result of a copy and paste): the documentation of Bitmap was the same as TrueType, while it should refer to the bitmap function instead of true_type.

Ah, thanks!


Don't trace OPTIONS and HEAD requests by default

We decided that OPTIONS requests shouldn't be traced by default since they eat into quota a lot for some users and don't add much value.
First implementation should not start transaction for OPTIONS and HEAD requests in Django, Flask and FastAPI integrations.
See https://github.com/getsentry/sentry-javascript/pull/5485
Also related to https://github.com/getsentry/sentry-ruby/issues/1864

Done, closing.


support for GCE-style disk spec in runtime block

My current WDL, which targets Google JES execution via Cromwell, will often contain something like this in the runtime block of a task:
    disks: "local-disk 375 LOCAL, /mnt/db 375 LOCAL"

On GCE, this attaches ephemeral local scratch disk, using one for the main drive, and another attached at a special mount point. GCE allows up to eight local disks per instance, and the Google backend for Cromwell ignores the disk size parameter if LOCAL is specified (since they're always 375GB on GCE).
dxWDL complains about this line in my runtime block, even if I manually specify dx_instance_type as well:
java.lang.Exception: Can't parse disk space specification local-disk 375 LOCAL, /mnt/db 375 LOCAL

On DNAnexus default AWS instances, the disks are always local (e.g. c3 instances) and you tend to gang them together in a RAID or something and present them as a single drive. I'm wondering if it would be possible for dxWDL to add the following abilities:

Successfully parse the above GCE-targeted disks specification without error (multiple disks, different mount points).
Use the sum of all requested disk sizes to decide on a standard dx_instance_type that presents a single drive with sufficient total storage.
Instead of presenting them as individually mounted file systems (as GCE does), keep it simple and just mkdir or symlink the requested mount points so that the code can at least write data to the path specified.

That would make it a bit easier to write portable WDL between the two execution environments. In these particular tasks, I'm unable to simplify the disks spec to a single drive and still execute on Cromwell+Google, because these tasks are much slower on Google's persistent network-attached drives, and a single Google local scratch disk is not big enough for the task.

@dpark01 This is an interesting, and new, portability problem. I think I can do what you are asking, although I want to think about it some more.
Aside from this problem, does your workflow run with dxWDL+DNAx?

This particular workflow does not compile, so I have not run it on DNAnexus. I have compiled others via dxWDL and have shared that project with you. The workflows I am attempting to compile are here. The fail-to-compile workflows are any of the ones involving kraken or demux. The traditional (non-WDL) DNAnexus workflows for the same processes are here.

Ok, while this still may be a nice feature to add later on, I've unblocked my own situation by moving this into the default runtime block in the config file for cromwell and adding some shell code to the task's command section to determine if certain mount points exist or not.

I would say that this is a portability problem, much better solved at the WDL specification level. Of course, that doesn't actually solve the problem.

On a personal level, I've found ways around this and am fine if we close the issue.


Issue with reading scale factor of "SymphonyMilkyWay" suite

Reading the scale factor of "SymphonyMilkyWay" suite will raise the following error:
ValueError: The halo in SymphonyMilkyWay/Halo023 does not belong to a recognized suite.
This error is caused by the following lines in the function scale_factors of symlib/lib.py:
suite_name = halo_dir_to_suite_name(dir_name)
if (suite_name in ["SymphonyLMC", "SymphonyGroup", "SymphonyMilkyWayDisk",
                   "SymphonyMilkyWayDiskDMO", "EDEN_MilkyWay_8K",
                   "EDEN_MilkyWay_16K"
                  "SymphonyMilkyWay", "MWest", "SymphonyMilkyWayLR",
                   "SymphonyMilkyWayHR"] or
    dir_name in ["SymphonyLMC", "SymphonyGroup", "SymphonyMilkyWayDisk",
                 "SymphonyMilkyWayDiskDMO", "EDEN_MilkyWay_8K",
                 "EDEN_MilkyWay_16K",
                 "SymphonyMilkyWay", "MWest", "SymphonyMilkyWayLR",
                 "SymphonyMilkyWayHR"]):

A comma is missing at he end of the fourth line.

Ah sorry that I missed this. I had covid. This should be fixed in the latest version.

Hi @phil-mansfield , thank you for looking into this.  I hope you have fully recovered now. I think the issue still exists in the latest version.

Oops, you're right. It was fixed on my local version, but I didn't push it. Can you check again (v1.4.17)?

Oops, you're right. It was fixed on my local version, but I didn't push it. Can you check again (v1.4.17)?

Yes, I just tried. The issue is fixed in v1.4.17. Thanks again.

Excellent. I'll close the issue now. Sorry again for the long initial wait!


Fixed bug #300: We need to beef up our logging of exceptions

PR is added to the queue for testing as 1. task. (982a57e)


Bigendian

A first try to get proper test coverage for alternate architectures:

Build the whole thing on these architectures and let the ctests run
Fix string constants, make them always UTF-16-LE

What is not yet addressed:

TestFreeRDPCodecProgressive has some big endian issue
TestBacktrace and TestASN1 fail on ppc64le
TestConnect has an issue with all of the builds, most likely some restriction of the container, needs to be investigated

@matoro do you know by chance why the asan stuff is failing? the library should be installed and I´ve already switched to clang as compiler, but still :/

@matoro do you know by chance why the asan stuff is failing? the library should be installed and I´ve already switched to clang as compiler, but still :/

No, I had to just chalk it up to QEMU issue.  Do you want to leave TestConnect?  This could be simply a race condition, there is a big perf hit due to QEMU so it could be that there just isn't enough delay for the server thread to spin up.

@matoro do you know by chance why the asan stuff is failing? the library should be installed and I´ve already switched to clang as compiler, but still :/

No, I had to just chalk it up to QEMU issue. Do you want to leave TestConnect? This could be simply a race condition, there is a big perf hit due to QEMU so it could be that there just isn't enough delay for the server thread to spin up.

would be nice to have that test working (test coverage around 60% of code, as a client and a server must be running and doing some protocol exchanges)
no other test does cover that much of the code base.
can try with extended timeouts though.

I can also do a manual ASAN run on real hardware to make sure there aren't any hits specific to BE codepath.
no need, it would only be great to have that integrated to the tests.

@matoro I can not find the issue the remaining architectures have, do you have any idea?

@matoro I can not find the issue the remaining architectures have, do you have any idea?

The ppc64le ones seem to be bogus qemu issues.  I just tested on real hardware and everything passes.  They might have to be excluded.
The big-endian one on s390 is legit though.  I reproduced on ppc64 BE and it says:
154/169 Testing: TestFreeRDPCodecProgressive
154/169 Test: TestFreeRDPCodecProgressive
Command: "/var/tmp/portage/net-misc/freerdp-9999/work/freerdp-9999_build/Testing/TestFreeRDPCodec" "TestFreeRDPCodecProgressive"
Directory: /var/tmp/portage/net-misc/freerdp-9999/work/freerdp-9999_build/libfreerdp/codec/test
"TestFreeRDPCodecProgressive" start time: Jan 24 16:00 UTC
Output:
----------------------------------------------------------
xxxxxxx [19:16] [PIXEL_FORMAT_BGRX32] 1F1E1C8A != 881D1FFF
<end of output>
Test time =   0.07 sec
----------------------------------------------------------
Test Failed.
"TestFreeRDPCodecProgressive" end time: Jan 24 16:00 UTC
"TestFreeRDPCodecProgressive" time elapsed: 00:00:00
----------------------------------------------------------

Is that enough of a clue?

@matoro ok, the progressive one was something I can work with, but the others were driving me nuts (aka I could not reproduce locally in my qemu)

@matoro ok, found the issues :)
there is a problem with unwind or execinfo not working in the containers, so I´ve deactivated backtracing functions.
now only thing left is the trigger for this, I think running for every pull request is overkill, but once a week on current master would give a nice status report.

@matoro ok, found the issues :) there is a problem with unwind or execinfo not working in the containers, so I´ve deactivated backtracing functions.
now only thing left is the trigger for this, I think running for every pull request is overkill, but once a week on current master would give a nice status report.

Wow, thanks so much for this huge effort!  Please let me know if there's anything I can help with, or feel free to reach out for checking on unusual arches!

split this: only the workflow files are part of this, code changes are in #9848


CURLOPT_FOLLOWLOCATION and open_basedir

Hey, great work on the library, we've been testing it and it seems to be pretty robust. We ran into a minor issue and I'd like to propose a change:
Basically, BaseHTTPQRCodeProvider crashes if the open_basedir php directive is set, and it doesn't appear that a workaround exists:
http://stackoverflow.com/questions/19539922/php-can-curlopt-followlocation-and-open-basedir-be-used-together
And while "just turn the directive off" can be a solution for some, like the thread there suggests it can be a security liability to get rid of it. So I would vote to remove CURLOPT_FOLLOWLOCATION from the class, unless there's a compelling reason to keep it. Google Charts seems to work fine at least without redirects.

I don't have a problem with removing the CURLOPT_FOLLOWLOCATION option. On it.

Here you go!

Hurray, thanks! :) We were already running this change in a hacky way, but it's always nicer to have it in the upstream version.


Adding some missing descriptions...

Adding some missing descriptions to match the documentation.
Also adding zoom.wheelDelta. Unsure of would be preferred, so I put it after zoom.filter to match the d3-zoom documentation.

These changes look good to me.


Solvers do not model uncertainty

Hi there!
Thanks for creating this toolbox!
I just tried the solvers for the first time and I'm running into some problems. My ODE is a Hodgkin-Huxley model (a model in neuroscience). Some specs (full code below):

the model has 4 states
it's non-linear
it's non-homogenous

When simulating it with the EK0 solver and low tolerance, the solution is the same as if I used a simple Euler() solver. The EK0 solution is the light blue curve in the figure below. However, when I increase the tolerance in the EK0 solver, the solution changes, but all samples are the same. These are the other five traces in the figure (you can see only the dark green one because they are almost, but not bitwise, identical).

Have you tried running the Hodgkin-Huxley model?
Should I expect different / better results when using https://github.com/probabilistic-numerics/probnum?
Have you observed this failure to model uncertainty?

Thanks!
Michael

Hi Michael, thanks for opening this issue! It's very helpful to see what problems people are working on and to get some feedback on these solvers.
Thanks also for providing all the code. I looked into the example and might have a few pointers on how to get more useful output, but let me first answer your questions.

Have you observed this failure to model uncertainty?

Unfortunately, yes. Model calibration is not perfect, and especially for high-tolerance regimes the model can become overconfident. For low tolerances however my current experience rather suggests underconfident uncertainties.

Should I expect different / better results when using https://github.com/probabilistic-numerics/probnum?

I don't think you should expect different results with ProbNum. It does provide some additional functionality, such as more priors (IOUP and Matern) and an unscented Kalman filtering-based solver, but I also think that these should not influence the qualitative behaviour of the calibration.
I still encourage you to have a look if you're interested in this topic, the documentation is much more polished and the tutorials might be helpful if you want to learn more about these solvers.

Have you tried running the Hodgkin-Huxley model?

I have not previously considered this model, but I suspect that it might be one of the more challenging settings for the EK solvers (compared to sampling-based solvers), especially with high tolerances, due to the fact that they are only able to model Gaussian uncertainties.
But, I was able to obtain a more useful output:
The EK0 with default diffusionmodel=:dynamic models the same uncertainty for each dimension, but the scales in the Hodgkin-Huxley example vary a lot between the dimensions. With EK0(diffusionmodel=:dynamicMV), the solver models the uncertainties in each dimension independently from each-other, resulting in larger uncertainties for the first component:

Now, the plot clearly shows that the solver is indeed very uncertain about the result.
It is still not perfect and the samples look very different from the actual solution, but that might be the point where you would need non-Gaussian multi-modal distributions, e.g. provided by sampling-based methods.
I hope that helped, let me know if you have more questions! And of course, please feel free to open issues about anything that is unclear or that doesn't work.

Hi Nathanael,
thanks for your reply and your efforts. It's pretty cool that there is uncertainty now! You are raising a good point about the Gaussianity...I'm pretty sure the "true" uncertainty-aware likelihood is non-Gaussian. When adding Brownian motion to the differential equation (which, from my layman's perspective seems very similar to sampling-based approaches), the marginal histogram over samples is highly non-Gaussian.
Regarding dynamicMV: I guess the scenario of having dimensions on different scales is pretty common. Is there a reason to not make dynamicMV the default?
Anyways, thanks a lot for your efforts, this is a really cool project!
Michael

Hi Michael, I'm glad to hear that my reply was helpful.
The dynamicMV diffusion model unfortunately does not work with the EK1, which is typically my preferred solver, therefore I made the dynamic diffusion model the default. But this is a great example on why I might want to re-evaluate this choice. A warning is also a great idea to communicate these model differences better to users.
So, thanks again for trying out our library and for opening this issue! It provided very helpful input for us :)
Feel free to reach out if you have any more questions or issues.


Two way audio cameras

Hello,
First of all thanks for all the work !
Also, I have reolink CX410, do you confirm the two way audio doesn't work ?
And do you have a list of working cameras with two way audio ? Or something to watch in the caracteristics of the camera ?
Thanks again

There are, tens and even hundreds of thousands of camera models in the world. Plus there are countless unnamed cameras from China. Plus depending on the firmware of the camera, its functionality may vary. Including Reolink cameras. I think it's impossible to collect a reliable list.
You can view the functionality of your camera in the go2rtc WebUI (info/probe).


Overall cleanup of HTTP/2 handlers / Do not send GOAWAY to a closed connection

Motivation:

We currently add 2 Http2Connection.Listeners, Http2GoAwayListener
and Http2(Request|Response)Decoder. We could add only a single
listener so that Netty does not waste time to notify the events which
is uninteresting to Http2GoAwayListener.
We handle GOAWAY frames in more than one place, which is not great for
readability.
Netty invokes onGoAwaySent() callback even if the connection has
been closed already and thus a GOAWAY frame is never sent. This makes
Http2GoAwayListener log a false-positive message at WARN level.

Modifications

Renamed Http2GoAwayListener to Http2GoAwayHandler.
Http2GoAwayHandler is not an Http2ConnectionAdapter anymore.
Moved all GOAWAY-related code to Http2GoAwayHandler.
Clean-up

Reordered some constructor parameters so that the parameters of superclass
appears first.
Removed Http2Connection from parameter list if there is
Http2ConnectionEncoder in the parameter list, because
Http2Connection can be retrieved via Http2ConnectionEncoder.connection().
Cleaned up the code that injects verbose debug data into
Http2Exception.

Miscellaneous:

Fixed a bug where HTTP/2 graceful shutdown timeout it set to 0 when
Armeria idle timeout is disabled.

Result:

Fewer number of Http2Connection.Listeners
Less scattered code
Less noisy log messages

Thanks for reviewing. Let me release 0.74.1 soon.


Update all dependencies, build binaries via go 1.11.1

and xenial, too!

LGTM!
I saw reports of increased virtual memory usage on 1.11.1 (https://github.com/golang/go/issues/28114), might be a non issue though.

Getting the most recent commit green required busting the cache :disappointed: I'm going to see about ensuring vendor/manifest is not overwritten at cache import time, or at least is restored prior to make deps.


RSDK-4053 add Camera::GetImages

Still need to add tests - will do that tomorrow morning

@cheukt FYI


Missing M1.5 arbour rod sizes for small winder set

For the smallest spectrum of winders, an M1.5 arbour rod would be ideal.

Resolved in latest release.


Validation loss and acc don't show

I'm trying to fine-tune squeeze net and i've edited the deploy.prototxt but only the training values are shown, not the validation. My validation set is a fraction of my testing set.
The following are the final layers of my network.
....
layer {
name: "relu_conv10"
type: "ReLU"
bottom: "conv10_new"
top: "conv10_new"
}
layer {
name: "pool10"
type: "Pooling"
bottom: "conv10_new"
top: "pool10"
pooling_param {
pool: AVE
global_pooling: true
}
}
layer {
name: "loss"
type: "SoftmaxWithLoss"
bottom: "pool10"
bottom: "label"
top: "loss"
include {
phase: TRAIN
}
}
layer {
name: “softmax”
type: "Softmax"
bottom: "pool10"
top: “softmax”
exclude {
phase: TRAIN
}
}
layer {
name: "accuracy"
type: "Accuracy"
bottom: "pool10"
bottom: "label"
top: "accuracy"
include {
phase: TRAIN
}
}
layer {
name: "accuracy_top5"
type: "Accuracy"
bottom: "pool10"
bottom: "label"
top: "accuracy_top5"
include {
phase: TRAIN
}
accuracy_param {
top_k: 5
}
}

Update: I commented out all the includes and the same thing is still happening

I'm a bit confused about the difference between a 'phase' and a 'stage'; phase: TRAIN and stage: deploy, for example.
@lukeyeager if you could clarify, that would be great.
Thank you

The documentation for this can be accessed when creating a custom network. See this screenshot: https://github.com/NVIDIA/DIGITS/pull/628#issuecomment-195137527


Implement new Loadable behavior for pages and sections

@natritmeyer
This PR adds a fully backwards compatible feature I call 'load validations' to SitePrism which we have been using internally for a while now with great success.  It provides a common abstraction for dealing with asynchronous behavior on pages, which can be a significant source of intermittent failures due to waiting for elements to be displayed.
This allows you to very cleanly abstract all of your waiting logic to a single place with a common pattern, and even allows for common validation logic to be shared between page if needed.  Additionally, it makes it very easy and clean to encapsulate this behavior in your tests without spreading boilerplate code throughout them.
The PR has thorough documentation and tests, please take a look.  I have additionally run our entire acceptance test suite against a branch of our SitePrism fork containing these changes, with the implementation of this in our test suite having been removed.
I can provide a more thorough contrived example if needed, but the basic gist assuming you have some pages defined with load validations is this:
it 'does something' do
  home_page.load do |page|
    # Passing a block to the load method loads the page as usual,
    # then implicitly runs load validations before executing the block.
    # Existing functionality of the `load` method is not affected.
    page.register_button.click
  end

  registration_page.when_loaded do |page|
    # The page instance is yielded into the `load` and `when_loaded` blocks.  This allows you to cleanly encapsulate
    # interactions with each page, which is especially nice when performing multiple interactions
    # with a single page.
    page.email.set<EMAIL_ADDRESS>    page.password.set 'hunter2'
    page.submit
  end

  # Here we call a helper method on a page object which uses when_loaded to allow this to be executed as a one-liner in the test:
  another_page_with_a_form.fill_and_submit

  yet_another_page.when_loaded do |page|
    # Here we call a series of helper methods and/or element interactions on the page object.
    # Some of these methods may wrap their implementation in `when_loaded` blocks so that they
    # can be used safely as either a one-liner in a test, or as part of a larger construct as it is here.
    # The load caching in the Loaded module makes these nested calls extremely efficient, as only
    # the outermost `when_loaded` call actually execute the load validations while inner calls use the
    # cached result of the outermost block.
    page.do_thing_1
    page.do_thing_2
    page.complete
  end
end

We have found this to be very useful because:

It removes boilerplate code for page readiness checks such as page.displayed? and/or testing for element existence in tests and/or helper methods prior to interacting with a page.
It avoids nested/duplicate page readiness checks in test and helper methods, which incur a performance hit performing multiple duplicate queries against the driver.
It provides a single, common abstraction for handling and using load validations for every page object.
It makes our tests very clean with obvious scoping for the interactions with each page

I additionally believe this is a sufficient solution for issues #48, #49, and #50, as it enables users to define how they want a page to be tested for readiness which may include checks of 0 or more elements as needed and may include recursive checks against sections which may additionally provide their own load validations if desired.
It is also much more efficient and useful than all_there?, which does not provide an implicit wait mechanism and is a sledgehammer where a scalpel is needed.  Checking for all elements when testing if a page is ready for interaction under normal circumstances incurs a significant negative performance penalty for all tests that use it, where testing for URL and usually one or two elements is usually sufficient.
Please let me know your thoughts and comments.  From a functional standpoint, this PR is fully backwards-compatible, fully tested, thoroughly documented and should be ready to merge pending your approval.
Cheers!

Thank you so much for this! This is something I've always implemented in a wrapper around loading pages - awesome to see it implemented this well right into site prism!

rubocop issues have been fixed.

@gerryster we chatted about this at windy city rails.

Any PR that references James Bond in the tests is worthy of serious consideration :)

Can this be merged released? Totally struggeling with bootstrap modals and waiting all over the place and not really getting anywhere :D

Just pushed the latest version

Ooooh this looks interesting @tmertens !  I'm going try to make time to read through the whole thing too in the next week or so!

@natritmeyer Any update on this?

Been playing around with it, looks great :) I've got a couple of friends testing it out and I'm waiting on feedback from them.
Anyone else had a play around with this? I'd be interested in what others think :)
Also, my wife was due to give birth yesterday so I'm kinda distracted ;)

@natritmeyer If it's happened already, congratulations! :smile:
I'll admit I haven't looked in-depth yet but it's now officially on my TODO.
(Shamelessly) if after the birth you somehow find time to merge/review mine as well I'd be pleased.  Although I suspect I'll have to redraw it after this.

@tgaff Have you had a chance to look at this yet?
We have 6 QA engineers and 50+ devs relying on a test suite that uses this feature extensively in our acceptance test suite for a few months now, so I'm quite confident in it ;)  It has made our tests substantially more reliable and easy to maintain.

@tgaff thanks, the boy has arrived safe and sound :)
@tmertens thanks for the reminder :) I've been spending the last week with my wife in the High Dependency Unit after she suffered some fairly major complications after an emergency c-section that went wrong. As excuses go, it's a pretty good one :)
I should be home in 2-3 days; I'll get to the rest starting from then :)
This comment has used up my daily emoticon quote

@natritmeyer That's why i didn't ask, I figured you were busy :)  I hope your wife is doing better now, and congratulations!

@tmertens Finally got a chance and ran into no issues; sorry for the continued delay.  I see this has been merged :+1:


Feature request: Kinesis Data Firehose event envelope and parser model

Issue number: 1555
Summary
Add Kinesis Data Firehose event envelope and parser model
Checklist
If your change doesn't seem to apply, please leave them unchecked.

[ ] Meet tenets criteria
[ ] I have performed a self-review of this change
[ ] Changes have been tested
[ ] Changes are documented
[ ] PR title follows conventional commit semantics

Is this a breaking change?
RFC issue number:
Checklist:

[ ] Migration process documented
[ ] Implement warnings (if it can live side by side)

Acknowledgment
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

View rendered docs/utilities/parser.md

@leandrodamascena we might want to do an E2E for this. Due to parser's nature, it has the highest risk of causing runtime disruption.

Codecov Report
Base: 99.73% // Head: 99.77% // Increases project coverage by +0.03% :tada:

Coverage data is based on head (2512ab9) compared to base (22459e7).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1556      +/-   ##
===========================================
+ Coverage    99.73%   99.77%   +0.03%
===========================================
  Files          124      126       +2
  Lines         5718     5757      +39
  Branches       651      656       +5
===========================================
+ Hits          5703     5744      +41
+ Misses           8        6       -2
  Partials         7        7

Impacted Files
Coverage Δ

...a_powertools/utilities/parser/envelopes/kinesis.py
100.00% <ø> (ø)

aws_lambda_powertools/logging/formatter.py
100.00% <100.00%> (ø)

aws_lambda_powertools/shared/constants.py
100.00% <100.00%> (ø)

aws_lambda_powertools/shared/functions.py
92.59% <100.00%> (+7.40%)
:arrow_up:

..._powertools/utilities/parser/envelopes/__init__.py
100.00% <100.00%> (ø)

...ols/utilities/parser/envelopes/kinesis_firehose.py
100.00% <100.00%> (ø)

...bda_powertools/utilities/parser/models/__init__.py
100.00% <100.00%> (ø)

...mbda_powertools/utilities/parser/models/kinesis.py
100.00% <100.00%> (ø)

...rtools/utilities/parser/models/kinesis_firehose.py
100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Hii @ran-isenberg thanks for one more PR :smiley:.. I'm starting to work on it now.
@heitorlessa in fact adding e2e tests is a big improvement in the parser area. I just need to check how much data we need to generate in Kinesis Firehose with PUT source and Kinesis Firehose with Kinesis Stream source to trigger an event in Lambda. I think we should open a new PR to include e2e tests in v2 branch while this is not the official branch.

+1 for E2E tests.
but on the flip side, it's going to make adding new services way harder for newcomers.

+1 for E2E tests. but on the flip side, it's going to make adding new services way harder for newcomers.

Yeah Ran, it makes sense and I understand your concern.. But for now we can write e2e tests when someone adds a new service, feature or even extends an existing feature and make life easier for new contributors.

+1 for E2E tests. but on the flip side, it's going to make adding new services way harder for newcomers.

Yeah Ran, it makes sense and I understand your concern.. But for now we can write e2e tests when someone adds a new service, feature or even extends an existing feature and make life easier for new contributors.

You use CDK right?

You use CDK right?

Yeah, we use CDK for e2 testing. But I have some considerations to make about e2e testing before we think about any implementation on Parser.

e2e tests will only work in v2, so in this PR we should not write end2end tests for the parser.
We have a release date for v2 later this month and we are working hard for it.
We believe that only the e2e tests are not enough in all cases. For e.g, if the parser fails for any change in the payload, we run a risk of breaking the function, because it is a runtime error. We need to write a RFC to think of a solution where we can update all lambda eventSources payload every X days/hours, because the payload can change on the service/lambda side and if we don't have the updated project it can be a problem. Maybe a separate project to deal with this, I'm not sure yet, but we'll start that discussion soon.

You use CDK right?

Yeah, we use CDK for e2 testing. But I have some considerations to make about e2e testing before we think about any implementation on Parser.

e2e tests will only work in v2, so in this PR we should not write end2end tests for the parser.
We have a release date for v2 later this month and we are working hard for it.
We believe that only the e2e tests are not enough in all cases. For e.g, if the parser fails for any change in the payload, we run a risk of breaking the function, because it is a runtime error. We need to write a RFC to think of a solution where we can update all lambda eventSources payload every X days/hours, because the payload can change on the service/lambda side and if we don't have the updated project it can be a problem. Maybe a separate project to deal with this, I'm not sure yet, but we'll start that discussion soon.

The way I'd handle schemas is to put the models in a new repo and maintain proper versions BUT it should be managed and updated by the AWS services, not the community. They should be aware of how we use and model the events since sometime the docs are not good enough.
If there are any unit/e2e tests, I'd expect the services to run them each time they make a change, some sort of  a  contract test. The E2E tests will run here per PR but you probably wont catch.

We need to update the code to account for differences between payloads.
Payload using Kinesis Firehose with a Kinesis Stream as source:
{
   "invocationId":"77c0024f-9869-4af0-951a-d83c102e8a63",
   "sourceKinesisStreamArn":"arn:aws:kinesis:us-east-1:200984112386:stream/kinesis-evento",
   "deliveryStreamArn":"arn:aws:firehose:us-east-1:200984112386:deliverystream/KDS-S3-10POr",
   "region":"us-east-1",
   "records":[
      {
         "recordId":"49633602406469139358835205474851822974501572780333465650000000",
         "approximateArrivalTimestamp":1664028793294,
         "data":"eyJ1c2VyX2lkIjoidXNlcjEiLCAic2NvcmUiOiAxMDB9Cg==",
         "kinesisRecordMetadata":{
            "sequenceNumber":"49633602406469139358835205474851822974501572780333465650",
            "subsequenceNumber":0,
            "partitionKey":"0",
            "shardId":"shardId-000000000003",
            "approximateArrivalTimestamp":1664028793294
         }
      },
      {
         "recordId":"49633602406469139358835205474853031900321189264934043698000000",
         "approximateArrivalTimestamp":1664028820148,
         "data":"eyJ1c2VyX2lkIjoidXNlcjEiLCAic2NvcmUiOiAxMDB9Cg==",
         "kinesisRecordMetadata":{
            "sequenceNumber":"49633602406469139358835205474853031900321189264934043698",
            "subsequenceNumber":0,
            "partitionKey":"0",
            "shardId":"shardId-000000000003",
            "approximateArrivalTimestamp":1664028820148
         }
      }
   ]
}

Payload using Kinesis Firehose with Direct PUT:
{
   "invocationId":"25326478-5c08-45ad-a34d-7342edbc942d",
   "deliveryStreamArn":"arn:aws:firehose:us-east-1:200984112386:deliverystream/PUT-S3-PCYyW",
   "region":"us-east-1",
   "records":[
      {
         "recordId":"49633602747871247603140515133584305740288993709252411394000000",
         "approximateArrivalTimestamp":1664029185290,
         "data":"eyJ1c2VyX2lkIjoidXNlcjEiLCAic2NvcmUiOiAxMDB9Cg=="
      },
      {
         "recordId":"49633602747871247603140515133585514666108608544585547778000000",
         "approximateArrivalTimestamp":1664029186945,
         "data":"eyJ1c2VyX2lkIjoidXNlcjEiLCAic2NvcmUiOiAxMDB9Cg=="
      }
   ]
}

We can go in the same direction as we went here #1499 and create separate classes for each type of payload.

I've modeled the metadata as optional so it works for both cases. The only reason to split them is if it's officially documented as such. do you think that's the case? I didnt find a mention that metadata params are applicable only for put events. If we add by ourselves, it's more likely to be broken in the future.
WDYT?

The way I'd handle schemas is to put the models in a new repo and maintain proper versions BUT it should be managed and updated by the AWS services, not the community. They should be aware of how we use and model the events since sometime the docs are not good enough. If there are any unit/e2e tests, I'd expect the services to run them each time they make a change, some sort of a contract test. The E2E tests will run here per PR but you probably wont catch.

I'll open an issue and add this discussion in there, so we can keep focus here in this PR. ok?

The way I'd handle schemas is to put the models in a new repo and maintain proper versions BUT it should be managed and updated by the AWS services, not the community. They should be aware of how we use and model the events since sometime the docs are not good enough. If there are any unit/e2e tests, I'd expect the services to run them each time they make a change, some sort of a contract test. The E2E tests will run here per PR but you probably wont catch.

I'll open an issue and add this discussion in there, so we can keep focus here in this PR. ok?

No problem.

@leandrodamascena added my small fix and fixed develop conflict

@leandrodamascena added my small fix and fixed develop conflict

Heyy good morning and happy monday @ran-isenberg!
If you don't have any other considerations, I think we are good to merge it!
Thanks for addressing all feedback so quickly. :fast_forward:

LGTM


Debianization for 1.0

Done in https://github.com/3Hren/blackhole/commit/d3a890fba8fef134145bcf1d52d8b7189c7442a2.


docker-compose -f setup.yml up does not work on Windows 10

docker-compose -f setup.yml up fails with the error
WARNING: The PWD variable is not set. Defaulting to a blank string.
Recreating<PHONE_NUMBER>5b_stack-docker_setup_1 ... error
ERROR: for<PHONE_NUMBER>5b_stack-docker_setup_1  Cannot create container for service setup: invalid volume specification: '/host_mnt:rw'
ERROR: for setup  Cannot create container for service setup: invalid volume specification: '/host_mnt:rw'
ERROR: Encountered errors while bringing up the project.

@SavvySkylark can you confirm that you followed the following steps in the README?

Windows Users must set the following 2 ENV vars:

COMPOSE_CONVERT_WINDOWS_PATHS=1
PWD=/path/to/checkout/for/stack-docker

for example I use the path: /c/Users/nick/elastic/stack-docker
Note: you're paths must be in the form of /c/path/to/place using C:\path\to\place will not work

You can set these two ways:

Temporarily add an env var in powershell use: $Env:COMPOSE_CONVERT_WINDOWS_PATHS=1
Permanently add an env var in powershell use: [Environment]::SetEnvironmentVariable("COMPOSE_CONVERT_WINDOWS_PATHS", "1", "Machine")

Note: you will need to refresh or create a new powershell for this env var to take effect

in System Properties add the environment variables.

PS C:\Users\myuser\stack-docker> echo $Env:COMPOSE_CONVERT_WINDOWS_PATHS
1

PS C:\Users\myuser\stack-docker> $PWD="/c/Users/myuser/stack-docker"

PS C:\Users\myuser\stack-docker> echo ${PWD}
/c/Users/myuser/stack-docker

PS C:\Users\myuser\stack-docker> docker-compose -f setup.yml up -d
docker-compose : The PWD variable is not set. Defaulting to a blank string.
At line:1 char:1
+ docker-compose -f setup.yml up -d
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (The PWD variabl...a blank string.:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

Creating stack-docker_setup_1 ...
Creating stack-docker_setup_1 ... error
ERROR: for stack-docker_setup_1  Cannot create container for service setup: invalid volume specification: '/host_mnt:rw'
ERROR: for setup  Cannot create container for service setup: invalid volume specification: '/host_mnt:rw'
Encountered errors while bringing up the project.

@fxdgear
Went back through the steps and  set the environment variables using the "Temporarily add an env var in powershell" method.
No longer getting the PWD error but now I'm getting.
setup_1  | cat: can't open './scripts/setup.sh': No such file or directory
below are the commands I put into powershell.
PS C:\git\stack-docker> $Env:COMPOSE_CONVERT_WINDOWS_PATHS=1
PS C:\git\stack-docker> $Env:PWD="/c/git/stack-docker"
PS C:\git\stack-docker> docker-compose -f setup.yml up
Starting stack-docker_setup_1 ... done
Attaching to stack-docker_setup_1
setup_1  | cat: can't open './scripts/setup.sh': No such file or directory
stack-docker_setup_1 exited with code 0

Seems like the volume bind mount is not working.
I'm using this version of docker for windows

below is the inspect command of the stack-docker_setup_1 container
[
    {
        "Id": "2cf5d837626daa86ed44aad12f5e2df746d20ab04e8aff9021c5170dc054fc8a",
        "Created": "2019-03-13T22:23:31.8617415Z",
        "Path": "/bin/ash",
        "Args": [
            "-c",
            "cat ./scripts/setup.sh | tr -d \"\\r\" | ash"
        ],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2019-03-13T22:24:57.4191388Z",
            "FinishedAt": "2019-03-13T22:24:57.4826454Z"
        },
        "Image": "sha256:86bf85b58bc85f693bca134e6334baac73b3a8461cfe6e48018f1da268801b75",
        "ResolvConfPath": "/var/lib/docker/containers/2cf5d837626daa86ed44aad12f5e2df746d20ab04e8aff9021c5170dc054fc8a/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/2cf5d837626daa86ed44aad12f5e2df746d20ab04e8aff9021c5170dc054fc8a/hostname",
        "HostsPath": "/var/lib/docker/containers/2cf5d837626daa86ed44aad12f5e2df746d20ab04e8aff9021c5170dc054fc8a/hosts",
        "LogPath": "/var/lib/docker/containers/2cf5d837626daa86ed44aad12f5e2df746d20ab04e8aff9021c5170dc054fc8a/2cf5d837626daa86ed44aad12f5e2df746d20ab04e8aff9021c5170dc054fc8a-json.log",
        "Name": "/stack-docker_setup_1",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/host_mnt/c/git/stack-docker:/c/git/stack-docker:rw",
                "/var/run/docker.sock:/var/run/docker.sock:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "stack-docker_default",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": [],
            "CapAdd": [
                "SYS_ADMIN"
            ],
            "CapDrop": null,
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/81110ea7d1bb48f6496ef5f3997abcc3def064b2c7571c2b6c92ea33c875e6a7-init/diff:/var/lib/docker/overlay2/3351e40034feab0713512568e0e9bb6e1c4422ea5d919040c16c062c63ccbb5d/diff:/var/lib/docker/overlay2/6aec6c943093bf340cc040850bfcd6d03018c0800ffbacc132796429d4b9c242/diff:/var/lib/docker/overlay2/36bc378c799877045620dd8a171a141ce6e8743549df493f483e070ed2d74f7a/diff",
                "MergedDir": "/var/lib/docker/overlay2/81110ea7d1bb48f6496ef5f3997abcc3def064b2c7571c2b6c92ea33c875e6a7/merged",
                "UpperDir": "/var/lib/docker/overlay2/81110ea7d1bb48f6496ef5f3997abcc3def064b2c7571c2b6c92ea33c875e6a7/diff",
                "WorkDir": "/var/lib/docker/overlay2/81110ea7d1bb48f6496ef5f3997abcc3def064b2c7571c2b6c92ea33c875e6a7/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/host_mnt/c/git/stack-docker",
                "Destination": "/c/git/stack-docker",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/run/docker.sock",
                "Destination": "/var/run/docker.sock",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "2cf5d837626d",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PWD=/c/git/stack-docker",
                "ELASTIC_PASSWORD",
                "ELASTIC_VERSION=6.6.0",
                "TAG=6.6.0",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "GLIBC=2.27-r0",
                "DOCKERBINS_SHA=1270dce1bd7e1838d62ae21d2505d87f16efc1d9074645571daaefdfd0c14054"
            ],
            "Cmd": [
                "cat ./scripts/setup.sh | tr -d \"\\r\" | ash"
            ],
            "Image": "docker/compose:1.21.2",
            "Volumes": {
                "/c/git/stack-docker": {},
                "/var/run/docker.sock": {}
            },
            "WorkingDir": "/c/git/stack-docker",
            "Entrypoint": [
                "/bin/ash",
                "-c"
            ],
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "1af6873a66f9821cf6d569643aec919b9a7c10d89c9cb2a607f50170924a93d8",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "stack-docker",
                "com.docker.compose.service": "setup",
                "com.docker.compose.version": "1.23.2"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "7627d0dd3c283af055f7aac7b2cbb0d8c28a3bd78a4fe8f4603e03450d6d4b38",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/7627d0dd3c28",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "stack-docker_default": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "2cf5d837626d",
                        "setup"
                    ],
                    "NetworkID": "0fc9448d03b8767f499f461c7c0c409c1fbdf67a496483c1e328742c5a141ee6",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]

sorry accidentally closed issue.

In Powershell ISE I was able to first paste the command and have it run as a single operation.
$Env:PWD='/c/Users/__user__/stack-docker'
docker-compose -f setup.yml up

@sprankle Hello, i have the same issue, i try with powershell and powershell ISE with the administrator mode.
PS C:\WS\stack-docker> $Env:PWD="/c/ws/stack-docker"
PS C:\WS\stack-docker> $Env:COMPOSE_CONVERT_WINDOWS_PATHS=1
PS C:\WS\stack-docker> docker-compose -f setup.yml up
Starting stack-docker_setup_1 ... done
Attaching to stack-docker_setup_1
setup_1  | cat: can't open './scripts/setup.sh': No such file or directory

I was able to resolve the problem "setup_1  | cat: can't open './scripts/setup.sh': No such file or directory" by editing setup.yml :
Seems to be windows can not handle "." as current working directory and can't find the file so I had to specify its location:
command: ['cat "$PWD/scripts/setup.sh" | tr -d "\r" | ash']
One additional thing, if you do this on Win10 then you get an apm-server error, that a file is missing:
setup_1  | If you wish to remove the setup containers please run:
setup_1  |      docker-compose -f docker-compose.yml -f docker-compose.setup.yml down --remove-orphans
ERROR: for apm-server  Cannot create container for service apm-server: b'Mount denied:\nThe source path "C:/Users/Attila_Szabo1/stack-docker/stack-docker/config/apm-server/apm-server.keystore"\ndoesn't exist and is not known to Docker'
ERROR: Encountered errors while bringing up the project.
So by removing all manually then recreating all again but nothing cleaning the orpans I made it working.

The can't open './scripts/setup.sh': No such file or directory"  error seems to only happen when using PowerShell. The setup process works correctly in the Command Prompt.
However, the actual docker-compose up -d command failed for me in the same way as @atekka85epam , but I was able to get it running by removing all the setup containers manually and then rerunning the setup with the --remove-orphans flag:
docker-compose -f setup.yml up --remove-orphans

After doing this, I just ran docker-compose up -d again and everything worked.
@fxdgear , any ideas why would the above be happening to us? Seems that the keystore file cannot be created in the local Windows file system for some of the Elastic services.

I wish I knew the "workings" of windows better. I know that it "works" in windows cause I was able to get it working using windows. But as for why sometimes it has to be retried... I'm not sure. It might have something to do with the way the hypervisor is mounting the file system? Usually the best course of action when it fails on windows is to delete everything (images, volumes, containers, networks etc..) and re-run it.


[Question] Persistence of custom models

Hi!
Thanks a lot for the nice library!
I see that, in the documentation, you write:

At the moment, skops cannot persist arbitrary Python code. This means if you have custom functions (say, a custom function to be used with sklearn.preprocessing.FunctionTransformer), it will not work.

I totally understand it, since this is far from being trivial.
However, I'm wondering if you have any plans on this topic.
I often find myself writing either custom sklearn models, by inheriting from sklearn.base.BaseEstimator, or wrappers of existing sklearn models. This, of course, complicates the model persistence problem when you want to use the model in another environment, without installing the package where the source code lives.
One solution is to bundle the model with the source code, but I have never found an easy and clean way to do it.
At the moment, I'm taking advantage of the experimental register_pickle_by_value function provided by cloudpickle. It works for simple use cases, but I don't really know what it's doing and all the cases where it might break.
I would like to ask if you think that, in the future, skops might help in this case as well.
Thanks a lot!

But if your requirement is to load the model somewhere that you can't install any packages, then that's tricky. We definitely plan to investigate how we could support this, but it's a rather tricky subject since it opens the door to all sorts of exploits.

Unfortunately, at the moment, this is the requirement. Just to give a little bit of context, let me explain why I raised this issue.
The current workflow we follow is as follows:

The data scientist trains a model inside a project-related repository, where she/he might need to write her/his own custom model.
The model is then serialized with cloudpickle and packaged into a zip file.
Later on, the model is loaded inside a home-made inference engine, which might also be deployed on embedded devices.
We would like to avoid having to install the data scientist's owned repository inside the runtime, as it might contain a lot of stuff not related to the model and it might cause a dependency management nightmare (in some cases, we cannot use docker). Also, this repository might be different for each project.
That's why, at the moment, the data scientist has to call register_pickle_by_value(my_package) before saving the pickle, where my_package is the package where the data scientist defined the custom model. This allows us to load the pickle file inside the runtime without installing the package. But it's not very robust and might break in multiple ways.

For these reasons, I've started looking for better alternatives.
In any case, thanks a lot for the quick answer!!

What you describe seems like a very non-trivial issue.

We would like to avoid having to install the data scientist's owned repository inside the runtime, as it might contain a lot of stuff not related to the model and it might cause a dependency management nightmare

It would be nice if the models could be factored out into their own package with only minimal dependencies. This package could be hosted on a private package index which is accessible from the embedded device. That would result in extra work for the data scientist but maybe it's an option. However, in that case pickle would also be an option, as there isn't much benefit of using skops.

It would be nice if the models could be factored out into their own package with only minimal dependencies

Yes, we will probably start looking into this. The definitive solution would be getting rid of python with something like ONNX, but then it becomes very hard to use custom models if they are written in pure python.
Thanks again for the answers and for the clarification!

 If that's the case, I guess it wold be useful for you to have a workflow where your DS would sign the pickle files, and then you check the signature before loading at least.
The skops.io module is not intended to support more than what pickle does really. You have a somewhat dangerous workflow ;) and it goes the opposite direction of what we're trying to go with this persistent format.
But I really appreciate you trying it out.

You have a somewhat dangerous workflow

We're at the early stages of development and our solution has been used only for demos so far, not real production environments. We're fully aware that the workflow is dangerous, and that's why we're looking for alternatives :)

If that's the case, I guess it wold be useful for you to have a workflow where your DS would sign the pickle files, and then you check the signature before loading at least.

This is loosely checked. But as long as the data scientist's code respects the input and the output, he can do whatever he wants in between. Still, that code that lives in between must be persisted somehow.
In any case, I appreciate a lot your suggestions!

In terms of handling dependencies in production and in your docker images / environments, it might be useful for you to have a look at how we do it: https://github.com/huggingface/api-inference-community/tree/main/docker_images/sklearn


dolibarr: init at 15.0.3, nixos/dolibarr: init

Description of changes
Taken from @aanderse branch, this is Dolibarr packaging & NixOS module.
TODO:

[x] applying the final polish in the module
[x] move to NGINX
[x] add manual notes
[x] adding a NixOS test module
[ ] fix it.
[ ] passthru the package for the NixOS test module
[ ] verify each TODO item

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[x] (Module addition) Added a release notes entry if adding a new NixOS module
[x] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

@aanderse If you do have some time, I would be interested into knowing if you could install Dolibarr and the conf.php, I'm getting errors (the NixOS test reproduce it) of the following form:
<!DOCTYPE HTML>

<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="generator" content="Dolibarr installer">
<link rel="stylesheet" type="text/css" href="default.css">

<link rel="stylesheet" type="text/css" href="../includes/jquery/css/base/jquery-ui.min.css" />

<title>Dolibarr install or upgrade</title>

15.0.3

Dolibarr install or upgrade - Configuration file
<form name="forminstall" style="width: 100%" action="step1.php" method="POST">
<input type="hidden" name="testpost" value="ok">
<input type="hidden" name="action" value="set">

Configuration file /var/lib/dolibarr/conf.php is not writable. Check permissions. For first install, your web server must be able to write into this file during configuration process ("chmod 666" for example on a Unix like OS).

<input type="hidden" name="selectlang" value="auto"></form>

My understanding though is that php-fpm is supposed to have the rights to edit the state directory, therefore, creating the configuration file, but I have no clue about what is going on, is it inheriting the rights from NGINX or something?

This should do it. dolibarr-config.service should have always been a oneshot.

This should do it.

apparently the installer requires that conf.php exists, even if empty... man_shrugging
dolibarr-config.service should have always been a oneshot
we don't need to run dolibarr-config.service explicitly, though it might be a good idea to run it once more at the end of this test

Trying it now.

Nice, it is writable, and I'm running into stuff I can grok at:
machine #   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0[   11.382551] nginx[921]: 2022/09/04 00:01:44 [error] 921#921: *8 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Uncaught mysqli_sql_exception: No such file or directory in /nix/store/xy62178cl1ynk86qf2if97hcpr1nlygj-dolibarr-15.0.3/htdocs/core/db/mysqli.class.php:218
machine # 100  1218    0  1179  100    39   7315    241 --:--:-- --:--:-[   11.391545] nginx[921]: Stack trace:
machine # [   11.392370] nginx[921]: #0 /nix/store/xy62178cl1ynk86qf2if97hcpr1nlygj-dolibarr-15.0.3/htdocs/core/db/mysqli.class.php(218): mysqli->__construct()
machine # - --:--:--  7612
machine # [   11.398301] nginx[921]: #1 /nix/store/xy62178cl1ynk86qf2if97hcpr1nlygj-dolibarr-15.0.3/htdocs/core/db/mysqli.class.php(93): DoliDBMysqli->connect()
machine # [   11.410389] nginx[921]: #2 /nix/store/xy62178cl1ynk86qf2if97hcpr1nlygj-dolibarr-15.0.3/htdocs/core/lib/functions.lib.php(86): DoliDBMysqli->__construct()


Fix nil pointer error in entrystream [phab:D479]

Created by schroederc at 2015-08-31 16:44:02:

zarko accepted this revision at 2015-08-31 16:45:59

schroederc closed via commit d4a3a0da1a4ffced80da6e62045430217d885457 at 2015-08-31 16:48:02


rdkafka 0.14.0.rc1 update

Hey,
I just released https://rubygems.org/gems/rdkafka/versions/0.14.0.rc1
Just pinging so you guys can test it alongside of others :)
Will promote to stable once I get :green_circle: from some folks.
Once that's released, will release 0.15.0 with librdkafka 2.3.0 so we have 2.1, 2.2 and 2.3.

Closing as I already released it. Should release 0.15 in the upcoming week.


CU-1gexut1 Added NFS and Samba deployment automation

Task linked: CU-1gexut1 deploy)_home: NFS and Samba on Home Server


[FLINK-6256] Fix outputTag variable name in Side Output docs

@kl0u Is this the issue of the JIRA or was there something else?

@zentol Yes, this is the main issue, but you could also use the same name for the output tags between Java and Scala. Java uses "side-output" while Scala uses "string-side-output".

@kl0u done; i went with "side-output".


Define measurement mechanism for standards compliance, logging and reporting

From discussion raised in Slack by @willhiorns

How will we know whether a standard is being met?
I suspect that without a data push from within the delivery team we'll not be able to collate the information across the estate.
So the user of the standard needs to know what to do to meet the standard, and how to record whether it is met.  That log of "standards met" informs the registers and heatmaps.
I think it needs to be logged by the user for later reference because log events will be spread through time, we won't test and verify all standards at the moment of the assurance gate. The snapshot summary of current state is what gets governed.

We need to define an approach to this, preferably something straightforward that can be managed in a single piece of guidance/standard, depending on the needs of people that will consume registers or other aggregate reports.

This issue has been raised in internal Slack (or similar variant of), where a user wants to take the whole of SEGAS as document for dissemination/baselining (or some other purpose that has not yet been fully defined).
Think we might benefit from a downloadable table/document of standards and requirements that can be used as a checklist by teams, maybe principles and patterns too.
Possible that versioning that document would have benefit as well, could consider a release process to do that.


Connect up augmentations from macro output through to host.

Hook up macro augmentation output end to end.
Add the capability to reference the dart_model schema from the macro_service schema, because Augmentation is in dart_model.
The wire format still does not carry the actual request type because it's not needed yet, there are few enough request types to always "guess" correctly. It will need the request type very soon :)

Thanks, PTAL.


UMLS and/or MESH ids can't be resolved

Hi there,
According to your prefix list: https://nodenormalization-sri.renci.org/1.1/get_curie_prefixes
The system supports UMLS and MESH ids  to be resolved. I have a long list that were not resolved successfully with your service.
Please check:
https://github.com/r76941156/assorted_data/blob/main/example_missing_ids.csv
The source is from UMLS. So, perhaps you can update to the latest version? Or this is a different issue? Thanks.

Hi @r76941156  -
I took a quick look at the example_missing_ids link.   From what I saw, the first column (UMLS) was mapped in the normalizer, but the second (MESH) was not.  From your note above I'm not sure that's right though?
Also in glancing through, it seemed like the unmapped MESH terms were Proteins.  The issue here is that MESH is not an allowed namespace for proteins in the data model that we're using (the biolink model).
I've made a PR on that model (https://github.com/biolink/biolink-model/pull/856) to address.
Please let me know if I've missed your point.

OK, the model PR has been merged, so we should be able to merge MESH identifiers into protein soon.

Thanks @cbizon . I will provide a better missing mapping list later.

We currently have most of those UMLS codes on our dev instance (with some exceptions, e.g. UMLS:C3178991), but we don't have most of those MESH identifiers (with some exceptions, e.g. MESH:D000070, MESH:D000992). I can come up with a more detailed report if that would be useful; otherwise, I'll focus on trying to figure out why those MESH identifiers are missing as a first step.


Day and date values do not match when used with negative offset timezones

version: 2.3.4
Issue:
•	When the device is set in a negative offset time-zone, we observed that for any date the day is shown ahead of the actual day. i.e. 20 November 2021 is shown on Friday instead of Saturday.
•	For a positive offset time-zone, we observed that it is displayed correctly.
Here are two screen shots:

@Tanuja0049 Thanks for your report. It has been fixed in v2.3.6 now.


Compatibility problems with sub-plug-ins and cordova-plugin-firebase

Your Environment

Plugin version:
Platform:   Android
OS version:
Device manufacturer and model:

Running in Simulator:
Cordova version (cordova -v): 8.0.0
Cordova platform version (cordova platform ls):
Plugin configuration options:
Link to your project:

Context

Compatibility problems with sub-plug-ins and cordova-plugin-firebase
I don't know if it's the plug-in problem or the cordova-plugin-firebase problem. I should give that plug-in feedback and ask the author to fix it.
This causes gradle to fail,
Expected Behavior

Actual Behavior

Possible Fix

Steps to Reproduce

Context

Debug logs

`Discovered saved plugin "mx.ferreyra.callnumber". Adding it to the project
Plugin "call-number" already installed on android.
Preparing Firebase on Android
Checking Java JDK and Android SDK versions
ANDROID_SDK_ROOT=undefined (recommended setting)
ANDROID_HOME=D:\android sdk (DEPRECATED)
Subproject Path: CordovaLib
Subproject Path: app
Starting a Gradle Daemon, 1 incompatible and 1 stopped Daemons could not be reused, use --status for details

Configure project :app
WARNING: Configuration 'compile' is obsolete and has been replaced with 'implementation' and 'api'.
It will be removed at the end of 2018. For more information see: http://d.android.com/r/tools/update-dependency-configurations.html
registerResGeneratingTask is deprecated, use registerGeneratedResFolders(FileCollection)
WARNING: API 'variant.getAssemble()' is obsolete and has been replaced with 'variant.getAssembleProvider()'.
It will be removed at the end of 2019.
For more information, see https://d.android.com/r/tools/task-configuration-avoidance.
To determine what is calling variant.getAssemble(), use -Pandroid.debug.obsoleteApi=true on the command line to display a stack trace.
WARNING: API 'variantOutput.getProcessResources()' is obsolete and has been replaced with 'variantOutput.getProcessResourcesProvider()'.
It will be removed at the end of 2019.
For more information, see https://d.android.com/r/tools/task-configuration-avoidance.
To determine what is calling variantOutput.getProcessResources(), use -Pandroid.debug.obsoleteApi=true on the command line to display a stack trace.
WARNING: API 'variantOutput.getProcessManifest()' is obsolete and has been replaced with 'variantOutput.getProcessManifestProvider()'.
It will be removed at the end of 2019.
For more information, see https://d.android.com/r/tools/task-configuration-avoidance.
To determine what is calling variantOutput.getProcessManifest(), use -Pandroid.debug.obsoleteApi=true on the command line to display a stack trace.
WARNING: API 'variant.getMergeResources()' is obsolete and has been replaced with 'variant.getMergeResourcesProvider()'.
It will be removed at the end of 2019.
For more information, see https://d.android.com/r/tools/task-configuration-avoidance.
To determine what is calling variant.getMergeResources(), use -Pandroid.debug.obsoleteApi=true on the command line to display a stack trace.
WARNING: API 'variant.getMergeAssets()' is obsolete and has been replaced with 'variant.getMergeAssetsProvider()'.
It will be removed at the end of 2019.
For more information, see https://d.android.com/r/tools/task-configuration-avoidance.
To determine what is calling variant.getMergeAssets(), use -Pandroid.debug.obsoleteApi=true on the command line to display a stack trace.
WARNING: API 'variant.getPackageApplication()' is obsolete and has been replaced with 'variant.getPackageApplicationProvider()'.
It will be removed at the end of 2019.
For more information, see https://d.android.com/r/tools/task-configuration-avoidance.
To determine what is calling variant.getPackageApplication(), use -Pandroid.debug.obsoleteApi=true on the command line to display a stack trace.
WARNING: API 'variant.getExternalNativeBuildTasks()' is obsolete and has been replaced with 'variant.getExternalNativeBuildProviders()'.
It will be removed at the end of 2019.
For more information, see https://d.android.com/r/tools/task-configuration-avoidance.
To determine what is calling variant.getExternalNativeBuildTasks(), use -Pandroid.debug.obsoleteApi=true on the command line to display a stack trace.
registerResGeneratingTask is deprecated, use registerGeneratedResFolders(FileCollection)

Task :app:preBuild UP-TO-DATE
Task :CordovaLib:preBuild UP-TO-DATE
Task :CordovaLib:preDebugBuild UP-TO-DATE
Task :CordovaLib:checkDebugManifest UP-TO-DATE
Task :CordovaLib:processDebugManifest UP-TO-DATE
Task :app:preDebugBuild
Task :app:preDebugBuild
FAILURE: Build failed with an exception.

What went wrong:
Could not resolve all files for configuration ':app:debugCompileClasspath'.

Could not resolve com.android.support:support-v4:24.1.1+. FAILED

Required by:
project :app

Failed to list versions for com.android.support:support-v4.
Unable to load Maven meta-data from https://jcenter.bintray.com/com/android/support/support-v4/maven-metadata.xml.
Could not get resource 'https://jcenter.bintray.com/com/android/support/support-v4/maven-metadata.xml'.
Could not GET 'https://jcenter.bintray.com/com/android/support/support-v4/maven-metadata.xml'.
Connect to jcenter.bintray.com:443 [jcenter.bintray.com/<IP_ADDRESS>, jcenter.bintray.com/<IP_ADDRESS>] failed: Connection timed out: con
nect
Failed to list versions for com.android.support:support-v4.
Unable to load Maven meta-data from https://jcenter.bintray.com/com/android/support/support-v4/maven-metadata.xml.
Could not get resource 'https://jcenter.bintray.com/com/android/support/support-v4/maven-metadata.xml'.
Could not GET 'https://jcenter.bintray.com/com/android/support/support-v4/maven-metadata.xml'.
Connect to jcenter.bintray.com:443 [jcenter.bintray.com/<IP_ADDRESS>, jcenter.bintray.com/<IP_ADDRESS>] failed: Connection timed out: con
nect

Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

Get more help at https://help.gradle.org

BUILD FAILED in 4m 35s
3 actionable tasks: 1 executed, 2 up-to-date
E:\AuszDriverApp\platforms\android\gradlew: Command failed with exit code 1 Error output:
FAILURE: Build failed with an exception.

What went wrong:
Could not resolve all files for configuration ':app:debugCompileClasspath'.

Could not resolve com.android.support:support-v4:24.1.1+.
Required by:
project :app
Failed to list versions for com.android.support:support-v4.
Unable to load Maven meta-data from https://jcenter.bintray.com/com/android/support/support-v4/maven-metadata.xml.
Could not get resource 'https://jcenter.bintray.com/com/android/support/support-v4/maven-metadata.xml'.
Could not GET 'https://jcenter.bintray.com/com/android/support/support-v4/maven-metadata.xml'.
Connect to jcenter.bintray.com:443 [jcenter.bintray.com/<IP_ADDRESS>, jcenter.bintray.com/<IP_ADDRESS>] failed: Connection timed out: con
nect
Failed to list versions for com.android.support:support-v4.
Unable to load Maven meta-data from https://jcenter.bintray.com/com/android/support/support-v4/maven-metadata.xml.
Could not get resource 'https://jcenter.bintray.com/com/android/support/support-v4/maven-metadata.xml'.
Could not GET 'https://jcenter.bintray.com/com/android/support/support-v4/maven-metadata.xml'.
Connect to jcenter.bintray.com:443 [jcenter.bintray.com/<IP_ADDRESS>, jcenter.bintray.com/<IP_ADDRESS>] failed: Connection timed out: con
nect

Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

Get more help at https://help.gradle.org

BUILD FAILED in 4m 35s
`

@mauron85 @globules-io

# Project target. target=android-28 android.library.reference.1=CordovaLib android.library.reference.2=app cordova.system.library.1=com.android.support:support-v4:24.1.1+ cordova.system.library.2=com.android.support:support-v4:27.+ cordova.gradle.include.1=cordova-plugin-firebase/driver-build.gradle cordova.system.library.3=com.google.android.gms:play-services-tagmanager:+ cordova.system.library.4=com.google.firebase:firebase-core:+ cordova.system.library.5=com.google.firebase:firebase-messaging:+ cordova.system.library.6=com.google.firebase:firebase-config:+ cordova.system.library.7=com.google.firebase:firebase-perf:+ cordova.system.library.8=com.android.support:support-annotations:27.+ cordova.system.library.9=com.google.android.gms:play-services-location:11+ cordova.system.library.10=com.android.support:support-v4:26+ cordova.system.library.11=com.android.support:appcompat-v7:26+ cordova.gradle.include.2=cordova-plugin-mauron85-background-geolocation/driver-dependencies.gradle cordova.system.library.12=com.android.support:support-v4:28.+ cordova.system.library.13=com.android.support:appcompat-v7:28.+


Accumulate Net Weight of Approved Weight Ticket Set Docs

Description
Populate the Net Weight panel based on the OKed weight ticket set documents net weights
Setup
make server_run
make office_client_run

Have a PPM with a payment requested
Change status of a weight ticket set document to OK
Net weight panel on PPM tab should have the net weight of that weight ticket set document

Code Review Verification Steps

[ ] Request review from a member of a different team.
[ ] Have the Pivotal acceptance criteria been met for this change?

References

Pivotal story for this change

@jim @LeDeep Added a unit test for the calculations and did some minor refactoring by using constants for the move doc status and types instead of strings


Flake: cmake/build/Release/concurrent_connectivity_test.exe times out

Timeout
D1108 22:19:32.048000000  3944 test_config.cc:388] test slowdown factor: sanitizer=1, fixture=1, poller=1, total=1
D1108 22:19:32.048000000  3944 dns_resolver.cc:307] Using native dns resolver
D1108 22:19:32.048000000  3944 concurrent_connectivity_test.cc:179] Wave 1
I1108 22:19:33.095000000  5616 subchannel.cc:683] Connect failed: {"created":"@1510208373.095000000","description":"OS Error","file":"T:\src\github\grpc\workspace_c_windows_opt_native\src\core\lib\iomgr\tcp_client_windows.cc","file_line":106,"os_error":"No connection could be made because the target machine actively refused it.\r\n","syscall":"ConnectEx","wsa_error":10061}
D1108 22:19:33.501000000  3944 concurrent_connectivity_test.cc:189] Wave 2

https://sponge.corp.google.com/target?id=6e3c6569-a4a5-44a0-a35c-0030fda857f8&target=github/grpc/c_windows_opt_native&searchFor=&show=FAILED&sortBy=STATUS

I am still seeing this on Windows (both Debug and Release).
There are other concurrent_connnectivity_test failures on other platforms that might be related  but not necessarily the same: https://github.com/grpc/grpc/issues/13125

I'm unable to reproduce it on my desktop, but I'm able to reproduce on a Kokoro VM (it doesn't happen very often though). Still looking.

Based on historic data, we haven't seen this since Nov 30 2017.


Add davidkel to Fabric Smart Client and Fabric Token SDK

Please include:

GitHub ID to add: davidkel
Lab to be added to: Fabric Smart Client and Fabric Token SDK
Role:

Read
Triage
Write

Thanks :)

Invited @davidkel to the two lab teams.


Bump version to v1.4.0

Pull Request Test Coverage Report for Build 424

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 60.088%

Totals

Change from base Build 414:
0.0%

Covered Lines:
1647

Relevant Lines:
2741

💛  - Coveralls


Merge pull request #3 from PowerShell/dev

Merge pull request #96 from camiloborges/dev

Hi @camiloborges, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!
The agreement was validated by Microsoft and real humans are currently evaluating your PR.
TTYL, MSBOT;

sorry...


Add my name into preface

Please consider my name in the acknowlegement if I may. Thanks !

Job d2l-zh/PR-831/1 is complete.
Check the results at http://preview.d2l.ai/d2l-zh/PR-831/


Had to add system config to get production working.

package.json
system-config.js

@matthewp pretty sure this is closed.  Please re-open if I'm wrong.

Not totally fixed, I need to redo the envs config as described in #57 and then put the vdom config in can's package.json.

Fixed by #57


Advanced clipping support for SVG

In Macaw, clip is a Locus which is currently quite limited. In SVG you can combine paths, make transformation, etc. to define a clip. It's a difficult task to implement all possible path transformations, however we can do a trick for better SVG support.
We can create a custom Locus which will be wrapped for UIBezierPath and use its power for difficult cases.

Hi @ystrot,
May I know if there will be a substantial explanation on this feature when it has been released? Perhaps a simple example showcasing the clip methods that will be available would be good.
I have a particular layout I wish to be able to draw using this library which involves two circular clips to a pair of roundedRects. May I know if the coming update will be able to support this layout?

Thank you and the team for this amazing library.

Hi John,
Actually your task doesn't require advanced clipping mentioned in this task. All you need is clip your shapes with a Path created using set of lines and arcs. Something like that:
let clipPath = MoveTo(x: 50, y: 50)
    .lineTo(x: 200, y: 50)
    .arcTo(rx: 10, ry: 10, angle: .pi / 2, largeArc: false, sweep: true, x: 210, y: 60)
    .lineTo(x: 210, y: 200)
    .arcTo(rx: 10, ry: 10, angle: .pi, largeArc: false, sweep: false, x: 210, y: 220)
    ...
    .build()

Current clipping already good enough. In case of some edge cases other issues may be opened.


Update README.md

After quite a bit of time spent analyzing what might be wrong as I did everything from HotSwiftUI repo's readme (and it didn't work),  I have re-read Injection III repo's readme where I found as below:
| As an alternative, this code is available in the HotSwiftUI Swift Package though you would have to remember to load the iOSInjection.bundle separately by  using the .loadInjection()
When I added that to the end of the view, injection started working.

Hi, thanks for your PR and I'm sorry you lost a bit of time there. As an alternative I've pushed a commit https://github.com/johnno1962/HotSwiftUI/commit/4ec71ac178d8d87742196a92bfcbff438852e39a that makes HotSwiftUI load the injection bundle by default even if you only use .eraseToAnyView(). I can't remember why I had reservations about doing this before.