Identical data is loaded into every session wasting memory

Describe the bug
Each botocore Session creates its own instance of a Loader within which JSON content loaded from botocore/data/ is cached by @instance_cache e.g. on methods like load_service_model & load_data_with_path.
This caching applies to many things including loading endpoints.json into an EndpointResolver which happens in every session and results in approx 6 MB of memory allocation (to load details of all the HTTP endpoints for every region/partition for all 300+ AWS services).
The JSON files shipped with botocore presumably do not change on disk at runtime. Nevertheless if you create several sessions within a process - e.g. in a multi-threaded app because sessions are not thread safe - this exact same data is loaded into memory multiple times and cached separately in each Session's Loader and in its EndpointResolver.
It seems therefore like a bug (of the wasteful memory usage variety) that the immutable JSON cache is per-session rather than per-process. In a multi-threaded app in a resource-constrained environment every 6MB really adds up.
Expected Behavior
When creating a 2nd (and any subsequent) Session, the data which has already been loaded from endpoints.json should be re-used, quickly and without unnecessary extra memory allocation.
Current Behavior
Instead, each new session actually loads the whole thing in again resulting in another ~6 MB of memory usage each time, storing it in a new EndpointResolver (and Loader) with the new Session. (The same issue exists for other JSON data such as service definitions but I'm just focussing on the most common & most impactful example that I observed.)
Reproduction Steps
import boto3.session

# Note: in real usage the below would be in separate threads (otherwise we could just re-use the Session)
# but the threading code is omitted from this example for brevity and because it does not affect the repro

# In one thread
session = boto3.session.Session(region_name='us-east-1')  # +6mb
client = session.client('s3')  # (+5mb as it happens)
# do stuff with client

# In another thread
session = boto3.session.Session(region_name='us-east-1')  # +6mb again = REPRO: this shouldn't need to load endpoints.json again
client = session.client('someotherservice')
# do stuff with client

Possible Solution
One solution would be to make the Loader process-wide with suitable locking on state as necessary. I imagine the small extra overhead is more than paid for by the memory savings if many sessions/clients are created.
A more radical alternative would be for the pre-processing step that generates the botocore/data/ to spit out, instead of each JSON file, a python module (.py file) containing a dict with the same data. Then Loader doesn't have to load JSON, it just lazily imports the python files it needs and python's importlib gives you the process-wide sharing and thread safety for free. I imagine this would be a much more difficult change having seen the existence of things like CUSTOMER_DATA_PATH (~/.aws/models/) so it may not be feasible - but I've included it if nothing else for hypothetical comparison and to illustrate the principle of the problem.
Additional Information/Context
https://github.com/boto/boto3/issues/1670 is very related - this ticket is an attempt at a detailed description of why each session increases memory usage so much and how this might be avoided.
Any
SDK version used
botocore==1.33.1 boto3==1.33.1
Environment details (OS name and version, etc.)
Windows 10 Ubuntu WSL / same happens on Amazon Linux 2

Thanks so much for getting back to me @tim-finnigan. I have not tried that, I assumed Loader was not thread safe (otherwise why would each session need its own?) - before I do could you clarify a couple of things:

is Loader thread safe?
if so, is there any reason not to make this the default behaviour? (i.e. all sessions using the same 'data_loader' component, I guess unless they specify a non-default 'data_path')

To respond to some of your other points:

Another option is using a single session to create multiple clients which get passed to the other threads

Sadly this is not really an option - the app in question is a multi-threaded web server and it is not possible to predict in advance which boto3 clients any given thread might need, so because Session is not thread safe, each thread has to create its own session in order to create the client(s) it needs. I am already caching that session using threading.local() so that subsequent client creations in the same thread don't need to make another session.

The endpoints.json file itself is relatively small and only a small fraction of what’s causing the memory usage.

It is 781KB (only surpassed by a handful of the service definitions) however loading it into memory in python results in nearly 6 MB of memory allocation according to analysis using the Austin profiler e.g. in the memory allocation profile trace below where I did session = boto3.session.Session(region_name='us-east-1') and then client = session.client('s3') you see that within create_default_resolver where it loads endpoints.json) there is 5.86 MB of memory allocation, which is quite a large fraction of the total memory allocated, the other major parts being smaller and s3 specific _load_service_model (4.71 MB) and _load_service_endpoints_ruleset (2.05 MB):

Have you tried sharing a single loader instance across several sessions? For example:
[snipped]

Found that each call of boto3.session.Session() (not using the default session) eat 200ms of wall clock time which lead me to this issue after I noticed all the stat/JSON parsing in my profiling; so not a RAM problem per se for me but a really expensive set up time.
The example was not exactly clear to me but did point me in the right direction on what I should be trying, so as a note for others, this got my ~200ms down to ~20ms per call to create and S3 Resource:
# preload and reuse the model to shave ~200ms each time we create a session
# https://github.com/boto/boto3/issues/1670
# https://github.com/boto/botocore/issues/3078
_loader = Loader()
# iterate contents of botocore/data/s3
for type_name in frozenset(['endpoint-rule-set-1', 'paginators-1', 'service-2', 'waiters-2']):
    _loader.load_service_model(service_name='s3', type_name=type_name)
# session *instantiation* is not safe either
_boto_session_lock = threading.Lock()
def _session():

    session = botocore_get_session()
    session.register_component('data_loader', _loader)
    with _boto_session_lock:
        return boto3.session.Session(region_name=region_name, botocore_session=session)

Then in your threads later you can use the following for a significantly faster setup time:
session = _session()
#session.events.register(...)
resource = session.resource('s3', config=config, endpoint_url=AWS_ENDPOINT_URL)


TASK LIST: Project 4 Ultradian analysis

Aggregate the interval data based on n-minute time intervals, return a time series.

def aggegate_interval(stain, mouse, feature, bin_width):
    """
    data loaded from data.load_intervals(feature)

    Parameters
    ---------------
    feature: {"AS", "F", "IS", "M_AS", "M_IS", "W"}
    bin_width: number of minutes of time interval for data aggregation
    start_time: time of day when system of sensors (F/W,etc) gets turned back on

    Returns
    ----------
    ts: pandas.tseries
        a pandas time series of length 12(day)\*24(hour)*60(minute)/n
    """

Detect period : Lomb-Scargle Algorithm

Reference
Comparison of Lomb-Scargle implementation in Python
Generalized vs Standard Lomb-Scargle in astroML package (with example of significance level calculation) [Another link]
(http://www.astroml.org/book_figures/chapter10/fig_LS_sg_comparison.html)
Example of Lomb-Scargle Algorithm in astroML package (with example of significance level calculation) Another link
Lomb-Scargle in scipy package

functions:

def plot_lomb_scargle(stain, mouse, day, feature, bin_width, start_time):
    """
    we detect ultradian period from 2 hours to 48 hours (to be decided)

    Returns
    ----------
    Periodograms: plot
        Lomb-Scargle Periodograms with horizontal line indicating
        significance level = 0.05, 0.01, 0.001, etc.
    significant period: numpy.array
        period in number of bin width
    """

Seasonal decomposition

@jarrodmillman @peterasujan @dylandaniels @shamindras We want to create a branch named ultradian under berkeley-stat222/mousestyles, we don't think we have access to do that. Could you please do that for us? Then we may push our changes on that branch.

@boyinggong - I'm not quite sure what you mean by not having >access to do that?
Generally the branch is created locally and pushed to your fork and then a pull request is submitted. And the branch is then created for you to make relevant changes based on code review. Sorry if I am misunderstanding your request.
I am on gitter at the moment if you want to clarify anything 😄


Make tests less flaky

Re-order logs with completing the task

@dotnet-bot test OSX 10.12 Debug Build

@halter73 merging this

post hoc :shipit:


isVisible doesn't check visibility due to CSS classes

What problem does this feature solve?
Making test for components like this
<template>

Header

        <button class="hidden">Test</button>

</template>

describe('Foo.vue', () => {
  test('Button', () => {
    const wrapper = shallowMount(Foo, {});
    console.log(wrapper.isVisible());
    const but = wrapper.find('button');
    console.log(window.getComputedStyle(but.element).display);
    console.log(but.isVisible());
  });
});

What does the proposed API look like?
Call of isVisible will be the same as before

This is not possible because of jsdom limitations as far as I know. I tried implementing this once. If you are interested in why, read a bit more here and see all the regular techniques for knowing is css is hiding an element here.
Normally you use things like getComputedStyle (as you suggested) or parentOffset, but since jsdom doesn't actually render anything, we cannot know if something is hidden or not based on css.
I'm sorry, for the limitations of jsom explained above, we will not be able to support this feature.

This appears to have been fixed in v2 with https://github.com/vuejs/test-utils/pull/454. Any chance of backporting to v1 for those of us still on Vue 2?

+1 to IGx89's sentiment

Sure thing, would you be interested in making a PR? If you do, please tag me in the PR, I can review it for you.

Sure can! Have unit tests passing, however it's failing when run in the browser using Karma -- breaks most isVisible tests. vue-test-utils v2 doesn't run tests in the browser so it doesn't have to deal with that issue. Will need to dig into it more to find the root cause -- sadly doesn't appear to be as simple as just copying the changes over.

Found the root cause, pretty interesting stuff :). PR submitted: https://github.com/vuejs/vue-test-utils/pull/2013.
Note that this will only work on attached elements, so the example above would need to have this change:
shallowMount(Foo, {
   attachTo: document.body
});

Hello,
I'm still getting this issue when I try to use isVisible when using a css class setting opacity to 0 after getting the 1.3.4 version.
Can it be due to sass ?

Getting the issue with V3, am using SASS as well.

It looks like opacity should work: https://github.com/vuejs/test-utils/pull/454/files
Please provide a minimal reproduction, that is the only way anyone can help.


Is there a way to export tracking data to a 3d software like 3d studio max and apply the tracking data to a 3d camera in the software?

Is there a way to export camera track data from the T265 to a 3d software package like 3d studio max of cinema 4d and apply the tracking data to a camera in the software?

I'd like to know this as well.  I was able to get the T265 ported into Unity and able to move a camera live within Unity, however, I wasn't able to record the camera position and keyframe it so that it would hold position.  I've been working on a different method by taking the CSV and stripping all the info in it, but I can't get the bag file converter that is included with librealsense to work.

Interesting, I wonder what kind of programmer we’d need to figure this out. Seems the csv method should be simple.
Frank

On Jul 27, 2019, at 5:45 PM, Randy Mills<EMAIL_ADDRESS>wrote:
I'd like to know this as well. I was able to get the T265 ported into Unity and able to move a camera live within Unity, however, I wasn't able to record the camera position and keyframe it so that it would hold position. I've been working on a different method by taking the CSV and stripping all the info in it, but I can't get the bag file converter that is included with librealsense to work.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Hi @frankywashere
There are a few different examples in librealsense that might be able to help here rs-trajectory, tracking and depth (if you want to bring in a depth camera into the system).  Also the realsense-viewer has the capability of saving bag files and playback within the viewer itself.  Currently we also have APIs to export/save a map for T265 but we do not provide the specifics or details of the maps configuration.  You would only be able to load it back onto the T265 for use, not able to use a 3rd party tool to see the map.  Lastly as discussed, taking the pose data in a CSV and using a 3rd party tool to map it out.
If there is anything else you need on this topic, let us know.
Thanks

@RealSenseCustomerSupport How would you propose exporting the pose data in CSV format?  The included rs-convert tool is not working currently.  Do you have any suggestions?  Thank you!

@RealSenseCustomerSupport hi, I am looking for a way create a CSV file with per frame position and orientation data to import in a 3rd party software. can you help me with my quest?


Fixed typo in docs part 2

Fixed another typo in docs similar to:
https://github.com/rubocop-hq/rubocop/pull/9522
Currently broken

This is an auto-generated doc at release, so #9522 was fine, but it's a typo anyway :-)

Yeah, I was going to say the same. That's going to be auto-fixed with the next release.

Yep, I missed that this was in the .adoc file 🤦♂


docs: add reference docs and guides for custom resources

@levfishbluefish Updated the PR, please re-review, thanks!

@levfishbluefish Friendly ping.


struct performance very poor

# python 2.7 32bit, windows 10
import numpy as np # version 1.10.1
import numba # version 0.22.1

numba.jit(nopython=True)
def sum_basic_array(a):
    ret = 0
    for aa in a:
        ret += aa
    return ret

numba.jit(nopython=True)
def sum_struct_array(a):
    ret = 0
    for aa in a:
        ret += aa['sub_0'] # or equally slow aa[0]
    return ret

x_basic = np.arange(2000*3, dtype=np.uint32)
x_struct = x_basic.view(dtype=np.dtype([('sub_0', np.int32),
                                        ('sub_1', np.float64)]))

%timeit sum_basic_array(x_basic) # 1.5ms
%timeit sum_struct_array(x_struct) # 22.0 ms

Both functions in the above code loop over the same chunk of data in memory, with the sum_struct_array version actually having 1/3 of the number of iterations (and using 1/3 of the actual data, although that's unlikely to be of much help).  However the struct version takes about 15x longer.
Is there another syntax I should be using to achieve the same result or is this simply something that hasn't been properly optimized for?
Edit: shortly after posting this I realized that the answer is to do some manual loop-invariant code motion:
numba.jit(nopython=True)
def sum_struct_array2(a):
    ret = 0
    sub_0 = a['sub_0']
    for aa in sub_0:
        ret += aa
    return ret
%timeit sum_struct_array2(x_struct) # 0.4ms

So, this version is indeed about 3x faster than the basic sum (and 45x faster than the naive struct version).  I guess that's the approach I will use for now, but surely numba could do this optimization automatically?  If not, then it's not really that helpful offering to support struct array types because you might as well just require that the caller passes each sub-view as a separate argument.

Did you forget the @ before numba.jit or is it just a copy/paste bug?

Note that I can't reproduce here: sum_struct_array is actually faster, probably due to the fact there are less array elements to sum.

OOOPS!

  I've updated the title and the description.


Ensure V4 to V4++ (Composer) compatibility

Ensure that the migration+integration path from V4 to Composer compatible productivity is smooth, easy, and fully supported.
Consideration: Team that has invested in developing substantial bot assets in V4 (or just completed migration from V3) would like to leverage the composer capabilities for all future development.
They would need a supported one swing "migration" to convert their assets to composer compatible. At minimum they'll need an easy integration path.

@scheyal I'm going to close this as it's likely an ask on you, and I don't think you use Composer to track your work. Please check-in with me if I'm unclear here.


Way to restart training from a certain point

Hi,
Ive ran tis_transformer command for a few day but the training got interrupted at fold 2 epoch 6 after 5 day (time limit of my HPC job), is there a way to restart the program to continue from that point or at least a way to restart from the beginning of fold2 without redoing fold 1 and 0
Ive tried setting the variable transfer checkpoint and settign transfer checkpoint and forcing --train --val and --test but in both case it seems to restart from 0
Thanks

Hey Nicolas,
I'm sorry, this feature is not supported right now, which I understand is problematic for people running everything on an HPC infrastructure, given the long training times.
I'll leave this issue open and hope to look at this in the near future. Ideally every model can be trained separately in parallel instead of sequentially. Similar to how RiboTIE allows optimization of separate datasets.

Alright thanks,
I might implement one when i have the time into a fork, if you want once its done i can do a pull request
I've run out of free time recently so it might be a while
Cheers
Nicolas

Of course.


set LC_ALL=C in all the scripts, to prevent locale issues

Description

Type:  Enhancement
Priority: Minor

Enhancement
Justification - why does the library need this feature?
As fixed and suggested in #1056 , it is better to set LC_ALL=C in all the scripts, for preventive issues, to avoid locale problems.
More information on LC_ALL can be found here

ARM Internal Ref: IOTSSL-1665

Looking quickly through the scripts, I see:

Dependency on collation order in generate-code.pl, affecting the output which needs to match exactly so it's a bad one. Solved by #1056.
Dependency on collation order in check-names.sh and the list-*.sh scripts. This affects the output of list-*, but for check-names.sh itself, it's only internal, so not important. Besides we're planning to replace check-names.sh, so no action needed.
Use of a-z to mean “all lowercase letters” (or likewise A-Z) in several scripts: compat.sh, check-names.sh and lits-*.sh. In check-names.sh this affects the behavior but is low-priority. For compat.sh, there's another usage problem of tr that needs to be fixed.


New 3rd party integration for winit, ash and vk_mem!

I publish a new integration, egui_winit_ash_vk_mem!
https://github.com/MatchaChoco010/egui_winit_ash_vk_mem
https://crates.io/crates/egui_winit_ash_vk_mem
I'll keep updating it, so I hope you'll add it to the Readme.
At the end, I would like to thank you for developing this wonderful library called egui.
Thank you.

Very cool, thank you! I added a link under https://github.com/emilk/egui#integrations


Update fs2-core to 3.1.5

Updates co.fs2:fs2-core from 3.0.6 to 3.1.5.
GitHub Release Notes - Changelog - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "co.fs2", artifactId = "fs2-core" } ]

labels: library-update, semver-minor

Superseded by #110.


ports check broken?

⚠ Please verify that this question has NOT been raised before.

[X] I checked and didn't find similar issue

🛡 Security Policy

[X] I agree to have read this project Security Policy

📝 Describe your problem
I have two port check items and both don't work (8920, 8096)
I checked the ports via https://portchecker.co/check-it and it correctly reported them as open.
Maybe a regression.
📝 Error Message(s) or Log
none
🐻 Uptime-Kuma Version
1.23.11
💻 Operating System and Arch
Ubuntu 22.04
🌐 Browser
Chrome
🐋 Docker Version
n/a
🟩 NodeJS Version
n/a

My URL is external and HTTPS Emby server

If the url is external, could you share the ip/port of said service?

If the url is external, could you share the ip/port of said service?

Let me look for a public server as an example (I'd like to avoid sharing mine publicly)

You can share it to me via my email adress if you'd prefer:

Check you email 😉

Sorry, but I cannot reproduce said ip responding to a tcp connection.

I think you are misunderstanding something:

TCP port is the equivalent of telnet IP PORT
https://portchecker.co/ checks for "open ports".

it is unclear to me what mechanism they use, to determine this, best I can think of is

@CommanderStorm
I am blocking non-US inbound traffic and that's why you could not telnet to it.
I disabled the block now.  Can you retry?
TIA

Pls paste your setup of this ping

If there is a field that needs better UX, feedback is appreciated

It's odd.  It used to work fine for me and then stopped and still does not.
I guess it's some combination of odd things.
What is the next way to get more detail logs from uptime kuma to troubleshoot?

If there is a field that needs better UX, feedback is appreciated

please remove IP !

used to work fine for me and then stopped and still does not

Software normally does not stop working without external effects ("admins/users mucking with it"). => Bisect the changes you did when the changes occured.

What is the next way to get more detail logs from uptime kuma to troubleshoot

You can exec inside the container and then progressively go outward.
We have documented how to do this partially in https://github.com/louislam/uptime-kuma/wiki/Troubleshooting#uptime-kuma-reports-down-but-the-service-can-be-accessed
It works on https://uptime.kuma.pet/ as expected.

used to work fine for me and then stopped and still does not

Software normally does not stop working without external effects ("admins/users mucking with it"). => Bisect the changes you did when the changes occured.

What is the next way to get more detail logs from uptime kuma to troubleshoot

You can exec inside the container and then progressively go outward. We have documented how to do this partially in https://github.com/louislam/uptime-kuma/wiki/Troubleshooting#uptime-kuma-reports-down-but-the-service-can-be-accessed
It works on https://uptime.kuma.pet/ as expected.

Not for me:

Tried on my network and on a different network

I restored an old version of UK from last November with the hope it would work, but no it shows the same problem.
Very interesting...

@CommanderStorm how can you check the port on the same node where UK runs via the command line tool?
Cam you try it for me pls?
Thx

I am blocking non-US inbound traffic

Note that you likely re-enabled this block. Remember that I am doing this from a residential, German internet access.

I don't run docker 😞
Anything else you can use?

Try any external port pls

I don't know what you mean by any external port. I tried this already as noted above:
The ip+port you specified are external to me, and they don't want to connect to me.
Here is what telnet should look like if a website wants to connect via TCP:

I don't run docker 😞 Anything else you can use?

You are likely searching at the wrong spot.
From my perspective, your service is either geo-locked, defective or not at said ip + port any more.

I don't run docker 😞 Anything else you can use?

You are likely searching at the wrong spot. From my perspective, your service is either geo-locked, defective or not at said ip + port any more.

My ports are definitely open.  Yes I block all traffic from non-US.
I suspect the issue is the way UK was deployed via https://tteck.github.io/Proxmox/
Sorry for being unclear.
All I need help with is to check a port via the command line (on Debian), so I can compare.
I bet the port won't check as open from the same node as UK is installed.
By "any" port I meant it does not have to be my port :)

Yup confirmed - the port is not checkable.
Why - who knows?!
Workaround - use local LAN IP address instead of an external host name


Fix the jenkins.sh script for a persistent workspace

Now that the Jenkins job definition has been changed in govuk-puppet
to not wipe the workspace, this directory needs deleting so that it
can be cloned.
It's probably possible to do this more efficiently, but this is a
simple way of making it work again.

This relates to the change here https://github.com/alphagov/govuk-puppet/pull/8773


OSX: Add support for enable / disable autofocus

getAutoProperty() and setAutoProperty() for CT_FOCUS_AUTO_CONTROL on OSX wasn't implemented.
Tested with a Logitech C930e on OSX 10.13 with QTCaptureTest.

Thanks, looks good! 👍

@trcwm well, that was quick, thanks!
What would be necessary to release a new version? I would like to get rid of my hacky local library setup.

I'll tag it for release tomorrow - it'll then get integrated into Jason's experimental build, if I'm not mistaken.


Add line numbers for functions

Line and column numbers should be added to the final JSONOutput in the field fn_lc.

Addressed in #21.


Write a test driver for opaque signatures without storage

Write a test opaque driver conforming to the interface defined in #3313 that supports the following functionality with at least one signature algorithm: key import, key generation, obtaining the public key, signing a hash, verifying a hash. This test driver will use Mbed TLS functions to perform the calculation.
The test driver will encode the key and let the core store the encoded key. The encoding can be anything as long as it's sufficiently distinct from the representation of transparent keys that confusion would be detected, for example a fixed header followed by rot13 of the transparent key representation.
Write tests that validate the interface (including some correct and error cases). Add a component to tests/scripts/all.sh that runs tests using this driver using the build framework provided by #3325.
Volatile keys are out of scope of this task.

I would like to assign this task to myself.
-Kjell Eirik

@kjeir Done


Create some form of MultiError type

Note: the scope of this issue has tightened, see https://github.com/pkg/errors/issues/15#issuecomment-217587756
This is an issue for discussion of a potential MutliError (or other name...) type API that allows the accumulation of multiple errors in a collection which still implements the error interface but allows retrieval of the individual errors.
There are a few other options out there: https://godoc.org/?q=multierror but since github.com/pkg/errors already implements, in my opinion, the canonical best practices for error handling in Go it would be be nice if it included this feature.

While this is definitely a constant problem in my experience... I'm not sure canonicalizing it here is super necessary.
Here's my example: https://play.golang.org/p/B9_v2E4Rrv
The only part that requires any logic is how you concatenate the error strings, which seems like the kind of bike-sheddy thing no one's going to agree on, so everyone will just write their own.
What would be nice is to canonicalize the interface for getting back the list of underlying errors.  This is actually true of all the common error handling interfaces for Dave's "check by behavior" - IsTemporary() bool etc - if they are dictated by the stdlib, a la io.Reader, then everyone will use them and we'll all be singing Happy Days.  If they're not, we'll have a hodgepodge and interoperability will still be a problem.

Thanks for raising this issue. I think there are several things going on here. The first is an error which represents a set of errors, aka, something like
var errs []error
for _, x := range things {
        err := x.Do()
        if err != nil {
              errs = append(errs, err)
        }
}
return errs

So some kind of errors.Wrap(errs...), thing. If this is what you're talking about then I'd like to take a pass. I think multierror is too complicated. Ie, if an error does not occur, then the number of errors returned will not match the number of operations attempted. So, does multierror track errors along with operations, or does it track only errors and you have to figure out how to tie them back to the operation that failed. This ambiguity makes me want to not touch this.
With that said, perhaps there is cause for another interface, like Cause() error, possibly Causes() []error for an error which is caused by multiple events. I dunno, It doesn't seem to solve the question above.
The other kind of multierror could be something like this
rc := SomeReadCloser()
_, err := rc.Read(...)
if err != nil {
     err2 := rc.Close()
     if err2 != nil {
           // what do we do here ?
     }
     return err
}
...

In this example, which we hit a few times in Juju the operation failed, then the cleanup operation failed -- now we have two errors that are related, call them a sequence, rather than a set (above), which one do return?
If we choose err then possibly we're ignoring the more serious, or more detailed error, returned from close. If we choose err2 then the caller sees that the "close" operation failed, but not the actual "read" which was the first failure.
At the moment i've seen this handled in a number of ways, none very good. The first one is to return the first error and log the second barf. The other is to return err.Error()+":"+err2.Error() or something.
This is a problem I think the errors package should try to solve, I'm not sure if that would solve the other use case at the same time.

I have found a collection of errors useful when doing input validation where the goal is to report all problems with the data rather than just the first problem. I am not sure that this use case is related closely enough to this package to make it a good fit though.

@ChrisHines exploring this idea a bit further, say we have
var errs []error = ... // from somewhere !?
err := errors.Wrap(errs...)

What does
errors.Cause(err)

return ?
We could make some kind of MultiError structure, so the underlying cause is something like
type Multierror []error

But that doens't seem like something that needs to be added to the errors package.

It's never popular to cite Java for prior art, but Java 7 added to class Throwable the getSuppressed() method. The motivation for this method and the broader concept of suppressed exceptions was to accommodate these failed cleanup attempts that occur in the finally block of a try-with-resources statement.
Translating your example above, in Java you'd get err thrown from the method, with err2 exposed as the only element in the array of err's suppressed exceptions.

I agree that this use case is orthogonal to the Wrap/Cause feature of this package. I guess it depends what you want the scope of pkg/errors to be. The package docs imply a general purpose set of error helpers.

Package errors implements functions for manipulating errors.

So although type Multierror []error is orthogonal to Wrap/Cause, maybe it belongs in pkg/errors if it adds another useful way to manipulate errors. Or perhaps pkg/errors should stay one dimensional (often a good thing) and the package docs should say something like "Package errors implements functions for creating and inspecting contextual errors."

@seh this is an interesting point. I think there are two ideas here, but first some background.
err := errors.Wrap(err, "oops")

We say that the error on the right is the 'cause', and the error on the left is a wrapper, or it wraps the cause. It gets a little more complicated if we have
err := errors.Wrap(errors.Wrap(errors.Wrap(err, "oh), "damn"), "blast)

So, the error in the middle is the cause, and the one on the left is a wrapper, so I guess that means all the others are wrappers, even if they are retrieved by the Cause behaviour.
So with that background, none of these wrappers are considered to be the cause of the error, they simply wrap the error, exploiting the error interface contract to add additional information.
What you're talking about is an error that is the cause, but also notes some other cause as related. So in the example I gave, and to make up a function name
if err := nil {
       err2 := thing.Close()
       if err2 := nil {
           err = errors.CausedBy(err2, err)
      }
      return err
}

This has a few problems. Firstly the name. Second, the arguments, they'll both be the same type so getting them in the wrong order is almost guaranteed; an api with a 50% chance getting it wrong is a bad idea. But I don't have any better ideas at the moment.

@ChrisHines i'm fine with a PR to change the comment on the package declaration.
I'm not sold that type Multierror []error is something that should be added until the result of passing it to errors.Cause is defined.

Java's Throwable has a complementary method addSuppressed(), for which the documentation is admirably thorough. This part is pertinent:

In the try-with-resources statement, when there are two such exceptions, the exception originating from the try block is propagated and the exception from the finally block is added to the list of exceptions suppressed by the exception from the try block. As an exception unwinds the stack, it can accumulate multiple suppressed exceptions.

An exception may have suppressed exceptions while also being caused by another exception. Whether or not an exception has a cause is semantically known at the time of its creation, unlike whether or not an exception will suppress other exceptions which is typically only determined after an exception is thrown.

Note that programmer written code is also able to take advantage of calling this method in situations where there are multiple sibling exceptions and only one can be propagated.

There they acknowledge a difference between accumulating siblings and causing those siblings to arise. The finally block would get executed whether the try block throws or not; that is, we're going to close the file whether or not we ran into an error reading from it once it was open. Hence, the read failure doesn't cause the failure to close the file. The read failure merely "dominates" the closing failure.
I see the quandary you face with parameter order; Java skirts that problem by dispatching via method on the accumulating exception instance. I'm not trying to sell you on accommodating this capability. I just want to point to other attempts at handling the same problem.

i'm fine with a PR to change the comment on the package declaration.

I would wait and see how the package evolves toward version 1.0. I was simply trying to gain some insight into the scope you have in mind for the package.

I'm not sold that type Multierror []error is something that should be added until the result of passing it to errors.Cause is defined.

Should Multierror implement errors.causer? My guess is probably not, in which case passing it to errors.Cause returns the Multierror itself.
However, my quick survey of the various multierrr packages indexed by godoc.org indicates that a type Multierror []error is probably just a collection of errors that implements the error interface with some opinion about formatting the aggregate error messages.
I have to agree with you, I don't see the value in adding it to pkg/errors yet.

Seems like there's two different types of multierrors - parallel and serial.  Parallel errors occur when you have one entrypoint that does N independent things, each of which can independently error or not, such as "read these 3 files".  Serial errors occur when one entrypoint has N errors related to different parts of the same logical operation, such as Dave's Read and Close errors.
Parallel errors really should just be a slice of errors - it's just one function call that returned N unrelated errors.  In that case, it's probably correct to correlate errors to input 1:1, which means some of the errors in the slice may be nil.  This is probably the less interesting of the two types... you should just be returning []error and not a single error as the signature of the function with this kind of failure is possible.
Serial errors are more interesting. They likely have one primary error, and some associated errors.  In this case, I think the main error may well have a cause (which is not any of the associated errors).  This type of multi error would then also have a slice of associated errors, each of which may also have a cause.
I'm not sure if the serial type of multierror should exist in this package or not.

@natefinch Re your first comment about a common interface to get the list of underlying errors, there's a Wrapper interface in the Hashicorp errwrap package:
https://github.com/hashicorp/errwrap/blob/master/errwrap.go#L48
While MultiError may not belong in this errors pkg, I think it's worth looking at the following for some ideas (though I don't agree with everything that happens in there either -- in the end it seems that everyone will cut & paste from multiple projects and make their own):
https://github.com/hashicorp/go-multierror
https://github.com/hashicorp/errwrap

@bits01 thanks for your reply. To make the scope of this issue clear, I do not want to add
 type MultiError []error

To this package -- it's not complicated enough to warrant someone having to depend on this package -- remember that any type which has an underlying type of []error can be converted to an []error so there is no need for everyone to agree on the name or location of that declaration.
wrt. the remaining bits in scope, @natefinch 's so named Serial error, I think this is something that the errors package should handle, assuming a non footgun API can be found for it.

over at https://github.com/contiv/errored we use this notion of .Combined
errored.Errorf("Error message").Combined(err)
(No, I am not here to promote my project, genuinely interested!)

@erikh thanks for your comment.
I've been thinking about this problem a lot over the last few weeks, and you're right that it really boils down to the API chosen for handling this. eg, it's either going to be
errors.CausedBy(err1, err2)

or your fluent suggestion of
errors.New("something happened).And(errors.New("this happened"))

To dispense with my former suggestion, I cannot bring myself to introduce an API that will have such a high rate of misuse -- which is more important, err1 or err2 ? Can you always remember the rule ? Even when this is the unlikely failure path. We actually have this in Juju's errors package, and it's used very infrequently.
So, to your suggestion. I'm not opposed to the fluent or builder pattern in principal, but it would mean that to make it work, New, Errorf, would have to return some concrete builder type. For the sake of argument that might be errors.ErrorBuilder (or errors.Error, but the former is clearer what is happening).
The problem I see with this approach is this pattern is the following code would not compile
err := errors.New("oops")
err = err.Add(errors.New("something else"))

The reason this does not compile is obvious, but will be a stone in the shoe of people who hit it. It would also mean that this code
err := errors.New("oops")
// something happens
err = errors.Wrap(err, "some more context")

Would also not compile. I see the latter as a common use case and that concerns me. It's not that either suggestions are bad or wrong, but they have some concerning usability issues.
So I think it might be time to step back and look at where I see this kind of function being used. Here is an example piece of code from Juju. It doesn't appear often, but always appears in this form
        if _, err := st.Model(); err != nil {
                if err := st.Close(); err != nil {
                        logger.Errorf("error closing state for unreadable model %s: %v", tag.Id(), err)
                }
                return nil, errors.Annotatef(err, "cannot read model %s", tag.Id())
        }

In other words, an error happened using a resource, then cleaning up the resource, a further error occurred.
Thinking about this some more I wonder if there is a logical error to assume anything about the state of st once the error occurred. That is, if st.Model failed, then one line of thinking would suggest that any other method on st will fail, including Close. In which case, what is the point of recording an error that is already known. The other line of thinking suggests that depending on how st is written, each operation is independent and a failure to call one method does not invalidate the object as a whole.
In summary, this pattern isn't as widespread as I first thought, and under investigation I'm not sure there is a one size fits all approach that works. @erikh i'd be interested in hearing your experiences as I am leaning towards closing without changing the errors package.

Sure. So, I can give you some concrete examples of how we use this;
perhaps it will help you.
We use it in this project: https://github.com/contiv/volplugin. Here are
a couple of chunks of code that use it.
https://github.com/contiv/volplugin/blob/master/volmaster/daemon.go#L428
https://github.com/contiv/volplugin/blob/master/volmaster/daemon.go#L622
Particularly, we wrap any error that is not an errored error with
.Combine. This way we get the basic functionality (stack traces and
some additional debug logging notably).
We also wrap errored errors occasionally to extend/augment the
debugging data, but this is less common, e.g. propagating up from the
storage driver(s).
We aren't really doing typed errors with errored though -- we still use
errors for that most of the time. Typed errors aren't very elegant for
the kind of diagnostics we're attempting to get out of the system.
Let me know if you want more, I'm sure I can find something.
-Erik

@erikh maybe we're talking about different things, this line from your sample
return errored.Errorf("Clearing volume records for %q", vc).Combine(err)

Would be written as
return errors.Wrapf(err, "Clearing volume records for %q", vc)
You're not combining two errors values, just wrapping an existing error with more context.

That's correct. We haven't integrated it but this is our solution for
multi-errors:
https://github.com/contiv/errored/blob/master/errored.go#L78
Sorry for the lack of clarity, I'm a little foggy. I'll follow up to any
replies tomorrow.
On 24 May 2016, at 1:14, Dave Cheney wrote:

@erikh maybe we're talking about different things, this line from your
sample
return errored.Errorf("Clearing volume records for %q",

vc).Combine(err)
Would be written as
return errors.Wrapf(err, "Clearing volume records for %q", vc)
You're not combining two errors values, just wrapping an existing
error with more context.

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/pkg/errors/issues/15#issuecomment-221197597

For what it's worth, our own error library has had a way to combine errors and it's been mostly helpful. https://godoc.org/github.com/spacemonkeygo/errors#ErrorGroup
The main downside is that, yeah, it makes it hard to programatically do anything with specific internal errors, but the times we've used it for a 90kloc project are in the high double digits, and the times we've needed to do anything with the errors are maaaaybe 1 or 2.

For things like form validation or other situations where lots of errors can occur, you always need to know failed and what didn't, and what 'things' the errors relate to.
In the past, I've used a map:
func ValidateForm(data map[string]interface{}) map[string]error {
  // validate each field and set errors keyed on the fields
}

This is nice because:

you can check the length of the map to see if any errors occurred,
range over it getting keys and errors, and
use native types (a map) without introducing anything new to learn.

errs := DoSomething(data)
if len(errs) > 0 {
  for field, err := range errs {
    log.Println("field", field, "invalid:", err)
  }
}

But this is different in every case, so in my view each codebase should solve it themselves in an appropriate way, and I'd keep it out of this package.

Another potential use case: start multiple goroutines to work on some task in parallel and then wait for them to finish. Some goroutines may return errors. It would be useful to take all the errors and package them all into a single error so it can be reported back as a single error while not losing details on each individual error.

All, please read the note at the top of the issue about scope, https://github.com/pkg/errors/issues/15#issuecomment-217587756

Specially, []error is out of scope. The remaining points are https://github.com/pkg/errors/issues/15#issuecomment-221194128

And thanks for your comments, please keep them coming.

Apologies - I was answering the MultiError case. @bits01 in that case, an error channel (<-chan error) has worked well for me in the past. Tweet me @matryer if you'd like me to explain more.

@erikh ping

Ah! Sorry I didn't get back to you. So, Combine(), Contains() both fill and access a []error and [][]string (stack traces). The struct is here: https://github.com/contiv/errored/blob/master/errored.go#L64-L74. So they are essentially "multi-errors" that evaluate to a lengthy combined error. There is also a pure errors generated error it keeps as well, for better or worse.
I'm not sure if that was the answer you were looking for or not. FWIW, I just peeked briefly at #34 and I will have to take a look at applying what zap is doing to errored too, we need this to improve our UX. There's a lot of matching functionality there that could be implemented.

@erikh thanks for your reply. I don't really grok fully what errored.Error is doing, it looks like a stack of errors, by some parts of the api look like a slice of errors. If it's the former then AFAICT we're doing the same thing, https://github.com/pkg/errors/issues/15#issuecomment-221197597. If it's the latter, a slice of errors that represent a set of errors that all occurred during a single operation, then that's out of scope for this issue.

yes. it is. Very sorry if I wasted too much of your time.

Not at all, it's not a waste of time to see how others have approached this problem. This package is very opinionated, it's never going to be possible to make it adapt to every requirement.

If this error package implemented hashicorp's WrappedErrors() []error style interface, the two could work nicely together, providing different styles of error wrapping, while still being able to represent the notion of wrapping consistently.
This would allow their multierror to play nice with this package, which could also be fit into those multi errors. The default wrapped error provided by hashicorp's errwrap package is much more simplistic than this package's approach to wrapping.
The nice thing about the errwrap package is that it isn't really one at all, it's just a convenience function justifying an interface

Would an errors.AndThen(err1, err2, "message") pattern work? This could return a new, 2-component error; first and second which could be printed out, similarly to a wrapped error but with the first's error stack followed by "and subsequently..." followed by the second stack.
The only issue I see with this is how Cause() would work.

@alanraison Sorry, but no. The reason for rejecting this idea is the same as declining errors.Wrap(err1, err2) -- there is no guideline as to which is the dominant error. This gives an API with a 50% chance of misuse.

@davecheney how about a simple AddSuppressed (or whatever you would like to name it) for the fundamental type?
func (f *fundamental) AddSuppressed(err error) *fundamental {
    f.suppressed = append(f.suppressed, err)
    return f
}

Pro:

Backwards compatible
Impossible to mistake parameter order

Con:

This method can only be used with errors created by err := errors.New("...")

What happens to suppressed errors? How are they recovered? Are they recoverable?

On 3 Dec 2017, at 01:22, max-rsYkRh<EMAIL_ADDRESS>wrote:
@davecheney how about a simple AddSuppressed (or whatever you would like to name it) for the fundamental type?
func (f *fundamental) AddSuppressed(err error) *fundamental {
f.suppressed = append(f.suppressed, err)
return f
}
Pro:
Backwards compatible
Impossible to mistake parameter order
Con:
This method can only be used with errors created by err := errors.New("...")
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

No less, one would have to interface assert the error to an AddSuppresseder in order to even access the method.
The answer here is not to expand the error interface or error library. If you are possibly going to return multiple errors, you should be returning a <-chan error, and for err := range errch it.
Best part of this chan error technique is that you never have to return a nil error, if you just close the channel, it will then start sending nil errors to all readers. So even if someone just uses return <-errch rather than ranging over it, it will return nil in a no error situation, and return only the first error put on the channel otherwise.

@davecheney i'm sorry to disturb you, (and i #15 (comment), it looks pretty correct, it's just that []error looks too small to be included in the package.
However, I find solution from hashicorp quite useful: it has convenient formatting, error checking, converting to normalized error, etc. etc. etc. What do you think about examining multierror package and trying to add something to this library? I'm ready to spend my time to suggest several PRs, because I'm absolutely sure: github.com/pkg/errors is the second most important package for error handling in the gophers community after stdlib errors, and the lack of a wrapper for multiple errors makes my life very difficult literally every day.


[DownloadBuildArtifactsV0] and [PublishBuildArtifactsV1] - workaround for '+' symbol issue under FF

Task name: DownloadBuildArtifactsV0 and PublishBuildArtifactsV1
Description: There is an issue, where customers cannot download part of the artifact if artifact contains '+' symbol in its name. Changes in encoding didn't work. It was decided to replace '+' symbol by its representation ' ' (space) in the artifactname variable. Changes were done both in DownloadBuildArtifactsV0 and PublishBuildArtifactsV1 to be coherent.
Documentation changes required: N
Added unit tests: N
Attached related issue: Link in the WI
Checklist:

[x] Task version was bumped - please check instruction how to do it
[x] Checked that applied changes work as expected

Testing:
Tested with different options of using and configuring the task:
publishLocation:

Container
FilePath

buildType:

current
specific

downloadType:

single
specific (multiple)

ForceEnableDownloadZipForCanary:

true
false

Agent type:

Self-hosted
MS-hosted

Agent OS:

Windows
MacOS
Ubuntu

Other:

Tried manually download part of artifact

Links to test pipeline runs:

https://abtttestorg.visualstudio.com/BuildArtifacts/_build/results?buildId=6460&view=results
https://abtttestorg.visualstudio.com/BuildArtifacts/_build/results?buildId=6461&view=results

Test pipeline runs:
Different options to download:

Download and publish to file path:


[Storage] remove the preview version tag from file-share NPM link

and from the npm install example

These files are not part of the package so should be okay to merge after release.

Merging as we are re-triggering the build pipeline and re-testing anyway


Fix implicit narrowing cast in foreachStatement

THIS IS IMPORTANT!
without this
sdc will segfault in test0077.d
which will break the new tester

done


Trying to use DropboxResponseError from transpiled TS code

Hey hi,
I'm using this client to handle OAuth2 for Dropbox in a dedicated internal API and for that purpose I need to map the lib's error codes to custom ones. My issue / question is about the DropboxResponseError class that is exposed as a .ts type but not exposed in the JavaScript .js index file.I'll explain my usage in the code below:
Version 10.x
TypeScript transpiled to Javascript
Linux
'use strict'

import * as r from 'ramda'
import { MyError } from './api-error'
import { MyErrorCodes } from './api-codes'
import { DropboxResponseError } from 'dropbox'

export function handleDropboxError(err: DropboxResponseError<any> | Error) {
    let errorCode = MyErrorCodes.UNKNOWN_ERROR
    const errorMessage = r.pathOr(
        'Unknown error',
        ['error', 'error_summary'],
        err
    )
    if (err instanceof DropboxResponseError && err.status === 401) {
        errorCode = MyErrorCodes.AUTH_ERROR
    }
    return new MyError(
        errorCode,
        errorMessage
    )
}

Here I'm trying to narrow the type of err to DropboxResponseError so that the err.status read is only performed when status exists, unfortunately I cannot use DropboxResponsError from the transpiled code as it is not exposed in the JS.
A workaround would be to use a type guard based on the property status presence in err, I like the instanceof as IMO it's easier to understand.
Is there a reason to the DropboxResponseError class not being exposed in the JS code ?
Thank you for your time and answers.

Thanks for the feedback! This was a bug on our end not exporting the DropboxResponse object and the DropboxResponseError object. I linked the PR that fixes this and should be resolved in v10.5.0! Please reopen if you continue having issues

Thank you for your reactivity! :champagne:


[Feat]: Form validation during registration

Thank you for your commitment to contributing your own ideas. Please complete the form below as accurately as possible to help us understand your request.
😒 Is your feature request related to a problem? Please describe.
It is currently possible to simply submit the registration form. If errors occur after submitting, a corresponding message is displayed, but it would make sense to validate the form on the front-end side.
🤔 Describe the solution you'd like
VeeValidate and Yup can be used to implement the validation, similar to how it is already integrated in the registration form.

We've created a FDFormBuilder which helps creating and validating Forms. This is now also used in the LoginView, but should also be implemented in the QuestionCreateView.


[PE-D][Tester C] edit command shows error when trying to edit 1 job listing to another job listing with the same title and description

When running edit 10 t/1 d/1, app shows error stating that changes did not modify original version even though title and description was changed.
Not sure if it is because there is another job listing with title 1, intended behaviour could be better explained.

However when running edit 10 t/2 d/1, the command is executed when it did not change the original value.

Labels: severity.Medium type.FunctionalityBug
original: randallnhr/ped#3

bug? if we change error message we will need to change the UG ones too


Start a Player at Fullscreenmode

I would like to have a version of Tooll3 as a standalone player. So that I can start in fullscreen mode for an installation without having to make any entries. That would be very helpful. Maybe you can also tell me where in the code I can find the appropriate settings so that I can program it myself. Many thanks.

Interesting. Can you elaborate on that? What to mean by " not any entries".
For an installation you should be able to export your project as an executable with the overhead of the editor... It will start in fullscreen mode by default. Also see: https://github.com/tooll3/t3/wiki/ExportingExecutables

Thank You very much thats what Iam looking for. Great.


Sync alarms in SECO

As for now we can get the android data using home assistance so i got idea to sync its alarm with Watcher as it has a buzzer and flashlight.


chore(package): update kronos-service to version 4.17.2

:tada: This PR is included in version 5.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Parsing HTML and then pretty printing it

I'm trying to parse some html text and then pretty print the entire document.  I couldn't tell what was the best way to traverse the hierarchy of nodes/elements and wasn't sure how to get the inner html content of a tag.  I'm posting here because I think this could improve the documentation for the API.
   let html = "**** put some html text here. ****"

       let doc = try HTMLDocument(string: html, encoding: NSUTF8StringEncoding)

        if let root = doc.root {
                let str = self.dumpElement(root)
                print(str)
        }

    func dumpElement(element:XMLElement) -> String {
        var str = ""

        str = "<\(element.tag!.uppercaseString)"
        for attr in element.attributes {
            str += " \(attr.0)=\(attr.1)"
        }
        str += ">"
        let nodes = element.childNodes(ofTypes: [.Text])
        for node in nodes {
            str += node.stringValue
        }

        for el in element.children {
            str += self.dumpElement(el)
        }
        str += "</\(element.tag!.uppercaseString)>"
        return str
    }

Is this correct?

What do you mean exactly by pretty print?
The code you shared actually does the following things:

Make the element's tag name uppercase
Ignored all child node that are not text nodes and elements(which includes CDATA, comment, etc.)
Dump all text nodes first, then elements after them, regardless of their order

So I don't really think it makes sense.

I guess you only want to pretty print some xml for while debugging? try the xmllint command.

How might you create the html String?  I'm using a separate class that inherits from NSObject to parse down a URL.
func httpGet(request: NSURLRequest!, callback: @escaping (String, String?) -> Void) {
var session = URLSession.shared
var task = session.dataTask(with: request as URLRequest){
(data, response, error) -> Void in
if error != nil {
callback("error", error?.localizedDescription)
} else {
var result = String(data: data!, encoding:
String.Encoding(rawValue: String.Encoding.ascii.rawValue))!
callback(result as String, nil)
}
}
task.resume()
}
in my ViewController I'm trying to :
let html = data
do {
let doc = try HTMLDocument(string: html, encoding: String.Encoding.utf8)
} catch {
}
but I get an error for the 'html' variable.
How do I set the html as a string from my initial URLrequest set up in my data service?

@hvtor you don't need to create the string if you have NSData. It is stated in the README that you can create a document with either a String, an NSData(Data for Swift 3), or [CChar] instance. Actually having a NSData instance is simpler since you don't have to specify the Encoding.
let doc = try HTMLDocument(data: data)

@cezheng Yes. Thank you. :) BTW, great documentation. Just a bit 😴 I guess.

@hvtor haha, it's true. Any suggestions on improving it?

No I meant I am 😴. It's great documentation. An example webpage would be great too! Showing how the elements can be mapped over. I'm not too familiar with
I'm trying to parse an IMDb list and it's a series of anchor tags. It's not clear to me how you select for specific tags.

Walking Dead
Stranger Things
goes to read the docs again
Parent. And then the child tags.


Revert back to small titles on mosaic view

@sreilly we would like to revert the mosaic view back to using small titles for the large stories. So, essentially making the large story's images slightly taller to line up the title box with that of small stories. Would this be an easy fix or do you think we should push this off to the 'iteration 1' milestone?
cc @chrismorast
(left: current, right: next)

should be a piece of cake!

@thm just to confirm, this was done earlier today and pushed to hockey.

Thanks, noticed it!


xpath 3.1

hello guys !!
PS as C# dont support xpath > 1.0 because it use old library and MS seem not upgrade here library. many Microsoft XML adopt LINQ to XML, which doesn't address the same problem-space.
now with xpath 3.1 we can query xml and also json easily and with powerfull mini-language.
its possible to adopt another library like saxon.net or others third party instead to old library ?

PowerShell usually follows .Net. So right quest is whether .Net support XPath 3.1?

@iSazonov
thanks
yes PS follows .NET but if .NET does not offer an official technology, it is not a reason not to implement it, especially if this technology is efficient. there are third party implementations of XPATH 3.1 like saxon.net and others.

I guess that the community is moving towards Json and no one wants to invest in XML.


OGG support

If possible adding OGG streaming support since it's what most talk streams and some music streams use

I tried to implement this, and doing so broke mp3 streaming, I may revisit this at a later time, but for now this isn't happening.


attribute constraint in application_information

for all attributes of application information have - minOccurs=0 in schema definition and table column definition is Null. Is it intended? or some constraint can be defined for this.

General comments on schema validation:
There are many styles of xsd and UML modeling in use in various information exchange domains. The ESPI standard schema was built with all optional fields. So, as we refined the xsd for ApplicationInformation and authorization we kept this model.
Clearly interoperability with all optional fields doesn't mean much. So, where are the actual interoperability agreements and how is it tested?
It turns out there are two types of content testing that need to be performed. The first is "type" testing where each element that is present is tested for its composition according to its type definition -- this is typically done with schema validation in XML documents.
The second is referential testing. That is, if the file has "this" it must also have "that" or "that" must contain a specific value(s). Such testing is not practical with schema validation alone. Hence, additional tools are necessary. Schematron, which is a "short hand" xml vocabulary for using the XPath standard query language for XML documents can be used for referential testing. The Green Button certification tests use this method extensively. Additionally, tests can simply verify the presence or absence of a list of required elements.
We use schema validation for type and multiplicity validation and use the other tools to go beyond that. This why the required is enforced in the implementation agreement and the test and not in the schema. To find the definitive definitions of interoperability for Green Button, the OpenADE task force is assembling GreenButtonTestPlan and corresponding GreenButtonTestCases.xlsm document (http://www.greenbuttondata.org/library/#testing).
Additional documentation hopefully will be made available in the future.
Recommend this issue be closed.

I find both the GreenButtonTestPlan and the GreenButtonTestCases.xlsm (referenced above) to be too opaque with respect to the documentation of the articulation of specific required elements.
Is there a way to make this information more accessible?  If not in the .xsd (where all might find it easily), then in a specific (maybe separate) document for ease of access?  I really would not know where to look in the two referenced docuemnts and, not knowing where to look, would quite likely not know what is, and is not, a required attribute of any of our ESPI resources.

Based on discussions on this week's OpenESPI call, there was a strong consensus that at least for ApplicationInformation and probably Authorization required fields should have a corresponding multiplicity of 1 in the schema. We will confirm on the OpenADE call tomorrow, January 6th, and accordingly we will make the update to the schema.
We will document the final set of required fields in this message thread before closing.

Updated schema commit 900d5a7079970481c7c2a9dcb20daa22d38b171a


Surfacing metrics faster than 10s

Describe the feature you'd like and the problem it will solve
I am diagnosing intermittent packetloss on an ethernet point to point link, and came across cloudprober which can send UDP and ICMP echo probes at a reasonably high rate. However the prometheus surfacer updates at a rate of 10s currently.
I saw sysvars_interval_msec which does increase the rate of sysvars, but it does not update the prometheus surfacer at a higher rate.
Description of what you want to happen and what problem will it solve for you.
I am looking to export /metrics at 1Hz so I can better diagnose this ethernet link going down intermittently (using a linkstate protocol called BFD, I do know that the loss of service on the link is larger than the BFD timeout which is currently at 2.0s). I would like to present the ethernet link provider with a clear graph that shows latency and loss at a per-second granularity.

Hello @pimvanpelt!
There is a probe level config option to control the frequency at which metrics are exported:
https://github.com/google/cloudprober/blob/3d5080b5dd0ee6a23395e6cf42a24c3e10557c2d/probes/proto/config.proto#L102
So your config will look like the following:
probe {
  name: ...
  type: ..

  # Run probe every 200 ms, with 100 ms timeout, and export results very 500ms.
  interval_msec: 200
  timeout_msec: 100
  stats_export_interval_msec: 500

  ping_probe {
  }
}

I think it should do what you're looking for.

Thanks @manugarg exactly what I was looking to achieve. Fixed a tiny broken window along the way, #512.


Selection after Reopen

Hi :-),
Is there any possibility to display the selected option after a selection when reopening the Selectbox ? Currently the tree is reset after a selection.
Thanks alot

$("#HeadingParentIdFilter").select2ToTree({ treeData: { dataArr: result }, maximumSelectionLength: 3 });
$("#select2-HeadingParentIdFilter-container").text("");
$("#HeadingParentIdFilter").val("")


Error Handling: Automate error reporting

We should consider making a unified Error() exception that encodes the following:

A stack trace to the point in the code where it occurred
A URL that, when clicked, will submit the stack trace anonymously to a bug reporter

The high-level goal is that if a user's app encounters a bug, and they look in the console and see this error, they'll see a helpful message saying something like "click here to report this bug to Blockstack", followed by a URL that encodes the error.
The URL should resolve to a form with a bug report, pre-filled with the data gathered from the stack trace.  The user would have a chance to consent to submit it.

Agreed. This can be really great for getting detailed error reports -- the idea being that we'd just prefill values using URL strings on the form: blockstack.org/bugs
A good place to start would be to look at how other frameworks have implemented this (in particular, I think there are tricks to making the console printout highly visible), and then try to emulate.

React does a pretty good job with error messages and warnings. I'm pretty sure if you just include a URL in the message, it gets formatted properly.
Here is an example of them creating an Error, and a warning.
I'm happy to help with this one.

Errors have been improved in https://github.com/blockstack/blockstack.js/pull/720
Although there isn't an automatically generated error reporting URL.


'WorkerThread' object has no attribute 'isAlive'

After updating to Python 3.9.5 and using Petastorm '0.11.1', I am getting an error 'WorkerThread' object has no attribute 'isAlive'. My code mostly follows the databricks example. Based off some other posts such as this, it looks like

It seems that thread's method isAlive() was renamed to is_alive() in python verson 3.9

Is there a temporary workaround?
Thanks
Full error:
AttributeError                            Traceback (most recent call last)
D:\Data\BayesNN.py in train_and_evaluate_SVI(self, svi, criterion, bs, ne, lr)
    141
--> 142                 train_loss = self.train_one_epoch_SVI(svi, train_dataloader_iter, steps_per_epoch, epoch, self.device)
    143

d:\data\venv\lib\site-packages\petastorm\spark\spark_dataset_converter.py in __exit__(self, exc_type, exc_value, exc_traceback)
    403         self.reader.stop()
--> 404         self.reader.join()
    405

d:\data\venv\lib\site-packages\petastorm\reader.py in join(self)
    658         """Joins all worker threads/processes. Will block until all worker workers have been fully terminated."""
--> 659         self._workers_pool.join()
    660

d:\data\venv\lib\site-packages\petastorm\workers_pool\thread_pool.py in join(self)
    186         for w in self._workers:
--> 187             if w.isAlive():
    188                 w.join()

AttributeError: 'WorkerThread' object has no attribute 'isAlive'

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
<timed eval> in <module>

D:\Data\BayesNN.py in train_and_evaluate_SVI(self, svi, criterion, bs, ne, lr)
    140                 if (epoch + 1) % 10 == 0: print('-' * 10)
    141
--> 142                 train_loss = self.train_one_epoch_SVI(svi, train_dataloader_iter, steps_per_epoch, epoch, self.device)
    143
    144                 #if (epoch + 1) % 10 == 0: val_loss = evaluate(model, criterion, val_dataloader_iter, validation_steps, device) #val_loss =

d:\data\venv\lib\site-packages\petastorm\spark\spark_dataset_converter.py in __exit__(self, exc_type, exc_value, exc_traceback)
    402     def __exit__(self, exc_type, exc_value, exc_traceback):
    403         self.reader.stop()
--> 404         self.reader.join()
    405
    406

d:\data\venv\lib\site-packages\petastorm\reader.py in join(self)
    657     def join(self):
    658         """Joins all worker threads/processes. Will block until all worker workers have been fully terminated."""
--> 659         self._workers_pool.join()
    660
    661     @property

d:\data\venv\lib\site-packages\petastorm\workers_pool\thread_pool.py in join(self)
    185         """Block until all workers are terminated."""
    186         for w in self._workers:
--> 187             if w.isAlive():
    188                 w.join()
    189

AttributeError: 'WorkerThread' object has no attribute 'isAlive'

I should note that downgrading to Python 3.8.10 works


arm 64 release include many amd64 bin, is it correct?

arm 64 release include many amd64 bin, is it correct?

The arm program packaged in ubuntu may not be correct, and it lacks equipment for compilation verification for the time being. Wait for everyone to test and ask questions, I will make changes. At present, it is difficult for one person to maintain it. I hope that some enthusiasts can participate in it one day.


Governance.getProposalStage returns incorrect data for half-way expired proposals

Expected Behavior
Governance.getProposalStage(id) should return Expired for dequeued proposals that return 'True' for: isDequeuedProposalExpired(id).
Current Behavior
For dequeued proposals, getProposalStage(id) just returns stage based on current time in the stages. So it will be incorrect for proposals that may not get approved or that may not pass.
While this doesn't seem to cause issues in Governance contract itself (since everything critical is manually checking isDequeuedProposalExpired), it causes unintentional mistakes in ContractKit, and it is also a potential time bomb waiting for someone to make a bigger mistake assuming getProposalStage returns correct data.

@yorhodes did you have a draft pr for this fix already?

@aslawson it's linked but I still want to add unit tests

PR is ready for review

@yorhodes is this "in review" or "closed"?

closed ;)


Parsers: Only partial test results shown

When running the test of a parser, the results are truncated.

Seems to have been corrected, watch to see if persists.


Update oci-java-sdk-objectstorage to 2.4.0

Updates com.oracle.oci.sdk:oci-java-sdk-objectstorage from 1.37.2 to 2.4.0.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.oracle.oci.sdk", artifactId = "oci-java-sdk-objectstorage" } ]

labels: library-update, semver-major

Superseded by #66.


Planning Module Configurations Apollo 5.0

Hello,
In planning/conf/Scenario, for example inside
bare_intersection_unprotected_config.pb.txt
In the beginning, I see the configuration values defined as
.................................................................................................................
scenario_type: BARE_INTERSECTION_UNPROTECTED
bare_intersection_unprotected_config: {
start_bare_intersection_scenario_distance: 25.0
enable_explicit_stop: false
min_pass_s_distance: 3.0
approach_speed_limit: 6.7056  # 15 mph
stop_distance: 0.5
stop_timeout_sec: 8.0
creep_timeout_sec: 10.0
}
......................................................................................................................
But my doubt here is, where are they used in the code? Can someone provide me the link of the code which uses these values?
Same doubt for all other config files inside planning/conf/Scenario

@ashwinsarvesh
The path of scenario conf file is defined by gflags:
https://github.com/ApolloAuto/apollo/blob/master/modules/planning/common/planning_gflags.h#L27
all the scenario conf files are assigned to scenario for loading during scenario registration method: https://github.com/ApolloAuto/apollo/blob/master/modules/planning/scenarios/scenario_manager.cc#L134
for this particular scenario conf, please see: https://github.com/ApolloAuto/apollo/blob/master/modules/planning/scenarios/bare_intersection/unprotected/bare_intersection_unprotected_scenario.cc#L46

Thank you


Remove disable mark related to 9.0.0

Closing in favor of https://github.com/IntersectMBO/cardano-node-tests/pull/2513


fix: added space between text in devision api

:tada: This issue has been resolved in version 1.4.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


shu codecov commit test

asdasdasd

Codecov Report

Merging #18 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #18   +/-   ##
=======================================
  Coverage   15.84%   15.84%
=======================================
  Files          16       16
  Lines         303      303
  Branches       37       37
=======================================
  Hits           48       48
  Misses        251      251
  Partials        4        4


/gallery/{folderid} is getting "all" instead

I tried, but is impossible.  I always get ALL the content, from all the folders

works for me


keybase login on linux not working

Can't login though the CLI. Running the latest client.
Username:      ***
Logged in:     no

Device:
    name:      ***
    ID:        ***
    status:    active

Session:       no session
    is valid:  no
    keys:      unlocked

Key status:
    stream:    not cached
    secret:    stored
    dev sig:   cached
    dev enc:   cached
    paper sig: not cached
    paper enc: not cached
    prompt:    show
    tsec:      not cached

When using the "keybase login" the prompt just returns without output:
$ keybase login
$ keybase device add
Starting `device add`...

(Please note that you should run `device add` on a computer that is
already registered with Keybase)
â¶ ERROR Login required
$

my log id: 6a9c11d7d7ee8b031905021c

thanks for the bug report, we'll look into it

in the meantime, do you know your password?  if so, do keybase logout and keybase login and this problem should go away...

Thanks, this solved the problem indeed!
Op do 12 apr. 2018 18:54 schreef Maxwell Krohn<EMAIL_ADDRESS>
in the meantime, do you know your password? if so, do keybase logout and keybase
login and this problem should go away...
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/keybase/client/issues/11376#issuecomment-380873737,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AWIG15pUNMvQTrQ1uz2-NI2v2O2jUo4Cks5tn4a4gaJpZM4TRT-Z
.


Parent dependency is resolved from cache after child has changed, instead of being recalculated

Description
In our project we're using dependencies that rely on other lower level dependencies. We're usually doing that by declaring @Dependency(\.child) property wrapper inside parent's implementation of liveValue.
But because parent's value is cached upon its first resolution -> no matter how we override the child's value, the parent will not be recalculated capturing its child's initial value.
This might produce unexpected side effects like in the example below.
Checklist

[X] I have determined whether this bug is also reproducible in a vanilla SwiftUI project.
[X] If possible, I've reproduced the issue using the main branch of this package.
[X] This issue hasn't been addressed in an existing GitHub issue or discussion.

Expected behavior
Parent.liveValue is recalculated after Child is overwritten.
Actual behavior
Parent.liveValue is resolved from cache capturing the initial Child value.
Steps to reproduce
// -------- Declaring dependencies

struct Parent: DependencyKey {
    let child: Child

    static var liveValue: Parent {
        @Dependency(Child.self)
        var child

        return Self(child: child)
    }
}

struct Child: DependencyKey {
    let id: String

    static let liveValue = Child(id: "Child")
}

// --------- Consuming dependencies

withDependencies {
    $0[Child.self] = Child(id: "Overriden Child")
} operation: {
    @Dependency(Parent.self)
    var parent

    print(parent.child.id) // "Overriden Child"
}

@Dependency(Parent.self)
var parent

print(parent.child.id) // Expected: "Child", Actual: "Overriden Child"

Dependencies version information
1.3.0
Destination operating system
iOS 17
Xcode version information
15.4

Hi @kirillsh, the behavior you have demonstrated above is to be expected. By doing this:
static var liveValue: Parent {
  @Dependency(Child.self) var child
  return Self(child: child)
}

…you have decided to resolve the Child dependency at this moment and pass a concrete version of the child to Parent. There will not be another resolution of this dependency at a later time, and so nothing can change its value. The Child object is held inside the Parent outside of the dependency system.
I'm not entirely sure what you are trying to accomplish in your real code. Perhaps you can share something more real world. But since this isn't an issue with the library I am going to convert it to a discussion, and please feel free to continue the conversation over there.


Fix minor alignment issue in Past Bronze Sponsors list

@DelazJ could you update your PR and I will merge it. Sorry.

@yjacolin No problem, it's already out of date and It isn't worth the fix.
@andreasneumann this PR was more about an issue on rendering of avioportolano icon (note the indentation) even though i fixed  table reordering.


Doesn't work for Windows Subsystem for Linux(WSL)

Hi,
If anyone has any idea how this library can be extended to work with WSL Ubuntu 20.04, that would be great. If someone does it, that's even better.

So do I, please give us some advice

So are you using it on WSL, or on windows to do work on the subsystem?
It's been a hot second, but if I remember correctly WSL is just the terminal (Has that changed?). If that is still the case I'm not sure I understand the use case.

Getting the same issue, using MobaXTerm and it says no .XAuthority file found. Creating this file under ~ will then run the code however it states that "Xlib.xauth: warning, no xauthority details available"

Following this guide helped. It works now https://micwan88.github.io/linux/xauth/2019/10/16/create-xauthority-manually.html

Following this guide helped. It works now https://micwan88.github.io/linux/xauth/2019/10/16/create-xauthority-manually.html

oh, thanks a lot, just help solve the problem

Following this guide helped. It works now https://micwan88.github.io/linux/xauth/2019/10/16/create-xauthority-manually.html

Oooh, this was great. Thank you very much  xD

Copied the manual over here just in case:
How to create .Xauthority manually
If .Xauthority does not exist in user home directory, it will show below error:
xauth
xauth:  file /root/.Xauthority does not exist
Using authority file /root/.Xauthority
Below are the steps to manually create the .Xauthority under user home directory
touch ~/.Xauthority

# Generate the magic cookie with 128 bit hex encoding
xauth add ${HOST}:0 . $(xxd -l 16 -p /dev/urandom)

# Verify the result and it shouldn't show any error
xauth list


microk8s dbctl backup Error

Summary

My env:

Microk8s 1.24
OS: Ubuntu 20.04.1 LTS
Kernel: 5.4.0-60-generic

I execute microk8s dbctl backup command. It print thie.
Backing up the datastore
INFO[0000] mode: backup-dqlite, endpoint: unix:///var/snap/microk8s/current/var/kubernetes/backend/kine.sock:12379, dir: /tmp/tmpg8rxtmqb/backup-2022-10-31-09-48-42
FATA[0012] couldn't create keyfile /tmp/tmpg8rxtmqb/backup-2022-10-31-09-48-42/508.key: open /tmp/tmpg8rxtmqb/backup-2022-10-31-09-48-42/508.key: too many open files
Backup process failed. 1

What Should Happen Instead?

Reproduction Steps

...
...

Introspection Report

Can you suggest a fix?

Are you interested in contributing with a fix?

Hi @logici, thank you for raising this issue.
The error message clearly indicates the problem, and that is the shortage of open files on your system. A workaround would be to increase this limit.
However, after taking a look at the code, it seems that we leave a few too many open files lying around, and https://github.com/canonical/go-migrator/pull/2/files should help with this.
After merging, the fix is typically released to 1.24/edge within a day, and then promoted to stable with the next minor release of Kubernetes.

Thanks for your reply.
And it means that I can't use microk8s1.24/stable with fixed backup tool. The fixed backup tool must be used in microk8s1.25/stable or microk8s1.24/edge. Is that so?

Hi, sorry for missing this.
Both 1.24/edge and 1.25/edge should get the fix (and, eventually, with the next Kubernetes patch release, it will move to 1.24/stable and 1.25/stable)

Thanks for your help.
And I will close this Issue.


https://mediafusion.fun host is down

yesterday the dns issue was preventing the url from being accessible. now the host seems to be down.

The server is unable to keep up on high load on peak time. The better alternative is to setup mediafusion in your local. The guide will be available on next release.

make few improvements further.

added hpa with cpu utilization based metric
use gunicorn with multi worker.
i can see its much better on peak time as of yesterday. so closing this as done


[RxMoya] Fix double response event in requestWithProgress.

As per #1094. This bug should be fixed, I've also added a test case to check correct number of events.

Codecov Report

Merging #1095 into master will increase coverage by 2.71%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1095      +/-   ##
==========================================
+ Coverage   73.79%   76.51%   +2.71%
==========================================
  Files          22       22
  Lines         729      728       -1
==========================================
+ Hits          538      557      +19
+ Misses        191      171      -20

Impacted Files
Coverage Δ

Sources/RxMoya/RxMoyaProvider.swift
94.59% <100%> (+52.48%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update caadf45...17e15a6. Read the comment docs.


PP-3707: Tagging script has been renamed

WHAT
See https://github.com/alphagov/pay-jenkins-library/pull/69/commits/50ac339ea3f832db53d7c0003f793c75c9c1add1
@oswaldquek

Approve/merge https://github.com/alphagov/pay-jenkins-library/pull/69 first!!!


bug: formatting in C# files do not work

Did you check docs and existing issues?

[X] I have read all the LazyVim docs
[X] I have searched the existing issues of LazyVim
[X] I have searched the existing issues of plugins related to this issue

Neovim version (nvim -v)
NVIM v0.9.4 Build type: Release LuaJIT 2.1.1699524327
Operating system/version
MacOS Ventura 13.6
Describe the bug
When I try to format a C# file, I get a lag for like half a second and then I get the message: Formatter 'csharpier' timeout.
When I execute the formatter on the command line (dotnet csharpier ..) it works without any issues.
Steps To Reproduce

Open a C# file
try to format

Expected Behavior
File should be formatted
Repro
no changes stock from starter repo

Did you install the omnisharp from LazyExtras?

I meant if you enabled the whole lang.omnisharp extra from LazyExtras. Just to clarify.

Yes, my LSP is working for C#.

Yes, it's enabled:

I don't know anything about C#, I just tried to confirm if the formatter worked or not. I installed dotnet for my distro (Linux Mint) and also enabled the extra lang.omnisharp and then created a project with dotnet new console -n test and the formatter works as it should (although I do observe about half a second delay for it to format, but I guess that's something related to csharpier itself). What is the output of LazyFormatInfo and ConformInfo?

Plz show output of LazyFormatInfo and ConformInfo. For me both show that csharpier is enabled for the buffer. Maybe something is wrong with your formatter?

LazyFormatInfo:

ConformInfo:

Tested on a new C# console project.

The output of the commands is the same for me as well. Unfortunately, I can't reproduce this. It works fine for me. Sorry I can't be of more help.

No worries, thanks for your time :)

Does :Mason offer an update for csharpier?
Perhaps your commandline version differs from the version Mason installed.

@abeldekat No, everything is up to date.

Which version of dotnet-sdk are you using?

This are my installed SDKs (arm64):

My projects for my work are using dotnet 6.

This are my installed SDKs (arm64):
My projects for my work are using dotnet 6.

I just tested again with dotnet-6.0. ConformInfo:

I tried reinstalling csharpier from Mason and it worked.

This are my installed SDKs (arm64):
My projects for my work are using dotnet 6.

I just tested again with dotnet-6.0. ConformInfo:
I tried reinstalling csharpier from Mason and it worked.

I had the same issue and reinstall helped me too, but I still get the timeout message in VIM but in the terminal it works.

Do :echo stdpath("data").

In which directory Mason installs the binaries?
....
$home/.config/local/share/nvim

That's a strange location given that your conform.log is in /Users/yourname/.local/state/nvim/conform.log.
I would expect the directory to be /Users/yourname/.local/share/nvim/mason/bin
In this comment you write:

I had the same issue and reinstall helped me too, but I still get the timeout message in VIM but in the terminal it works.

Am I correct to understand that the formatting now works in Neovim, but you still get the timeout message?
Would you be able to try using stock lazyvim starter, only adding the minimum required plugins, like lang.omnisharp?

Maybe try csharpier instead of omnisharp ?

In which directory Mason installs the binaries?
....
$home/.config/local/share/nvim

That's a strange location given that your conform.log is in /Users/yourname/.local/state/nvim/conform.log. I would expect the directory to be /Users/yourname/.local/share/nvim/mason/bin
In this comment you write:

Weird I didn't noted that, but I tried it again and the path was right this time:

I had the same issue and reinstall helped me too, but I still get the timeout message in VIM but in the terminal it works.

Am I correct to understand that the formatting now works in Neovim, but you still get the timeout message?

No, in that case there were an error when I executed csharpier in the terminal (a lib was missing), but after reinstall it started to work in the terminal, but not in VIM.

Would you be able to try using stock lazyvim starter, only adding the minimum required plugins, like lang.omnisharp?

I will try that.

Maybe try csharpier instead of omnisharp ?

I try it with csharpier..

So you have 2 versions of csharpier installed? One local which you installed yourself and one installed by Mason via the extra lang.omnisharp?

So you have 2 versions of csharpier installed? One local which you installed yourself and one installed by Mason via the extra lang.omnisharp?

That's right. I used dotnet tool install -g csharpier to install it for my system globally.

What does :!which csharpier say?

What does :!which csharpier say?

:!which csharpier
shell returned 1

I found a solution.
The issue was that conform does not properly pipe the buffer to stdin, because of that I was receiving the timeout message (default: 500ms).
I think there is a issue in comform, I will try to open a issue there.
My current workaround is to use the file path instead of the pipe:
  {
    "stevearc/conform.nvim",
    opts = {
      formatters_by_ft = {
        cs = { "csharpier" },
      },
      formatters = {
        csharpier = {
          command = "dotnet-csharpier",
          args = { "--write-stdout", "$FILENAME" },
        },
      },
    },
  },


Small fixes to detection of package installed nix and nixbld users

If a user installs nix via nix, then which nix will show that nix but not the system nix prevent the detection of nix.
And in environments with nss modules, nixbld users may not be in /etc/passwd or /etc/group, so do the detection of those the same ways apps do.

Thank you very much!


(Documentation) #Add Code of Conduct

Hey 🥳! I see that repository is missing code of conduct. I would like to add a code of conduct developed from scratch for your project. Please let me know if it is okay to proceed. Please assign to me and add GSSoC24 Ext & level1 Labels.

@KiranBaliga work on it

@Yashgabani845 Please review the PR and action accordingly.
Please add level1 and gssoc-ext to the linked PR as well.
Thank you!


Configure SAML is not working with DEX

Hi All,
I just configure dex to integrated with my corp's IDP service, through the saml connector example, seems not working right now. dex will redirect to IDP, but no login screen appears.
Backgroud:
Our IDP service is co-work with UAA and provide the SSO for Cloud Foundry platform, the configureation need to involves trust configuration between them which is work perfectly. I want to use dex to replace UAA part with the IDP to provide the auth for kubernetes cluster in my testing env.
The trust configuration in both UAA and IDP is achieved by importing a metadata.xml file, in UAA side, it will add following four parameters:
 - Single Sign-On URL (Redirect Binding)
 - Single Sign-On URL (Post Binding)
 - Single Logout URL (Redirect Binding)
 - Single Logout URL (Post Binding)

with same value: https://csc-devops-test.<IDP domain>/saml2/idp/sso/csc-devops-test.<IDP domain>
and plus an signning certificates of the IDP URL
in IDP part, sill several configuration is needed:

Field
Description
value

Name
The entity ID of the service provider.
csc-devops-test-pt1.canary

Assertion Consumer Service Endpoint
The SP's endpoint URL that receives the response with the SAML assertion from Identity Authentication.
https://csc-devops-test-pt1.<UAA domain>/saml/SSO/alias/csc-devops-test-pt1.canary

Single Logout Endpoint
The SP's endpoint URL that receives the logout response or request (for a multiple SPs scenario) from Identity Authentication for the termination of all current sessions.
it has two items configured,  both HTTP-Post and HTTP-Redirect are set to https://csc-devops-test-pt1.<UAA domain>/saml/SingleLogout/alias/csc-devops-test-pt1.canary,

Signing Certificate
A base64-encoded certificate used by the service provider to sign digitally SAML protocol messages sent to Identity Authentication.
certificates of the UAA url

Dex Configuraion:
as switched to DEX, thant is repllceing UAA with DEX to integrate with IDP service. I set  dex with following parameters:
    issuer: https://dex.<my domain>
    storage:
      type: etcd
      config:
        # list of etcd endpoints we should connect to
        endpoints:
          - https://<etcd addr>:2379
        namespace: dex
        ssl:
          caFile: /etc/etcd/etcd-ca
          keyFile: /etc/etcd/etcd-key
          certFile: /etc/etcd/etcd-cert
    web:
      http: <IP_ADDRESS>:5556
      #tlsCert: /etc/dex/tls/tls.crt
      #tlsKey: /etc/dex/tls/tls.key
    connectors:
    - type: saml
      # Required field for connector id.
      id: saml
      # Required field for connector name.
      name: SAML
      config:
        ssoURL: https://csc-devops-test.<IDP domain>/saml2/idp/sso/csc-devops-test.<IDP domain>
        ca: /tmp/idp-ca.pem
        redirectURI: https://dex.<my domain>/callback
        entityIssuer: https://dex.<my domain>/callback
        ssoIssuer: https://csc-devops-test.<IDP domain>/saml2/idp/sso/csc-devops-test.<IDP domain>
        usernameAttr: name
        emailAttr: mail
        groupsAttr: Groups # optional
        groupsDelim: ","
    nameIDPolicyFormat: persistent
    #oauth2:
      #skipApprovalScreen: true
    logger:
      level: debug
      format: text
    staticClients:
    - id: kubernetes
      redirectURIs:
      - 'http://<IP_ADDRESS>:5555/callback'
      name: 'kubernetes'
    enablePasswordDB: true

on the IDP side,the name is set to saml,   three URL I set is https://dex.<my domain>/callback, and import the dex certificates.
Problem
When I use example-app  to get the token, it will redirect to the URL https://csc-devops-test.<IDP domain>/saml2/idp/sso/csc-devops-test.<IDP domain>， but neither login screen appears nor redirect to https://dex.<my domain>/callback, just gave me an error

Can  any one help me to configure it?
mages.githubusercontent.com/10169236/34887793-b39256e8-f802-11e7-8db1-d4adfdc1ac83.png)
Can  any one help me to configure it?
The main problems is:
when login with example-app, via http://ip:5555/login, it will redirect to https://dex.<my domain>/auth?client_id=kubernetes&redirect_uri=http%3A%2F%2F<IP_ADDRESS>%3A5555%2Fcallback&response_type=code&scope=audience%3Aserver%3Aclient_id%3Akubernetes+openid+profile+email+offline_access&state=I+wish+to+wash+my+irish+wristwatch and then when I choose Login with SAML then redirect to https://dex.<my domain>/auth/saml?req=va46pbjbpmyt7e22v2wb4j7gz and then https://csc-devops-test.<IDP domain>/saml2/idp/sso/csc-devops-test.<IDP domain>. but for a usable redirect, it should be https://csc-devops-test.<IDP domain>/saml2/idp/sso/csc-devops-test.<IDP domain>?SAMLRequest=va46pbjbpmyt7e22v2wb4j7gz, append ?SAMLRequest=va46pbjbpmyt7e22v2wb4j7gz to the ssoURL, but actually not.
useable example of UAA quest:
https://csc-devops-test.<IDP domain>/saml2/idp/sso/csc-devops-test.<IDP domain>?SAMLRequest=jZJRT9swFIX%2FiuX3JE6adJnVFHWrEJVgRCTsYW%2B3tmktJXbwdarx73HTwpgYaK%2F2Oee7uucuLn73HTkoh9qaiqYxo0QZYaU2u4ret5dRSS%2BWC4S%2Bywa%2BGv3e3KnHUaEnwWiQn34qOjrDLaBGbqBXyL3gzermmmcx44Oz3grbUbJCVM4H1HdrcOyVa5Q7aKHu764ruvd%2BQJ4kAkWkeoxGlKq3EfoshgBWxmsBR3OMMMR7MBBbEyRgZCxsnxxHSZrmNoFOA%2F4rRgSPe6JkHebXZsp6jz3ihLCj8Zgz9h6RJVoOCaL9Hwcll9YJNS2uog%2FQoaJks64oFLn8Akrst6xkooBcyrwod8V2N58XX8sgwhoQ9UH9sSGOamPQg%2FEVzVhaRiyN0lnLMp6nnBXxbD77RUl9Xvc3bU41ftbN9iRCftW2dVTfNi0lP1%2FOIQjouXw%2B0d3b1j8Phpeq6fLDJhbJ2%2BzXM%2FsRwjbr2nZaPB3314P%2FmJXG6fSiZfQwSXlYvu5WUjqFSJPlGfL39S6fAQ%3D%3D&RelayState=cloudfoundry-uaa-sp&SigAlg=http%3A%2F%2Fwww.w3.org%2F2000%2F09%2Fxmldsig%23rsa-sha1&Signature=ZJSC2ngUWtpyIssJaemDxlD5JAnqhSBIDfb8yhaBsVMibr%2Fw4wWBswUqpwpIKzHtxAdhvPbBPbrH6gF%2BkUr6PdZ9s1KXy7ZYFnh0He6%2FqHB1hO24kFzl7%2BoW5AYdoI4J6vW2%2BMK9ELlF2B1qQZsL2cXyb8ZdDc1aVcSwgs9AiAb0p5gBE7pDiuETDI8GI65u9Ck2NljlLp%2BXhl9rjcq8UKKO7AJbc1RXKOwQiPYqSQLLrioooTLaxRJSlFTQNknx1jUXOXu7ZMF8I%2FFcj06vFMpnPNfhKsA07G0yCCyPCrnT0U4hxktcYMTFcINPs7gV4VKpjqRPc8%2F%2BOB5VyXwWrQ%3D%3D

/ help wanted

Hey @jenningsloy318
I see you're using on demand solution of SAP Cloud Identity, I had the same error which in fact tells nothing...
In my case this error occurred when :

I had wrong SP in AuthnRequest url (it has to match name in the SAML2 configuration of your application(Service Provider), you can check it in admin console of your tenant in SCP. I have multiple applications registered, therefore I have to add it as query param in url.

I did not have proper issuer in AuthnRequest - In dex configmap I had to add entityIssuer property with value of registered SP so it looks like entityIssuer: MY_SP_NAME. If you have your SP registered with name https://dex. I think you should change your entityIssuer from https://dex./callback to https://dex.

My working dex configuration with SCP IAS :
      config:         # Issuer for SAML Request         entityIssuer: dex.{{ .Values.global.domainName }}         # I have multiple SP registered so I have to specify which one to use in SCP IAS         ssoURL: https://{{ .Values.idp.tenant }}.{{ .Values.idp.domain}}/saml2/idp/sso?sp=dex.{{ .Values.global.domainName }}         ca: {{ .Values.idp.ca }}         redirectURI: https://dex.{{ .Values.global.domainName }}/callback         usernameAttr: first_name         emailAttr: mail         groupsAttr: groups
And I did not have to change bindings, works out of the box :)

— @DebugIT Thanks for info. I will test it after the vocation.

@DebugIT I tried your recommendations, but still got no luck. Can you advise if I'd make some changes on the IAS side ?


Get visualization auth tokens from DB

Closes #9308

Rake already ran, so all password protected visualizations have auth_token set. Please CR @gfiorav

📦 it for the 🆓 🌍 but only once 🍏