Fix TR NullPointerException in LetsEncryptDnsChallengeWatcher

What does this PR (Pull Request) do?
Fixes a NullPointerException in LetsEncryptDnsChallengeWatcher which I believe is breaking the Let's Encrypt functionality.
Because databasesDirectory is already a property of the extended class AbstractServiceUpdater, it should not be redeclared in LetsEncryptDnsChallengeWatcher. The Spring framework was setting the value in LetsEncryptDnsChallengeWatcher, but the AbstractServiceUpdater uses its own value which was null because setDatabasesDirectory was overridden by LetsEncryptDnsChallengeWatcher.
Also, log exceptions when catching them in order to capture the full stacktraces in the log.
Which Traffic Control components are affected by this PR?

Traffic Router

What is the best way to verify this PR?
Run TR, view the logs to ensure that the following NPE does not occur periodically:
2020-12-01 16:30:33,834 [ERROR] traffic_router.core.loc.AbstractServiceUpdater: [] Failed updating database!
java.lang.NullPointerException
	at java.nio.file.Files.provider(Files.java:97)
	at java.nio.file.Files.exists(Files.java:2385)
	at com.comcast.cdn.traffic_control.traffic_router.core.loc.AbstractServiceUpdater.updateDatabase(AbstractServiceUpdater.java:104)
	at com.comcast.cdn.traffic_control.traffic_router.core.loc.AbstractServiceUpdater$1.run(AbstractServiceUpdater.java:86)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

and instead see logs like this:
2020-12-01 16:33:06,704 [INFO ] traffic_router.core.util.Fetcher: GETing: https://trafficops.example.org/api/2.0/letsencrypt/dnsrecords/; timeout is 15000
2020-12-01 16:33:06,900 [INFO ] traffic_router.core.loc.AbstractServiceUpdater: [LetsEncryptDnsChallengeWatcher] database unchanged.

Verify the TR unit and integration tests pass.
If this is a bug fix, what versions of Traffic Control are affected?

master
5.0.x

The following criteria are ALL met by this PR

[ ] This PR includes tests OR I have explained why tests are unnecessary
[x] bug fix, no docs necessary
[x] unreleased bug, no changelog necessary
[x] This PR includes any and all required license headers
[x] This PR DOES NOT FIX A SERIOUS SECURITY VULNERABILITY (see the Apache Software Foundation's security guidelines for details)

I see, this PR partially reverses the changes in #5280, so the timeout is just #5069 again. So not actually a new problem.

So, if I understand correctly, #5280 "fixed" the external integration test timeout by causing a NullPointerException in the LetsEncryptDnsChallengeWatcher, and by fixing that, the external integration test still hangs again.
What's strange is that seems to be a CIAB issue, because when I run the integration/external tests locally (excluding that one failing RouterTest), they pass.


Add files via upload

Edited to remove CSV reader error (Windows)

Hi @sammiebuzzard, I can't check the diff because it's too big, and also don't have a Win machine to test. Should I just merge?

Hi again @sammiebuzzard,
After diffing locally it seems that your PR mainly fixes char encoding by doing
with open(CORD19_CSVFILE, newline='',encoding='Latin1') as csvfile:

Is that correct?
If the issue using the notebook on Windows persists, I suggest to create a new PR based on the current state of main, as this PR has fallen behind quite a bit now.
What do you think?

Sorry I missed this yesterday @sdruskat, the char encoding was the one remaining problem I found for Windows machines, (the other issues I had were local dependencies so nothing to fix there). Would be useful to have in if it doesn't break anything elsewhere- which I don't expect it should- so I'll submit a new PR for the latest version.


Error hook with vite buildEnd?

Describe the bug
Refer to https://github.com/vitejs/vite/discussions/8293#discussioncomment-2805229
I don't know how it works, but following the guide, I found that it only works in dev mode, so is this hook used correctly?
Using setTimeout is too brute force.😅
Reproduction
...
System Info
...

Used Package Manager
pnpm
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guide.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A, please open a GitHub Discussion instead.
[X] The provided reproduction is a minimal reproducible of the bug.

My mistake, sorry!


Discussion: Adding support for new features to MSIX manifests

Discussion: Adding support for new features to MSIX manifests
Currently, only Microsoft can add new features to the MSIX manifest schema, through extending the code and data inside AppxPackaging.dll. However, some users have asked for MSIX to be able to install more esoteric payloads, such as MMC snapins or Windows credential providers. Currently, the only way one could try to do this is to use customInstallActions and pack an MSI into the MSIX package. While this would work (assuming App Installer permits installation of apps that declare that rescap, which I am unsure of), IMO it goes against the design of MSIX. Furthermore, waiting for Microsoft to add new features into Windows core goes against the spirit of Project Reunion. 😄
One way new features in MSIX could be implemented in a backwards-compatible way would be to to invent a namespace not natively supported by Windows, and put custom features into that. Then users would add its prefix into IgnorableNamespaces so that Windows doesn’t complain about not recognizing it. Finally, the app would bundle a small component run using customInstallActions that parses the manifest file and processes the elements during the installation process. This small component should be extensible, so that third parties can add support for their specific needs by writing a plugin DLL, instead of having to fork and vendor the entire tool.
I have a good enough grasp of this process to start on a preliminary implementation in my own repo, but I don’t yet have enough detail for a formal Feature Proposal just yet.

Check out https://github.com/microsoft/msix-packaging - that's a great repo in which to try out some of these ideas before integrating into the overall product package manager.
What would you want to see in a Reunion-style package manager?
Your observation is correct - MSIX ensures that no (or very little, tightly controlled) app code is run during install and uninstall.  Those are tricky times and it's challenging to write code that runs correctly on half-staged / broken-needs-repair / being-uninstalled, along with the testing on all the various variants of systems.
MSI's custom actions ended up being very attractive and people built entire "sub installers" based on them.
What would be some rules for the kinds of code that apps can run during install & uninstall?

I’m not proposing that I add arbitrary code to run during install/uninstall; the use of customInstallActions is simply a necessary reality given how today’s system MSIX support is written. All the logic would be put in the AppxManifest.xml file, just as we do today for MS-supported installation features. I actually really like the design of MSIX as it is currently designed, including the declarative installation system; I just know that IT admins have been saying they want to make it able to install more kinds of software than it can today. (There was a post in the MSIX techcommunity outlining this issue, but I cannot seem to find it right now. I do remember it specified MMC and credential providers specifically, which is why I mentioned those in the OP.)
I understand that msix-packaging includes support for packing and unpacking MSIX files; it also includes MsixCore for use on downlevel OSes. However, MsixCore does not work on Windows 10 (it is programmed to always delegate to the OS if running on a recent enough build), so adding features there would seem to be a non-starter. How do you suggest I work on this, beyond simply opening design issues there? If that means forking MsixCore to make it work on Windows 10 and then support a superset of what AppxPackaging does, that’s fine by me. Thanks!

as the pm who put custom actions in MSI i can tell you it didn't work out how the 29 year old me expected :)
IT admins can run scripts with MSIX using the Package Support Framework (PSF) https://github.com/microsoft/MSIX-PackageSupportFramework/tree/master/samples/PSFScriptingSample allows you to run a Powershell onFirstRun, Run and exit
I would look at the PSFbut also feel free to provide feedback/code to MSIX Core so that we can determine how best to undock MSIX for you to use with Reunion

What are the next steps here? @wjk - were you planning to try out some ideas with the packaging repo @jonwis pointed to and submit a formal proposal, or work with MSIX Core per @aclinick's suggestion?

@stevewri Unfortunately, I don't quite understand what you’re asking. As I said above, the MSIX Core repo doesn't have the ability to change the OS implementation of the MSIX packager AFAIK. The package-support framework won’t work for my needs either. While I can run code on first-run to modify the system as needed, there is no way to ensure that the changes are then cleaned up on uninstall, as PSF only supports code running when the application exits. Dangling references to package files will still be a problem, as described in #97.
However, the implementation I described in the OP probably won’t work, as I remember reading somewhere that the customInstallActions rescap is protected by the system. Only apps specially blessed by Microsoft can successfully use it; random sideloaded packages will not install if they declare this capability. If I am mistaken here, I’d be glad to hear it. Thanks!

You're correct, customInstallActions requires a specific capability and approval as it explicitly breaks the "MSIX install/uninstall are programmatically driven" proposition.
The packaging system can trigger a background task when your app has been updated. Maybe we could define another background task - "you're being uninstalled" - that would let you do your cleanup? The main issue we have is preventing unbounded/uncontrolled work in servicing operations - Contoso shouldn't be able to camp in their background task for 20m and use that time to beg the user to cancel/reinstall.
For now, can you enumerate the features you'd want to see? The packaging team (hi @jvintzel and @aclinick !) monitors these threads and we're always looking for ways to make MSIX more useful, even if we can't deliver that as part of Project Reunion directly (yet.) :)

@jonwis An on-uninstallation background task would work quite well here. The specific scenarios I am interested in are as follows:

Unregister ETW manifests (#97). This may be better handled via a new, dedicated AppX manifest element rather than programmatic install/removal, but I don't know how difficult that is/how long it would take to implement. (Please see also https://github.com/MicrosoftDocs/winrt-related/issues/201, as while the serverpreview namespace has supported this task since it was introduced, Windows 10 did not appear to honor <serverpreview:EventProvider> elements in the AppX manifest when I tested it.)
Remove licensing information from shared user cache (cf #13), which may involve sending a "deactivation" signal to a license server so I can decrement the used-license count and allow installation elsewhere.

In terms of preventing abuse of this feature, I would recommend that on-uninstall background tasks be strictly forbidden to show UI of any kind to the user. (Enforcing this requirement when the task is running in full-trust, as would be required for ETW support, would be difficult.) To prevent apps from simply stalling indefinitely in their uninstall routines without showing UI, I would recommend that Windows display a dialog box stating "Please wait while Contoso is uninstalling..." if the background task doesn't return within x seconds. I don't think this dialog should allow for the uninstall process to be interrupted, as that may leave the system in a bad state that is difficult to recover from.
Finally, given that I am planning on storing licensing information in the Shared Local folder, I am wondering if it would be possible to add an element in the AppX manifest that would instruct Windows to automatically enable the Shared Local policy during install, obviating the need for the workaround described in https://github.com/microsoft/ProjectReunion/issues/13#issuecomment-632273438. Thanks!

@wjk An on-uninstallation background task would work quite well here
Do you have additional thoughts about such an uninstall task?
Execute it before uninstall?

what if it takes a while or hangs? Should it be time boxed? How big a box?  Terminated after timeout and proceed with the uninstall?
What if it crashes (or otherwise 'unexpected prematurely terminates')? Just one bite at the apple or retry N times? Can it fail and block/stop the uninstall?

Execute it after uninstall?

should it be time boxed?
what if the package is reinstalled while the uninstall task is running?
where do we find the bits if we uninstalled the package? Do we need to delay removing the disk files until the uninstall task completes?
does the uninstall task need to be an EXE linked with /SWAPRUN so it can be run just before we delete the files w/o being impacted (or impacting) those deletes?

No-UI is a good policy though it can be challenging to enforce, and there's plenty of non-UI potentially lengthy operations e.g. network operations.
Asynchronous is ideal but hard to see how Windows could be entirely fire-and-forget. Maybe fire-and-be-patient(-but-not-too-patient :P) ?
Best-effort vs must-succeed/can stop the uninstall? The latter poses obvious concerns. What would happen in your scenario if the uninstall task failed to complete? Under what conditions could you craft it to be OK if it doesn't complete successfully? How much latitude would you need to ensure it did its important work?
if it would be possible to add an element in the AppX manifest that would instruct Windows to automatically enable the Shared Local policy during install
An interesting idea. Or something equivalent e.g. the policy only applies to packages without <custom:Capability Name="sharedLocalFolderRegardlessOfPolicy">. +@jvintzel  for follow up

There is no current UX for MSIX uninstall and we should not add it - at least not this way.  The only visible behavior to the user is either Remove-AppxPackage take a while (there's lots of files and content to delete, processes to tear down) or the start menu item lingers around a while longer.  So no, app UX during uninstall should be prevented; I suspect our friends in lifecycle management can help with that.
The uninstall task should happen before final unregistration, as the task might need content from the Desktop Bridge registry & filesystem redirection in order to complete its work.  I suggest it be boxed, and we work with our friends in lifetime management to effectively enforce this boxing.  A failure to run the task is ignored and the uninstall proceeds. An app should not be allowed to hang around by crashing in its uninstall handler.

@jonwis @DrusTheAxe @wjk - nothing new here for several months. Let me try to sum up where we are so we can convert this into a proposal; let me know if I've missed anything:

We should support an on-uninstallation background task triggered by the packaging system (@jvintzel @aclinick)
Need help from lifecycle team (@andreww-msft) to timebox the task and to prevent it from creating UI
I'm unclear whether we need a similar on-installation background task as well?

@stevewri

We should support an on-uninstallation background task triggered by the packaging system (@jvintzel @aclinick)
Need help from lifecycle team (@andreww-msft) to timebox the task and to prevent it from creating UI
I'm unclear whether we need a similar on-installation background task as well?

#3 it's come up though unclear the priority - and restrictions.
We currently have ` but that's only available to preinstalled (and inbox) packages, it's not a general "on install" trigger. The windows.updateTask only fires on package update e.g. when registering foo-v2 when you have foo-v1 registered. It doesn't fire if you register foo-v2 when you have no foo package registered.
A generic windows.installTask poses questions regarding login performance. Is the perf impact and potential for abuse not the concern they once were? Can use be (reasonably) limited via custom capabilities?

@DrusTheAxe Your message is slightly garbled. As to perf impact, can't these be run asynchronously in the background and not block login? If the user tries to open an app that hasn't completed its on-install code yet, the user will get a progress dialog until then.
Also, since opening this issue I have found alternatives to on-install/uninstall tasks that serve the same function without needing inbox support. This issue is now personally almost zero priority. Thanks nevertheless!

@wjk  asynchronously - yes but you still have the ThunderingHerd(TM) problem. Login is a very busy period of time so we continually strive to shrink cpu/io/memory hit at login. The trick is to do so w/o limiting functionality. A tough act to juggle.


Payment fixes for frontend

sourceType, targetType changed from string to ParticipantType enum
include amountValue in PaymentLocal and PaymentDetailsLocal that is like amountDescription but without the + or -.
fix bug where target, targetType were empty when sending from keybase -> stellar.

cc @buoyad

CI failed because of flow...

Ok, @buoyad amountDescription no longer has +/-.  Please use delta.


Add link to github

Add a footer that links to this repo, so it can be easily found from the website.
Also extract the #history-div that is used by JS, so the footer does get in its way.
It looks like this:

I like the idea of linking to the GitHub repo, but having it under the tag list makes it look like it's part of the OSM data. Can it be moved up to the header? Maybe in the upper right corner?

On my screen (width) that would be cluttered and could be interpreted as part of the header.
I thought the footer-gray styling makes it different enough to not be confused as content.
But maybe an icon instead?


Add a Debian Slim variant

Firstly, I saw issue #78 and the reason to close it, however, the success of this library has preceded itself and other contributers are using it as a base template for their images. Like https://github.com/docker-library/golang.
At this point Golang is using the SCM variant of this set for their Debian builds and Alpine for their slim builds but it would be really nice to have a Debian slim build to be able to utilize the pro's of Debian with the reduced size of Alpine in, at least, that specific branch of the SCM tree.
Please reconsider adding a slim branch in this repository for a buster-(slim-)curl and a buster-(slim-)scm image.
I volunteer contribution by the way if it is required.

This repository currently has ~30 unique variations already -- IMO the value of having something that's "batteries included, but somewhat slimmer" is a bit too low for us to want to maintain such a thing (we already support far too much here :disappointed:).


Humio as storage backend

Humio is great at storing and querying large amount of data, but lacks visualization for the data. We therefore want to add visualization capabilities to Humio using Jaeger. @Xitric and I are working on a storage backend for Humio using the grpc plugin. Our project can be seen at https://github.com/em135/humio-jaeger-plugin.
We have implemented the spanstore.Reader and dependencystore.Reader interfaces. We are currently able to visualize traces by querying Humio. However, the visualization of dependency graphs are insufficient in a large-scale production system.

What's the action item behind this ticket? Would you like to see your storage incorporated in the core? Or is it open so that interested folks can exchange experiences?

The storage should not be incorporated in the core. We have made a MVP Humio storage plugin to visualize our data. This issue and our work should serve as a discussion board for a Humio storage plugin (similar to the other open storage plugin issues).

Closing as done: https://github.com/em135/humio-jaeger-plugin


Script loaders

Investigate the different script loaders, such as LABjs, require.js and head.js.
Could Cassette provide automatic support for these? Would that even make sense, given Cassette's module-based approach to scripts?

I too would like this feature.
I'm planning on writing my own code to emit a single head.js call for all included bundle URLs (regardless of location).
I haven't yet figured out how difficult that will be.


Can I use vuetify + weex?

Hello,
Is it possible to use weex and vuetify?

I am not familiar with vuetify, but all the vue syntax that Weex supports is listed here


[ Question ] Isn't the mid-training evaluation of the SWA performance corrupting the batchnorm running averages ?

Hello,
Thank you for your code.
I believe some mechanism is needed to reset the batchnorm running averages to their original values after a mid-training evaluation of the SWA performance (i.e. use of the bn_update method).
Otherwise, evaluating the SWA performance is tampering with the training procedure.
Maybe a buffer could be created when calling bn_update, and then used when the user calls a new method in the flavour of bn_reset before resuming to the training ?
Best,
Alex

Hi Alex,
Thank you for your question.
The mid-training evaluation of the SWA performance does not really have any effect on the training. We only apply bn_update to swa_model which is separated from model trained by SGD. Essentially, parameters of swa_model implement the buffers where we store the averaged weights of SGD iterates.
Let me know if that answers your question.
Best regards,
Timur

Hello Timur,
Sorry I had not seen your response. Thank you very much for it.
I think the buffers are only implemented for the parameters directly optimized by the chosen optimizer (i.e. optimized with a gradient-based approach). This would exclude the running averages of the batch norm layers.
Therefore, I believe after calling the bn_update method in the middle of the training, the running averages of batch norm layers are not reset to their original values and left "corrupted".
Are some buffers really dedicated to the batch norm running averages ?
Best,
Alex
 def swap_swa_sgd(self):
    r"""Swaps the values of the optimized variables and swa buffers.

    It's meant to be called in the end of training to use the collected
    swa running averages. It can also be used to evaluate the running
    averages during training; to continue training `swap_swa_sgd`
    should be called again.
    """
    for group in self.param_groups:  # in init: "self.param_groups = self.optimizer.param_groups"
        for p in group['params']:
            param_state = self.state[p]
            if 'swa_buffer' not in param_state:
                # If swa wasn't applied we don't swap params
                warnings.warn(
                    # "SWA wasn't applied to param {}; skipping it".format(p))
                    "SWA wasn't applied; skipping")
                continue
            buf = param_state['swa_buffer']
            tmp = torch.empty_like(p.data)
            tmp.copy_(p.data)
            p.data.copy_(buf)
            buf.copy_(tmp)

Hey @alexandrebone, we realized you are asking about the torchcontrib SWA implementation and not the implementation in this repo. In any case, in the torchcontrib implementation we assume that the non-SWA model is not going to be used after training. The batchnorm statistics do not affect the behavior of the model in train mode, as in train mode the batchnorm layers use batch statistics. So, the behaviour of the non-SWA model is unchanged by calling swap_swa_sgd and bn_udate.

Thank you very much for your answer ! Very clear.
Best,
Alex


Update/create a build task to: compile sass into css, add map and minify the resulting css file.

Problem
Hello, I would like to update/create a build gulp task to compile sass into css, add map and minify the resulting css file. My problem is about npm/github.
How can I make a npm install of my forked slate-tools modules into my theme directory?
ie, in the theme directory run npm install my-username/slate-tools.
Thank you.
Replication steps
n/a
More Information
n/a

You need to reference a local clone of your forked Github since your fork isnt published on NPM. It would look something like this:
"dependencies": {
    "bar": "file:../slate-forked/packages/slate-tools"
  }

This Stackoverflow thread has more details.
Typically you could reference the forked Github directly, but since we're using a monorepo structure for Slate this doesn't work. Hopefully NPM or Yarn will find a solution for this soon!


Support the addTelemetryInitializer boolean delegate

The addTelemetryInitializer function defines the following signature:
public addTelemetryInitializer(telemetryInitializer: (item: ITelemetryItem) => boolean | void)

[...] If one of the telemetry initializers returns false or throws an error, then the telemetry item will not be sent.

I don't believe there currently exists a way to utilise this boolean delegate such that telemetry items can be conditionally excluded or otherwise handled. This would be extremely useful.
I realise there may be JS interop limitations which immediately prevent this capability (serialization of a Func<> argument), but raising this issue in the hopes that some level of equivalence could be achieved.

I attempted to solve this a few times, the main issue is the call is synchronous and JSInterop must be async. I was not able to get this to work :(

I don't understand how sync/async issues could be blocking this; would you mind giving some more detail?

App Insights calls the telemetryInitializer synchronously, in that method, we need to call JSInterop into Blazor, this call has to be done async.

I see. From the documentation, .NET provides a synchronous version of the invocation method: DotNet.invokeMethod() (see here). Has this been explored?

I haven't as it only works on WASM, but that would likely work.


[BackEnd] Change "seatNum" into "row & col" and Optimization about query

Update for seat info type by using "row & col"
Optimization at /manager-event

update for seat info

make seat id "-" to "-:*"
means "eventId-seatNum" to"eventId-row:col"

❗ TODO : front-end update need

update for optimization when using query at /manager-event

when someone register at /register, update index by using CREATE INDEX
when manager request participants' information who buy some tickets about manager's event, get users' info by using  SELECT * FROM

You should make PRs separately if there are completely different topic of commits.
And you can mention someone who needs to check something, like this.
@greenapple-18 Please recognize context above!


Recognizes ipv6 hosts, adds an error field for optional error checkin…

…g; Adds mutation functions, URIFromAddress, Valid, Error methods.

Coverage remained the same at 100.0% when pulling 19a65ffe37885ab7f4c1ff716eb7324d31d6f5be on yuce:uri-updates into 9e38e81bd015db250f907d583edac9190bbd82cd on pilosa:master.

Coverage remained the same at 100.0% when pulling c564bfa3a583574bed41722078891bb785479e8a on yuce:uri-updates into 9e38e81bd015db250f907d583edac9190bbd82cd on pilosa:master.

Coverage remained the same at 100.0% when pulling 6ddfb4ff54d155e2dc2139a1b05572d047e97d66 on yuce:uri-updates into 9e38e81bd015db250f907d583edac9190bbd82cd on pilosa:master.


Refactor dosomething_user_address_country variable

The function introduced in #3249 adds some duplication, we need just one place to check for a country code.
This issue is to replace the calls using the variable dosomething_user_address_country with the new dosomething_settings_get_affiliate_country_code function.
Also the dosomething_settings_is_affiliate function can now be refactored to use  dosomething_settings_get_affiliate_country_code() != US, so we wont need the is_affiliate variable.

this has been refactored and will be deprecated once we are ~one site~


Create common file for /bin files.

There are some things in common between the scripts, like the version manipulation.
I propose creation of a file with some utilities, like writeln($message, $stream = STDOUT), increaseVersion($version, $type = 'patch') and ask($question).

Agreed!


Updated validations for polling source and http source + documentation

Updated documentation:
Added a tutorial on how to setup your environment to develop this provider locally. (Feel free to change/improve it)
Functions added:
Resource http source:
Added TimeZone validation against Sumologic server
Resource polling source:
All fields in func resourceSumologicPollingSource() *schema.Resource are being validated against Sumologic Server.
Update and Delete method working in Polling Source
Minor refactorings with naming consistency.

@josacar Hope that should be enough, thank you for your feedback and patience :)

Looks good to me

@erikvanbrakel can you take a look? I think it's all good :)
any changes just ask

Looks good! Will merge.


Custom response header

Question
Is there a possibility to return data (e.g. via headers) to client from hook?
Details
My current setup is a client sending a file to tusd inside a docker-container and a hook service inside a docker container.
Hook service checks if the user is authenticated and authorized on pre-create, and finally calls an API on post-finish (that could be changed to pre-finish for blocking). The API generates a unique id for further handling of the given file.
Can I somehow transfer the unique id back to the client, for example by setting a custom header "X-id" in the hooks response to tusd and tusd passing it to the client?

I hope this pull request fulfills the requirements. I'm a go as well as a github beginner

@tim3trick Interesting. I have a very similar requirement; interested to see what happens to your PR.
Separately, can you share whether in your Docker scenario you are actually storing the uploaded files outside of the container? I would need to and haven't been able to figure out how to get that accomplished.

Hi @netaisllc, couldn't you do that with a volume bount to the containers files directory? I'm currently planning to create a (cron)job for regular cleanup of the files directory, and my first idea was to use a volume to be able to access the files from the outside and either use a cronjob on the host or setup another container with the single purpose of cleaning up the files

Is there a possibility to return data (e.g. via headers) to client from hook?

Just to quickly answer that question: No, such a way does not exist right now. Application-specific communication from the server to the client must be handled outside of tusd for now.

Yes, that would be the easiest solution, I second that, but that would also require me to either open the hook service to the outside world (including the setup for the proxy, including load balancing etc) or bring another service into play that handles this. And finally I would have to include a storage that keeps the relation between source file and my application-specific data persistent, even though I neither need that relation nor the source file any longer, once the workflow is completed.
I hope that by just adding the "bypass" in tusd for passing data from the hook to the client, I keep the changes to tusd as generic as possible and do not violate it's purpose.

I'm not sure to follow what's the advancement stage of this issue, but just to confirm, there is absolutely no way of sending any custom data to the client right now? Not even Through headers?
I need to make a shortened URL for the image, the only possible thing is to make another request by the client when the upload is finished?

but just to confirm, there is absolutely no way of sending any custom data to the client right now?

No, there is currently no easy way to do this. The only possible approach is to implement your own middleware which catches the responses from tusd and adds application-specific headers to it. But that is very error-prone and might break easily.

@Acconut , Is this issue in plan in near future? We @zulip thinking of integrating TusD for chunk upload support. The problem we are facing is we are moving files to a different location after upload finishes using pre-finish hook, and want to send the new location of the file. If we can send using hook(or in response), than it would be much easier to handle things at client side.

Yes, there is a new hook system in v2 (https://github.com/tus/tusd/pull/672), which allows you to customize the responses from inside the blocking hooks. It's not yet released, but feel free to try it out. Documentation is lacking so far, but the examples might help you to get started: https://github.com/tus/tusd/tree/v2/examples
Also, it's nice to hear that Zulip is interested. I used your services frequently in the past :)

Thank you @Acconut, Can you tell when(estimated) the next release will come out?

There is no clear release date but we hope to finish v2 in the next two months.

Any update on this issue? The Zulip implementation for this is almost complete, and this issue is the main thing that's ugly about the implementation, so if v2 is on track to be ready for production use in the next month or so, we'll probably want to hold off on merging our tusd integration until this is merged.

The v2 release is shaping up well and is already feature-complete. It is mostly documentation that has to be added at this point. The release will be finished soon and I would recommend you to try out tusd v2. Once it is released, we will not be able to support tusd v1 anymore.


Using SDDM with custom themes

Describe the bug
On unstable, when trying to use a custom theme on SDDM, it fails to load it with this custom error message:
The theme at "/run/current-system/sw/share/sddm/themes/catppuccin-mocha" requires missing "/nix/store/f03hm28dsz6v0d9zdwdlfmrra25k2w7l-sddm-wrapped/bin/sddm-greeter". Using fallback theme.
I'm using this custom package for theming: https://github.com/ElysaSrc/nixos-config/blob/main/modules/home/packages/catppuccin-sddm.nix
Steps To Reproduce
Steps to reproduce the behavior:

Use plasma + sddm on unstable (qt6 versions).
It loads the default theme, this error shows up.

Expected behavior
The theme sould load

Add a :+1: reaction to issues you find important.

Yep, this is extremely annoying. Seems like the greeter is there but it has qt6 suffix.

Yep, this is extremely annoying. Seems like the greeter is there but it has qt6 suffix.

Would it be hard to modify the derivation to add a symlink ?

I was stuck after updating to Plasma6 today at the login screen. This is the fallback in the screenshot. I had the same error:  The theme at "/usr/share/sddm/themes/Utterly-Nord" requires missing "/usr/bin/sddm-greeter" . Using fallback theme.
I resolved it by creating a new user, logging in and changing the sddm theme to the default one. Now im back up and running.

I was stuck after updating to Plasma6 today at the login screen. This is the fallback in the screenshot. I had the same error: The theme at "/usr/share/sddm/themes/Utterly-Nord" requires missing "/usr/bin/sddm-greeter" . Using fallback theme.
I resolved it by creating a new user, logging in and changing the sddm theme to the default one. Now im back up and running.

I'm on a freshly installed computer (it never was on plasma 5) and I'm not sure why creating a new user would work on my case ? I can give it a try.

This happens because 3rd party themes have not been updated to use the QT6 version.
You have 2 options:

Set the sddm package back to older package. This is needed because enabling the plasma6 module also sets the sddm package to the newer, kdePackages.sddm

services.xserver.displayManager.sddm.package = libsForQt5.sddm

Another option is to keep using kdePackages.sddm, but override it and pass in qt5 libraries, as it will automatically enable the QT6 flag for SDDM if the QT version is 6

Port the theme you are using to QT6. For most themes, it only consists of

Replacing all 'import QtGraphicalEffects' with  'import Qt5Compat.GraphicalEffects'
Adding a metadata.desktop file at the theme root folder (if it doesn't already have one) and add 'QtVersion=6'

This happens because 3rd party themes have not been updated to use the QT6 version.
There are 2 options:

Set the sddm package back to older package. This is needed because enabling the plasma6 module also sets the sddm package to the newer kdePackages.sddm package.

services.xserver.displayManager.sddm.package = libsForQt5.sddm

Another option is to keep using kdePackages.sddm, but override it and pass in qt5 libraries, as it will automatically enable the QT6 flag for SDDM if the QT libraries version is 6

Port the theme you are using to QT6. For most themes, it only consists of:

Replacing all 'import QtGraphicalEffects' with  'import Qt5Compat.GraphicalEffects'
Adding a metadata.desktop file at the theme root folder (if it doesn't already have one) and add 'QtVersion=6'

This is right. Maybe this issue can be closed then.

error: builder for '/nix/store/kfjifv45gnmf4yx0b1hsz01ah3xnnsg1-sddm-wrapped.drv' failed with exit code 1;
       last 3 log lines:
       > Error: detected mismatched Qt dependencies:
       >     /nix/store/2mcqdmp7kb8pfkibxsm10z3s94j80s6y-qtbase-6.6.3
       >     /nix/store/62jj0ky87hh1rxrjcz6w44ix2798skc2-qtbase-5.15.12-dev
       For full logs, run 'nix log /nix/store/kfjifv45gnmf4yx0b1hsz01ah3xnnsg1-sddm-wrapped.drv'.
error: 1 dependencies of derivation '/nix/store/bbr1ryr00ps0k6mrm4nxs9az5kwvxgf5-dbus-1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/kxvf6n7j9mcx2iih02niqfx658rwky8f-dbus-1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/n3b6zqnvsg5h2w1a04h8p0lw95byrsh1-system-path.drv' failed to build
error: 1 dependencies of derivation '/nix/store/w62ja5yx5nb7gr9wm6nn7fd5n8shh5jz-system-path.drv' failed to build
error: 1 dependencies of derivation '/nix/store/1yrhaf4ivqnyi84h4yc1vnicn6yiy1il-tmpfiles.d.drv' failed to build
error: 1 dependencies of derivation '/nix/store/hfx1bjp3c29yqsw8bn9j1ai0917zq8sa-tmpfiles.d.drv' failed to build
error: 1 dependencies of derivation '/nix/store/z9gc35w5a5gs7ss1fgpyk2i168xzylwp-nixos-system-pavel-nix-24.05.20240329.0f20cd1.drv' failed to build

Well, if I try to downgrade sddm, I get this.

You are experiencing this issue likely because Plasma 6 sets services.displayManager.sddm.extraPackages and includes Qt6 packages. If you set that option to mkForce [], the error would likely disappear. Of course other issues might crop up if you were implicitly relying on those packages existing.

You are experiencing this issue likely because Plasma 6 sets services.displayManager.sddm.extraPackages and includes Qt6 packages. If you set that option to mkForce [], the error would likely disappear. Of course other issues might crop up if you were implicitly relying on those packages existing.

I have apply lib.mkForce in my code like this:
services = {
    xserver.enable = true;

    desktopManager.plasma6= {
        enable = true;
        enableQt5Integration = true;
    };

    # Enable Sddm
    displayManager.sddm = {
       enable = true;
       package = lib.mkForce pkgs.libsForQt5.sddm;
       theme = "${import ./sddm-theme.nix { inherit pkgs;}}";
    };
};

but nix still complain
error: builder for '/nix/store/gmcpabrr9aa24gsh5c6iw3266psxmirg-sddm-wrapped.drv' failed with exit code 1;
       last 3 log lines:
       > Error: detected mismatched Qt dependencies:
       >     /nix/store/r8a6ss3xnc9j4yrgajbvy8wnn359cpkq-qtbase-6.7.0
       >     /nix/store/pblcbcbida7wmpyb6brhcasrj8rg5jpk-qtbase-5.15.12-dev
       For full logs, run 'nix log /nix/store/gmcpabrr9aa24gsh5c6iw3266psxmirg-sddm-wrapped.drv'.

did I miss something?

FWIW I managed to get my theme working with this:
  services.displayManager.sddm = {
    package = pkgs.lib.mkForce pkgs.libsForQt5.sddm;
    extraPackages = pkgs.lib.mkForce [ pkgs.libsForQt5.qt5.qtgraphicaleffects ];
    enable = true;
    theme = "corners";
  };

The extraPackages you need may vary with your theme...I started by setting it empty and then just adding the packages it told me were missing.

I'm experiencing this issue with the catppuccin-sddm package in nixpkgs on Hyprland, however, using this catppuccin-sddm package from an old PR works fine for me.
May 13 22:50:27 will-desktop sddm[1770]: The theme at "/run/current-system/sw/share/sddm/themes/catppuccin-mocha" requires missing "/nix/store/3a8ilym4yp223ppv4pijmnmi2paxr5hq-sddm-wrapped/bin/sddm-greeter-qt6" . Using fallback theme.

I'm experiencing this issue with the catppuccin-sddm package in nixpkgs on Hyprland, however, using this catppuccin-sddm package from an old PR works fine for me.
May 13 22:50:27 will-desktop sddm[1770]: The theme at "/run/current-system/sw/share/sddm/themes/catppuccin-mocha" requires missing "/nix/store/3a8ilym4yp223ppv4pijmnmi2paxr5hq-sddm-wrapped/bin/sddm-greeter-qt6" . Using fallback theme.

By default, SDDM falls back to Qt5. You have to override the package version with this:
displayManager.sddm.package = pkgs.kdePackages.sddm;

I'm experiencing this issue with the catppuccin-sddm package in nixpkgs on Hyprland, however, using this catppuccin-sddm package from an old PR works fine for me.
May 13 22:50:27 will-desktop sddm[1770]: The theme at "/run/current-system/sw/share/sddm/themes/catppuccin-mocha" requires missing "/nix/store/3a8ilym4yp223ppv4pijmnmi2paxr5hq-sddm-wrapped/bin/sddm-greeter-qt6" . Using fallback theme.

By default, SDDM falls back to Qt5. You have to override the package version with this:
displayManager.sddm.package = pkgs.kdePackages.sddm;

That fixed it, thanks for your help. Strangely the theme didn't apply until I rebooted as well, previously I was just exiting hyprland with hyprtcl dispatch exit, which did work for applying some other sddm themes I tried.


add missing event parameter to touchEndEvent

This fixes a ReferenceError: event is not defined error I was hitting.

Thanks for your help 😀

A new version v0.2.4 has published on npm 😉

Awesome, thanks!


Update create_commit output type to return something even when no PR is created

Related to PR https://github.com/huggingface/huggingface_hub/pull/888.
When create_commit is used, it will return a URL (str) if create_pr is True and None otherwise. It feels weird as we should be able to return something (like commit id or commit url for instance). Other subsequent methods have the same behavior (upload_folder, upload_file). Might be awkward  to handle backward compatibility here.
For the record, output from the API is this one (internal link).

For the record, the returned pr_url is used once in transformers here


2022.11.33 - Sensor Parsing Errors

Loaded up 2022.11.33. Logs filling up with errors around parsing sensor data, much like the initial two included below (they seem to fire every second or two).
path To files:          =================
   Hue Lights                      detailed logging        /Library/Application Support/Perceptive Automation/Indigo 2022.1/Logs/com.nathansheldon.indigoplugin.HueLights/plugin.log
   Hue Lights                      testing logging levels, for info only:
   Hue Lights                      logger  enabled for     INFO           ==> TEST ONLY
   Hue Lights Warning              logger  enabled for     WARNING        ==> TEST ONLY
   Hue Lights Error                logger  enabled for     ERROR          ==> TEST ONLY
   Hue Lights Error                logger  enabled for     CRITICAL       ==> TEST ONLY
   Started plugin "Hue Lights 2022.11.33"
   Hue Lights                      debug areas:[]
   Hue Lights                      bridge lights groups scenes users resources rules schedules sensors hostIds / user names ------------------- ip Numbers ---- BridgeIds-------
   Hue Lights                      #0         47     52     69   200        15   180        12      72 eKZfuqKfR1VZ0bQBq27UT8BVXhC4OHmBK18f3QOw <IP_ADDRESS>       001788FFFEA8EB67
   Hue Lights                      #1          2      1      9    64         9    74        14      15 KtZynLv1ZU4gwBknjmWjj01zNkd5odsP608vP6tk <IP_ADDRESS>       001788FFFEA252CB
   Hue Lights                      ... initialized
   Hue Lights                      Hue Lights  "Yard - Hue Motion Sensor - Luminance" received: sensor update to 766 lux
   Hue Lights Error
Traceback (most recent call last):
  File "plugin.py", line 7336, in parseOneHueSensorData
    if logChanges: self.indiLOG.log(20, "Hue Lights  \"{}\" received:{}  {}".format(device.name), shortcut['buttontexts'][lastButtonPressed],  shortcut['textSuffix'][eventType+1])
IndexError: Replacement index 1 out of range for positional args tuple
   Hue Lights Error
Traceback (most recent call last):
  File "plugin.py", line 7336, in parseOneHueSensorData
    if logChanges: self.indiLOG.log(20, "Hue Lights  \"{}\" received:{}  {}".format(device.name), shortcut['buttontexts'][lastButtonPressed],  shortcut['textSuffix'][eventType+1])
IndexError: Replacement index 1 out of range for positional args tuple
   Hue Lights                      Hue Lights  "Name Redacted Room- Hue Motion

for the time being switch off all logging in config. I shall post a version today that fixes it .
it is one misplaced “)” in a format statement for a device type&event type I don’t have installed.
Karl
ps the next version can:

add new devices to the bridge
delete devices from bridge
rename them
stet on/off,

those are  all option available on a bridge that you can do  to / with devices on the bridge.

On Jul 19, 2022, at 04:39, virtzilla @.***> wrote:
Loaded up 2022.11.33. Logs filling up with errors around parsing sensor data, much like the initial two included below (they seem to fire every second or two).
path To files: ================= Hue Lights detailed logging /Library/Application Support/Perceptive Automation/Indigo 2022.1/Logs/com.nathansheldon.indigoplugin.HueLights/plugin.log Hue Lights testing logging levels, for info only: Hue Lights logger enabled for INFO ==> TEST ONLY Hue Lights Warning logger enabled for WARNING ==> TEST ONLY Hue Lights Error logger enabled for ERROR ==> TEST ONLY Hue Lights Error logger enabled for CRITICAL ==> TEST ONLY Started plugin "Hue Lights 2022.11.33" Hue Lights debug areas:[] Hue Lights bridge lights groups scenes users resources rules schedules sensors hostIds / user names ------------------- ip Numbers ---- BridgeIds------- Hue Lights #0 47 52 69 200 15 180 12 72 eKZfuqKfR1VZ0bQBq27UT8BVXhC4OHmBK18f3QOw <IP_ADDRESS> 001788FFFEA8EB67 Hue Lights #1 2 1 9 64 9 74 14      15 KtZynLv1ZU4gwBknjmWjj01zNkd5odsP608vP6tk <IP_ADDRESS> 001788FFFEA252CB Hue Lights ... initialized Hue Lights Hue Lights "Yard - Hue Motion Sensor - Luminance" received: sensor update to 766 lux Hue Lights Error Traceback (most recent call last): File "plugin.py", line 7336, in parseOneHueSensorData if logChanges: self.indiLOG.log(20, "Hue Lights "{}" received:{} {}".format(device.name), shortcut['buttontexts'][lastButtonPressed], shortcut['textSuffix'][eventType+1]) IndexError: Replacement index 1 out of range for positional args tuple Hue Lights Error Traceback (most recent call last): File "plugin.py", line 7336, in parseOneHueSensorData if logChanges: self.indiLOG.log(20, "Hue Lights "{}" received:{} {}".format(device.name), shortcut['buttontexts'][lastButtonPressed], shortcut['textSuffix'][eventType+1]) IndexError: Replacement index 1 out of range for positional args tuple Hue Lights Hue Lights "Name Redacted Room- Hue Motion
—
Reply to this email directly, view it on GitHub https://github.com/kw123/Hue-Lights-Indigo-plugin/issues/19, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEH2SNFLA2GIILXQLGA37H3VUYIOJANCNFSM536L2RTA.
You are receiving this because you are subscribed to this thread.

v2022.12.36 @ 2022-07-20
should fix that

On Jul 20, 2022, at 00:45, virtzilla @.***> wrote:
Installed 2022.11.34 and no more red in the logs.
There is a bit of strangeness observed when the plugin fired up:
Hue Lights                      Hue Lights  "Dining Room - Switch" received:button2  release
Hue Lights                      Hue Lights  "Entry - Switch" received:button3  release
Hue Lights                      Hue Lights  "Front Yard - Outside Light Switch" received:button1  release
Hue Lights                      Hue Lights  "Kitchen - Hue Dimmer Remote" received: Hue/off  press with short release
Those remotes haven't been touched for at least 24 hours (some days or longer), so it seems odd that they are appearing unless they got stuck in a queue (which also would seem odd). Point of concern, or as expected?
In terms of "stet on/off," you wrote above - could you elaborate on what that is?
—
Reply to this email directly, view it on GitHub https://github.com/kw123/Hue-Lights-Indigo-plugin/issues/19#issuecomment-1189622807, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEH2SNH677PNZANWHGIUBEDVU4VZDANCNFSM536L2RTA.
You are receiving this because you commented.

It did.

are the messages just once or ongoing?
Re set on-off
There is now a menu option to disable / enable switches / sensors on the bridge.
So if you are home you could disable a motion sensor and then actions on the bridge get disabled. Eg  light on if motion
detected.
Karl

On 20.07.2022, at 00:45, virtzilla @.***> wrote:

Installed 2022.11.34 and no more red in the logs.
There is a bit of strangeness observed when the plugin fired up:
Hue Lights                      Hue Lights  "Dining Room - Switch" received:button2  release
Hue Lights                      Hue Lights  "Entry - Switch" received:button3  release
Hue Lights                      Hue Lights  "Front Yard - Outside Light Switch" received:button1  release
Hue Lights                      Hue Lights  "Kitchen - Hue Dimmer Remote" received: Hue/off  press with short release
Those remotes haven't been touched for at least 24 hours (some days or longer), so it seems odd that they are appearing unless they got stuck in a queue (which also would seem odd). Point of concern, or as expected?
In terms of "stet on/off," you wrote above - could you elaborate on what that is?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.


Tenant specific media and trans files not showing up

Hi,
I am building a multi-tenancy application using Laravel 5.5 and hyn/multi-tenancy (5.0.6.) and 5.0.15 (I upgraded after reporting this issue, but there is no improvement.)
The multi-tenancy works fine for the database: if I login to the application for a specific tenant through the url for the tenant, changes to the data are nicely reflected in the database of the tenant.
However, I try to display a tenant specific logo.jpg in the header and I created a tenant specif translation file.
As in the documentation, I created the folders "media" and "trans" in the tenant folder ("storage/app/tenancy/tenants/b015cd66499f41d39ed3c8d08ed4708a").
Unfortunately, I cannot get this to work. I chatted about this issue with Luceos and he asked me to create this issue.
In my blade.php I have the following code. The logo.jpg shows in the browser for the tenant in these situations:

And:
img src="storage/media/logo.jpg" alt="Cash Flow Forecaster LogoX"
But the logo.jpg file is always served from this folder:
"storage/app/public/media"
and not from
"storage/app/tenancy/tenants/b015cd66499f41d39ed3c8d08ed4708a",
what I expect.
I cannot figure out what's wrong.

Laravel :5.5

Multi-tenancy: 5.0.15

database driver and version: MariaDb 10.2.10

webserver software and version: nginx 1.13.3
config.zip

php version: 7.1.10

Vagrant box / Homestead
Ubuntu 16.04

tenancy.php config

webserver.php config

Error log
There are no errors in the log related to this issue.

Hello,
I don't know if it's related but did you try with: asset('media/logo.jpg') ?
I use this path on my app and it's work fine.

Hi Julian,
Yes, I have tried that too. If I use asset('storage/media/logo.jpg') the
logo in the default folder is displayed, if I follow your suggestion, the
logo.jpg is not found at all.
(sorry for the late response, my Windows 10 crashed and I am
reinstalling.... :-( )
On Fri, Dec 8, 2017 at 3:36 AM, Julian Renard<EMAIL_ADDRESS>wrote:

Hello,
I don't know if it's related but did you try with: asset('media/logo.jpg')
?
I use this path on my app and it's work fine.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/hyn/multi-tenant/issues/344#issuecomment-350155766,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AZDp0wpiHzP6r8WrigoJ-xVI1xc0yFgHks5s-KDEgaJpZM4Q5ryK
.

--
Mat beschte Gréiss
Kind regards
Reinier Bloem
Téléphone: +352 273 950 59
Mobile: +33 7800 22533

Sorry for my late response too, pretty busy this weekend...
Do you have this instruction in your nginx tenant config:
location ~* ^/media/(.+)$ {         alias /home/vagrant/code/storage/app/tenancy/tenants/1c5b76eed9f5422a9d983fdeb08e991c/media/$1;     }  ?

@ReinierB you have to use a direct link:

Please note this only works currently if you use the apache or nginx webserver configurations. Also you might need to touch the website model once you created the media folder, it will only add the alias to the internal storage folder if the media folder exists in the tenant directory.

Hi,
I have solved the problem for the /media:

I have added the line of code as suggested by @julianfox:

location ~* ^/media/(.+)$ {
alias /home/vagrant/code//storage/app/tenancy/tenants/1c5b76eed9f5422a9d983fdeb08e991c/media/$1;
}
to the "1c5b76eed9f5422a9d983fdeb08e991c.conf" file

"a.cff.dev" is the tenant web site. There were 2 files with the same name ("a.cff.dev") in the /etc/nginx/sites-available and /etc/nginx/sites-enabled folders containing  "server_name a.cff.dev;". After removing these files (or removing / changing the "server_name" to something else), the tenant specific image appears on the page correctly.

This is the line of html code in the blade file, as posted by @luceos and as in the documentation, that I use now:
Thank you both luceos and julianfox for helping me.
The /trans (tenant specific translations) do not show up. This issue is still open. I'll investigate further.

Related to trans can you explain the steps to reproduce? What key do you use, what trans file did you add, etc.

@luceos:
Here is my setup:
Inside the tenants folder, I created  "/lang/en" and I have place the translation file "setup.php" in there:
storage/app/tenancy/tenants/102fd25696904985bbd8558651fc4fae/lang/en/setup.php
Some of the translation strings in the setup.php (see zip below) file are:
'Company' => 'Compaan',
'Companies' => 'Comps',
'Label' => 'Naam',

Translations work fine in the "main" site (standard Laravel), but not for tenants
setup.zip
tenancy.zip
Additionally, I have tried:
storage/app/tenancy/tenants/102fd25696904985bbd8558651fc4fae/lang/setup.php
storage/app/tenancy/tenants/102fd25696904985bbd8558651fc4fae/trans/en/setup.php
storage/app/tenancy/tenants/102fd25696904985bbd8558651fc4fae/trans/setup.php
storage/app/tenancy/tenants/102fd25696904985bbd8558651fc4fae/trans/lang/en/setup.php
storage/app/tenancy/tenants/102fd25696904985bbd8558651fc4fae/trans/lang/setup.php

What's the translation key you used in your views, code?

What do you mean by translation key?
A translation string example I use in a blade file is:
@lang('setup.Company')
The translation as in the default setup.php file is displayed instead of the translation from the tenant specific file.

@ReinierB the dev-master branch holds the MediaController. I've added a test to confirm it's working as intended.


Refactorings

Update lots of comments
Small local refactorings
Update ghost dependency function to no longer add links to all files when a namespace is not found anywhere
Change the name of the test flag DumpGraph -> DumpCheckingGraph

I didn't use Fantomas, because when I ran it, I got changes in 25 files and also in lines I didn't modify, so I wasn't sure which of those I was supposed to commit.

I can't seem to build the project when checking out this branch:

Did you push everything?

I didn't use Fantomas, because when I ran it, I got changes in 25 files and also in lines I didn't modify, so I wasn't sure which of those I was supposed to commit.

If unrelated files changed, I might mean that you were invoking Fantomas from another location than the repository root.
TL;DR there are known issues with the .fantomasignore file, it only works properly if you run dotnet fantomas src -r from the repository root.

scenario "A open statement that leads nowhere should link to every file that came above it." failed locally for me. This test covers ghost dependencies specifically.

But what really disturbed me was that none of my local projects were running in graph mode. Because --deterministic+ is enabled by default and you need to explicitly turn that off before graph can kick in.
Only when setting --deterministic- will --test:GraphBasedChecking do anything.
This should be added for the methodOptions in CompliationTests.fs, but maybe we also want to print a warning when --deterministic+ is combined with --test:GraphBasedChecking.

This is something I should fix in the main branch (or here) - perhaps when the feature is enabled via the dedicated flag, it should stay enabled even in deterministic builds, and only the experimental flag should have the implicit disable possible.

I revert controversial changes and fixed the build.

If unrelated files changed, I might mean that you were invoking Fantomas from another location than the repository root.
TL;DR there are known issues with the .fantomasignore file, it only works properly if you run dotnet fantomas src -r from the repository root.

I ran the following:
PS C:\projekty\fsharp\nojaf> dotnet tool run fantomas --version
Fantomas v5.0.3+f1ec0088672e4b075aab51d5b66a99c7ae538260

PS C:\projekty\fsharp\nojaf> git reset --hard
PS C:\projekty\fsharp\nojaf> dotnet tool run fantomas -r .

and this gives the following result:
PS C:\projekty\fsharp\nojaf> git status
On branch graph-tc-new-tweaks
Your branch is up to date with 'safesparrow/graph-tc-new-tweaks'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   vsintegration/tests/FSharp.Editor.Tests/BraceMatchingServiceTests.fs
        modified:   vsintegration/tests/FSharp.Editor.Tests/BreakpointResolutionServiceTests.fs
        modified:   vsintegration/tests/FSharp.Editor.Tests/CompletionProviderTests.fs
        modified:   vsintegration/tests/FSharp.Editor.Tests/DocumentDiagnosticAnalyzerTests.fs
        modified:   vsintegration/tests/FSharp.Editor.Tests/DocumentHighlightsServiceTests.fs
        modified:   vsintegration/tests/FSharp.Editor.Tests/EditorFormattingServiceTests.fs
        modified:   vsintegration/tests/FSharp.Editor.Tests/FsxCompletionProviderTests.fs
        modified:   vsintegration/tests/FSharp.Editor.Tests/GoToDefinitionServiceTests.fs
        modified:   vsintegration/tests/FSharp.Editor.Tests/HelpContextServiceTests.fs
        modified:   vsintegration/tests/FSharp.Editor.Tests/Helpers/RoslynHelpers.fs
        modified:   vsintegration/tests/FSharp.Editor.Tests/Hints/HintTestFramework.fs
        modified:   vsintegration/tests/FSharp.Editor.Tests/Hints/InlineParameterNameHintTests.fs
        modified:   vsintegration/tests/FSharp.Editor.Tests/Hints/InlineTypeHintTests.fs
        modified:   vsintegration/tests/FSharp.Editor.Tests/IndentationServiceTests.fs
        modified:   vsintegration/tests/FSharp.Editor.Tests/LanguageDebugInfoServiceTests.fs
        modified:   vsintegration/tests/FSharp.Editor.Tests/QuickInfoProviderTests.fs
        modified:   vsintegration/tests/FSharp.Editor.Tests/QuickInfoTests.fs
        modified:   vsintegration/tests/FSharp.Editor.Tests/SemanticClassificationServiceTests.fs
        modified:   vsintegration/tests/FSharp.Editor.Tests/SignatureHelpProviderTests.fs
        modified:   vsintegration/tests/FSharp.Editor.Tests/SyntacticColorizationServiceTests.fs

I'll keep reverting those changes, but I can't see what I'm doing wrong.

Oh, that is interesting that. I always run Fantomas on the src folder. vsintegration is listed in the .fantomasignore but I'm not picked up. I'm not sure if that is to be expected or not. Maybe they shuffled stuff around. I'll look into this.


Reuse util.ContainerIDForPID in enricher

What type of PR is this?
/kind cleanup
What this PR does / why we need it:
We now reuse the utility function to cleanup parts of the enricher
implementation.
Which issue(s) this PR fixes:

None
Does this PR have test?
None

Special notes for your reviewer:
None
Does this PR introduce a user-facing change?

None

Codecov Report

Merging #637 (9d12496) into master (0193bbf) will decrease coverage by 0.40%.
The diff coverage is 66.66%.

:exclamation: Current head 9d12496 differs from pull request most recent head 3d8d390. Consider uploading reports for the commit 3d8d390 to get more accurate results

@@            Coverage Diff             @@
##           master     #637      +/-   ##
==========================================
- Coverage   43.76%   43.36%   -0.41%
==========================================
  Files          29       29
  Lines        1565     1545      -20
==========================================
- Hits          685      670      -15
+ Misses        834      831       -3
+ Partials       46       44       -2

/retest

/test pull-security-profiles-operator-test-e2e

/retest

/test pull-security-profiles-operator-test-e2e

@pjbgf @jhrozek @JAORMX PTAL


Machine Frame doesn't have a localized name

As the title says, the Machine Frame needs a localized name. Right now it is tile.machine_frame.name

Fixed next release


Module should return IP address of service providers for further action

Hello. Thanks for the awesome module. However, it doesn't return an IP address.
Here's what Bonjour returns:
{ addresses:
   [ 'fe80::1067:1df6:2e65:440e',
     '<IP_ADDRESS>',
     'fe80::f4ae:26ff:fe41:63fd',
     'fe80::4006:72c4:f4b7:6292',
     'fe80::cd1b:e77e:f9bb:6ae6',
     'fe80::608e:f9b2:c226:272b',
     'fe80::f287:c988:48f9:c88b',
     'fe80::1daf:386d:b81:afc6',
     'fe80::aede:48ff:fe00:1122' ],
  name: 'RoomOS',
  fqdn: 'RoomOS._http._tcp.local',
  host: 'Damilola-10.local',
  referer:
   { address: '<IP_ADDRESS>',
     family: 'IPv4',
     port: 5353,
     size: 568 },
  port: 3000,
  type: 'http',
  protocol: 'tcp',
  subtypes: [],
  rawTxt: <Buffer 0e 47 72 65 65 74 69 6e 67 3d 48 45 4c 4c 4f>,
  txt: { greeting: 'HELLO' } }

Here's what the module returns:
{eventName: "propertyChange", object: Zeroconf, propertyName: "serviceFound", value: Object, oldValue: undefined}
serviceFound: {"eventName":"propertyChange","object":{"_observers":{"propertyChange":[{}]},"serviceType":"_http._tcp.","netServiceBrowser":{}},"propertyName":"serviceFound","value":{"name":"RoomOS","type":"_http._tcp.","host":"Damilola-9.local.","port":3000}}

Hey @DamilolaJegede, to be honest haven't looked at this for a while, but I see the module returns {"name":"RoomOS","type":"_http._tcp.","host":"Damilola-9.local.","port":3000}, so can't you just connect to host/port Damilola-9.local.:3000? If Bonjour is working correctly this should be resolvable if I remember correctly?

You’re very correct. Worked well. Thanks a billion!

On Dec 12, 2018, at 11:14 AM, Jonathan Salomon<EMAIL_ADDRESS>wrote:
Hey @DamilolaJegede, to be honest haven't looked at this for a while, but I see the module returns {"name":"RoomOS","type":"_http._tcp.","host":"Damilola-9.local.","port":3000}, so can't you just connect to host/port Damilola-9.local.:3000? If Bonjour is working correctly this should be resolvable if I remember correctly?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.


gogo removal part 2: avoid all gogo-specific features

This removes the dependency on anything special gogoproto can do.  Phase 3 actually uses the standard compiler instead of gogo.

Just out of curiosity, will this break the go-client? (I know we've moved a lot to internal but just from the use of the go-client in tests, seems like a bunch of definitions have changed)


NullPointerException in latest build

TestNG Version

Note: only the latest version is supported

Expected behavior
the java package is expected to run as usual after updating the build
Actual behavior
getting NullPointerException soon after updating the TestNG new build in eclipse
Is the issue reproductible on runner?

[x] Eclipse

Test case sample

Please, share the test case (as small as possible) which shows the issue

Hello,
I have same issues with the latest version of TestNG.
The NullPointerException supposed to be corrected but the version 7.0.0-beta7 is unrunable in eclipse IDE.
There is  an example of my test case
You can aslo find erros I had in my project.
Thanks for help.

@shah4corp @campao - Please file a bug in the eclipse TestNG plugin project https://github.com/cbeust/testng-eclipse


Sort By Column / Script to apply parent-child hierarchy in Tabular Editor

Is there any way to sort a column  by another column in Tabular Editor?
I´m new to it and couldn´t find this property.
My ultimate goal is to automatically generate the DAX PATTERN https://www.daxpatterns.com/parent-child-hierarchies/ through a Tabular Editor script.. but I have a requirement that the order of the columns of each level must be ordered by each Key (not alphabetically)

Sort by property of columns. Look again :)
On Wed, Dec 30, 2020, 12:59 natolira<EMAIL_ADDRESS>wrote:

Is there any way to sort a column by another column in Tabular Editor?
I´m new to it and couldn´t find this property.
My ultimate goal is to automatically generate the DAX PATTERN
https://www.daxpatterns.com/parent-child-hierarchies/ through a Tabular
Editor script.. but I have a requirement that the order of the columns of
each level must be ordered by each Key (not alphabetically)
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/otykier/TabularEditor/issues/723, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AGTQLM4C6AZ53RG2WNISS6LSXMIZ5ANCNFSM4VOICPYA
.

Yeah... I had to enable the Experimental Power BI features. Now I can see it :)
Thanks.

Yeah... I had to enable the Experimental Power BI features. Now I can see it :)
Thanks.


Automatically import $(MSBuildProjectName).props/.targets if found

The IDE(s) will typically modify the .csproj whenever you add/remove stuff and save. Sometimes, it's not 100% full-fidelity (i.e. it will modify the newlines in attributes you might have carefully crafted).
In addition, sometimes you just want to keep the declarative project structure and props in the csproj, and leave the augmented behavior in a separate .targets file alongside the project (I do this all the time for VSIX projects, for example, or to separate the packaging aspects from the build aspects of the project).
It would be useful if the Common.props and Common.targets already imported those files automatically, like:
<Import Project="$(MSBuildProjectDirectory)\$(MSBuildProjectName).props"
        Condition="Exists('$(MSBuildProjectDirectory)\$(MSBuildProjectName).props')" />

and
<Import Project="$(MSBuildProjectDirectory)\$(MSBuildProjectName).targets"
        Condition="Exists('$(MSBuildProjectDirectory)\$(MSBuildProjectName).targets')" />

There are already a lot of extension points. I would think Directory.Build.props/targets might be close? Unless you don't want any imports? In which case I would use an SDK from a NuGet package or something like that.

Indeed, there are. Just thinking that something seamless like the auto-correct importing of directory.build.props/targets would make sense at the project level too.

Closing. This is easily achievable with those two lines in Directory.Build.props and Directory.Build.targets, so ¯_(ツ)_/¯


WebGL IE11 link broken

The link under Known Issues for WebGL (http://connect.microsoft.com/IE/feedback/details/795172/ie11-fails-more-than-half-tests-in-official-webgl-conformance-test-suite) return 401 Unauthorized. Don't know if this is temporary, but right now it doesn't work for me.

The link loads just fine for me currently.

Sure does, I'll close this.


[SPARK-22029][PySpark] Add lru_cache to _parse_datatype_json_string

What changes were proposed in this pull request?
_parse_datatype_json_string is called many times for the same datatypes.
By cacheing its result we can speed up pySpark internals.
Speed up is bigger for complicated SQL schemas.
Test
import time
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()

df = spark.range(1000000).selectExpr("struct(struct(id, id), id) as id0", "struct(struct(id, id), id) as id1", "struct(struct(id, id), id) as id2", "struct(struct(id, id), id) as id3", "struct(struct(id, id), id) as id4", "struct(struct(id, id), id) as id5", "struct(struct(id, id), id) as id6", "struct(struct(id, id), id) as id7", "struct(struct(id, id), id) as id8", "struct(struct(id, id), id) as id9").cache()
df.count()
for i in range(10):
  start = time.time()
  df.rdd.map(lambda x: x).count()
  print(i, time.time() - start)

Before:
0 12.123183250427246
1 11.405049562454224
2 11.22128415107727
3 11.24196171760559
4 11.492472410202026
5 11.242795705795288
6 11.167386531829834
7 11.119175672531128
8 11.130866765975952
9 11.13940691947937

After:
0 11.548425674438477
1 10.786834239959717
2 10.491389751434326
3 10.409717082977295
4 10.756452083587646
5 10.595868825912476
6 10.374756097793579
7 10.407423257827759
8 10.379964113235474
9 10.382996797561646

How was this patch tested?
Existing tests.
Performance benchmark.

Test build #81846 has started for PR 19255 at commit c903860.

Test build #81846 has finished for PR 19255 at commit c903860.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81846/
Test FAILed.

Merged build finished. Test FAILed.

Could you mark [PySpark] in the title? cc @ueshin

This PR also needs pref tests and the results in the description.

@HyukjinKwon
I added perf tests.

Jenkins, retest this please.

Test build #82977 has started for PR 19255 at commit 679141d.

Test build #82976 has started for PR 19255 at commit 679141d.

Test build #82976 has finished for PR 19255 at commit 679141d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82976/
Test PASSed.

Test build #82977 has finished for PR 19255 at commit 679141d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82977/
Test PASSed.

Merged build finished. Test PASSed.

Ping @HyukjinKwon

Ditto: if you'd like to leave it as is, I think we should better close it.

I'd say we should close this, too.

OK. Let's close it.


403 Forbidden Errors

Currently, there is an issue regarding mass 403 errors. I believe that this may be due to CloudFlare now enforcing the use of cookies or some other mechanism to prevent web scraping from functioning. From my investigation, it seems that certain URLs processed with the downloader produces 403 Forbidden errors which can be reproduced in web browsers when the link is opened directly within the browser. However, when the link is opened by opening up the post then opening the link in another tab within that post, the link opens successfully (note that this link is the same link that gives the 403 errors). Afterwards, opening the same link will not give a 403 error but this may be due to caching. The lack of cookies leading to this issue is just speculation as it is the major difference between a successful request and a forbidden request in their request header. However, it is certain that some modifications to the download protocol is needed. I will continue to work on a solution which should hopefully be ready some time this month.

From some investigation, I believe that the root cause of the issue is that an adjustment was made to click events on images. When you click on an image, an API call is made to a server which flips an access flag for your IP to true. One of the API's response headers suggests that this is the case:

Why IP address and not cookies? Both chrome and firefox (different cookies) both see a post as forbidden. The moment that an API call is made, the post becomes accessible on both browsers immediately.
The main issue is deciphering the API call. An example call is included below (Safe content which is just a poll):
https://kemono.party/onomek?download=https://kemono.party/data/d5/97/d5970115cb05cd4c8c811010fbaf991c902c6dd2a1a29d9e77a098283994ebea.png?f=K-009.png&idsite=1&rec=1&r=345242&h=12&m=30&s=37&url=https://kemono.party/patreon/user/17913091/post/69241378&urlref=https://kemono.party/patreon/user/17913091&_id=f2cd244219ced97e&_idn=0&send_image=0&_refts=1664980598&_ref=https://www.google.com/&pdf=1&qt=0&realp=0&wma=0&fla=0&java=0&ag=0&cookie=1&res=2560x1440&pv_id=l6OxAh
Parsed format:

I haven't yet delved deep enough into these parameters; however, they are a pain to deal with. One such issue is the r paramater which seems to be some sort of random number. It'll take some time to figure out how the API call works. An alternate solution aside deciphering the API call is using Selenium. However, Selenium is slow (software controlled browser, think of a fast human downloading content for you). It's going to take some time for a solution on top of other things in life so an update won't be out for some time.


Adds metrics for network bandwidth on all requests

Adds metrics for request and response size in bytes, measured from the rust object and not the encoded objects.

Couple of quick comments:

Can we also add an info line / span around each of the input functions in the authority_server to do some good old fashion logging on the basic method / time / maybe tx and basic parameters?
I am a little wary of computing the sizes in a different way from what we are actually serializing / deserializing / transmitting / receiving. It also adds a lot of code to maintain, which we might forget to update. Maybe @huitseeker & @bmwill could provide some pointers on how to re-use the length of the data that we are receiving / sending (I am also not familiar with the grpc framework sadly).

I am a little wary of computing the sizes in a different way from what we are actually serializing / deserializing / transmitting / receiving. It also adds a lot of code to maintain, which we might forget to update. Maybe @huitseeker & @bmwill could provide some pointers on how to re-use the length of the data that we are receiving / sending (I am also not familiar with the grpc framework sadly).

We can use the TraceLayer in tower-http to get this info i believe. But if we do migrate away from grpc (which i believe may be the plan) then we'll need to re-do some of this work with the new stack.

Can we also add an info line / span around each of the input functions in the authority_server to do some good old fashion logging on the basic method / time / maybe tx and basic parameters? (if that is too much data - it might - we could sample 1/100 or 1/1000?)

Sure, such as the time taken for each request.

I am a little wary of computing the sizes in a different way from what we are actually serializing / deserializing / transmitting / receiving. It also adds a lot of code to maintain, which we might forget to update.

Yes, after going through and seeing the density of our structures that we are sending around, another approach would be strongly preferred. @bmwill let's sync on future plans for the networking stack.


Clarify direction on vignettes

See also:

#893 (Currently in vignettes_old: Not sure what needs to be updated.)
#1233

Should we work on R examples that parallel the python, rather than focusing on stand-alone vignettes?
Or would some vignettes still be useful, even if the main goal is to present the APIs in parallel?

Now that we have an approach for embedding test-able R examples in multi-tabs, I think it would be better to consolidate on multi-tabs throughout. Vignettes will have the same tradeoff as embedded notebooks, in that they only support one language, and the text will have to be duplicated. This makes me dislike having a lot of our documentation in notebooks, and I'm hesitant to commit to doing the same in R for vignettes. I also don't like the vignette editing experience with the mixed pseudo-latex formatting.

Great -- Just filed #1258 for the follow-up work. Since the direction is clear, this can be closed.


Uri.toString() for "mailto" does not encode spaces properly (+ instead of %20)

Hi.
This ticket was first opened on flutter bug tracker: https://github.com/flutter/flutter/issues/68406
It seems Uri.toString() convert spaces to + in subject and body for mailto scheme.
final uri = Uri(
  scheme: 'mailto',
  path<EMAIL_ADDRESS>  queryParameters: {
    'subject': 'Example Subject & Symbols are allowed!'
  }
);

print(uri.toString());
// mailto:smith@example.com?subject=Example+Subject+%26+Symbols+are+allowed%21

This seems to be a problem, when the link is opened with url_launcher for example then the subject displayed in the mail app is like this:

Example+Subject+&+Symbols+are+allowed!

For some reason, the app does not handle + symbols in the uri as expected. However, it works fine if %20 is used instead.
I tested this with url_launcher v5.7.5, flutter v1.23.0, dart v2.10.0, Android v8.0.0 and opening the link with Gmail app.
Should the spaces be automatically converted to %20 instead to ensure portability?

The Dart Uri class does not recognize most schemes specially. The mailto: scheme is treated the same as any other scheme, so the query part is encoded using URL encoding, which is why it uses + for spaces.

Same here. How to fix it?

The lazy approach would be:
print(uri.toString().replaceAll("+", "%20"));

A more intricate approach could be:
final uri = Uri.parse('mailto:smith@example.com?subject=' +
   'Example Subject & Symbols are allowed!'.replaceAll(" ", "%20"));

print(uri.toString());

or
final uri = Uri(
  scheme: 'mailto',
  path<EMAIL_ADDRESS>  query:
    'subject=Example Subject & Symbols are allowed!'.replaceAll(" ", "%20")
);

print(uri.toString());

There currently is no way to use the queryParameters argument and get what you want. There is no way to configure the URI to not use standard URL encoding for query parameters.
(If we do a complete rewrite of the Uri class at some point, which we want to for several reasons, mainly compatibility with the browser Url class, then we will likely start recognizing "web" URIs - http and web sockets - specifically, and not treat all other URIs as if they are necessarily the same as HTTP. Until such time, we are unlikely to make any breaking changes to the Uri class.)

I'm personally constructing the uri string myself by calling Uri.encodeComponent() for each parameter.
final subject = Uri.encodeComponent("Example Subject & Symbols are allowed!")
final uri = "mailto:smith@example.com?subject=$subject";

It seems to work fine.

The Dart Uri class does not recognize most schemes specially. The mailto: scheme is treated the same as any other scheme, so the query part is encoded using URL encoding, which is why it uses + for spaces.

This is backwards though, as the generic form of URL encoding is percent encoding (click on your link there, then click on "URL encoding" in that text, which leads to https://en.wikipedia.org/wiki/Percent-encoding). The issue isn't that mailto is getting special handling, it's that all schemes are being treated as if they are HTML form query strings, which is where the use of + comes from.

There is no way to configure the URI to not use standard URL encoding for query parameters.

The correct statement is that there is no way to configure Uri to use standard URL encoding for query parameters, rather than applying HTML form encoding to every scheme.

A more intricate approach could be:
final uri = Uri.parse('mailto:smith@example.com?subject=' +
   'Example Subject & Symbols are allowed!'.replaceAll(" ", "%20"));

print(uri.toString());

or
final uri = Uri(
  scheme: 'mailto',
  path<EMAIL_ADDRESS>  query:
    'subject=Example Subject & Symbols are allowed!'.replaceAll(" ", "%20")
);

print(uri.toString());

These suggestions are both incorrect, as they will not encode anything else. Notably in this case, the subject will just be "Example Subject ", because the unencoded & will end that query argument.

Until such time, we are unlikely to make any breaking changes to the Uri class.

Perhaps an alternate constructor that does the right thing as an interim solution? This issue makes it very hard for people to correctly use the Uri class for anything that's not an http* URI.

(Also, perhaps [the encodeComponent](docs https://api.dart.dev/stable/2.12.4/dart-core/Uri/encodeComponent.html) could be updated in the short term; currently they say "For encoding the query part consider using encodeQueryComponent." which is actively misleading advice for any non-http URI. It could say "For encoding the query part of an HTTP or HTTPS URI, consider ...")

The simplest safe solution I can see at this point to use Uri for anything that has query parameters but isn't an http* URL is to manually re-implement what queryParameters should do inline, using encodedComponent. E.g.:
final parameters = <String, String>{
  'subject': 'Example Subject & Symbols are allowed!',
  'body': 'Here too! &+=',
};
final uri = Uri(
  scheme: 'mailto',
  path<EMAIL_ADDRESS>  query: parameters.entries.map((e) => '${Uri.encodeComponent(e.key)}=${Uri.encodeComponent(e.value)}').join('&'),
);
print(uri.toString());

For url_launcher I think we'll probably need to provide a utility method to do that query encoding, since that's a lot to put in a recommended code samples.

Until such time, we are unlikely to make any breaking changes to the Uri class.

Perhaps an alternate constructor that does the right thing as an interim solution? This issue makes it very hard for people to correctly use the Uri class for anything that's not an http* URI. (Exhibit A: the fact that your example workarounds are not actually right, and will break things in other ways.)

Or, alternately, switch on the scheme and keep the existing behavior for http/https, and do a breaking change for the others? While it is a breaking change, it's hard to imagine many scenarios where anyone would prefer the current behavior for other schemes.

https://url.spec.whatwg.org/ is the controlling standard here and it describes our current behaviour as invalid. Since we are not following the standard, we should fix our behaviour accordingly.

Changing our behavior to match the whatwg spec was one of our plans for Dart 2.0, but it didn't pan out (we significantly over-estimated how much change we could do in the time available). See https://github.com/dart-lang/sdk/issues/29420
It's still a worthy goal, and would make web compilers save a lot of code, but also a significantly breaking change to anyone using the existing Uri class.
One strategy (which we have used before with varying success) could be to copy the existing Uri class to a package, and re-implement the API with whatwg-URL compatible semantics in the platform libraries. People depending heavily on the existing behavior can migrate to the package then.
It's going to be a big undertaking. The Uri class is the largest class in the platform libraries, by a significant margin. The uri.dart file is larger than all other .dart files in the SDK except the auto-generated html_dart2js.dart. It's bigger than http_impl.dart!
(Reducing the complexity was one of the reasons for rewriting Uri, to not have to do copious amounts of up-front normalization by default).

Can you comment on the possiblity of doing one of the two small changes I suggested above (a new named constructor for non-web Uris, or a breaking change for the constructor just for non-web schemes)?
I understand there are other issues that make a major rewrite desirable, but if we could avoid letting the perfect being the enemy of the good in fixing this particular issue in the short term it would make it feasible to use Uri directly for common cases like mailto:, which currently is not the case.

a highly breaking change to anyone using the existing Uri class.

I don't see how making our behaviour less broken could be considered a breaking change. Could you elaborate on the flow that you're concerned would break?

If we don't normalize on creation (and the whatwg spec doesn't), then people relying on creating two Uris and checking whether they are equal will start seeing different results for things that were equivalent, but not equal.
That is, unless we make == normalize first, or do a comparison which recognize equivalences of characters, which is probably as expensive as normalizing (except for not allocating the result, but therefore also having to do it every time you call ==).

A new constructor which does only rudimentary normalization (and the normalize() would have to do full normalization) could be an option.
Perhaps Uri.raw(String scheme, String content) where we behave "normally" for http, https, ws, wss, and file URIs, perhaps also package ones (with normal being Uri.parse("$scheme;$content")). For the rest, we create a URI with scheme as scheme and content as path, no matter what the content contains. I guess we need to make % escapes valid, and escape non-printable ASCII characters, possibly also encode non-ASCII letters.
Something like:
  /// Creates a URI with the given scheme and content.
  ///
  /// The [scheme] must be a valid, non-empty scheme and
  /// the [content] is everything which occurs
  /// after the colon following the scheme.
  ///
  /// If [scheme] is a *recognized scheme*,
  /// one of `http`, `https`, `ws`, `wss`, `file` or `package`,
  /// the URI is normalized as normal for `Uri.parse("$scheme:$content")`.
  /// Otherwise the URI is assumed to be non-hierarchical and
  /// the only normalization which happens is scheme normalization,
  /// percent-escape normalization of existing escapes,
  /// and percent-escaping of non-visible-ASCII characters of the content,
  /// where non-ASCII characters are UTF-8 encoded first.
  ///
  /// The entirety of the normalized content will be accessible as the [path]
  /// of the created URI.
  ///
  /// Example:
  /// ```dart
  /// var uri = Uri.raw("Unknown", "#!%%0a?/\xa0::\n[]");
  /// print(uri.path); // #!%25%0A?/%C2%A0::%0A[]
  /// ```
  factory Uri.raw(String scheme, String content) => ...

That should cover mailto URIs and other unknown URI schemes. WDYT?

A new constructor which does only rudimentary normalization (and the normalize() would have to do full normalization) could be an option.

I'm confused; I don't think this issue has anything to do with normalization. AFAICT it's purely a question of whether queryParameters provided in the constructor are encoded using percent escaping (standard URI encoding; what encodeComponent currently does) or HTML form encoding (what encodeQueryComponent currently does).
Consider this code:
// Case a.
final parameters = <String, String>{
  'subject': 'a & b',
  'body': 'c & d',
};
var uri = Uri(
  scheme: 'mailto',
  path<EMAIL_ADDRESS>  queryParameters: parameters,
);
print(uri.toString());

// Case b.
uri = Uri(
  scheme: 'mailto',
  path<EMAIL_ADDRESS>  query: parameters.entries.map((e) => '${Uri.encodeComponent(e.key)}=${Uri.encodeQueryComponent(e.value)}').join('&'),
);
print(uri.toString());

// Case c
uri = Uri(
  scheme: 'mailto',
  path<EMAIL_ADDRESS>  query: parameters.entries.map((e) => '${Uri.encodeComponent(e.key)}=${Uri.encodeComponent(e.value)}').join('&'),
);
print(uri.toString());

It prints:
mailto:smith@example.com?subject=a+%26+b&body=c+%26+d
mailto:smith@example.com?subject=a+%26+b&body=c+%26+d
mailto:smith@example.com?subject=a%20%26%20b&body=c%20%26%20d

The first two are identical (presumably the first is actually using encodeQueryCompontent internally?), and wrong. The third is the correct encoding—if it's being normalized internally on construction, that normalization is harmless for the purposes of this issue.
I'm simply suggesting a new named constructor that is identical to the Uri constructor except that it does percent encoding—not HTML form encoding—for queryParameters.

Perhaps Uri.raw(String scheme, String content) [...]
That should cover mailto URIs and other unknown URI schemes. WDYT?

That seems equivalent to Uri.parse(Uri.encodeFull('$scheme://$content')), so doesn't really solve this problem; the developer is still entirely on their own in correctly handling query arguments.

To elaborate on the background here, as an illustration of the use case I'm trying to solve: the url_launcher plugin for Flutter takes a URL (currently as a String, but ideally a Uri in the future), and hands it off to the OS for handling. These can be anything, but are commonly http(s), mailto, or sms. For mailto (and to a lesser extend sms) we frequently see issues because many, many developers—unsurprisingly, as it's non-trivial—do not understand how to correctly construct correctly non-trivial URLs. The two most common failures are:

var myURL = 'mailto:$emailAddress?subject=$subject&body=$body'; - Usually there are spaces in subject or body so this isn't even remotely valid as a URL; on iOS at least this will simply fail to launch.
var myURL = Url.encodeFull([Insert string from above here]); - Works a lot of the time, but fails for certain special characters, most notably & as discussed above (it will launch, but won't have either the subject or body they intended).

To help users avoid these pitfalls, we added an example to the plugin README that demonstrates how to correctly (we thought; I didn't realize that Uri used the wrong encoding method for queryParameters) construct a URL with query parameters using Uri without having to understand intricacies of how individual components of URLs are encoded. That example is the code block that's in the OP here.
My goal here is to have a similarly-high-level example code to replace that incorrect example. Given that the issue here is with Uri, my preference would be for that similarly-high-level code to be part of Uri rather than the plugin providing its own URL construction helper.
Anything that involves the developer already having had to correctly construct a properly encoded query string from individual, unencoded components doesn't meet that goal.

I don't see why we need a new constructor. The current one is just behaving incorrectly. I could see an argument for accepting an optional named boolean parameter, which is basically what the standard does, that enables the +-for-space thing, but that's about it IMHO.

Changing broken behaviour should not be a non-starter. What we have today is a bug. We are corrupting user data.

The Uri class was never WHATWG URL compliant, and has never tried to be. It's implementing RFC 3986, and it probably predates the WHATWG URL spec by at least a year.
The encoding of queryComponents is intended to match the behavior of a browser submitting a form with those values. It uses "www-form-urlencoded" representation of the query components deliberately to do that. It's not the only possible option, but itt was the one chosen at the time.
If that's not what users are expecting, then that is a usability problem, but it is not a bug.
We can add a flag to the constructor to make it use percent normal encoding instead of +-for-space. That's non-breaking as long as it chooses www-form-urlencoded as the default.
In the longer run, we should do a full breaking change to become WHATWG URL compatible. It would help our JS compilers and probably confuse users less. (Again, I'd have preferred to do that at the Dart 2.0 release, but it didn't make the cut).
Maybe we should introduce a Url class for that, to follow the WHATWG naming as well.
That will likely still be a breaking change if we change the signature of methods expecting Uri to expect Url, even if we keep Uri and makes it implement Url. Not an easy change, no matter how we cut it (maybe if we can get extension types with constructors).

RFC 3986 doesn't specify any sort of + encoding for 0x20. It's actually less compliant if the target is that RFC. Indeed, that RFC defines a normalization scheme that considers + and %20 in the query component to be distinct. There is a nod to the concept of "scheme-based normalization" which could have (though wasn't) used to define this +-for-space encoding convention for http and https in section 6.2.3, but that's really the closest that the RFC comes to this concept and it's not sufficient to claim that our behaviour is conforming to that RFC.

That's non-breaking as long as

The current behaviour is already broken. This isn't a situation where we are doing something valid but want to change to something else valid and we have to worry about compatibility. We are literally misencoding ASCII space and causing user data corruption as a result. It's just a bug.

We can add a flag to the constructor to make it use percent normal encoding instead of +-for-space. That's non-breaking as long as it chooses www-form-urlencoded as the default.
It's easy for the Uri constructor, harder for Uri.http/Uri.https because they already have an optional positional parameter, so adding another boolean parameter would need to make it positional too.
For those, a breaking change might actually be easier, and then we might as well break the default for every constructor taking a queryParameters argument.

What's the next step here? E.g., is there a formal process for deciding whether to make the breaking change here?

What's the next step here? E.g., is there a formal process for deciding whether to make the breaking change here?

@lrhn Ping on this question? We're continuing to have users bitten by this issue, and I need to know if this is going to be fixed in Dart, or a workaround added to url_launcher.

I guess the next step is either:

A breaking change request (followed up by implementing a breaking change to the current behavior), or
Deciding on a usable non-breaking workaround and implementing that.

The breaking change can be either:

Just changing the behavior to not do form-encoding (spaces become %20 instead of + when encoding the query).
Do that, but add an extra flag to trigger the old behavior, so users who depend on it (if any) can keep using it.
Only changing the behavior for non-http/https schemes (maybe a small list of other similar schemes). Likely needs the flag.

A non-breaking change could be:

Adding the flag to allow changing the behavior, but keep the current default (for constructors only, it's breaking to add it to, say, replace).
Create an entirely new Url class with different behavior. (Long term goal!)

The easiest to implement is just a clean break switching to new behavior. If you want + in your query, you need to put it there yourself.
The least-breaking-but-still-breaking approach would be to change the behavior for non-http-like URIs, but keep the current behavior for http-like ones.
So the next step is to decide which solution to go for, then we can file a breaking change request for that (if necessary, which is probably likely, the non-breaking approaches are not great).

Any updates on this? having this problem with the url_launcher plugin

Just changing the behavior to not do form-encoding (spaces become %20 instead of + when encoding the query).
Do that, but add an extra flag to trigger the old behavior, so users who depend on it (if any) can keep using it.

Either of these options sounds good to me. Who are the other stakeholders who need to weigh in prior to making the breaking change request?

Ping. :eyes:


gateway fqdn is not working CLI

Describe the bug
grid cli, v0.11.2
the gateway fqdn command is not working.
tf-grid-cli deploy gateway fqdn -n gatewaytest --node 14 --backends http://<IP_ADDRESS>:80 --fqdn example.com

the fqdn you provide should point to the node 14 ip


Invalid data request format when creating service with network

Description
Creating a service using a network results in the error "Invalid data request format"
Steps to reproduce the issue:
Create a new service
Choose a network
Hit "Create service"
Technical details:
Portainer version: 1.14.0
Target Docker version (the host/cluster you manage): 17.09.0-ce, build afdb6d4
Platform (windows/linux): Debian 8
Command used to start Portainer: docker run -p 9000:9000 portainer/portainer
Browser: Chrome stable

Hi @JackHull
Thanks for the report, unfortunately I'm not able to reproduce this. Could you give me more details? Also, could you try with the latest version of Portainer (1.14.3) ?

ping @JackHull

Closing this issue due to inactivity.


Log Rotation

I'm having some trouble interpreting the Log Rotation Setting documentation. Specifically the rotate_age option. The docs specific this can be an integer or string value. My understanding is if this field is an integer value, the field indicates "how many logs to keep before removing the oldest" but, when this field is a string, (ex "daily") the field indicates "when to rotate a log file". These don't seem like they are the same function to me?
Essentially what I'm trying to achieve is a daily log rotation, keeping only the last 7 logs (1 week). If I were to set the rotate_age to "daily", I'm not sure how I would specify the "age" (number) of files to keep. Is this configuration possible with some combination of these settings?

Sorry, wrong repository