Search SDK: Indexes.ExistsAsync faulting

I am getting a TargetInvocationException on an index with the innerExcpetion of "Cannot get SerializationBinder because an ISerializationBinder was previously set." from Newtonsoft.Json.
The code is pretty straight forward:
        try
        {
            if(await SearchClient.Indexes.ExistsAsync(INDEX))
            {
                Logger.LogInformation("{0}", "Deleting index..\n");
                await SearchClient.Indexes.DeleteAsync(INDEX);
            }
            if(await SearchClient.DataSources.ExistsAsync(DATA_SOUCE))
            {
                Logger.LogInformation("{0}", "Deleting data source...\n");
                await SearchClient.DataSources.DeleteAsync(DATA_SOUCE);
            }
            if(await SearchClient.Indexers.ExistsAsync(INDEXER))
            {
                Logger.LogInformation("{0}", "Deleting indexer...\n");
                await SearchClient.Indexers.DeleteAsync(INDEXER);
            }
        }
        catch (Exception ex)
        {
            Logger.LogError("Error deleting indexing resources: {0}\r\n", ex.Message);
            throw;
        }

Any help is appreciated
rich
PS - I'm using Windows.Azure.Search (2.0.6-preview)

It appears that downgrading to 9.0.1 from 9.0.2-beta resolves the issue

@fuocor We're aware of this issue. It seems like a compatibility issue with the latest beta of JSON.NET, but we haven't been able to track it down yet.

thanks

@fuocor I'm having trouble getting a minimum repro of this problem. Would you be willing to share your code, or whatever minimal subset of it triggers the issue? Just calling ExistsAsync in a console app doesn't seem to be enough.

I am getting the same error. Following this example:
https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-csharp-template?toc=%2Fazure%2Fvirtual-machines%2Fwindows%2Ftoc.json
Console app .NET Core 1.1.0
My project.json file content:
{
"version": "1.0.0-*",
"buildOptions": {
"emitEntryPoint": true
},
"dependencies": {
"Microsoft.Azure.Management.ResourceManager": "1.4.0-preview",
"Microsoft.IdentityModel.Clients.ActiveDirectory": "3.13.6",
"Microsoft.Rest.ClientRuntime.Azure": "3.3.3",
"Newtonsoft.Json": "9.0.2-beta1",
"Microsoft.NETCore.App": {
"type": "platform",
"version": "1.1.0"
},
"Microsoft.Azure.Management.Storage": "5.0.2-preview",
"Microsoft.Azure.Management.Network": "8.0.0-preview",
"Microsoft.Azure.Management.Compute": "13.1.0-prerelease"
},
"runtimes": {
"win10-x64": {}
},
"frameworks": {
"netcoreapp1.0": {
"imports": [
"dnxcore50",
"portable-net45+win8"
]
}
}
}

Just downgrade to 9.1 of NewtonSoft to work around it
Rich
Sent from my Windows 10 phone
From: Larry<EMAIL_ADDRESS>Sent: Sunday, November 20, 2016 6:14 PM
To<EMAIL_ADDRESS>Cc: Richard<EMAIL_ADDRESS><EMAIL_ADDRESS>Subject: Re: [Azure/azure-sdk-for-net] Search SDK: Indexes.ExistsAsync faulting (#2552)
I am getting the same error. Following this example:
https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-csharp-template?toc=%2Fazure%2Fvirtual-machines%2Fwindows%2Ftoc.json
Console app .NET Core 1.1.0
My project.json file content:
{
"version": "1.0.0-*",
"buildOptions": {
"emitEntryPoint": true
},
"dependencies": {
"Microsoft.Azure.Management.ResourceManager": "1.4.0-preview",
"Microsoft.IdentityModel.Clients.ActiveDirectory": "3.13.6",
"Microsoft.Rest.ClientRuntime.Azure": "3.3.3",
"Newtonsoft.Json": "9.0.2-beta1",
"Microsoft.NETCore.App": {
"type": "platform",
"version": "1.1.0"
},
"Microsoft.Azure.Management.Storage": "5.0.2-preview",
"Microsoft.Azure.Management.Network": "8.0.0-preview",
"Microsoft.Azure.Management.Compute": "13.1.0-prerelease"
},
"runtimes": {
"win10-x64": {}
},
"frameworks": {
"netcoreapp1.0": {
"imports": [
"dnxcore50",
"portable-net45+win8"
]
}
}
}

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/Azure/azure-sdk-for-net/issues/2552#issuecomment-261813909, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AO7kwRGkFBq18Uqfy1VU5SR2QapUuqUKks5rANRLgaJpZM4K0sJB.

Thanks @laultman. This confirms that it is a more general issue not specific to the Azure Search SDK. I'll update the title of this issue accordingly and try to get the right people looking into it.
In the meantime, the workaround is to downgrade as @fuocor suggested.

FYI @hovsepm @markcowl

Description
Get a minimal repro of the issue and determine the root cause.  File issues for fix.
Cost: 2

Thanks guys you save me a lot of time.

Ah snap, I have the same problem with Newtonsoft.Json v10.0.2
Downgrading to 9.0.1 fixes the problem.

Also seeing this issue with Newtonsoft.Json v10.0.2.

Is there any fix in sight for this bug? It's not always possible to downgrade to Newtonsoft v9.
For instance, F# 4.1 requires Newtonsoft.Json v10.0: https://github.com/JamesNK/Newtonsoft.Json/issues/1180

@markcowl Any update on this?

Just in case it's a useful data point, I've encountered this in what might be an error scenario (with Json.NET 10.0.2) because it looks like a CloudErrorJsonConverter was involved:
System.AggregateException: One or more errors occurred. ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.InvalidOperationException: Cannot get SerializationBinder because an ISerializationBinder was previously set.
   at Newtonsoft.Json.JsonSerializer.get_Binder()
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.RuntimePropertyInfo.GetValue(Object obj, Object[] index)
   at Microsoft.Rest.Azure.JsonSerializerExtensions.WithoutConverter(JsonSerializer serializer, JsonConverter converterToExclude)
   at Microsoft.Rest.Azure.CloudErrorJsonConverter.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Microsoft.Rest.Serialization.SafeJsonConvert.DeserializeObject[T](String json, JsonSerializerSettings settings)
   at Microsoft.Azure.Management.Compute.VirtualMachinesOperations.<GetWithHttpMessagesAsync>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Valhalla.Web.SystemTests.Tests.Pre.DeprovisionSingleVmSet.<CheckVmInAzure>d__9.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Valhalla.Web.SystemTests.Tests.Pre.DeprovisionSingleVmSet.RunWeb()
   at Valhalla.Web.SystemTests.Tests.TestBaseWithWebDriver.RunCore()
---> (Inner Exception #0) System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.InvalidOperationException: Cannot get SerializationBinder because an ISerializationBinder was previously set.
   at Newtonsoft.Json.JsonSerializer.get_Binder()
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.RuntimePropertyInfo.GetValue(Object obj, Object[] index)
   at Microsoft.Rest.Azure.JsonSerializerExtensions.WithoutConverter(JsonSerializer serializer, JsonConverter converterToExclude)
   at Microsoft.Rest.Azure.CloudErrorJsonConverter.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Microsoft.Rest.Serialization.SafeJsonConvert.DeserializeObject[T](String json, JsonSerializerSettings settings)
   at Microsoft.Azure.Management.Compute.VirtualMachinesOperations.<GetWithHttpMessagesAsync>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Valhalla.Web.SystemTests.Tests.Pre.DeprovisionSingleVmSet.<CheckVmInAzure>d__9.MoveNext()<---

@fuocor @brjohnstmsft This is firmly in plan to look at in May, and from there we will have a plan to fix in runtime and/or generated code.  It is absolutely in our mission to support the latest version

This is great news as this issue is impacting our development here (downgrade JSON is not an option right now).
Thanks

thanks, just hit this. Saved me tons of times!

Guys, any updates on that? using v10+ still having the same error...
Thank you.

@galvesribeiro Which Azure library are you using, and which version is it?

@brjohnstmsft
<PackageReference Include="Microsoft.Azure.Management.Fluent" Version="1.0.0" />
<PackageReference Include="Newtonsoft.Json" Version="9.0.1"/>

I can only get it to work with 9.0.1. I tried with 10+ and same error as state on this issue.

@galvesribeiro It might be because Microsoft.Azure.Management.Fluent depends on an older version of Microsoft.Rest.ClientRuntime.Azure. Try adding a binding redirect to version 3.3.6 and see if that helps.

@brjohnstmsft I'm on .Net Core... No binding redirects :(

@galvesribeiro Got it. Sorry for the inconvenience. :(
Since my last reply I've seen an update from @markcowl that this work is still in progress.

Ok, thanks @brjohnstmsft will keep the older version for now.

Same here. The problem occurred when I added 2 fields to an index. I was on NewtonSoft.Json 10, a downgrade to 9 made it work again.
Probably it works when updating to NewtonSoft.Json 10 if the index is untouched. Changing the index causes the problem.

@bottomup Note that this issue covers all Azure .NET libraries. It sounds like you're talking about the Azure Search library in particular -- is that correct? If so, which version are you using?

@brjohnstmsft correct. I'm using Microsoft.Azure.Search 3.0.3.
However, I'm also using other Microsoft.Azure packages, but didn't experience these issues there.

@bottomup Can you try upgrading to 3.0.4 and let me know if you still experience the problem?

I'm having the same issue with Azure Search. Upgraded to 3.0.4 and I get the error "Cannot get SerializationBinder because an ISerializationBinder was previously set.". Using Newtonsoft.Json version 10.0.2.

Its important to note that this is not a particular nuget issue... Any nuget from Azure SDK that is using 9.0.1 and you try to use 10.x.x+, you will get that problem...

Thanks for the info. So do we need to wait until a new version comes out that uses 10.x.x+ ? I can't downgrade to 9.

Yeah, unfortunately... I did rollback to 9+ for a particular project that use those packages in order to be unblocked...

I tried adding the following binding redirect, but it didn't work:
<dependentAssembly>
  <assemblyIdentity name="Microsoft.Rest.ClientRuntime.Azure" publicKeyToken="31bf3856ad364e35" />
  <bindingRedirect oldVersion="<IP_ADDRESS>-<IP_ADDRESS>" newVersion="<IP_ADDRESS>" />
</dependentAssembly>

@markcowl Any idea when a new version will be released which supports NewtonSoft.Json v10?

@brjohnstmsft I tried your suggestion and upgraded to 3.0.4, but the same issue arises. Downgraded again to NewtonSoft.Json 9.0.1

@bottomup Thanks for confirming. I wasn't sure if the issue with the Azure Search SDK is related to the issue affecting all Azure .NET SDKs, but it looks like it is related. @markcowl is working on a fix.

Just adding to this - I see this, but only when I generate an invalid search query in the Azure Search SDK - then it comes back with this error (rather than the "real" error).

This is an issue in ClientRuntime.Azure and will be fixed shortly.

@hovsepm Thanks Hovsep. The Azure Search SDK does some custom JSON serialization stuff as well, so once the issue in ClientRuntime.Azure is fixed, I still have to verify whether more work needs to be done to fix the issue in Microsoft.Azure.Search.

@hovsepm: Will #3304 solve the other nugets (like compute, networking, etc...) as well? As I mentioned, it is a general issue...

We misunderstood the root cause for this, which is why we had the additional delay.  Hovsep's fix will allow those using NetCore and those using binding redirects in net452 to use Newtonsoft.Json.  Those who cannot use binding redirects will need us to republish the individual management libraries, which will happen next month.
@brjohnstmsft Code the depends on ClientRuntiem exception handling should be fixed automatically with an upgrade.  Code that uses properties deprecated in Version 10 will have to be updated.

@galvesribeiro Those libraries are compatible with the new runtime, so you should be able to upgrade.  Note that you will need to be using NetCore or assembly binding redirects to take this fix, because the assembly identity for Newtonsoft.Json has changed.

@markcowl thankfuly I'm free from binding redirect hell by using .Net Core :)
Thanks! Looking forward for the update.

The issue is resolved. Please use Microsoft.Rest.ClientRuntime.Azure v 3.3.8 or later.

@hovsepm @markcowl Not sure this should be closed until the libraries depending on Microsoft.Rest.ClientRuntime.Azure have been republished, unless there is another issue tracking this work...? I'd love to republish the Search libraries, but I see that AzSdk.reference.props in the vs17Dev branch hasn't been updated with the new references to Microsoft.Rest.ClientRuntime.Azure and NewtonSoft.Json yet. What's the ETA for this?

@shahabhijeet

@hovsepm No, but the minimum required version is 3.3.6, not 3.3.8. A few of our customers have already reported running into this issue as soon as they upgrade to JSON.NET 10. Referencing 3.3.8 resolves it, but in the meantime it creates extra support traffic for us.

@brjohnstmsft following [fix] (https://github.com/JamesNK/Newtonsoft.Json/pull/1341) fixes underlying error. New set of Clientruntime libraries are on track to be republished by next week.
In the mean time, upgrading to newtonsoft.json 10.0.3 will fix the issue whoever wants to upgrade to newtonsoft 10+

@shahabhijeet Awesome, thanks for the update!


[Bug] java.lang.InternalError: Memory Pool not found

Search before asking

[X] I had searched in the issues and found no similar issues.

Apache SkyWalking Component
Java Agent (apache/skywalking-java)
What happened
report the error per second
ERROR 2022-03-24 10:51:56:698 SkywalkingAgent-4-JVMService-produce-0 JVMService : JVMService produces metrics failure.
java.lang.InternalError: Memory Pool not found
at sun.management.MemoryPoolImpl.getUsage0(Native Method)
at sun.management.MemoryPoolImpl.getUsage(MemoryPoolImpl.java:96)
at org.apache.skywalking.apm.agent.core.jvm.memorypool.MemoryPoolModule.getMemoryPoolMetricsList(MemoryPoolModule.java:57)
at org.apache.skywalking.apm.agent.core.jvm.memorypool.MemoryPoolProvider.getMemoryPoolMetricsList(MemoryPoolProvider.java:48)
at org.apache.skywalking.apm.agent.core.jvm.JVMService.run(JVMService.java:101)
at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
What you expected to happen
don't report error
How to reproduce
version 8.3.0
Anything else
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

I found you've also issued a ticket there https://github.com/alibaba/arthas/issues/2120
Does the same solution work this time?

We are just accessing through JDK XBean.
This looks like a JVM internal error. Not sure what we can do.

I solved the problem of increasing memory ago in arthas ,but this The remaining memory is 200M, still report error，I'll try updating a JDK version and observe


Check the lazyload on medias

Acceptance criteria:

[ ] all images are lazy-loaded - start
[ ] if it's a UI issue - create one in UI repository and reference it here as a blocker

Checked on baby category, media are lazy-loaded.


Is there an easy way to Switch account?

Is there a better way to do when creating multiple repo with multiple user
i keep getting this error
Error creating repository: Not Found (HTTP 404)
Not Found

only the first github account would work and all the organization on it...
i hope there is an easy way to switch an account
like hub auth-switch
FYI
My ssh config is properly configure , and i can do the normal stuff like pushing and pulling no trouble
only when creating a new repo is that im facing it
im using ssh protocol by default...
Can someone sugguest me an easier way of doing this...
SO i wont need to go to each github account login then logout
then create new repo

Hi, there is currently no functionality to easily switch users. You can hack something up yourself:

Have two authentication tokens (belonging to separate users) available, then swap the value of GITHUB_TOKEN environment variable around in your shell to switch between users;
Store two separate authentication tokens in separate config files (see ~/.config/hub), then swap the value of HUB_CONFIG when you want hub to use an alternate config file.


Query with time < now() produces negative timestamps and null data

InfluxDB 1.3.2 on Debian 9.
This query worked as I expect it to (only return existing data and fill between first and last points) in 1.2.x, but now it starts with completely odd negative timestamp (like -9223372036) which breaks my stuff entirely.
Was this an expected change, as I don't see anything particular in the changelog.
The full query is: SELECT time, MAX(value) FROM concurrent_players_daily WHERE appid = '440' AND time < now() - 1d GROUP BY time(1d) fill(linear)
This appears to directly related to GROUP BY clause as removing fill(linear) does not change anything.

We added the ability to query from times before 1970 so the lower limit of a query is now approximately -9223372036. I don't know if that's the exact number, but I believe it's close to there.
You must specify a lower limit. There is a bug fix in 1.4 that will throw an error if you don't specify a lower limit on this type of query though to prevent this kind of error.

@jsternberg So what's the approach running a query that would return all the stored data? The query without a lower bound worked before upgrading to 1.3.

I'm not sure what you mean, but the lower timestamp has been that way since 1.0. The issue documenting the fix is in #7212.
If you want something that aggregates all data, you can use fill(none) and that will not generate the extra intervals when there are no points.
It also appears like I may be wrong about the lower limit in 1.4. I was going to add that limitation to prevent overflowing queries, but it seems like I never did and I'll avoid adding that limitation so that this kind of query can work. But, keep an eye out for the changelog on this because it's very easy for someone to forget to limit their query and create a slow query if they don't specify fill(none).

This definitely worked on 1.2 as I used the exact same query which started returning negative timestamps on 1.3 with the exact same query, so something must have changed in that regard.

I'm curious why this was working for you in 1.2. Can you give me an example with some sample data so I can confirm this isn't a regression? The behavior I discussed should have existed since 1.0.

The query I used is in the first post. Data would be just couple of numbers for couple of days. I don't think the data even matters.

@jsternberg So what would be a solution to select all existing data and grouping it without knowing the specific timestamp where the data starts?
I can't do something like this either because influxdb does not allow selecting just time:
time > (SELECT first(value) FROM concurrent_players_daily WHERE appid = '440')

@xPaw if you can change fill(linear) to fill(none), that would skip empty intervals. However it also changes the behavior for missing intervals in the more recent data.
@jsternberg Any luck reproducing this?

@mark-rushakoff - can we make sure we have this query pattern in h/t?

I bisected it back to timezone support: 347b018.
I'll start looking into it.

So... this gets a little trippy. There are two bugs. There are two bugs in 1.2 and one bug in 1.3. The two bugs, in combination with each other, caused the behavior that @xPaw saw. That behavior is incorrect in the current query engine.
Before I continue, I actually see @xPaw's concern and I think we should consider ways to make his behavior be correct. I've got an idea, but I'd prefer to focus on fixing the bugs first.
Here's the fixed bug:
        if itr.opt.Ascending {
-               itr.window.time = p.Time + int64(itr.opt.Interval.Duration)
+               itr.window.time += int64(itr.opt.Interval.Duration)
        } else {
-               itr.window.time = p.Time - int64(itr.opt.Interval.Duration)
+               itr.window.time -= int64(itr.opt.Interval.Duration)
+       }

So the timezone feature fixed one of the two bugs because I was running around that section of the code. The next interval for a fill iterator is supposed to be based on the last interval. Both sections of code should work the same when there are no other bugs.
But, I discovered a second bug. The initial timestamp of the interval window is being calculated incorrectly with Window(). When the time is the MinTime, this underflows and resets to the highest time. The fill iterator has a feature where if points are interspered before the expected next window, it will output them. So the fill iterator is calculating, due to underflow, that the window should be completely empty, but the fill iterator is still letting other points through because of that other feature.
We're going to need to fix the underflow problem first and then I think we can make it so when the fill iterator sees the min time, it calculates the time from the first point in the interval instead of using the start time. We use that in some other areas too so we already have precedent for doing that.

I appear to have made a mistake while looking at the 1.2 code. There were two bugs, but the underflow bug was present in 1.2 and was fixed later so it doesn't exist in 1.3.
I'll start looking into a way that we can make a non-existent lower bound to be the most recent window. About a month ago, I merged a change that made it an error not to specify a lower bound to avoid this kind of issue.
Sorry about that. I actually forgot I made that change and didn't realize it until I tried the query on the master branch today. I'll try to turn the 1.2 behavior from what was a bug into something that is a feature.

@xPaw can you try #8866 and see if that works for you?

@jsternberg I am not familiar with running influxdb from source, it might be too much hassle. But the PR looks solid to me.


Update set-up-your-lab.md

Updated Security > Windows Authorization to Windows Authentication

@soyalejolopez : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#sign-off
Once merged, this should be live with the next daily publishing process. Right now that should be tomorrow (Wednesday, 10 July) around 11:00 Pacific time. No further action needed from you.


SSL: CERTIFICATE_VERIFY_FAILED - when using SplunkStream output mode

When using outputMode = splunkstream I get SSL issues
root@eventgen:/eventgen# python bin/eventgen.py conf/data.conf
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/eventgen/lib/outputworker.py", line 40, in run
    self.worker.run()
  File "/eventgen/lib/outputworker.py", line 74, in run
    self.real_run()
  File "/eventgen/lib/outputworker.py", line 96, in real_run
    plugin.flush(deque(queue[:]))
  File "/eventgen/lib/plugins/output/splunkstream.py", line 110, in flush
    splunkhttp.connect()
  File "/usr/local/lib/python2.7/httplib.py", line 1263, in connect
    server_hostname=server_hostname)
  File "/usr/local/lib/python2.7/ssl.py", line 363, in wrap_socket
    _context=self)
  File "/usr/local/lib/python2.7/ssl.py", line 611, in __init__
    self.do_handshake()
  File "/usr/local/lib/python2.7/ssl.py", line 840, in do_handshake
    self._sslobj.do_handshake()
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)

I've raised a pull request which fixes the issue for me.
https://github.com/splunk/eventgen/pull/65

Can you let me know if this is still an issue in eventgen 6?


Typo

"Microsoft 365 compliance enter" in two places

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 643cb341-cf88-69d6-561a-04f0eff3fe3e
Version Independent ID: fe07a988-a832-92d8-9852-9956ecb2f655
Content: Close, reopen, and delete Core eDiscovery cases - Microsoft 365 Compliance
Content Source: microsoft-365/compliance/close-reopen-delete-core-ediscovery-cases.md
Service: o365-seccomp
GitHub Login: @markjjo
Microsoft Alias: markjjo

@hwkwicre Fixed and will be live by EOD. Thanks for heads up.


MCMC tutorial

Added an MCMC jupyter notebook to the tests.

I'm having trouble importing seaborn. I'm not sure this is my problem or something generic. I get the following warning message:

It would be good to make the ticks and labels larger here:

I think it would be good to clarify that the "data" you are using is the shear auto-spectrum without tomography (one big redshift bin) and the range of ells you are using.

I would consider adding some baseline shape noise in the covariance. It would make this more realistic. The relevant numbers are in confluence.

I get an error at the following point, there seems to be a problem with the definition of icov. Do you know where this might be coming from?

I'm having trouble importing seaborn. I'm not sure this is my problem or something generic. I get the following warning message:

It seems like you're using an old version of ipython (4.0) the latest stable version is 6.1. Could you try to update it and try again?

I'm having trouble importing seaborn. I'm not sure this is my problem or something generic. I get the following warning message

It seems you have an old version of ipywidgets
Can you try pip install ipywidgets and run again?

It would be good to make the ticks and labels larger here:

Things will look different once seaborn is imported, can you update this when that works? (you might still have the same comments...)

Have you tried changing the burn-in requirement? Is 50 sufficient?

I would consider adding some baseline shape noise in the covariance. It would make this more realistic. The relevant numbers are in confluence.
I actually used the covariance from there previously, but is it all okay to make public? I thought it might be safer to use data that is not super meaningful yet good enough as a proof of concept throughout.

@hsnee @elisachisari @mishakb I got stuck when running following:
In [ * ] :  sampler = emcee. Ensemblesampler( nwalkers, ndim, Inprob)
Run 50 steps as a burn-in

It seems not getting through. I tried the notebook both on my Mac and on NERSC. Both stop at this step. I checked that emcee has been installed on my Mac and on NERSC. And ipywidgets is already installed in my Mac.
If I stop and run it, that step will get through, but it will get stuck again in the next step. The first step of importing the modules run successfully. Am I missing anything?

Hi @WeikangLin, How exactly does it get stuck? Does it just take a large amount of time to run, or does the kernel die? I would expect that step to take ~2 hours on a NERSC node so if it's just taking a lot of time, that's normal. If the kernel dies, though, the RAM is most likely not enough to run it.

Hi @hsnee, thanks! It keeps running, and the kernel is alive. So, it maybe just taking a long time. I will let it run for a while, and let you know how it goes.

Hi @hsnee , After running the notebook on NERSC for~ 2 hours, the kernel dies.

Hi @hsnee @WeikangLin @elisachisari I have been testing the notebook at NERSC and with the default transfer function (CLASS) the kernel dies after a while (I don't know exactly why yet but I think it's because I am using an old version of the master branch). In any case, I have a couple of suggestions:

I would change the transfer function type to 'eisenstein_hu'. This way the notebook runs faster (and it doesn't die).
I think it would be useful to add one line with the timing for the function lnprob, e.g., you can add a cell with:

%time lnprob(p0[0,:])
This will give you the time that it takes to evaluate the likelihood for one iteration. You can also do %timeit lnprob(p0[0,:]) and it will evaluate several iterations and give you the best/worst cases and the average time.

Hi @hsnee @fjaviersanchez @elisachisari, I changed the transfer function type from "boltzmann" to "bbks" (as @fjaviersanchez suggested, but I used bbks instead), and the notebook finishes in ~20 mins. So, it is probably not enough RAM as @hsnee suggested?
If we copy the notebook to a python script, and submit a job to NERSC, would it help? I currently don't know how to submit a job on NERSC yet. Any tutorial about it?

@WeikangLin It will definitely help. You can request one or more nodes with 128 GB each and I don't think that will run out of memory. You can find some examples on how to send jobs to NERSC here: http://www.nersc.gov/users/computational-systems/cori/running-jobs/batch-jobs/
Also about the RAM, I think that might be true. All the jupyter jobs at NERSC run in certain node (that is shared with a lot of users) and running out of RAM is a possibility. I sometimes run the same program at night Pacific Time with no problem and during business hours it says memory error...

making a .py file out of it and running it should work, but it kind of goes against the fact that this is supposed to be a tutorial rather than a script (but at least for testing it is helpful). You can also get an allocation for an interactive job on NERSC http://www.nersc.gov/users/computational-systems/cori/running-jobs/interactive-jobs/, which you can then use to run the jupyter notebook (this is how I've run the tutorial in the past -- as @fjaviersanchez said running it on the head node might be troublesome memory-wise if a lot of other people are using a lot of memory at the same time.)
also thanks for the suggestions on using a different transfer function and %time! I'll edit the notebook to include them.

Thanks @hsnee and @fjaviersanchez for the links.

@hsnee @fjaviersanchez @WeikangLin It looks like one option concerning this PR is to use BBKS in the notebook for now and address the memory leak in a separate issue. Are you happy with this plan? Would this be ready to merge then?

Personally, I am happy with this plan and I think that this is ready to merge once these changes are addressed.

I just pushed the notebook to the mcmc branch with the suggestions edited in -- do you think it's ready?

@elisachisari I think this is ready to merge. @WeikangLin what do you think?

@fjaviersanchez @hsnee @elisachisari, The notebook tutorial reads good, and I have run the notebook on NERSC successfully. I think it is ready to merge too.


[Type Resolution] Emit a tailored diagnostic for the incorrect any P? syntax.

The compiler now emits the following error for the incorrect any P? syntax, with a fix-it to insert parenthesis:
protocol P {}

var p: any P? = nil // error: optional 'any' type must be written '(any P)?'

Resolves: rdar://90558740

@swift-ci please smoke test


chore: update build tasks, package.json and changelog

PR Checklist

[ ] Unit Tests have been added for new changes
[ ] All commented code has been removed
[ ] If you've added a dependency, you've ensured license is compatible with Apache 2.0 and clearly outlined the added dependency.

What are you changing?

Anything the reviewer should know when reviewing this PR?
If the there are associated PRs in other repositories, please link them here (i.e. open-amt-cloud-toolkit/repo#365 )

:tada: This PR is included in version 5.6.10 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Fixes related to ITensorMPS module

Fixes of test errors in test/ITensorMPS/Ops/ and the ITensorGaussianMPS package.
Also attempting to fix the error seen when running the ITensorGPU tests.

Codecov Report
Attention: Patch coverage is 0% with 3 lines in your changes are missing coverage. Please review.

Project coverage is 53.34%.  Consider uploading reports for the commit d4b4826 to get more accurate results

Files
Patch %
Lines

src/ITensorMPS/dmrg.jl
0.00%
3 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1362      +/-   ##
==========================================
- Coverage   53.45%   53.34%   -0.11%
==========================================
  Files          99       99
  Lines        8452     8452
==========================================
- Hits         4518     4509       -9
- Misses       3934     3943       +9

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[test ITensors mps]

[test ITensorGaussianMPS]

So the test failure here for the MPS tests is strange: apparently allequal was not defined in Base until Julia 1.8. But it is used in some @assert's in the file abstractprojmpo/projmposum.jl. So I'm surprised these tests were ever passing.
The question is whether we should:

replace the use of allequal with something else there or
raise the minimum Julia version to 1.8

I'd be fine with either but I think for now better to just replace allequal if we are just using it in a small number of places.

[test ITensors mps]

[test ITensorGaussianMPS]

I went ahead and replaced allequal but let's discuss which choice would be best.

So the test failure here for the MPS tests is strange: apparently allequal was not defined in Base until Julia 1.8. But it is used in some @assert's in the file abstractprojmpo/projmposum.jl. So I'm surprised these tests were ever passing.
The question is whether we should:
1. replace the use of `allequal` with something else there or

2. raise the minimum Julia version to 1.8

I'd be fine with either but I think for now better to just replace allequal if we are just using it in a small number of places.

I've been using the Compat.jl package to address issues like that. Compat.jl makes some subset of newer Julia functionality (like allequal) available in older versions of Julia. Probably it was working before because we load Compat in ITensors.jl, but it isn't being loaded in the ITensorMPS module now. So we could add using Compat: allequal at the top of the abstractprojmpo/projmposum.jl file.

[test ITensors mps]

We should change the trigger comment for this workflow to [test ITensorMPS], it just requires changing this line: https://github.com/ITensor/ITensors.jl/blob/v0.3.57/.github/workflows/comment_trigger_test_itensors_mps.yml#L12.

[test ITensorGaussianMPS]


Remove the GAEN Error From Last Mile Failure

This PR removes the GAEN error message for the last-mile failure since it can't be resent.  It must be resent via resubmission from the sender.
Test Steps:

./gradlew testIntegration

Changes

Include a comprehensive list of changes in this PR
(For web UI changes) Include screenshots/video of changes

Checklist
Testing

[x] Tested locally?
[x] Ran ./prime test or ./gradlew testSmoke against local Docker ReportStream container?
[ ] (For Changes to /frontend-react/...) Ran npm run lint:write?
[ ] Added tests?

Process

[ ] Are there licensing issues with any new dependencies introduced?
[ ] Includes a summary of what a code reviewer should test/verify?
[ ] Updated the release notes?
[ ] Database changes are submitted as a separate PR?
[ ] DevOps team has been notified if PR requires ops support?

Linked Issues
- Fixes #issue
To Be Done
Create GitHub issues to track the work remaining, if any

#issue

Specific Security-related subjects a reviewer should pay specific attention to

Does this PR introduce new endpoints?

new endpoint A
new endpoint B

Does this PR include changes in authentication and/or authorization of existing endpoints?
Does this change introduce new dependencies that need vetting?
Does this change require changes to our infrastructure?
Does logging contain sensitive data?
Does this PR include or remove any sensitive information itself?

If you answered 'yes' to any of the questions above, conduct a detailed Review that addresses at least:

What are the potential security threats and mitigations? Please list the STRIDE threats and how they are mitigated

Spoofing (faking authenticity)

Threat T, which could be achieved by A, is mitigated by M

Tampering (influence or sabotage the integrity of information, data, or system)
Repudiation (the ability to dispute the origin or originator of an action)
Information disclosure (data made available to entities who should not have it)
Denial of service (make a resource unavailable)
Elevation of Privilege (reduce restrictions that apply or gain privileges one should not have)

Have you ensured logging does not contain sensitive data?
Have you received any additional approvals needed for this change?

@oslynn I concur with @GilmoreA6's suggestion above. Please update the documentation so it's clear why we're ignoring certain errors.


ConPTY: process not exist  after I close tab or terminus

I use windows 10,
When I close tab or terminus, many process is not exist, they still running in background.

Apparently ConPTY is still not stable enough in Windows builds before 18309, so I'll be adding a corresponding warning to the settings UI.


Different layout for the footer on mobile?

Currently, the footer start/center/end sections are always placed next to each other. On mobile, when using more than 1 section (as the default does using start/end, or our own docs use even all three), this can become quite "crammed":

Should we start allowing those to be on separate lines (vertically instead of horizontally) at a certain breakpoint?

We should be using the existing breakpoints to determine this layout change.
Related - it would be good to also do an overall pass at the theme for mobile and small screens as I know we have recently uncovered a number of items and layouts that needed adjustments for these.
We could add this to the accessibility related working items.

Tagging @smeragoel for visibility as this will more than likely need some design input


Adding Rebus Carpool

Added Rebus with hint.
Related Issue: #121

Pull Request Test Coverage Report for Build 898

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 87.143%

Totals

Change from base Build 514:
0.0%

Covered Lines:
158

Relevant Lines:
180

💛  - Coveralls


Optimized a bunch of imports

Mainly, removed * wildcards from imports.

LGTM.
Do imports like java.util.* instead of explicit class incur additional penalties in general?

I don't know if there are additional penalties with *, but it's usually easier to understand imports that don't have it.


Add This Amazing Blocklist.

Please! rather than maintaining a blocklist of you own you can use the tiarap dns blocklist containing over 300k domains . this list is update everyday by the tiarap dev .
Here's the File

Also can you please try to block recaptcha and those captchas


Examples website is not available

Hello,
I wanted to have a look at examples on redis. However it seems the live version on Vercel is currently removed.
Best.

this example is throwing errors as well

hello, @Bakhtiyar-Garashov, we are having a look, sorry for the late response. Thank you for bringing this to our attention.
@hofewo is there a specific example that doesn't work for you?

If it helps with anything, it's not a problem with redis, but nextjs.
getServerSideProps() is called multiple times.
Verify this:
export async function getServerSideProps() {
const data = await redis.incr('counter');
console.log(`getServerSideProps.data: ${data}, ${new Date().toISOString()}`);
return {  props: {  hits: data,  },  };
}
So the result isn't reliable 😞


mobile: allow api users to decide when to SwapBuffers.

While fixing app on x11/egl and porting it to x11/glx I found some need for a redesign.
See https://go-review.googlesource.com/#/c/1851 for some discussion.
Right now the idea is to always vsync on back buffer swap.
This will make any app run at the display devices (probably) fixed refresh rate.
An app that takes 1.1 frames to render will wait 0.9 frames for the next vblank causing your app to run at half of the display's refresh rate when you could be running much faster.
Refresh rates are not required to be 60hz, so you would have to measure your framerate every frame to figure out if you are running slower or faster.
I recommend letting the user trigger a draw and swap and the user running their own timer.
The triggers for draw and swap should be separate so people have the choice to use single buffered contexts (which may be making a comeback oddly).
For running your own timer I recommend timer.NewTicker().
I found it works perfectly as a frame timer with desktop gl.
For controlling whether SwapBuffers() vsync's
This simple function is available with nearly the same definition for egl/glx/wgl.
https://www.khronos.org/registry/egl/sdk/docs/man/html/eglSwapInterval.xhtml
It just takes an integer >= 0 that is the number of vblanks SwapBuffers should wait for.
Something similar exists for darwin but ios looks a bit ugly.

I am unwatching/subscribing from this issue, if no one else cares you can delete it.


[MBL-871] Add compose visuals based on design system

📲 What
Added many default views to be used in the future, added a screen where you can preview all the default views and typography
🤔 Why
We need to prepare for our future visuals building
🛠 How
Compose and a bit of learning
👀 See
In App on internal builds: Go to internal tools and design system
Story 📖
MBL-871

We could probably add to jacoco.gradle any new or modified folder containing only UI, that or create a spike ticket to explore UI testing with compose. The only UI testing we currently have it's screenshot testing and it's not adapted to compose yet.

Small comments but overall looks fantastic!


connection to msssql 2014

Hello
i have problem to connnect  to mssql 2014 enterprise in Always on group
Error in plugin [inputs.sqlserver]: read tcp x.x.x.200:44787->x.x.x.41:34001: read: connection reset by peer
server is with SSL certificate
connection string looks like that
"Server=x.x.x.x.41;Port=34001;User Id=telegraf;Password=telegraf;app name=telegraf;log=32;certificate=ca-chain.crt;",
Server force encryption and i try with encrypt=true and  TrustServerCertificate=true  and is same problem

try installing latest service pack

we try it ..
but is same
is this driver works on Always on group ? i check alot of setups but no one talks about always on

Yes,  we use it with an Always On Availability Group in a mission-critical, high-volume, production  environment.  However, we don't use SSL certificates.  So I think the problem must be with your certificate-based authentication rather than the AG functionality.

Ah, wait, one important thing to note:
AG's work by having a multiple-IP DNS name, with the addresses of all the servers in the cluster.  The driver tries to connect simultaneously to all of them.  The current master will accept a connection and all the others wont.  Therefore, you cannot use a fixed IP address to connect to an AG cluster.

Have you tried it with a DNS name for the AG cluster rather than an IP address?

we use DNS name
"Server=hostname.domain.com;Port=25001;User Id=telegraf;Password=telegraf;app name=telegraf;log=2;TrustServerCertificate=true;encrypt=true;",
anything else that im missing ?

If the DNS name in question resolves to all of the IP addresses in the cluster, that's all that you should need from an AG standpoint.  We don't use encryption so possibly that is causing some problems, but I wouldn't expect so.


Can't find type for var used in c:Ecto.Adapter.checkout/3

Current Behavior
I'm using Ecto as the "test repository" for the next release of IntelliJ Elixir.  As part of that, I've noticed, I can't resolve var in
defmodule Ecto.Adapter do
  ...
  @callback checkout(adapter_meta, config :: Keyword.t(), (() -> result)) :: result when result: var
end

I thought var might be a built-in typespec I was just missing, but I don't see it listed.
Further enforcing var as unresolvable is that ex_doc doesn't generate link for it either.

Expected Behavior
If var is a built-in type ex_doc should link to it the way it does for term, etc.  If that is the case, this is an ex_doc bug and can be transferred there.

If it's not a built-in type, then its definition is missing and the callback needs to be fixed by adding a @type var :: ... that is can refer to or the result:  changed to result: term() or some other existing type.

Hi @KronicDeth! var is not listed as a built-in type because it is not really a type but rather a type variable. The docs mention it here:

Type variables with no restriction can also be defined using var.
@spec function(arg) :: [arg] when arg: var

https://hexdocs.pm/elixir/typespecs.html#defining-a-specification

We could define it as a builtin type but I am not quite sure it would be correct. Perhaps we should change ExDoc to handle it anyway?

If var only works in a when clause (I've been calling that a "type restriction") then making it a built-in type is probably wrong as it can't be used in all places that term() or any() would work.

What is it being translated to in spec that Dialyzer can read?
Or is var inherited from Erlang so it already works for Dialyzer as var?

I can special-case it in IntelliJ Elixir. I already have special handling for the optional and required pseudo-functions used for map keys, so that they just don't generate references and therefore aren't marked as unresolvable.  I can do the same for var.
Two of the reasons to make something a built-in type or have a fixed definition in general is so that you can (1) do Go To Definition, so you can understand it, but also going the opposite direction (2) Find Usages of the type.  I don't think it makes sense to try to Find Usages of var since each usage is potentially a different concrete type.
Given the above reasoning I think the follow changes would make sense

[ ] ex_doc

[ ] Link var in a type restriction to https://hexdocs.pm/elixir/typespecs.html#defining-a-specification. If that isn't close enough, we'll have to modify the format of those docs to give us a closer anchor.  This will mirror built-in types linking to https://hexdocs.pm/elixir/typespecs.html#built-in-types.

[ ] Add a hover text "No restriction" to mirror "Built-in Type" shown when hovering over built-in types.

[ ] IntelliJ Elixir

[ ] Return null for var's reference.

Does this sound like a good course of actions @josevalim?
Would you like me to make the ex_doc changes?

If var only works in a when clause

This is the case.
In Erlang, there is no such concept, because you would write this:
  @callback checkout(adapter_meta(), config(), (() -> Result)) :: Result

But because in Elixir there is ambiguity between a variable and a type, then such conflict arises. It is really a flaw in your typespecs syntax.
That said, I trust your judgement here. So feel free to send PRs that you find relevant. A PR for ExDoc is certainly welcome!

I have a test passing in the ex_doc repo, but it's getting late here. I'll test it against Ecto tomorrow.

Potential nested commas, parentheses, braces, and brackets are messing with a correct solution when.  Trying a non-Regex solution that builds up the Macro.to_string output instead, but it's a larger change 😬

Thank you! Closing this here then as it is not an Ecto issue. :)


feature: barbar.nvim

Did you check the docs?

[X] I have read all the lazy.nvim docs

Is your feature request related to a problem? Please describe.
My workflow often requires that I can move tabs with files of the same type together. bufferline.vim does not have the functionality to move tabs forward and backward. This is why I have to use barbar.nvim.
Of course I have placed a feature request for bufferline.vim to add the required functions. But apparently this would require to rewrite bufferline.vim, which I do not expect to happen.
It would be great if lazy.nvim could also support barbar.nvim.
Describe the solution you'd like
It would be great if lazy.nvim could also support barbar.nvim.
Describe alternatives you've considered
Using barbar.vim.
Additional context
Very often I do open configuration files (Neovim, Docker, Kubernetes, Saltstack, Linux) of two or more different configurations to compare them and to copy configuration parts from one file to another.
Example: find ~/.config/nvim.{1,2} -name *.lua | xargs vim
This opens all lua files from the neovim configurations nvim.1 and nvim.2. Then I want to compare the files of the same type. So I want to move the tabs for all init.lua together and the same for all options.lua and keyaps.lua.
Of course I can open a split and compare the files of the same type side by side. But sometimes it is better to switch between files of the same type w/o using window splits and w/o cycling through many open buffers / tabs.
barbar.vim provides the functions BufferMoveNext, BufferMovePrev, BufferMoveStart and BufferMoveEnd of barbar.nvim which can be mapped to keys to easily move buffers.
This is the reason why I am using barbar.nvim and not bufferline.nvim
Thank you for sharing your great work if lazy.nvim!
Kind regards,
Uri

Hello @talentix-ch,
How is this feature request related to lazy.nvim, the plugin manager?

Of course this issue can be closed.


A bug in getting from peers

Reproduce steps
pool := groupcache.NewHTTPPool("http://<IP_ADDRESS>:"+os.Getenv("PORT"))
cache := groupcache.NewGroup("cacher", 64<<20, someGetterFunc)
pool.Set("http://<IP_ADDRESS>:50000", "http://<IP_ADDRESS>:50001")

and run it in two port :50000 and :50001
Now, if use /xxx as a key, then the groupcacher will act like there is no any peer.
i.e. It will only get data from locally, instead of getting from peers.
The problem is the slash in the head of the key (i.e. if the key is xxx then everything goes right), and I still can't figure out why.

This is 2015 and this bug is 2013, very funny. I began to use this in my minicdn project recently and meet this problem, too. I almost reviewed the whole code. The fix is very simple and I already send a pull request, waiting for merge.

Sounds like this was fixed already but this bug wasn't closed.


CTEs cause a database error when using fields with non-default names

Environment

Elixir version (elixir -v): 1.9.2
Database and version (PostgreSQL 9.4, MongoDB 3.2, etc.): MySQL 8
Ecto version (mix deps): 3.2.0
Database adapter and version (mix deps): myxql 0.2.9, ecto_sql 3.2.0
Operating system: MacOS 10.14.6

Current behavior
Using a CTE with a schema that has differently named fields in the schema (using source) causes the CTE to select wrongly named fields.
Here is an example:
  @primary_key {:id, :id, source: :F_GroupID, autogenerate: true}
  schema "T_Groupstructure" do
    field :name, :string, source: :F_GroupName

    # Group hierarchy
    belongs_to :parent, __MODULE__, source: :F_GroupParent
    has_many :children, __MODULE__, foreign_key: :parent_id
  end

  def ancestors_query(%Group{} = group) do
    initial_query =
      Group
      |> where([g], g.id == ^group.id)

    recursion_query =
      Group
      |> join(:inner, [parent], tree in "group_tree", on: parent.parent_id == tree.id)

    ancestors_query = union_all(initial_query, ^recursion_query)

    {"group_tree", Group}
    |> recursive_ctes(true)
    |> with_cte("group_tree", as: ^ancestors_query)
  end

This causes this error: ** (MyXQL.Error) (1054) (ER_BAD_FIELD_ERROR) Unknown column 'g0.F_GroupParent' in 'where clause'
Looking at the generated SQL, the problem is that the initial_query gets turned into something like:
SELECT T0.F_GroupID AS id, T0.F_GroupName as name (etc)
However, the main SELECT statement outside the CTE still uses the real names for the fields (e.g. SELECT G0.F_GroupID`), but by this point they have been renamed, causing the error.

Here is a full example of some generated SQL that fails:
WITH RECURSIVE `group_tree` AS (
 SELECT T0.`F_GroupID` AS `id`, T0.`F_GroupName` AS `name`, T0.`F_GroupType` AS `type`, T0.`F_DeletedAt` AS `deleted_at`, T0.`F_GroupParent` AS `parent_id` FROM `T_Groupstructure` AS T0 WHERE (T0.`F_GroupID` = 51)
 UNION ALL
 SELECT T0.`F_GroupID`, T0.`F_GroupName`, T0.`F_GroupType`, T0.`F_DeletedAt`, T0.`F_GroupParent` FROM `T_Groupstructure` AS T0 INNER JOIN `group_tree` AS g1 ON T0.`F_GroupParent` = g1.`id` WHERE (T0.`F_GroupID` = 51)
)

SELECT g0.`F_GroupID`, g0.`F_GroupName`, g0.`F_GroupType`, g0.`F_DeletedAt`, g0.`F_GroupParent` FROM `group_tree` AS g0

I think we could fix this by disabling source aliasing for recursive queries.

If you can point me in the right direction I could have a go at fixing this (no promised though :) )

I think it is an issue on Ecto.Query.Planner and unfortunately it is not straight-forward to fix it there, sorry. But if you want to give it a try, notice how some of the queries have the AS and the others do not, we need to find out why some do, and the others do not, and make them consistent.

Btw @ccapndave, it would help me a ton if you provide a sample app that reproduces this error. This way I can focus my efforts on fixing the bug instead of reproducing it. Can you provide so?

Sure thing - I'll do it first thing next week.

@josevalim Here is a minimal example with two tests - the first passes using a fragment with manual SQL, and the second fails using composition.  Hope it helps!
https://github.com/ccapndave/cte_161_example

@ccapndave I think I am missing something, because both queries in your example app is failing.
Here is MySQL version:
$ mysql --version
mysql  Ver 8.0.18 for osx10.14 on x86_64 (Homebrew)

I'm on the same version.  That's weird, let me double check it.

What error is it failing with?

➜  cte_161_example git:(master) MIX_ENV=test mix ecto.create
The database for Cte161Example.Repo has been created
➜  cte_161_example git:(master) MIX_ENV=test mix ecto.migrate
➜  cte_161_example git:(master) MIX_ENV=test mix test

  1) test it fails with composition (CTE161Test)
     test/cte_161_example_web/cte_161_test.exs:41
     ** (MyXQL.Error) (1054) (ER_BAD_FIELD_ERROR) Unknown column 'g0.F_GroupID' in 'field list'
     code: result = Repo.all(query)
     stacktrace:
       (ecto_sql) lib/ecto/adapters/sql.ex:629: Ecto.Adapters.SQL.raise_sql_call_error/1
       (ecto_sql) lib/ecto/adapters/sql.ex:562: Ecto.Adapters.SQL.execute/5
       (ecto) lib/ecto/repo/queryable.ex:177: Ecto.Repo.Queryable.execute/4
       (ecto) lib/ecto/repo/queryable.ex:17: Ecto.Repo.Queryable.all/3
       test/cte_161_example_web/cte_161_test.exs:57: (test)

.

Finished in 0.1 seconds
2 tests, 1 failure

Randomized with seed 672677

Sorry for not pasting the error:

  1) test it works with a fragment (CTE161Test)
     test/cte_161_example_web/cte_161_test.exs:21
     ** (MyXQL.Error) (1064) (ER_PARSE_ERROR) You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'RECURSIVE `group_tree` AS (SELECT * FROM T_Groups WHERE F_GroupID=?
                 ' at line 1
     code: result = Repo.all(query)
     stacktrace:
       (ecto_sql 3.2.1) lib/ecto/adapters/sql.ex:629: Ecto.Adapters.SQL.raise_sql_call_error/1
       (ecto_sql 3.2.1) lib/ecto/adapters/sql.ex:562: Ecto.Adapters.SQL.execute/5
       (ecto 3.2.5) lib/ecto/repo/queryable.ex:177: Ecto.Repo.Queryable.execute/4
       (ecto 3.2.5) lib/ecto/repo/queryable.ex:17: Ecto.Repo.Queryable.all/3
       test/cte_161_example_web/cte_161_test.exs:35: (test)

  2) test it fails with composition (CTE161Test)
     test/cte_161_example_web/cte_161_test.exs:41
     ** (MyXQL.Error) (1064) (ER_PARSE_ERROR) You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'RECURSIVE `group_tree` AS (SELECT T0.`F_GroupID` AS `id`, T0.`F_GroupName` AS `n' at line 1
     code: result = Repo.all(query)
     stacktrace:
       (ecto_sql 3.2.1) lib/ecto/adapters/sql.ex:629: Ecto.Adapters.SQL.raise_sql_call_error/1
       (ecto_sql 3.2.1) lib/ecto/adapters/sql.ex:562: Ecto.Adapters.SQL.execute/5
       (ecto 3.2.5) lib/ecto/repo/queryable.ex:177: Ecto.Repo.Queryable.execute/4
       (ecto 3.2.5) lib/ecto/repo/queryable.ex:17: Ecto.Repo.Queryable.all/3
       test/cte_161_example_web/cte_161_test.exs:57: (test)

Wait. It is something weird on my end:
{:ok,
 %MyXQL.Result{
   columns: ["@@version"],
   connection_id: 62,
   last_insert_id: nil,
   num_rows: 1,
   num_warnings: 0,
   rows: [["5.7.24"]]
 }}

It is running 5.7.24 even if I don'thave it anywhere in my system?

Ok, I got it. Thanks for confirming it was rather me. :D

So this is unfortunately not trivial to fix. The issue is that often you want to keep the aliases and it is hard to know when to keep or when not to. So you have two options:

Use a fragment
Use a query but build the select yourself %{F_GroupID: ...}

For now, I will document this limitation in the docs. Thank you for reporting and for the sample app.

Thanks very much for looking into it.  I don't think I'm going to be able to compose queries that way I want to using either fragments or manual selects, so I'm going to have a go creating VIEWs in the database that rename the columns to what Ecto expects.
Cheers!

In case anyone else encounters the same problem, I can confirm that making a VIEW with columns renamed to Ecto defaults works around this problem neatly.


tsconfig.json [compilerOptions.target] should be "es2019", but found "5".

"5" is ts.ScriptTarget.ES2018, pikapkg should transform back the enum value to "ES2018"

Bump...I'm getting the same error, but with different versions:
tsconfig.json [compilerOptions.target] should be "es2019", but found "8". You may encounter problems building.

+1, this should be an easy one if anyone wants to help tackle

Made PR #95 for that
» tsconfig.json [compilerOptions.target] should be "es2019", but found "es2015". You may encounter problems building.
» tsconfig.json [compilerOptions.module] should be "esnext", but found "commonjs". You may encounter problems building.

Thanks @yumauri!  Just merged, will do a release later today to get this out


duplicate entry 'code' in validation file for Media

Hi,
I've found a bug in Media.yml file :
properties:
        code:
            - NotBlank:
                message: 'bitbag_sylius_cms_plugin.media.code.not_blank'
                groups: ['bitbag']
            - Length:
                min: 2
                max: 250
                minMessage: 'bitbag_sylius_cms_plugin.media.code.min_length'
                maxMessage: 'bitbag_sylius_cms_plugin.media.code.max_length'
                groups: ['bitbag']
        code:
            - NotBlank:
                groups: ['bitbag']

there is 2 'code' entries, the second one should be removed.
Best regards,
David

Thanks


Suggestion: Implement pagination support

Hi, i like this package, ty for creating it!
Would it be possible to support pagination params?
Thank you :)

Pagination should be implemented for the various endpoints under the options param. Lmk if you see otherwise. @FlurryNight


Add support for output formatters using plugins

It would be great to have a nice bubbletea renderer, but instead of making glazed even bigger, let's start experimenting with plugins.

Maybe this would work: https://github.com/hashicorp/go-plugin

We could add plugins by loading wasm to run them inside geppetto...


[CIVP-10391] Add Python 2.7 compatibility

I think I addressed all of the comments.

@jseabold , I poked through the documentation and didn't see anything about installation requirements. Maybe it's worth mentioning Python version requirements (Python >= 3.4.2 or Python 2.7, AFAICT) in the "Installation" section of index.rst?

There's an os.makedirs call in civis/ml/_model.py which uses the "exist_ok" parameter. That's not present in Python 2.7. You'll need to remove the parameter and put it behind an if not os.path.exists.

There's an os.makedirs call in civis/ml/_model.py which uses the "exist_ok" parameter. That's not present in Python 2.7. You'll need to remove the parameter and put it behind an if not os.path.exists.

Fixed.

Added Python versions to docs

Comments should be addressed.


Autogenerate representation for .mp4 file

Hello Guys,
I am using your library to create HLS files. It works great. I want to create .mp4 to multiple bitrates using this same library. Is it possible? Can I do something like
$video->hls()
->encryption($save_to, $url)
->x264()
->setFormat($format)
->addRepresentations($rep)
->autoGenerateRepresentations([r_720p,r_240p])
->save(null , $destinationURL);
For mp4 source file and output as mp4 file itself in 720 and 240. I know I can use ffm-peg but from my understanding, I have to generate the individual file for each representation.
Regards,

"MP4 output format" is not in the scope of this project. You should use PHP-FFMpeg for this use case.
use FFMpeg\Coordinate\Dimension;
use FFMpeg\Filters\Video\ResizeFilter;
use FFMpeg\Format\Video\X264;

$ffmpeg = FFMpeg\FFMpeg::create();

$video = $ffmpeg->open('/var/media/video_1080p.mp4');

$video
    ->addFilter(new ResizeFilter(new Dimension(426, 240)))
    ->save((new X264)->setKiloBitrate(150), '/var/media/video_240p.mp4');

$video
    ->addFilter(new ResizeFilter(new Dimension(1280, 720)))
    ->save((new X264)->setKiloBitrate(2048), '/var/media/video_720p.mp4');

You can also use AutoReps class to generate the Representation objects as an iterator. After generating representations, you can encode MP4 format files into multiple bitrates.


New version: NextDNS.NextDNS.Prerelease version 3.0.4

[x] Have you signed the Contributor License Agreement?
[x] Have you checked that there aren't other open pull requests for the same manifest update/change?

[x] Does your manifest conform to the 1.2 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Closes #82792

Microsoft Reviewers: Open in CodeFlow

@wingetbot waivers Add Policy-Test-2.8


op_edit errors on comma three

Loaded SA-master on 8/23/2021 around 10:30am Eastern time
Tried running op_edit and got the following...
ccomma@tici:/data/openpilot$ python op_edit.py
Traceback (most recent call last):
File "op_edit.py", line 3, in
from common.op_params import opParams
File "/data/openpilot/common/op_params.py", line 7, in
from selfdrive.hardware import TICI
File "/data/openpilot/selfdrive/hardware/init.py", line 5, in
from selfdrive.hardware.eon.hardware import Android
File "/data/openpilot/selfdrive/hardware/eon/hardware.py", line 10, in
from cereal import log
ImportError: cannot import name 'log' from 'cereal' (unknown location)
comma@tici:/data/openpilot$
Then I tried it with sudo and got the following:
comma@tici:/data/openpilot$ sudo python op_edit.py
File "op_edit.py", line 237
print(msg, flush=True, end='\n' + end)
^
SyntaxError: invalid syntax

I'm wondering if it is still compiling.  It's been stuck on the screen with the big comma and no progress bar for a while...

Does it need to be run with sudo?

What does tmux a say when the logo comes up with no progress bar?

Waiting for wayland
wayland ready
adjusting time from '2021-07-21 15:00:39.508255' to '2021-08-23 15:15:00'
Mon 23 Aug 2021 03:15:00 PM UTC
INFO:root:System time valid
/data/openpilot has been modified, skipping overlay update installation
Cloning into '/data/openpilot/cereal'...
fatal: unable to access 'https://github.com/sshane/cereal.git/': Could not resolve host: github.com
fatal: clone of 'https://github.com/sshane/cereal.git' into submodule path '/data/openpilot/cereal' failed
Failed to clone 'cereal'. Retry scheduled
Cloning into '/data/openpilot/laika_repo'...
fatal: unable to access 'https://github.com/commaai/laika.git/': Could not resolve host: github.com
fatal: clone of 'https://github.com/commaai/laika.git' into submodule path '/data/openpilot/laika_repo' failed
Failed to clone 'laika_repo'. Retry scheduled
Cloning into '/data/openpilot/opendbc'...
fatal: unable to access 'https://github.com/commaai/opendbc.git/': Could not resolve host: github.com
fatal: clone of 'https://github.com/commaai/opendbc.git' into submodule path '/data/openpilot/opendbc' failed
Failed to clone 'opendbc'. Retry scheduled
Cloning into '/data/openpilot/panda'...
fatal: unable to access 'https://github.com/commaai/panda.git/': Could not resolve host: github.com
fatal: clone of 'https://github.com/commaai/panda.git' into submodule path '/data/openpilot/panda' failed
Failed to clone 'panda'. Retry scheduled
Cloning into '/data/openpilot/rednose_repo'...
fatal: unable to access 'https://github.com/commaai/rednose.git/': Could not resolve host: github.com
fatal: clone of 'https://github.com/commaai/rednose.git' into submodule path '/data/openpilot/rednose_repo' failed
Failed to clone 'rednose_repo'. Retry scheduled
Cloning into '/data/openpilot/cereal'...
fatal: unable to access 'https://github.com/sshane/cereal.git/': Could not resolve host: github.com
fatal: clone of 'https://github.com/sshane/cereal.git' into submodule path '/data/openpilot/cereal' failed
Failed to clone 'cereal' a second time, aborting
Traceback (most recent call last):
File "./build.py", line 10, in
from common.basedir import BASEDIR
File "/data/openpilot/common/basedir.py", line 4, in
from selfdrive.hardware import PC
File "/data/openpilot/selfdrive/hardware/init.py", line 5, in
from selfdrive.hardware.eon.hardware import Android
File "/data/openpilot/selfdrive/hardware/eon/hardware.py", line 10, in
from cereal import log
ImportError: cannot import name 'log' from 'cereal' (unknown location)

That looks like it is progressing.  Should I reboot after it finishes?

That did it.  Based on the tmux, it looks like the wifi had not connected yet when it was first trying to update the submodules.


Could not find gem 'wdm (>= 0.1.0)' in locally installed gems. (Bundler::GemNotFound) - Al-folio Docker Image

Describe the bug
[+] Running 2/2
 - Network jekyll-al-folio-blog_default  Created                                                                   1.0s
 - Container al-folio-website            Created                                                                   2.6s
Attaching to al-folio-website
al-folio-website  | bundler: failed to load command: jekyll (/usr/local/bin/jekyll)
al-folio-website  | /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/resolver.rb:293:in `raise_not_found!': Could not find gem 'wdm (>= 0.1.0)' in locally installed gems. (Bundler::GemNotFound)
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/resolver.rb:346:in `block in prepare_dependencies'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/resolver.rb:331:in `each'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/resolver.rb:331:in `map'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/resolver.rb:331:in `prepare_dependencies'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/resolver.rb:51:in `setup_solver'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/resolver.rb:28:in `start'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/definition.rb:554:in `start_resolution'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/definition.rb:289:in `resolve'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/definition.rb:507:in `materialize'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/definition.rb:197:in `specs'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/definition.rb:254:in `specs_for'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/runtime.rb:18:in `setup'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler.rb:170:in `setup'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/setup.rb:20:in `block in <top (required)>'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/ui/shell.rb:159:in `with_level'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/ui/shell.rb:111:in `silence'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/setup.rb:20:in `<top (required)>'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/cli/exec.rb:56:in `require_relative'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/cli/exec.rb:56:in `kernel_load'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/cli/exec.rb:23:in `run'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/cli.rb:491:in `exec'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/cli.rb:34:in `dispatch'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/cli.rb:28:in `start'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/exe/bundle:45:in `block in <top (required)>'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/exe/bundle:33:in `<top (required)>'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/exe/bundler:4:in `load'
al-folio-website  |     from /var/lib/gems/2.7.0/gems/bundler-2.4.6/exe/bundler:4:in `<top (required)>'
al-folio-website  |     from /usr/local/bin/bundler:23:in `load'
al-folio-website  |     from /usr/local/bin/bundler:23:in `<main>'
al-folio-website exited with code 1

To Reproduce
Steps to reproduce the behavior:

Cloned the repository
docker-compose up
Saw the error

Expected behavior
I expected the instructions given on the main page to work.
System (please complete the following information):

Host OS in which docker is running on is Win 10

Additional context
It seems that instructions in the Dockerfile installs Ruby 2.7.0.
Referring to the following instruction:
RUN apt-get install --no-install-recommends ruby-full build-essential zlib1g-dev -y
In addition, the following is the content of my Gemfile
source 'https://rubygems.org'
group :jekyll_plugins do
    gem 'jekyll'
    gem 'jekyll-archives'
    gem 'jekyll-diagrams'
    gem 'jekyll-email-protect'
    gem 'jekyll-feed'
    gem 'jekyll-imagemagick'
    gem 'jekyll-minifier'
    gem 'jekyll-paginate-v2'
    gem 'jekyll-scholar'
    gem 'jekyll-sitemap'
    gem 'jekyll-link-attributes'
    gem 'jekyll-twitter-plugin'
    gem 'jemoji'
    # gem 'mini_racer'
    gem 'unicode_utils'
    gem 'webrick'
end
group :other_plugins do
    gem 'httparty'
    gem 'feedjira'
    gem 'wdm', '>=0.1.0'
end

If you are using docker then you don't need gem 'wdm (>= 0.1.0)' in your gemfile.
I had this error when I moved from running bundle exec jekyll serve to docker-compose up.
Deleting the line from the gemfile solved the error.


Unable to open 'X.view.xml' with the layout editor

Please report a bug in our Jira with your project attached as zip file:
https://jira.tools.sap/projects/DEVXBUGS/issues
The right component is CA-BAS-W5G.

Hi. Since this is not a documentation issue, I'm closing the issue and encourage you to follow up by opening a JIRA ticket, as Licam described. Thanks and best wishes.


Add builtin yq support to limactl edit command

Also add --tty parameter, similar to the start command, for avoiding user interaction asking whether to start it.
$ limactl edit --tty=false --set ".cpus = 2" default
INFO[0000] Instance "default" configuration edited
$ limactl edit --tty=false --set ".cpus = 2" default
INFO[0000] Aborting, no changes made to the instance

Follow-up to:

#1359

I think limactl edit --set ... should not invoke the editor. I can't think of any situation where I want to set some settings via --set, but also want to use the editor as well; I would simply not bother writing the --set options in the first place.
If you ever wanted to do both (which I believe would be extremely uncommon, but maybe from a script), you could still call the command twice:
limactl edit --tty=false --set ".cpus = 3" default
limactl edit default

Even when --set is specified, we should use the --tty setting to determine if we should ask to start the instance or not.

Hmm, for start we open the editor after --set ? That was the behaviour that was copied here.

for start we open the editor after --set ?

No, we don't. We ask if the user wants to edit the YAML file. But I see the point that maybe that shouldn't happen either. But that should be a separate issue.
For edit we always open the editor right away (because that is the point of the command), so it is more jarring that I have to close the editor again even if I didn't want to open it (because I used --set). And if I specify --tty=false to prevent it, then it won't ask me to start the VM, and I have to start it manually as well.


Is possible run HLS Fair Play on Safari with useNativeHlsOnSafari: false?

What version of Shaka Player are you using?
2.5.21
Please ask your question
Native support for HLS on Safari has disadvantages eg. in custom HttpXHRPlugin I can't detect manifests or segments requests.
So, is possible run HLS Fair Play on Safari with useNativeHlsOnSafari: false?
If yes, have you any examples?

It is not possible to do this. Failed to get FairPlay to work with MSE yet.

ok, but generally is it possible for this time? If yes, is it on Shaka Player roadmap?

Correct.  We only support FairPlay with src= at the moment.  But it seems with #3346, it may be possible in the future.


serialization: Drop rootfs.type (which had only one legal value)

With 0c52d5de (serialization: remove windows-specific layers+base rootfs, 2016-08-26, #211) this field became a single-choice field.  We could keep the field to make the current "layers" approach explicit in the face of future rootfs types, but we'd have to solidify the wording around it to cover what should happen when rootfs.type was not set, or if rootfs.type was an unrecognized value.  It seems easier to remove it and make the "layers" approach implicit default, so that's what this commit does.  We can restore the field later if/when we have another rootfs type without breaking forward compatibility.
Code translating between Docker and OCI image renderings can drop/restore this field while it is translating Docker ↔ OCI media types, so I don't expect compatibility issues from this change.
Spun off from this discussion.

cc @stevvooe

👎
I made my opinion clear in https://github.com/opencontainers/image-spec/pull/211#discussion_r76713977. I'll repeat here, since you seem to have missed it:

This field was added as part of the modifications that came out with Docker 1.10 to address long standing image format issues. The rootfs provide image provenance over the root filesystem components that make up a given image. The type field provides some measure of algorithmic agility. Removing this field probably has unintended consequences.

Recently, this value for the field was removed from the docker landscape due to a better solution for identifying base image layers that could not be distributed. The field value would never have been seen in production ready versions of docker and has no place in OCI.

Note that, in practice, there is the possibility of encountering other values of this field, but it is unlikely. However, if they are encountered, that conversion cannot safely proceed. Removing this field is more likely to be actively harmful than leaving it in place.
I'm not defending whether or not this field is useful. I'm trying to stem the tide of unnecessary change. This is particularly pathological, as it is subtly different and not actively harmful.

@stevvooe I am having a hard time understanding the meaning of this response. You are saying that this field is never used and shouldn't be interpreted but we should keep it so we don't drop it in serialization and deserialization? Sorry, just having a difficult time understanding the danger in dropping it. Is there a concrete example?

On Wed, Aug 31, 2016 at 01:54:04PM -0700, Stephen Day wrote:

I made my opinion clear in
https://github.com/opencontainers/image-spec/pull/211#discussion_r76713977. I'll
repeat here, since you seem to have missed it…

I saw it, I'm just not convinced yet.  Handling this issue by itself
in this PR should let us get to the bottom of our concerns without
holding up other work.

… Removing this field probably has unintended consequences.

I was looking for some details behind “unintended consequences” 1,
since I don't see any.

Note that, in practice, there is the possibility of encountering
other values of this field, but it is unlikely. However, if they are
encountered, that conversion cannot safely proceed. Removing this
field is more likely to be actively harmful than leaving it in
place.

The current image-spec wording does not say anything like “if you see
an unrecognized value, do …”.  I'm not even sure what you're
suggesting should happen in that case.  “cannot safely proceed” sounds
like “die with an error” (which makes sense to me).  “leaving it in
place” sounds like “silently pass unrecognized values” (which does not
make sense to me).

I'm not defending whether or not this field is useful. I'm trying to
stem the tide of unnecessary change.

This spec is seeded by the Docker spec; it's not bound to exactly
match the Docker spec.  When the translation is unambiguous (replace
media types following this mapping, add/remove rootfs.type, …), I
don't see a problem with divergence.  Where that divergence makes the
OCI spec simpler (and keeps the mapping deterministic), I think we
want to do that.

I'm also having great difficulty understanding the rationale for keeping
this; if this value would never have been seen in production ready versions
of Docker, what is the concern? It seems like an extreme/obscure edge case.
If we really should retain it for some reason we should explicate why in
the spec.
On 31 August 2016 at 22:54, Stephen Day<EMAIL_ADDRESS>wrote:

👎
I made my opinion clear in #211 (comment)
https://github.com/opencontainers/image-spec/pull/211#discussion_r76713977.
I'll repeat here, since you seem to have missed it:
This field was added as part of the modifications that came out with
Docker 1.10 to address long standing image format issues. The rootfs
provide image provenance over the root filesystem components that make up a
given image. The type field provides some measure of algorithmic agility.
Removing this field probably has unintended consequences.
Recently, this value for the field was removed from the docker landscape
due to a better solution for identifying base image layers that could not
be distributed. The field value would never have been seen in production
ready versions of docker and has no place in OCI.
Note that, in practice, there is the possibility of encountering other
values of this field, but it is unlikely. However, if they are encountered,
that conversion cannot safely proceed. Removing this field is more
likely to be actively harmful than leaving it in place.
I'm not defending whether or not this field is useful. I'm trying to stem
the tide of unnecessary change. This is particularly pathological, as it is
subtly different and not actively harmful.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/opencontainers/image-spec/pull/224#issuecomment-243898097,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACewN9p2X_fiqOQ4pgJTY-vlvul6LJVyks5qlentgaJpZM4JwxBj
.

I never said it wasn't interpreted. Docker uses it to internally interpret the meaning of rootfs. The field's role is to declare the meaning of the DiffIDs array.  If this gets modified or fudged in any way, there is the possibility to open the format to malicious injection.
The fact is, the field exists in the wild, with varying values, for Windows container images. Though it isn't in "production", it may be seen as part of an earlier release. If these diff ids are interpreted incorrectly, an image may be incorrectly assembled. Retaining this field with this values makes it absolutely explicit what these diff ids actually mean.
This is particularly dangerous due to the way Go processes JSON, which would lead to this field being ignored in practice. If a config with a different value is encountered in the wild, it will be impossible to detect without making that definition explicit. Are you willing to take a risk that code converting from the docker image format always gets this subtlety just right?
I understand this may offend one's sense of aesthetics but leaving this in place and making it explicit is just the safest option.

@stevvooe Do you have links to the code or whatever that caused this issue? Then we can make a big footnote for future readers.

@philips Yea, let me see what I can find here.

On Wed, Aug 31, 2016 at 05:20:15PM -0700, Stephen Day wrote:

The field's role is to declare the meaning of the DiffIDs array.
If this gets modified or fudged in any way, there is the
possibility to open the format to malicious injection.

But as image-spec stands (with or without this PR), there's only one
possible meaning for the DiffIDs array.

Retaining this field with this values makes it absolutely explicit
what these diff ids actually mean.

As does not setting the field and referencing the config with a
descriptor holding the
application/vnd.oci.image.serialization.config.v1+json media type.

This is particularly dangerous due to the way Go processes JSON,
which would lead to this field being ignored in practice.

With the language in-flight in #164 (which I think should cover the
config JSON too 1), such unrecognized keys MUST NOT lead to errors.
If somebody points to a Docker config (including the rootfs.type
field, possibly with some other value) and claims it is an
application/vnd.oci.image.serialization.config.v1+json, they should
not be surprised when the tooling treats the referenced config as an
OCI config and ignores the Docker-only field.
The right approach is to translate the Docker config to an OCI config,
and then point at the OCI config and claim it's an
application/vnd.oci.image.serialization.config.v1+json.  But the
current PR doesn't make this clear.  I'll add notes to 2 tomorrow to
clarify the new translation step.

@wking I understand what you're saying and you're right from a theoretical perspective. However, we need to take a harm reduction approach to security. Introducing a potential security land mine then saying "whoops, should have read this part more carefully" isn't acceptable. The result is bad and we've gained nothing from having a "simpler" specification.

On Wed, Aug 31, 2016 at 06:52:40PM -0700, Stephen Day wrote:

Introducing a potential security land mine then saying "whoops,
should have read this part more carefully" isn't acceptable.

So let's try and figure out what the side effects of a sloppy
translation are.  In the current docker/docker master:
$ git log --oneline -G 'layers+base'
f342b27 Removing old custom RootFS behavior on Windows.
4fa0ecc Update image specification for content-addressability changes
d45a26d Windows: Support non-base-layered images
94d70d8 Windows libcontainerd implementation
6bb0d18 Move Container to its own package.
4352da7 Update daemon and docker core to use new content addressable storage
01ba0a9 Add image store
$ git describe --contains 01ba0a9
v1.10.0-rc1~520^2~4
$ git describe --contains f342b27
fatal: cannot describe 'f342b27145d8f5af27cd5de1501551af275e899b'
$ git describe --tags f342b27
docs-v1.12.0-rc4-2016-07-15-403-gf342b27
So the removal will presumably land in Docker 1.13.  That means that
Docker ≥ 1.10 and < 1.13 will respect (and possibly generate) images
using the layers+base type.
What were the semantics of the layers+base type, and what happens if
you interpret a layers+base config as a layers config?  I'm not
familiar enough with the Docker code to get much from reading
docker/docker#25090, but I expect the fallout is that the rootfs is
built without the base layer underneath it.  I expect the base layer
is a critical piece of any Windows images that used it, and containers
launched with a rootfs that was missing that expected base layer would
immediately die due to missing libraries or similar.  That seems like
a sane way to handle “you botched the translation to an OCI format”.
Or is the base layer just some security patches, and layers+base
containers missing that base layer will silently run with those
security issues unpatched?  If so, can you find an image like this?

On Wed, Aug 31, 2016 at 06:44:57PM -0700, W. Trevor King wrote:

I'll add notes to [2] tomorrow to clarify the new translation step.

Added with cb9a5a9 → 5d032cb, which also rebases onto the (rapidly
evolving) master ;).

I've received a little more information on the role of this field. It actually selects the hashing algorithm used for the image ID/rootfs.  If that algorithm changes, this field changes value. Removing it will make otherwise compatible images completely incompatible.
What reason do you have for making this incompatible? Is this just really about field hygiene?

On Fri, Sep 02, 2016 at 03:35:44PM -0700, Stephen Day wrote:

I've received a little more information on the role of this
field. It actually selects the hashing algorithm used for the image
ID/rootfs.

Can you link to code?  I'm not sure why you'd need something other
than the manifest digest for an image ID.

If that algorithm changes, this field changes value. Removing it
will make otherwise compatible images completely incompatible.

It's hard to say without looking at code, but if the field consumers
have a valid reason for switching hashing algorithms based on this
field, I expect we want to keep the field and document that logic.
If they use different hashing agorithms based on historical reasoning
that no longer holds water, then I'm likely to still be ok dropping
the field (but it would be nice to understand what “completely
incompatible” meant in more detail).
And translation is still deterministic and straightforward 1.  I am
absolutely fine with making translation a concious choice, otherwise
what's the point of having two different specs?  We'd just have the
OCI cite Docker (or vice versa).

What reason do you have for making this incompatible? Is this just
really about field hygiene?

Yes.  Each property defined by the spec should have a clear purpose
and well-defined behavior.  Currently rootfs.type has neither.  In
that case, I think it's good to remove it, and see what happens.
You're arguing strongly for keeping it, but I still don't have a clear
handle on what goes wrong if we drop it (and users forget to
translate?  And translating is somehow more difficult than 1?).
If/when you paint a clear picture for why the property is needed, I
think we want it in the spec with that picture.  For example:
type (required, string) One of layers, other-value.  This
selects the hashing algorithm used for the image
ID/roofs, which allows you to {FIXME:
why do we care?}.  Libraries unpacking images MUST generate en error
if they do not recognize the value.

@wking At this point, I have painted a very clear picture of why this field is needed and you're simply rejecting the premises.
I'm sorry, but you're creating work and subtlety for no good reason.

On Fri, Sep 02, 2016 at 04:01:11PM -0700, Stephen Day wrote:

@wking At this point, I have painted a very clear picture of why
this field is needed and you're simply rejecting the premises.

Ah, you're saying 1 that touching the config at all changes the
ImageID 2.  That makes a lot of sense (obviously the config hash
will change if you remove a field from the config JSON).  What's the
downside to that?  You get a different config hash, which gives you a
different manifest hash (but you already have a different manifest
hash, because the manifest layer descriptors have Docker→OCI media
types).  The config JSON is tiny, so refetching that blob isn't a
big cost.
Maybe there are systems which handle trust by signing configs?  In
that case, you could translate the rest of the Merkle tree to OCI
values, and as long as the config didn't change, those old
signatures would still validate [3].  Those signatures would break if
the registry removed the rootfs.type property.
But if the rootfs.type value is ‘layers’, the registry can leave the
config alone and call it an application/vnd.oci.image.config.v1+json.
With the wording in flight with #164 (which I still [4] think should
cover the config JSON too [5]), implementations would ignore the
unknown property.
And if the rootfs.type value is not defined in this spec or an
extension understood by the implementation, then you're going to have
to get a new signature after a more invasive config translation.
Is that config-signature-preservation issue the only problem you see?
I'm personally fine requiring folks who don't use ‘layers’ to re-sign
after translating to an OCI config format.  Folks using images that
rely on other values [6] should avoid using OCI-only implementations
for those images until they've had time to convert them.
[3]: I'm concerned about tarbomb exposure in that case and would like
to see signatures on at least the manifest level, but that's a
different issue.
[4]: https://github.com/opencontainers/image-spec/pull/224#issuecomment-243953152
[5]: https://github.com/opencontainers/image-spec/pull/164#issuecomment-243952187
[6]: https://github.com/opencontainers/image-spec/pull/224#issuecomment-243941065

I think @wking is raising a lot of valid points here. @stevvooe can you please give a tangible example of how this can cause a security issue? If it really is a potential landmine then we should be crystal clear in the spec about why it's so important so implementers can take appropriate caution. As it stands the implications are not obvious.

@jonboulle The value type=layers expresses the ChainID hash algorithm used as the image ID. If this is incorrect, it will change how the image is identified and verified in certain implementations.  Put differently, for the purposes of image verification, it can only be used if and only the value of type is equal to the value layers. Making assumptions about this value when it is omitted leaves the possibility of error when addressing possible future security concerns. This is particularly problematic if there is a vulnerability found in both docker and OCI. If docker's import assumes the value is layers for OCI images but it was really updated due to some change, these images may be imported with malicious identifiers.
If you're asking for an example of an active exploit that involves some contrived sequence of events that lead to an injection, you're going to be disappointed. That simply isn't how security works. If we knew there was an exploit, we'd suggest a full fix for it. In this case, this PR is suggesting that we take implicitly, what was explicit before, which creates a possible risk that needs mitigation.
The safest option here is to leave the field in place and mark for removal in a later version if it can be shown that it is truly not needed. The best part is that this option requires no work at all, other than me, sitting here, asserting that this community should not take undue security risks.
The approach here is not dissimilar to putting ashtrays in airplane lavatories.  Just because the implications of removing the ash tray aren't obvious doesn't mean we remove the ash tray.

Here is an example of value of Type causing unspecified problems: https://github.com/docker/docker/pull/25806. Likely fix, https://github.com/Microsoft/docker/commit/710b456b5bf87c3c17fea3120e90f65b059287f9, is to add an explicit filter on a value of Type.

On Tue, Sep 06, 2016 at 12:56:27PM -0700, Stephen Day wrote:

@jonboulle The value type=layers expresses the ChainID hash
algorithm used as the image ID.

The code in 1 is not compatible with our ChainID definition 2,
because 2 does not say:
ChainID(layerN) is always the empty string if rootfs.type is
‘layers+base’ and the implementation OS is Windows.
Which seems like a pretty crazy thing to say (1 is pretty
straightforward, why such an odd special case?).

If this is incorrect, it will change how the image is identified and
verified in certain implementations.  Put differently, for the
purposes of image verification, it can only be used if and only the
value of type is equal to the value layers. Making assumptions
about this value when it is omitted leaves the possibility of error
when addressing possible future security concerns.

I'm not clear on how the ChainID is ever useful for validating
images.  It seems like it would only be useful for validating stacks
of layers, and then it's (obviously?) only useful for validating the
layers that you tell it about.  The general principle is:
When naming an object, hash something high enough up the Merkle tree
to contain all that object's constituent parts.
I don't think the ChainID would ever fit that criterion for an image
(because it was missing ‘user’, etc.).  If there are config media
types which allow layers+base but did not contain a reference to base,
then signing one of those config instances would not fit that
criterion either.

This is particularly problematic if there is a vulnerability found
in both docker and OCI. If docker's import assumes the value is
layers for OCI images but it was really updated due to some
change, these images may be imported with malicious identifiers.

If the OCI updates the config spec with different layer semantics, we
had better be bumping our media type too.  If Docker imports a
application/vnd.oci.image.config.v2+json (where the correct Docker
analog is not rootfs.type = layers), and says “hmm, never heard of OCI
config v2.  I'll just use my OCI config v1 importer logic”, then that
is pretty obviously Docker's problem, and not a security flaw in the
OCI spec.
One way to avoid confusion would be to clearly document (and provide a
Go API for?) which media types do meet the constituent parts criterion
for naming images.  With the current OCI spec, that includes
application/vnd.oci.image.manifest.v1+json,
application/vnd.oci.image.manifest.list.v1+json, and (if you don't
mind the tarbomb exposure 3 or the inability to lookup blobs
directly by DiffID) application/vnd.oci.image.config.v1+json.

@wking I am not going to repeat my points again, so I would please ask that you take some time to read and understand my position before responding.

@wking wants to clarify that things MUST error out if this field is missing. @stevvooe and @wking will work to close this one and open a new one with that error out language or equivalent.

On Wed, Sep 07, 2016 at 02:32:26PM -0700, Brandon Philips wrote:

@wking wants to clarify that things MUST error out if this field is
missing. @stevvooe and @wking will work to close this one and open a
new one with that error out language or equivalent.

I've filed this alternative approach as #277 (is that 50 issues/PRs in
the last 9 days?  I hope that settles down before we start cutting 1.0
RCs ;).  I'm fine with having this PR closed now, or it can wait until
#277 lands (or maybe folks will swing back over to this approach after
we kick #277 around for a bit ;).

@wking Closing this PR. If #277 stalls, we can re-open.


Point logs to somewhere Docker can read them

Current attempt doesn't work and is disable. Consider also setting the default log location be a volume in case only we can do is duplicate the logs to syslog so we should try to avoid making the container huge with logs and instead make the volume location a separate volume. This might be helpful https://mariadb.com/kb/en/mariadb/overview-of-mariadb-logs/

The fuzzy nginx image is using this, which could be helpful here too:
# Forward request and error logs to docker log collector.
ln -sf /dev/stdout /var/log/nginx/access.log && \
ln -sf /dev/stderr /var/log/nginx/error.log