DRILL-4704 fix

calculate precision from integer value, instead of using faulty
precision of zero

This patch is missing a unit test..can you add a unit test for this in TestDecimal.java ?

I'm getting errors like this one even from unit tests in TestDecimal that I have not modified.  Anyone have an explanation?
Tests run: 7, Failures: 0, Errors: 7, Skipped: 0, Time elapsed: 25.801 sec <<< FAILURE! - in org.apache.drill.exec.physical.impl.TestDecimaltestCastFromFloat(org.apache.drill.exec.physical.impl.TestDecimal)  Time elapsed: 6.876 sec  <<< ERROR!^Morg.apache.drill.exec.rpc.RpcException: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: PatternSyntaxException: Unexpected internal error near index 1\ ^
[Error Id: 2b33914f-2262-4370-8d1b-b5e554b02c4c on DaveOshinsky-PC.gp.cv.commvault.com:31010]        at org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:60)        at org.apache.drill.exec.client.DrillClient$ListHoldingResultsListener.getResults(DrillClient.java:424)        at org.apache.drill.exec.client.DrillClient.runQuery(DrillClient.java:321)        at org.apache.drill.exec.physical.impl.TestDecimal.testCastFromFloat(TestDecimal.java:102)Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: PatternSyntaxException: Unexpected internal error near index 1\ ^
On Friday, June 17, 2016 10:05 AM, Aman Sinha<EMAIL_ADDRESS>wrote:

This patch is missing a unit test..can you add a unit test for this in TestDecimal.java ?—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

The tests passed for me (I am running on a linux box).
mvn install -DskipTests
mvn -Dtest=TestDecimal surefire:test -DfailIfNoTests=false
Running org.apache.drill.exec.physical.impl.TestDecimal#testCastFromFloat
Running org.apache.drill.exec.physical.impl.TestDecimal#testComplexDecimal
Running org.apache.drill.exec.physical.impl.TestDecimal#testSimpleDecimalArithmetic
Running org.apache.drill.exec.physical.impl.TestDecimal#testComplexDecimalSort
Running org.apache.drill.exec.physical.impl.TestDecimal#testSimpleDecimalMathFunc
Running org.apache.drill.exec.physical.impl.TestDecimal#testSimpleDecimal
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.566 sec - in org.apache.drill.exec.physical.impl.TestDecimal
Results :
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0
I see control chars in your error stack .. what platform are you using ? Is it windows ?

Yes, Windows 7.  java version "1.8.0_91",  Maven 3.3.9, running build under cygwin
doshinsky@DaveOshinsky-PC /cygdrive/c/apache/drill-1.6.0/exec/java-exec$ mvn -Dtest=TestDecimal surefire:test -DfailIfNoTests=false[INFO] Scanning for projects...[INFO][INFO] ------------------------------------------------------------------------[INFO] Building exec/Java Execution Engine 1.6.0[INFO] ------------------------------------------------------------------------....Running org.apache.drill.exec.physical.impl.TestDecimal#testComplexDecimalRunning org.apache.drill.exec.physical.impl.TestDecimal#testCastFromIntRunning org.apache.drill.exec.physical.impl.TestDecimal#testSimpleDecimalArithmeticRunning org.apache.drill.exec.physical.impl.TestDecimal#testComplexDecimalSortRunning org.apache.drill.exec.physical.impl.TestDecimal#testSimpleDecimalMathFuncRunning org.apache.drill.exec.physical.impl.TestDecimal#testSimpleDecimalTests run: 7, Failures: 0, Errors: 7, Skipped: 0, Time elapsed: 30.062 sec <<< FAILURE! - in org.apache.drill.exec.physical.impl.TestDecimaltestCastFromFloat(org.apache.drill.exec.physical.impl.TestDecimal)  Time elapsed: 8.202 sec  <<< ERROR!org.apache.drill.exec.rpc.RpcException: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: PatternSyntaxException: Unexpected internal error near index 1\ ^
[Error Id: 8a0fff12-6dea-407d-99f2-12ea0479b0f7 on DaveOshinsky-PC.gp.cv.commvault.com:31010]        at org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:60)        at org.apache.drill.exec.client.DrillClient$ListHoldingResultsListener.getResults(DrillClient.java:424)        at org.apache.drill.exec.client.DrillClient.runQuery(DrillClient.java:321)        at org.apache.drill.exec.physical.impl.TestDecimal.testCastFromFloat(TestDecimal.java:111)Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: PatternSyntaxException: Unexpected internal error near index 1\ ^
On Friday, June 17, 2016 2:10 PM, Aman Sinha<EMAIL_ADDRESS>wrote:

The tests passed for me (I am running on a linux box). mvn install -DskipTests
mvn -Dtest=TestDecimal surefire:test -DfailIfNoTests=falseRunning org.apache.drill.exec.physical.impl.TestDecimal#testCastFromFloat
Running org.apache.drill.exec.physical.impl.TestDecimal#testComplexDecimal
Running org.apache.drill.exec.physical.impl.TestDecimal#testSimpleDecimalArithmetic
Running org.apache.drill.exec.physical.impl.TestDecimal#testComplexDecimalSort
Running org.apache.drill.exec.physical.impl.TestDecimal#testSimpleDecimalMathFunc
Running org.apache.drill.exec.physical.impl.TestDecimal#testSimpleDecimal
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.566 sec - in org.apache.drill.exec.physical.impl.TestDecimalResults :Tests run: 6, Failures: 0, Errors: 0, Skipped: 0I see control chars in your error stack .. what platform are you using ? Is it windows ? —
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

unfortunately, I don't have access to a windows machine.  I would suggest a few options:

I just noticed that your test run shows 7 tests.  Mine had 6 tests.  I suppose you added a new one.  Can you first try a clean version without your new test ?
Could you try it on a different machine with either mac or linux ?
Maybe you can seek help from others in the community (I believe @adityakishore runs on windows + cygwin).

If I have some extra time to set this up on a Mac or Linux, I will.  That's not so convenient at the moment.  Besides, this should work on Windows, shouldn't it?  I noticed that there's an RpcException coming in from the server side (the drill-bit), but the exception includes nothing like a stack trace to figure out where it's failing.  How can I debug the server side?  Is there a log file?  Or which source file can I change to catch the exception on server side, and log a stack trace?
Regarding the 7th test, yes I added it.  I tried a fresh git clone to Windows from apache/drill master, and that TestDecimal (with 6 tests) fails exactly the same way.  It does appear to have something to do with the different newline convention on Windows (CR-LF), but I need something like a server-side stack trace to localize the problem.  I tried hacking TestDecimal.java to change all \n to \r\n (CR LF) in the query string that is sent to the server (Windows only), but that did not help.
On Friday, June 17, 2016 6:01 PM, Aman Sinha<EMAIL_ADDRESS>wrote:

unfortunately, I don't have access to a windows machine. I would suggest a few options:

I just noticed that your test run shows 7 tests. Mine had 6 tests. I suppose you added a new one. Can you first try a clean version without your new test ?

Could you try it on a different machine with either mac or linux ?

Maybe you can seek help from others in the community (I believe @adityakishore runs on windows + cygwin). —
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

I saw no emails yesterday associated with my adding a unit test here for DRILL-4704 fix.  Just in case those got lost somewhere, please review.

Thanks for adding the tests.  I will take a look likely by tomorrow.

Now that 1.7 release dust has hopefully settled down (congrats on release!), please review added unit test.

Regarding Aman's last comment, please add the following... In the scenario addressed by this fix, there is a const integer that came from the SQL select where clause, and that is what is being casted into a decimal, for comparison with each a decimal value in each row.   Since the integer is constant, a thoroughly optimized piece of code would cast it just once to a decimal, and use the stored decimal to compare with the decimal value in each row.  It appears that that is not what this code is doing. It is casting repeatedly, for each row.  So, rather than be concerned with just a few added integer divisions by 10 (a small number around log10(the const integer) + 1 divides), this code should ideally be (more) extensively modified to do the cast operation just once.  That's too big a change for me to make for this fix.  Besides, which is preferable?  Code that does not function properly (DRILL-4704), and is not well optimized?  Or code that behaves correctly, and is only slightly less well optimized?

I added a print to System.out for each time that the cast (including the precision calculation I added) is called.  I found that the cast is done once for each input integer value from the SQL select statement.  It is NOT done once per row in the Parquet file(s) that Drill is scanning.  So, there should be absolutely no performance issue with doing the precision calculation as I am doing it.  Again, this is NOT done for every row.  Please review.

I agreed with Aman that the eval() of CastIntDecimal28Sparse() should not ignore the specified scale/precision, if they are passed as parameters.
As you analyzed, the problem is the scale/precision is 0 for a int constant value, which is not right. We should fix place where scale/precision is specified wrongly.  Ignoring scale/precision and re-calcualting them in eval() means Drill has to apply that logic per input row.
The reason that you saw eval() is called only once:
Filter employee_id = 170,
is   employee_id = castIntDecimal28Sparse(170, scale, precision). // scale and precision is wrongly 0.
Since RHS is a constant, Drill will evaluate only once, even you have multiple rows. However, this is not true, if it's not a constant expression.
To me, you point to the right cause of the problem. But we need get better place to apply the fix ( not in eval() ) method.
The place where we get scale/precision as 0
https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/ExpressionTreeMaterializer.java#L315-L316

Thanks Jinfeng for the very clear explanation, and pointer to ExpressionTreeMaterializer.java.  Rather than change ExpressionTreeMaterializer.java, with which I am completely unfamiliar, I added another change to CastIntDecimal.java to only calculate the precision when the supplied one is zero.  I hope the amended fix is sufficient for now.  There might be other cases where the supplied precision is invalid (zero), and this change will handle those.
On Wednesday, August 3, 2016 7:18 PM, Jinfeng Ni<EMAIL_ADDRESS>wrote:

I agreed with Aman that the eval() of CastIntDecimal28Sparse() should not ignore the specified scale/precision, if they are passed as parameters. As you analyzed, the problem is the scale/precision is 0 for a int constant value, which is not right. We should fix place where scale/precision is specified wrongly. Ignoring scale/precision and re-calcualting them in eval() means Drill has to apply that logic per input row.The reason that you saw eval() is called only once:
Filter employee_id = 170,
is employee_id = castIntDecimal28Sparse(170, scale, precision). // scale and precision is wrongly 0.Since RHS is a constant, Drill will evaluate only once, even you have multiple rows. However, this is not true, if it's not a constant expression. To me, you point to the right cause of the problem. But we need get better place to apply the fix ( not in eval() ) method. The place where we get scale/precision as 0 https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/ExpressionTreeMaterializer.java#L315-L316—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

I attached debugger to Drill and found it's passing through line 172 (not 315-316) of ExpressionTreeMaterializer.java, and that the "toType" object is one of those runtime generated ones (name like TypeProtos$MajorType@8570).  Rather than attempt to change that code, I made another change to CastIntDecimal.java which avoids repeated calculation of the precision when it is already provided.  Please review.

Line 172 was called after scale/precision are set on line 315-316.
I understand your new change will kick in when the passed precision= 0. However, I still think that the right way is to pass in the correct scale/precision when call that cast function, in stead of figuring out on-the-fly in the middle of that function.  If someone pass a 0 precision (which is not right), we need fix that in the place where 0 precision is specified.
Also, will your code work correct for negative integer input?

No, negative precision will not work properly.  I committed another change to PR-517 to handle this.
Quite honestly, I do not know how to fix the problem as you suggest in ExpressionTreeMaterializer.java.  First, I set a breakpoint on line 315, and it never fired.  It went to the breakpoint on line 172, never passing line 315 (where I set a breakpoint).  I see that code at line 172 is making a call to a virtual function toType.getPrecision() to obtain the precision, from an object of a type that is only known at runtime.  Based on what I saw in the debugger, the type of the "toType" object at line 172 was something like TypeProtos$MajorType@8570, which looked like a piece of runtime generated code (as often seen in Drill), and the debugger would not let me single step into it.  That toType object did have both scale and precision = 0.  Now, if I could obtain the integer value of the toType object, I could calculate the precision right there (if the getPrecision() returns 0).  But, I don't see how to get the integer value at line 172.  In fact, I'm guessing that "getting an integer value" only applies in some cases, for some kinds of values that are being "materialized".
To summarize, I have not been looking at Apache Drill nearly as long as you have, and don't understand it nearly as well as you.  Yet, you are asking me to change a fix that already works, to fit a model you appear to have in mind for a fix as you would do it.  I simply don't have the level of understanding of the Drill code to do the fix as you imagine it should be done.  Moreover, I am a volunteer, being paid by my employer to work mainly on things other than Apache Drill.  As such, the amount of additional time I can spend on this fix is limited.  My fix already works AFAIK, and is not inefficient.  I suggest you take it, and if necessary, then change it to your liking.  The unit test I've included is a decent test of the DRILL-4704 problem.
On Thursday, August 4, 2016 4:06 PM, Jinfeng Ni<EMAIL_ADDRESS>wrote:

I attached debugger to Drill and found it's passing through line 172 (not 315-316) of ExpressionTreeMaterializer.java, and that the "toType" object is one of those runtime generated ones (name like TypeProtos$MajorType@8570). Rather than attempt to change that code, I made another change to CastIntDecimal.java which avoids repeated calculation of the precision when it is already provided. Please review.
Line 172 was called after scale/precision are set on line 315-316.I understand your new change will kick in when the passed precision= 0. However, I still think that the right way is to pass in the correct scale/precision when call that cast function, in stead of figuring out on-the-fly in the middle of that function. If someone pass a 0 precision (which is not right), we need fix that in the place where 0 precision is specified.Also, will your code work correct for negative integer input?—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

The proximate cause of this particular issue is the mechanism by which the cast function is being generated. As others have noted, the code is passing through lines 316-316. My guess is that the materializer is working out how to call this function:
Decimal28SparseFunctions.Decimal28SparseEqual( Decimal28Sparse left, Decimal28Sparse right )
The materializer sees: D28SEqual( Decimal28Sparse, Integer )
The code wants to convert the Integer to a Decimal28Sparse. It asks the argument for its precision and scale. But, since the argument just specifies the "minor type" (SQL type), it does not have the extended attributes of precision and scale. Thus, the annotations in the function definition are the ultimate cause of the precision being 0:
@Param Decimal28SparseHolder left;
@Param Decimal28SparseHolder right;
Now, we might argue that the code should be using, as the target (p, s) the (p,s) of the left argument (the one that is already a decimal). But, the materializer does not know that this is an operator; it treats all functions the same (operators and otherwise.) So, the materializer does not know that we should treat arguments in this way. All it knows is that it has an Integer and wants a D28S (of, presumably, any p & s).
Next, we can ask how DS28Equal handles the (p,s) of the two arguments: must they be the same or can they be different? Looking at the code (as best as I can tell), the code will align values with different (p,s) values. Thus, say, values of Decimal28( 10, 2 ) and Decimal28( 8, 0 ) will compare properly.
So, back to the cast from Integer to D28S. As Dave suggests, we can interpret (precision == 0) to mean "you figure it out": we can set (p,s) way we want.
Dave's fix computes the precision based on the actual integer value. But, since the comparison can handle any valid (p,s), we might as well set them to a constant. Since the maximum Integer value is 2 billion, we can just use a constant (10,0).
If the selected precision is constant, then the cost is low and the solution is fine for both for constant integers (as in the original case), and for per-row conversions (as in Aman's concern.)

The plot thickens. I tried the fix of setting the precision to a constant of 10. This uncovered a larger issue. The template in question generates cast functions for (INTEGER, BIGINT) x (DECIMAL9, DECIMAL18, DECIMAL28) and perhaps others. The constant of 10 does not, of course, work for BIGINT (long) values.
The trick is that precision=10 won't work for DECIMAL9 either. Dave's solution has a similar problem. Dave sets the precision to whatever is right for the input value, which seems great. But, that value could be too large for the output DECIMAL type.
What we need is to set the precision to the min( max int precision, max decimal precision ). Or, if we use Dave's proposed solution, max( input arg precision, max decimal precision ).
In either case, the code must handle overflow. Passing a Long.MAX_VALUE or even Integer.MAX_VALUE to CastBigIntDecimal9( ) should cause an overflow error or data truncation. I'll research how that worked previously to see if we've uncovered a new issue, or if a solution already exists.

Thanks for looking at this, Paul.  If there are problems with comparing an input decimal (from the "where clause", for example) against the data, it is because of the algorithms for comparing decimal values of different precisions, not because the proper precision was computed for the input decimal (e.g., based on the actual numerical value, as my PR-517 fix is doing when no precision is provided initially).  I was going to comment about the idea of using a fixed value of precision=10, but you beat me to it.  It does not suffice for larger integers, like those that require a "long" instead of an "int" to represent.  So, I suggest that we take a "divide and conquer" approach with this problem.  First, compute a proper precision for the value casted from int or BigInt.  Next, make sure the comparison with other decimal values works for any other decimal value precision (a separate issue).
Decimal support in Drill is disabled by default for good reasons.  The (decimal) design itself is fundamentally flawed, and it has all sorts of issues related to code maintainability as well as runtime performance.  I have been experimenting in a Drill "play build node" with having a single, one-size-fits-all variable width vector class that represents any decimal value efficiently (I call it "VarDecimal").  It's difficult to even get this to compile (with all of the FreeMarker use, and the numerous conversions between all of the decimal and non-decimal types), but eventually I hope to experiment with this to deal with the DRILL-4184 problem I encountered a number of months ago (for which I have a pull request PR-372 with a short-term, but not a clean, fix).
As food for thought, while looking at this experimental "VarDecimal" thing, I noticed that there are (numerous) casts/conversions between all of the pairwise combinations of different decimal precisions (DECIMAL9, DECIMAL18, DECIMAL28, DECIMAL38, both dense and sparse, from what I recall), and between all of these and all other types that have numeric interpretation.  Eventually replacing all of these different DECIMAL* fixed width representations with a single VarDecimal variable width representation (one-size-fits-all) would be much more efficient (memory-wise) at runtime (for typical scenarios where most actual numeric values don't require the full precision to represent), would greatly simplify parts of the Drill code, and would fix DRILL-4184 cleanly.
Until some major re-work can be done for Drill decimal support, we will probably have to settle for small, incremental improvements.  I would say that PR-517 represents one such improvement.
On Thursday, August 4, 2016 9:31 PM, Paul Rogers<EMAIL_ADDRESS>wrote:

The plot thickens. I tried the fix of setting the precision to a constant of 10. This uncovered a larger issue. The template in question generates cast functions for (INTEGER, BIGINT) x (DECIMAL9, DECIMAL18, DECIMAL28) and perhaps others. The constant of 10 does not, of course, work for BIGINT (long) values.The trick is that precision=10 won't work for DECIMAL9 either. Dave's solution has a similar problem. Dave sets the precision to whatever is right for the input value, which seems great. But, that value could be too large for the output DECIMAL type.What we need is to set the precision to the min( max int precision, max decimal precision ). Or, if we use Dave's proposed solution, max( input arg precision, max decimal precision ).In either case, the code must handle overflow. Passing a Long.MAX_VALUE or even Integer.MAX_VALUE to CastBigIntDecimal9( ) should cause an overflow error or data truncation. I'll research how that worked previously to see if we've uncovered a new issue, or if a solution already exists.—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Hi Dave,
I agree that Drill Decimal support clearly needs attention. I'm expermenting with adjusting the template code to at least generate a meaningful conversion for all int x decimal types when the values fit.
Looks like current behavior will be that if we convert an int that is too large for a scale of 0, we won't adjust the scale. That is, if we always leave scale=0, then we'll do the following obviously wrong converstion:
big int: 123,456,789,012
Dec 9: 456,789,012
We want a scale of 3, so that the answer is:
Dec 9: 123,456,789,000
And, of course, we should round the least significant digit, which it seems we don't do.
So, our fix will work only for the simple case, not for the more complex case.
We could leverage your fix to properly set the scale as well as the precision.

Based on above discussion, I played around with the proposed solution. The following takes into consideration the max precision of both the input and output types. It DOES NOT handle truncation properly (but nor do any of the other solutions.)
Here is the revised code:
    if (precision.value <= 0) {
<#--
        // since input precision is nonpositive, calculate precision for this integer value
        long precisionTmp = in.value == 0 ? 1 : in.value;  // precision for value 0 will be 1
        int precisionCounter = 0;
        while (precisionTmp != 0) {
            ++precisionCounter;
            precisionTmp /= 10;
        }
        out.precision = precisionCounter;
-->
        // Since precision is not provided, select one appropriate for the
        // largest integer value and the target decimal type.

        // Note: if the integer is too large for the target precision, silent
        // truncation of the highest part of the number will occur. This is
        // obviously bad. What should happen is we shift scale and discard
        // the low-order digits.
        // That is, if we convert 123,456,789,012 to Dec9, the result is
        // 456,789,012 which is very obviously very wrong. It should be
        // 123,456,789,000.
        <#-- A correct solution requires a modification of the code
             commented out above. -->

      <#-- Precision needed by the largest from type value. -->
      <#if type.from.equals( "Int" )>
        <#assign inPrec = 10>
      <#elseif type.from.equals( "BigInt" )>
        <#assign inPrec = 19>
      <#else>
        <#-- Yes, create invalid syntax: that way the compiler will tell us we missed a type. -->
        <#assign inPrec = "Unexpected from type: ${type.from}">
      </#if>

      <#-- Maximum precision allowed by the to type. -->
      <#if type.to.startsWith("Decimal9")>
        <#assign maxPrec = 9>
      <#elseif type.to.startsWith("Decimal18")>
        <#assign maxPrec = 18>
      <#elseif type.to.startsWith("Decimal28")>
        <#assign maxPrec = 28>
      <#elseif type.to.startsWith("Decimal38")>
        <#assign maxPrec = 38>
      <#else>
        <#-- Yes, create invalid syntax: that way the compiler will tell us we missed a type. -->
        <#assign maxPrec = "Unexpected to type: ${type.to}">
      </#if>
      <#-- Note that this calculation is done here, rather than in static variables,
           because static members are not allowed for this function. -->
      <#if inPrec < maxPrec>
        // Maximum precision needed by the largest ${type.from} value.
        out.precision = ${inPrec};
      <#else>
        // Maximum precision allowed by the ${type.to} type.
        out.precision = ${maxPrec};
      </#if>
    }
    else {
        // since input precision is positive, assume it is correct, and use it
        out.precision = (int) precision.value;
        out.scale = (int) scale.value;
    }

As a next step, using Dave's solution would better handle truncation: based on the precision of the input value, adjust the scale to truncate low-order digits.

To clarify the previous comment... In the case where the maximum required precision for the int type is greater than the maximum precision of the target decimal type, we use Dave's code to figure out the actual precision of the int value. Then, we figure out scale as:
scale = value precision - max decimal precision

This, if you try to store 123,456,789,012 into an Dec9, you would get 123,456,789,000 (truncate the lower-order digits.) The solution does not handle rounding, but that can be added.
Note that the decision to use Dave's code would be in the template: it would apply for, say, BigInt to Dec9 and that kind of thing. The logic to detect this case is already in the suggested code snippet.
  <#if inPrec < maxPrec>
    ...
  <#else>
    // Put Dave's find-the-precision logic here
    if ( out.precision > ${maxPrec} ) {
        out.scale = out.precision - ${maxPrec};
        out.precision = ${maxPrec};
    }
  </#endif>

I have not tested the above; it is just a pseudocode suggestion.

The scale is a schema-level property of that column of data, that is never changed on the fly, to deal with problematic truncation, or otherwise.  If anybody is interested in a detail explanation, or other tidbits about how decimal types work, just let me know (daveoshinsky@yahoo.com).
On Saturday, August 6, 2016 3:27 PM, Paul Rogers<EMAIL_ADDRESS>wrote:

To clarify the previous comment... In the case where the maximum required precision for the int type is greater than the maximum precision of the target decimal type, we use Dave's code to figure out the actual precision of the int value. Then, we figure out scale as:scale = value precision - max decimal precision
This, if you try to store 123,456,789,012 into an Dec9, you would get 123,456,789,000 (truncate the lower-order digits.) The solution does not handle rounding, but that can be added.Note that the decision to use Dave's code would be in the template: it would apply for, say, BigInt to Dec9 and that kind of thing. The logic to detect this case is already in the suggested code snippet.  <#if inPrec < maxPrec>
...
<#else>
// Put Dave's find-the-precision logic here
if ( out.precision > ${maxPrec} ) {
out.scale = out.precision - ${maxPrec};
out.precision = ${maxPrec};
}
</#endif>
I have not tested the above; it is just a pseudocode suggestion. —
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Good point. So, we'd want to issue an overflow error in the cast method, but it seems Drill has no way to do that. So, for now, maybe just leave it at the version that works when numbers are in range, and file a JIRA for the out-of-range case.

As a follow-up, the case we're focusing on is an implicit cast: one created by the system to convert an integer to a decimal type in order to do an equality check. To avoid the scale issue, the code that does the explcit cast should check the type of integer and select a target decimal type that will hold all values of that int type with a scale of 0. Does the code do that now? It may be that the decimal type must also be cast so that the equality has the same types. That is, if we compare Dec9 with Integer, we need a scale of 10 for the Int, so we must cast the Int to Dec18. As a result, we must cast the Dec9 to Dec18.
The reason for all of this is that the casts are done statically. A more efficient solution would be for the the run-time code to make a decision based on actual values, but Drill is not set up that way. Another solution would be to have an in-memory form of a decimal that caries the precision and scale so we can set the scale per value. But, Drill is not set up that way either.
So, the change we've been discussing (setting scale based on int and decimal types, ignoring overflow for now) seems the only short-term solution for 1.8.

The change to the cast function implementation will impact not only implicit cast, but also explicit cast. You are right that the test case of employee_id = 170 will trigger implicit cast. However, if you have an explicit cast  "select cast(int_column as decimal)  from ...", you will also see the impact of the code change.
I could not understand what you mean by "casts are done statically".  For implicit cast, the cast function is inserted dynamically in execution time, only after Drill knows the incoming batch schema.

"Dynamic" meant "for each data value." "Static" meant "at compile time." We make our type decisions at compile time. That is too early to adjust scale based on actual input values. It is a real nuisance that scale is associated with the type, not the value.
Dave filed a bug to track the additional issues. See DRILL-4834.

What is the resolution here? Dave, do you want to review & adopt the suggested code several posts back? That code passed the unit tests that you provided, plus those aready in Drill (using the unit tests you suggested in an earlier post.) Then, update this PR with the results. Sounds like Jinfeng can then review the code & accept the PR. As we've said, the fix is not perfect, but it does work for every int/decimal type if the user is careful to provide int values that fit within the target decimal type.

Paul, do you mean the long piece of code with conditions on various destination precision types, that sets inPrec and maxPrec, and then selects the smaller of the two for the resulting precision?  I think it's just way simpler to use the calculated precision, as I suggested earlier.  That also works, and it's less code to look at, and less code to potentially be broken.  I do have a bias against the heavy, heavy use of FreeMarker everywhere.  It's just a recipe for bugs piled on top of bugs, and all of the numerous FreeMarker conditions are not likely to be tested for quite some time.
On Monday, August 8, 2016 1:48 PM, Paul Rogers<EMAIL_ADDRESS>wrote:

What is the resolution here? Dave, do you want to review & adopt the suggested code several posts back? That code passed the unit tests that you provided, plus those aready in Drill (using the unit tests you suggested in an earlier post.) Then, update this PR with the results. Sounds like Jinfeng can then review the code & accept the PR. As we've said, the fix is not perfect, but it does work for every int/decimal type if the user is careful to provide int values that fit within the target decimal type.—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

I'm not fully convinced that we should check the precision for each input value for this castIntDecimal function.
The argument of proposed patch is parameter "precision=0" is not valid, and has to be calculated on-the-fly or "dynamically" for each input value.  To me, 1) if "precision=0" is a wrong input to this function, then we should fix the place where precision=0 is passed in.  2) why would you treat "precision=0" only?  what if I pass in "precision = 1", and a integer value of 123456? in such case, do we think "precison=1" is valid or not?
Regarding the overflow problem, that seems to be a separate issue, and probably is true for most of the existing decimal functions.

The overflow problem will require separate changes to fix, from any changes we make in the short-term to fix this issue (see DRILL-4834).  However, if I understand some of Paul's findings earlier, the two issues are related as follows.  If we choose a large precision based only on the kind of integer that is input to the cast (either an int, or a long, but not minimized to represent the integer value to be casted), then we are more likely to encounter the overflow issue, because this precision is more likely to exceed the capacity of the destination decimal of the cast.  Is that right, Paul?  Given the risk of overflow, it is safer to choose the absolute minimum precision possible, which is what the changes I made to CastIntDecimal.java are doing.  We can't avoid overflow in every situation, but we can try to minimize the chances of it.  My understanding is that the actual integer value to be casted isn't available in ExpressionTreeMaterializer.java; at least, I did not see how to get it.  And that integer value is required to compute the minimum precision to represent that value.  I don't see how to fix this problem reliably in ExpressionTreeMaterializer.java, as I said earlier.  If you (Jinfeng) know how to do that, why don't you just take the changes in PR-517, and then check changes in on top of those to fix the problem your way?  At least, you get my unit test, which was for a condition (DRILL-4704) that was never tested by any earlier unit test.
On Monday, August 8, 2016 3:25 PM, Jinfeng Ni<EMAIL_ADDRESS>wrote:

I'm not fully convinced that we should check the precision for each input value for this castIntDecimal function. The argument of proposed patch is parameter "precision=0" is not valid, and has to be calculated on-the-fly or "dynamically" for each input value. To me, 1) if "precision=0" is a wrong input to this function, then we should fix the place where precision=0 is passed in. 2) why would you treat "precision=0" only? what if I pass in "precision = 1", and a integer value of 123456? in such case, do we think "precison=1" is valid or not? Regarding the overflow problem, that seems to be a separate issue, and probably is true for most of the existing decimal functions. —
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

This is the prototype patch I tried [1]. I replied with the link in our prior direct communication.
[1] https://github.com/jinfengni/incubator-drill/commits/review/DRILL-4704

Got it.  I added a commit, but cleaned up your changes to centralize the duplicated code in a function.  As discussed, this method might increase the chances of an overflow, but it avoids changing the cast.  Also, if there are any other places in the code that pass an invalid (zero) precision to the cast, those will now exhibit the same bug, as the cast is vulnerable (without my earlier fix) to such a bad argument.
On Monday, August 8, 2016 3:58 PM, Jinfeng Ni<EMAIL_ADDRESS>wrote:

This is the prototype patch I tried [1]. I replied with the link in our prior direct communication. [1] https://github.com/jinfengni/incubator-drill/commits/review/DRILL-4704—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Since we will have a discussion for long term plan of decimal type support (tomorrow hangout), I think it makes sense to apply a small-scoped change to fix this incorrect query result issue. For the overflow problem or decimal type design in general, we could  continue our discussion either in hangout or dev list.
Based on that, I'm inclined to merge this patch. @amansinha100 , could you please take a look at the new change, and see if it addressed your prior concern?

couple of minor comments; otherwise LGTM.

I'll make minor change to the commit based on Aman's comment, and re-run regression and merge if everything is fine.


Bug: Error when trying to filter language to 'csharp' when also using '-d' filter

When filtering results, if using the language csharp on its own, it is fine, but if I add -d today, I get the error seen in the details dropdown below:
Error details

 `-l csharp -d today`
zsh  starcli -l csharp -d today
Traceback (most recent call last):
  File "/home/mosthated/.pyenv/versions/3.9.9/bin/starcli", line 8, in <module>
    sys.exit(cli())
  File "/home/mosthated/.pyenv/versions/3.9.9/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/mosthated/.pyenv/versions/3.9.9/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/mosthated/.pyenv/versions/3.9.9/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/mosthated/.pyenv/versions/3.9.9/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/mosthated/.pyenv/versions/3.9.9/lib/python3.9/site-packages/starcli/__main__.py", line 183, in cli
    tmp_repos = search_github_trending(
  File "/home/mosthated/.pyenv/versions/3.9.9/lib/python3.9/site-packages/starcli/search.py", line 233, in search_github_trending
    gtrending_repo_list = fetch_repos(
  File "/home/mosthated/.pyenv/versions/3.9.9/lib/python3.9/site-packages/gtrending/fetch.py", line 30, in fetch_repos
    raise ValueError(f"Invalid language argument: {language}")
ValueError: Invalid language argument: csharp

This demonstrates it not working, as well as working with the omission of the -d filter

# This works fine:
starcli -l csharp

# This does not
starcli -l csharp -d today

# I also tried a few others, but had the same results
starcli -l csharp -d this-week
starcli --lang csharp -d this-week
starcli -d today -l csharp
starcli -d this-week -l csharp

Just to check, I also tried the following, and it worked fine:
starcli -l rust -d today

Python version
3.9.9

your operating system (and terminal type or shell of needed)

System Details
mosthated@pop-os
----------------
OS: Pop!_OS 21.10 x86_64
Host: X570 AORUS ELITE -CF
Kernel: 5.15.15-76051515-generic
Uptime: 7 days, 16 hours, 39 mins
Packages: 3947 (dpkg), 194 (brew), 76 (flatpak), 13 (snap)
Shell: zsh 5.8
Resolution: 1920x1080, 2560x1080, 1920x1080
DE: GNOME 40.5
WM: Mutter
WM Theme: Materia-dark-compact
Theme: Materia-dark-compact [GTK2/3]
Icons: Flatery-Orange-Dark [GTK2/3]
Terminal: alacritty
CPU: AMD Ryzen 9 3950X (32) @ 3.500GHz
GPU: NVIDIA GeForce RTX 2070
Memory: 53130MiB / 64314MiB

starcli version
starcli==2.17.1

Hi, I think I know what's causing this inconsistent behaviour. I'll get to it as soon as I get the chance, thanks!

It seems like the GitHub search API uses 'csharp' whereas GitHub Trending uses 'c#'. To have consistency we could change 'csharp' to 'c#' before calling the gtrending functions. Note that it appears there's also inconsistency with 'cpp' vs 'c++', possibly others too. It'd be nice to have all the language parameters for GitHub search work for GitHub Trending as well.
Apologies for the delay @Treehole, are you still interested in working on this?

Hello there!


Add Ankur to Contributors list

could you please add your name to https://github.com/Roshanjossey/first-contributions/blob/master/Contributors.md rather than creating a new file?

@Ankurk99, I'm closing this pull request as you've already done this in another pull request.


JAX-RS endpoints without @ApiResponse annotation get a single "default" response code instead of "200"

Example endpoint implementation:
	@GET
	@Produces(MediaType.APPLICATION_JSON)
	@Operation(summary = "Retrieve an existing metro")
	public MetroDto get() {
		MetroVO vo = load();
		return MetroDto.from(vo);
	}

Generated openapi:
    "/metros/{id}" : {
      "get" : {
        "summary" : "Retrieve an existing metro",
        "operationId" : "get_3",
        "parameters" : [ {
          "name" : "id",
          "in" : "path",
          "required" : true,
          "schema" : {
            "type" : "integer",
            "format" : "int32"
          }
        } ],
        "responses" : {
          "default" : {
            "description" : "default response",
            "content" : {
              "application/json" : {
                "schema" : {
                  "$ref" : "#/components/schemas/MetroDto"
                }
              }
            }
          }
        }
      },

The openapi spec says:

The default MAY be used as a default response object for all HTTP codes that are not covered individually by the Responses Object.
The Responses Object MUST contain at least one response code, and if only one response code is provided it SHOULD be the response for a successful operation call.

https://github.com/OAI/OpenAPI-Specification/blob/main/versions/3.1.0.md#responsesObject
The Swagger OpenAPI guide (https://swagger.io/docs/specification/describing-responses/) says:

You can use the default response to describe these errors collectively, not individually. “Default” means this response is used for all HTTP codes that are not covered individually for this operation.

Consequently:

JAX-RS endpoint is responding with 200
spec requires at least one success code in Responses Object
"default" is a fallback not to be interpreted as "success" and allowed only in addition to other responses

=> "default" should not be used as a single response code in Responses Object.

#4482 introduces configuration property defaultResponseCode which allows to define the HTTP Status Code to use for such cases, replacing the default default.
Being this possibly opinionated and to maintain backward compatibility default is used when no config property is provided
Closing ticket, please reopen if still experiencing issues


ベンチマーク用CI修正

ベンチマーク用のCIがコメント部分で落ちていたので修正した。

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (dd227b1) 70.25% compared to head (a1041f8) 70.25%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main       #2   +/-   ##
=======================================
  Coverage   70.25%   70.25%
=======================================
  Files           5        5
  Lines         279      279
=======================================
  Hits          196      196
  Misses         70       70
  Partials       13       13

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


About US UI Improvements

#1
There is a   About US  page , app/about/page.js , Your Task is to make it Better
Comment below /Attempting and Start working on it

Go On,
All the Best.
After Contributing Please Add your data in    Contributing

Assign me this issue


Question: type checking on create/find/update/delete input props

Is it possible somehow to have type checking on these situations?
import  { User } from './models/User';

await User.create({
    // <-- type checking here for User props
});

User.destroy({
            where: {
                // <-- type checking here for User props
            }
        });

// etc

Hey @spinlud, yeah it is indeed possible, but need to be implemented in sequelize. (Since version 1.0.0 sequelize-typescript uses the official typings provided by sequelize). I'm planning to create a PR for that.

Ok! Let me know if I can help in anyway ;-)


Issue running x64 ELF since 1803 Build Windows [Version 10.0.17134.1]

Microsoft Windows [Version 10.0.17134.1]

Since the latest Upgrade of Windows I have trouble running ELF provided by a custom yocto sdk. This used to work before the latest Windows Update.  This looks exactly like Issue #1048

/opt/sdk-linux/usr/bin/arm-my-linux/arm-my-linux-gcc
cannot execute binary file: Exec format error

Some output:
readelf -e /opt/sdk-linux/usr/bin/arm-my-linux/arm-my-linux-gcc

ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x4047f0
  Start of program headers:          64 (bytes into file)
  Start of section headers:          852312 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         9
  Size of section headers:           64 (bytes)
  Number of section headers:         30
  Section header string table index: 29

[...]

 INTERP         0x0000000000000238 0x0000000000400238 0x0000000000400238
                 0x0000000000001000 0x0000000000001000  R      1
      [Requesting program interpreter: /opt/sdk-linux/lib/ld-linux-x86-64.so.2]

ldd /opt/sdk-linux/usr/bin/arm-my-linux/arm-my-linux-gcc
        linux-vdso.so.1 =>  (0x00007fffd6021000)
        libstdc++.so.6 => /opt/sdk-linux/usr/bin/arm-my-linux/../../lib/libstdc++.so.6 (0x00007fb85b5c0000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fb85b2a0000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fb85b080000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb85acb0000)
        /opt/sdk-linux/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007fb85ba00000)

/opt/sdk-linux/lib/ld-linux-x86-64.so.2 /opt/sdk-linux/usr/bin/arm-my-linux/arm-my-linux-gcc
arm-my-linux-gcc: fatal error: no input files
compilation terminated.

I'm also having an issue with yocto sdk since last update.
When checking for compiler ($CC) after sourcing the yocto sdk i get:
-bash: /mnt/e/edws/svn-edws/s3/S3Trunk/ThirdPartyTools/yocto/sdk/sysroots/x86_64-pokysdk-linux/usr/bin/arm-poky-linux-gnueabi/arm-poky-linux-gnueabi-gcc: cannot execute binary file: Exec format error

A regression with > PATH_MAX (4096) is possible as that was the fix there but most likely, I suspect you need this great workaround for 32 bit support: https://github.com/Microsoft/wsl/issues/2468#issuecomment-374904520.

After downgrade to previous the Windows Version I can run the binary again. No Workaround or anything required.

I confirm what @jebos said.
I revert the Windows to the previous build 1709, and it is working fine again.
Hope this issue will be solved soon, otherwise i cannot update Windows to 1803 build.

The issue can't be fixed if we don't know where the fail is. Hence the strace and repro steps. We have clarity on the version of Windows that has the issue at least.

Seem to have the same issue here on Windows 1803 (build 17134.48), this used to work for me on 1709:
/opt/sdk/sysroots/x86_64-sdk-linux/usr/bin/protoc-c
-bash: /opt/sdk/sysroots/x86_64-sdk-linux/usr/bin/protoc-c: cannot execute binary file: Exec format error

strace /opt/sdk/sysroots/x86_64-sdk-linux/usr/bin/protoc-c
execve("/opt/sdk/sysroots/x86_64-sdk-linux/usr/bin/protoc-c", ["/opt/sdk/sysroots"...], [/* 18 vars */]) = -1 ENOEXEC (Exec format error)
write(2, "strace: exec: Exec format error\n", 32strace: exec: Exec format error
) = 32
exit_group(1)                           = ?
+++ exited with 1 +++

readelf -e /opt/sdk/sysroots/x86_64-sdk-linux/usr/bin/protoc-c
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x404840
  Start of program headers:          64 (bytes into file)
  Start of section headers:          222128 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         9
  Size of section headers:           64 (bytes)
  Number of section headers:         29
  Section header string table index: 28

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL<PHONE_NUMBER>000000  00000000
<PHONE_NUMBER>000000<PHONE_NUMBER>000000           0     0     0
  [ 1] .interp           PROGBITS<PHONE_NUMBER>400238  00000238
<PHONE_NUMBER>001000<PHONE_NUMBER>000000   A       0     0     1
  [ 2] .note.ABI-tag     NOTE<PHONE_NUMBER>401238  00001238
<PHONE_NUMBER>000020<PHONE_NUMBER>000000   A       0     0     4
  [ 3] .note.gnu.build-i NOTE<PHONE_NUMBER>401258  00001258
<PHONE_NUMBER>000024<PHONE_NUMBER>000000   A       0     0     4
  [ 4] .hash             HASH<PHONE_NUMBER>401280  00001280
<PHONE_NUMBER>0004d0<PHONE_NUMBER>000004   A       5     0     8
  [ 5] .dynsym           DYNSYM<PHONE_NUMBER>401750  00001750
<PHONE_NUMBER>000a68<PHONE_NUMBER>000018   A       6     1     8
  [ 6] .dynstr           STRTAB<PHONE_NUMBER>4021b8  000021b8
<PHONE_NUMBER>00161f<PHONE_NUMBER>000000   A       0     0     1
  [ 7] .gnu.version      VERSYM<PHONE_NUMBER>4037d8  000037d8
<PHONE_NUMBER>0000de<PHONE_NUMBER>000002   A       5     0     2
  [ 8] .gnu.version_r    VERNEED<PHONE_NUMBER>4038b8  000038b8
<PHONE_NUMBER>0000d0<PHONE_NUMBER>000000   A       6     3     8
  [ 9] .rela.dyn         RELA<PHONE_NUMBER>403988  00003988
<PHONE_NUMBER>000108<PHONE_NUMBER>000018   A       5     0     8
  [10] .rela.plt         RELA<PHONE_NUMBER>403a90  00003a90
<PHONE_NUMBER>000738<PHONE_NUMBER>000018  AI       5    24     8
  [11] .init             PROGBITS<PHONE_NUMBER>4041c8  000041c8
<PHONE_NUMBER>000017<PHONE_NUMBER>000000  AX       0     0     4
  [12] .plt              PROGBITS<PHONE_NUMBER>4041e0  000041e0
<PHONE_NUMBER>0004e0<PHONE_NUMBER>000010  AX       0     0     16
  [13] .text             PROGBITS<PHONE_NUMBER>4046c0  000046c0
<PHONE_NUMBER>028222<PHONE_NUMBER>000000  AX       0     0     16
  [14] .fini             PROGBITS<PHONE_NUMBER>42c8e4  0002c8e4
<PHONE_NUMBER>000009<PHONE_NUMBER>000000  AX       0     0     4
  [15] .rodata           PROGBITS<PHONE_NUMBER>42c900  0002c900
<PHONE_NUMBER>00458d<PHONE_NUMBER>000000   A       0     0     32
  [16] .eh_frame_hdr     PROGBITS<PHONE_NUMBER>430e90  00030e90
<PHONE_NUMBER>00054c<PHONE_NUMBER>000000   A       0     0     4
  [17] .eh_frame         PROGBITS<PHONE_NUMBER>4313e0  000313e0
<PHONE_NUMBER>0028ac<PHONE_NUMBER>000000   A       0     0     8
  [18] .gcc_except_table PROGBITS<PHONE_NUMBER>433c8c  00033c8c
<PHONE_NUMBER>0016ac<PHONE_NUMBER>000000   A       0     0     4
  [19] .init_array       INIT_ARRAY<PHONE_NUMBER>635d90  00035d90
<PHONE_NUMBER>000010<PHONE_NUMBER>000008  WA       0     0     8
  [20] .fini_array       FINI_ARRAY<PHONE_NUMBER>635da0  00035da0
<PHONE_NUMBER>000008<PHONE_NUMBER>000008  WA       0     0     8
  [21] .jcr              PROGBITS<PHONE_NUMBER>635da8  00035da8
<PHONE_NUMBER>000008<PHONE_NUMBER>000000  WA       0     0     8
  [22] .dynamic          DYNAMIC<PHONE_NUMBER>635db0  00035db0
<PHONE_NUMBER>000240<PHONE_NUMBER>000010  WA       6     0     8
  [23] .got              PROGBITS<PHONE_NUMBER>635ff0  00035ff0
<PHONE_NUMBER>000010<PHONE_NUMBER>000008  WA       0     0     8
  [24] .got.plt          PROGBITS<PHONE_NUMBER>636000  00036000
<PHONE_NUMBER>000280<PHONE_NUMBER>000008  WA       0     0     8
  [25] .data             PROGBITS<PHONE_NUMBER>636280  00036280
<PHONE_NUMBER>000010<PHONE_NUMBER>000000  WA       0     0     8
  [26] .bss              NOBITS<PHONE_NUMBER>6362a0  00036290
<PHONE_NUMBER>000350<PHONE_NUMBER>000000  WA       0     0     32
  [27] .gnu_debuglink    PROGBITS<PHONE_NUMBER>000000  00036290
<PHONE_NUMBER>000010<PHONE_NUMBER>000000           0     0     1
  [28] .shstrtab         STRTAB<PHONE_NUMBER>000000  000362a0
<PHONE_NUMBER>00010c<PHONE_NUMBER>000000           0     0     1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), l (large)
  I (info), L (link order), G (group), T (TLS), E (exclude), x (unknown)
  O (extra OS processing required) o (OS specific), p (processor specific)

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x0000000000400040 0x0000000000400040
                 0x00000000000001f8 0x00000000000001f8  R E    8
  INTERP         0x0000000000000238 0x0000000000400238 0x0000000000400238
                 0x0000000000001000 0x0000000000001000  R      1
      [Requesting program interpreter: /opt/sdk/sysroots/x86_64-sdk-linux/lib/ld-linux-x86-64.so.2]
  LOAD           0x0000000000000000 0x0000000000400000 0x0000000000400000
                 0x0000000000035338 0x0000000000035338  R E    200000
  LOAD           0x0000000000035d90 0x0000000000635d90 0x0000000000635d90
                 0x0000000000000500 0x0000000000000860  RW     200000
  DYNAMIC        0x0000000000035db0 0x0000000000635db0 0x0000000000635db0
                 0x0000000000000240 0x0000000000000240  RW     8
  NOTE           0x0000000000001238 0x0000000000401238 0x0000000000401238
                 0x0000000000000044 0x0000000000000044  R      4
  GNU_EH_FRAME   0x0000000000030e90 0x0000000000430e90 0x0000000000430e90
                 0x000000000000054c 0x000000000000054c  R      4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     10
  GNU_RELRO      0x0000000000035d90 0x0000000000635d90 0x0000000000635d90
                 0x0000000000000270 0x0000000000000270  R      1

 Section to Segment mapping:
  Segment Sections...
   00
   01     .interp
   02     .interp .note.ABI-tag .note.gnu.build-id .hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn .rela.plt .init .plt .text .fini .rodata .eh_frame_hdr .eh_frame .gcc_except_table
   03     .init_array .fini_array .jcr .dynamic .got .got.plt .data .bss
   04     .dynamic
   05     .note.ABI-tag .note.gnu.build-id
   06     .eh_frame_hdr
   07
   08     .init_array .fini_array .jcr .dynamic .got

Going back to 1709 fixed this for me as well.

Ok, a larger strace is needed. Should of provided more clarity. strace -f -o strace -o. Create a gist and link to it here. You should have both parent and child processes captured. Since it's known to work on 1709. Performing the strace on both would prove helpful in seeing the delta.

An strace isn't going to be useful. Would it be possible to upload the failing binary in question?

Here is the binary in question (zipped):
protoc-c.zip

@nsgundy - Thanks for sharing the binary, that made my life easy.  Looks like there was an off-by-one error introduced to our ELF parser.  I have submitted a fix that will get this working.
I'm of the opinion that we should backport this to 1803 but in order to get signoff I'll need to make a case.  So far, as I understand it, some yocto tools that are provided by their SDK do not run.  Those with more context on this thread, could you please provide me more context or details that I can use to help justify releasing this fix via Windows Update?
Thanks!

@benhillis Thanks, this would be great.
My case is that I'm using wsl on my daily work, cross-compiling for arm embedded linux. Without the patch I need to stay away from 1803. So 1803 stops me from working in Windows 10. Staying away from 1803 for long (until FALL) won't be an option due to our policies ect. Having a patch for 1803 would be really appreciated.
If there is no patch incoming, I fear that I will be forced to switch back to my native Linux for daily development work... that would be a shame since WSL worked flawless for me since over a year.
Thanks  again. Give me a note if you need more input on making a case for the backport.

@jebos - That is exactly what I was looking for, thank you!

Just stumbled on this myself.   This also breaks the Zephyr (https://www.zephyrproject.org/ , https://github.com/zephyrproject-rtos/) SDK, which is another Yocto-derived toolchain suite.

Great finding @benhillis . Thank you
Me and my team use WSL as our development environment to do cross-compiling for ARM processors and we use yocto toolchain for that.
It would be great to fix this in 1803, otherwise I cannot update Windows 10.
At this moment it is essential for me to use WSL.
I hope we can see this fixed in 1803 soon :)
Thanks again

@benhillis Thanks for looking into this! I am in the same boat as @jebos. As it so happens, got my fingers crossed for a patch...

Thanks everyone, I think we have a very good case for getting this fix backported.  The fix for this has been checked in and will be making its way to the flighting branch in the coming weeks.  After that fix is public it would be useful for somebody on this thread to ensure this fixes all the issues they are having.

Standing by :) 1803 is now knocking on my door every day and I have to tell it to go away manually since I cannot change the update settings as that's controlled centrally.

Hey all, I'm affected by this as well with a Yocto built SDK that no longer functions after updating to 1803.  Looking forward to the fix. Member of our development team that made the decision to run Windows are disrupted.
Downgrading to 1709 resolved this.

Downgrading to 1709 fixed it.
Pause updates checked, hopefully it gets fixed by 6/22/18!

Hey here,
opened #3259 by mistake (didn't see that one).
exactly the same problem here.
Will follow this ticket to be notified when the fix is back-ported to 1803.
@benhillis : any idea about an expected ETA?
thanks!

This is fixed in 17686, please let me know if this resolves your issues and I will begin the backporting process.

@benhillis Unfortunately I'd like to stick to the stable releases... is there a less-intrusive way to test this?

Unfortunately not.

Second the issue regarding the Zephyr SDK mentioned by @andyross. Our toolchain is yocto-based and we suffer from the same bug  in 1803, which is preventing Zephyr developers from building on WSL.

Thanks for confirming, I've started the backporting process.

My work's domain policy does not allow Insiders program. Is there a way to manually apply a fix as I cannot revert back to 1709?

There isn't a way to manually apply the fix. But maybe Ben can give some details about what makes these Yocto binaries so "weird" in the first place. Possibly you can just use the stock Ubuntu arm gcc cross compiler as a work-around. I am not clear on whether there is something special about Yocto's gcc that makes using their compiler build (contrast sdk) strictly necessary.
It would be better if Yocto's binary just worked, natch. But since you're locked into a "can't go back can't go forward" situation, I'm just looking for possible solutions in the interim. If the difference in their binary isn't critical, maybe it could even be addressed upstream. Anyway just spit-balling.

@benhillis is there any ETA for the backport to 1803? Thanks for your effort.

@jebos - It's in the pipeline, but I don't know much beyond that.

Just upgrade to 17134.191 today, still not fixed yet. Doesn't the backport process take too long time? More than one month already, the best bet is next Tuesday update, though it's two weeks away, yet who knows whether it will be there. Maybe we should wait another month?
Does microsoft force all users to go insider? This is not practical, the new half-year release/upgrade cycle is already fast enough for most users.

Any news?

@benhillis Any news on the backport ? It's been a couple of months since any update here and I'm still experiencing this exact issue on 1803. Thanks

Unfortunately due to some circumstances out of my control this fix got pushed to the next servicing Window. October is the last date I heard. Sorry for the delay, I'm frustrated with this as well.

@benhillis does 1809 fix this issue?

@carlescufi - Yes the fix for this issue is in 1809.

I'm assuming that Windows server 2019 that is releasing imminently will already have this fix in it ? We are moving out build servers to that once it releases to take advantage of WSL instead of requiring a separate Linux or Docker setup.

@chrisella - correct, Server 2019 has the fix as well.

Not fixed for ELF32, at least not for me:
readelf -e arm-v7a8v4r3-linux-gnueabi-gcc
ELF Header:
Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
Class:                             ELF32
Data:                              2's complement, little endian
Version:                           1 (current)
OS/ABI:                            UNIX - System V
ABI Version:                       0
Type:                              EXEC (Executable file)
Machine:                           Intel 80386
Version:                           0x1
Entry point address:               0x804cf58
Start of program headers:          52 (bytes into file)
Start of section headers:          1591256 (bytes into file)
Flags:                             0x0
Size of this header:               52 (bytes)
Size of program headers:           32 (bytes)
Number of program headers:         8
Size of section headers:           40 (bytes)
Number of section headers:         38
Section header string table index: 35

@ad-on-is That's because you need to use qemu for 32 bit support. See #2468.


Missing simlink for panel.h to brew --prefix/incude

panel.h is missing in the include directory. Breaks llvm-lldb (and standard distros do place panel.h at the top of /usr/include, ex: centos)

Another file is missing, nvm


Link to further explanation of The Elastic Band

Towards #125. @rbyers PTAL.

Codecov Report

Merging #132 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #132   +/-   ##
=======================================
  Coverage   60.69%   60.69%
=======================================
  Files          35       35
  Lines        1768     1768
  Branches      134      134
=======================================
  Hits         1073     1073
  Misses        695      695

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4a6f57e...e72dd79. Read the comment docs.


[POC] performance improvements to reduce resource footprint

Fixes: Issue #N
Relates to: Issue #N, PR #N, ...
Changes introduced with PR works only with the disk image created with SNC PR
More details are documented here at this doc
Solution/Idea

Reduce resource footprint for CRC users
Enable CRC users to customize their Single node OpenShift environment

Proposed changes
At this time, these POC changes work only with SNC disk image created with this PR.

Reduce resource footprint for CRC users
Enable CRC users to customize their Single node OpenShift environment. With this customization, users would have the ability to (a) disable OLM (b) disable cluster monitoring

Testing

Users can disable OLM with crc config set disable-olm flag
Users can disable Cluster monitoring with crc config set disable-cluster-monitoring flag
Users can specify memory which can be less than 9 GB

Can one of the admins verify this patch?

/ok-to-test

Flag for enabling cluster monitoring is now in master.

/cat

/meow

/meow

sorry for spamming this PR! - I was showing the plugin :|


Fix load to respect existing environment variables over env file when doing variable interpolation

Closes issue #319.

:wave: @cbjoseph !
Apologies that this has been sitting for so long, I'm part of a trio of new maintainers of this project, and we're trying to clear out the backlog of folks waiting for responses.
There appears to be a merge conflict associated with some of our work clearing up said backlog.  Could you please resolve the conflict so I can review?
Thank you!

@cadwallion No Problem, added new pull request #332 with no merge conflicts

@cadwallion resolved conflicts! Thanks for taking a look.

@cbjoseph it looks like there is one rubocop error:
lib/dotenv/substitutions/variable.rb:44:1: C: Trailing whitespace detected.

I believe fixing that up should be the last step.

@jonmagic oops! fixed rubocop error, should be all good now

I think that something is still off with the implementation.
balrog:~ n$ cat .env
TEST=5
TEST_COMBINED=combined_${TEST}
balrog:~ n$ TEST=10 ruby dotenv.rb
Before loading .env...

TEST=10
TEST_COMBINED=

{"TEST"=>"5", "TEST_COMBINED"=>"combined_10"}
After loading .env...

TEST=5
TEST_COMBINED=combined_10

#!/usr/bin/env ruby
# frozen_string_literal: true

def display_envs()
  puts
  puts "TEST=#{ENV['TEST']}"
  puts "TEST_COMBINED=#{ENV['TEST_COMBINED']}"
  puts
end

puts "Before loading .env..."
display_envs

require 'dotenv'
puts Dotenv::Environment.new('.env', true).inspect
ENV.update(Dotenv::Environment.new('.env', true))

puts "After loading .env..."
display_envs

/cc @cbjoseph @cadwallion

@cadwallion Can you comment on expectation of this Environment class? My assumption was that it's an internal tool, since it seems undocumented. Could you provide more context on its use case?

This change just caught us off guard as well with a gem update.
@cbjoseph: Is it documented anywhere that Dotenv::Environment is internal and subject to breaking changes?
Ideally, any changes to a method signature should be backwards compatible.
What does everyone think about making the is_load parameter here default to true?
i.e.
    def initialize(filename, is_load = true)
      @filename = filename
      load(is_load)
    end

I also wanted to jump in that this broke many setups for fastlane and fastlane.ci, as it was introduced with a minor version bump. Adding a default parameter with https://github.com/bkeepers/dotenv/pull/345 sounds like a great fix 👍

Curious to know if Dotenv::Environment should be considered public given
that it's not documented.


Bump copyright year in README.md

Description
This PR bumps the copyright year in the README.md file.
Happy new year! 🎉
Note
This PR should only be merged once a change to any content of this repository was done.

Internal Tracking ID: EXPOSUREAPP-14521

@Ein-Tim We can close this one, right?

@dsarkar
Yes, as this was done with https://github.com/corona-warn-app/cwa-hotline/pull/34.
Seems like I'll never get the Contributor badge in this repository 🥲 :D


Fix links in CONTRIBUTING.md

CHANGES and NEWS were renamed into CHANGES.md and NEWS.md; this commit
updates the references inside CONTRIBUTING.md to fix broken links.
Checklist

[x] documentation is added or updated

This pull request is ready to merge

I will soon merge this!

El El lun, 27 abr 2563 BE a las 08:08, Nicola Tuveri <
<EMAIL_ADDRESS>escribió:

I will soon merge this!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/openssl/openssl/pull/11634#issuecomment-619942978,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJL2S2BUT3KWDK3BAZCHA63ROVYTHANCNFSM4MQHR6DA
.

--
Jackson Chung

Merged to master with ee6c7cde752992f0fb63820a2c5e63496fd686b1
Thanks!


SQL Imports: Another iteration on the UX and underlying API

Description
Building on #655, This change accomplishes a few things:

Move the responsibility for updating the step state to the upper level (e.g. no progress setting scattered throughout client-file-uploader, search & replace, & sql validation functionality)
Simplify the code needed in the upper level (e.g. only needing to set the success and failure status with the lib handling more of the mechanics)
Combine the output of the "preflight" steps that are hard-coded in the client with the importing status that comes from the server. (Try as I might, I could not get the single-log-line lib to play nicely unless all the input was combined prior to writing to stdout.)

Utilize a prefix and suffix to enable printing lines before and after the single-log-line writes.

Make the ProgressTracker an object instance and consolidate state inside it
Pull in enhancements from https://github.com/Automattic/vip/pull/650

Steps to Test

Check out PR.
Run npm run build
Run ./dist/bin/vip-import-sql.js (with required fields)

Verify the output is an improved experience

While an import is running run ./dist/bin/vip-import-sql-status.js

Verify the output is an improved experience

Attempt to cause failures

They should be handled gracefully and attempt to show actionable steps

We'll probably want to handle that error more gracefully too (in another PR).
We could also check for that case as well ! fs.statSync(filePath).isDirectory()

We'll probably want to handle that error more gracefully too (in another PR).
We could also check for that case as well ! fs.statSync(filePath).isDirectory()

Directory ("not a file") gating added in PR #660

Directory ("not a file") gating added in PR #660


feat: trpc with github fetching

:tada: This PR is included in version 0.1.0-dev.7 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

:tada: This PR is included in version 0.1.0-stage.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

:tada: This PR is included in version 0.1.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


High memory consumption in python xgboost

I'm working on python AutoML package and one of my users reported very high memory usage while using xgboost.
I've made an investigation to show memory consumption by xgboost. You can find the notebook here. From the code, you can see that the model allocate over 7GB of RAM memory. When I save the model to hard disk (5 kB !) and then load it back I can save a huge amount of RAM.
For me, it look's like xgboost is storing the copy of data in its structure? Am I right?
Is there any way to slim down memory usage by xgboost? Do you think that saving model to the hard drive and then loading it back is way to handle this issue?

@pplonski We are trying to eliminate the copy for histogram algorithm.  It'a working in progress.  For GPU is mostly done: https://github.com/dmlc/xgboost/pull/5420 https://github.com/dmlc/xgboost/pull/5465
CPU still have some more work to do.

@pplonski, we implemented reducing memory consumption on CPU also in this PR https://github.com/dmlc/xgboost/pull/5334, but for 'hist' method only. It's included in master only for now, but I hope it will be a part of the future release.

memory, Kb
Airline
Higgs1m

Master
28311860
1907812

This PR:
16218404
1155156

reduced:
1.75
1.65

Agree with @trivialfis, there are many things to do in the area.

Hi, I have recently faced a similar high memory problem with xgboost. I am using 'gpu_hist' for training.
I notice large system memory spikes when train() method is executed, which leads to my jupyter kernel crashing.

Is it correct to say that Xgboost is making a copy of my data in system RAM (even when I'm using 'gpu_hist')?
I was under the assumption that xgboost loads the entire training data to the GPU. Is that also incorrect?

The memory usage has since improved a lot with inplace predict and DeviceQuantileDMatrix.  Free free to try them out.  One issue remain is sometimes CPU algorithm uses memory linear to n_threads * n_features.

xgboost.train() also consume a lot of memory(not gpu memory) when making a copy of Booster as returned model , in my case 9GB before bst.copy() and 34G after.
What about not making copy as a option?

What about not making copy as a option?

I'm working on it.

I am not really sure whether Booter.save_model() would produce exactly the same model file If bst.copy() is ommited when Booster.train() return. Would it be safe that Bootsr.train() simply return bst directly without copy If nothing happen between Bootster.train() and Bootser.save_model()? @trivialfis

Copy is exact, no change happens during the copy.

Copy is exact, no change happens during the copy.

Thanks for the reply, really appreciate it. Looking forward for the upcoming release!

We have implemented QuantileDMatrix for hist tree method and set it as default for sklearn, along with which, inplace prediction is used for sklearn whenever feasible. Other than these, some additional memory reduction optimizations for hist are implemented. Lastly, UBJSON is used as the default serialization format for pickle. I think we can close this issue and consider other memory reduction work as further optimization.


Text for cata parser.

Pull Request Description
Brings back support for parsing text literals
Important Notes
This is draft. Although all tests are passing, some functionality like correct offsetzip and multiline text parsing are missing.
Checklist
Please include the following checklist in your PR:

[x] The documentation has been updated if necessary.
[x] All code conforms to the Scala, Java or Haskell style guides as appropriate.
[x] All code has been tested where possible.

Please label your PRs.


add initial image argument to neural-style example

This adds a feature to set the initial image to be optimized from the command line for the neural-style example. This can be useful if you want to first run the program with a smaller max_long_edge, then use that for the initial image for a longer max_long_edge as this can help with speed optimizations. This may also be useful for general experimentation.

line 140, it should be 'arg.initial_image' rather than 'arg.start_image' @JacobEdelman

Fixed.


Recommended practice for handling unknown escape sequences

I see in #125 that sometimes tcell needs new entries for escape sequences. However I also gather from other issues that tcell won't support some escape sequences as policy - for not being ubiquitous enough.
In particular I'm seeing that CtrlPgUp and CtrlPgDn generate a sequence of runes, indicating that these keys aren't supported. Ideally I'd like to offer these keys in my app, but at the least I'd like a way to not have these keys trigger irrelevant key events.
What's the recommending thing to do in this case?

Probably we could add extra symbols and you can define terminals that have
them. These will be highly non portable and unlikely to work in many
terminal or window environments.
I am not opposed to adding these if we know what the values are. I just
think most programs should avoid them because they will often fail to work.
On Fri, Jul 20, 2018, 11:24 PM Thomas Buckley-Houston <
<EMAIL_ADDRESS>wrote:

I see in #125 https://github.com/gdamore/tcell/pull/125 that sometimes
tcell needs new entries for escape sequences. However I also gather from
other issues that tcell won't support some escape sequences as policy - for
not being ubiquitous enough.
In particular I'm seeing that CtrlPgUp and CtrlPgDn generate a sequence
of runes, indicating that these keys aren't supported. Ideally I'd like to
offer these keys in my app, but at the least I'd like a way to not have
these keys triggered irrelevant key events.
What's the recommending thing to do in this case?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/gdamore/tcell/issues/228, or mute the thread
https://github.com/notifications/unsubscribe-auth/ABPDff3bZEGIc8hC8bYW6pEWNR9C__oxks5uIskygaJpZM4VZg26
.

Thanks. So I was wondering more about a general solution. For instance would it be possible to use the existing escape sequence detection code and have it match an escape sequence regex (if such a thing is possible)? Then an extra method could be added to tcell.EventKey, like .EscapeSequence()? This way all current and future sequences could be handled and it would avoid all these random .Rune()s appearing when an unrecognised escape sequence is sent.

Ok, thanks. I'm also happy to have a go at putting together a PR, if you decide on an approach.

So the status update here is that the database have changed quite a bit.  We should add new definitions where appropriate.  If you want to start a PR please do so.

Please see #369, which I think completely addresses this  (at least for modern emulators that are conformant with xterm).

Closing this old issue.


Refine and cleanup DeprecationWarnings.

all warnings should mention a version, either since when things are
deprecated, or when they will be removed. Otherwise they are not
actionable by users or future devs without looking through git
history, which nobody does and then get never cleaned up.

Things that we say we would removed have not been removed, and thus
removing them.

(2) seem to indicate that in (1) we should not try to say things about
the future, and it's better to use version when things are deprecated as
this is factual and we can change our mind

Thanks for pushing this towards some consistency @Carreau !
I think we only just pushed a change that ensure that DeprecationWarnings are issued in napari right. So I'm a little worried that we're removing some things without warning to some people.
Also, one other thing that comes to mind is that targeting removal of deprecated things in patch versions seems aggressive - given napari's current version numbering, I'd prefer to target minor version changes. And I think we should try to include the target version of removal. So maybe we can delay the removals until 0.5?
Also, I think we're trying to follow the numpydoc style guide (though I'm not sure), in which case we may also want to consistently update the docstrings.

(2) seem to indicate that in (1) we should not try to say things about
the future, and it's better to use version when things are deprecated as
this is factual and we can change our mind

I do not agree. For me as a plugin maintainer who would like to support more than one napari version, it is really important until which version of napari my code will work. It allows me to plan when I need to focus on updating my plugin.

I think we only just pushed a change that ensure that DeprecationWarnings are issued in napari?

I'm not even sure this has been done yet.

[...] Deprecated things in patch versions seems aggressive [...]

We had much bigger changes in patch versions, maybe it's time to do less patch release and more minor ones? Napari 0.4.0 was released in 2020...

I do not agree. For me as a plugin maintainer who would like to support more than one napari version, it is really important until which version of napari my code will work. It allows me to plan when I need to focus on updating my plugin.

Well then we should stick to the removal plan. And then I think the information of since when it is deprecated is as useful if not more, because by the time we publish napari 0.6.31, if you see "foo is deprecated in favor of bar and removed in 0.6.61", you have no clue since when it's deprecated and since when bar is avail. So really you want:

since when things get deprecated.
since when the replacement is available.
Potentially when the deprecation will become an error.

My comment of these three, most of the time 1 and 2 are the same, and when people include 3 in the message most of the time it is not respected. See for example what is in this PR with multiple things marked as removal several versions ago.
I think a good middle ground is to have a clear deprecation policy (say deprecated features stay for at least X minor versions). That way you can state something factual (deprecated since Y), and provide guarantees to consumers.
For napari, I don't particularly care as long as every deprecation warning have a version somewhere, for either deprecation or removal.

If we assume that everything is deprecated until the next major release and one release before the major release we do not deprecate anything I will agree. But a common thing in napari history was deprecating things in minor releases.

My comment of these three, most of the time 1 and 2 are the same, and when people include 3 in the message most of the time it is not respected.

Good example of deprecation without replacement is _qt_viewer.
And as I remember most of the deprecations that touch me was respecting the time of removal.

https://github.com/napari/napari-console/pull/19 will need to be released first, as the warnings were silenced.

Let's start with #4757 that only contains update to messages to add "since X.y.z", that should be non-controversial.

[...] Deprecated things in patch versions seems aggressive [...]

We had much bigger changes in patch versions, maybe it's time to do less patch release and more minor ones? Napari 0.4.0 was released in 2020...

yeah, definitely.  though we three numbers and two dots, our versioning has almost zero semantics :joy:. i'm in favor both of just deprecating things in patch releases (if we stick with our current version bumping hesitations) and being more willing/less scared of minor releases

I have now merged #4757 to keep this going. @Carreau if you can clean up and fix merge conflicts that would be great. It would be nice to get this in, including dropping the deprecated API.
First thing in the Readme is

We're developing napari in the open! But the project is in an alpha stage, and there will still likely be breaking changes with each release.

and I feel like we've worked very hard to have give these warnings the past two years, but we have to still keep moving forward


App controller methods are being called multiple times

These tests are performed on a homepage
I have the following structure:
app/controllers/App.php
app/controllers/Home.php
...
class App extends Controller {
    public function app_test()
    {
         error_log('called App::test');
    }
}

...
class Home extends Controller {
    public function home_test()
    {
         error_log('called Home::test');
    }
}

the result of the error log for a single request is:
called App::test
called App::test
called Home::test
called App::test

So far I have been debugging I found out the following:
\Sober\Controller\loader function adds filters for
sage/template/app-data/data (App controller)
sage/template/home-data/data (Home controller)
Sage's template_include filter then runs the above filters (it actually calls several other based on the body classes)
the filter then creates a new controller for each of the filters it added, in our case it can be translated as
    $controller = new App\Controllers\App(); // First instance of App being created
    $controller->__setup();
    return array_merge($loader->getAppData(), $loader->getPostData(), $controller->__setTreeData($data), $controller->__getData());

and
    $controller = new App\Controllers\Home(); // First instance of Home being created
    $controller->__setup();
    return array_merge($loader->getAppData(), $loader->getPostData(), $controller->__setTreeData($data), $controller->__getData());

So to clarify: We call the __setup method on the controller once for App and then once for Home.
The problem is in the array_merge when $loader calls getAppData which in turn calls __setup on App again
This results in the following calls:
__setup from the sage/template/app-data/data filter on App class (first App instance)
__setup from getAppData on new $this->instances['app']() from sage/template/app-data/data filter (second instance of App being created)
__setup from the sage/template/home-data/data filter on Home class (first instance of Home)
__setup from getAppData on new $this->instances['app']() from filter sage/template/home-data/data (third instance of App)
So I have a solution to migrate the controllers to singletons (not sure if this is viable as we would need protected contructors for a clean design and this might break already existing solutions based on the current controller pattern) - this could also make it harder for dependency injection if it was to be implemented later on
So if anyone has a better idea please share.
I will submit a PR for this in a bit in case nobody comes up with a better solution

Hi, did this make it into 2.0.0? Looks like it from the code but changelog isn't updated. I would also like to try 2.0.0 👍

@slackday yea this working as should now. I forgot to update the changelog, will do that.

@darrenjacoby is this fix will be released on 9.0.0 also (sage user here).


Add default to .getComponentBy*()

This PR introduces default kwarg to the .getComponentBy*() methods which is returned instead of the schema object in case the latter is to be returned.
The aim here is to ease migration from older pyasn1 API where .getComponentBy*() used to return None, then switched to return noValue sentinel:
x.getComponentByName('field')  # -> noValue
x.getComponentByName('field', default=None)  # -> None

Codecov Report

Merging #100 into devel-0.4.1 will increase coverage by 0.02%.
The diff coverage is 95.83%.

@@               Coverage Diff               @@
##           devel-0.4.1     #100      +/-   ##
===============================================
+ Coverage        85.45%   85.48%   +0.02%
===============================================
  Files               29       29
  Lines             3968     3976       +8
===============================================
+ Hits              3391     3399       +8
  Misses             577      577

Impacted Files
Coverage Δ

pyasn1/type/univ.py
83.4% <95.83%> (+0.11%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cddc8b8...aa64175. Read the comment docs.


Tilde prefix not considered absolute path

This relates to https://docs.gradle.org/current/javadoc/org/gradle/api/Project.html#file-java.lang.Object-
It seems that paths starting with tilde(~) are not considered absolute path but relative to the project.
Steps to Reproduce

Create a simple project with gradle init
Patch app/build.gradle.kts with the following (adapt second path to your local config):

application {
    // Define the main class for the application.
    mainClass.set("gradle.path.AppKt")
    println("Tilde absolute path: ${file("~/.gitconfig").absolutePath}")
    println("Absolute path: ${file("/Users/nico/.gitconfig").absolutePath}")
}

Expected Behavior
Tilde absolute path: /Users/nico/.gitconfig
Absolute path: /Users/nico/.gitconfig

Current Behavior
Tilde absolute path: /Users/nico/dev/gradle-path/app/~/.gitconfig
Absolute path: /Users/nico/.gitconfig

Context
Is it an intended behavior or a bug?

Your Environment
macOS 11, gradle 7.0
Build scan URL:

The tilde ~ symbol is a bash-specific expansion. You shouldn't expect any Java program to interpret it, since its meaning is dynamically defined by the bash environment.

This is the expected behavior.
If you want to access the user home, you should use System.getProperty("user.home") instead, which has semantics defined by the JVM.


Extend is_valid_ram_address to LP (RTC) memories

At the moment if we have part of the backtrace in LP memory esp-backtrace will fail.

You might get a stack trace in this region if you call a function that's placed in this ram region, e.g: https://github.com/esp-rs/esp-hal/blob/edd6c16637f9639c2e4de9f7c664c398794c2594/esp32c3-hal/examples/ram.rs#L94-L97

IIRC is_valid_ram_address is only used to check the stack frame address - not the program counter address.
I tested with a panic inside function_in_rtc_ram on C3 and S3 and I got the backtrace.
Do you have an example where it failed to produce a backtrace while it should be able to generate it?

I don't have a concrete case, because I was just theorizing that it was possible :D. I think the case I had in my mind was if function_in_rtc_ram called another function inside rtc ram. I think then you'd get a stacktrace which was partially in rtc ram?

Should work - at least in my test it does: (tested on ESP32-S3 but shouldn't matter)
0x420179a3 - core::panicking::panic
    at C:\Users\Bjoern\.rustup\toolchains\esp\lib\rustlib\src\rust\library\core\src\panicking.rs:117
0x42001e59 - ram::in_flash
    at C:\projects\esp\esp-hal\esp32s3-hal\examples\ram.rs:101
0x600fe00d - ram::function_in_rtc_ram
    at C:\projects\esp\esp-hal\esp32s3-hal\examples\ram.rs:96
0x420027f1 - ram::__xtensa_lx_rt_main
    at C:\projects\esp\esp-hal\esp32s3-hal\examples\ram.rs:81
0x42015193 - Reset
    at C:\Users\Bjoern\.cargo\registry\src\index.crates.io-6f17d22bba15001f\xtensa-lx-rt-0.16.0\src\lib.rs:70
0x403787cf - ESP32Reset

Nice, in which case we can close this! Thanks for looking into it :).


Is the fully connected layer faster than using the OpenBLAS equivalent?

No it's slower unfortunately.

Thank you for reporting this. Fixed in a344d23816542a1c91bc99c238fa520ac4936f9a

I suggest you take a look at XNNPACK library, which is a successor to NNPACK.

Oh nice! Great I'll try it out. :)


Image urls on homepage are hardcoded

Needs to use get_template_directory_uri()

Actually, this is related to #1 since the images are hard-coded via the Custom Fields.

Yes, ACF stores image fields as IDs, so this issue would be resolved by switching to ACF. However, since Custom Fields Suite (the current custom fields plugin) stores images as URLs, we can't simply map existing custom fields to ACF fields as we originally planned. There is definitely going to be some data re-entry involved.

Not that much though. It's just for the homepage. That's why I want to get it done now before we build out more pages and make it even MORE complex.


Build in Travis CI only on master

Build only when we actually want a snapshot built as part of .buildscript/deploy_snapshot.sh

This may not have worked...

Also disabled here https://travis-ci.com/square/okhttp/settings, can reenable if we don't want to land this.

cc @swankjesse


How to center the slider on a page

Hi. Thank you for this slider.
I'm trying to work out how to center the slider when adding to a webpage ie. center on the horizontal between the left and right sides in the browser window.
There's nothing I can seem to find in the code or config which would allow this to happen.
Please advise. Thank you.

Try adding:  .lslide {text-align: center;} to your stylesheet


Figma tokens | Input component

Additional figma tokens need to be created for the input component.

--component-input-read_only_info-border: #FBF6BD
--component-input-read_only_info-background: #FBF6BD
--component-input-read_only_confirm-border: #D4F9E4
--component-input-read_only_confirm-background: #D4F9E4
The figma tokens then have to be taken into use by the component's css.

Added in https://github.com/Altinn/figma-design-tokens/commit/a46dd9e8651b4c780d79dafbba4b52f01652397e


deschedule/balance order

Also generalized RunDeschedulerLoop and RunProfiles for when/if we start working on the descheduler/kube-scheduler simulator.
fix #979

Both RunDeschedulerLoop and RunProfiles can be turned into methods with an PoC new data type definition for the framework. At the end a user is expected to invoke NewFramework or a similar function that will follow with fmwrk.DescheduleOnce(...) or similar.

I think before that we would need to properly transform Descheduler into a kubernetes controller to stop using wait.NonSlidingUntil and wire the other things around.

I think before that we would need to properly transform Descheduler into a kubernetes controller to stop using wait.NonSlidingUntil and wire the other things around. (In short - I think it should come with another PR/PRs)

Ok. Let's focus on:

making the new functions internal so we can revisit the parameters later
reducing the cost of re-creating a profile for each extension point so it can be later extended with other batch of EPs

Just adding a comment to say I decided to go back and turn the functions into methods with an PoC new data type. I am calling it Descheduler, and we have NewDescheduler to get an instance of it populated with all needed stuff, and RunDeschedulerLoop and RunProfiles as methods under it.

/assign @ingvagabund
/cc @damemi @a7i

/test all

Closed in favor of https://github.com/kubernetes-sigs/descheduler/pull/1177. @knelasevero thank you for all the help and work here.


Template changes for Scaffolding

Update the scripts section to match Templates
Fix Edit view template to always add a hidden input for the primary key
Fix select tag helper for foreign keys in the Create and Edit view templates

cc @rustd @sayedihashimi @abpiskunov

cc @balachir @phenning

cc @Eilon, @DamianEdwards  (for RC2 ask mode approval).

Tagging aspnet/Release/issues/4

Why does the Edit view need a hidden input with the primary key? Isn't the primary key already in the URL?

Or is it for cases when the ID isn't called id?

Yes. Its also needed for the form submit.

Ok :shipit: approved for RC2.


Fixed 'SyntaxError: Unexpected end of JSON input' error when lambda r…

…esult is not valid json
Issue #, if available:
Description of changes:
Currently if a lambda fails when running amplify mock function <lambda> and returns a non valid JSON response (such as null or string or Error) then the nodejs lambda invoke fails with this error message:
SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at ChildProcess.<anonymous> (/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-nodejs-function-runtime-provider/lib/utils/invoke.js:45:37)
    at ChildProcess.emit (events.js:315:20)
    at ChildProcess.EventEmitter.emit (domain.js:485:12)
    at maybeClose (internal/child_process.js:1051:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:287:5)

This change catches this issue and outputs the value instead without an attempt to parse to JSON
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

I just ran into this aswell, thanks man, hope they'll merge this soon.

I think I'm also running into this. My lambda returns a string, and I'm invoking it via @function in graphql, and amplify mock. In my case the amplify mock graphql server just crashes with this error:
2020-08-11T17:53:33.835Z - error: uncaughtException: Unexpected end of JSON input date=Tue Aug 11 2020 11:53:33 GMT-0600 (Mountain Daylight Time), pid=60534, uid=501, gid=20, cwd=/Users/hisham/src/ess-app, execPath=/Users/hisham/.nvm/versions/node/v13.14.0/bin/node, version=v13.14.0, argv=[/Users/hisham/.nvm/versions/node/v13.14.0/bin/node, /Users/hisham/.nvm/versions/node/v13.14.0/bin/amplify, mock], rss=346918912, heapTotal=269488128, heapUsed=261629800, external=3655064, arrayBuffers=1612264, loadavg=[2.337890625, 3.1083984375, 3.93701171875], uptime=237844, trace=[column=null, file=null, function=JSON.parse, line=null, method=parse, native=false, column=29<EMAIL_ADDRESS>function=null, line=23, method=null, native=false, column=20, file=events.js, function=ChildProcess.emit, line=315, method=emit, native=false, column=12, file=domain.js, function=ChildProcess.EventEmitter.emit, line=485, method=emit, native=false, column=16, file=internal/child_process.js, function=maybeClose, line=1026, method=null, native=false, column=11, file=internal/child_process.js, function=null, line=441, method=null, native=false, column=20, file=events.js, function=Socket.emit, line=315, method=emit, native=false, column=12, file=domain.js, function=Socket.EventEmitter.emit, line=485, method=emit, native=false, column=12, file=net.js, function=null, line=674, method=null, native=false], stack=[SyntaxError: Unexpected end of JSON input,     at JSON.parse (<anonymous>),     at ChildProcess.<anonymous> (/Users/hisham/.nvm/versions/node/v13.14.0/lib/node_modules/@aws-amplify/cli/node_modules/amplify-nodejs-function-runtime-provider/src/utils/invoke.ts:23:29),     at ChildProcess.emit (events.js:315:20),     at ChildProcess.EventEmitter.emit (domain.js:485:12),     at maybeClose (internal/child_process.js:1026:16),     at Socket.<anonymous> (internal/child_process.js:441:11),     at Socket.emit (events.js:315:20),     at Socket.EventEmitter.emit (domain.js:485:12),     at Pipe.<anonymous> (net.js:674:12)]
This didn't happen before. I think this bug was introduced by @akshbhu in this change: https://github.com/aws-amplify/amplify-cli/commit/3733a161ea09472fe52d5fc8a6515a3e4e726338#diff-08b966f21a964a857ed548461ff5f9a4

@edwardfoyle @SwaySway just bumping this in the hope it can make it into the next release. 👍