[WIP] fix for #2013 Support descending ordered queries

Can you please provide a description of what functionality you are implementing as well as a high level overview of the approach?

This looks like a great start on re-ordering data to return in reverse-time order.
One big thing that this highlighted for me, however, is how the descending flag is being delivered.  It definitely works to add it as has been done, but every time we want to add a new flag like this, we shouldn't have to extend every single query to support it.  So that got me thinking which of the parts of the query would be best to house that flag.  Immediate options as I see them are

granularity - given that the descending flag applies to how the time grain is ordered, adding it here could make sense.  But, at the same time, that reason seems weak.
intervals - kinda the same logic as granularity: it's something that deals with time...
context - it's an extra flag that changes the behavior of queries.  But, so far things in context are somewhat meta.  They are extra things specific to a query type or specific to how Brokers/Historicals comunicate, etc.

After going through all these options and realizing that I don't like any of them, what do people think about introduce a new element to the Query to replace granularity and intervals.  We could call it the timeSpec and it would have 3 fields for now

granularity
intervals
ascending

As we get more things related to how a specific query processes time, we could add them here.  What do people think?  Am I over-thinking it?

One thing I wonder about is how the merge happens when we go in a different time-sort order.  I notice that you haven't made any changes to the comparators used for the merges that take place.  I think those will also have to be adjusted for reversing the time order, but maybe not.

Implemented descending order only for search/timeseries/timebound/topn queries and caching is disabled for them. It's already grown too big for me to handle.

@cheddar I've also thought on the position of descending field and QuerySegmentSpec in BaseQuery looked good position of that (granularity also can be included in it). But I've decided not to make too many changes for now. We can discuss on this when the functionality is settled.
For merging works, I thing I've changed related codes(comparators, etc.) but cannot sure I've done it right. I'll add more tests on current supported query types and also will make group-by and select queries support descending. Thanks.

@cheddar I'm not sure how I feel about a "timeSpec" as the grouping. "intervals" is technically a filter and "granularity" and "direction" both define the formatting of the result. I think a "resultSpec" or "applySpec" is more intuitive to users. At some point we should have a "filterSpec" with "intervals" and "dimensionFilters" in it. Would love to get some feedback from @vogievetsky as well.

I think of all operations in terms of (filter-)split-apply-combine where for timeseries:
Filter - self explanatory (intervals + filter)
Split - how to bucket the data (granularity)
Apply - what to compute per bucket (aggregations + postAggregations)
Combine - how to sort, order, and limit the buckets
The proposed descending flag fits squarely in the combine category of my clarification.
As such I would advice against a timeSpec as it is arbitrary to try and group all time related stuff together.
I would recommend there to be a sortSpec or that direction just be a top level flag on the timeseries query only (just like threshold is on topN)

descending would be about the order of processing, not about ordering of result. So it can be applied to all kind of queries but whether it has any meaning is a different question.
For example, descending processing of time-series queries will make descending ordered result(good). But for group-by queries, the order of processing has no meaning and there are another ordering spec for result ordering in it. For search queries? because druid just lookups index and dictionary, it just not have meaning of order of processing.
I think time-series and search queries can make differences by this. But for others, I don't know.

Ok, timeSpec is a horrible idea ;).
I kinda like the combine idea, but @navis 's comments about the ordering affecting the processing order rather than the response order has also got me thinking.  As I think about it, I think I've convinced myself that even though it is affecting the processing order and not just the result order, I think that's an implementation detail/optimization...
I wonder if maybe we should look at creating a query with the chunks as Vadim has them laid out "split/apply/combine/filter".  We could rewrite that into whatever queries we have right now as an initial implementation and then eventually implement it to actually run against segments too?
If we were to take this approach, I think that the way you are doing it now would still make sense for the long-term ('cause eventually maybe those queries would be hidden behind the split/apply/combine/filter query?).  What do you guys think?

@cheddar +1. The way I always think about Druid's current query API is that it is the low level API and over time we should migrate a higher level API that is much easier to reason about and extend. However, what do you want to do about the changes required in this current PR?

If we want to take the approach of trying to do a split/apply/combine/filter query to replace them "all", then I think having it at the query level like it is can make sense.  So let's just leave it there and maybe try to get @vogievetsky to propose how he would prefer to specify his split/apply/combine/filter queries?

Yes, descending-processing would be a part of execution environment for S/A/C/F, not to be accessed by user directly. But can we include this just for time-series queries for now? There are some requests for it.

This generally makes sense to me.  I think that the way you went about adding the descending property to all of the function calls is actually not necessary, but I won't know without messing around with the code a bit.  I'm gonna try forking your branch and editing it up some to see if I'm smoking crack or not.

Ok, yeah, I was right in that there was a simpler way to do things.  It required a bit of butchering of interfaces.
Tthere were methods on QueryToolChest that shouldn't've been there.  Those methods were breaking the abstraction and needed to be eliminated in order to make the simplified changes.  It probably wasn't readily apparent that the problem was the bad methods on the interface, but once they are cleaned up, the code cleans up quite a bit.  I did a PR against your PR branch, you can see the changes here:
https://github.com/navis/druid/pull/1
Let me know what you think.

Merged @cheddar's patch and rebased on master. Let's see the test results.

There's still two comments I'd like to see addressed, but once they are in I'll be :+1:

@navis I have a comment to please update the documentation so people can know how to get results in reverse order

@cheddar @navis does this only work for timeseries? how hard is it to get it working for other query types?

should the cacheKeys returned by CacheStrategy.computeCacheKey also include the order of results in which they are stored in cache ?

@nishantmonu51 that's a very good point

I am :+1: on this. @himanshug ? I think we should file issues to support other types of queries in reverse order

I'm :+1:

:+1: beside a nit and https://github.com/druid-io/druid/pull/2014#discussion_r48860921
you might want to cleanup the commit history.

I can only see the result order being part of cacheKey for only TimeSeriesQuery,
I believe this needs to be added for all other queries as well since the descending flag is part of BaseQuery. Also we seem to be missing unit tests for caching of reverse queries.

@nishantmonu51 added descending to all cache keys. And there is a tiny test case in CachingQueryRunnerTest#testTimeseries() for descending timeseries query cache.

Updated patch. I'll squash commit log when there is no more comments.

@nishantmonu51 do you have any more comments?

@navis there's some merge conflicts now, hopefully they are small

@navis any chance of resolving merge conflicts and finishing this one up?

@fjy fixed conflict and addressed comments.

@navis latest changes looks fine to me, can we squash commits before merging. @cheddar there have been quite a few changes since your last thumbs up, can you have a second look?

I'll squash commits when @cheddar approves.

I have verified that all of navis's changes since I looked last are good on the basic functionality.  I did not verify that the caching looks good, but I'm happy with everyone else's eyes on that.  so I'm :+1:

Rebased on trunk & squshed.

@fjy my comment here was not addressed https://github.com/druid-io/druid/pull/2014/files#r49129298
Can we please make sure to review all comments before hitting merge?

@xvrl Totally my bad, sorry. I've added patch for it in #2326.


Bring "cancel" up to spec

This adds "killnowait", and some backwards-compatibility for "abort" -- a different PR shortly with no backwards-compatibiltiy code (take your pick :).

Codecov Report

Merging #1047 into master will decrease coverage by <.01%.
The diff coverage is 50%.

@@            Coverage Diff             @@
##           master    #1047      +/-   ##
==========================================
- Coverage   42.35%   42.35%   -0.01%
==========================================
  Files          69       69
  Lines       12825    12830       +5
  Branches     2078     2079       +1
==========================================
+ Hits         5432     5434       +2
- Misses       7002     7005       +3
  Partials      391      391

Impacted Files
Coverage Δ

autobahn/wamp/protocol.py
12.91% <0%> (-0.02%)
:arrow_down:

autobahn/wamp/message.py
58.84% <60%> (-0.03%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 57fd716...624e471.  I merged the wrong one.

too subtle for me .. 2 PRs to pick from.


Remove "Begone, Evil! (Disable Corruption Spread)", duplicated function in Quality Of Terraria

Quality Of Terraria has the same functionality and as such this mod is redundant

Thank you, I'll directly commit the changes.


Simple way to change page title

First, very nice project! Thanks for sharing.
Have been trying to find simple way to add form field, which will allow changing of the default encrypted page title.
Currently set to <title>Password Protected Page</title> and like to have form field entry change it when encryption is processed.
Any tips to accomplish will be much appreciated.
Susan

Thanks anyway, found a better solution for the teachers to use when protecting their HTML quiz's.


Load markdown text from a file.

I'm not very php oriented but I believe this is not that wrong. I used this to parse a bunch of markdown files into an Html page.

I think this is maybe best done by the user :)
You can always write your own function to instantiate, configure, and call Parsedown's methods on your behalf if you find yourself repeating the same lines of code a lot.


Drop support for advanced proxy

Adding the advanced proxy was a mistake, and most of the time it isn't "advanced" enough for people using it.
There needs to be a new solution.

The advanced configuration is very simple if you're using it like the old advanced proxy. Please see the migration guide, it should be a copy and paste.

For anyone looking for the migration guide, it's here

the generated README.md should be updated


[SUPPORT] Can't query Redshift rows even after downgrade from 0.10

Describe the problem you faced
After the upgrade to Hudi 0.10, I faced the https://github.com/apache/hudi/issues/4283 issue in my environment, so my AWS Glue tables were working fine on AWS Athena, but were returning 0 rows with Redshift Spectrum.
After that, I downgraded the tables to version 2 using Hudi-CLI, and returned to Hudi 0.9. I immediately tried to query the tables again after the downgrade, but i kept the same behavior, no errors and no rows returned.
But, after doing a new load on this table, some rows appeared on the Redshift Spectrum query, not all of them, basically, i'm seeing a different number of rows on Redshift Spectrum (way less rows) in comparison with Athena (the complete set of rows).
The strange thing is:
The table i'm facing this error have lots of upserts on lots of partitions (> than 1000 partitions) per batch, usually the updated partitions are days on the last 2 months, but there is some days on other dates aswell.
The most recent partitions, are with the number of records ok, and big difference occurs on the old ones.
Does Hudi only "fix" the rows of the Redshift Spectrum incompatibility when there is a new load on a specific partition?
There is any quick workaround to fix it?
To Reproduce
Steps to reproduce the behavior:

Upgrade AWS Glue table to Hudi 0.10
Load the table using Hudi 0.10
Downgrade AWS Glue table to Hudi 0.9 with Hudi-CLI
Load the table using Hudi 0.9
The row count from Athena to Redshift Spectrum does not match anymore.

Expected behavior
All the rows should be visible on redshift spectrum after the downgrade.
Environment Description

Hudi version :
0.10/0.9
Spark version :
3.1.1
Hive version :
2.3.7-amzn-4
Hadoop version :
3.2.1
Storage (HDFS/S3/GCS..) :
S3
Running on Docker? (yes/no) :
no

@nochimow this is likely caused by the same issue in https://issues.apache.org/jira/browse/HUDI-3056 where the timeline's time precision is not handled properly. But when you write data with 0.9.0, the old time precision was used then partial data is shown in redshift. While we can't troubleshoot any aws services as we're open-source, I'm adding this issue to the ticket to push up the priority.

@xushiyan I opened a ticket to AWS support about this too to try to get more priority too.
Can you confirm what is the Hudi behaviour in this case? If i write new data into a partition, all this partition will be visible to Redshift again or just the updates/inserted rows?

@nochimow I can confirm due to time precision change in 0.10.0 there needs to be some fix for redshift. I can't confirm the exact behavior, since the behavior is under the influence of the bug reported in that JIRA, also this is in redshift which i can't verify with. Hope you understand. We just need to prioritize the fix and follow up aws support. Please close this if you don't have further questions.

Probably you might have to do a restore to a older commit. Can you give it a try.
add a savepoint to a commit which was created w/ 0.9.0. and then trigger a restore to that savepointed commit.
it should restore your entire table to a older snapshot. (its destructive operation though. something to keep in mind)
essentially restore will delete all data files and timeline files from now until the savepoint to which you are looking to restore.

@nochimow : let us know how did it go. or if you need hudi-cli commands to do restore, let us know.

Hello,
I did a workaround to fix this issue that i didnt need to restore my table.
Apparently, after the hudi table downgrade to 0.9 when I do a Hudi upsert operation into a partition, the rows of the partition become visible for the Redshift Spectrum again.
I'm still trying to correct all the rows all my partitions. But 95% of my rows are already normal again, maybe the remaining 5% is something i'm still missing on my side that i'm currently checking.

So for now the status is that using 0.10.x of Hudi is incompatible with redshift spectrum? We are exploring using Hudi for our data lake and came across this very same problem which confused me a lot. I'll do the downgrade now.

@nsivabalan Considering that we seem to be at AWS mercy here, could there be an option to have some config flag which uses the old precision? Or is there some new features relying on this change to be present?
We have this chicken and egg problem with adopting dbt now. We want to use redshift spectrum for queries and dbt spark for transforms. (Which was added in 0.10) But that renders spectrum unusable.
Best regards, Jørgen

yes, makes sense. I will see where we can document this.

@xushiyan Is it possible to try to escalate this issue with AWS again? There is still no return from their side?

@umehrot2 any updates on HUDI-3056 ?

yes, makes sense. I will see where we can document this.

Can we add it here https://hudi.apache.org/releases/release-0.10.0#writer-side-improvements ?

Hello Guys, any updates on this? :)

@codope @nsivabalan do you have updates on this subject? We stuck on Hudi 0.9 because of this :(

Same here. Still waiting for an update to upgrade our Hudi from 0.9

@nochimow Thought you had opened an AWS Support ticket.  If yes, any update from AWS Support? Curious to know if Amazon Redshift Spectrum team is planning to do something on this issue.

@pomaster Yes i did, Their reply was that the product team is aware of this issue, but there is no ETA to fix this yet.

@nochimow Thanks for the info!  When was the last update from AWS Support on your ticket?  We are planning to reach out to AWS Redshift engineering team for help.  Don't want to waste their time if there has already been a recent status update (e.g. < 1 month).  Can you share your AWS Support ticket #?  You can e-mail me at<EMAIL_ADDRESS>

@nochimow @rubenssoto and others: Looks like hudi 0.10.0 is supported from the docs https://docs.aws.amazon.com/redshift/latest/dg/c-spectrum-external-tables.html#c-spectrum-column-mapping-hudi
Can you folks give it a try and let us know if things are working well w/ hudi 0.10.0.

Even with AWS saying that only 0.10.0 is "supported", I did some compatibility tests with Hudi 0.10, 0.11 and 0.12.
All versions worked fine, like it wasn't before. (Prior to that, any table with Hudi version > 0.9 was returning 0 rows on Redshift Spectrum)
The only detail here is that the Redshift version must be with the patch >=169. (Got this requirement from the AWS support)

thanks @nochimow for the update. appreciate it.

Great to know, I will test, thank you so much guys!


z-index with IR headers

The IR headers are showing above the daterange picker:

The z-index of the IR is set to 300, so you could fix it by updating the CSS to:
.daterangepicker {
  position: absolute;
  color: #393939;
  background: #fff;
  border-radius: 2px;
  display: none;
  /*width: 278px;*/
  padding: 8px;
  margin-top: 1px;
  top: -100%;
  left: -100%;
  z-index: 400;  /*add this line*/
}

Hi,
i use "Left Side Column" as site template and the daterange pickers are in the "Left Column". Here the z-index of 400 is still not enough as the popup is displayed behind the left column:

I changed it to 500 and this worked:

Thank you for your contribution @maxime-tremblay and @ma-schroeder. I will be releasing bug fixes this week.

z-index 700 is default since 1.1.0


Fixed styling for the dynatree nodes in the Entry point popups

<fieldset> was removed when we added Patternfly buttons to the form, therefore the dynatree CSS that was using fieldset had to be fixed.
#1292

@epwinchell @dclarizio Please review.

@AparnaKarve
As discussed, can we make the "highlight" of the selected node more visible.  Making the text bold does not really show up that well, especially if there is only 1 node with no siblings next to it to be able to tell the difference.
In addition, clicking on nodes that are not eligible to be applied should not show them as selected.  Hopefully that can be done with the existing dynatree control.

@dclarizio Please test the new changes.
The node highlight should be prominent now. Thanks @epwinchell .
Also added functionality to highlight only the eligible nodes.

@AparnaKarve Seems to all be working great.  Not sure I like the highlight, it's almost TOO prominent having the entire row containing the tree node being colored:

Can we do something more like the "Sub-item 4.1" node in the tree on this page? https://code.google.com/p/dynatree/ Where only the text of the tree node is highlighted instead of the entire row.
Thx, Dan

@dclarizio Made suggested change. Please verify.

Perfect! :+1:


License

Hello, I would like to test this model but I notice there is no license attached. Please could you add one to clarify what usage is allowed?

Hi, it's MIT license and I just uploaded it to the repo.


typo: intellisense -> IntelliSense

Removed BOM as well

@nschonni : Thanks for your contribution! The author, @mumian, has been notified to review your proposed change.


fix(docs): bump release level to GA

@soldair before landing the GA label on Bunyan and Winston, I thought it would be good for us to address the race condition you brought up last week that results in an uncaught rejection.

Trying not to lose track of this one.  Do we have an issue tracking the aforementioned race condition?  @bcoe @soldair

@JustinBeckwith https://github.com/googleapis/gcp-metadata/pull/273 I believe I patched the bug @soldair reported in gcp-metadata last week, and it will go out early this week.


BugFixes

Mantis#909793 - SFSP - "Escalate To Incident" playbook should correlate MITRE modules correlations of alerts to incidents

Validated that the MITRE modules such as Software, Technique, Sub-Technique and Mitigation have been correlated to the incident from the escalated alert

Mantis#0911534 -> SFSP - Rename "08 Utilities > Fetch and Link Team to Related Records" Playbook

Validated that the 08 - Utilities > Fetch and Link Team to Related Records is renamed to Cascade Permissions to all Related Records

Reopen PR


JAVA-933: Consider allowing users to override default codecs.

Overview of changes:

The overriding of a single codec takes place in CodecRegistry.register() : if the exact same CQL-Java mapping already exists, the existing codec is replaced by the new one (note the additional log warning).
The on the fly  overriding of collection codecs is more complex: there is now a new CodecFactory class that can be subclassed for this purpose.
Behavior in case of overlapping codecs didn't change: it is still possible to register more than one codec for the same CQL type.

Note: the CodecFactory class allow users to override the on-the-fly instantiation mechanism, but we could live without it. If the user knows which exact list types he needs to handle, the following code:
CodecFactory factory = new DefaultCodecFactory() {
    protected <T> TypeCodec<List<T>> newListCodec(TypeCodec<T> eltCodec) {
        return new MyListCodec<T>(eltCodec);
    }
};
CodecRegistry registry = new CodecRegistry(factory);

Could easily be replaced with:
CodecRegistry registry = new CodecRegistry();
registry.register(new MyListCodec(IntCodec.instance));
registry.register(new MyListCodec(VarcharCodec.instance));
// etc. register all list types required by the application

Closing as the corresponding ticket has been closed as Won't Fix.


Instructions for generating and using a Maestro key

After both Atlas and Maestro release, let's add instructions for how to generate and use a Maestro key with Atlas.

Closing as it's already accounted for. Please feel free to reopen this issue in case I am missing something.


Lmb 686/banner for environment

Description
We want to show a banner when we are in a different environment.
How to test
Forcefully change the enviroment name to DEV, QA, PROD, LOCAL
Links to other PR's

https://github.com/lblod/app-lokaal-mandatenbeheer/pull/215

Attachments

You'll have to fix the pipeline, because it can't do the npm install --legacy-peer-deps


Black screen on Ubuntu 24.10 arm64

Describe the bug
Freelens does not render anything but black window
To Reproduce
Start Freelens.
Expected behavior
Normal screen.
Environment (please complete the following information):

Freelens Version: 0.1.0-nightly-2024-10-20
OS: Ubuntu 24.10 arm64
Installation method: DEB

Additional context
It is fine when app is started with option --ozone-platform=wayland.
https://www.reddit.com/r/swaywm/comments/191zdgs/support_both_wayland_and_x11_when_using/ gives some hint how to handle it more universally.

It looks like the problem is only when GL device is used.


User cannot log in to venmo account via my app

So I have successfully installed the venmo sdk on my app and it works fine when I use the Sample app venmo id, but when i use my own venmo app id, the app either crashes or doesn't log the user to his/her venmo account. It keeps coming back to the loggedoutviewcontroller after the "Authorize app" page comes up. Please can you help me out with this. Secondly how can I add like a service charge on every transaction via my app? By the way, is it possible that a wrong web re deirect url can cause this issue

Can you give us a little more info on the crash? A stack trace would be great.
You cannot charge a service charge directly through the Venmo app.

Sorry for the late response. I’m going to close the issue because it’s old, but please feel free to reopen it if it is still an issue. Thanks!


[FEATURE] Allow CNN + MLP torsos before the action heads

Currently, several CNN-based torsos: e.g. CNN and VisualResnet, do not allow for an MLP layer between the outputs of the convolutional features and the action head (either CategoricalHead or DiscreteQNetworkHead). I think this is important for reproducing the architectures that are widely used where there is 1) A convolutional feature extractor; 2) an MLP projector; and finally 3) the action head.
It would be great to be able to combine torsos in the config files (I am not 100% sure if this is doable already?).
An alternative solution would be to modify the action heads (e.g. CategoricalHead, DiscreteQNetworkHead) to take layer_sizes as optional inputs to pre-append them before the output layers.
Edit: I realized VisualResNet actually has this, so it's only missing for the standard CNN

Hey, you are 100% right and this is a feature that I've been meaning to do actually (I just forgot about it ... ) It's quite a small change so I'll try get around doing it sometime Sunday. Thanks for reminding me :)

So basically, I've been thinking about it and i actually want to maybe do a slightly big change on how networks are constructed - it would be nice to easily be able to chain together any and all torsos but I'm not exactly sure on the best way to do it. I have no idea though when i will get around to this. For now, I've simply added the MLP torso to the end of the CNN torso and put in the config a hidden_sizes argument like that of the visual resnet. I hope this is okay for now. If you want to have a more complicated torso, i'd recommend just simply copying and pasting the most relevant one and editing it there and making a new config for it. I'll merge the PR when it passes the tests.


Separate ByteCode from wasm opcode and define own opcode for ByteCode…

… design

The main update of this patch is

clearly divide wasm opcode and bytecode because our bytecode is not mapped to wasm-instruction one by one
split chunk operations such as MemoryLoad, MemoryStore, BinaryOperation, UnaryOperation into each individual operation

@zherczeg
This patch fixed a lot code part of Walrus, so please review this and share your view.
I hope this patch helps you design/implement JITC
As a following patch, I'm also considering removing all vtable in ByteCode structures because vtable takes additional space but its not necessary I think.


GraphQL helpers

The same as with Redux, do you have any workaround about Web Components + GraphQL?
Maybe with something like apollo-client.
Thanks in advance!

@abdonrd you can use apollo-client from web components and with lit-element.
Create an apollo client
import { ApolloClient } from 'apollo-client';
import { HttpLink } from 'apollo-link-http';
import { InMemoryCache } from 'apollo-cache-inmemory';
import { ApolloLink, concat } from 'apollo-link';

const httpLink = new HttpLink({ uri: '/graphql' });
const cache = new InMemoryCache();
...
...
const link = authFlowLink.concat(httpLink);

export const client = new ApolloClient({
  link: link,
  cache
});

Import it where you need it and do query
import { client } from "./apollo-client.js";

...
constructor() {
    this.actionsQuery = client.watchQuery({
      query: gql`
        {
          actions {
            date
            msg
          }
        }
      `
    });
  }
...
  userLoggedIn({ user }) {
    this.actionsQuery.subscribe({
      next: ({ data }) => {
        this.actions = data.actions;
      }
    });
  }
  userLoggedOut() {
    if (this.actionsQuery) this.actionsQuery.unsubscribe();
  }

There's this also: https://github.com/bennypowers/lit-apollo

@eskan your example doesn't work because ES Modules.

Uncaught (in promise) SyntaxError: The requested module '../../../../fast-json-stable-stringify/index.js' does not provide an export named 'default'

Another reference: https://github.com/apollographql/apollo-client/issues/3571

@abdonrd this sample is a bit outdated with latest Apollo-client but it works like a charm. The error you've mentioned seems not related to apollo ?

@eskan That error occurs just with:
import ApolloClient from 'apollo-boost';
const client = new ApolloClient();

it should be related to the way you serve/package your application.
How do you serve files ?

With npm start (polymer serve), the project is built with the PWA Starter Kit.

Reproduction: https://jsfiddle.net/va2uxesr/

Here i write a sample that can help you : https://stackblitz.com/edit/lit-demo-base-ts-graphql
I know it's not an easy way but you shoud think to switch to webpack instead
npm i -D webpack@latest webpack-dev-server@latest webpack-cli@latest
a simple webpack.config.js
const path = require("path");

module.exports = {
  entry: {
    app: "./src/components/my-app.js"
  },
  output: {
    filename: "[name].js",
    path: path.resolve(__dirname, "build"),
    publicPath: "build"
  }
};

and change the import in your index.html to
<script type="module" src="build/app.js" crossorigin></script>

@abdonrd  Did you manage to resolve the issue?

@hashamhabib not in a easy way for the PWA Starter Kit.😕

@hashamhabib is not really an issue related to pwa-helpers or lit-element. You're able to use many  other libraries to handle data : locally/remotely
Use apollo in a lit-element is not a substitute to redux. To replace redux in an apollo way, you need to use apollo-link-state https://www.apollographql.com/docs/link/links/state.html
it allows you to query local data from cache with the @client
query getState {
    currentPage @client
    currentUser @client {
      dn
  }
}

IMHO Apollo-client doesn't need helpers. Uses of Vanilla client is a perfect way to deal with apollo.


(maint) Return 400 for incomplete URL to certificate_statuses

Previously, we were falling through to the default handler of our
web-routes if the certificate_statuses endpoint did not include its
required -- but ignored -- trailing key segment.  The default handler
returns a 404.
This provides a default handler for the certificate_statuses endpoint
that will catch malformed URLs and return a 400 instead of a 404.
It also extracts providing a generic 400 response into a method that
is reused where appropriate.

Per suggestion in https://github.com/puppetlabs/puppet-server/pull/980#discussion_r56898773

:see_no_evil:

:see_no_evil: just one last comment on adding an assertion to the test

Awesome, updated according to comments!


I get errror information when I applying realsense-metadata patch.

Required Info

Camera Model
D435i

Firmware Version

Operating System & Version
ubuntu18.04

Kernel Version (Linux Only)
4.9.140

Platform
NVIDIA Jetson TX2

SDK Version
v2.22

Language

Segment

Issue Description
In order to use this guide on  TX2(Jetpack4.2+kernel4.9.140+librealsense_v2.22),I have changed some code.Everything is ok,but applying realsense metadata patch.There are some error I get.

How can I fix it?I never change anything under patches folder.

I have fixed it successfully!


fix: 🐛 Undo Redo broken with CapsLock

Summary
Undo Redo broken with Caps Lock
How did you test this change?
By manually.

This breaks the behavior of Mod-Shift-z to redo a change.

Fixed.

Seems still not working on macOS.

I think Mod-Shift-z may conflict with Mod-Z.

Yes, I think it's an issue of prosemirror-keymap.


Introducing Modal component

This introduces a new Modal component to anchor-ui. It is a simplified version of the Modal components found in Wharf/Buoy, preventing users from having to compose multiple pieces of the modal in order to create the modal in its entirety.
Screenshot

Converting to draft - need to expand the functionality by including a LegacyModal for a more plug-and-play migration path.


Wrong header information displayed with snap_maint -vv when Cloud build option is enabled

2 lines read from Registers are wrong.

FPGA Release v1.5.0 is correct
Distance 0 is correct
GIT 0xee882a6e while ee882a6 is expected (not wrong but taking 1 digit more than standard short release number)
SNAP FPGA Build date and time are outdated (expected 2018/11/08)

This happens almost for image built on Nimbix using cloud build option using KU3 card. Problem not seen when building an image on Nimbix w/o the cloud build option.

image executed on cloud or on-premises show the same erroneous information.
synthesis created w/ and w/o cloud option shows data ok in snap_core_types.vhd

Seems that the snap_build process doesn't take data from snap_core_types.vhd file

The date contained in the binary code when using cloud_image option is the date of the generation of the static block containing PSL+ MMIO registers. 2 options:

no change

find a way to know that image was built in cloud and adapt the text to display

Displaying 7 digits instead of 8 as per GIT short number is modified in branch Issue_845

no change
7 digits corrected in branch improve_branch


Module not found: Can't resolve '@material-ui/core/Stack'

[ ] The issue is present in the latest release.
[ ] I have searched the issues of this repository and believe that this is not a duplicate.

  Output from `npx @material-ui/envinfo` goes here.

Hi @Nondukishor, could you provide us with more information about the error please? this happens in v5?

Closing due to lack of activity.


Support multiple comments with a title identifier

I keep looking at the build logs and can't figure out why the build is failing. I don't see any errors. Any ideas, @rickardl?


idempotent indexing for a time averaged field gives different results [when subtracting]

Reported by Remik Ziemlinski on 9 Feb 2005 18:29 UTC
The results of differencing are different when a time averaged field is
explicitly indexed.  Sample script below:

NOAA/PMEL TMAP
FERRET v5.70 (gcc3.2)
Linux(g77) 2.4.18 - 10/08/04
9-Feb-05 13:25
use "in.nc"
let ave  = air[dif  = air[d=1](d=1,t=@ave]
let) - ave
! incorrect?
let dif2 = air[- avel=1
set window 1
plot dif[difx=85w,y=45n
set window 2
plot dif2[dif2x=85w,y=45n
Migrated-From: http://dunkel.pmel.noaa.gov/trac/ferret/ticket/1170

Comment by<EMAIL_ADDRESS>on 9 Feb 2005 22:56 UTC
I don't think this is a bug -- but it is a common gotcha
with the delayed-evaluation philosophy of Ferret.  The
"t=@ave" gets applied to the current context, which for
ave[is just the first time level.  So while dif
represents the deviation from the full time average, dif2
just represents the deviation from the first time level.
You can prevent this "fuzzy math" by being explicit about
the time averaging interval.  So if you have 1200 time levels,
do:
let ave = aird=1,l=1:1200@ave

Attachment from<EMAIL_ADDRESS>on 9 Feb 2005 18:32 UTC
plot dif
REPLACE THIS TEXT WITH UPLOADED FILE ./attachments/TRAC_1170_GIT_534/dif.gif

Attachment from<EMAIL_ADDRESS>on 9 Feb 2005 18:33 UTC
plot dif2
REPLACE THIS TEXT WITH UPLOADED FILE ./attachments/TRAC_1170_GIT_534/dif2.gif

Attachment from<EMAIL_ADDRESS>on 9 Feb 2005 18:33 UTC
sample data for test script
REPLACE THIS TEXT WITH UPLOADED FILE ./attachments/TRAC_1170_GIT_534/in.nc


xgboost Distributed Calculates Wrong Gradients/Hessians

I've tried using a  simple example for distributed XGBoost and I'm getting wrong gradients
I'm using the 1.4 release branch of xgboost
In short, I'm using 100 samples from Iris dataset and try to train distributed xgb model. Code is below:
import argparse
import logging
import xgboost as xgb
import traceback
import pandas as pd
from sklearn import datasets
import socket

HOST = socket.gethostname()
HOSTNAMES = ['opuscydvux0221.optiver.us', 'opuscydvux1220.optiver.us']
assert HOST in HOSTNAMES

logger = logging.getLogger(__name__)

pd.set_option('display.max_rows', 500)
pd.set_option('display.max_columns', 500)
pd.set_option('display.width', 10000)
pd.set_option('max_colwidth', 100)
pd.set_option('precision', 5)
pd.set_option('display.max_rows', 4)

def read_train_data(rank, num_workers, partition):
    """
    Read file based on the rank of worker.
    We use the sklearn.iris data for demonstration
    You can extend this to read distributed data source like HDFS, HIVE etc
    :param rank: the id of each worker
    :param num_workers: total number of workers in this cluster
    :param path: the input file name or the place to read the data
    :return: XGBoost Dmatrix
    """
    iris = datasets.load_iris()
    iris_df = pd.DataFrame(iris.data, columns=iris.feature_names)
    if num_workers > 1:
        partition = rank
    assert partition < 2

    separate = 100
    train_df = iris_df.iloc[:separate]
    test_df = iris_df.iloc[separate:]

    if partition < 0:
        print("Using full data")
    else:
        separate_train = 60
        separate_test = 30
        if partition == 0:
            train_df = train_df.iloc[:separate_train]
            test_df = test_df.iloc[:separate_test]
        else:
            train_df = train_df.iloc[separate_train:]
            test_df = test_df.iloc[separate_test:]

    def get_dmatrix(df):
        x = df[df.columns[:-1]]
        y = df[df.columns[-1]]
        print(x)
        return xgb.DMatrix(data=x, label=y)

    dtrain = get_dmatrix(train_df)
    dtest = get_dmatrix(test_df)
    return dtrain, dtest

def train(args):
    """
    :param args: configuration for train job
    :return: XGBoost model
    """

    world_size = args.world_size
    assert world_size <= len(HOSTNAMES)
    port = 9095

    rank = 0 if world_size == 1 else HOSTNAMES.index(HOST)
    # addr = args.hostnames[0]
    addr = '<IP_ADDRESS>'

    print("Rank=", rank, addr)

    rabit_tracker = None

    try:
        """start to build the network"""
        if world_size > 1:
            if rank == 0:
                logger.info("start the master node")

                # rabit = RabitTracker(hostIP="<IP_ADDRESS>", nslave=world_size,
                #                     port=port, port_end=port + 1)
                rabit = xgb.tracker.RabitTracker(hostIP=addr, nslave=world_size,
                                                 port=port, port_end=port + 1)
                rabit.start(world_size)
                rabit_tracker = rabit
                logger.info('###### RabitTracker Setup Finished ######')
            envs = [
                'DMLC_NUM_WORKER=%d' % world_size,
                'DMLC_TRACKER_URI=%s' % addr,
                'DMLC_TRACKER_PORT=%d' % port,
                'DMLC_TASK_ID=%d' % rank
            ]

            logger.info('##### Rabit rank setup with below envs #####')
            for i, env in enumerate(envs):
                logger.info(env)
                envs[i] = str.encode(env)

            xgb.rabit.init(envs)
            logger.info('##### Rabit rank = %d' % xgb.rabit.get_rank())
            rank = xgb.rabit.get_rank()
            print("RR", rank)
        else:
            world_size = 1
            logging.info("Start the train in a single node")

        dtrain, dtest = read_train_data(rank=rank, num_workers=world_size, partition=args.partition)
        params = {'max_depth': 2,
                  'eta': 1,
                  'silent': 1,
                  "tree_method": "hist",
                  "objective": "reg:linear",
                  }
        logging.info("starting to train xgboost at node with rank %d", rank)
        evals_result = {}
        bst = xgb.train(
            dtrain=dtrain,
            num_boost_round=int(args.n_estimators),
            params=params,
            # evals=[(dtrain, 'train'), (dtest, 'test')],
            # evals_result=evals_result
        )
        print(evals_result)

        model = bst

        logging.info("finish xgboost training at node with rank %d", rank)

    except Exception as e:
        logger.error("something wrong happen: %s", traceback.format_exc())
        raise e
    finally:
        logger.info("xgboost training job finished!")
        if world_size > 1:
            logging.info('finalizing rabit')
            xgb.rabit.finalize()
        if rabit_tracker:
            rabit_tracker.join()
            logging.info('rabit_tracker join')

    logging.info("Out of here")
    return model

def main(args):
    logging.info("starting the train job")
    model = train(args)

    if model is not None:
        dump = model.get_dump(with_stats=True)
        for i in dump:
            print(i)

if __name__ == '__main__':
    parser = argparse.ArgumentParser()

    parser.add_argument(
        '--n_estimators',
        help='Number of trees in the model',
        type=int,
        default=1
    )

    parser.add_argument(
        '--world_size',
        help='World Size',
        default=1,
        type=int
    )

    parser.add_argument(
        '--partition',
        help='Partition',
        default=-1,
        type=int,
    )

    logging.basicConfig(format='%(message)s')
    logging.getLogger().setLevel(logging.INFO)
    main_args = parser.parse_args()
    main(main_args)

When I run non-distributed version
python simple_distributed_xgb.py --world_size 1
I get this tree:
0:[petal length (cm)<3] yes=1,no=2,missing=2,gain=28.5088348,cover=100
        1:[petal length (cm)<1.70000005] yes=3,no=4,missing=4,gain=0.0223081112,cover=50
                3:leaf=-0.25999999,cover=44
                4:leaf=-0.142857149,cover=6
        2:[petal length (cm)<4.19999981] yes=5,no=6,missing=6,gain=0.343410492,cover=50
                5:leaf=0.620000005,cover=19
                6:leaf=0.903124988,cover=31

Seems correct since I'm using 100 examples.
However, then I run
python simple_distributed_xgb.py --world_size 2
On the two machines that I use and I get this tree:
0:[petal length (cm)<3] yes=1,no=2,missing=2,gain=19.2424183,cover=200
        1:[petal length (cm)<1.70000005] yes=3,no=4,missing=4,gain=0.0223081112,cover=50
                3:leaf=-0.25999999,cover=44
                4:leaf=-0.142857149,cover=6
        2:[petal length (cm)<3.5999999] yes=5,no=6,missing=5,gain=9.38419342,cover=150
                5:leaf=0.294339627,cover=105
                6:leaf=0.841304362,cover=45

Notice cover=200. Gain is also wrong initially(and not halfed. so gradients also wrong or am I missing a regularization?),  which causes wrong leaf values on the right side of the tree.
I'd appreciate your help figuring out what am I doing wrong

It appears that the following line should be removed from updater_quantile_hist.cc:
histred_.Allreduce(&grad_stat, 1);
I checked that grad_stat is already correct prior to this line. Most likely because histograms are already syncronized by this point. So, this allreduce just doubles stats (since I used two nodes) incorrectly.
If I remove that line, distributed works as expected producing the same tree as non-distributed:
0:[petal length (cm)<3] yes=1,no=2,missing=2,gain=28.5088348,cover=100
        1:[petal length (cm)<1.70000005] yes=3,no=4,missing=4,gain=0.0223081112,cover=50
                3:leaf=-0.25999999,cover=44
                4:leaf=-0.142857149,cover=6
        2:[petal length (cm)<4.19999981] yes=5,no=6,missing=6,gain=0.343410492,cover=50
                5:leaf=0.620000005,cover=19
                6:leaf=0.903124988,cover=31

Thanks for raising the issue, I will look into it and add some tests tomorrow.

@trivialfis Did you have a chance to look at it?

Yes, trying to clear out the to-do items. ;-)

@Denisevi4 I think we can't avoid this synchronization histred_.Allreduce(&grad_stat, 1);
As grad_stat is a local variable: https://github.com/dmlc/xgboost/blob/master/src/tree/updater_quantile_hist.cc#L316
Each worker calculate the statistic on local data subset and it should be decently  synchronized.
As for difference in batch and distributed trained model, seems the most possible reason is difference in quantile matrix construction. The is no expectation that quantization process is similar for distributed and batch case (dependency on number of workers).

@ShvetsKS Hessian equals to 100 prior to this line on both nodes. However, one node has 60 samples locally, the other has 40 samples. So, they both must have gotten stats from the other node already. I presume this happens during histogram building when all histograms are synchronized and stats are summed.

@Denisevi4  for current else branch https://github.com/dmlc/xgboost/blob/master/src/tree/updater_quantile_hist.cc#L327 we definitely have to keep this synchronization as it accumulates the sum of local gpair_h vector.

But you understand what I'm saying right? Hess=100 means that each node counts that it already used 100 events in the stats. Which is a full training dataset. Just making sure.

@Denisevi4 sorry for not accurate look at Dense branch. As a local grad_stat is accumulated from hist what was synchronized in hist_synchronizer_->SyncHistograms call already. Seems yes, grad_stats.GetHess() after synchronization histred_.Allreduce(&grad_stat, 1); is just multiplied by number of local workers. And it doesn't look like expected behavior. Thanks!

@ShvetsKS Would you like to create a PR?  Would be great to have a test in dask.

@ShvetsKS  I dunno about https://github.com/dmlc/xgboost/blob/master/src/tree/updater_quantile_hist.cc#L327 example that you posted.
Because in my example it went to this if condition
if (data_layout_ == DataLayout::kDenseDataZeroBased ||
        data_layout_ == DataLayout::kDenseDataOneBased)

But maybe we need to keep Allreduce inside that else condition like you mentioned?

@trivialfis sure https://github.com/dmlc/xgboost/pull/7120

@Denisevi4 yes, that's exactly what I wanted to do (https://github.com/dmlc/xgboost/pull/7120)

@ShvetsKS Nice. Maybe you can also add a couple of distributed unittests for both Dense and not dense cases to make sure this stays correct in future.

@ShvetsKS Nice. Maybe you can also add a couple of distributed unittests for both Dense and not dense cases to make sure this stays correct in future.

Yes. It's in progress. Just waiting for finishing current tests.

@Denisevi4 , @trivialfis unfortunately the problem with different workers count is still exist even after applied fix. If we use larger than iris datasets (make_regression(1000000,100) as an example) and test it with different num_workers, models will be different. And seems the reason is in the quantization process (different bin matrices produce different models).
Now we can only guaranty that cover value on root will be the same regardless of the number of workers.

That's fine.  Different number of workers will generate different models for training, it's a known limitation in the algorithm.

Closed by https://github.com/dmlc/xgboost/pull/7120 .


merge bitcoin#17192, 16240: rpc backports

uhh lol, that's a lot bigger of a PR now 😂

Is there a reason why 16240 and 19386 (should be marked partial btw) are incomplete?

16240 was marked as partial as b6fb617aaaad5f9cdd7f2ad2825b253ca792055d was excluded as this PR favours logical backports as it is a dependency for two other PRs and a complete backport would invite a lot of potential merge conflicts (19250 needed me to resolve conflicts multiple times, for instance)
The same applies for 19386's fa7592bfa8691eb0289b21da3571709a18391b0f.


Cannot perform flutter run against Android Emulator with API Level 29

It seems Android API Level 29 AVDs do not work with flutter run.  Downgrading to API Level 28 works as intended.
I learned about the workaround from @dustinrwh's comment in #19056.
Steps to Reproduce

Create AVD with API Level 29 (I chose Nexus 5X)
Boot AVD and wait for it to load
Run flutter run and note the error below

Logs
$ flutter run
Using hardware rendering with device Android SDK built for x86. If you get graphics artifacts, consider enabling
software rendering with "--enable-software-rendering".
Launching lib/main.dart on Android SDK built for x86 in debug mode...
Initializing gradle...                                              0.9s
Resolving dependencies...                                           3.0s
Running Gradle task 'assembleDebug'...
Running Gradle task 'assembleDebug'... Done                        11.7s
Built build\app\outputs\apk\debug\app-debug.apk.
Installing build\app\outputs\apk\app.apk...                         2.8s
Error connecting to the service protocol: HttpException: Connection closed before full header was received, uri =
http://<IP_ADDRESS>:59823/M7bHVz9iLGQ=/ws

$ flutter doctor -v
[√] Flutter (Channel stable, v1.7.8+hotfix.3, on Microsoft Windows [Version 10.0.17134.885], locale en-US)
    • Flutter version 1.7.8+hotfix.3 at E:\dev\flutter
    • Framework revision b712a172f9 (13 days ago), 2019-07-09 13:14:38 -0700
    • Engine revision 54ad777fd2
    • Dart version 2.4.0

[√] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
    • Android SDK at C:\android
    • Android NDK location not configured (optional; useful for native profiling support)
    • Platform android-28, build-tools 28.0.3
    • ANDROID_HOME = C:\android
    • ANDROID_SDK_ROOT = C:\android
    • Java binary at: C:\Users\rathe\AppData\Local\JetBrains\Toolbox\apps\AndroidStudio\ch-0\183.5692245\jre\bin\java
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1343-b01)
    • All Android licenses accepted.

[!] Android Studio (version 3.4)
    • Android Studio at C:\Users\rathe\AppData\Local\JetBrains\Toolbox\apps\AndroidStudio\ch-0\183.5692245
    X Flutter plugin not installed; this adds Flutter specific functionality.
    X Dart plugin not installed; this adds Dart specific functionality.
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1343-b01)

[√] IntelliJ IDEA Ultimate Edition (version 2019.1)
    • IntelliJ at C:\Users\rathe\AppData\Local\JetBrains\Toolbox\apps\IDEA-U\ch-0\191.7479.19
    • Flutter plugin version 37.1.3
    • Dart plugin version 191.7830

[√] Connected device (1 available)
    • Android SDK built for x86 • emulator-5554 • android-x86 • Android 10 (API 29) (emulator)

Note: I use IntelliJ to manage my AVDs (and oddly enough, I DO have those Android Studio plugins installed...but whatever).

Thanks for the report, though there's already an open issue covering this.
/dupe #33050.

@DanTup #33050 is reported on May 20  (2 months ago) and still not assigned to anyone.

@DanTup Thanks.  Can you edit #33050 to have the text "API Level 29" in there as well.  Reason I didn't see it in past issues, is I searched by that text.

@AlexanderArendar I believe it's an Android issue and not something fixable in Flutter. Check the last comments in the linked issue, it seems like it may be fixed in Beta 5, though I'm not sure what the process is for updating with the beta.
@julian-sf done!


fix(shield): add missing components, missing var to brand docs js

fixes https://github.com/patternfly/patternfly-next/issues/2249

:tada: This PR is included in version 2.31.9 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


What is the difference between $item and $model in the event function params?

There are 2 params for the events https://github.com/angular-ui/ui-select/wiki/ui-select#events: $item and $model. When i log these they both contain the same object.
Are they supposed to contain different objects or what is the person of having both of them?

Why was this closed with no answer?

After a year I don't think it's getting answered. It doesn't look like it's causing a bug/problem either.
Just tidying up...

Well maybe it's pointless to have both if they are the same thing? This could be refactored to remove one and make the code simpler?
If there is a difference it'd be nice to have it documented.

Feel free to submit a PR :smiley:


[BUG] shenyu-admin pg sql script miss alert_template table

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior
create-table.sql miss alert_template table structure
Expected Behavior
No response
Steps To Reproduce
No response
Environment
ShenYu version(s): 2.5.0

Debug logs
No response
Anything else?
No response

can you fix it?


The bundle references removed images from the FrameworkBundle

The images removed in https://github.com/symfony/symfony/pull/19996 are still referenced :)
web/config.php

see symfony/symfony-standard#995

See #292.


Fix p2f root children checking

Fixed condition for checking directions of the root node children
Exception message states clearly that it is intended for at least one of them to be sufficient

Thank you!

btw any chance for a new release with this patch?


Removes IPVS prerequisites in "Validate IPv4/IPv6 dual-stack"

This is a Bug Report

Problem:
In Dualstack doc updates for 1.18 #20694, it only removes IPVS prerequisites in https://kubernetes.io/docs/concepts/services-networking/dual-stack/ but not remove it in https://kubernetes.io/docs/tasks/network/validate-dual-stack/, you still can find "Kube-proxy running in mode IPVS" in the doc.
Proposed Solution:
remove "Kube-proxy running in mode IPVS" in https://kubernetes.io/docs/tasks/network/validate-dual-stack/
Page to Update:
https://kubernetes.io/docs/tasks/network/validate-dual-stack/

/language en
/priority important-soon
/kind bug
@lachie83 – Can you verify the issue described is correct? If so we can tag it as a good first issue and have someone tackle it.

Hi @celestehorgan and @hobbytp. Thank you very much for raising this. I can indeed validate that the issue is correct and that the proposed solution is correct and would be more than happy to review a PR if someone would be happy to contribute. Thanks again

/assign

@lachie83 – no further action from you needed, this was a small fix. Thank you!


Doesn't work

I have turned on external access a month ago and just tried to add my personal skype, but it says "We are unable to set up the conversation because the contact you entered wasn't found".  So, the documentation needs to be clearer.  If you can't use teams to communicate with personal skype users, you should make that clear (also make clear when it will become possible) and save us all wasting our time trying until it is ready.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 59900927-5e5f-77a4-04be-e05a91330fca
Version Independent ID: 1e4d6b87-f57c-fa45-1630-3c32b821d93c
Content: Communicate with Teams users in another organization
Content Source: Teams/let-your-teams-users-communicate-with-other-people.md
Service: msteams
GitHub Login: @tonysmit
Microsoft Alias: tonysmit

@officedocsbot unassign @yogkumgit

@officedocsbot assign @scanum

Hi @bcraigie
Thanks for reaching out!
Currently, there is no integration between public Skype (not Skype for Business) and Teams. There is a user voice created to implement this feature and it is under review now. Please refer to the below URL.
https://microsoftteams.uservoice.com/forums/555103-public/suggestions/16958347-skype-integration-consumer-federation
Hope this information helps. Kindly confirm. Thank you.

@bcraigie Thank you for submitting feedback. We understand that this issue has been resolved.
Please feel free to re-open this issue if there is a specific area of the docs that we can improve or make better. Thank you.

@officedocsbot close


Synchronous XHR warning

When I source browser-source-map-support.js in Chrome 42, I get the following warning in the JS console:

Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user's experience. For more help, check http://xhr.spec.whatwg.org/.

 Logged a bug against the spec: https://www.w3.org/Bugs/Public/show_bug.cgi?id=28505. We'll see how it goes.

I was hoping to use this with Zombie headless browser but that doesn't support synchronous XMLHttpRequests: https://github.com/assaf/zombie/issues/417

This should be easy to fix - https://github.com/evanw/node-source-map-support/blob/master/source-map-support.js#L99
Either have an install and an installAsync - or the inverse to be more like Node (but more of a compatibility break)

@DominicTobias Were you able to get a fix working?


await-promise is incompatible with the Promise/A+ spec

Bug Report

TSLint version: 5.8.0
TypeScript version: 2.6.1

TypeScript code being linted
  const tweetsIterator = twitterClient.scanUserTimeline(account.twitter)
  while (true) {
    const tweets = await tweetsIterator.next()
    if (tweets.value && tweets.value.length) {
      await db.bulkInsertTweets(twitterArchiveIndex, tweets.value)
    }
    if (tweets.done) {
      break
    }
  }

Actual behavior
Invalid 'await' of a non-Promise value. (await-promise)
  68 |       const tweetsIterator = twitterClient.scanUserTimeline(account.twitter)
  69 |       while (true) {
> 70 |         const tweets = await tweetsIterator.next()
     |                       ^
  71 |         if (tweets.value && tweets.value.length) {
  72 |           await db.bulkInsertTweets(twitterArchiveIndex, tweets.value)
  73 |         }

Expected behavior
The error is triggered because the object awaited is a Funfix Future which does not inherit from Promise.
According to the Promises/A+ specification the Promise class is NOT required for await to work. You only need a "thenable" object, meaning a type that implements then with the described behavior.
In TypeScript this is exposed as PromiseLike:
interface PromiseLike<T> {
    /**
     * Attaches callbacks for the resolution and/or rejection of the Promise.
     * @param onfulfilled The callback to execute when the Promise is resolved.
     * @param onrejected The callback to execute when the Promise is rejected.
     * @returns A Promise for the completion of which ever callback is executed.
     */
    then<TResult1 = T, TResult2 = never>(onfulfilled?: ((value: T) => TResult1 | PromiseLike<TResult1>) | undefined | null, onrejected?: ((reason: any) => TResult2 | PromiseLike<TResult2>) | undefined | null): PromiseLike<TResult1 | TResult2>;
}

Possibly related: https://github.com/palantir/tslint/issues/3381
Please make this check look for PromiseLike and not for Promise.
Thanks,

You can configure what the rule should treat as promise: https://palantir.github.io/tslint/rules/await-promise/

What's the motivation for Thenable to not e the default?

The rule doesn't check for assignability / compatibility. It just checks the name. Thenable is a builtin type and even if it was, you are not required to implement that interface.

You mean it literally checks that the type's name is that string?

You mean it literally checks that the type's name is that string?

Yep, it checks the name of every base type and recurses into union and intersection types.

How's this for a docs update to clarify the issue? https://github.com/palantir/tslint/pull/3519


Can't switch the camera

When I try to switch the camera, goes back to the back camera

interesting, what device and windows version are you on?

Lumia 730, Windows 10

Sometimes I can switch, but takes a lot of time trying and pressing the camera switch buttom

I have a L730 here too and it works normally


Running Classic-McEliece-8192128f StackOverflow Exception

KEM.cs
Exception of type 'System.StackOverflowException' was thrown.
Line 226
When using this:
using (KEM client = new KEM("Classic-McEliece-8192128f"),
server = new KEM("Classic-McEliece-8192128f"))

Seems to occur for all Classic-McEliece variants

The Classic McEliece implementations in liboqs have large stack usage (see notes at https://github.com/open-quantum-safe/liboqs/blob/main/docs/algorithms/kem/classic_mceliece.md) which is probably causing this problem. Are you able to increase the stack allocation at runtime?

yes, using editbit to set a stack size for the .exe built by VS made this work. Sample command line:
editbin /stack:50000000 C:\Users\User\Downloads\liboqs-dotnet-master1\liboqs-dotnet-master\dotnetOQSSample\bin\Debug\net5.0\dotnetOQSSample.exe
This sets the built executable's stack to 50 MB - the Classic McEliece tests then pass.
Thanks


Strange lifecycle problem when using ThorntailTestRunner

Describe the bug
Thorntail appears to stop before running tests using ThorntailTestRunner.class
To Reproduce
Steps to reproduce the behavior:
Appears to be an edge case and appears extremely difficult to reproduce I will do my best to find a way to reproduce, but the repo the error is on is closed-source so I cannot simply link to code.
Expected behavior
I expected tests to run and thorntail to spin down after test cases were executed
Desktop (please complete the following information):

OS: Fedora 28
Browser N/A

Stacktrace(s)
INFO  : Mo Okt 08 12:27:28 PDT 2018 [io.thorntail.jpa] THORN-007011: initializing persistence contexts
INFO  : Mo Okt 08 12:27:28 PDT 2018 [io.thorntail.kernel] THORN-000013: phase [after start] completed in 0.98s
INFO  : Mo Okt 08 12:27:28 PDT 2018 [io.thorntail.kernel] THORN-000999: Thorntail started in 0.400s
INFO  : Mo Okt 08 12:27:28 PDT 2018 [io.thorntail.kernel] THORN-000011: Thorntail stopping
INFO  : Mo Okt 08 12:27:28 PDT 2018 [org.hibernate.ogm.datastore.mongodb.impl.MongoDBDatastoreProvider] OGM001202: Closing connection to MongoDB
INFO  : Mo Okt 08 12:27:28 PDT 2018 [org.mongodb.driver.connection] Closed connection [connectionId{localValue:10, serverValue:54}] to localhost:27017 because the pool has been closed.
INFO  : Mo Okt 08 12:27:28 PDT 2018 [io.thorntail.kernel] THORN-000012: Thorntail stopped
Tests run: 3, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 0.391 sec <<< FAILURE!
testAuthenticateWrongPasswordShouldFail(com.closedsource.code.AuthenticationTest)  Time elapsed: 0.001 sec  <<< ERROR!
java.lang.IllegalStateException: EntityManagerFactory is closed

Additional context

Update: I've managed to eliminate this particular problem by doing three things:

Downgrading Hibernate components to all stable versions (including OGM) in Thorntail v4
Rebuilding Thorntail v4 and installing to local cache
Adding the following to the project's pom.xml under <plugins />:

			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-surefire-plugin</artifactId>
				<version>2.22.0</version>
				<configuration>
					<forkMode>always</forkMode>
					<reuseForks>true</reuseForks>
				</configuration>
			</plugin>


Refactoring - take out of the library all related to sTLOS and wTLOS

Description
The Antelope Library attempts to become a common store library for every Antelope-compatible network, therefore all non-common features should be implemented outside the library. However, the library may provide a way to facilitate the task of incorporating particular features like having two special tokens such as sTLOS and wTLOS.
In this particular case, we need to remove everything related to these two specific tokens and instead provide an elegant solution for the incorporation of specific tokens' info.
https://github.com/telosnetwork/telos-wallet/blob/dcbccf75dbbb6917657577448dce36c985d5a993/src/antelope/chains/EVMChainSettings.ts#L92-L93
https://github.com/telosnetwork/telos-wallet/blob/dcbccf75dbbb6917657577448dce36c985d5a993/src/antelope/chains/evm/telos-evm-testnet/index.ts#L89-L95
https://github.com/telosnetwork/telos-wallet/blob/dcbccf75dbbb6917657577448dce36c985d5a993/src/antelope/chains/evm/telos-evm/index.ts#L89-L95

The referenced code is not present anymore. I close this issue


final missing admin bits

Adds some final missing admin bits: picard, and the export tool that comes the selfregistration app.
Additions

URL patterns, redirects, and OPTIONAL_APPS config for picard and selfregistration
some sample scripts for picard
adds 'task Launcher' (aka Picard) to django admin nav

Testing

check out CFPB/django-cfpb-common and pip install -e <path> it
check out CFPB/Picard and pip install -e <path> it
check out CFPB/cfgov-selfregistration and pip install -e <path> it
pip install django-braces==1.0.0
cfgov/manage.py migrate
cfgov/manage.py runserver
Observe that http://localhost:8000/picard/ redirects to localhost:8000/tasks/ (will require login if you aren't already logged in), and that running any tasks produce the expected output "this script only exists to prove that Picard is working"
fill out this form: http://localhost:8000/company-signup/
go to http://localhost:8000/selfregs/export/ and click 'export all'
dig that sweet CSV file.

Review

@kave @kurtw @richaagarwal

Checklist

[ ] Changes are limited to a single goal (no scope creep)
[x] Code can be automatically merged (no conflicts)
[ ] Code follows the standards laid out in the front end playbook
[ ] Passes all existing automated tests
[ ] New functions include new tests
[ ] New functions are documented (with a description, list of inputs, and expected output)
[ ] Placeholder code is flagged
[ ] Visually tested in supported browsers and devices
[ ] Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)

add Task Launcher to the django admin navigation

@rosskarchner i get a 404 page doesn't exist when it redirects to /task/

everything works, except for posting in the company form. getting a 403 forbidden response...

Everything works for me 👍

@kave I couldn't replicate the problem, and it currently works on build. I'm gonna merge.


Potentially false validation error if image size > maxResourceSize

Running dEQP-VK.pipeline.render_to_image.core.2d_array.huge.width_height_layers.r8g8b8a8_unorm produces a validation error:

ERROR: Object: 0x0 | CreateImage resource size exceeds allowable maximum Image resource size = 0x400000000, maximum resource size = 0x80000000 (code 0x00000000 from Image at Image:867)

when trying to create an image using the following info:
VkImageCreateInfo
{
    VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO,
    0,
    0,
    VK_IMAGE_TYPE_2D,
    VK_FORMAT_R8G8B8A8_UNORM,
    { 4096, 4096, 1 },
    1,
    256,
    VK_SAMPLE_COUNT_1_BIT,
    VK_IMAGE_TILING_OPTIMAL,
    VK_IMAGE_USAGE_TRANSFER_SRC_BIT | VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT,
    VK_SHARING_MODE_EXCLUSIVE,
    0,
    0,
    VK_IMAGE_LAYOUT_UNDEFINED
}

I don't think this is quite right. Specification says:

There is no mechanism to query the size of an image before creating it, to compare that size against maxResourceSize. If an application attempts to create an image that exceeds this limit, the creation will fail and vkCreateImage will return VK_ERROR_OUT_OF_DEVICE_MEMORY.

Even if vkGetPhysicalDeviceImageFormatProperties2 returns success and the parameters to vkCreateImage are all within the returned limits, vkCreateImage must fail and return VK_ERROR_OUT_OF_DEVICE_MEMORY if the resulting size of the image would be larger than maxResourceSize.

On this device imageGranularity is 1 so it should not affect this in any way.

4096 x 4096 x 256 x 4 = 0x4 0000 0000, which is greater than 0x8000 0000(=231),  so that checks out.
It should be a WARNING, not ERROR, oterwise I don't see what the problem is?

Spec quote from @pppessi is out of date, potentially. Current language is:

There is no mechanism to query the size of an image before creating it, to compare that size against maxResourceSize. If an application attempts to create an image that exceeds this limit, the creation will fail or the image will be invalid. While the advertised limit must be at least 2^(31), it may not be possible to create an image that approaches that size, particularly for VK_IMAGE_TYPE_1D.

Since this causes either failure or an invalid image I like having it as an ERROR.
Tagging @chrisforbes for input as this is a deqp test.

@tobine Is that unreleased version of the spec? Sounds like nonsense though; "no mechanism to query the size" vs "or the image will be invalid", which makes all images in quantum superposition of being valid and invalid.

Yes, I'm sorry. I was quoting the last version of the 1.0 spec. I see language @pppessi is quoting from the latest 1.1 spec.
Given this requirement:

Even if vkGetPhysicalDeviceImageFormatProperties2. returns success and the parameters to vkCreateImage are all within the returned limits, vkCreateImage must fail and return VK_ERROR_OUT_OF_DEVICE_MEMORY if the resulting size of the image would be larger than maxResourceSize.

I agree that the validation check should be changed to a warning as in #2487. Thanks!

@tobine Right, it was apparently fixed recently in the 70, which is not exposed in 1.0 for technical reasons.
The new quote is unguarded by macro though, so it applies to 1.0 too.

@tobine I'm torn between converting this to a warning and getting rid of it entirely.
Reasons for the latter:

Drivers are required to check this condition, so we shouldn't need to.
We don't know the actual size -- the layers complain based on an estimate that's oblivious to tiling/alignment/etc constraints. It's probably pretty close for a single 2D layer, but there's no reason to believe it's even close for multi-level and/or multi-layer scenarios.

@chrisforbes good point. After a bit of thought I still slightly favor WARNING over killing the check. I just have a fear that some driver, esp. older ones, may not return error correctly so the warning is a nice backup.
I agree that validation layers really shouldn't have to check the condition and even the existing check is just a guess, though, so I'm fine killing if that's your preference.
For the near term I'll approve/push #2487, but if you want me to kill I can follow-on with another PR.

The drivers return cookie-cutter error for the case.
It is still nice to know the real reason for the error. Assuming some good lower-bound for the resource use can be "guessed".
It might better fit one of the proposed "guidelines" layer, though trying to make a resource of > 231 bytes (or as in the OP even 16 GB) on a contemporary desktop could in of itself be considered likely error (i.e. warning).

:+1:

WARNING update #2487 from @krOoze was pushed. Closing this issue. Thanks, everyone!


[SPARK-12139] [SQL] REGEX Column Specification

What changes were proposed in this pull request?
Hive interprets regular expression, e.g., (a)?+.+ in query specification. This PR enables spark to support this feature when hive.support.quoted.identifiers is set to true.
How was this patch tested?

Add unittests in SQLQuerySuite.scala
Run spark-shell tested the original failed query:
scala> hc.sql("SELECT (a|b)?+.+ from test1").collect.foreach(println)

Can one of the admins verify this patch?

ok to test

Test build #77062 has started for PR 18023 at commit af55afd.

Test build #77062 has finished for PR 18023 at commit af55afd.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class UnresolvedRegex(expr: String, table: Option[String]) extends Star with Unevaluable

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77062/
Test FAILed.

Please update the PR title to [SPARK-12139] [SQL] REGEX Column Specification

Test build #77063 has started for PR 18023 at commit 43beb07.

Test build #77063 has finished for PR 18023 at commit 43beb07.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77063/
Test FAILed.

Test build #77068 has started for PR 18023 at commit 7699e87.

Test build #77068 has finished for PR 18023 at commit 7699e87.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77068/
Test PASSed.

Test build #77081 has started for PR 18023 at commit 6e37517.

Like what we did for * in Column.scala, we also need to handle the Dataset APIs. You can follow the way we handle star there.
df.select(df("(a|b)?+.+"))

Test build #77081 has finished for PR 18023 at commit 6e37517.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class UnresolvedRegex(regexPattern: String, table: Option[String])

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77081/
Test PASSed.

Test build #77086 has started for PR 18023 at commit bee07cd.

Test build #77086 has finished for PR 18023 at commit bee07cd.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77086/
Test FAILed.

Merged build finished. Test FAILed.

Test build #77099 has started for PR 18023 at commit 979bfb6.

Test build #77099 has finished for PR 18023 at commit 979bfb6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77099/
Test PASSed.

Test build #77108 has started for PR 18023 at commit 612bedf.

LGTM pending Jenkins
cc @hvanhovell @cloud-fan

Test build #77109 has started for PR 18023 at commit 48c54aa.

Test build #77108 has finished for PR 18023 at commit 612bedf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77108/
Test PASSed.

Test build #77109 has finished for PR 18023 at commit 48c54aa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77109/
Test PASSed.

Test build #77192 has started for PR 18023 at commit 0284d01.

Test build #77192 has finished for PR 18023 at commit 0284d01.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class UnresolvedRegex(regexPattern: String, table: Option[String])

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77192/
Test PASSed.

@cloud-fan, there are difference between xyz and xyz. we always need to extract xyz part from xyz, so we will need to pattern match. if we do not get pattern matched, we know it is not regex, then why bother to use UnsolvedRegex, which will need to project list expansion later.
Hive supper regex column specification, see https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Select.

Test build #77442 has started for PR 18023 at commit 779724d.

Test build #77442 has finished for PR 18023 at commit 779724d.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77442/
Test FAILed.

Test build #77443 has started for PR 18023 at commit a27023c.

Test build #77443 has finished for PR 18023 at commit a27023c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77443/
Test FAILed.

Test build #77548 has started for PR 18023 at commit a0e3773.

Test build #77548 has finished for PR 18023 at commit a0e3773.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77548/
Test PASSed.

Merged build finished. Test PASSed.

Ping. I went through the individual reviewers' comment, it seems I addressed every one. Not sure if I am missing anyone, please let me know. Thanks!

Will review it this weekend. Thanks for your work!

Could you please add another case when no column is matched into query_regex_column.sql? Thanks!

Test build #78645 has started for PR 18023 at commit da60368.

Test build #78645 has finished for PR 18023 at commit da60368.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78645/
Test PASSed.

Merged build finished. Test PASSed.

do we need to consider case sensitivity when matching the regex?

Yes. We need to consider it.

(?i) makes the regex case insensitive.

Test build #78773 has started for PR 18023 at commit f98207b.

Test build #78773 has finished for PR 18023 at commit f98207b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78773/
Test PASSed.

Merged build finished. Test PASSed.

Test build #78815 has started for PR 18023 at commit 321211d.

Test build #78815 has finished for PR 18023 at commit 321211d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78815/
Test PASSed.

Test build #78955 has started for PR 18023 at commit 4e36ed9.

Test build #78955 has finished for PR 18023 at commit 4e36ed9.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class UnresolvedRegex(regexPattern: String, table: Option[String], caseSensitive: Boolean)

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78955/
Test FAILed.

Merged build finished. Test FAILed.

Jenkins test this please

retest this please

Test build #79003 has started for PR 18023 at commit 4e36ed9.

Test build #79004 has started for PR 18023 at commit 448c3e2.

Test build #79003 has finished for PR 18023 at commit 4e36ed9.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class UnresolvedRegex(regexPattern: String, table: Option[String], caseSensitive: Boolean)

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79003/
Test PASSed.

Test build #79004 has finished for PR 18023 at commit 448c3e2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79004/
Test FAILed.

Merged build finished. Test FAILed.

Let me post a confusing error message.
    withSQLConf(SQLConf.SUPPORT_QUOTED_REGEX_COLUMN_NAME.key -> "true") {
      Seq((1, 1)).toDF("key", "value").createOrReplaceTempView("test")
      sql("select `key` from test where `key` > 3").show()
    }

The error message is:
Invalid call to dataType on unresolved object, tree: key
org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to dataType on unresolved object, tree: key
	at org.apache.spark.sql.catalyst.analysis.Star.dataType(unresolved.scala:250)
	at org.apache.spark.sql.catalyst.expressions.BinaryOperator.checkInputDataTypes(Expression.scala:517)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:92)

Users might use quoted attributes in any place of a query.

Fixed the failed testcases pointed out by gatorsmile.

Test build #79238 has started for PR 18023 at commit d65c462.

Test build #79238 has finished for PR 18023 at commit d65c462.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79238/
Test FAILed.

Test build #79239 has started for PR 18023 at commit 65e5eec.

Test build #79239 has finished for PR 18023 at commit 65e5eec.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with Logging
        throw new AnalysisException(s\"UDF class $
        throw new AnalysisException(s\"It is invalid to implement multiple UDF interfaces, UDF class $
              throw new AnalysisException(s\"UDF class with $
            throw new AnalysisException(s\"Can not instantiate class $
      case e: ClassNotFoundException => throw new AnalysisException(s\"Can not load class $
        throw new AnalysisException(s\"class $className doesn't implement interface UserDefinedAggregateFunction\")
      case e: ClassNotFoundException => throw new AnalysisException(s\"Can not load class $
        throw new AnalysisException(s\"Can not instantiate class $
class SparkSqlParser extends AbstractSqlParser
class SparkSqlAstBuilder extends AstBuilder
class VariableSubstitution

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79239/
Test FAILed.

Test build #79240 has started for PR 18023 at commit 65886cd.

Test build #79240 has finished for PR 18023 at commit 65886cd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79240/
Test PASSed.

Left a comment https://github.com/apache/spark/pull/18023#discussion_r126079978

Do we still care hive.support.quoted.identifiers? If no, please update the PR description accordingly.

@gatorsmile, regarding:

Left a comment #18023 (comment)
I have already fixed the issue.

Could you add some test cases in the other parts of the query? For example, group by clauses.

@viirya regarding

Do we still care hive.support.quoted.identifiers? If no, please update the PR description accordingly.

Yes, we still have hive.support.quoted.identifiers. only when spark.sql.parser.quotedRegexColumnNames = true, ... in SELECT statements will be treated in regex as in Hive

@gatorsmile regarding

Could you add some test cases in the other parts of the query? For example, group by clauses.

I added some tests for aggreation. But for group by, we cannot have regex there. group by requires the fields to be orderable. As hive states and our previous comments, we should only support regex in SELECT.

@janewangfb That is fine we do not support, but we still need to add test cases for these negative cases. Thanks!

@gatorsmile, regarding:

@janewangfb If we turn on the flag spark.sql.parser.quotedRegexColumnNames by default, the > following test cases failed. Could you do some investigations? Thanks!
org.apache.spark.sql.SQLQuerySuite
the struct type was not supported now in the regex
some special characters has different meaning in regex.

org.apache.spark.sql.DataFrameSuite
some special characters has different meaning in regex.

org.apache.spark.sql.SingleLevelAggregateHashMapSuite
org.apache.spark.sql.DataFrameAggregateSuite
org.apache.spark.sql.TwoLevelAggregateHashMapSuite
org.apache.spark.sql.TwoLevelAggregateHashMapWithVectorizedMapSuite
org.apache.spark.sql.DataFrameNaFunctionsSuite
org.apache.spark.sql.DataFrameStatSuite
These four failed for the same testcase. in AS alias, regex is not allowed.

org.apache.spark.sql.SQLQueryTestSuite
This suite has the same behavior wether spark.sql.parser.quotedRegexColumnNames default value is true/false.

org.apache.spark.sql.execution.datasources.json.JsonSuite
for map struct, regex should not be allowed in A[B] part.

org.apache.spark.sql.DatasetSuite
Expected. Explicitly set the spark.sql.parser.quotedRegexColumnNames = false to false for those tests.

org.apache.spark.sql.sources.TableScanSuite
some special characters has different meaning in regex

org.apache.spark.sql.execution.datasources.parquet.ParquetFilterSuite
regex is not allowed in where.

@gatorsmile, regarding

That is fine we do not support it, but we still need to add test cases for these negative cases. Thanks!

Yes, I have added testcases.

@gatorsmile regarding:

Could you add some test cases in the other parts of the query? For example, group by clauses.
Yes, added.

Test build #79476 has started for PR 18023 at commit 8adad7c.

Test build #79476 has finished for PR 18023 at commit 8adad7c.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AstBuilder(conf: SQLConf) extends SqlBaseBaseVisitor[AnyRef] with Logging
class CatalystSqlParser(conf: SQLConf) extends AbstractSqlParser
class SparkSqlParser(conf: SQLConf) extends AbstractSqlParser
class SparkSqlAstBuilder(conf: SQLConf) extends AstBuilder(conf)
class VariableSubstitution(conf: SQLConf)

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79476/
Test FAILed.

Test build #79503 has started for PR 18023 at commit 56e2b83.

Test build #79503 has finished for PR 18023 at commit 56e2b83.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ArrowSerializer(FramedSerializer):

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79503/
Test FAILed.

Merged build finished. Test FAILed.

Test build #79504 has started for PR 18023 at commit d613ff9.

Test build #79504 has finished for PR 18023 at commit d613ff9.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79504/
Test FAILed.

Merged build finished. Test FAILed.

@gatorsmile regarding:

Could you revert back all the unneeded changes? (in JsonSuite.scala).

(I saw this comment in email but didn't find in the PR) I have reverted the unneeded changes.

Test build #79535 has started for PR 18023 at commit a5f9c44.

Test build #79535 has finished for PR 18023 at commit a5f9c44.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79535/
Test PASSed.

Merged build finished. Test PASSed.

The last comment is about DataFrameNaFunctions.fill. It does not work when spark.sql.parser.quotedRegexColumnNames is on. Could you resolve that in the follow-up PR?

LGTM

Thanks! Merging to master.

@gatorsmile
Sure, I could have a follow-up PR to resolve DataFrameNaFunctions.fill.
thanks for reviewing this PR.


Make extension signing compatible with current Safari releases

Sign the compressed table of contents in the xar archive
using 'openssl dgst -sign' rather than 'openssl rsautl -sign'.
See https://github.com/avast/safariextz/issues/1#issuecomment-131229944
and the linked StackOverflow post at http://stackoverflow.com/a/16264508
for an explanation of the difference.
This requires an update to xar from
https://github.com/robertknight/xar/commit/847103b787cfddf200c61d7d93fb57f993bdd976
that fixes the --dump-toc-raw option.

For our needs this PR and the one for xar isn't needed any more as we should be able to use the xar-js package. However, I would still be interested to know why the changes here worked for us but not in your case @pavelstudeny . In xar-js I'm signing the package using the same method as in this PR, except using node's crypto APIs directly instead of shelling out to openssl.

xar-js used in release 2.0.1 - the reason why this PR didn't work was probably unexpected certificate format (DER instead of PEM), which was the case also when testing xar-js


Address the case where the root span parentId is not empty

There may be cases where the parentId of the root span is not empty but traceId..
This PR addresses that case.

This has been resolved by #3507 instead


How do you use it to make it predict a sentence?

Could you also tell on how to use the link you provided for each models?

Same question: "How do you use it to make it predict a sentence?"


has_and_belongs_to_many error: Mongo::OperationFailure: invalid query

I have the following association:
class Account
has_and_belongs_to_many :followed_by, class_name: Account.to_s
end
running the following:
a = Account.new
a.followed_by
gives the error:
Mongo::OperationFailure: invalid query
from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongo-1.3.1/lib/mongo/cursor.rb:101:in next_document' 	from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongo-1.3.1/lib/mongo/cursor.rb:248:in each'
from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/cursor.rb:48:in each' 	from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/contexts/mongo.rb:212:in iterate'
from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/criteria.rb:143:in block in each' 	from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/criteria.rb:143:in tap'
from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/criteria.rb:143:in each' 	from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/relations/targets/enumerable.rb:153:in each'
from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/relations/targets/enumerable.rb:223:in entries' 	from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/relations/targets/enumerable.rb:223:in inspect'
from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/relations/referenced/many.rb:394:in method_missing' 	from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/railties-3.1.3/lib/rails/commands/console.rb:45:in start'
from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/railties-3.1.3/lib/rails/commands/console.rb:8:in start' 	from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/railties-3.1.3/lib/rails/commands.rb:40:in <top (required)>'
from script/rails:11:in require' 	from script/rails:11:in '
Also,
doing:
a.followed_by << Account.first
and then the following gives the same Mongo::OperationFailure: invalid query:
a.followed_by
doing a.reload works.

Are you potentially not using MongoDB 2.0?

This is the query:
MONGODB myapp_development['accounts'].find({:_id=>{"$in"=>nil}})

Hey Fredrik. When customizing the name of a relation, it's safest to
provide the inverse_of option. If you add that, does it fix the problem?
On Dec 5, 2011 3:11 PM, "Fredrik Bjrk" <
<EMAIL_ADDRESS>wrote:

I have the following association:
class Account
has_and_belongs_to_many :followed_by, class_name: Account.to_s
end
running the following:
a = Account.new
a.followed_by
gives the error:
Mongo::OperationFailure: invalid query
from
/Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongo-1.3.1/lib/mongo/cursor.rb:101:in
next_document'        from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongo-1.3.1/lib/mongo/cursor.rb:248:in each'
from
/Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/cursor.rb:48:in
each'        from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/contexts/mongo.rb:212:in iterate'
from
/Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/criteria.rb:143:in
block in each'        from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/criteria.rb:143:in tap'
from
/Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/criteria.rb:143:in
each'        from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/relations/targets/enumerable.rb:153:in each'
from
/Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/relations/targets/enumerable.rb:223:in
entries'        from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/relations/targets/enumerable.rb:223:in inspect'
from
/Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/relations/referenced/many.rb:394:in
method_missing'        from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/railties-3.1.3/lib/rails/commands/console.rb:45:in start'
from
/Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/railties-3.1.3/lib/rails/commands/console.rb:8:in
start'        from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/railties-3.1.3/lib/rails/commands.rb:40:in <top (required)>'
from script/rails:11:in require'        from script/rails:11:in '
Also,
doing:
a.followed_by << Account.first
and then the following gives the same Mongo::OperationFailure: invalid
query:
a.followed_by
doing a.reload works.

Reply to this email directly or view it on GitHub:
https://github.com/mongoid/mongoid/issues/1478

I tried this:
has_and_belongs_to_many :followed_by, class_name: Account.to_s, inverse_of: nil
but I still get this error:
Mongo::OperationFailure: invalid query
from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongo-1.3.1/lib/mongo/cursor.rb:101:in next_document' 	from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongo-1.3.1/lib/mongo/cursor.rb:248:in each'
from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/cursor.rb:48:in each' 	from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/contexts/mongo.rb:212:in iterate'
from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/criteria.rb:143:in block in each' 	from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/criteria.rb:143:in tap'
from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/criteria.rb:143:in each' 	from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/relations/targets/enumerable.rb:153:in each'
from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/relations/targets/enumerable.rb:223:in entries' 	from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/relations/targets/enumerable.rb:223:in inspect'
from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/mongoid-2.3.3/lib/mongoid/relations/referenced/many.rb:394:in method_missing' 	from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/railties-3.1.3/lib/rails/commands/console.rb:45:in start'
from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/railties-3.1.3/lib/rails/commands/console.rb:8:in start' 	from /Users/fredrik/.rvm/gems/ruby-1.9.2-p180/gems/railties-3.1.3/lib/rails/commands.rb:40:in <top (required)>'
from script/rails:11:in require' 	from script/rails:11:in '

Which version of Mongoid are you using? With 2.3.4 , it works for me
as expected:
class Account
  include Mongoid::Document
  has_and_belongs_to_many :followed_by, class_name: Account.to_s,
inverse_of: nil
end

a = Account.new
=> #<Account _id: 4edd504a2b58af0a10000001, _type: nil,
followed_by_ids: []>

a.followed_by
=> []

2.3.3, was it fixed in 2.3.4?

To be honest, I don't remember this being a problem for a long time.
Do you have anything else in your model that might be affecting this?

Same issue with, 2.3.4 could it be something with mongo driver 1.3.1? which driver do you use?

1.4.1 , I believe. I'm in my phone right now, though, so that's from memory.
On Dec 5, 2011 6:33 PM, "Fredrik Bjrk" <
<EMAIL_ADDRESS>wrote:

Same issue with, 2.3.4 could it be something with mongo driver 1.3.1?
which driver do you use?

Reply to this email directly or view it on GitHub:
https://github.com/mongoid/mongoid/issues/1478#issuecomment-3025515

Ok, since 1.4.1 is pulled I cannot test with it. Are you on 1.4.0 or perhaps 1.5.1?

I just checked: 1.5.1 !

I cannot replicate this against master... Could this have been the same as #1481?

Yeah, #1481 looks like the same problem. I can't reproduce this on master anymore, either.
@fbjork Can you confirm that this problem no longer occurs for you using the master branch of Mongoid?

Going to close.