METRON-1526: Location field types cause DocValuesField appear more than once error

Contributor Comments
Solr automatically expands location type fields internally when a document is indexed.  This causes a problem in our DAO layer because we don't do partial updates (we reindex the whole document) and these expanded fields are included in the updated document.  This PR strips these expanded fields out in the SolrUpdateDao before updating.
I also added a unit test to cover this case.
Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our Development Guidelines for the complete guide to follow for contributions.
Please refer also to our Build Verification Guidelines for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
For all changes:

[x] Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
[x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

For code changes:

[ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?

[ ] Have you included steps or a guide to how the change may be verified and tested manually?

[ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh

[x] Have you written or updated unit tests and or integration tests to verify your changes?

[x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

[x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

For documentation related changes:

[] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html:
cd site-book
mvn site

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that travis-ci is set up for your personal repository such that your branches are built there before submitting a pull request.

It seems as though we should be checking the type for isPolyField() (see here.
The problem, as I see it, is that we don't ahve access to the type directly.  So, in order to have the type, we may need a type cache.  I'd suggest a LFU caffeine cache that will retrieve the schema for the sensor from solr.  Thoughts?

Is it possible for parser writers to create other fields that have this issue?

This causes a problem in our DAO layer because we don't do partial updates (we reindex the whole document) and these expanded fields are included in the updated document.

@merrimanr Can you elaborate on this a bit? I'm not sure I follow the full scope of the problem. Per @ottobackwards comment, would new fields introduced by parsers also cause issues? What are the parameters around when a user would "step in it?" What prophylaxis do we need to ensure, or at least reduce the risk, that this can happen for other fields?
This makes me think of another topic that's been discussed recently about message envelopes (e.g. syslog and parser chaining) that potentially wrap multiple other log types that need to be parsed. If the wrapped messages all routed to the same index, is that a problem?

@ottobackwards At the very least, if this is indeed a problem with all PointType fields and not just lat/long fields, then this may impact a common data science workflow:

Generate a model which outputs a vector representation for some input
Enrich data with that model (e.g. generate a vector per data point)
Index the data as a Point in solr
Find similar data by doing nearest neighbor searches

I am thinking of the metron as a framework POV here.  Right now, someone can create a parser, an enrichment a (in this PR's world) solr/es schema for their input.
This PR makes assumptions about a single use case, but the platform allows the capability for this to happen in many other ways that are beyond the control or knowledge of the 'core' platform through the platform's extensibility.
That makes any remediation of this issue a 'one' off it assumes knowledge of the data, which the platform cannot have beyond the parsers that ship.  And even then that knowledge ( based on the known configuration ) is not reliable because they are examples and can be modified.
Does that make sense or am I missing something?

There are a finite number of field types that ship with Solr so the fact that someone can create an infinite number of fields really isn't an issue.  Any field that is created and stored in Solr has to be one of the finite field types.  I agree that we can't assume knowledge of data hence handling all field types that could be potentially problematic.
Are there other solutions to this problem that we're not thinking of?

@cestella I think that must be the case, given that we cannot assume the fields present beyond a certain set.

I'm sure you did this, @merrimanr, but I googled your error and ran upon this.  Are we certain that we're not in this situation where a simple migration to solr.LatLonPointSpatialField isn't the solution?

I did see that but that field type doesn't in Solr 5.5.

Yes, but according to that thread, this became a problem in 6.5.  In 5.5, presumably, this worked.  Should we, perhaps, have different schema for 5.x vs 6.x?

One thing that I would question, looking at that answer, is whether it's just a LatLonType problem or if it really is a problem with all poly fields.  I would suggest tryign to replicate with a currency type, which is also a polyfield.  If the same error does not happen, then we may be safe with just handling this for point types.

Another possibility, which these guys did is disabling docValues for the spatial field.  I'm not clear on the consequences of that.

An interesting consequence of not using docValues is https://lucene.apache.org/solr/guide/6_6/docvalues.html#DocValues-RetrievingDocValuesDuringSearch

When useDocValuesAsStored="false", non-stored DocValues fields can still be explicitly requested by name in the fl param, but will not match glob patterns ("*").

This sounds more like what we want, right?  From my selfish metaalerts perspective.  Why should be be returning the subfields of polyfields to the user?  My concerns are more around if this causes perf problems around sorting those fields.

After doing more research on this, managing fields included when updating a document looks to be more complex than we thought.  The isPolyField() method is part of the internal API and we don't have access to it with SolrJ.  Also, I found another field that is stored in multiple fields internally (date).  I suspect we will find more special cases like this and if there are cases we don't find, it could break things later if a user adds a field with that type.
Should we investigate adding partial update support to the SolrDao?  There is a patch method in the IndexDao so the interfaces would not need to change.  The downside is this will introduce some constraints on schema design:  https://lucene.apache.org/solr/guide/6_6/updating-parts-of-documents.html#UpdatingPartsofDocuments-FieldStorage.

Before going down the path of partial document update:

Can you confirm that this isn't an issue when you set docValues to false, like in my previous comment?
Are date fields polyfields?
Is this a problem in 5.x or just 6.x?

Is there a workaround for dates like there is for latlon in 6.x (e.g. another type)?  If so, wouldn't differentiating schema per version solve the issue?

If you did want to go with partial updates:

how do you propose to pull the JSON Patch apart into just the selected fields?
Since we will possibly expose these generated fields (perhaps not for latlon, but possibly for other polytypes), aren't we in the same spot of having a way to update both a generated field and a derived field if the users can update derived fields?

If the answer to that is to whitelist updates on derived fields, aren't we back in the same spot as we are currently in?

It might also be possible to do this the other way, rather than getting a list of all fields then figuring out which are polyFields, we might be able to get just all valid fields.  Substantially more validation would have to be done on what I've tested here though to make sure it's valid and I didn't skip over something important.
I threw this together in one of the metaalerts tests:
    MetronSolrClient client = solr.getSolrClient();
    client.setDefaultCollection("test");
    LukeRequest lukeRequest = new LukeRequest();
    lukeRequest.setShowSchema(true);
    LukeResponse result = lukeRequest
        .process(client);
    Map<String, FieldInfo> fieldInfo = result.getFieldInfo();
    for(Entry<String, FieldInfo> entry : fieldInfo.entrySet()) {
      System.out.println("KEY: " + entry.getKey());
      System.out.println("VALUE NAME: " + entry.getValue().getName());
      System.out.println("FLAGS: " + entry.getValue().getFlags());
      System.out.println();
    }

Output is

KEY: new-field
VALUE NAME: new-field
FLAGS: [INDEXED, STORED, OMIT_NORMS, OMIT_TF, SORT_MISSING_LAST]
KEY: score
VALUE NAME: score
FLAGS: [INDEXED, STORED, DOC_VALUES, OMIT_NORMS, OMIT_TF]
KEY: version
VALUE NAME: version
FLAGS: [DOC_VALUES]
KEY: threat:triage:score
VALUE NAME: threat:triage:score
FLAGS: [INDEXED, STORED, DOC_VALUES, OMIT_NORMS, OMIT_TF]
KEY: root
VALUE NAME: root
FLAGS: [INDEXED, OMIT_NORMS, OMIT_TF, SORT_MISSING_LAST]
KEY: location_point
VALUE NAME: location_point
FLAGS: [INDEXED, STORED, OMIT_TF]
KEY: name
VALUE NAME: name
FLAGS: [INDEXED, STORED, OMIT_NORMS, OMIT_TF, SORT_MISSING_LAST]
KEY: guid
VALUE NAME: guid
FLAGS: [INDEXED, STORED, OMIT_NORMS, OMIT_TF, SORT_MISSING_LAST]
KEY: metaalerts
VALUE NAME: metaalerts
FLAGS: [INDEXED, STORED, MULTI_VALUED, OMIT_NORMS, OMIT_TF, SORT_MISSING_LAST]
KEY: source.type
VALUE NAME: source.type
FLAGS: [INDEXED, STORED, OMIT_NORMS, OMIT_TF, SORT_MISSING_LAST]
KEY: timestamp
VALUE NAME: timestamp
FLAGS: [INDEXED, STORED, DOC_VALUES, OMIT_NORMS, OMIT_TF]

Notice there's no "_coordinate" fields. Having "location_point" defined in my schema with:
  <field name="location_point" type="location" indexed="true" stored="true"/>
  <dynamicField name="*_coordinate" type="pdouble" indexed="true" stored="false"/>
  ...
  <fieldType name="location" class="solr.LatLonType" subFieldSuffix="_coordinate"/>

Resulted in the "_coordinate" fields being present in this query:
    client = solr.getSolrClient();
    client.setDefaultCollection("test");
    SolrQuery query = new SolrQuery("*:*");
    query.set("fl", "*");
    QueryResponse response = client.query(query);
    System.out.println(response.getResults());

Like so (with all the extra stripped out so it's easier to read here):
{
  ...,
  docs=[
    SolrDocument{threat:triage:score=0.0, location_point=10,20, guid=message_0, metaalerts=[meta_active], source.type=test, timestamp=1524157278555, ip_src_port=8010, location_point_1_coordinate=20.0, _version_=1598194742551642112, location_point_0_coordinate=10.0, ip_src_addr=<IP_ADDRESS>},
    ...
  ]}

There is a catch: the schema has to be updated and I'm not sure it's compatible with both 5.x and 6.x
<requestHandler name="/admin/luke" class="org.apache.solr.handler.admin.LukeRequestHandler" />

I'm not following. A patch would only include the field to be added, changed, or removed.

JSONPatch, the input that we accept for the patch command, offers more than add, change and remove operations.  There is also move and add supports adding into an array.  My point is that the format is more complex than just passing in a list of fields to add, change or remove.

If for some reason a user is trying to directly update a generated field that is incorrect and a user error.
Consider this from a user perspective for a moment.  How would a user know if a field is generated or not? If we could determine which fields were generated to ensure the users aren't given the ability to change them, then we could whitelist them generally and solve this problem.

I'm not opposed to partial updates, but I don't see that as a solution to the problem from this PR.  It feels like kicking the can down the road and making it the user's problem, which I don't agree with.

@justinleet Looks like LukeRequestHandler has been around since Solr 1.2.

The schema declaration itself might have changed between 5.x and 6.x.  I saw something about it, but didn't double check since I hadn't looked in to the request at all yet. The request itself should be fine though

@justinleet Your example suggests that by using a LukeRequest and calling getFieldInfo() on the result, we should be able to naturally obtain the filtered list of fields that can be used as a whitelist when constructing the index update? And it's available since Solr 1.2? This sounds pretty clean to me and it seems to solve the problem in a general way without exposing Solr implementation details to the end user (whether by API or documentation).

I agree, I'm not seeing how partial updates solves the general problem of "We need to avoid updates against derived fields".  End users aren't going to necessarily know what a generated field is.  Say a data scientist writes a machine learning model, and an analyst wants to update some output of the model.  They may not know, or even be able to easily discover, what fields are poly types and what aren't. They'll just get an error.  I'm not even convinced it'll be an easily understandable error to anyone not familiar with Solr.
It seems like there one real, solution to this problem that doesn't compromise usability: Don't pass sub fields back in queries.
This can be something in the schema itself (e.g. does docValues change work).  It could also be something determined while running (e.g. LukeRequest if it turns out to be viable or some other request / query).

@justinleet will this return dynamic fields?  Why did you change the schema?

@mmiklavc The main worry is I'm not honestly sure if that's exactly how it works.  I didn't dig into the caveats or any issues that we might hit with it.  I also don't know what constraints get placed on general polyfields in terms of whether they get returned by that or not.  Assuming it works, it would be fairly clean.   But that'll require a bit of research.

@merrimanr Not sure if it returns dynamic fields.  I didn't see them, but there might be a different way to grab them.  Again, it's a really cursory idea that needs more research if other things fall through (e.g. docValues)
I changed the schema in two ways

To add the ability to make the call
To add the dynamic "*_coordinate" field.  This is (or at least was at some point) required.  Check out https://wiki.apache.org/solr/SpatialSearch#QuickStart.  Assuming this is true, we likely need to add this to all our schemas anyway.

To add onto my last comment, the subfields probably get created right now, but get caught by the "*" field that's type "ignored", which seems wrong.

@justinleet

They may not know, or even be able to easily discover, what fields are poly types and what aren't.

@cestella, @merrimanr

Are date fields polyfields?

I don't know. At this point I am not sure how to determine if a field is a polyfield or not. The isPolyField method is internal and I don't have access to it in Solrj.

I think the proof of @justinleet 's point is made abundantly clear by our very own discussion on the subject.

@merrimanr Re: dynamic fields.  Looks like you can get them in 6.6, but possibly not 5.5
https://lucene.apache.org/solr/6_6_0/solr-solrj/org/apache/solr/client/solrj/response/LukeResponse.html#getDynamicFieldInfo--
http://lucene.apache.org/solr/5_5_2/solr-solrj/org/apache/solr/client/solrj/response/LukeResponse.html#getFieldInfo()
Not sure what it returns, whether it's the actual fields or just what the schema defines though.

@justinleet I tested that and it looks like it's just what the schema defines.  The only way we could use those is if we did a regex max on the actual fields in a document.

Can we get this problem exhibited in the integration tests?  It would help us try potential solutions out easily.

If you add the "-h node1" option to the start command in install_solr.sh you can develop locally against Solr in full dev.  The line ends up looking like:
su $SOLR_USER -c "bin/solr -e cloud -noprompt -h node1"

@merrimanr Ok, that's fine, we can do it on full-dev.  Can you please submit a step-by-step manual testing script to exhibit this issue?  I was asking for an integration test because then it'll be run every build and easy to adjust the code, but this will work too.

I've been changing/uploading schemas in full dev with the Solr CLI and using the Solr Admin console to test.  I can write up with a list of commands that mimic what I'm doing in the console or create a simple integration test.  Which one do you prefer?

Just added an integration test that demonstrates this behavior.  I had to switch from the test schema to the schema that is actually deployed and remove the HBaseDao from this particular test to get it to work.  We will want to expand this test to all types once we agree on a good solution to this problem.  It also might be a good idea to get rid of these test schemas and just use the real ones at some point.

I did some additional testing and here are my notes.  I've attempted to compile a list of all the proposed solutions and their outcomes.

checking the type for isPolyField()

This method is not available in the Solrj client library and would require adding a dependency on solr-core which I don't think we want to do.  Compiling a list of field type to polyfield mappings would need to be something we do outside of runtime and maintain across Solr upgrades.  I did this and found that out of the field types include with Solr 6.6, these are polyfields:

CurrencyField
DateRangeField
LatLonPointSpatialField
LatLonType
PointType
SpatialRecursivePrefixTreeFieldType

I also added any types not currently defined in our schemas and indexed sample data.  Only LatLonType and PointType resulted in expanded fields.

migrate to solr.LatLonPointSpatialField and maintain separate schemas

The solr.LatLonPointSpatialField type worked without issue in Solr 6.6 as expected.  I installed a separate 5.5 instance and indexed the same data.  There is no error thrown but the data ends up looking like this in the index:
{
  "enrichments.geo.ip_dst_addr.location_point_0_coordinate": [
    "34.0438",
    "34.0438"
  ],
  "enrichments.geo.ip_dst_addr.location_point_1_coordinate": [
    "-118.2512",
    "-118.2512"
  ],
  "enrichments.geo.ip_dst_addr.location_point": "34.0438,-118.2512"
}

That does not look correct to me.  If this were to be acceptable what would our approach be for testing and maintaining separate schemas for different versions?  Our code does not compile with version 5.5.

disabling docValues for the spatial field

The docValues setting is already disabled in our schemas.

using LukeRequest to get field type information

This does not handle the case of dynamic fields so we will need to come up with a strategy to map dynamic fields to the field types we care about.  I think we may need to do this regardless of which approach we decide on.  It will be necessary to match dynamic field types to fields that are actually present in a document.
Am I missing any solutions that have been proposed?  Let me know I will investigate.
I also looked into the date field issue I noticed.  Turns out I was wrong and this was actually a result of a copyField being defined.  This is another case we will need to handle.  If a field is a copyField you cannot reindex that field or you will get an error.

Another potential solution I thought of is to only return the intersection of the fields defined in the schema with fields that are present in a document.  This could potentially eliminate the need to keep track of polyfields and make things simpler.  We would still need to handle dynamic fields and copyFields but I think we'll need to do that anyways.  Still thinking through edge cases but let me know if we think this might be an option worth exploring.
Given all this information, where do we go from here?

{
  "enrichments.geo.ip_dst_addr.location_point_0_coordinate": [
    "34.0438",
    "34.0438"
  ],
  "enrichments.geo.ip_dst_addr.location_point_1_coordinate": [
    "-118.2512",
    "-118.2512"
  ],
  "enrichments.geo.ip_dst_addr.location_point": "34.0438,-118.2512"
}

That does not look correct to me.

@merrimanr - can you be more specific? I gather it should not be an array, but simple string value for the derived fields, e.g.
{
  "enrichments.geo.ip_dst_addr.location_point_0_coordinate":"34.0438",
  "enrichments.geo.ip_dst_addr.location_point_1_coordinate":"-118.2512",
  "enrichments.geo.ip_dst_addr.location_point": "34.0438,-118.2512"
}

What was your input doc when doing this test on 5.5? Did you included the expanded fields as part of the insert? To me your results make it look like Solr treated the values as an additive operation vs overwrite, but it's hard for me to say without more detail.

Yes it should be simple string values or, in other words, the doc should look the same minus the changes.  I should have stated this in my previous comment but ideally those *_coordinate fields wouldn't come back at all.
The test case is:

read back a doc that is already indexed
change or add field (not location type)
reindex the doc
read back the doc and verify it is unchanged minus the change in step 2

To answer your question, yes the expanded fields are included in the insert because we are reindexing the whole document.  I agree with you, it looks like Solr treats it as an additive operation.

For the integration test, adding the specific dynamic fields for the underlying LatLon seems to take care of the problem for that type.
e.g. this makes the test you added pass:
  <!-- Catch all, if we don't know about it, it gets dropped. -->
  <dynamicField name="*_coordinate" type="pdouble" indexed="true" stored="false" docValues="false"/>
  <dynamicField name="*" type="ignored" multiValued="false" docValues="true"/>

It appears docValues is not disabled for the subfields, because they get caught by the "*" dynamic fields, so they end up being docValues=true.
My concern would be making sure we document what has to happen for LatLon, and making sure that the other types you laid out can be managed in a similar way (and document appropriately),

Adding this dynamic field worked for me as well.  I tested in a Solr installation as well as the integration test.  I believe this will solve our issue with the older LatLon type.  The only other field type that resulted in
How should we document this?  Comments in our schemas?  We don't have a dedicated document for creating Solr schemas and I don't think we should maintain detailed documentation on the topic.  Maybe we add a section to the Solr README with tips for Solr schema creation as it relates to Metron (would include this issue)?

If we're going to the docValues=false + documentation route,  we should have the documentation be more generic than just the LatLon and Point (e.g. custom types that may get used in more advanced use cases).
I'm also a little surprised that LatLon and Point are the only two types that result in additional fields, given that there were more poly types listed above. Do we know why Currency and other types don't result in additional fields?  Also, is there anything different under the hood about Points vs LatLon?
General questions for people interested

Are we okay with docValues=false and documenting (I personally am, assuming we can provide relatively general guidance, not just something that only works for a couple types).
If LatLon and Point are the only two mainline fields that produce subfields, should we also provide an example of suppressing the subfields of Point, even though it's not in use right now?

Just to clarify, both docValues and stored needs to be set to false.  If one or the other are true then it fails.
At this point I question whether a field being a polyfield is causing this issue.  The testing I've done does not support that.  Javadocs for FieldType.isPolyField() state:

A "polyField" is a FieldType that can produce more than one IndexableField instance for a single value

The issue here is with stored fields because that's what is returned in a search query.  Indexed fields are not.

To give an idea of how I arrived at the conclusion that other fields are not a problem, this is how I tested it.  I will use currency as an example.  First I added this to our bro schema:
<dynamicField name="*.c"   type="currency" indexed="true"  stored="true"/>
<fieldType name="currency" class="solr.CurrencyField" precisionStep="8"
             defaultCurrency="USD" currencyConfig="currency.xml" />

I also added the currency.xml file from the example configs that ship with Solr.  Then I added that field to the integration test added in this PR (SolrUpdateIntegrationTest.suppress_expanded_fields).  It ends up looking like:
@Test
  public void suppress_expanded_fields() throws Exception {
    dao = new MultiIndexDao(createDao());
    dao.init(accessConfig);

    Map<String, Object> fields = new HashMap<>();
    fields.put("guid", "bro_1");
    fields.put("source.type", SENSOR_NAME);
    fields.put("ip_src_port", 8010);
    fields.put("long_field", 10000);
    fields.put("latitude", 48.5839);
    fields.put("score", 10.0);
    fields.put("is_alert", true);
    fields.put("field.location_point", "48.5839,7.7455");
    fields.put("field.c", "1000,EUR");

    Document document = new Document(fields, "bro_1", SENSOR_NAME, 0L);
    dao.update(document, Optional.of(SENSOR_NAME));

    Document indexedDocument = dao.getLatest("bro_1", SENSOR_NAME);

    // assert no extra expanded fields are included
    Assert.assertEquals(9, indexedDocument.getDocument().size());
  }

If we get the same number of fields back that we indexed originally, we know no extra fields are being returned.

A note about the added integration test.  I moved the test to the SolrUpdateIntegrationTest because this issue is specific to Solr.  Adding an additional test exposed a problem in our Solr in memory component where reset wasn't working correctly.  I had to fix that to get all tests to pass.
If verifying this in full dev, the "*_coordinate" fields should no longer be present when searching or looking up documents either through our REST endpoint or the Solr admin console.

Thanks for the thorough reply @justinleet.  Everything you have said makes sense to me.
In my testing I also found copy fields to be an issue if they were set to stored.  Same problem as LatLon types.  Is it worth adding those to our README in this PR as well?

@justinleet - Thanks for that write up - that helps considerably.
Just to round out @merrimanr 's comment about polyfields:

At this point I question whether a field being a polyfield is causing this issue. The testing I've done does not support that. Javadocs for FieldType.isPolyField() state:

A "polyField" is a FieldType that can produce more than one IndexableField instance for a single value

So, from the last few examples discussed it suggests to me that being a polyfield is actually a problem, but it's only part of the total reason for the problem. Under the hood they are setting stored=false and docValues=false for currency and some of the other polyfields, whereas they aren't doing this for LatLonType and Point. Does that sound about right? I saw mention of Date also, and some comments about LatLonType being the only problem data type, so it would be good to summarize again what field types are specifically a problem.
One bit I'm not completely clear on after the rounds of discussion is when it's desirable to return the dynamic/subfields generated by virtue of being a polyfield, and further how the doc update code is managing that differently from a normal search. For a standard doc query in context of performing an update, we don't want dynamically generated fields returned because that will bork the re-index. For a user glob query, we do want then returned because they may be useful to some users? Is that hovering around accurate?

@mmiklavc That sounds right, yes. For returning dynamic fields (at least on the known problematic fields), I don't think it's pretty much ever advantageous. At most it avoids a split on the LatLon result to pull out just latitude or longitude. I think the newer version of geospatial, LatLonPointSpatialField, is in the list not problematic list, so presumably this realization was made and it's handled similar to others.
As an aside, looking through the javadocs, there's this remark on PointFields

FieldCache is not supported for PointFields, so sorting, faceting, etc on these fields require the use of docValues="true" in the schema.
Even though we don't use PointField in our schemas, we may still want to make that note in our docs.  Seems like a fairly important caveat.

We could also potentially, based on https://lucene.apache.org/solr/guide/6_6/docvalues.html#DocValues-RetrievingDocValuesDuringSearch, set useDocValuesAsStored="false", which might sidestep this problem entirely.  The catch is I think there's a whole set of considerations regarding performance and querying that I do not have the expertise to assess.
@merrimanr Are there any circumstances where we'd want copy fields?  It seems like we could just direct users to make a copy of the field in Metron and just index the copy fields directly. The main concern would be if we're able to manage globs in there, but even if we don't it doesn't seem like super important functionality to support.  At least at a glance; I may definitely be missing use cases.

So, from the last few examples discussed it suggests to me that being a polyfield is actually a problem, but it's only part of the total reason for the problem. Under the hood they are setting stored=false and docValues=false for currency and some of the other polyfields, whereas they aren't doing this for LatLonType and Point. Does that sound about right? I saw mention of Date also, and some comments about LatLonType being the only problem data type, so it would be good to summarize again what field types are specifically a problem.

Yes that sounds right to me.  The issue with Date was actually a result of a copy field being defined with stored set to true.  The only types that returned extra fields in my testing were LatLonType and PointType.

One bit I'm not completely clear on after the rounds of discussion is when it's desirable to return the dynamic/subfields generated by virtue of being a polyfield, and further how the doc update code is managing that differently from a normal search. For a standard doc query in context of performing an update, we don't want dynamically generated fields returned because that will bork the re-index. For a user glob query, we do want then returned because they may be useful to some users? Is that hovering around accurate?

The root of the problem is that updates in our DAOs involve reindexing the whole document.  If internal fields are returned in the lookup it will bork the re-index.  With a couple exceptions, document lookups are done with the SolrClient.getById method which I believe returns all fields.  I don't know that there is a case where we want to return these dynamically generated fields.  In fact, I'm now convinced the LatLonType field was just not configured correctly and is an isolated issue.

@merrimanr Are there any circumstances where we'd want copy fields? It seems like we could just direct users to make a copy of the field in Metron and just index the copy fields directly. The main concern would be if we're able to manage globs in there, but even if we don't it doesn't seem like super important functionality to support. At least at a glance; I may definitely be missing use cases.

I believe copy fields is a commonly used feature in Solr.  It can be useful for indexing a value in different ways, with a different chain of analyzers.

@merrimanr Can the copyFields destination be docValues=false?  I'm inclined to just document that requirement (assuming it works), and not worry too much about it until someone actually needs something that can't be done via the main Metron processes?

I think, based on my understanding, that we document around the copy fields limitation.
@cestella @mmiklavc are you both good with the approach of documenting the schema definitions for LatLon and Point along with the restrictions on copy fields?  Are there any outstanding questions or issues you have that need to be addressed? Or new ones to be added?

Latest commit fixes the copy/paste error and adds copyFields documentation to the README.

@justinleet @merrimanr Nope, I'm good.  I'll give my +1 (pending acceptance by the rest of the people who have commented here).

@mmiklavc let me know if the recent commit satisfies your feedback.

Thanks @cestella and @mmiklavc.  Just waiting on a +1 from @justinleet and we'll call it done.

+1 by inspection. Thanks a lot for the work on this.


Skip max_docs query string params

This commit skips max_docs as query string params on DeleteByQuery,UpdateByQuery and ReindexOnServer.
The parameter can also be defined in the request body, so prefer this.

ported to master


[Add] Waterox

Hello,
Its possible to add waterfox ? https://www.waterfox.net/
Thanks

Thank you for the suggestion! On the list in #66


[Error - input.scss] Undefined variable: "$--input-clear-hover-color"

Versions
element-theme: 2.0.1
element-ui: 2.0.7
element-theme-chalk: 2.0.7
Problem
When I run et -i, I get the file element-variables.scss. This file doesn't contain the variable $--input-clear-hover-color. And when I run et, I get the following output:
$>et

\ build element theme

events.js:160
      throw er; // Unhandled 'error' event
      ^
Error: node_modules\element-theme-chalk\src\input.scss
Error: Undefined variable: "$--input-clear-hover-color".
        on line 19 of node_modules/element-theme-chalk/src/input.scss
>>       color: $--input-clear-hover-color;
   -------------^

    at Object.module.exports.renderSync (C:\Users\rvanrent080817\AppData\Local\Yarn\config\global\node_modules\node-sass\lib\index
.js:439:16)
    at DestroyableTransform._transform (C:\Users\rvanrent080817\AppData\Local\Yarn\config\global\node_modules\gulp-sass\index.js:1
57:36)
    at DestroyableTransform.Transform._read (C:\Users\rvanrent080817\AppData\Local\Yarn\config\global\node_modules\readable-stream
\lib\_stream_transform.js:182:10)
    at DestroyableTransform.Transform._write (C:\Users\rvanrent080817\AppData\Local\Yarn\config\global\node_modules\readable-strea
m\lib\_stream_transform.js:170:83)
    at doWrite (C:\Users\rvanrent080817\AppData\Local\Yarn\config\global\node_modules\readable-stream\lib\_stream_writable.js:406:
64)
    at writeOrBuffer (C:\Users\rvanrent080817\AppData\Local\Yarn\config\global\node_modules\readable-stream\lib\_stream_writable.j
s:395:5)
    at DestroyableTransform.Writable.write (C:\Users\rvanrent080817\AppData\Local\Yarn\config\global\node_modules\readable-stream\
lib\_stream_writable.js:322:11)
    at write (C:\Users\rvanrent080817\AppData\Local\Yarn\config\global\node_modules\gulp\node_modules\readable-stream\lib\_stream_
readable.js:623:24)
    at flow (C:\Users\rvanrent080817\AppData\Local\Yarn\config\global\node_modules\gulp\node_modules\readable-stream\lib\_stream_r
eadable.js:632:7)
    at DestroyableTransform.pipeOnReadable (C:\Users\rvanrent080817\AppData\Local\Yarn\config\global\node_modules\gulp\node_module
s\readable-stream\lib\_stream_readable.js:664:5)

How am I supposed to fix this?

I downgraded to<EMAIL_ADDRESS>same issue for $--checkbox-bordered-height missing in th element-variables.scss file. I had to go back to<EMAIL_ADDRESS>to fix all this.

+1

Please verify if node_modules/element-theme-chalk/src/common/var.scss has these variables. If not, make sure you have latest element-theme-chalk installed.

I have the latest version of element-theme-chalk and I do have both variables that element-theme is missing on my file node_modules/element-theme-chalk/src/common/var.scss. Still I get the error mentioned above.

I couldn't reproduce this on my computer. Could you minify your project and upload it to GitHub?

I found the problem for me.
I installed the<EMAIL_ADDRESS>globally, that is why I am missing the variables.
Removing the global package solved the problem.

Not my case:
$ npm -g list --depth=0
C:\Users\<username>\AppData\Roaming\npm
`--<EMAIL_ADDRESS>

@Elfayer
I did as you suggest:
Remove node_modules
Run npm i
Run et -i
Run et
But I still have missing variables while execute et. It should be the problem of file element-variables.scss. It was generated with missing variables by et -i.
And I still don't know how to go over this issue.

@snakepoon Correct, had to update again myself, there is a new issue. :weary:
In my case it seems like element-theme is generating a file with the exact modifications I did before... But I am generating a new file, so where does it take this information from? :tired_face:

@snakepoon I made some testing and it finally works. I am not sure why, can you try the following and tell me if it fixes it for you?
Previous posts not working?
Skip steps that do not concern your case.

Remove folder /node_modules
Remove folder /theme
Run npm i
Run npm i element-theme-chalk -D
Make sure your version of element-ui is the same as element-theme-chalk in your package.json
Rename your file element-variables.scss to _element-variables.scss (to keep your modifications)
Run et -i (to create a new element-variables.scss file)
Replace manually the new file variables to the ones you have in your old file
Run et


STY standardize plotting styles

Closes #415 and closes #256
Here's what I've done:

Capitalize plot titles/legends/axis labels correctly

For titles, legends and labels, capitalize only the first letter of the first word. No other capitalizations unless its a proper noun/acronym (specifically: Sharpe, Bayesian, PnL)

Ensure proper rotation of axis labels

No rotation at all, unless the labels overlap

Standardize legend style (I'm personally a fan of the transparent background-box)

All legends should have frameon=True, framealpha=0.5

Standardize font sizes (might already be done)

Always use matplotlib defaults. Never manually set font sizes.

To-do:

[x] Fix colors. Always use matplotlib named colors or color palettes. At no point should we be using our own hex RGB values.
[x] Run a full tear sheet on an example and actually see if the style actually looks nice.

Notes:

Removed plot_monthly_returns_timeseries, plot_returns_similarity and plot_multistrike_cones from plotting.py. They don't show up in any other module, so I'm not sure what they're doing there. If we actually do need them, I made them separate commits so we can safely revert them.
Also did some minor formatting for readability

Not yet actually ready for review yet btw, just wanted to run a travis build so I'm sure that deleting those 3 functions doesn't actually mess anything up. I'll request reviews when ready :)

Nice! Would you mind posting a couple screenshots of sample plots with the new legend? What's your opinion on how it looks?

I haven't actually tested it out yet! I'll do that once I fix the colors, which will be probably the day after tomorrow.

Also while we're at it, all fill_between should have alpha=0.4. It would be nice to have the edge have a slightly higher alpha value, but that's rather complicated and not for right now.

This looks great. Agree with @gusgordon, should update the Notebooks and see what the changes look like.

Yes, thats a good idea. Unfortunately, the full_tear_sheet_example notebook must be run on Q research environment, since it uses a backtest object to create a full tear sheet. I can't run my PR version of pyfolio on research. Is it possible for someone at Q to do this? I'll take care of the rest.
Can I also suggest deleting the portfolio_volatility_weighted_example notebook? It illustrates portfolio allocation strategies, not how to actually use pyfolio. Academia might find it useful, but it doesn't really belong in pyfolio documentation.

We can just leave that one for now but update the other ones. And yes, agreed about deleting the vol-weighted NB.

FYI: the issue with #256 was not that we are using indistinguishable colors, it is that we are setting alpha to be too low. Someone must have decided to use a low alpha for e.g. the sector allocation plot because there are so many lines being plotted at once.
From now on:
Try not to lower alpha: it defaults to 1.0
If there is a genuine concern that we will be unable to distinguish lines, use alpha=0.6

In the meantime, here are some screenshots of what the newest style looks like:

Also removed 2 pngs from the examples directory. Not sure what they were doing there.

Nice, this looks much better! Ready to merge?

@twiecki yes!
I'm not very experienced with zipline, so I can't fix up the zipline example notebook. I'll file a separate issue for that though, I think there's enough in this PR to merge as it is.

Thanks @eigenfoo!

@eigenfoo We're currently using plot_multistrike_cones in an internal project, should we add it back before https://github.com/quantopian/pyfolio/pull/427?

@yankees714 I don't think that code is used currently? Let's chat offline.


Misc. typo fixes

Found via codespell -q 3 -I ../python-fluids-word-whitelist.txt --skip="./fluids/data" whereby the whitelist consists of:
ans
cas
nd
que
te

Thanks Luzpaz! Really neat tool.


métodos criados e testados.

close #7

Codecov Report

Merging #8 (d19fd7d) into main (e34987e) will increase coverage by 22.43%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##             main       #8       +/-   ##
===========================================
+ Coverage   63.41%   85.84%   +22.43%
===========================================
  Files           3        3
  Lines          41      106       +65
===========================================
+ Hits           26       91       +65
  Misses         15       15

Impacted Files
Coverage Δ

ifrn_estatistica/__init__.py
100.00% <ø> (ø)

ifrn_estatistica/emparn.py
0.00% <ø> (ø)

ifrn_estatistica/descriptive_table.py
97.82% <100.00%> (+5.23%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e34987e...d19fd7d. Read the comment docs.


Bug: JsonDecoder on Map

This works:
@main def main =
  val decoder = summon[Schema[Map[String,String]]]
  println(JsonDecoder.decode(decoder, """{}"""))

But JsonDecoder can't decode "{}" as Map[String, String] in a case class field
case class A(
    a: Map[String, String]
) derives Schema

@main def main =
  val decoder = summon[Schema[A]]
  println(JsonDecoder.decode(decoder, """{"a": {}}"""))

But this works:
case class A(
    a: Map[String, String]
) derives Schema

@main def main =
  val decoder = summon[Schema[A]]
  println(JsonDecoder.decode(decoder, """{"a": []}""")) // note here, [] is valid for maps, i think maybe Map is decode as Seq of Tuple

The reason is field schema of case class if Lazy, it does not match one of int string long

Solved


fix: Minimap panning fix. issue #3625

Fix to issue https://github.com/xyflow/xyflow/issues/3625

Thanks @mirisony we will check it!

Thanks! Works better for larger flows now :)


Replace deprecated package with kubevirt package in rest of the file

Replaced the deprecated package k8s.io/utils/pointer with kubevirt package kubevirt.io/kubevirt/pkg/pointer in the rest of the file.

What this PR does
Before this PR:

In some remaining file  we used k8s.io/utils/pointer package which is deprecated.

After this PR:

Now, it is replaced with kubevirt package kubevirt.io/kubevirt/pkg/pointer

fixes #12949
Why we need it and why it was done in this way
The following tradeoffs were made:
The following alternatives were considered:
Links to places where the discussion took place:
Special notes for your reviewer

Checklist
This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

[ ] Design: A design document was considered and is present (link) or not required
[ ] PR: The PR description is expressive enough and will help future contributors
[ ] Code: Write code that humans can understand and Keep it simple
[ ] Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
[ ] Upgrade: Impact of this change on upgrade flows was considered and addressed if required
[ ] Testing: New code requires new unit tests. New features and bug fixes require at least on e2e test
[ ] Documentation: A user-guide update was considered and is present (link) or not required. You want a user-guide update if it's a user facing feature / API change.
[ ] Community: Announcement to kubevirt-dev was considered

Release note

None

Could you pleasee take a look ?

@fossedihelm could you please take a look

@anishbista60 I think something went wrong w/ the rebase.
I'd expect to not see any change in the /staging folder.
Thank you

There is not problem with rebasing. Should I leave /staging ?

Exactly. /staging cannot be addressed with these changes

/test al

/test all

/lgtm

@alicefr could you please take a look ?

/approve


Sync .github/workflows directory with azure-sdk-tools for PR 8566

Sync .github/workflows directory with azure-sdk-tools for PR https://github.com/Azure/azure-sdk-tools/pull/8566  See eng/common workflow

/check-enforcer override

/check-enforcer override


Issue22

Transition Travis builds to use tox and pytest instead of having a separate build process maintained in parallel.
Closes issue 22

Coverage decreased (-0.7%) to 98.707% when pulling 73d3348134063d74a4db6e2f5c0493a1f4a93d20 on issue22 into 03f7f0530189d07df8ab6cfe7cfc266f9910f2f9 on master.


how to use listStatusChanges to get envelopes that were sent to me?

I can only see the envelope status of envelopes I sent. What option is needed to include all the envelopes in the inbox?

envelopesApi.listDocuments is the call you are looking for. Please refer to the listDocuments unit test in SdkUnitTests for a sample of its usage


[Radio] Accept number value

[x] I have searched the issues of this repository and believe that this is not a duplicate.

Expected Behavior
I would have expected to be able to use value={100} inside a RadioGroup and FormControlLabel.
The docs state, that value is a string https://material-ui-1dab0.firebaseapp.com/api/radio-group/ https://material-ui-1dab0.firebaseapp.com/api/form-control-label/ , but I wonder if this is on purpose or accident.
Current Behavior
It actually works just fine, but throws type errors in the console.
If you also think that value should accept numbers, i'd be happy to create a pr.
Best Regards,
lukas

I agree, I think that we could be allowing a number too. This can be serialized in the native input.

Just for people stepping over this:
Using numbers as radio values will be problematic. Even when you provide a number as value the native onChange will return a stringified version of that which will then fail in the value === value comparison for checked state (you know because "123" !=== 123). So you would have to use Number.parseInt(value) onChange to set a valid state which also seems error prone.

The value is inlined into a native input. I don't think that we can do anything about it. You would have the same behavior without Material-UI.

tl;dr;
Just don't use numbers as values for radios. You're probably better off without them.

What should I do, If I need to pass bool to value, cause I need by default initialize the value for radio with redux-form. In my case I use react16, reduz-form, redux
My component
<Field  style={{paddingLeft: 20}} name='activationOutputDoor' component={renderRadioGroup}>
                            <Radio
                                value='false'
                                label={<FormattedHTMLMessage id='passage.activateOutputClosesDoor'/>}
                                disabled={!enableControlDoor|| !enableOutputSettings}
                            />,
                            <Radio
                                value='true'
                                label={<FormattedHTMLMessage id='passage.activateOutputOpensDoor'/>}
                                disabled={!enableControlDoor|| !enableOutpu />

When I inisialized this value in state
export const mapStateToProps = (state, {match: {params: {deviceId}}}) => {
    let initialValues = deviceId && state.entities.devices[deviceId]
    if (!initialValues)
        initialValues = {
            standAloneMode: true,
        }
    return {
        initialValues,
        deviceId,
        form: `device-${deviceId}`,
    }
}

In my case, if I pass to value bool true or false one value Radio will be checked as well, bu to choose the second on Value is not possible. How to pass bool to Radio value?

@palaniichukdmytro I just came here for the exactly same reason you did, I wanted to pass bool values in there. My actions shows the payload as {type: "@@redux-form/CHANGE", meta: {…}, payload: "true"}
I guess we could convert the data as @sakulstra wrote, before you update your store. But then again, why? I think keeping string is actually a good option here. I will convert my data later on, when I need to call an api with bool values.


Capi-ml-common is depending on capi-base-common

in the capi-ml-common pc file we should be requiring capi-base-common and our meson should not directly require capi-base-common

:octocat: cibot: Thank you for posting issue #57. The person in charge will reply soon.


Unescape all exception messages

As to avoid parsing error conditions in Grafana logs.

Detected two unrelated issues:

latest version of witnet-toolkit binary is not compatible w/ container's base distro
bridge jumps over new queries if an rpc provider error arises while getting the query's bytecode, whereas it should jump over only if an evm revert is produced, and retry as many times as required otherwise

* latest version of witnet-toolkit binary is not compatible w/ container's base distro

Fixed in #58

* bridge jumps over new queries if an rpc provider error arises while getting the query's bytecode, whereas it should jump over only if an evm revert is produced, and retry as many times as required otherwise

That'll be https://github.com/witnet/witnet-rust/issues/2388. Working on it!

New base docker image included within fix #58, works fine and smoothly.

Related PRs:

https://github.com/witnet/witnet-price-feeds-poller/pull/59

Updating all testnets to use:

Bridge v1.6.6
Poller v2.0.5


Add Simplified Chinese translation

Summary of the Pull Request
Add Simplified Chinese (zh-cn) translation.
PR Checklist

[x] Applies to #179
[x] CLA signed. If not, go over here and sign the CLA
[ ] Tests added/passed
[ ] Requires documentation to be updated
[ ] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx

Info on Pull Request
What does this include?
Validation Steps Performed
How does someone test & validate?

@olingCat, while greatly appreciated but we're currently going through our internal translation systems as we've discussed in issue #199.  This should be starting to bubble into builds for 0.23 and 0.24.
Reason why we like this route is we can stay up-to-date when we do verbiage tweaks and add in new features across all languages and it is all built into our build pipeline.
Once we have our translations done, if there are small tweaks needed, we'll happily accept the adjustments.

@crutkas OK, thank you.
I'll help to improve the coming translation. 😜

@crutkas
I'm complete the Simplified Chinese translation in 3rili@e600e4b.
So, if someone is working on the Chinese translation, they can refer this translation.

look forward to the translation system ,i can help for the translation too


Regexp injection

This:
PasswordBlacklist::Checker.new.blacklisted? ')'

will cause a
RegexpError: unmatched close parenthesis: /)/

Similarly
PasswordBlacklist::Checker.new.blacklisted? '.*'

Always returns true because of it is not interpreted as a verbatim string.
Could you please use
/#{Regexp.quote(password.to_s)}/

In your check?
Thank you!

This has been fixed! Thanks for reporting the issue!

That was quick :) Thank you!


[Feature] - Saving transcription as a file.

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
I just want to have an option of saving the transcription as a file in my server instead of uploading it, since when it tries to upload the transcription, it always lags out, causing duplicate channel being created and making the bot not responsive during that period.
Describe the solution you'd like
Basically in src/utils/close.ts, add another function where if the configured function in config.jsonc for saveTranscriptionAsFile: true, it will save the file in the bot installation location folder, or it can be configured via transcriptionFilePath: /your/path/here
Describe alternatives you've considered
By using https://github.com/ItzDerock/discord-html-transcripts library, those feature can be implemented easily.
Additional context
Forcing the self-hoster to upload transcription when its enabled or by option when clicking Save Transcription button is not ideal as there is no option to save it in the hosting location. Also, instead of relying on upload, transcription can be saved easily and faster.

Uploading transcripts to a server is the best way to make them accessible to everyone (PC, Mac, smartphone, tablet). What's more, uploading transcripts to a remote server doesn't slow down ticket closing, as the file uploaded is only about ten KB, excluding attachments. However, you say that when you close tickets there are problems, and I can help you with that.

idk, I feel like uploading an HTML file as discord attachment wouldn't be that bad and even helps you save bandwidth. The only downside is that user will now need to download the file to access it oppose to direct access via a link.

I usually use transcript as a log for every ticket, instead of making it available for users. Hence having this feature is necessary as there is multiple hierarchy in a team and the one that read those transcript are definitely not for normal people. But I guess I'll have to modify it myself as its not planned nor will be made in the future.

I don't need transcripts to be accessible to everyone, but I'd like to log them to a file in case I need to look at them again for some reason. I don't want to upload them to a third party site. Is there any option for this yet?

@Sayrix I think we should consider adding this to the re-write. It just makes more sense to allow more advanced user to control how the transcript is provided.
Since we're in the process of re-write, we can just add this to the queue.

Transcripts are saved for 365 days, it's only the media which is saved for 90 days with sponsor


Update to address Issue #14 and Issue #23

See issue comments for description.

Should be fixed in release 1.3.2.


Thank you for your great work
I want to run the inference demo by using my own images. Could you please tell me how I can do that?

Hi! If you are talking about EPro-PnP-Det, you can run the command python demo/infer_imgs.py -h to view the available options. Note that camera intrinsic matrix is required to run the inference, and the camera should have zero pitch/roll angles relative to the ground.

thank you very much!


Get javadocs implemented with standard Flutter website style.

Get javadocs implemented with standard Flutter website style.

CC @xster for visibility. The JavaDoc pages are standard JavaDoc style, which feels completely unrelated to the main Flutter docs.

This is about finding some css theme for javadocs that look closer to the dartdoc sites right?

I think this should be more than just styling. I think we should migrate the docs so that they are a part of the standard flutter.dev website. In other words, I don't think the javadocs should be located in a fundamentally different place, or feel fundamentally different than Dart docs and any other docs that are core to Flutter. I don't want to be too prescriptive because there may be other concerns for us to weigh here, but generally speaking, the closer the JavaDocs are to DartDocs in every way, the better.

BTW, this is what we have now: https://api.flutter.dev/javadoc/

Do you mean also finding FlutterActivity when searching from https://api.flutter.dev?

Being able to search javadocs from the main Flutter website would certainly be one expectation, but that's just one detail. I would expect similar styling of the docs (but not necessarily identical). I would expect Flutter's main website navigation to surround the javadocs. At this point, it would be a shorter list to say what our current javadocs have correct in terms of styling and the surrounding website.

not sure if this is still valuable

@reidbaker what's up?!


[Bug]: S220 cameras reported as not compatible?

What happened?
I get the following warning for all four of my S220 cameras on a HomeBase 3:
"WARN This accessory is not compatible with HomeBridge Eufy Security plugin: <camera_name> Type : 63"
When I go to bropat's supported devices page (https://bropat.github.io/eufy-security-client/#/supported_devices), I can see that the S220 has a green checkmark next to.
Device Type
Floodlight (Supported)
Plugin Version
v2.0.x (Supported)
HomeBridge Version
v1.6.x (Supported)
NodeJS Version
other (UnSupported)
Operating System
Raspbian 10+ (Supported)
Relevant log output
[02/16/2024, 13:10:58]  [EufySecurity]  INFO    Updating existing accessory: HomeBase3
[02/16/2024, 13:11:01]  [EufySecurity]  WARN    This accessory is not compatible with HomeBridge Eufy Security plugin: South Sideyard Type: 63
[02/16/2024, 13:11:01]  [EufySecurity]  WARN    This accessory is not compatible with HomeBridge Eufy Security plugin: Driveway Type: 63
[02/16/2024, 13:11:01]  [EufySecurity]  WARN    This accessory is not compatible with HomeBridge Eufy Security plugin: North Sideyard Type: 63
[02/16/2024, 13:11:01]  [EufySecurity]  WARN    This accessory is not compatible with HomeBridge Eufy Security plugin: Front Yard Type: 63
[02/16/2024, 13:11:38]  [EufySecurity]  INFO    Looking for old cached accessories that seem to be outdated...
[02/16/2024, 13:11:38]  [EufySecurity]  INFO    No outdated cached accessories found.

config
{
    "username": "*redacted*",
    "password": "*redacted*",
    "country": "US",
    "deviceName": "HomeBase2",
    "platform": "EufySecurity",
    "ignoreStations": [],
    "ignoreDevices": [],
    "cleanCache": true,
    "cameras": [
        {
            "serialNumber": "T813******81",
            "rtsp": false,
            "talkback": false
        }
    ]
}

i have the same issue

I have the same issue. I hope for a good integration of the S220 in this plugin. It's the best plugin for me. Nice job.

Seem like someone added support. Not sure how to install his version though.
https://github.com/homebridge-eufy-security/plugin/pull/511


ehancement Stats preheating as FE booted

Proposed changes

Support prefetch some column stats when FE booted, it would load column stats that was got updated recently according to the comment of PR #18460 from @morrySnow
Refactor stats cache, split histogram cache from column stats, so that we could avoid some redundant query for column statistics table,for example, update the histogram or column stats only, in the previous implementation a united cache loader would send query request to both column stats table and histogram table,
Extract some common logic to StatsUtil
Remove some useless codes in unit tests, those codes is hard to maintaince and it's not a good idea for testing the accurracy of stats estimation according to the advise from @englefly
Add field type restriction when create analysis tasks to avoid unnecessary failure

Problem summary
Describe your changes.
Checklist(Required)

[ ] Does it affect the original behavior
[ ] Has unit tests been added
[ ] Has document been added or modified
[ ] Does it need to update dependencies
[ ] Is this PR support rollback (If NO, please explain WHY)

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

run buildall

run buildall

run buildall


Disclosure Accessibility isues: missing some or all of the required children

Hi, I've noticed this error pop up when running a lighthouse report on the disclosure component example:
Elements with an ARIA [role] that require children to contain a specific [role] are missing some or all of those required children.

It's probably because of the role="menu" that is being applied by the panel, but I'm not sure which other children are required.

Thanks, I'll look into it - my initial reaction is that the menu role is probably being set when it shouldn't be.

Yes, that was it - thanks again for the report!


Model Runner

Support model operation JIT in controller
Scope: function type controller

Outstanding:

[ ] Many to many
[ ] Relation builder improvement


[BUG]: commaize not working in RN Android

Package Scope

Package name<EMAIL_ADDRESS>Describe the bug

commaizeNumber didn't work at Android which written by RN.
The function returns String(value).replace(/(\d)(?=(\d\d\d)+(?!\d))/g, '$1,') when input argument value's type is string, but, it's weired that convert string type variants to string.
Expected behavior

Here is my code.
const SOME_NUMBER = 1000000;

return (
    <Text>{commaizeNumber(SOME_NUMBER}/<Text>
);

I expected it works as same in Android and iOS, but it didn't work at Android.

iOS case

Android case

It works when I change code above like below.
const SOME_NUMBER = 1000000;

return (
    <Text>{commaizeNumber(String(SOME_NUMBER)}/<Text>
);

To Reproduce

Change commaizeNumber functions as below. Delete unnecessary type conversion. Converting value only when value type is number.
export function commaizeNumber(value: string | number) {
  if (typeof value === 'number') {
    return String(value).replace(/(\d)(?=(\d\d\d)+(?!\d))/g, '$1,');
  }

  return value.replace(/(\d)(?=(\d\d\d)+(?!\d))/g, '$1,');
}

Possible Solution

I made PR to solve this issue #104 .
Additional context

closed with #104


Allow creating an Expression<Func<>> instead of a IQueryable Projection

It is possible to get a IQueryable<B> ProjectToB(IQueryable<A> query) which can be used as
context.ASet.ProjectToB().ToListAsync();

Is your feature request related to a problem? Please describe.
We have a generic class like follows:
class BaseRepository<TEntity, TModel>
{
    public BaseRepository(DbSet<TEntity> dbSet, Func<IQueryable<TEntity>, IQueryable<TModel>> projectToModel) { ... }

    public async Task<List<TModel>> GetAll()
    {
        return await _projectToModel(
                _dbSet.Where(x => x.SomeCondition)
            )
            .Where(x => x.IsSomething)
            .ToListAsync();
    }
}

class UserRepository : BaseRepository<UserEntity, UserModel>
{
    public UserRepository(DbContext c) : base(c.Users, UserMapper.ProjectToModel) {}
}

static partial class UserMapper
{
    public static partial IQueryable<UserModel> ProjectToModel(this IQueryable<UserEntity> q);
}

Describe the solution you'd like
I would prefer being able to create an Expression<Func<A, B>> and give it to my base repository.
class BaseRepository<TEntity, TModel>
{
    public BaseRepository(DbSet<TEntity> dbSet, Expression<Func<TEntity, TModel>> projectToModel) { ... }

    public async Task<List<TModel>> GetAll()
    {
        return await _dbSet
            .Where(x => x.SomeCondition)
            .Select(_projectToModel)
            .Where(x => x.IsSomething)
            .ToListAsync();
    }
}

class UserRepository : BaseRepository<UserEntity, UserModel>
{
    public UserRepository(DbContext c) : base(c.Users, UserMapper.CreateToModelProjection()) {}
}

static partial class UserMapper
{
    public static partial Expression<Func<UserModel, UserEntity>> CreateToModelProjection()
}

This will make the query cleaner and easier to read.
Describe alternatives you've considered
The aforementioned code works, but I'd prefer the new variant.
Additional context
I'm happy to hear your opinions on this matter.

I understand your use case and I think this could probably be helpful in other scenarios. However, as I think this is an edge case, it is not a priority for us. But I would be happy to accept a PR that implements this.

I'd be happy to do this, should be easy to add 🤞

nvm might be a bit trickier than I thought, sorry

The Expression must already be created in order to use IQueryable<>, so I figured it might not be too difficult. If I find some free time this month I might dive into the code

Yeah, I think the logic is identical to IQueryable. I'd naively thought it would be a quick, simple check for expressions and then write some tests.
I hadn't registered that I'd need to edit the method extractor, maybe add a new object method builder and maybe change an interface.
It's possible to add, but not a quick 15 minute change

I started experimenting on this. I stumbled on some assumptions that currently seem to be baked in, but so far nothing major; in fact with some hacks I managed to get it somewhat working:
//HintName: Mapper.g.cs
// <auto-generated />
#nullable enable
public partial class Mapper
{
    private partial global::System.Linq.IQueryable<global::B> Map(global::System.Linq.IQueryable<global::A> source)
    {
        return System.Linq.Queryable.Select(source, MapToExpression(42));
    }

    private global::System.Linq.Expressions.Expression<global::System.Func<global::A, global::B>> MapToExpression(int source)
    {
#nullable disable
        return x => new global::B()
        {
            Parent = x.Parent != null ? new global::B() : default,
        };
#nullable enable
    }
}

I'll try experimenting some more, but I would like to know if this direction looks somewhat reasonable. I am aiming for something like:
//HintName: Mapper.g.cs
// <auto-generated />
#nullable enable
public partial class Mapper
{
    private partial global::System.Linq.IQueryable<global::B> Map(global::System.Linq.IQueryable<global::A> source)
    {
        return System.Linq.Queryable.Select(source, MapToExpression());
    }

    private global::System.Linq.Expressions.Expression<global::System.Func<global::A, global::B>> MapToExpression()
    {
#nullable disable
        return x => new global::B()
        {
            Parent = x.Parent != null ? new global::B() : default,
        };
#nullable enable
    }
}

(aka no hacked source). Later on I would like to memoize the expression, but that might not be really relevant (it would be a minor optimization for the allocations, definitely optional).

@ranma42 looks good so far 😊 Thank you for your efforts.
IMO the externalized expression method only needs to be generated if there is a partial method definition for it:
[Mapper]
public partial class Mapper
{
    public partial IQueryable<B> Map(this IQueryable<A> source);
}

should generate
[Mapper]
public partial class Mapper
{
    public partial global::System.Linq.IQueryable<global::B> Map(this global::System.Linq.IQueryable<global::A> source)
    {
#nullable disable
          return System.Linq.Queryable.Select(source, x => ...);
#nullable enable
    }
}

but
[Mapper]
public partial class Mapper
{
    public partial Expression<Func<A, B>> Mapper();
}

should generate
[Mapper]
public partial class Mapper
{
    public partial Expression<Func<A, B>> Mapper()
    {
#nullable disable
          return x => ...;
#nullable enable
    }
}

If both are declared, the queryable method could call the expression builder, but IMO this would be optional for a first version and could be an optimisation later on.


Error when you click run

Describe the bug
Error
To Reproduce
Steps to reproduce the behavior:

When you click run

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots

Platform
Windows 10 - PT-BR

Thank's for the report!
I'm currently debugging this ! A fix will soon be released

Thank you

This is fixed ! A release is coming soon


Plularization of the queries

English language as many others have plural versions.
What it means that findTask is different than findTasks operation can be come
find Object [s,ss,sh] etc. Having plurals brings confusion to the spec - especially when developers dealing with models that might not be easily pluralized.
Current implementations in javascript using pluralize library that has some specific rules but other languages will have different ones.
We are hitting this issue when building client side queries for graphback.
Solution:

Do not use plural form getTask and task - as the plural query.
Always add s even if the form will not be valid
Others?

@machi1990 @craicoverflow This is something I have totally overlooked. What would be your take?

Solution:

Do not use plural form getTask and task - as the plural query.

+1 on this solution.

find Object [s,ss,sh] etc. Having plurals brings confusion to the spec - especially when developers dealing with models that might not be easily pluralized.

This is the main concern I had with pluralising.

English language as many others have plural versions.

Wait till you hear the different varieties there in French language.

I was talking @ralfhandl the other day about how OData deals with this and there is an explicit definition of the model - model can be singleton or array - In case of the graphql crud we assume that everything is array and there are no singletons, but the benefit of splitting model definition and it's the actual name in query, update thing can be beneficial.
This is also what @danielrearden mentioned to me - names should be controllable - so we do not have any artificial variants that differ only by name.
However I find it hard to reflect plural in SDL. Would something like this make sense:
type User @instance(name: "Users") {
}

Seems overkill to me

Another crazy idea. Instead of dealing with names we can specify operations and mappings - and even build specific implementation for the language of the choice. This will mean that spec itself will abstract from the name - what you will get is just crud operations with certain capabilities and names itself would be coming from individual implementation.
This will allow us to define mappings to various different solutions - we can map (and transform) from Prisma, Hasura, AppSync in seconds - or throw errors if someone using capability that other platform is not using.

I don't see this as an issue with GraphQLCRUD. I see this as an issue with how it is implemented in Graphback.
"""
@model(name: "Utilisateurs")
"""

Removing all pluralization will become confusing for users of the API:
findTask(...)
getTask(...)

If above were to happen, it is not obvious what the difference between both queries are.
Pluralization is very important in identifying the action of a resolver/method. If everything is singular the schema API becomes less clear.

@craicoverflow but following your comment. How do you see us to remove ambiguity in spec.
We are building now forms solution that works with GraphQL CRUD. How we can build queries that will be compatible without using Graphback. Do you see Graphback being used as source for generating schema everywhere?

FWIW, some APIs (like Hasura's) don't bother with the distinction between singular and plural at all.
If you did drop the pluralization, something like listTask vs taskById might be sufficiently clear. Just my two cents.

WRT Graphback specifically, you could add a plugin config option for an operation name template map with some reasonable defaults
operationNames:
  findById: '${MODEL}ById'
  findAll: 'find${MODEL}'
  # create, update, etc.

I think we want to go above graphback here. Graphback (or more specifically GraphQL-serve) is a tool that allows people to print out compatible schema, but if you as the creator will need to adjust your own tool to be compatible just by names that will be bummer - Introducing breaking changes to users.
I will check how we can do this mapping - in both documentation and tooling and see if that is doable

WRT Graphback specifically, you could add a plugin config option for an operation name template map with some reasonable defaults

Great idea. We need here short term plan and long term. For the short term, I propose non pluralized names. For the long term, we will introduce concept of features and how they are composed in queries which will allow us to map any names used for queries in implementations like AppSync, Hasura, Prisma etc.
Any opinions?

For the short term, I propose non pluralized names.

For the short term this would mean completely changing the semantic structure of the query names so that they make sense.
Simply dropping plurality from the existing names mean that they will single v plural queries become indistinguishable, eg: getTask v findTask.
So as @danielrearden suggested a way to distinguish these is to use something like taskList over findTasks.

WRT Graphback specifically, you could add a plugin config option for an operation name template map with some reasonable defaults

Great idea!
We need here short term plan and long term.
For the short term, I propose non pluralized names.
For the long term, we will introduce the concept of features and how they are composed in queries which will allow us to map any names used for queries in implementations like AppSync, Hasura, Prisma etc.
Any opinions?

The short term solution should be easy to map and possibly the default one once the long term solution is there.
To avoid ambiguities of the queries, I second what @craicoverflow @danielrearden proposed, instead of findTask, we can have listTask.

After today's call, taskList seems to fit best. It makes sense from a grammatical point of view - it is never going to change depending on the word used (Person/People).

Perfect! This change would be really beneficial.


[GHSA-mw9h-hcp7-fgc6] Exposure of Sensitive Information in OPCFoundation.NetStandard.Opc.Ua.Server

Updates

CVSS
Severity

Comments
CVSS string was incorrectly copied? score is 5.3, not 7.5

Looks like the CVSS we used came in from NIST on this one
https://nvd.nist.gov/vuln/detail/CVE-2022-33916#VulnChangeHistorySection


Ctrl-C Overwrites Multiple Line Inputs and Hisotry Search

Pretty like #35, There are two Ctrl-C overwrite issues.
(III means the cursor position)

when cancel an input:

$ echo III you typed a long long
line command, and decide to cancel with Ctrl-C
# Ctrl-C makes prompt overwrite `line command, ...` line.

and 2. when searching history with Ctrl-R:
(reverse-i-search)`curl': curl III -i http://a-very-very-
long-url.com/which-wraps-lines/
# Ctrl-C also overwrite the second line

I'm working on a big set of changes right now and, while the solution to this issue is fairly small, it does touch a few pieces of code that are being moved around, so I'm going to hold off until that big changeset lands to save myself further merge headaches. Hopefully, I don't take too long to finish up the big changes.
You can leave this issue open until then.


Add mtk.jl for AutoModelingToolkit as AD backend support

This PR moves implementation of AutoModelingToolkit to similar interface as the other AD backends and also adds support for constraints.

Unfortunately with the constraints support there's a dependency on https://github.com/SciML/ModelingToolkit.jl/pull/1563 and that depends on this PR..


MINGW MSYS2 and CLANG64 Could NOT find Libev (missing: LIBEV_LIBRARY LIBEV_INCLUDE_DIR)

MINGW MSYS2 and CLANG64 Could NOT find Libev (missing: LIBEV_LIBRARY LIBEV_INCLUDE_DIR)
$ CC=clang CXX=clang++ cmake . --fresh
-- Building for: Ninja
CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

-- The C compiler identification is Clang 18.1.5
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/msys64/clang64/bin/clang.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
CMake Error at C:/msys64/clang64/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (messag
e):
  Could NOT find Libev (missing: LIBEV_LIBRARY LIBEV_INCLUDE_DIR)
Call Stack (most recent call first):
  C:/msys64/clang64/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESS
AGE)
  cmake/Modules/FindLibev.cmake:25 (find_package_handle_standard_args)
  src/CMakeLists.txt:7 (find_package)

-- Configuring incomplete, errors occurred!

我也遇到了此问题


For UiTPAS and Paspartoe is always shown when query is "uitpas=1" on agenda/search

When the query in the url is "uitpas=1" on agenda/search, the active filters block always shows "For UiTPAS and Paspartoe". Despite when you change the value in the filter-form block

Fixed with this commit a68ec7315d6cf5209be7b38008cdc912c042a832


Threat generation does not work - results.map is not a function

Describe the bug
Clicking on "+ STRIDE per element..." in the manage threat section does not generate any threat. Happens on Windows and MacOS with version 1.5.0.
Expected behaviour
Expected to have some threats automatically generated.
Environment

Version: 1.5.0
Platform: Desktop App
OS: MacOS  & Windows
Browser: -

To Reproduce

Open Demo model
Open the example diagram
Select "STRIDE" as model.
Add a new element and select it
Click "+ STRIDE per element..." in the "Manage Threats" section

Any additional context, screenshots, etc
Error message from the developer tools console on MacOS (same error also occurs on Windows):
/Applications/OWASP-Threat-Dragon.app/Contents/Resources/app.asar/core/services/threatengine.js:34 Uncaught (in promise) TypeError: results.map is not a function
at onCompleted (/Applications/OWASP-Threat-Dragon.app/Contents/Resources/app.asar/core/services/threatengine.js:34)
onCompleted @ /Applications/OWASP-Threat-Dragon.app/Contents/Resources/app.asar/core/services/threatengine.js:34
Promise.then (async)
generateThreats @ /Applications/OWASP-Threat-Dragon.app/Contents/Resources/app.asar/core/diagrams/diagram.js:235
scope.onSuggestThreats @ /Applications/OWASP-Threat-Dragon.app/Contents/Resources/app.asar/core/diagrams/elementpropdirectives.js:88
(anonymous) @ /Applications/OWASP-Threat-Dragon.app/Contents/Resources/app.asar/node_modules/angular/angular.js:17073
callback @ /Applications/OWASP-Threat-Dragon.app/Contents/Resources/app.asar/node_modules/angular/angular.js:29123
$eval @ /Applications/OWASP-Threat-Dragon.app/Contents/Resources/app.asar/node_modules/angular/angular.js:19523
$apply @ /Applications/OWASP-Threat-Dragon.app/Contents/Resources/app.asar/node_modules/angular/angular.js:19622
(anonymous) @ /Applications/OWASP-Threat-Dragon.app/Contents/Resources/app.asar/node_modules/angular/angular.js:29127
defaultHandlerWrapper @ /Applications/OWASP-Threat-Dragon.app/Contents/Resources/app.asar/node_modules/angular/angular.js:3891
eventHandler @ /Applications/OWASP-Threat-Dragon.app/Contents/Resources/app.asar/node_modules/angular/angular.js:3879

Thank you for the report!
Interestingly, I found that a couple of unit tests are failing with this same issue, but only when run on windows, not on linux.

hmmm, yes, I updated  json-rules-engine and at the same time I broke the unit tests so did not spot this
json-rules-engine has gone from 3.1.0 to 6.1.2 and I have not tested this breaking change properly - will look for a fix

I find it interesting that it was passing on linux though... It makes me wonder about some of the tests.  I've been digging into them a bit and we're using at least one or two extremely out of date and/or unmaintained libraries for testing.  Probably due for an overhaul at some point in the near future. :)
Let me know what I can do to help, @jgadsden !

Well, the fix is really easy. Line 34 in threatengine.js changes from
return results.map(function(result) {
to
return results.events.map(function(result) {
However json-rules-engine has also changed the path condition helper, so that needs a fix up as well ...

Thank you for the fast fix @jgadsden. Will there be a new release of 1.5.0 with that fix included? I think it is an essential feature for creating threat models.

We have the fixes in place, and we expect a new version 1.5.1 to be released in the next few days.
@lreading is updating the release process, so that snaps are signed, Windows NSIS is signed and the MacOS image is notarized ... nearly there oin all of this

@uncatchable-de - Version 1.5.3 has been released, which contains the fix for this issue.,  Thanks again for reporting this!


No validation on the RemoteReadSpec fields

What happened?
I set this remoteread detail in the prometheus CR
  remoteRead:
    - url: http://prom1:9090/api/v1/read
      read_recent: true
      name: prom1
    - url: http://prom2:9090/api/v1/read
      read_recent: true
      name: prom2

and successfully create the prometheus CR.
Did you expect to see something different?
I would expect that if I set a key that is not valid for remoteRead spec the creation should fail.
How to reproduce it (as minimally and precisely as possible):
Create prometheus CR using the remoteRead above.
Environment
KGE k8s 1.15, prometheus operator latest which uses prometheus v2.15.2.

Prometheus Operator version:
install it from stable helm chart https://github.com/helm/charts/tree/master/stable/prometheus-operator

quay.io/coreos/prometheus-operator:v0.37.0
+ kubectl describe pods chart1-prometheus-operator-operator-9dbc95d66-hhm7n
Name:           chart1-prometheus-operator-operator-9dbc95d66-hhm7n
Namespace:      default
Priority:       0
Node:           gke-cluster-shay-pool-1-9e99da38-fx13/<IP_ADDRESS>
Start Time:     Tue, 28 Apr 2020 10:18:01 +0300
Labels:         app=prometheus-operator-operator
                chart=prometheus-operator-8.12.12
                heritage=Helm
                pod-template-hash=9dbc95d66
                release=chart1
Annotations:    kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container prometheus-operator; cpu request for container tls-proxy
Status:         Running
IP:             <IP_ADDRESS>
IPs:            <none>
Controlled By:  ReplicaSet/chart1-prometheus-operator-operator-9dbc95d66
Containers:
  prometheus-operator:
    Container ID:  docker://fa9978f5137c5bca74dbb649874d21634445e8ab3c19a738c923fd8758abc4d0
    Image:         quay.io/coreos/prometheus-operator:v0.37.0
    Image ID:      docker-pullable://quay.io/coreos/prometheus-operator@sha256:51c4b3180aa4cb819ae918da49776dba9a17e11d5a5d1eb22d878e1141aa23c9
    Port:          8080/TCP
    Host Port:     0/TCP
    Args:
      --manage-crds=true
      --kubelet-service=kube-system/chart1-prometheus-operator-kubelet
      --logtostderr=true
      --localhost=<IP_ADDRESS>
      --prometheus-config-reloader=quay.io/coreos/prometheus-config-reloader:v0.37.0
      --config-reloader-image=quay.io/coreos/configmap-reload:v0.0.1
      --config-reloader-cpu=100m
      --config-reloader-memory=25Mi
    State:          Running
      Started:      Tue, 28 Apr 2020 10:18:05 +0300
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        100m
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from chart1-prometheus-operator-operator-token-lsvdv (ro)
  tls-proxy:
    Container ID:  docker://5140e6a046cec5e3692b2db15571086552831e1551854f0bbcf37876d77e05d2
    Image:         squareup/ghostunnel:v1.5.2
    Image ID:      docker-pullable://squareup/ghostunnel@sha256:70f4cf270425dee074f49626ec63fc96e6712e9c0eedf127e7254e8132d25063
    Port:          8443/TCP
    Host Port:     0/TCP
    Args:
      server
      --listen=:8443
      --target=<IP_ADDRESS>:8080
      --key=cert/key
      --cert=cert/cert
      --disable-authentication
    State:          Running
      Started:      Tue, 28 Apr 2020 10:18:08 +0300
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        100m
    Environment:  <none>
    Mounts:
      /cert from tls-proxy-secret (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from chart1-prometheus-operator-operator-token-lsvdv (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  tls-proxy-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  chart1-prometheus-operator-admission
    Optional:    false
  chart1-prometheus-operator-operator-token-lsvdv:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  chart1-prometheus-operator-operator-token-lsvdv
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

Kubernetes version information:
kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:20:10Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.10-gke.27", GitCommit:"145f9e21a4515947d6fb10819e5a336aff1b6959", GitTreeState:"clean", BuildDate:"2020-02-21T18:01:40Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes cluster kind:

GKE

Manifests:

+ kubectl describe prometheus chart1-prometheus-operator-prometheus3rr
Name:         chart1-prometheus-operator-prometheus3rr
Namespace:    default
Labels:       app=prometheus-operator-prometheus3rr
Annotations:  <none>
API Version:  monitoring.coreos.com/v1
Kind:         Prometheus
Metadata:
  Creation Timestamp:  2020-04-28T13:04:52Z
  Generation:          1
  Resource Version:    14657263
  Self Link:           /apis/monitoring.coreos.com/v1/namespaces/default/prometheuses/chart1-prometheus-operator-prometheus3rr
  UID:                 d95d4825-8950-11ea-b3d9-42010aa80034
Spec:
  Alerting:
    Alertmanagers:
      API Version:   v2
      Name:          chart1-prometheus-operator-alertmanager
      Namespace:     default
      Path Prefix:   /
      Port:          web
  Enable Admin API:  false
  External URL:      http://chart1-prometheus-operator-prometheus3rr.default:9090
  Image:             quay.io/prometheus/prometheus
  Listen Local:      false
  Log Format:        logfmt
  Log Level:         debug
  Paused:            false
  Pod Monitor Namespace Selector:
  Pod Monitor Selector:
  Port Name:  web
  Remote Read:
    Name:         prom1
    read_recent:  true
    URL:          http://chart1-prometheus-operator-prometheus1:9090/read
    Name:         prom2
    Read Recent:  true
    URL:          http://chart1-prometheus-operator-prometheus2:9090/api/v1/read
  Replicas:       1
  Retention:      10d
  Route Prefix:   /
  Rule Namespace Selector:
  Rule Selector:
    Match Labels:
      Nothing:  nothing
  Security Context:
    Fs Group:            2000
    Run As Non Root:     true
    Run As User:         1000
  Service Account Name:  chart1-prometheus-operator-prometheus
  Service Monitor Namespace Selector:
  Service Monitor Selector:
    Match Labels:
      Nothing:  nothing
  Version:      v2.15.2
Events:         <none>

Prometheus Operator Logs:

Anything else we need to know?:

cc @pgier @dgrisonnet with the recent CRD changes this should now be validated as expected right?

Since #3107 was reverted by #3132 because we needed to upgrade CRD to v1 before upgrading controller-gen I don't think this is validated right now. However, since we recently upgraded CRD to v1 in #3175 we might be able to update controller-gen again.
I am concerned that upgrading controller-gen will break our v1beta1 CRDs again. I am not sure if controller-gen provides an opt-in way to remove the problematic x-kubernetes-list-map-keys field. Should I work something out to generate v1 and v1beta1 CRDs with different controller-gen version?

If that’s the case, why don’t we drop v1beta1 entirely? It seems like keeping it is keeping us from marking progress. People who have to use v1beta1 will just have to use an older version of prometheus operator compatible with their Kubernetes version.

Sounds good to me.

Thanks guys for your responsiveness.
So does this bug is going to be fixed?

It would be great if you could test that once #3187 merges :)

Sure @brancz. Also reviewed #3187
But can you please clarify how merging #3187 will fix this issue?

#3187 actually won't cut it just yet, (the theory is) we also need to upgrade the controller tools as @dgrisonnet mentioned.

ok, thanks for the clarification.

Hi @shay-berman, sorry I misunderstood your initial problem. In order to validate that all the fields exist, we use Kubernetes CRD pruning. However, this feature is only available after Kubernetes v1.15.0. Because of that, we disabled it in kube-prometheus #2924 but I don't know how it is done in the helm chart.
Thinking about it we might want to revert https://github.com/coreos/kube-prometheus/pull/408 in kube-prometheus versions that should not be compatible with Kubernetes v1.14. What do you think @brancz?

Helm does nothing here.  If the CR has admission controller that fail on wrong field also the helm should fail.

What @dgrisonnet meant is we have no idea if the helm chart is correctly synced, it's not maintained by us as we don't use it.

Thinking about it we might want to revert coreos/kube-prometheus#408 in kube-prometheus versions that should not be compatible with Kubernetes v1.14. What do you think @brancz?

I really think we should move beyond 1.14 at this point. I don't think we should put much more work into supporting it.

I created https://github.com/coreos/kube-prometheus/pull/519 which reverts the previous compatibility changes.  This enables validation, but the CRD will no longer apply in kube 1.14 and earlier.

@pgier \ @brancz
You mentioned that prometheus-operator-helm-chart does not maintained by you(prometheus operator maintainers).
So i wonder which one should I use - the prometheus-operator-helm-chart or the kube-prometheus?
from the prometheus-operator README it looks like both are valid and maintained by the community.
What are the differences between prometheus helm vs kube-prometheus in terms of features and maintainability? (except that one is helm method and the other is jsonnet)

The jsonnet version of kube-prometheus is maintained by the same people who maintain the prometheus operator. You need to make your own opinion of that information (obviously we recommend the jsonnet approach, but that's our opinion). We simply don't believe in helm as a technology which is why we don't use it or support it.
That said, often the jsonnet version is more up to date as it's kind of the "upstream" for the helm packaging, and the helm version is updated as the jsonnet version introduces features (things have happened the other way around as well though).

Thanks @brancz  for the clarification.
Anyway - @pgier@dgrisonnet  should we keep this issue open?

I think we can close this issue as this validation is enabled in the manifests provided by this repository.
As a side note, this validation requires Kubernetes v1.15+.


Problem downloading a marketplace app with no image associated

Description
If you have a marketapp with no image associated, when you click on download to your cloud button for the first time, it shows the datastores. If you go back and then click again everything works.
To Reproduce
Click on an app with no image associated (EC2 app) then click on download button.
Expected behavior
The datastores shouldn't be shown, because this app has no image associated.
Details

Affected Component: Sunstone
Version: development and 5.8

Progress Status

[ ] Branch created
[ ] Code committed to development branch
[ ] Testing - QA
[ ] Documentation
[ ] Release notes - resolved issues, compatibility, known issues
[ ] Code committed to upstream release/hotfix branches
[ ] Documentation committed to upstream release/hotfix branches

Wrong behaviour first time

Expected behaviour second time


A tiny mistake in tiramisu-pytorch.ipynb

In update_viz_summary_plot fuction:
`
def update_viz_summary_plot(self):
    trn_loss = self.loss_history['train'][-1]

    val_loss = self.loss_history['val'][-1]

    trn_err = self.error_history['train'][-1]

    val_err = self.error_history['val'][-1]

    txt = ("""Epoch: %d

        Train - Loss: %.3f Err: %.3f

        Test - Loss: %.3f Err: %.3f""" % (self.epoch,

        trn_loss, trn_err, tst_loss, tst_err))

    window = self.visdom_plots['summary']

    return viz.text(

        txt,

        win=window,

        env=self.name

    )

`
there are no tst_loss and tst_err variables.
So I think they should be replaced by val_loss and val_err respectively, right?

Yes i had this issue as well! Need to Replace it with val_loss and val_err.

Cleaned up the notebook and removed the visdom stuff for simplicity.


Fix build for py3k and pip inside a virtualenv.

pip installing SciPy within a virtualenv with Python 3.2 fails with the following error:
ValueError: 'build/py3k/scipy' is not a directory
Earlier versions of Python 3k are probably affected, though I didn't test.
This patch is really just a port of the numpy patch, since numpy suffered from the same issue: https://github.com/numpy/numpy/commit/dba98cc1.

Thanks for the PR. I tested this and pushed it as 5635506.


Coverage on Builds

Currently we do not have a coverage report being generated for TolaActivity.
We can use TravisCI to generate the coverage. So we need to set up Coverage reports for build and identify a place to publish the report once it is generated.

I updated README-TESTING.md to describe how to generate coverage reports locally. I'd be happy to take on adding it to Travis. Couldn't be that hard.


adding bhakti profile

title

Hi @bhaktishh I merged this PR, but currently, our deployment has some issues which I have reported to techstaff. So it may take a few days for the changes to show up.


Update link to .NET Core tools

http://dotnet.github.io/getting-started/ redirects to https://www.microsoft.com/net/core#windows

Thanks! Looks good to me.


tilemap breaking changes

scene.setTileMap is no longer available
Same for:
scene.getTilesByType
scene.setTile

Did you import the color coded tile map extension? There's an upgrade rule that should be loading it in old projects as needed

Yes, I just did that, got rid of the build problems, but things are not working.


Fixed sync with rebar2 port_compiler

There was a bug in the handling of multiple specs, which has since been corrected. I've also bumped the vsn to 1.4.0, but it seems no tag later than v1.1.0 has been pushed to the repo yet.
@tsloughter, can you please (1) push your local tags, (2) review, and (3) release 1.4.0? I've verified a multi-spec project like gen_socket and also single-spec projects to work with the fix.

yea, I'll do that today.

I haven't done any tags actually, since I just have it bump the version in .app.src, commit that and publish to hex.


Add an optimised Aarch64 assembly implementation of Int8 bitpacking.

What do these changes do?
This PR adds an Aarch64 assembly implementation of Int8 bitpacking.
How Has This Been Tested?
CI. Specific tests have been added for Int8 Aarch64 bitpacking.
Benchmark Results
These benchmarks were taken with the LCE benchmarking tool on my Android phone (a OnePlus 6), running the QuickNet models (modified to use Int8 instead of float) single-threaded. I used num_runs=250 and report the average latency and the standard deviation.

Model
Baseline (#492)
PR

QuickNet
17.18 +- 0.09
16.22 +- 0.05

QuickNetLarge
24.59 +- 0.10
23.21 +- 0.07

QuickNetXL
43.27 +- 0.09
40.32 +- 0.06

With the Ruy profiler, I've observed that with these changes the LceQuantize op accounts for 0.57% of the single-threaded Int8 QuickNet latency.
Related issue number
N/A.

Wow that did not work how I expected it would. This was a PR into the larq:optimised-int8-kernel branch (the #492 PR branch), and as a result of that PR being merged the branch got deleted and so this PR got automatically closed. I thought usually GitHub was clever enough to 'redirect' the PR to point to the master branch, but apparently not if it's a PR from a fork.


Go through Ledger (hardware wallet) documentation

Identify requirements for making a connection with Ledger hard wallet device.
Resources:
https://github.com/near/near-ledger-js
https://docs.near.org/docs/tutorials/ledger
https://blog.neufund.org/integrating-ledger-nanos-into-your-dapp-using-neufunds-js-wallet-provider-ecb71ca8f7b6
https://www.youtube.com/watch?v=V9eRdPCE2lo
https://portal.productboard.com/lnpb6dhpyz225uxhuo74d32p/c/25-support-active-ledger-transports?utm_medium=social&utm_source=portal_share
https://github.com/wandevs/wan-web-wallet/blob/master/src/wallets/ledger.ts

Researched and implemented basic Ledger connection demo

Researched and found Ledger Hardware emulator that may be used for testing purpeses.

Emulator documentation https://speculos.ledger.com/


chore: update plugin for IntelliJ 2024.3

IntelliJ 2024.3 has been released. This PR updates this plugin metadata to confirm it supports 2024.3.

Thank you @charleskorn! Is this ready for merge?

Thank you @charleskorn! Is this ready for merge?

Yep, should be good to go.

Thanks for the contribution! I'll get this released.


Add .nvmrc file to fix development node version

Currently, 18 node is not supported by Cloudflare:
https://github.com/igorkamyshev/farfetched/pull/101
https://github.com/cloudflare/pages-build-image/discussions/1
So we need to wait Cloudflare image update, and after we can add .nvmrc file

So, I think, Couldflare cannot make our life easier in this case, so I set up GitHub Action instead — https://github.com/igorkamyshev/farfetched/commit/f3f6ceadd13c6495828653f8326827e2c853ae13