Metadata caching - Parsed SQL and prepared statement handle caching

Still a prototype, opened PR to allow for conversation.

Codecov Report

Merging #272 into master will increase coverage by 0.37%.
The diff coverage is 18.87%.

@@             Coverage Diff              @@
##             master     #272      +/-   ##
============================================
+ Coverage     33.44%   33.82%   +0.37%
- Complexity     1483     1525      +42
============================================
  Files            97      101       +4
  Lines         23390    23623     +233
  Branches       3840     3877      +37
============================================
+ Hits           7823     7990     +167
- Misses        14001    14062      +61
- Partials       1566     1571       +5

Flag
Coverage Δ
Complexity Δ

#JDBC41
33.73% <18.87%> (+0.36%)
1519 <4> (+39)
:arrow_up:

#JDBC42
33.71% <18.87%> (+0.34%)
1519 <4> (+38)
:arrow_up:

Impacted Files
Coverage Δ
Complexity Δ

...om/microsoft/sqlserver/jdbc/SQLServerResource.java
100% <ø> (ø)
4 <0> (ø)
:arrow_down:

...t/sqlserver/jdbc/SQLServerConnectionPoolProxy.java
13.61% <0%> (-0.22%)
12 <0> (ø)

...va/com/microsoft/sqlserver/jdbc/SQLServerBlob.java
26.06% <0%> (-0.29%)
13 <0> (ø)

...m/microsoft/sqlserver/jdbc/KerbAuthentication.java
0% <0%> (ø)
0 <0> (ø)
:arrow_down:

...crosoft/sqlserver/jdbc/dns/DNSKerberosLocator.java
0% <0%> (ø)
0 <0> (?)

...com/microsoft/sqlserver/jdbc/dns/DNSUtilities.java
0% <0%> (ø)
0 <0> (?)

...ava/com/microsoft/sqlserver/jdbc/KerbCallback.java
0% <0%> (ø)
0 <0> (?)

...com/microsoft/sqlserver/jdbc/dns/DNSRecordSRV.java
0% <0%> (ø)
0 <0> (?)

...oft/sqlserver/jdbc/SQLServerPreparedStatement.java
31.59% <100%> (+2.19%)
89 <2> (+6)
:arrow_up:

...in/java/com/microsoft/sqlserver/jdbc/IOBuffer.java
37.87% <100%> (+0.96%)
0 <0> (ø)
:arrow_down:

... and 19 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d8e711e...418528b. Read the comment docs.

If the parse cache is going to be shared across multiple connections you'll need a way to evict invalid entries. Otherwise the following will break a long running application until it restarts:

App executes SQL
SQL metadata is cached with value M1
DDL change on database - now SQL would parse to M2
Usage of cached M1 metadata leads to an exception

Step 4 would repeatedly give an error with no way to succeed till the app is bounced and the cache is reset.
One solution to this would be to have entries evicted if methods that are using the cache throw an exception. An error would still happen if the cache is out of date but at least it'd only happen once per running application. Subsequent attempts to repeat the same operation would then succeed as the next attempt would get fresh metadata.
Alternatively, having the cache be at the Statement level mostly avoids this issue as you'd only run into it if the DDL change happens in the middle of repeated usage of the same Statement. As most JDBC usage involves creating Statements and then closing them when the unit of work is complete, the errant window would be much smaller.
Breaking it out into pros / cons:

Shared Connection Cache

Pro: Most memory efficient
Pro: Highest cache hits as shared across entire pool
Con: Requires more complicated eviction to deal with DDL changes (evict on error?)
Con: Must be thread safe (not a huge deal but still a minor con)

Per Connection Cache

Pro: Medium memory efficiency
Pro: Medium cache hit rate as there will be reuse for pooled connections and shared across Statements.
Pro: Eviction from connection pool provides convenient place to get rid of a stale cache (i.e. just rebuild the pool)
Pro: Don't have to worry about thread safety or synchronization as Connections are only meant to be used by one Thread at a time.
Con: Requires more complicated eviction to deal with DDL changes
Con: May require time based eviction to prevent stale SQL from remaining cached forever though could defer to connection pool (e.g. configure your pool to evict connections from the pool after a max age)

Per Statement Cache

Pro: Simplest to implement
Pro: Smallest "error window" when dealing DDL changes that should lead to cache invalidations
Pro: Don't have to worry about thread safety or synchronization as Statements are only meant to be used by one Thread at a time.
Con: Least efficient memory wise as each Statement would have its own cache.
Con: Least efficient cache hit wise as each Statement would have its own cache.

Note that I'm not advocating one over the other ... just providing some clarity on options. If it can be disabled via connection properties and gracefully handle errant evictions, the shared cache would likely be the best for the common case where DDL changes are not happening.

Wow, @brettwooldridge & @sehrope, thanks for all of the feedback!
I just added a simple implementation of actual reuse of created server-side prepared statement handles. The prepared statment handle cache is currently controlled by the SQLServerConnection.preparedStatementHandleCacheSize_SHOULD_BE_CONNECTION_STRING_PROPERTY field, a value > 0 means cache is on :-). This cache should further improve performance.
We now have two caches:

SQLServerPreparedStatement.parsedSQLCache (renamed per feedback) - Multi-connection/static cache. Contains metadata from parsing the SQL text on the client-side to avoid repeated parsing.
SQLServerConnection.preparedStatementHandleCache - Per connection cache. Contains prepared statement handles to be re-used.

Trying to reply as best I can:

Curious on why not to take the dependency on the Guava LRU cache. Is this mainly about download size?
@brettwooldridge, I renamed preparedStatementMetadataSQLCache to parsedSQLCache per request, way clearer, thanks!
I don't see why the parsedSQLCache should not be used across connections (and be static), it is simply tracking results of parsed strings and does not rely on connection properties. The new prepared statement handle cache I added needs to be per connection.
@brettwooldridge, good point of the ensureSQLSyntax() method, will start tracking that one as well.
@brettwooldridge, agree on not using the SQL text as key (doing POC first). Will move to a hash in a subsequent commit.
@brettwooldridge, sqlCommand is the original SQL text before processing, that is what we need to build the key on. Can't find it based on the post-processed SQL string. Agree that naming of userSQL vs. sqlCommand is not clear here (userSQL in fact NOT being user-provided-SQL...)
@sehrope, given that the parsedSQLCache has no relationship to the server at all, all it does is track metadata related to client-side-only parsing. So it does not need to be invalidated based on what happens on the server :-) Do you think we should clarify the name further to make it clearer that it is caching related to metadata from SQL-text only parsing?

Thanks!
-Tobias

@TobiasSQL
Regarding the Guava dependency, a few things.

Yes, one is size.  That is a 2.5MB dependency for an LRU cache that can be implemented in ~100 lines of code.
Another is that projects using the current v6 driver (or before) cannot simply drop it in, as they will be broken without that dependency.
As far as I know, no other JDBC drivers allow external dependencies.  If they do in fact require them, they shade them.  Not that I am suggesting shading Guava, because of its size -- even if you only wanted the dependent graph of classes for Cache.

Regarding the static cache, I think you are correct.  As a pure parsed SQL cache, I don't see a reason not to share VM-wide.  I was thinking ahead, and injecting my own assumptions about implementation, assuming that the metadata cache would eventually be married with the parsed SQL cache.  But that is not necessarily so, clearly.

Regarding the sqlCommand, the only place it is referenced outside of the constructor is buildExecuteMetaData():
988  private ResultSet buildExecuteMetaData() throws SQLServerException {
989     String fmtSQL = sqlCommand;
990     if (fmtSQL.indexOf(LEFT_CURLY_BRACKET) >= 0) {
991        fmtSQL = (new JDBCSyntaxTranslator()).translate(fmtSQL);
992     }
993
994     ResultSet emptyResultSet = null;
995     try {
996        fmtSQL = replaceMarkerWithNull(fmtSQL);
      ...

But it seems the result of line 991 must be the same as userSQL in the constructor, right?
199     userSQL = translator.translate(sql);

Great to see the appearance of a metadata cache ... or the beginnings of one!  🎉
I'll need a closer reading before, so take these comments with a grain of salt...
I think the prepared statement handles can be shared across Connections, right?  I believe the server-side access plans are shared across connections, so it should be possible -- assuming an access scheme keyed by the hostname/port/database like that discussed above.
I also believe that in the end, one of the biggest performance wins will come from caching the metadata returned by sp_prepare with option RETURN_METADATA along side the prepared handle.  I'm not sure, but one short-circuit location would likely be in buildParamTypeDefinitions(), and another in buildExecuteMetaData().

Thx @brettwooldridge!
For Guava your point on not being a simple drop-in replacement anymore settles it for me. Will replace :-)
Execution plans are shared across connections, and even users but prepared handles are not. They are tied to a specific connection. You can see if you try to do sp_prepare on two connections that both will get handle number 1 back.
Grain of salt applied :-) WRT to the metadata from sp_prepare we actually never call sp_prepare. All of our drivers now call sp_prepexec instead (which always returns metadata). I.e. we don't do the prepare until the first execution and then we prepare and execute at the same time. Following executes call sp_execute. buildExecuteMetaData seems to be only used if you want metadata for a statement without executing (in fact this should be changed from using set fmtonly to sp_describe_first_result_set).
Good point on row 991! Like you say it should be changed to:
fmtSQL = userSQL;
Made the change :-)
Thanks!
-Tobias

@TobiasSQL Most Java ORMs call the equivalent of Connection.prepareStatement() followed by stmt.getParameterMetaData().  It was the poor performance of that pattern in jTDS that led to my implementing metadata caching back in the day.  However we implement the metadata caching, it is good to keep that common pattern in many NLD.

@TobiasSQL Could you please open this pr to dev branch? Thank you.

@v-afrafi, changed to dev, thanks for noticing!

As an alternative to Guava's cache, how about caffeine? ( https://github.com/ben-manes/caffeine ) It is a near drop-in replacement for Guava's cache,  it was written by the guy that made the original guava cache (with lessons learned), and it has a much smaller footprint in general compared to guava.

A synchronized LRU is supported by Java's LinkedHashMap as a tiny extension, and can be written in less than 10 LOC. However, it does not perform well under concurrent workloads.
Attempting to solve that led to ConcurrentLinkedHashMap. This was used heavily (Cassandra, etc), and often embedded (Groovy, Camel, etc). It is small and JDK6 based.
Guava's Cache was retrofitted with a CLHM's design and embedding many features worked out custom decorators. This provides many best practices, such as using memoization rather than racy get-compute-put. Unfortunately its original design around soft references hindered our efforts, since we had to work within that framework rather than from a stronger foundation.
Caffeine is a JDK8 rewrite that improves on the above. It uses a modern eviction policy, offers more features, and has had very healthy adoption. However, this richness comes at the cost of a modestly sized jar. Some do shade it, but its probably not ideal for a jdbc driver.
I'd recommend embedding CLHM as it is probably the best fit.

@ben-manes & @cogman, thanks! Given concurrency is important for at least the parsed SQL text cache, CLHM sounds like a good option to embed. I will take a closer look at it once I get stuff into reasonable shape :-)
@brettwooldridge , thanks for pointing out the ORM pattern using getParameterMetaData(), makes sense. I reworked things to also handle caching this metadata.
Overall I am reworking a bunch of stuff from first prototype as it has a few issues, should hopefully be able to push some changes soon.

Cleaned up prototype and fixed a bunch of issues. Depending on feedback next POR is to replace cache impl. and start using hash for cache key.

I would suggest against using Sha-1 for the hash key.  Sha-1 was designed to reduce collisions in a cryptographically safe manor, but it is slow.  Faster algorithms exist which have low collision rates.  Particularly, fnv3 or murmur would be preferable as high performance hashing algorithms with low collision rates.
With that said, it probably would be a better idea to use the string as the key but limit the size that will be cached, since large statements are less likely to be reused.  If you are concerned about size, then you could compress the statement.  But I wouldn't do that as a first step.

@cogman, after running my perf suite I agree with you. Using SHA1 more or less undid the perf gain from caching the statement handle in the first place :-( Also, after pondering this a bit more, given the string is anyway used by the prepared statement instances it seems like adding a hash will actually use more memory...

@TobiasSQL Your hash code had a bug that caused 100% cache miss.  I suggest trying it again.  A quick and dirty bench on my circa 2013 i5 desktop yielded 1000 SHA1 hashes in 1.7ms, or roughly 1.7 microseconds per.

@TobiasSQL Dump the Apache Commons Digest, drop the use of ByteBuffer its equals() and hashCode() are unnecessarily expensive.
Try:
static private Cache< Sha1HashKey, ParsedSQLCacheItem> parsedSQLCache;
...
private static class Sha1HashKey {
   private byte[] bytes;

   Sha1HashKey(String s) {
      this.bytes = getSha1Digest().digest(s.getBytes());
   }

   public boolean equals(Object obj) {
      return Arrays.equals(bytes, (byte[]) obj);
   }

   public int hashCode() {
      return Arrays.hashCode(bytes);
   }

   private MessageDigest getSha1Digest() {
      try {
         return MessageDigest.getInstance("SHA-1");
      }
      catch (final NoSuchAlgorithmException e) {
         // this is not theoretically possible, but we're forced to catch it anyway
         throw new RuntimeException(e);
      }
   }
}

Thanks @brettwooldridge, this is a bit strange since the first thing I tried was close to exactly the code that you propose (using Arrays.equals and Arrays.hashCode) which didn't work. I am sure I did something wrong, will give this a try :-)
Meanwhile I have replaced the use of the Guava cache to the CLHM tha @ben-manes suggested :-)

@cogman
At 1.7 microseconds overhead per statement, I have a hard time believing that SHA1 calculation "undoes the perf gain from caching the statement handle"...
I would suggest that possibly other inefficiencies in the implementation, including this code which seems to result in a 100% cache miss, the use of ByteBuffer as a cache key which has a fairly high overhead for equals()/hashCode() are much more likely to have affected benchmarks.
How are you benchmarking?    Warm-up is essential to benchmarking on the JVM; any kind of "cold" test is going to give skewed results ... potentially very skewed depending on the code in question.

@TobiasSQL Curious about the statement:

given the string is anyway used by the prepared statement instances it seems like adding a hash will actually use more memory...

Why is the string used anyway?  After transformation, there seems no point to holding onto the original SQL string anywhere in the prepared statement instance, right?

Good point @brettwooldridge. First off I was assuming it was public which it is not. After my changes so far it is only used in buildExecuteMetaData where it seems useless to me. If I remove that reference you are right, there seems like no point to keep track of it and it can be removed.
For the statement handle cache it is different since it uses the processed SQL (userSQL) as the key. For that case using a hash still seems unnecessary from memory usage POV.

@brettwooldridge, thanks for the catch of the 100% cache misses for the parsed SQL text. That was clearly the main issue, sorry for wasting your time!
After re-running my simple perf suite (with warm-up) I get inclusive results on hash vs. not (hash clearly adds overhead, not clear on %), need to run some real benchmarking on this when I get back from vacation :-)
Will push the change including CLHM replacement for Guava and hashing for parsed SQL text lookup. @brettwooldridge et al, if you have time to compare the two (simple change in SQLServerPreparedStatement in 3 places) that would awesome :-)

@TobiasSQL Re: taking at look, will do.
Re: the handle cache using the processed SQL (userSQL) as the key, does it need to?  Or can it use the user provided SQL, in which case maybe the SHA1 hash could be reused and the memory overhead cut?

@brettwooldridge, you are right, I can just hash once and keep the hash around and use it for both caches. Very good point!

FYI, trying to push but connection is very slow from this Caribbean cruise ship I am on, git isn't liking it...

@brettwooldridge  Yeah, mainly my problem with SHA-1 wasn't so much the overhead, just that it is the wrong tool for this sort of a job.  The same as it would be wrong to use a List where a Set is wanted and vice versa.
I question the usage of it, mainly because it looks like it is mostly there for memory saving purposes.  As @TobiasSQL points out, it looks like the string is being saved off anyways as part of the statement.  If that weren't the case, I would still suggest using a non-cryptographic hashing algorithm.  We don't really care if an attacker can cause a collision.

@cogman I wouldn't say SHA1 is the wrong tool necessarily, after all git itself uses SHA1 hashes.  Certainly it is worth exploring faster hashes, but the collision rates of hashes like CityHash, Murmur3, etc. are still unacceptably high.  Any 128-bit hash is likely insufficient.  FNV in the 256-bit variant might fit.  How the performance of FNV-256 compares to SHA1 (which enjoys some specific x86 CPU instructions), I do not know.

10 was a random pick (and hasn't really been picked yet...), we haven't discussed a default size but now is as good a time as any :-)
This cache doesn't really use a lot of memory on the client side after we hashed the key and removed the need for keeping additional copies of the SQL text. On the client-side each cache entry really only includes the SHA1 hash (20 bytes), the handle (4 bytes) as well as a few booleans and the reference to the connection plus whatever overhead is used by the cache item instances and the cache itself. Overall cheap. If we also cache parameter metadata (after a request to getParameterMetadata) then all of the information from that instance is also part of the cache entry.
All of this is per connection so with lots of connections it adds up, depending on the H/W you run on (thinking devices/sensors) we may want to be conservative. On the server side we basically keep the hash of the SQL text, the handle, as well as the actual SQL text. So the server-side is for sure more expensive. The server side also tends to be the critical resource since it is shared.
10 seemed like a safe first choice which should provide good handle re-use for many workloads.

@brettwooldridge  " I wouldn't say SHA1 is the wrong tool necessarily, after all git itself uses SHA1 hashes"
Git doesn't use SHA1 hashes for cache lookups.  Nor is it particularly chuffed about the performance of it's hashing algorithm.  SHA1 in git is to provide a unique key with low collision probability, data verification, and until recently, for commit signing.

After googling the SHA extensions on Intel and Ryzen, I must say I'm really impressed at what Intel and AMD have done here.  I was going to say something like "A multiplication + xor is going to be hard to beat" for FNV, however, the SHA extensions are really crazy optimized.  Some of my reading put it at around 10 cycles/byte.  Really not that bad, especially considering statements shouldn't generally be many bytes.

I have a concern about the com.googlecode.* classes.  If the user has an incompatible version of those classes in their classpath before the driver's jar, we will get unexpected failures.  If we include these files, they should be relocated to a different package, such as mssql.googlecode.* or similar.

Good catch @brettwooldridge, thx!

As an FYI the commits coming in now are fixes based on findings from our test suites that have not been GitHub:ed yet.

Got it.  I'll run the benchmark later today.

@TobiasSQL Have you had a look at my metadataCaching2 branch?   I replaced the SHA1 hash with CityHash128 which performed very well, with no collisions over the corpus.  In a local SHA1 vs. CityHash128 benchmark, CityHash128 was ~10x faster, at roughly 160ns per hash.
The reference counting is tightened up further still.

Here are the benchmark results, Best of Three runs:
mssql-jdbc-6.1.7.jre8.preview.jar
22:39:42,654 (DBWorkload.java:614) INFO  - Workload Histograms:
Completed Transactions:
Persist/01           [11697] *******************************************************************************
Retrieve/02          [11536] ******************************************************************************
Update/03            [11774] ********************************************************************************
Delete/04            [11657] *******************************************************************************

mssql-jdbc-6.1.8-SNAPSHOT.jre8.jar (mssql-jdbc/metadataCaching branch):
22:49:37,774 (DBWorkload.java:614) INFO  - Workload Histograms:
Completed Transactions:
Persist/01           [12321] ********************************************************************************
Retrieve/02          [12120] ******************************************************************************
Update/03            [12112] ******************************************************************************
Delete/04            [12080] ******************************************************************************

mssql-jdbc-6.1.8-SNAPSHOT.jre8.jar (brettwooldridge/metadataCaching2 branch):
22:58:19,075 (DBWorkload.java:614) INFO  - Workload Histograms:
Completed Transactions:
Persist/01           [12564] *******************************************************************************
Retrieve/02          [12595] ********************************************************************************
Update/03            [12457] *******************************************************************************
Delete/04            [12579] *******************************************************************************

Previous benchmark was the oltpbenchmark jpab benchmark.  This is the oltpbenchmark epinions benchmark:
mssql-jdbc-6.1.7.jre8.preview.jar
23:59:17,238 (DBWorkload.java:614) INFO  - Workload Histograms:
Completed Transactions:
GetReviewItemById/01 [33985] ***************************************
GetReviewsByUser/02  [33999] ***************************************
GetAverageRatingB... [33961] ***************************************
GetItemAverageRat... [33762] ***************************************
GetItemReviewsByT... [33722] ***************************************
UpdateUserName/06    [33919] ***************************************
UpdateItemTitle/07   [33828] ***************************************
UpdateReviewRatin... [33584] ***************************************
UpdateTrustRating/09 [68105] ********************************************************************************

mssql-jdbc-6.1.8-SNAPSHOT.jre8.jar (mssql-jdbc/metadataCaching branch):
23:50:19,993 (DBWorkload.java:614) INFO  - Workload Histograms:
Completed Transactions:
GetReviewItemById/01 [34305] ****************************************
GetReviewsByUser/02  [34392] ****************************************
GetAverageRatingB... [34216] ***************************************
GetItemAverageRat... [34175] ***************************************
GetItemReviewsByT... [34087] ***************************************
UpdateUserName/06    [34530] ****************************************
UpdateItemTitle/07   [34278] ****************************************
UpdateReviewRatin... [34270] ****************************************
UpdateTrustRating/09 [68526] ********************************************************************************

mssql-jdbc-6.1.8-SNAPSHOT.jre8.jar (brettwooldridge/metadataCaching2 branch):
23:39:41,015 (DBWorkload.java:614) INFO  - Workload Histograms:
Completed Transactions:
GetReviewItemById/01 [34670] ****************************************
GetReviewsByUser/02  [34595] ***************************************
GetAverageRatingB... [34911] ****************************************
GetItemAverageRat... [35003] ****************************************
GetItemReviewsByT... [34670] ****************************************
UpdateUserName/06    [35248] ****************************************
UpdateItemTitle/07   [34690] ****************************************
UpdateReviewRatin... [34823] ****************************************
UpdateTrustRating/09 [69330] ********************************************************************************

Left comments for your PR @brettwooldridge, sorry for the delay! I am hoping we get the current caching finalized for release next week. The week after should hopefully work great, I am thinking you can then submit a PR against the dev branch (rather than metadataCaching). We mainly need to make sure all issues found by the other test cases are resolved first.

@TobiasSQL Sure, sounds great.  I'm still testing and tweaking my speedify branch. 👍  But current results are looking outstanding.
oltpbench/jpab benchmark (Hibernate) running a transaction mix of 5/75/15/5 (Persist/Retrieve/Update/Delete):
mssql-jdbc-6.1.7.jre8.preview.jar 🐢
INFO  - Workload Histograms:
Completed Transactions:
Persist/01           [ 4848] *****
Retrieve/02          [72439] ********************************************************************************
Update/03            [14412] ***************
Delete/04            [ 4844] *****

mssql-jdbc-6.1.8.jre8.preview.jar (speedify) 🏇
INFO  - Workload Histograms:
Completed Transactions:
Persist/01           [ 6364] *****
Retrieve/02          [96709] ********************************************************************************
Update/03            [19439] ****************
Delete/04            [ 6487] *****

Beautiful! @brettwooldridge, is the comparison branch based on current state of metadataCaching, dev, or master, or something else?

The comparison was with v6.1.7.

@brettwooldridge, you can now open a PR against dev for your improvements :-)

Never mind, we have some more things to go through before the merge. Expecting it next week.

@brettwooldridge, this is a great point. Agree with the implementation, makes sense with calling setCapacity to me.
I was noodling on this as well, not sure about tying it to the discard queue they seem to have at best a distant relationship ;-)
Two options come to mind:

Expose a separate static configuration knob for the parse cache.
Tie the parse cache to the statement cache size (for 5x statement cache size, potentially expose the ratio)

Looking around, additionally I think we need to provide a limit for the max statement text size to place in the parse cache to prevent accidental memory hogging. My suggestion is to start with a conservative and fairly large value (say 10k), we can later make it configurable in a following release.
Thoughts?

Given that the statement cache isn't static (which I forgot) it seems to me that only option 1 above makes sense, i.e. separate configuration knob.


Make WindowBuilder and HeadlessRendererBuilder derive Clone.

This should deal with #779. I'm somewhat uneasy it touches platform specific code. It is really trivial tough.

Forgot to mention, I included HeadlessRendererBuilder for symmetry.

Thanks


Improve report ordering, status labels etc...

The report should be ordered according to the time the screenshot was taken.
The report should have a passed/failed label on each screenshot.
When a screenshot matches, we don't need to store the screenshot or the diff. We can just use the base image. This will speed up the tests

Resolved by #92


Loading large models with fabric, FSDP and empty_init=True does not work

Bug description
I'm following litgpt for how to load a large model with FSDP and I'm getting an error related to shapes. See example script and error message
What version are you seeing the problem on?
v2.2
How to reproduce the bug
#!/usr/bin/env python
import os.path as osp
import torch
from lightning.fabric.strategies import DDPStrategy, FSDPStrategy
from litgpt.lora import Block as LoraBlock
from litgpt.model import Block
import lightning as L
from litgpt.model import GPT
from litgpt.lora import Config

from torch.distributed.fsdp import MixedPrecision

def main():

    strategy = FSDPStrategy(auto_wrap_policy={Block, LoraBlock},
                                sharding_strategy='HYBRID_SHARD',
                                mixed_precision=torch.distributed.fsdp.MixedPrecision(param_dtype=torch.bfloat16),
                                use_orig_params=True)

    fabric = L.Fabric(
        accelerator="gpu",
        num_nodes=1,
        devices=8,
        strategy=strategy,
        precision='bf16-mixed')

    fabric.launch()

    llm_config_path = '.../checkpoints/mistralai/Mistral-7B-v0.1/model_config.yaml'
    config = Config.from_file(llm_config_path)
    with fabric.init_module(empty_init=True):
        model = GPT(config)
    print('setting up')
    fabric.setup_module(model)

    llm_path = osp.join(osp.dirname(llm_config_path), "lit_model.pth")
    fabric.load_raw(llm_path, model)
    print('done')

main()

Error messages and logs
  File "test_init.py", line 42, in <module>
    main()
  File "test_init.py", line 38, in main
    fabric.load_raw(llm_path, model)
  File "lightning/fabric/fabric.py", line 816, in load_raw
    self._strategy.load_checkpoint(path=path, state=obj, strict=strict)
  File "lightning/fabric/strategies/fsdp.py", line 548, in load_checkpoint
    _load_raw_module_state_from_path(path, module=state, world_size=self.world_size, strict=strict)
  File "/lightning/fabric/strategies/fsdp.py", line 888, in _load_raw_module_state_from_path
    _load_raw_module_state(state_dict=_lazy_load(path), module=module, world_size=world_size, strict=strict)
  File "lightning/fabric/strategies/fsdp.py", line 896, in _load_raw_module_state
    module.load_state_dict(state_dict, strict=strict)
  File "torch/nn/modules/module.py", line 2153, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for GPT:
        size mismatch for lm_head.weight: copying a param with shape torch.Size([32000, 4096]) from checkpoint, the shape in current model is torch.Size([32768512]).
        size mismatch for transformer.wte.weight: copying a param with shape torch.Size([32000, 4096]) from checkpoint, the shape in current model is torch.Size([0]).
        size mismatch for transformer.ln_f.weight: copying a param with shape torch.Size([4096]) from checkpoint, the shape in current model is torch.Size([0]).

Environment
	- lightning:         2.2.3
	- lightning-cloud:   0.5.64
	- lightning-utilities: 0.9.0
	- lion-pytorch:      0.1.4
	- pytorch-lightning: 2.2.1
	- pytorch-wpe:       0.0.1
	- torch:             2.2.2+cu121

More info
Looking into the implementation of fabric.load_raw, it seems to first unwrap compiled objects and then call FSDPStrategy.load_checkpoint, looking at the latter there is the following sentence in the doc string:
The strategy currently only supports saving and loading sharded checkpoints which are stored in form of a directory of multiple files rather than a single file.

which seems to contradict the doc of load_raw():
Use this for loading a raw PyTorch model checkpoint created without Fabric.
        This is conceptually equivalent to ``obj.load_state_dict(torch.load(path))``, but is agnostic to the strategy
        being used.

Very confused what the right approach is supposed to be.

Hi! I am facing a similar issue which might be related. Is there anything I can do at the moment to make this work?


Invoking commands using JSON is painful and error prone (CON-1194)

@wqx6
Converting the request handle data to JSON and then having send_request() convert that JSON to TLV is just painful to do. You are going to have to build custom code for every parameter list.
https://github.com/espressif/esp-matter/blob/main/examples/light_switch/main/app_driver.cpp#L211
Isn't the requestData on request_handle_t an EncodableType? So something like my old code should be possible...
It doesn't work anymore because Controller has been made internal. Old code....
I got this to compile again with "#include <controller/InvokeInteraction.h>"
    else if (cmd_handle->command_path.mClusterId == LevelControl::Id)
    {
        switch (cmd_handle->command_path.mCommandId)
        {
        case LevelControl::Commands::MoveWithOnOff::Id:
        {
            // level_control::command::group_send_move_with_on_off(fabric_index, cmd_handle->group_id);
            Controller::InvokeGroupCommandRequest(&exchangeMgr, fabric_index, cmd_handle->command_path.mGroupId,
                                                  *((LevelControl::Commands::MoveWithOnOff::Type *)cmd_handle->request_data));
            break;
        };
        case LevelControl::Commands::MoveToLevelWithOnOff::Id:
        {
            // level_control::command::group_send_move_with_on_off(fabric_index, cmd_handle->group_id);
            Controller::InvokeGroupCommandRequest(&exchangeMgr, fabric_index, cmd_handle->command_path.mGroupId,
                                                  *((LevelControl::Commands::MoveToLevelWithOnOff::Type *)cmd_handle->request_data));
            break;
        };
        case LevelControl::Commands::StopWithOnOff::Id:
        {
            // level_control::command::group_send_stop_with_on_off(fabric_index, cmd_handle->group_id);
            Controller::InvokeGroupCommandRequest(&exchangeMgr, fabric_index, cmd_handle->command_path.mGroupId,
                                                  *((LevelControl::Commands::StopWithOnOff::Type *)cmd_handle->request_data));
            break;
        };
        }
    }

It would be much easier to directly construct the TLV in light_switch/main/app_driver.cpp than to use JSON as an intermediary.  You might want to leave the JSON version and add a TLV version.
The new code would look something like this
    else if (cmd_handle->command_path.mClusterId == LevelControl::Id)
    {
        switch (cmd_handle->command_path.mCommandId)
        {
        case LevelControl::Commands::MoveWithOnOff::Id:
        {
            // level_control::command::group_send_move_with_on_off(fabric_index, cmd_handle->group_id);
            send_request(fabric_index, cmd_handle->command_path,
                                                  ((LevelControl::Commands::MoveWithOnOff::Type *)cmd_handle->request_data))->Encode());
            break;
        };
        case LevelControl::Commands::MoveToLevelWithOnOff::Id:
        {
            // level_control::command::group_send_move_with_on_off(fabric_index, cmd_handle->group_id);
            send_requestt(fabric_index, cmd_handle->command_path,
                                                  (((LevelControl::Commands::MoveToLevelWithOnOff::Type *)cmd_handle->request_data))->Encode());
            break;
        };
        case LevelControl::Commands::StopWithOnOff::Id:
        {
            // level_control::command::group_send_stop_with_on_off(fabric_index, cmd_handle->group_id);
            send_request(fabric_index, cmd_handle->command_path,
                                                  (((LevelControl::Commands::StopWithOnOff::Type *)cmd_handle->request_data))->Encode());
            break;
        };
        }
    }

Something like that since this stuff is templated instead of using virtual methods.

The request_data field in request handle doesn't need to be a JSON string. We just provide a API that could accept a JSON string input and use that API in our client examples. This API could be used for any clusters so that it can avoid so many if-elses or switch-cases in our code. And that API could also be used when CONFIG_ENABLE_CHIP_CONTROLLER_BUILD is not set. If you want to call the APIs which uses the Encoded types from Matter in controller/InvokeInteraction.h you can just set the Encodes types as the request data in request handle and do the converting in your command callback.

I agree that I don't like all of the if-else/switches in the command callbacks. There is something wrong with the CHIP architecture which is causing this. I think the design problem is that you should TLV encode the command parameters before the command is submitted instead of after. Then in the command callback the data would already be encoded and there would be no need for the if/else/switch.  request_data would be a pointer to the TLV instead of the command structure. This is along similar lines to what you are doing with JSON.
Of course this change would have to be made in the CHIP core.

@wqx6
This is getting fixed inside CHIP and you won't need the JSON.  There is a new API, https://github.com/project-chip/connectedhomeip/blob/master/src/app/CommandSender.h#L379-L393
You'll need to do some adjustments to use this, but this allows the parameters to be encoded BEFORE submitting the command. Now when the command ends up in the callback you don't need the giant if/else/switch since you will be able to directly Invoke using the EncodableToTLV thus avoiding all of the templated Invoke commands.

Thanks! I will update the API to use the virtual class when we update the upstream repo.


[DO NOT MERGE] Fix brittle time-out test.

Trying to see if I can fix the AppVeyor brittle timeout test. Using this PR as a way to run the build on AV.

Come on yak shaver ;)
-----Original Message-----
From: "Ian Battersby"<EMAIL_ADDRESS>Sent: 29/05/2015 15:54
To: "iancooper/Paramore"<EMAIL_ADDRESS>Subject: [Paramore] [DO NOT MERGE] Fix brittle time-out test. (#85)
Trying to see if I can fix the AppVeyor brittle timeout test. Using this PR as a way to run the build on AV.
You can view, comment on, or merge this pull request online at:
https://github.com/iancooper/Paramore/pull/85
Commit Summary
Fix brittle timeout test.
File Changes
M Brighter/paramore.brighter.commandprocessor.tests/Timeout/Test Doubles/MyFailsDueToTimeoutHandler.cs (6)
Patch Links:
https://github.com/iancooper/Paramore/pull/85.patch
https://github.com/iancooper/Paramore/pull/85.diff
—
Reply to this email directly or view it on GitHub.

@holytshirt OMG, we need to get funding for AppVeyor, this is bloddy painful.


DnD support.

This might be of more value starting from other forms of documentation but I wanted to be able to demo this alongside apidocs.
There's no drop-cue but there's enough information in the state to draw one.

Lots of this has made it back to main in a different form, but leaving open for the CM DND state management as we do plan to revisit dnd on #335

Closing this as I'm going to open a new draft PR, but keeping the branch around for now. We might want the decoration/state prep that's here.


feat(balance): add an unauthenticated getBalance method PE-6630

:tada: This PR is included in version 1.12.0-alpha.1 :tada:
The release is available on:

npm package (@alpha dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.12.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


client refactor: remove mirrors

Addresses #1307
Description of the changes being introduced by the pull request:
Drop "mirrors" support.
Updater no longer supports downloading each metadata file from a list of user-provided mirrors. All downloads are performed from a single metadata url.
Target files are downloaded either from a single default url or an optional url is passed for each target file on download.
More info in: comment, comment
API is changed the following way:
class Updater:
     def __init__(
         self,
         repository_name: str,
-        repository_mirrors: Dict,
+        metadata_url: str,
+        default_target_url: Optional[str] = None,
         fetcher: Optional[FetcherInterface] = None,
     ):

    def download_target(
         self,
         targetinfo: Dict,
         destination_directory: str,
+        target_url: Optional[str] = None,
     ):

Edit: The exact metdata_url and target_url format needs either better documentation for the user or better normalization functions provided by TUF.
Please verify and check that the pull request fulfills the following
requirements:

[ ] The code follows the Code Style Guidelines
[ ] Tests have been added for the bug fix or new feature
[ ] Docs have been added for the bug fix or new feature

This is a pleasing diff stat. I think we might be able to reduce it further

Absolutely: e.g. we can now get rid of almost all manual file object handling

Since Teodora is away for the week I'll take this and continue with suggested fixes (and removing loads of temp file handling). I think I'll do a new PR and close this (but will keep these commits so review can "continue")

superseded  by #1373


Why the class name is NettyRemotingAbstract instead of AbstractNettyRemoting？

Why the class name is NettyRemotingAbstract instead of AbstractNettyRemoting？

I don't know why, but it ended up like this ：）


ImageCard-Weather: ERROR: WeatherAdaptiveCardExtension is not initialized

Sample
ImageCard-Weather
Authors
@aakash_316
Expected or Desired Behavior
ACE should render after saving properties
Observed Behavior

Steps to Reproduce

Package and Deploy Solution to App Catalog
Add Weather Ace to Viva Connections Dashboard
Set Properties including image url, location, Bing and Azure API keys
Weather Ace renders
Save changes to dashboard
Weather ACE throws above error

Environment Details (Development & Target environment)
OS: Windows 11
Target: M365 Viva Connections
Browser: Edge
Tooling:
VS Code with
SP Framework 1.15.2
Node.js v16.20.0
Gulp 2.3.0

@aakashbhardwaj619 Can you please take a look at this and see what the issue might be?


Gzip manifest.json before uploading

Description

Fixes WB-NNNNN
Fixes #NNNN

What does the PR do? Include a concise description of the PR contents.

[ ] I updated CHANGELOG.md, or it's not applicable

Testing
How was this PR tested?

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#7402  👈
#7401
#7400
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @moredatarequired and the rest of your teammates on  Graphite


getting sqlalchemy datetime type ignores None timezones

When converting to datetime64, pandas does:
        if col_type == 'datetime64':
            try:
                tz = col.tzinfo
                return DateTime(timezone=True)
            except:
                return DateTime
        if col_type == 'timedelta64':

2 problems

The exception catch is way to broad. I think this was probably supposed to catch an AttributeException (For col with no tzinfo attribute at all)
columns can have a tzinfo =None, in which case a Datetime(timezone=False) should be created instead.

Did you have an issue with this with an actual case? Because I don't think this code is working at all, as columns (series) have never a tzinfo attribute.
I also mentioned this in #9086, the issue for timezone support in the sql functions.

This didn't trigger an actual bug, its more that i was looking through other issues and i noticed this bit of code didnt make sense....

It indeed does not make much sense ..
I am going to close this, as there is already #9086 to track this.


Candidate Ranking on insuranceqa V1 dataset

I encountered the following error trail while running the ranking routine on V1 dataset
`File "/home/ubuntu/.virtualenvs/env3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/home/ubuntu/.virtualenvs/env3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/ubuntu/.virtualenvs/env3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [13100,100] rhs shape
= [19283,100]
[[Node: save/Assign_5 = Assign[T=DT_FLOAT, _class=["loc:@embeddings"], use_locking=true, validate_shape=true, _device="/job:localhost/
replica:0/task:0/device:CPU:0"](embeddings, save/RestoreV2:5)]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "run_reranking_server.py", line 213, in
run()
File "/home/ubuntu/.virtualenvs/env3/lib/python3.5/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/home/ubuntu/.virtualenvs/env3/lib/python3.5/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/home/ubuntu/.virtualenvs/env3/lib/python3.5/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/ubuntu/.virtualenvs/env3/lib/python3.5/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "run_reranking_server.py", line 206, in run
training.start(model, data, sess)
File "/home/ubuntu/faq/acl2017-non-factoid-qa/Candidate-Ranking/experiment/qa/train/init.py", line 176, in start
self.state.load(sess, weights='last')
File "/home/ubuntu/faq/acl2017-non-factoid-qa/Candidate-Ranking/experiment/qa/train/init.py", line 344, in load
self.saver.restore(session, restore_path)
File "/home/ubuntu/.virtualenvs/env3/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1752, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/ubuntu/.virtualenvs/env3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/home/ubuntu/.virtualenvs/env3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/home/ubuntu/.virtualenvs/env3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/home/ubuntu/.virtualenvs/env3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [13100,100] rhs shap$
= [19283,100]
File "/home/ubuntu/.virtualenvs/env3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3414, in create_op
op_def=op_def)
File "/home/ubuntu/.virtualenvs/env3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1740, in init
self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access
InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [13100,100] rhs shape= [19283,100]
[[Node: save/Assign_5 = Assign[T=DT_FLOAT, _class=["loc:@embeddings"], use_locking=true, validate_shape=true, _device="/job:localhost/
replica:0/task:0/device:CPU:0"](embeddings, save/RestoreV2:5)]]
`

From the above stacktrace my guess would be that after model training the vocabulary changed. Was the model trained on a different dataset?

Thank you for the reply. The model was trained on the insuranceqaV1 dataset only. It turns out that if the model was trained on the V2 dataset first, then later training it on the V1 dataset with elicit the above error trail. If one changes the path in config file to a new checkpoint directory, the problem seems to be solved.

To enable training on one dataset and evaluation on another you would need to adapt experiment.qa.data.init.py and remove the vocab argument in  read_embeddings(...). You would also need to join self.archive.vocab with the vocab of embeddings_dict in line 65 (for loop).


No match in data for Zuma's Revenge Origin version

I have the Origin version of Zuma's Revenge and the tool doesn't find any matches.

exe: https://www.mediafire.com/file/siluf6tpex53v64/ZumasRevenge.exe/file

Thanks for providing the game file, but I couldn't find a way to patch Origin version.
It worked with Popcap version and Steam version.
Here are the patched files: ZumasRevenge.zip

Thanks for providing the game file, but I couldn't find a way to patch Origin version.
It worked with Popcap version and Steam version.
Here are the patched files: ZumasRevenge.zip


How can I uninstall everything?

How can I uninstall everything that was just installed on my system?

There's no easy way to uninstall "everything". Read through the script and reverse the steps. https://github.com/thoughtbot/laptop/blob/master/mac
This will uninstall a big chunk of the script as it removes all of Homebrew:
https://github.com/Homebrew/homebrew/blob/master/share/doc/homebrew/FAQ.md#how-do-i-uninstall-homebrew
You'll need to revert your shell back to whatever it was before zsh (probably bash). Remove anything added to your ~/.zshrc file.
That should get you 95% of the way there.

Thanks for the help.

The reason I am asking is because I did a fresh reset and reinstall of my system. Ran the commands in the instructions. Now everytime I start my terminal I get this message: http://shel.io/f/CrMI.png
The mtr thing is me. But the other stuff I have no clue. Please help.

Thanks @Seralto.

I'm not sure if the exec bash -l command would have given a different result, but I had to create .bash_profile, I copied the contents from .zshrc. Then since the terminal would go to my home dir every time I opened a new tab I had to change $HOME/.bin with ~/.rbenv/shims

Providing an automated uninstall would be difficult because we don't actually know what the start state of your machine is.
I would reverse the steps in the script somewhat manually:

chsh /bin/bash to switch back to bash
rm -rf ~/.rbenv to remove installed rubies/gems
rm ~/..zshrc to remove the zsh config we add (assumes you had no zsh config to start with)
uninstall homebrew (assumes homebrew was not installed to start with) or the individual formulas you want to remove.

... etc ...


test: moving waitForCounter calls off of using sleep

Risk Level: n/a (test only)
Testing: many tsan runs
Docs Changes: n/a
Release Notes: n/a

@jmarantz feel free to ignore this until next week given you're already looking at the accessor PR - you just seem like the right reviewer for this one.
If we like it, I can move gauge lookups over at some point

going back to condvar for now sgtm
On Wed, May 27, 2020 at 11:06 AM alyssawilk<EMAIL_ADDRESS>wrote:

@alyssawilk commented on this pull request.
In test/integration/server.h
https://github.com/envoyproxy/envoy/pull/11289#discussion_r431214011:

void waitForCounterEq(const std::string& name, uint64_t value) override {

TestUtility::waitForCounterEq(statStore(), name, value, time_system_);

while (statsAllocator().getCounter(name) == nullptr ||

      statsAllocator().getCounter(name)->value() != value) {

 AwaitData data = std::make_tuple(&statsAllocator(), &name, value);

 statsAllocator().mutex().Await(absl::Condition(+[](AwaitData* data) {

Ok if I revetr this while that lands? it was a WIP and again I think it
decreases readability.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/envoyproxy/envoy/pull/11289#discussion_r431214011,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAO2IPNE332GT3Y3FSUFQCTRTUUANANCNFSM4NHA2YUQ
.

Ok, had to do the dreaded force push to unwind the half-done counter PR, but honestly given the stats move back to the cc file may be easier to look at all diffs rather than the delta (which is from your first round of review)

heh. the lack of sleeps appears to be causing some flakes where there were unnoticed races.  Will upload when I have those sorted out.

It's awesome how going into the deterministic world shines a light on the messy dark corners :)
Looking forward to the update and thanks for doing this!

Should be deflaked - PTAL when you get a chance

I'll take a look but something in this PR has significantly increased memory usage for stats, which is why 'released' failed in CI:
   Which is: 52702
  44491

IMO yes: the purpose of this test is to measure memory overhead and I think we should try to do it faithfully. Is it a huge amount of work to plumb some bools through the constructors?

needs format fix.

Thinking about this more, there is no reason to wait for the counter to
exist. You can just create if needed, and return the notifying version.
The advantage is you would avoid exposing the class mutex and condvar.
Wdyt?
On Wed, Jun 3, 2020, 5:47 PM Joshua Marantz<EMAIL_ADDRESS>wrote:

needs format fix.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/envoyproxy/envoy/pull/11289#issuecomment-638479121,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAO2IPJRSPMO3NGHQVYM3T3RU3AIFANCNFSM4NHA2YUQ
.

creating the stat correctly requires the  tag_extracted_name and stat_name_tags which aren't available there.  I'd prefer to not shave more yaks here to avoid a second wait.

Merging master should fix CI.
/wait


[WIP][v1.3]Adding comments to the Browserpanel

Pull Request Test Coverage Report for Build 837

6 of 6 (100.0%)  changed or added relevant lines in 3 files are covered.
14 unchanged lines in 2 files lost coverage.
Overall coverage decreased (-0.9%) to 82.416%

Files with Coverage Reduction
New Missed Lines
%

src/main/java/seedu/address/logic/commands/RegisterCommand.java
6
0.0%

src/main/java/seedu/address/logic/parser/EventManagerParser.java
8
72.41%

Totals

Change from base Build 808:
-0.9%

Covered Lines:
1931

Relevant Lines:
2343

💛  - Coveralls


Fix global filter in dashboard of multi datasources with association

Description

Fix global filter in dashboard of multi datasources with association
Related Issue : #4004

How Has This Been Tested?

See the issue #4004
Need additional checks?
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ ] I have read the CONTRIBUTING document.
[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have added tests to cover my changes.

Additional Context

deploy to 1


⚠ Bofrost.at has degraded performance

In cfcd2ee, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1895 ms

Resolved: Bofrost.at performance has improved in 79b0748.


Fix redirect when only saml allowed

fix for issue #1584

cherry-picked to develop, no longer need this PR.


What should I do if redis has a password

perhaps it's a bit late but we support it for the connection pool that uses Redis Sentinel. For the simple pool you can easily copy the code and implement it, I can accept a PR


Simple fix to not re-running load

Explored dependency injection, which would require importing dagger/hilt and is probably beyond min ship.
How's this for a simple check to prevent repeat calls to load logic?

I would maybe recommend using this logic in the MainActivity, to be clear that load is only meant to run once per app session.
var isInitialized by rememberSaveable { mutableStateOf(false) }
LaunchedEffect(Unit) {
    if (!isInitialized) {
        DefaultSampleInfoRepository.load(context)
        isInitialized = true
    }
}

Apologies for missing your earlier comment on instantiating the json object.
Ready for re-review @sorenoid!


Fix simple typo for other archive URL from comits to commits.

Change the URL for other archive for Apache Twill commits list:
http://twill-commits.markmail.org

LGTM


Exception: Cannot fetch the page from Google Scholar.

Dear developers,
I tried to use your library but I get an error when running the following code:
from scholarly import scholarly
search_query = scholarly.search_pubs('Perception of physical stability and center of mass of 3D objects')
print(next(search_query))

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-4-b0177aafdf62> in <module>
----> 1 search_query = scholarly.search_pubs('Perception of physical stability and center of mass of 3D objects')
      2 print(next(search_query))

~/anaconda3/envs/deep-learning/lib/python3.6/site-packages/scholarly/_scholarly.py in search_pubs(self, query, patents, citations, year_low, year_high)
    185         # improve str below
    186         url = url + yr_lo + yr_hi + citations + patents
--> 187         return self.__nav.search_publications(url)
    188
    189     def search_single_pub(self, pub_title: str, filled: bool = False):

~/anaconda3/envs/deep-learning/lib/python3.6/site-packages/scholarly/_navigator.py in search_publications(self, url)
    500         :rtype: {_SearchScholarIterator}
    501         """
--> 502         return _SearchScholarIterator(self, url)
    503
    504     def search_author_id(self, id: str, filled: bool = False) -> Author:

~/anaconda3/envs/deep-learning/lib/python3.6/site-packages/scholarly/publication.py in __init__(self, nav, url)
     23         self._url = url
     24         self._nav = nav
---> 25         self._load_url(url)
     26
     27     def _load_url(self, url: str):

~/anaconda3/envs/deep-learning/lib/python3.6/site-packages/scholarly/publication.py in _load_url(self, url)
     27     def _load_url(self, url: str):
     28         # this is temporary until setup json file
---> 29         self._soup = self._nav._get_soup(url)
     30         self._pos = 0
     31         self._rows = self._soup.find_all('div', class_='gs_r gs_or gs_scl')

~/anaconda3/envs/deep-learning/lib/python3.6/site-packages/scholarly/_navigator.py in _get_soup(self, url)
    445     def _get_soup(self, url: str) -> BeautifulSoup:
    446         """Return the BeautifulSoup for a page on scholar.google.com"""
--> 447         html = self._get_page('https://scholar.google.com{0}'.format(url))
    448         html = html.replace(u'\xa0', u' ')
    449         res = BeautifulSoup(html, 'html.parser')

~/anaconda3/envs/deep-learning/lib/python3.6/site-packages/scholarly/_navigator.py in _get_page(self, pagerequest)
    256                 self._new_session()
    257
--> 258         raise Exception("Cannot fetch the page from Google Scholar.")
    259
    260     def _check_proxy(self, proxies) -> bool:

Exception: Cannot fetch the page from Google Scholar.

I installed it once via pip install  -U git+https://github.com/OrganicIrradiation/scholarly.git.
Do you have any idea what can I do?
Cheers

Ok, it's due to a blocked connection. A hint in the exception would be helpful 👍 . Thank you!


Truly freeze the eye images when the camera rig type is FREEZE

Oculus-specific; uses a layer flag that disables warping
For https://github.com/Samsung/GearVRf/issues/1157

GearVRf-DCO-1.0-Signed-off-by: Mihail Marinov<EMAIL_ADDRESS>

Planning to merge Tuesday morning.


Error when no headers in request to API gateway

I've got a normal setup, based closely on the example in magnum-cli and terraform docs on AWS lambda.  Using the AWS api gateway test button throws this error:
# loosely translated from the API gateway log
Endpoint response body before transformations:
  errorMessage:
    message: 'NoneType' object has no attribute 'get'
    errorType: AttributeError
    stackTrace: >
      File "/var/task/mangum/adapter.py", line 40, in __call__
        raise exc
      File "/var/task/mangum/adapter.py", line 35, in __call__
        response = self.handler(event, context)
      File "/var/task/mangum/adapter.py", line 50, in handler
        response = handle_http(self.app, event, context)
      File "/var/task/mangum/protocols/http.py", line 58, in handle_http
        server, client = get_server_and_client(event)
      File "/var/task/mangum/utils.py", line 22, in get_server_and_client
        server_addr = event["headers"].get("Host", None)

Adding Accept:application/json to the textbox for headers resolves the issue, and I get my expected { greeting: "hello" } JSON back.

I'm still having this error (with the same workaround of addding the header ) though I'm using 0.7.0

@allan-simon can you post your full traceback using 0.7.0? I can take a look.

{"errorMessage": "'NoneType' object has no attribute 'get'", "errorType": "AttributeError", "stackTrace": ["  File "/var/task/mangum/adapter.py", line 60, in call
raise exc
", "  File "/var/task/mangum/adapter.py", line 58, in call
response = self.handler(event, context)
", "  File "/var/task/mangum/adapter.py", line 72, in handler
response = self.handle_http(event, context)
", "  File "/var/task/mangum/adapter.py", line 84, in handle_http
server, client = get_server_and_client(event)
", "  File "/var/task/mangum/adapter.py", line 24, in get_server_and_client
server_addr = event["headers"].get("Host", None)
"]}

it seems it's only a problem for the first call , and after the second call on api gateway it seems to work without additional headers

adapter.py line 24 yep, the index-style dict access could fail there. Proposal: replace all the index-style dict access with .get(key, with_fallback).

@SKalt  thanks, I've proposed a PR here https://github.com/erm/mangum/pull/84


Review use of BigNumber

Currently we use json-bigint to parse JSON replies from the Watcher, which gives us BigNumber types.
However, BigNumber doesn't really play well with the bn.js, which is what web3 uses.
It would be nice to switch to using bn.js everywhere

I'm not aware that bg.js is used by web3.. it might be wise to stick to other web3 dependencies moving forward as well (as far as Root Chain lib goes)


Unable to build googleapis library with local gapic-generator-java

Error Message for building local library:
Starting local Bazel server and connecting to it...
DEBUG: Rule 'com_googleapis_gapic_generator_go' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "d9fa55ef3bc14e1c3c15870ef1080a29e6af2b996e37a1d7043f35a61aa1e869"
DEBUG: Repository com_googleapis_gapic_generator_go instantiated at:
  /Users/lawrenceqiu/IdeaProjects/googleapis/WORKSPACE:168:13: in <toplevel>
Repository rule http_archive defined at:
  /private/var/tmp/_bazel_lawrenceqiu/e651f412bb9d781d683087b8b31b9bbe/external/bazel_tools/tools/build_defs/repo/http.bzl:372:31: in <toplevel>
INFO: Repository com_google_api_gax_java instantiated at:
  /Users/lawrenceqiu/IdeaProjects/googleapis/WORKSPACE:289:13: in <toplevel>
Repository rule http_archive defined at:
  /private/var/tmp/_bazel_lawrenceqiu/e651f412bb9d781d683087b8b31b9bbe/external/bazel_tools/tools/build_defs/repo/http.bzl:372:31: in <toplevel>
WARNING: Download from https://github.com/googleapis/gapic-generator-java/archive/v2.15.4-SNAPSHOT.zip failed: class java.io.FileNotFoundException GET returned 404 Not Found
ERROR: An error occurred during the fetch of repository 'com_google_api_gax_java':
   Traceback (most recent call last):
	File "/private/var/tmp/_bazel_lawrenceqiu/e651f412bb9d781d683087b8b31b9bbe/external/bazel_tools/tools/build_defs/repo/http.bzl", line 132, column 45, in _http_archive_impl
		download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://github.com/googleapis/gapic-generator-java/archive/v2.15.4-SNAPSHOT.zip] to /private/var/tmp/_bazel_lawrenceqiu/e651f412bb9d781d683087b8b31b9bbe/external/com_google_api_gax_java/temp11953110264635311995/v2.15.4-SNAPSHOT.zip: GET returned 404 Not Found
ERROR: /Users/lawrenceqiu/IdeaProjects/googleapis/WORKSPACE:289:13: fetching http_archive rule //external:com_google_api_gax_java: Traceback (most recent call last):
	File "/private/var/tmp/_bazel_lawrenceqiu/e651f412bb9d781d683087b8b31b9bbe/external/bazel_tools/tools/build_defs/repo/http.bzl", line 132, column 45, in _http_archive_impl
		download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://github.com/googleapis/gapic-generator-java/archive/v2.15.4-SNAPSHOT.zip] to /private/var/tmp/_bazel_lawrenceqiu/e651f412bb9d781d683087b8b31b9bbe/external/com_google_api_gax_java/temp11953110264635311995/v2.15.4-SNAPSHOT.zip: GET returned 404 Not Found
ERROR: Error computing the main repository mapping: no such package '@com_google_api_gax_java//': java.io.IOException: Error downloading [https://github.com/googleapis/gapic-generator-java/archive/v2.15.4-SNAPSHOT.zip] to /private/var/tmp/_bazel_lawrenceqiu/e651f412bb9d781d683087b8b31b9bbe/external/com_google_api_gax_java/temp11953110264635311995/v2.15.4-SNAPSHOT.zip: GET returned 404 Not Found

Do you still have the issue if this line is commented out?

Do you still have the issue if this line is commented out?

Error:
Starting local Bazel server and connecting to it...
DEBUG: Rule 'com_googleapis_gapic_generator_go' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "d9fa55ef3bc14e1c3c15870ef1080a29e6af2b996e37a1d7043f35a61aa1e869"
DEBUG: Repository com_googleapis_gapic_generator_go instantiated at:
  /Users/lawrenceqiu/IdeaProjects/googleapis/WORKSPACE:168:13: in <toplevel>
Repository rule http_archive defined at:
  /private/var/tmp/_bazel_lawrenceqiu/e651f412bb9d781d683087b8b31b9bbe/external/bazel_tools/tools/build_defs/repo/http.bzl:372:31: in <toplevel>
INFO: Repository com_google_api_gax_java instantiated at:
  /Users/lawrenceqiu/IdeaProjects/googleapis/WORKSPACE:288:13: in <toplevel>
Repository rule http_archive defined at:
  /private/var/tmp/_bazel_lawrenceqiu/e651f412bb9d781d683087b8b31b9bbe/external/bazel_tools/tools/build_defs/repo/http.bzl:372:31: in <toplevel>
WARNING: Download from https://github.com/googleapis/gapic-generator-java/archive/v2.15.4-SNAPSHOT.zip failed: class java.io.FileNotFoundException GET returned 404 Not Found
ERROR: An error occurred during the fetch of repository 'com_google_api_gax_java':
   Traceback (most recent call last):
	File "/private/var/tmp/_bazel_lawrenceqiu/e651f412bb9d781d683087b8b31b9bbe/external/bazel_tools/tools/build_defs/repo/http.bzl", line 132, column 45, in _http_archive_impl
		download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://github.com/googleapis/gapic-generator-java/archive/v2.15.4-SNAPSHOT.zip] to /private/var/tmp/_bazel_lawrenceqiu/e651f412bb9d781d683087b8b31b9bbe/external/com_google_api_gax_java/temp1286758972211290600/v2.15.4-SNAPSHOT.zip: GET returned 404 Not Found
ERROR: /Users/lawrenceqiu/IdeaProjects/googleapis/WORKSPACE:288:13: fetching http_archive rule //external:com_google_api_gax_java: Traceback (most recent call last):
	File "/private/var/tmp/_bazel_lawrenceqiu/e651f412bb9d781d683087b8b31b9bbe/external/bazel_tools/tools/build_defs/repo/http.bzl", line 132, column 45, in _http_archive_impl
		download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://github.com/googleapis/gapic-generator-java/archive/v2.15.4-SNAPSHOT.zip] to /private/var/tmp/_bazel_lawrenceqiu/e651f412bb9d781d683087b8b31b9bbe/external/com_google_api_gax_java/temp1286758972211290600/v2.15.4-SNAPSHOT.zip: GET returned 404 Not Found
ERROR: Error computing the main repository mapping: no such package '@com_google_api_gax_java//': java.io.IOException: Error downloading [https://github.com/googleapis/gapic-generator-java/archive/v2.15.4-SNAPSHOT.zip] to /private/var/tmp/_bazel_lawrenceqiu/e651f412bb9d781d683087b8b31b9bbe/external/com_google_api_gax_java/temp1286758972211290600/v2.15.4-SNAPSHOT.zip: GET returned 404 Not Found

I updated to this:
_gapic_generator_java_version = "2.15.4-SNAPSHOT"

maven_install(
    artifacts = [
        "com.google.api:gapic-generator-java:" + _gapic_generator_java_version,
    ],
    #Update this False for local development
    fail_on_missing_checksum = False,
    repositories = [
        "m2Local",
        "https://repo.maven.apache.org/maven2/",
    ]
)

local_repository(
    name = "gapic_generator_java",
    path = "/Users/lawrenceqiu/IdeaProjects/gapic-generator-java",
)

# gax-java is part of gapic-generator-java repository
local_repository(
    name = "com_google_api_gax_java",
    path = "/Users/lawrenceqiu/IdeaProjects/gapic-generator-java/gax-java",
)

and this works find to build

@blakeli0
This changed worked for me (not sure if there is a better way):
_gapic_generator_java_version = "2.15.4-SNAPSHOT"

maven_install(
    artifacts = [
        "com.google.api:gapic-generator-java:" + _gapic_generator_java_version,
    ],
    #Update this False for local development
    fail_on_missing_checksum = False,
    repositories = [
        "m2Local",
        "https://repo.maven.apache.org/maven2/",
    ]
)

local_repository(
    name = "gapic_generator_java",
    path = "/Users/lawrenceqiu/IdeaProjects/gapic-generator-java",
)

# gax-java is part of gapic-generator-java repository
local_repository(
    name = "com_google_api_gax_java",
    path = "/Users/lawrenceqiu/IdeaProjects/gapic-generator-java/gax-java",
)


Update Make predictions with a trained model

Add note about PredictionEnginePool
Review outdated content and update as necessary

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 30e7cf89-1a90-7693-db50-11b770c09160
Version Independent ID: 21c26fd2-c6ad-2c2f-166b-03ac0590a1c2
Content: Make predictions with a trained model - ML.NET
Content Source: docs/machine-learning/how-to-guides/machine-learning-model-predictions-ml-net.md
Product: dotnet-ml
GitHub Login: @luisquintanilla
Microsoft Alias: luquinta

Merged changes as part of #25540


Add some enhancements to monument modes boss bar

This should fix be the bar being deleted and sent at the same time, when all modes has ran. I'd like anyone to confirm testing

:+1:

@Pablete1234 If you could confirm if this fixes your issue, I'll merge this.

@twizmwazin it works, but I think the wither distance from the player should be bumped up to prevent the player seeing the particles

@Pablete1234 yes like that : http://i.unixfox.eu/2015-09-01_15-07-05.gif

@unixfox like that, and also while falling from high places

@Pablete1234 It needs to be kept at a certain distance so users even with 2 chunks render distance can see the boss bar.

@twizmwazin maybe the server should check the render distance of the client to see if it's really small, and if so leave it like that, or else move it away 50 blocks
idk if it's possible, but i don't think the server sends more chunks than what the client requests, so i think it should be possible

@twizmwazin I believe Overcast keeps it at 32 blocks away (2 chunks) but I'm not sure if there's more problems with not seeing the boss bar when you make your render distance that low.

@iEli2tyree011 they have it 50 blocks away (when we had tm i did the test), look straight up, /minecraft:tp ~ ~45 ~ and you could see the particles

@Pablete1234 It's probably at 48.

I don't think this is related with the actual issue, but well. Should I change it to 48 blocks far or keep it as it's now (32)?

@Alfalik Unless you can find a reliable way to get client render distance, keep it at 32 blocks.

@Alfalik how about the pos of the wither instead of being (pos of the player + vector of where he is looking32) be (pos of the player + vector of where he is looking32 + vector of movement of the player)
that would mean, if the player is moving down, the wither would spawn lower, if player is moving foward, the wither will spawn foward

I won't have the time to code or use a computer until 17 of september (I'm going to the United States). Hope you can code a thing that get the render distance of a player @Pablete1234.

@Alfalik this PR fixes monument modes, but not core modes

Aren't they the same?

@Alfalik they should be, but it still happens on monument modes with this PR

@twizmwazin @Alfalik when shooting you sometimes see the arrow fall down (because it's hitting the wither client-side) and it is a very annoying bug, what could fix it? cough 48blocks cough either that or moving the player a bit up/down so it's not on the sight of arrows

@Pablete1234 Create a new issue. I don't think this is related with the actual issue, besides, I won't fix the arrows hiting the fake wither or you seeing the particles, because I don't have the time to. Thanks
Back to the topic, does this pull request fix the actual issue?

@Alfalik that was on a server that i had putted a cardinal ver with this fix, but someone else "updated" cardinal to a version without it
This PR does fix the wither blinking issue, but it's called "enhancements" so yeah, another enhancment won't hurt anyone

@Pablete1234 In-case you will fix the wither particles being rarely and arrows hiting the wither, use this http://wiki.vg/Protocol#Client_Settings

@Alfalik the other common issue in bossbars atm is that long tl bossbars don't get shown the last 5 seconds

It's a math problem. This is the actual formula:
int percent = (int) ((100F * timeLeft) / TimeLimit.getMatchTimeLimit());

If the int timeLeft is a low number compared with a high number (in this case a long time limit as you said), it'll return to 0F, and the bar won't be show. Here's a clearly example with a 3 hours time limit:
int percent = 5s / 10800s = 4,69x10^-4 = 0,000469 We'll ignore the percent, we want the number

It's a very little number, so it will be approximate to 0.

merged

As an additional note, to fix @Pablete1234's mentioned issue about the bar dissapearing, mathematically it makes sense to approximate to 0, however the health should always be kept at 1, even if it should be less than this, to keep showing the bar. Someone will need to do this.

@twizmwazin So basically if health=0 add one?

@SungMatt Yeah, pretty much. Not really anything complicated.


[Migrated] Slow import of zappa.asynchronous due to session.client() calls

Originally from: https://github.com/Miserlou/Zappa/issues/2072 by schuyler1d
Context
Merely importing from zappa.asynchronous import run takes several seconds due to the aws_session.client('sns'), etc. at the top of the page.  It's a worthy goal to keep these client sessions global/in-memory, precisely because of this delay, however we should only tax the delay if/when we actually need those sessions.
Expected Behavior
from zappa.asynchronous import run should be fast/immediate
Actual Behavior
It takes several seconds for the session to 'initialize'
Possible Fix
I propose we memoize the sessions instead.
Steps to Reproduce

Your Environment

Zappa version used: 0.51.0

Operating System and Python version: 3.6 (ubuntu/linux)

The output of pip freeze:
argcomplete==1.11.1
asgiref==3.2.3
boto3==1.12.4
botocore==1.15.4
certifi==2019.11.28
cfn-flip==1.2.2
chardet==3.0.4
Click==7.0
coverage==5.0.3
coveralls==1.11.1
Django==3.0.3
docopt==0.6.2
docutils==0.15.2
durationpy==0.5
entrypoints==0.3
flake8==3.7.9
Flask==1.1.1
future==0.18.2
hjson==3.0.1
idna==2.9
importlib-metadata==1.5.0
itsdangerous==1.1.0
Jinja2==2.11.1
jmespath==0.9.4
kappa==0.6.0
MarkupSafe==1.1.1
mccabe==0.6.1
mock==4.0.1
nose==1.3.7
nose-timer==0.7.5
pip-tools==4.5.0
placebo==0.9.0
pycodestyle==2.5.0
pyflakes==2.1.1
python-dateutil==2.6.1
python-slugify==4.0.0
pytz==2019.3
PyYAML==5.3
requests==2.23.0
s3transfer==0.3.3
six==1.14.0
sqlparse==0.3.0
text-unidecode==1.3
toml==0.10.0
tqdm==4.43.0
troposphere==2.5.3
urllib3==1.25.8
Werkzeug==0.16.1
wsgi-request-logger==0.4.6
zipp==3.0.0

Link to your project (optional):

Your zappa_settings.json:

duplicate:
https://github.com/zappa/Zappa/issues/832


blog.BlogCategory.None

I want to display blog Post category name in this way in templates
{{ blog_post.categories }}
but is display blog.BlogCategory.None instead of category name

{{ blog_post.categories.all }} should do the trick. Have a look at blog/templates/blog/blog_post_list.html line 89 ff.
For further questions like this lets move to https://groups.google.com/forum/#!forum/mezzanine-users.


feature request: ignore EXTERNAL LINK updates unless URL changes

What is the problem?
The problem is, if I click on “Update now!” more or less frequently, i.e. daily (or weekly), then after each update I see a few titles of new or updated chapters that reside directly on “Baka-tsuki” and then literally hundreds of “Updated Chapter” messages from EXTERNAL LINK chapters.
And it's not like these external chapters really are daily (or weekly) updated in hundreds. They are actually just hosted on sites that use engines that generate pages on-the-fly (such as Wordpress engine) with new banners, new voting blocks, new (randomly chosen) colorful page headers, and then new (current) datetime headers that force web browsers to refrain from using any old (cached) versions of those web pages.
Any real updates become totally buried in these hundreds of (mostly false) messages of updated external chapters.
Therefore I need a new option that would force BakaReader EX to completely ignore updates of EXTERNAL LINK chapters (unless something exceptional, such as an URL change, happens). This option should also prevent BakaReader EX from querying external sites for dates of pages (and thus should greatly reduce the time that “Update Now!” spends from its start to its finish).

Try Settings -> Update and Notification -> Include External Chapter = False?
If the url changes, most likely it will detected as new chapter instead, cos no wiki id I can use to check whether this is new chapter or not.

Thanks. (I have totally forgotten that existing “Include external chapter” setting, sorry.)

Reopening.
I have an impression that, while unchecking “Include External Chapter” indeed ceases messages about new (=updated) external chapters, it also shuts down messages about any appearance of new (=previously nonexistent) external chapters.
Is that impression correct?

No, that impression is not correct.
Some chapters just appeared on http://saekanothetranslation.wordpress.com/ before they were referenced from Baka-Tsuki.
I now see them in BakaReader EX as “New Chapter” / “EXTERNAL LINK” entries as well.
This issue can now be closed.


Metacritic Entry Not Found

Hi,
depending on the games, it might happen that no metacritic entry is available or that they simply cannot be found. If that happens the sensor cant be updated or setup at all if this happens after adding the integration.
I suggest to implement a fallback if the metacritic cannot be found. Unfortunatly, I rarely coded in Python at all. But I think it would be sufficient to wrap these lines with a try-catch (like the one for failed parsing).
search_url = f"https://www.metacritic.com/search/game/{game_name}/results?plats[3]=1&search_type=advanced"
response = requests.get(search_url, headers=self.headers)
response.raise_for_status()

The URL that currently cannot be found: https://www.metacritic.com/search/game/Tandem:%20A%20Tale%20of%20Shadows/results/?plats%5B3%5D=1&search_type=advanced
best regards,
SpiGAndromeda

Can confirm. Same for me:
Logger: homeassistant.components.sensor
Source: helpers/entity_platform.py:573
Integration: Sensor (documentation, issues)
First occurred: 15:48:13 (1 occurrences)
Last logged: 15:48:13

epic_games: Error on device update!
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 573, in _async_add_entity
    await entity.async_device_update(warning=False)
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 993, in async_device_update
    await hass.async_add_executor_job(self.update)
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/epic_games/sensor.py", line 176, in update
    [
  File "/config/custom_components/epic_games/sensor.py", line 182, in <listcomp>
    rating=self.get_score_metacritic(game.get("title")),
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/epic_games/sensor.py", line 139, in get_score_metacritic
    response.raise_for_status()
  File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://www.metacritic.com/search/game/Turnip%20Boy%20Commits%20Tax%20Evasion/results/?plats%5B3%5D=1&search_type=advanced

Hi,
The issue is still with us:
File "/config/custom_components/epic_games/sensor.py", line 139, in get_score_metacritic     response.raise_for_status()   File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status     raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url
URL pointing to:
https://www.metacritic.com/search/game/Surviving the Aftermath/results/?plats[3]=1&search_type=advanced
URL if i search the game:
https://www.metacritic.com/game/surviving-the-aftermath/


Support decode SM2 parameters

Checklist

[x] tests are added or updated

This PR is in a state where it requires action by @openssl/otc but the last update was 30 days ago

This pull request is ready to merge

Merged, thanks the addition.


First time logging in in 4 months and get a scam mail within couple hour

hello,
so i have no idea where to report any of this or if this is runelite related but
for the first time in 4 months i come back to osrs with runelite (i logged in about a month ago in rs3 same account)
and within a couple hours i receive a scam mail involving osrs
now i dont believe in coincidence so either my computer got infected with something today
or runelite might have a email data breach

Well, we are not sending scam emails to anyone, and we do not have any "email data breach", so you either have wrong version of runelite, or its simply coincidence, idk. Regardless of that, this is issue tracker, so closing


Error when building app: Could not find com.ad4screen.sdk:A4SSDK:3.7.2

OS: Android only (iOS version builds without errors)
I am building/distributing my android app through microsofts appcenter.
About 3 weeks ago I executed a build which worked fine and without any errors. I now try to execute a new build, without any change to the dependencies or their usage, but now it throws an error each time and I am unable to complete the build.
The dependencies are located in app/build.gradle:
    compile'com.ad4screen.sdk:A4SSDK:3.7.2'
    compile('com.ad4screen.sdk:plugin-play-services:3.1.1')

And this doesn't happen when I build locally through android studio
Error from buildserver:
FAILURE: Build failed with an exception.

* What went wrong:
A problem occurred configuring project ':app'.
> Could not resolve all dependencies for configuration ':app:_debugApkCopy'.
   > Could not find com.ad4screen.sdk:A4SSDK:3.7.2.
     Required by:
         project :app
   > Could not find com.ad4screen.sdk:A4SSDK:3.7.2.
     Required by:
         project :app > com.ad4screen.sdk:plugin-play-services:3.1.1 > com.ad4screen.sdk:plugin-play-services-advertiserid:3.1.1
         project :app > com.ad4screen.sdk:plugin-play-services:3.1.1 > com.ad4screen.sdk:plugin-play-services-geofence:3.1.1
         project :app > com.ad4screen.sdk:plugin-play-services:3.1.1 > com.ad4screen.sdk:plugin-play-services-location:3.1.1
         project :app > com.ad4screen.sdk:plugin-play-services:3.1.1 > com.ad4screen.sdk:plugin-play-services-push:3.1.1
         project :app > com.ad4screen.sdk:plugin-play-services:3.1.1 > com.ad4screen.sdk:plugin-play-services-advertiserid:3.1.1 > com.ad4screen.sdk:plugin-play-services-base:3.1.1

I am a bit lost on this because it seems to be unable to find the dependency even though it is located in the app/build.gradle file, and since building works locally and has worked through the appcenter in the recent past.

After reading the changelogs of your SDK, it seems that version 3.7.2 has been removed completely and 3.8.0 should be used instead.
Feel free to close the issue.


Conversation greyed out even when containing a new email.

This is the counterpart of #79 :
I'm having a folder, github, which was marked with a (1) but by clicking on it to open this folder, all conversations (only 5 conversations) were displayed "grey" (meaning: no new mail in the converation) not "black and white" (meaning : there's an unread email in this thread).
I had to click on all conversations to see which one (the last one obviously, the one on the top, but I started from the bottom) to see which one contained a new message. I was more seraching for a way to drop the (1) than to find an unread messages, as I though I faced issue #79 at first.
Desktop (please complete the following information):

OS: Debian
Browser Firefox
Version 61.0.1
URL mail.protonmail.com

So in this case the (1) was correct  but the conversation highlighting was off?

Yes the (1) was right as far as I can tell [1], but the conversation was not highlited properly, it was grey like a conversation with no unread messages.
[1]: My brain first expected #79 when I spotted it so there's a little chance I got confused. Very little but existing, I should video-capture everything þ

Got the bug today again, with a screenshot this time:

Here's the (1) near github is right, I'm having one unread email in the OsmAnd is slow on Android O conversation.
While opening this conversation, the (1) next to github dissapeared while the last mail (the unread one) showed-up.

Ignoring the sidebar count for a moment, was the unread message in the github folder or in inbox? I ask because conversations can include messages from different folders, whereas the sidebar just counts unread for that folder/label, i.e., they are context dependent. So you could have a conversation with an unread message in inbox, all messages in 'github' read, and it would show up as grey (no unread messages) in the github folder listing.

The unread message was the last one of a "thread" of emails in the github folder.

So you could have a conversation with an unread message in inbox, all messages in 'github' read, and it would show up as grey (no unread messages) in the github folder listing.

I don't think it's my case as my github folder is a part of my sieve filter:
[...]
elsif header :contains "list-id" ".github.com>" {
    fileinto "github";
}
[...]

Yes, I ocasionally move emails from my github folder to my inbox (a way for me to "mark them as TODO" / to make them more visible when I have something to do with them), but I don't mark them as unread when I do, and I had nothing "todo" with this email, so it can't be the case.

OK, just wanted to make sure.

Spotted it again today with a specific configuration:

The thread was all in the Trash directory
Except for the unread email which was in the github directory.
The conversation was showing in the github directory, greyed as when read, marked as containing 3 emails, and containing the icons of the directory and the icon of the trash.
When clicking the conversation, the single non-trashed unread email showed up, correctly prefixed by a "This conversation contains trashed messages", the (1) unread flag correctly dissapeared.

I do not exclude that all previous cases were similar (whole conversation in trash except a single unread email in folder).

@JulienPalard Unread counter is specific per folder. So if you have a thread in Trash and one of the message is unread and is in github, unread counter equals 0 for this conversation.
If you go to github folder and check this conversation, then it will be marked as unread.

Hi @EpokK, I do not understand the wontfix label, my experience is:

Open mailbox
See folder with (1) unread message, click on it.
See 10 conversations, no conversation are maked with unread message, have to click on every one of them to find the hidden unread message.

Can this really be considered a feature? From my user experience, it's clearly a problem.
see:


package cleanup

Coverage decreased (-0.03%) to 75.31% when pulling bbc30175319f8caccec3f5b4103f2138bb0327bc on hundeboll/package-cleanup into 3c6f39e72f719b0b8a3bf8ffeb07754d2a42e911 on master.

Coverage decreased (-0.03%) to 75.31% when pulling bbc30175319f8caccec3f5b4103f2138bb0327bc on hundeboll/package-cleanup into 3c6f39e72f719b0b8a3bf8ffeb07754d2a42e911 on master.

ping

@r-rojo @tswhison @anandaravuri please review (again)
I've rebased on current master, and added commits to use opae_python_install() for opae.admin and pacsign.
Thanks.

Rebased on current master to fix conflict from https://github.com/OPAE/opae-sdk/pull/1897

@michelleyho @r-rojo bump?


Extra  in inner html

I am sure that I have must be doing something wrong.  I am attempting to replace the inner html of an existing div with new content.
IHtmlDocument myDocument = new HtmlParser().Parse(myFileStream);
IElement e = myDocument.GetElementById("id");
e.InnerHtml = newHtml;

I look at newHtml in the debugger and it is fine.
I look at e.InnerHtml and it has newHtml with extra  appended to the end

Can you work out a MWE? Distill the original source and the text represented by newHtml. I'll look into it then.

I have attached a repro.  The webpage and replacement html are files in the project and they copy to the output folder so all you have to do is run the project.
Thanks
InnerHtmlRepro.zip

Two things:

If you want support / keep AngleSharp alive then star it (its the currency on GitHub; repos then are not starred are abandoned)
Next time please try to create a real MWE (as concise as possible) and bring everything together in a single source (please do not upload zip files!)

So this is the MWE for the described scenario:
var parser = new HtmlParser();
var document = parser.Parse("<!DOCTYPE html>

");
var element = document.GetElementById("outputPanel");
element.InnerHtml = "

";
Console.WriteLine(element.InnerHtml);

Yes, looks like a bug to me. I will investigate and fix it asap.

I wanted you to have my data, just in case it was somehow data specific.
What you reduced this to is indeed the very basic
Thanks

Roy Chastain
From: Florian Rappl<EMAIL_ADDRESS>Sent: Tuesday, February 2, 2016 08:52
To: AngleSharp/AngleSharp<EMAIL_ADDRESS>Cc: Roy Chastain<EMAIL_ADDRESS>Subject: Re: [AngleSharp] Extra  in inner html (#260)
So this is the MWE for the described scenario:
var parser = new HtmlParser();
var document = parser.Parse("");
var element = document.GetElementById("outputPanel");
element.InnerHtml = "";
Console.WriteLine(element.InnerHtml);
Yes, looks like a bug to me. I will investigate and fix it asap.
—
Reply to this email directly or view it on GitHubhttps://github.com/AngleSharp/AngleSharp/issues/260#issuecomment-178582488.


How to use self trained weight testing videos

I use my trained weights in 'tracking/run_ video_ demo.sh' Modified the weight path in demo. sh and tested the video by executing 'tracking/run_ video_ demo. sh ', why did I find that I still use pre trained weights? Because whether using pre trained weights or my own weights, the coordinates of the output bbox results are exactly the same, even accurate to 9 decimal places. This is unreasonable, and what is the reason?

Using pre trained weights and one's own weights, the predicted bbox for each frame only has different confidence levels, resulting in identical output results


ERROR: Method 'load:' not defined in Plugin 'OpenWithPlugin'

Hello, I get this error under IONIC v3, Platform iOS.
The handler get this intent item:
{"name":"","uti":"public.jpeg","utis":["public.jpeg"],"uri":"file:///private/var/mobile/Containers/Shared/AppGroup/9A08219E-0506-4667-8D2E-287DB167B89B/Library/Caches/82DF64B5-4476-422B-AE54-DD6C3CBC5CE0..com.apple.Foundation.NSItemProvider.ZiNZID.jpeg","type":"image/jpeg"}
but I get an error at this line:
`
(window).plugins.openwith.load(intent.items[0], function (data, item) {
// data is a long base64 string with the content of the file
console.log("the item weights " + data.length + " bytes");
alert("uploadToServer(item)");
        // "exit" when done.
        // Note that there is no need to wait for the upload to finish,
        // the app can continue while in background.
        if (intent.exit) {
          (<any>window).plugins.openwith.exit();
        }
      });

`
thanks

Do you referring to the wrong object on your window variable ?
In the config xml the target for this plugin is cordova.openwith and not plugins.openwith.
Have a look here:
https://github.com/j3k0/cordova-plugin-openwith/blob/master/plugin.xml#L48
Try to change your code like this:
(window).cordova.openwith.load(intent.items[0], function (data, item) ...

also you can declare a global cordova variable like this:

declare var cordova:any;

then you can write you code like this:
 cordova.openwith.load(intent.items[0], function (data, item) ...


Active Subscription type(monthly/Yearly) required

Describe the bug
The subscription plan type (monthly/yearly) is not being saved for active subscriptions. As a result, when a user has an active subscription, the plan status is shown as active for both Monthly and Yearly plans, regardless of the actual subscription type.

[x] Ensure the active plan type (monthly/yearly) is saved correctly.
[x] Display the active plan based on the correct selected plan type.

Created new subscriptions for android to differentiate between yearly and monthly plans


Full screen mode

Please, add button "full screen mode".

+++

Please, add buttom "full screen mode".

Here is a self-implement solution for anyone looking, requires Screenfull.js
To add it, add fullscreen somewhere into your Option's buttons element. After that you need to add the following code into your allButtons element of the Options object.
fullscreen: {
	title: "Fullscreen",
	buttonText: "Fullscreen",
	cmd: function() {
		if (screenfull.enabled) {
			screenfull.toggle($('.wysibb')[0]);
		}
	}
}


Tests also have to be re-configured after the structure

[x] Tests also have to be re-configured after the structure

[x] All the Tests has to pass

Only 3 tests are failing and have to be fixed.

all tests are passing.
can be closed


Change columnProperties to rowProperties

I think the second parameter of the formatter functions contain details (/ data) about the row instead of the column.

Thanks!


Missing attributes on VideoGrant

Hi, thanks for building this SDK. 💯
My problem is that I want to create a camera-only token and send back to client, in the doc I have read that we can use the attribute canPublishSources, I found it in the NodeJS package but for our python we don't have that. Am I missing something ?

I have created a quick PR fix for that, please help me to verify that. Thank you.
https://github.com/livekit/livekit-server-sdk-python/pull/10


Add Authentication type column in table azure_app_configuration

Is your feature request related to a problem? Please describe.
Add Authentication type column in table azure_app_configuration
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

https://github.com/Azure/azure-sdk-for-go/issues/21404


Feature/client rewrite

:tada: This PR is included in version 0.4.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


file_system: cleanup eBPF on server shutdown instead of on thread exit

Signed-off-by: Yi Wu<EMAIL_ADDRESS>What problem does this PR solve?
Previously we count the threads using biosnoop and cleanup the eBPF when the count drops to 0. This fails to take into account the main thread may also need to access biosnoop and the thread IdxWrapper does not get drop after program exit. This patch change to let TiKV server cleanup eBPF on shutdown. This change also fixes the broken biosnoop unit test.
It is still a problem if TiKV gets killed and the eBPF won't be cleaned up, though.
Problem Summary:
What is changed and how it works?
What's Changed: This patch change to let TiKV server cleanup the eBPF for biosnoop on shutdown.
Related changes
No
Check List
Tests
build and run test in environment with bcc support and see the test pass. https://gist.github.com/yiwu-arbug/35f52102defe25a7257f1a5082b54e0f
Release note

No release note

/test

/test