FTBFS with openssl 1.1.0

voms does not compile with openssl 1.1.0, as reported to Debian BTS:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=828595

Hello Mattias,
this is a known issue, but we hope to fix with the next VOMS release (that will just provide Openssl 1.1.0 compatibility) and which is scheduled for mid-October (i.e. in a month from now).

I know you are aware of it. I just wanted to keep track of it and be able to mark the BTS bug as forwarded.

Hi!
Debian just increased the severity of the bug in BTS so it is now release critical. If not fixed before 2016-11-25, the package - and those depending on it - will be removed.

Hi Mattias,
we're working on it. I think we can make a release before the 2016-11-25 deadline.

Hi Andrea,
could let me know when you have a reasonably working development version, so that I can also runtime check LCMAPS and the like, which has the same deadline ?

I will.
A.

Hi Mattias
I've started the porting, but I'm encountering a few problems with the new OpenSSL API that I'm not able to solve, even after looking into the source code. What's the best place to ask for some assistance? Should I put a comment in the original Debian bug? Can I contact directly the reporter of that bug (Kurt Roeckx)? something else?

Hi Francesco,
in my experience the best is to ask on the openssl-dev mailing list. What type of issues do you encounter? Most of the changes are now opaque struct members, for which there are mostly getters, plus the deprecation of the asn1_mac.h macros. Perhaps Mattias has better suggestions?

Hi Mischa, the key of the problem is the word "mostly" :-) I can't find a way to express what is at https://github.com/italiangrid/voms/blob/master/src/socklib/Server.cpp#L362. I'm writing some notes on my attempts; I'll post them later.
The removal (not only deprecation) of the asn1_mac.h macros is another problem that I've not yet addressed. But in the worst case I can always copy those macros into the voms code base and remove them later.

Hi Francesco,
from looking around a bit in the openssl-1.1 code, e.g. https://github.com/openssl/openssl/blob/OpenSSL_1_1_0b/crypto/bio/bss_sock.c#L56-L65, I would think it's better to run BIO_new() with a custom BIO_METHOD (created using BIO_meth_new()), for which you can get/set the struct elements using BIO_meth_get_write()/BIO_meth_set_write() etc.
Not sure if this is the best way, but looks doable and reasonable.
I would certainly not try to reinsert the deprecated asn1_mac.h macros...
I updated lcmaps-plugins-verify-proxy but that needed only subset (https://ndpfsvn.nikhef.nl/viewvc/mwsec/trunk/lcmaps-plugins-verify-proxy/src/verify-proxy/verify-lib/src_internal/_verify_proxy_certinfo.c?annotate=18683 in particular after lines 87 and 143).
Globus has updates in https://github.com/globus/globus-toolkit/blame/globus_6_branch/gsi/proxy/proxy_ssl/source/library/proxycertinfo.c and that should probably have anything you need concerning the asn1 macro replacements.

That's what I'm trying to do, but the API doesn't use const coherently, so, for example, the getter BIO_meth_get_write(BIO_METHOD *biom) takes a non-const BIO_METHOD; so I cannot use what I get back from const BIO_METHOD *BIO_s_socket(void). Unless I use a cast, of course, which is what I'll do since I know from the code that the getter doesn't change it, but it's not clean. Apart from that, I'm not sure which type and name to use for the call to BIO_METHOD *BIO_meth_new(int type, const char *name); I would like to take them from what BIO_s_socket gives me, but there are no getters available for that, they exist only for the BIO that uses that BIO_METHOD!
Thanks for the hints on the macros.

That's what I'm trying to do, but the API doesn't use const coherently. For example BIO_meth_get_write(BIO_METHOD *biom)) takes a non-const parameter, so I cannot pass what const BIO_METHOD *BIO_s_socket(void); returns. I can (and will) use a cast in this case, because I know from the source code that the argument is not modified, but it's not clean.
Then, I'm not sure what to pass as type and name to BIO_METHOD *BIO_meth_new(int type, const char *name). I would like to take them from the BIO_METHOD returned by BIO_s_socket, but there are no getters for those; curiously they exist for a BIO, but I would prefer not to build a temporary BIO just to get those.
Thanks for the hints on the macros.

If I understand correctly, BIO_s_socket() returns the default  BIO_METHOD which is rightfully const, but you want to make your own, since you want to adjust the bwrite/bread functions, so you need to call BIO_meth_new with the next free type, which you can get using BIO_get_new_index() plus a new name, e.g. "Globus I/O" or something like that ?

That's also what I'm doing for the type in fact, but I'm not sure that's correct. Thanks for confirming that that's the right way.
About const, I agree that BIO_s_socket() does the right thing; it's the various getters that seem wrong (from what I understand).
Anyway, from this discussion I think I can move forward. Thanks.

Note that I'm not a real expert, it's also for me just looking at code and manpages...
What do you think is wrong with the various getters (and which ones actually?).
You can always ask on the openssl-dev mailing list...

This is what I want to do
#include <openssl/bio.h>

int main()
{
  int const type = BIO_get_new_index();
  BIO_METHOD* voms_biom = BIO_meth_new(type, "VOMS I/O");
  BIO_METHOD const* sock_biom = BIO_s_socket();
  BIO_meth_set_write(voms_biom, BIO_meth_get_write(sock_biom));
  /* BIO_meth_set_write(voms_biom, BIO_meth_get_write((BIO_METHOD*)sock_biom)); */
}

But it doesn't compile. It does if you apply a cast to sock_biom to remove the const. Not nice.

I agree that the signatures of the getters should use const. The getters do not modify the object.

Ah, I think I agree: the BIO_meth_get_write() should have a const BIO_METHOD * argument, since it does not change it. Likewise obviously for the BIO_meth_get_read().
That's probably something to report to openssl (as issue via github).

We agree (-; For the time being an explicit cast should be harmless.

I've just stumbled on this progress and ticket. If it helps for the sslutils part, take a look at canl-c, the openssl-1.1 branch where we pushed the necessary changes earlier this week.

Thanks, Daniel. I've already done sslutils part, but I'll compare with what you have done to see if I've made any mistake. Unfortunately the missing part concerns the contents of proxycertinfo.c, which you have in your repo but apparently don't use so you didn't need to do the porting.

Hi Francesco,
I would certainly also look at the Globus version (see my previous comment above) which contains also a proxycertinfo.c, although the VOMS and Globus versions have diverged quite a bit over the years, you can still get a reasonable idea how one can do things.

Globus's reimplementation of proxycertinfo and proxypolicy for openssl 1.1 is here:
https://github.com/globus/globus-toolkit/tree/globus_6_branch/gsi/proxy/proxy_ssl/source/library
Globus now also uses the proxycertinfo struct and the associated functions that are defined in the openssl library in many places. The definition in the globus-proxy-ssl is marked as deprecated, and is mainly used for the implementation of non-RFC proxies.

Thanks Mattias, I found the globus code. I'll try to integrate it as much as I can. After that, the next hurdle is the encoding/decoding of the AC, which currently uses the removed macros. I'm reimplementing it in the "right" way. If only I could find some documentation on it, instead of looking into the code...

Hi!
How are things progressing? Can I help in any way?

Hi Mattias.
There is some progress. The code using the removed macros has been ported to OpensSSL 1.1 for the encoding/decoding of the proxy information, but not yet fully for the AC. But porting is just the first step, so there is still quite some work to do.
As soon as there is a stable situation I'll let you know, so that you can at least try the build.

In order not to be removed from Debian I built voms using the ssl11 branch in Debian unstable. This compiled, so I could close the FTBFS bug. But it is not fully functional, so It should be updated again when it works. The full freeze for debian 9 is 2017-02-05, and migration from unstable to testing takes 10 days, so the absolutely last day to upload the version that will be in Debian 9 is 2017-01-25. And preferably it should not be done at the very last minute...

I wonder what is preventing Debian to have an openssl-1.0 compatibility package, like Fedora does AFAIR. I am quite sure we won't make the deadline, as more work is still to be done for a properly tested transition to 1.1...

There is an openssl 1.0 compatibility library in debian too. However, since there are a lot of software (lcgdm, myproxy, fts, ...) that uses both the globus libraries and the voms libraries it would be very problematic to have the globus libraries compiled against openssl 1.1 and the voms libraries compiled against openssl 1.0 and then compile against both the globus and voms libraries since you would then have both openssl library versions in the same process...

So the idea is that we have a problem on fedora as well...

Yes, but the deadlines are not as close in time. (alpha freeze 2017-02-28, beta freeze 2017-04-25, final freeze 2017-05-23).

Even if VOMS manages, dependencies will probably not be able to, in such short notice...
I do fully agree with Mattias that building VOMS against 1.0 while globus is built against 1.1 will be a nightmare (read: probably one big segfault).

Sure. We'll do our best to meet the deadline.

We will provide you a tag tonight that builds and where the API and server works as expected. No effort will be invested in checking that the old clients work as expected, even if I do not expect big regressions there (and anyway, these are not officially supported by us).

The tag is https://github.com/italiangrid/voms/releases/tag/v2.1.0-rc0

Many thanks.
I uploaded the new version to Debian unstable last night:
https://packages.qa.debian.org/v/voms/news/20170125T213541Z.html
With 10 days in unstable before migrating to testing this should just about make it before the freeze on February 5.
I also built it for Fedora Rawhide this morning:
https://koji.fedoraproject.org/koji/buildinfo?buildID=836347

Thank you Mattias.
@msalle, let us know if VOMS 2.1.0 breaks lcmaps...

Hi,
It seems to work fine!
I found one minor issue
https://github.com/italiangrid/voms/blob/develop-2.1.x/src/sslutils/proxycertinfo.c#L350
This assertion goes wrong if you're also using globus (as we are doing in the lcas-lcmaps-gt-interface) since globus already creates the NID. Adding something like
  if (OBJ_txt2nid(PROXYCERTINFO_OLD_OID)!=NID_undef)	{
      init_done = 1;
      return;
  }

just before the OBJ_create solves the problem.
I also checked the clients and they actually seem to work too. It only produces a harmless error, which comes from https://github.com/italiangrid/voms/blob/develop-2.1.x/src/sslutils/sslutils.c#L480 (see also line before), which is not it self part of the clients (before you say that the clients are no longer supported (-: )

Thanks for the good news!
We'll go through all the asserts. I often put them in the code as a placeholder for proper error handling and indeed some may need to be reviewed.
Mattias, can you remind me what the procedure is now for modifications?

Hi,
I'm afraid I found one more issue: https://github.com/italiangrid/voms/blob/develop-2.1.x/src/api/ccapi/voms_apic.h#L41
gssapi.h defines it with a starting and a trailing underscore. See e.g. https://github.com/globus/globus-toolkit/blob/globus_6_branch/gsi/gssapi/source/library/gssapi.h and https://github.com/krb5/krb5/blob/krb5-1.15/src/lib/gssapi/generic/gssapi.hin

But do you see a specific build- or run-time problem? In fact the globus gssapi.h defines both forms (https://github.com/globus/globus-toolkit/blob/globus_6_branch/gsi/gssapi/source/library/gssapi.h#L18). BTW, technically speaking identifiers starting with an underscore followed by a capital letter are reserved and shouldn't be used (at least in C++).

Yes, I've seen it fail: if I build against my system gssapi.h, i.e. not the globus one (this was an LCMAPS plugin which does not really use Globus itself). That gssapi.h only defines _GSSAPI_H_, i.e. with a starting and ending underscore, conform https://github.com/globus/globus-toolkit/blob/globus_6_branch/gsi/gssapi/source/library/gssapi.h#L20-L24
But since https://github.com/italiangrid/voms/blob/develop-2.1.x/src/api/ccapi/voms_apic.h#L34-L41 only checks for either starting or ending, it fails.

@msalle , no changes were introduced with this version that affect that header, so I would say this issue is not 2.1.0 specific. Can you confirm whether you observe the build failure also with 2.0.14, for instance? Thanks

Hi,
You're right, I agree it's indeed an older issue and indeed also fails with 2.0.14.
For a simple test program:
#include <gssapi.h>
#include <voms/voms_apic.h>

int main(void)	{ return 0; }

Compiling against krb-based gssapi.h fails, compiling against globus-based gssapi.h succeeds.

Ok, thanks @msalle. I've created a separate issue to track this problem (#54)

My observation so far.
The command line client seems to work. When using voms-proxy-init the following lines are printed to the screen:
oid exists
Function: OBJ_create
but execution continues, and the proxy that is created seems to be correct.
The default proxy version in the absence of any command line options is still the legacy proxy though, which is different form the java client that makes RFC proxies by default.
The biggest issue I have found so far is with myproxy. When myproxy is compiled with voms support using the new library version the test suite fails. Voms support in myproxy is implemented using a plugin loaded at runtime which uses the voms library. If I move the plugin out of the way so that it can not be loaded and rerun the test suite it succeeds. This might be caused by the same issue previously mentioned by @msalle regarding using globus and voms libraries together.

Using the change to src/sslutils/proxycertinfo.c proposed by @msalle fixes the issue with the myproxy test suite. I created pull request #55 with this change.

Hi. I propose a larger change; see issue #60 and pull request #61.

I have also create pull request #62 to fix issue #54. Could you please have a look?

It looks fine to me, thanks!

Last Wednesday 1 February, I built an updated voms package for Fedora Rawhide. This package corresponds to the 2.1.0rc0 version + the pull requests #55, #56, #57 and #58. I then built the latest myproxy version in Fedora Rawhide using this voms version and the test suite was successful (didn't trigger an assertion in the voms library).
Early yesterday morning, the voms package with the 2.1.0rc0 version migrated to Debian testing after 10 days in Debian unstable. Late last night the new Debian release was declared to be in Final Freeze, and all further migrations to testing must have an explicit unblock approval from the Debian Release Team.
Yesterday I uploaded an updated package to Debian containing the same patches as the Fedora update mentioned above. I then requested an unblock approval for this update which was granted, so it looks like there will be a working voms library in the next Debian Release.

I agree that #61 is a more complete fix than #55 and should be preferred for the final release.

I didn't realize there were some PR's pending. They look good to me. The VOMS code base would need some cleanup, especially for the use of C++. I'll see if I can spend some time on it in the near future.


TypeError: Cannot read properties of null (reading 'useState')

facing this issue

Hi @callmejm,
Your issue might be related to https://github.com/facebook/react/issues/13991.
What bundler are you using? I bet it's create-react-app. You may find a lot of different ways to resolve the issue in the link pasted above.
One very easy workaround is to upgrade your react and react-dom to 18.2.0 or above. Please see https://stackblitz.com/edit/react-ts-xvw4qv?file=package.json,App.tsx for examples.


Move off eslint-plugin-fp

Hello there!
Sorry to file so many issues, I just haven't had time to/forgot to before now :)
eslint-plugin-fp is a handy plugin sheriff uses that was made by the the wonderful @jfmengels, who wisely chose to move on from JS to the fantastic land of Elm, but unfortunately in doing so left his JS repos unmaintained. 🙃
However, we in Elm land (read: me) often still need to write JS, but we'd like to do so in a functional style reminiscent of Elm. Therefore, I use sheriff as a handy helper to enforce a modern, functional style.
I recently discovered eslint-plugin-functional, which is a modern, maintained (not drop-in) replacement for eslint-plugin-fp. Would be interested in a PR migrating us?

I know about eslint-plugin-functional.
Some years ago i was comparing eslint-plugin-fp and eslint-plugin-functional.
In the end i opted for eslint-plugin-fp because it seemed more straightfoward.
I had another look yesterday at eslint-plugin-functional. It got recently updated and surely improved from the last time i looked at it.
But i still think it's fundamentally incompatible with Sheriff.
Taken from the docs:

Sheriff don’t offer a “recommended” or “strict” config. This config is a predefined set of rules meant to act as guidelines for a functional-light programming style.

Sheriff is meant to encourage light-functional-programming practices.
Every single eslint-plugin-functional is too fp-hardcore for Sheriff.
And YET, if anybody want to go for the fp-hardcore path, they CAN. Sheriff doesn't impede anybody from installing and using eslint-plugin-functional. Anybody can use it alongside Sheriff without issues, if they wish so.
But it's just not a right default for Sheriff, it doesn't belong here.
I think the only mildly interesting rule that i saw in eslint-plugin-functional is prefer-tacit.
Actually, this got me thinking that we can probably even remove eslint-plugin-fp. We are currently using eslint-plugin-fp just for 3 basic rules that could easily be re-implemented with no-restricted-syntax.

What do you think?

Also, as a side-note, i'm currently working on eslint-plugin-remeda.
When that will be ready (actually it's already working, but i'm thinking of adding some extras and polishing it a bit before considering it production ready. Contributions are welcome ofc), i'll add it to Sheriff as an opt-in.

Ok then. I assumed we we using the recommended config, and was thinking that no-classes, etc could be switched over.
However, I agree that no-restricted-syntax is probably better, as it's lighter. I mainly just want to reduce the amount of lodash in my bundle.
I noticed that. Remeda looks nice, but I'd also prefer to keep the number of copies of those sorts of libraries in my node_modules down. I've also been slowly migrating code to Effect-TS which looks like the two might conflict a bit there.

@lishaduck yeah, definitely you should only use Effect-TS or Remeda, not both at the same time.
Effect-TS kinda scares me, it seems very heavy.
How are you liking it so far?

@lishaduck yeah, definitely you should only use Effect-TS or Remeda, not both at the same time.
Effect-TS kinda scares me, it seems very heavy.
How are you liking it so far?

@lishaduck yeah, definitely you should only use Effect-TS or Remeda, not both at the same time. Effect-TS kinda scares me, it seems very heavy. How are you liking it so far?

It's nice. I like it. It's certainly not something I want to bundle, so I kept in in scripts, and I want to start replacing zod with it. My main thing is that there isn't a platform-deno yet, which is preventing me from doing to much. It's not bad to work around, I just don't want to write a bunch of code that'll need refactoring again soon.


lettuce doesnt queue commands to buffer in a failover scenario

Bug Report
We have triggered redis failover api in AWS elasticache redis, while the client is in a K8 environment
From configuration point of view, we have

auto reconnect is enabled
disconnect behaviour is set as default(accept command when disconnected)
request-queue-size is marked at 200(higher than the commands which we have triggered)
adaptive-refresh-triggers-timeout-in-seconds as 30secs
All triggers are enabled for adaptive refresh
periodic refresh is disabled
cancel commands on reconnect failure is false
suspend reconnect on protocol failure is false

Current Behavior
When we trigger failover from AWS console. Though in the prev logs, channel was closed/ channel was inactive, still the commands are triggered on redis server and below exception was observed.
logs:
logs:
08:22:36.486 [lettuce-nioEventLoop-4-2] DEBUG i.l.core.protocol.CommandHandler - [channel=0x4f03c02c, /<IP_ADDRESS>:50644 -> redisforfailovervalidation-0003-002.redisforfailovervalidation.run3wt.use1.cache.amazonaws.com/<IP_ADDRESS>:6379, chid=0x48] channelInactive() done
08:22:36.486 [lettuce-nioEventLoop-4-2] DEBUG i.l.core.protocol.ConnectionWatchdog - [channel=0x4f03c02c, /<IP_ADDRESS>:50644 -> redisforfailovervalidation-0003-002.redisforfailovervalidation.run3wt.use1.cache.amazonaws.com/<IP_ADDRESS>:6379, last known addr=redisforfailovervalidation-0003-002.redisforfailovervalidation.run3wt.use1.cache.amazonaws.com/<IP_ADDRESS>:6379] channelInactive()
08:22:36.486 [lettuce-nioEventLoop-4-2] DEBUG i.l.core.protocol.ConnectionWatchdog - [channel=0x4f03c02c, /<IP_ADDRESS>:50644 -> redisforfailovervalidation-0003-002.redisforfailovervalidation.run3wt.use1.cache.amazonaws.com/<IP_ADDRESS>:6379, last known addr=redisforfailovervalidation-0003-002.redisforfailovervalidation.run3wt.use1.cache.amazonaws.com/<IP_ADDRESS>:6379] Reconnect scheduling disabled
08:22:36.486 [lettuce-nioEventLoop-4-2] DEBUG i.l.core.protocol.CommandHandler - [channel=0x4f03c02c, /<IP_ADDRESS>:50644 -> redisforfailovervalidation-0003-002.redisforfailovervalidation.run3wt.use1.cache.amazonaws.com/<IP_ADDRESS>:6379, chid=0x48] channelUnregistered()
08:22:36.487 [lettuce-eventExecutorLoop-1-3] DEBUG io.lettuce.core.RedisChannelHandler - closeAsync()
08:22:36.487 [lettuce-eventExecutorLoop-1-3] DEBUG i.l.core.protocol.DefaultEndpoint - [channel=0x3a3a51d2, /<IP_ADDRESS>:50608 -> redisforfailovervalidation-0002-003.redisforfailovervalidation.run3wt.use1.cache.amazonaws.com/<IP_ADDRESS>:6379, epid=0x37] closeAsync()
08:22:36.487 [lettuce-nioEventLoop-4-3] DEBUG i.l.core.protocol.ConnectionWatchdog - [channel=0x3a3a51d2, /<IP_ADDRESS>:50608 -> redisforfailovervalidation-0002-003.redisforfailovervalidation.run3wt.use1.cache.amazonaws.com/<IP_ADDRESS>:6379, last known addr=redisforfailovervalidation-0002-003.redisforfailovervalidation.run3wt.use1.cache.amazonaws.com/<IP_ADDRESS>:6379] userEventTriggered(ctx, SslCloseCompletionEvent(java.nio.channels.ClosedChannelException))
08:22:36.487 [lettuce-nioEventLoop-4-3] DEBUG i.l.core.protocol.CommandHandler - [channel=0x3a3a51d2, /<IP_ADDRESS>:50608 -> redisforfailovervalidation-0002-003.redisforfailovervalidation.run3wt.use1.cache.amazonaws.com/<IP_ADDRESS>:6379, chid=0x47] channelInactive()
08:22:36.487 [lettuce-nioEventLoop-4-3] DEBUG i.l.core.protocol.DefaultEndpoint - [channel=0x3a3a51d2, /<IP_ADDRESS>:50608 -> redisforfailovervalidation-0002-003.redisforfailovervalidation.run3wt.use1.cache.amazonaws.com/<IP_ADDRESS>:6379, epid=0x37] deactivating endpoint handler
08:22:36.487 [lettuce-nioEventLoop-4-3] DEBUG i.l.core.protocol.CommandHandler - [channel=0x3a3a51d2, /<IP_ADDRESS>:50608 -> redisforfailovervalidation-0002-003.redisforfailovervalidation.run3wt.use1.cache.amazonaws.com/<IP_ADDRESS>:6379, chid=0x47] channelInactive() done
08:22:36.488 [lettuce-nioEventLoop-4-3] DEBUG i.l.core.protocol.ConnectionWatchdog - [channel=0x3a3a51d2, /<IP_ADDRESS>:50608 -> redisforfailovervalidation-0002-003.redisforfailovervalidation.run3wt.use1.cache.amazonaws.com/<IP_ADDRESS>:6379, last known addr=redisforfailovervalidation-0002-003.redisforfailovervalidation.run3wt.use1.cache.amazonaws.com/<IP_ADDRESS>:6379] channelInactive()
08:22:36.488 [lettuce-nioEventLoop-4-3] DEBUG i.l.core.protocol.ConnectionWatchdog - [channel=0x3a3a51d2, /<IP_ADDRESS>:50608 -> redisforfailovervalidation-0002-003.redisforfailovervalidation.run3wt.use1.cache.amazonaws.com/<IP_ADDRESS>:6379, last known addr=redisforfailovervalidation-0002-003.redisforfailovervalidation.run3wt.use1.cache.amazonaws.com/<IP_ADDRESS>:6379] Reconnect scheduling disabled
08:22:36.488 [lettuce-nioEventLoop-4-3] DEBUG i.l.core.protocol.CommandHandler - [channel=0x3a3a51d2, /<IP_ADDRESS>:50608 -> redisforfailovervalidation-0002-003.redisforfailovervalidation.run3wt.use1.cache.amazonaws.com/<IP_ADDRESS>:6379, chid=0x47] channelUnregistered()
08:22:36.490 [lettuce-eventExecutorLoop-1-3] DEBUG i.l.c.c.PooledClusterConnectionProvider - closeStaleConnections() count before expiring: 10
08:22:36.490 [lettuce-eventExecutorLoop-1-3] DEBUG i.l.c.c.PooledClusterConnectionProvider - closeStaleConnections() count after expiring: 10

Stack trace
```java
Caused by: io.lettuce.core.RedisCommandTimeoutException: Command timed out after 1 minute(s)
at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51)
at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:120)
at io.lettuce.core.cluster.ClusterFutureSyncInvocationHandler.handleInvocation(ClusterFutureSyncInvocationHandler.java:123)
at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
at com.sun.proxy.$Proxy17.set(Unknown Source)
at com.pega.fnx.datagrid.serviceredis.client.lettuce.commands.LettuceKeyValueCache.executeSetCommand(LettuceKeyValueCache.java:62)
... 20 common frames omitted
```

Input Code

Input Code
        RedisClusterClient client = RedisClusterClient.create(clientResource.get(), redisURI.get());
        client.setOptions(clientOptions.get());
        StatefulRedisClusterConnection connection =  clusterClient.connect(RedisCodec.of(
                new StringCodec(StandardCharsets.UTF_8),
                new ByteArrayCodec()));
         RedisClusterCommands clusterCommands = clusterClientConnection.sync();
         SetArgs setArgs = SetArgs.Builder.px(60000); //1min
         String setCmdStatus = clusterCommands.set("key", "value".toByteArray(), setArgs);

Expected behavior/code
I was expecting that the commands sent during the failover time will be queued to the buffer and will be executed once server is connected back. But that was not the case, command was sent to the node even though the connection was not established.

Environment

Lettuce version(s): 5.3.0.RELEASE
Redis version: AWS Elasticache Redis 5.0.6

Commands are queued during a disconnect but that doesn't pause the timeout scheduler. From a caller side, you don't want that the duration for a command is depending on the fact whether the connection is connected or disconnected. Instead, you want a reliable timeout cap so you can reason about service levels and a maximal latency.
If the server comes back during the configured timeout of one minute, the command will be re-sent. If the server is disconnected for longer than 1 minute, your application will see timeouts.

From a caller side, you don't want that the duration for a command is depending on the fact whether the connection is connected or disconnected. Instead, you want a reliable timeout cap so you can reason about service levels and a maximal latency.
yes, and should be able to tell if the command can be retriggered from caller end or not.
If the server comes back during the configured timeout of one minute, the command will be re-sent. If the server is disconnected for longer than 1 minute, your application will see timeouts.
Is the above described behaviour already in place. And will the commands which are queued be removed once timeout is reached or will the command be in the queue even after the timeout.

I have tested this behaviour. I see that caller will receive timeout exception, but command will still be processed once the connection is established after the timeout period as well. Do we have a way to cancel the command as well when timeout is reached. Because this is giving a wrong message to the caller.

Commands that time out are completed either exceptionally with TimeoutException or get cancelled when using the synchronous API. DefaultEndpoint removes cancelled/completed commands to avoid replaying completed/cancelled commands.
Can you provide a small reproducer if you still can observe the problem?

I have created a sample app to reproduce the issue. You can run the program, you should see below messages
"Enter
1.perform put operation 2. terminate "

choose 1, it will create a connection and update the "key" in redis cluster. Slot number will be available in the log as shown below
Performing put operation
Key will be put in slot: 12539

Now you can connect to master node of this slot, and flush this key(remove the key), check the client list, client name is testClient. Now you can kill the client. Auto reconnect is enabled with lower bound as 30 secs. And command timeout as 5 secs. So now again, immediately perform the put operation. Since connection is disconnected. We will get timeout exception after 5 secs. And after 30 secs we see logs stating reconnection established. After connection is reestablished, you can check for keys in redis master node. You should be seeing that key is available. Let me know if you need more details
micronautapp.zip

@mp911de Could get some time to look into this. Can we mark the issue as bug, since I have also provided a reproducible steps and app ?

Hey @ALSowjanya ,
I see this is an issue from 2020 - did you manage to solve the problem? Have you tried newer versions of the driver?

Hey @ALSowjanya ,
I've used your example and it works exactly how you described, but this behaviour is expected:

You send a command to the server
You flush the database and kill the client connection from another client
You attempt to send a command again

What I see locally is that the driver re-establishes connection and sends the command.
I never see a timeout being thrown for the command.
If I do change the timeout to some very low value, e.g. 1 ms (to quick for the reconnect to succeed) I get the TimeoutException and then I see no modifications in the database.
I am closing this for now because I can't see a problem with the driver.


🐞 beta.97 | breaking AOT Compiler on SvelteKit

What happened?
After the update to beta.97 SvelteKit doesn't start and breaks with the error Error: Dynamic require of "path" is not supported.
Version
2.0.0-beta.97
Relevant log output
failed to load config from /Users/x/Desktop/github/x/vite.config.js
error when starting dev server:
Error: Dynamic require of "path" is not supported
    at file:///Users/x/Desktop/github/x/node_modules/.pnpm/techor@1.0.5/node_modules/techor/dist/index.mjs:1:382
    at file:///Users/x/Desktop/github/x/node_modules/.pnpm/techor@1.0.5/node_modules/techor/dist/index.mjs:1:994
    at file:///Users/x/Desktop/github/x/node_modules/.pnpm/techor@1.0.5/node_modules/techor/dist/index.mjs:1:462
    at file:///Users/x/Desktop/github/x/node_modules/.pnpm/techor@1.0.5/node_modules/techor/dist/index.mjs:2:62587
    at ModuleJob.run (node:internal/modules/esm/module_job:193:25)
    at async Promise.all (index 0)
    at async ESMLoader.import (node:internal/modules/esm/loader:530:24)
    at async loadConfigFromBundledFile (file:///Users/x/Desktop/github/x/node_modules/.pnpm/vite@4.0.3/node_modules/vite/dist/node/chunks/dep-0bae2027.js:62058:21)
    at async loadConfigFromFile (file:///Users/x/Desktop/github/x/node_modules/.pnpm/vite@4.0.3/node_modules/vite/dist/node/chunks/dep-0bae2027.js:61943:28)
    at async resolveConfig (file:///Users/x/Desktop/github/x/node_modules/.pnpm/vite@4.0.3/node_modules/vite/dist/node/chunks/dep-0bae2027.js:61564:28)
 ELIFECYCLE  Command failed with exit code 1.

What browsers are you seeing the problem on?
No response

@oskar-gmerek We also found this problem and are urgently fixing it!

The bug has been fixed in v2.0.0-beta.98


Update mlbparks-binary-build.adoc

fixes #150

We updated the procedure on the new ocp-4.5 branch


unexpected result on our own street survelliance data

Hi, thanks for your great work!
I tried the pretrained models on BDD100K seg_track valset, the reuslts are the same with the demos.
But on our own videos (static cameras, 2560x1440), the detected objects are not enough. And some cars only got part bounded.

Is it just because the trained bdd dataset is different from our own data?
Do you have other preptrained models which is suitable for this kind of city street survelliance videos?
By the way, I don't know what is the difference between pcan_pretrained_model.pth and 'segtrack-fixed-new.pth' but they seem perform similar results. Thanks!

I suggest you to retrain the model based on you images because the two datasets have very different angles when taking photos. 'pcan_pretrained_model.pth' is the final model of BDD100K MOTS while 'segtrack-fixed-new.pth' is the initial model weights trained on BDD100k MOT tracking set.

@SRainGit  Interesting application!   Based on my experience, doing object detection in a bird-eye view video is very challenging.  You better transform the picture's angle a bit before inference.    At evaluation stage, I would feed these samples into a typical object detection network and see if interested objects could be effective recognized.
Another challenge is that in your sample pics, there are lots of cargo tricycles, which is quite common in China cities.  Unfortunately, this is not even a category in most public datasets.


make a comment explaining decrement/increment monster in IdViewStateBase

these are moved to the mixin now, i dont think it's necessary


matcher graph generator TLC

The matcher graph generator  is creaking a little.  It could do with some care.

If you run it without Graphviz installed, it claims to have generated a PDF, but has not.
Ideally, we would be able to optionally specify a pipeline date as a CLI argument (e.g. to compare differences between a live and non-live pipeline).

I achieved this by just hardcoding the date I wanted, but a "proper" way would be better

Appending "(suppressed)" to suppressed images makes the HTML labels invalid, and prevents dot creating the image.

an HTML-Like label can either be text or a table, not both,  so it needs to be inserted into the table, not appended afterwards as it is for text labels.
When this happens, it fails in the same manner as 1, above, rather than informing the user of any problems.

Thinking about the pipeline date thing.  I think the main reason for this is to compare between multiple existing pipelines, so perhaps having some incantation that tells it to generate multiple graphs would be nice.

If we're going to be working on this tool, I wonder if it would be useful to annotate the edges with the reasons at the same time? Or have a flag for doing that?


Bump tar from 2.2.1 to 2.2.2

Bumps tar from 2.2.1 to 2.2.2.

Release notes
Changelog
Commits

updated-dependencies:

dependency-name: tar
dependency-type: indirect
...

Signed-off-by: dependabot[bot]<EMAIL_ADDRESS>

3


fix(deps): update icr.io/appcafe/open-liberty/samples/getting-started:latest docker digest to 3999aa8

This PR contains the following updates:

Package
Update
Change

icr.io/appcafe/open-liberty/samples/getting-started
digest
91589e8 -> 3999aa8

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline

:tada: This PR is included in version 1.4.26 :tada:
The release is available on:

GitHub release
v1.4.26

Your semantic-release bot :package::rocket:


Use path.join for default build_dir

Description
The default build directory in cli.js was set to ".\build\contracts\", which is a windows notation. This caused the default command to fail on my MacBook as it couldn't find the build directory. I changed this to use path.join for a platform independent path.
Types of changes

[x] Bug fix (non-breaking change, which fixes an issue)

Codecov Report

Merging #19 into develop will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop      #19   +/-   ##
========================================
  Coverage    35.63%   35.63%
========================================
  Files           20       20
  Lines          188      188
  Branches        34       34
========================================
  Hits            67       67
  Misses         115      115
  Partials         6        6

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7c967f7...1b335e5. Read the comment docs.


Zero values for attention and meditation

Hello Brianna and Jim Roskind,
Firstly, I would like to thank you for making your code for reading data from the ‘Force Trainer II’ public and freely available here on Github. 😊
I decided to do a similar task for a school project, for which I have programmed a number of video games to use the ‘attention’ and ‘meditation’ values from the headset. Unfortunately, I am unable to get the Arduino code to output correct attention and meditation values to the computer at all. Regardless of whether I am wearing the headset or not, the values for attention and meditation remain at '0'.
I have attached your code here (I made a few small changes, the issue existed prior to these modifications) as well as a text file demonstrating the sort of output I am receiving from the force trainer headset.
Any help at all would be greatly appreciated!
Force Trainer Code
Force Trainer Output

Hello,
It seems that I have partially resolved the issue. The attention and meditation values only seem to be sent by the NeuroSky chip to the Arduino when the “poor signal” value is perfectly zero ‘0’. This takes some time and adjustment of the headset to achieve. Strangely enough, other Android and iOS Bluetooth enabled EEG apps report attention and meditation data even when the ‘poor signal’ value is not perfect (value > 0). I still have not been able to figure out exactly why this is the case, but the Arduino software does work - it just requires a perfect signal. 🙂


Fix the missing slashes in locations

Our common path calculation has a slash missing:

This seems to be the behaviour of Python's os.path.commonpath, which was was written for paths, not URLs.
Move to our own common_uri function, and add regression tests.

Reported in #279


Crashed during Mass Detection

mzmine_0_0.log
mzmine_1_0.log
mzmine_2_0.log
Windows Server 2019 Standart
Procesor: AMD Opteron Processor 6272
RAM: 128 GB
Mzmine 3.2.8
What happened
I tried to analyze 256 files in .mzXML format with a total size of 13.6 GB. It crashed already during the Mass Detection step in the MS1 scan (noise level 5.0E4).
No error is displayed.Program crashed and nothing happens.
When I analyzed less and smaller data, it ran normally. Does MZmine have any file limitations?
Thank you for your help

MZmine does not have any file limitations. It seems strange that the program would just crash suddenly. Could you try to run it on a different computer to see if you can reproduce the error?

Make sure to setup the temp path in the preferences to a fast local drive with enough space.
We are saving temp files with the spectral data and the filtered mass lists. Maybe your drive is full at one point.


[PoC] replace serializable with pipeline test for prototype datasets

Proof of concept for https://github.com/pytorch/pytorch/pull/74958#issuecomment-1085547766.

Superseded by #5711.


[BUG] bounded_blocking has a different metric name

Describe the bug
The default buffer used in Data Prepper is the BlockingBuffer. The plugin name is bounded_blocking and yet metrics appear as <pipeline>.BlockingBuffer.recordsInBuffer.count
Expected behavior
I would expect the naming to be consistent with plugin name and the metrics that are published.
Additional context
Relevant link to the issue:
https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/blocking-buffer/src/main/java/org/opensearch/dataprepper/plugins/buffer/blockingbuffer/BlockingBuffer.java#L69

Perhaps we can provide a couple options here to help with the breaking change.

Add a new metric in addition to the old.
Create a setting in Data Prepper to support the correct name. This could be useful for other metric changes. For example, the proposal for #1025 would change metrics for pipelines. This could be breaking if not done with a setting.


Seems that it is not caching fontist/relaton content?

https://github.com/metanorma/iso-15926-6/actions/runs/4361918960/jobs/7626263817
Run actions-mn/cache@v1
Run actions/cache@v3
  with:
    path: ~/.metanorma
    key: metanorma-home
    restore-keys: metanorma-home
    enableCrossOsArchive: false
    fail-on-cache-miss: false
/usr/bin/docker exec  140ed7e261088967f50d182b336461f6158069895aa59eb7c0b1230366b70908 sh -c "cat /etc/*release | grep ^ID"
Cache not found for input keys: metanorma-home, metanorma-home
Run actions/cache@v3
/usr/bin/docker exec  140ed7e261088967f50d182b336461f6158069895aa59eb7c0b1230366b70908 sh -c "cat /etc/*release | grep ^ID"
Cache not found for input keys: metanorma-relaton, metanorma-relaton
Run actions/cache@v3
/usr/bin/docker exec  140ed7e261088967f50d182b336461f6158069895aa59eb7c0b1230366b70908 sh -c "cat /etc/*release | grep ^ID"
Cache not found for input keys: metanorma-fontist, metanorma-fontist
Run actions/cache@v3
/usr/bin/docker exec  140ed7e261088967f50d182b336461f6158069895aa59eb7c0b1230366b70908 sh -c "cat /etc/*release | grep ^ID"
Cache not found for input keys: metanorma-ietf-workgroup-cache, metanorma-ietf-workgroup-cache

Originally the action developer for the non-container environment. I will add paths that make sense inside docker container

Still doesn't work:
Warning: Path Validation Error: Path(s) specified in the action for caching do(es) not exist, hence no cache is being saved.
Post job cleanup.
Warning: EACCES: permission denied, lstat '/root/.fontist'
Post job cleanup.
Warning: EACCES: permission denied, lstat '/root/.relaton'
Post job cleanup.
Warning: EACCES: permission denied, lstat '/root/.metanorma'

To be investigated

Fixed pathes on docker and now it works
Tested on https://github.com/metanorma/mn-samples-iso/actions/runs/4391108515/jobs/7972061875
Log details
...
Run actions/cache@v3
/usr/bin/docker exec  521e7edfa0f927c302dbc8320a0032f8b3dea0b1ba5c091dffb0209a8f95c476 sh -c "cat /etc/*release | grep ^ID"
Received 0 of 536016558 (0.0%), 0.0 MBs/sec
Received 67108864 of 536016558 (12.5%), 31.9 MBs/sec
Received 134217728 of 536016558 (25.0%), 42.0 MBs/sec
Received 209715200 of 536016558 (39.1%), 49.4 MBs/sec
Received 268435456 of 536016558 (50.1%), 50.7 MBs/sec
Received 356515840 of 536016558 (66.5%), 56.2 MBs/sec
Received 398458880 of 536016558 (74.3%), 53.9 MBs/sec
Received 402653184 of 536016558 (75.1%), 47.6 MBs/sec
Received 461373440 of 536016558 (86.1%), 48.6 MBs/sec
Received 528482304 of 536016558 (98.6%), 50.1 MBs/sec
Received 536016558 of 536016558 (100.0%), 48.4 MBs/sec
Cache Size: ~511 MB (536016558 B)
/bin/tar -xf /__w/_temp/dd670cdd-2bab-43a1-8b9a-c1a9645080e8/cache.tgz -P -C /__w/mn-samples-iso/mn-samples-iso -z
Cache restored successfully
Cache restored from key: metanorma-home
Run actions/cache@v3
/usr/bin/docker exec  521e7edfa0f927c302dbc8320a0032f8b3dea0b1ba5c091dffb0209a8f95c476 sh -c "cat /etc/*release | grep ^ID"
Received 325505 of 325505 (100.0%), 0.7 MBs/sec
Cache Size: ~0 MB (325505 B)
/bin/tar -xf /__w/_temp/8d4a036c-53e6-4f34-b1c8-48a7b6fd52f7/cache.tgz -P -C /__w/mn-samples-iso/mn-samples-iso -z
Cache restored successfully
Cache restored from key: metanorma-relaton
Run actions/cache@v3
/usr/bin/docker exec  521e7edfa0f927c302dbc8320a0032f8b3dea0b1ba5c091dffb0209a8f95c476 sh -c "cat /etc/*release | grep ^ID"
Received 0 of 1199669[29](https://github.com/metanorma/mn-samples-iso/actions/runs/4391108515/jobs/7972061875#step:8:32) (0.0%), 0.0 MBs/sec
Received 71303168 of 119966929 (59.4%), 34.0 MBs/sec
Received 119966929 of 119966929 (100.0%), 44.5 MBs/sec
Cache Size: ~114 MB (119966929 B)
/bin/tar -xf /__w/_temp/a86e21d2-a0ef-465c-88fc-46ac5ef71ee8/cache.tgz -P -C /__w/mn-samples-iso/mn-samples-iso -z
Cache restored successfully
Cache restored from key: metanorma-fontist
Run actions/cache@v3
/usr/bin/docker exec  521e7edfa0f927c302dbc8320a0032f8b3dea0b1ba5c091dffb0209a8f95c476 sh -c "cat /etc/*release | grep ^ID"
Cache not found for input keys: metanorma-ietf-workgroup-cache, metanorma-ietf-workgroup-cache


DAG-1901: Add Getting Help section to evg-module-manager README

evergreen merge

LGTM but looks like there were some issues installing atomicwrites on windows.

Fixed


Double-Square cards

Add double-square cards that enable players to move to the second square of that color instead of the first

Feature implemented


A splash screen can be added to improve the first user experience with the app.

A splash screen can be added to improve the first user experience with the app.
Create a design for the splash screen and add it to the project.

Can I take up this issue?

Go ahead. Ping me up on slack for any design issues!

@garimasingh128 Please close this, it is completed.


add .travis.yml / fix -fPIC for clang

adds a simple travis-ci.org config file & fixes compile with clang

Out of date and no longer required (clang compiles work fine in current master)

When I got a free moment I'll have a look if we can support the current travis.


set netconf_version and capabilities manually

Description
Issue #28 .  This PR allows setting netconf_version manually for situation where it is not ideal to detect the server capabilities and dynamically set client capabilities.  It preserves the dynamic detection and setting of client capabilities when a manual setting is not set.  This helps to interact with misbehaving devices that advertise Netconf 1.1 but doesn't fully support it.
Type of change
Please delete options that are not relevant.

[x] New feature (non-breaking change which adds functionality)

How Has This Been Tested?
Tested against a device that advertise Netconf 1.1 capabilities but subsequently timeout when the server capabilities are parsed dynamically.  Setting conn.netconf_version = NetconfVersion.VERSION_1_0 manually prior to conn.open() allows the device to communicates succesfully.
Checklist:

[x] My code follows the style guidelines of this project (no GitHub actions compalints! run make lint before
committing!)
[x] I have commented my code, pydocstyle and darglint are happy, docstrings are in google docstring format, and all
docstrings include a summary, args, returns and raises fields (even if N/A)
[ ] I have added tests that prove my fix is effective or that my feature works, if adding "functional" tests please
note if there are any considerations for the vrnetlab setup
[x] New and existing unit tests pass locally with my changes

Nice! This looks like a great addition. Just want to let you know it may be a few weeks till I get my life together and get this merged (hopefully not that long, just want to set expectations!). I think I may also want to add this as a NetconfDriver option -- something like preferred_netconf_version or something like that. Maybe this would be an Optional[str] w/ default of None -- if None we just do the "normal" thing, but if set to 1.0 we prefer that. So I may play around w/ that if thats OK with you?
I'm also wondering if it makes sense to add some sanity check that ensures that the manually provided version is in the server capabilities? I guess 1.0 will always be there on everything ive ever seen but maybe thats not universally true?
Carl

Nice! This looks like a great addition. Just want to let you know it may be a few weeks till I get my life together and get this merged (hopefully not that long, just want to set expectations!). I think I may also want to add this as a NetconfDriver option -- something like preferred_netconf_version or something like that. Maybe this would be an Optional[str] w/ default of None -- if None we just do the "normal" thing, but if set to 1.0 we prefer that. So I may play around w/ that if thats OK with you?
I'm also wondering if it makes sense to add some sanity check that ensures that the manually provided version is in the server capabilities? I guess 1.0 will always be there on everything ive ever seen but maybe thats not universally true?
Carl

Just an FYI - I am going to try to get scrapli overhaul things merged to develop across the board (here/communiyt/nornir) and pre-released this weekend. I probably won't look at this till next weekend is my guess. Sorry for the delay :(

Just an FYI - I am going to try to get scrapli overhaul things merged to develop across the board (here/communiyt/nornir) and pre-released this weekend. I probably won't look at this till next weekend is my guess. Sorry for the delay :(

Gah. Deleting that branch closed this, my bad. I will make a fork off this branch and update/merge next weekend if you dont get to it first. Sorry about that!

Gah. Deleting that branch closed this, my bad. I will make a fork off this branch and update/merge next weekend if you dont get to it first. Sorry about that!

No worry.  Give me some time to look into possibly adding it as the preferred_netconf_version instead.  Hope you have a good weekend.

No worry.  Give me some time to look into possibly adding it as the preferred_netconf_version instead.  Hope you have a good weekend.


ImageDataGenerator flow function continue to increase the amount of memory usage

My keras version is 2.0.1.  When I using the ImageDataGenerator flow function to train the model (train_on_batch), it will continue to increase the amount of memory usage. But the 1.2.0 version does does not appear this problem.

Do you have a reproducible script to help to identify the problem? For example, same script results memory leak in 2.0.1 but not in 1.2.0.

@joelthchao
def extract_imagelist(target_path):
 with open(target_path, 'r') as f:
        images, labels = [], []
        for l in f.readlines():
            l = l.strip('\n').split()
            images.append(l[0])
            labels.append(int(l[1]))

 image_len = len(images)
 all_images = np.empty([image_len, 224, 224, 3])
 all_labels = []
 for i in range(image_len):
     img = image.load_img(images[i], target_size=(224, 224))
     img = image.img_to_array(img)
     img = np.expand_dims(img, axis=0)
     img = preprocess_input(img)
     lab = labels[i]
     all_images[i, :, :, :] = img
     all_labels.append (lab)
 all_labels = to_categorical(all_labels)
 #yield (all_images, all_labels)
 return all_images, all_labels

files_list = "train.txt"
X, Y = extract_imagelist(files_list)
datagen = ImageDataGenerator(horizontal_flip=True)
net.compile(optimizer=SGD(lr=0.001, momentum=0.9), loss='categorical_crossentropy')
for e in range(10):
batches = 0
for X_batch, Y_batch in datagen.flow(X, Y, batch_size=16):
loss = net.train_on_batch(X_batch, Y_batch)
if batches % 20 == 0:
print ('Epoch', e, 'iteration:', batches, 'loss:', loss)
batches += 1
if batches >= len(X) / 16:
break

Are you using theano as backend? May relate to #5935.

@joelthchao No, I use tensorflow as backend.

I have the same issue when I used theano as backend，but once I changed to tensorflow，the issue is gone.

@joelthchao thanks for your time on all these issues. You always provide great feedback. As many people have mentioned there seems to be many of us having memory issues when we switch from Keras 1.2 to Keras 2.0. I had to make the switch because of the bug of Batch Normalization + TimeDistributed which was fixed on Keras 2.0.
Below you have my system specs, and my main fit_generator function. I'm using all using Xception model and keras default flow_flow_directory generators. The code works on Keras 1.2 and on a smaller subset of the data without changing a single line of code. I narrow the error to the last batch of training, which means it's related with validation_data evaluate_generator. I load a queue of 10k images, this takes about 20/256GB in the computer, but when it's time to validate it fills my RAM 256/256GB + my swap, and system kills the process cuz memory full. On a smaller dataset it works because the validation dataset is not big enough to fill the entire memory. Example training takes like 10GB, but validation takes 40GB on small dataset (20 times smaller than my real dataset).
Conclusion validating the data it's taking exponentially more resources than my training. Somehow is trying to load all the data into memory even if there is a fix queue size. Or is leaking memory somewhere. If the parameters are exactly the same and it takes 40GB to load the queue in validation why would it take more if the dataset is bigger, unless it's growing the queue to a larger size than specified?
System Specs:
Ubuntu 16.04
CPUs: 44
GPUs: 4
Keras: 2.0.3
Tensorflow: 1.0.1
 gpu_model.fit_generator(train_generator,
                            steps_per_epoch=10761,
                            epochs=5,
                            validation_data=validation_generator,
                            validation_steps=3179,
                            max_q_size=10000,
                            workers=40,
                            callbacks=callbacks_list)

@abnera What is the difference between your train_generator and validation_generator? I need to clarify if the problem comes from setting of generator or validation process itself.

@joelthchao good question. I apologize for not making this clear. The main difference is shuffle=False/True and The parameters of ImageGenerator. Below is my exact code for them.
train_datagen = ImageDataGenerator(rescale=1. / 255,
                                       rotation_range=FLAGS.transformation_ratio,
                                       shear_range=FLAGS.transformation_ratio,
                                       zoom_range=FLAGS.transformation_ratio,
                                       cval=FLAGS.transformation_ratio,
                                       horizontal_flip=True)
    train_generator = train_datagen.flow_from_directory(train_dir,
                                                        target_size=(FLAGS.img_height, FLAGS.img_width),
                                                        batch_size=FLAGS.batch_size,
                                                        class_mode='categorical',
                                                        shuffle=True)

    validation_datagen = ImageDataGenerator(rescale=1. / 255)
    validation_generator = validation_datagen.flow_from_directory(validation_dir,
                                                                  target_size=(FLAGS.img_height, FLAGS.img_width),
                                                                  batch_size=FLAGS.batch_size,
                                                                  class_mode='categorical',
                                                                  shuffle=False)

As you mention, Keras 1.2 has no memory issue. I inspect the code change of ImageDataGenerator between 1.2.0 and 2.0.3 and I think the memory leak issue is not related to it since there are only style, naming changes. I will continue to check other part of code change.

I would suggest to do following things to clarify the root cause:

Change different models to see if it is model independent.
Make train/val generator flow from same data directory to make it data independent.
Use same generator for train/val to eliminate configuration issue.
Change worker number to clarify if is an issue involve multithreading
If possible, switch backend, data_format to see if it is still leaking.
Use callback to monitor memory usage

similar issue: #5902

@joelthchao thanks for your support. This makes a lot of sense. I literally, just managed to get it to work by using pickle_safe=True. I'll try my best to find the root of the problem to help others, but it takes about 2 hours to run a epoch. I'll update in a couple of days.

In Keras 2.0.8 the memory leak issues still occur, I don't have any memory leaks issues before adding image augmentation. My code is similar to @abnera as well, using fit generator and ImageDataGenerator

Hi there, any updates about this issue? I got a similar memory leak problem in Keras 2.0.8 with either Tensorflow 1.3.0 or Theano 0.9 (Centos 7, CUDA 8.0, cudnn 5.1). If needed, I would provide more details. Any comments or suggestions are highly appreciated.

I have the exact same issue, anyone found a solution?

Same issue, which makes it impossible to train with ImageNet

You might be super late @Dref360 - but this helped me! :)

A similar issue is apparently present in Keras 2.1.6. @Dref360, your suggestion looks reasonable, but in my case I cannot load the whole dataset at a time. It is way too large.  In my code, I break dataset into chunks, load the chunk into the memory, pre-process it, and then try to use ImageDataGenerator's flow() to further augment the data.
So, the problem is most likely coming from the flow() function. What is its behavior when called multiple times?

I have the same problem with Keras 2.2.4. It will increase the memory usage each time I call the generator.
test_datagen  = ImageDataGenerator(preprocessing_function=preprocess_input)
I do not know what the problem is and how to free up some memory.

I have the same problem with Keras==2.2.2 and tensorflow==1.11.0. I am using this generator https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly and have use_multiprocessing=False and workers=1.
Also I'm on Tensorflow backend. The issue happens when I use training size of .20 which only manages to go for about 200 epochs and then crashes, while .99 goes on for 500 epochs.
Batch size is 20 images.
Changing type to float as @linan7788626 did not help.

I fixed the memory problem with using Python functions. I define a function, and generated the data and run my algorithm on the batch data each time the function is called. Hence after returning from the function, Python removes local variables and hence the memory will be freed up.

@alitk I'm using functions within a class (see link above for full code):
import numpy as np
import keras

class DataGenerator(keras.utils.Sequence):
    'Generates data for Keras'
    def __init__(self, list_IDs, labels, batch_size=32, dim=(32,32,32), n_channels=1,
                 n_classes=10, shuffle=True):
        'Initialization'
        self.dim = dim
        self.batch_size = batch_size
        self.labels = labels
        self.list_IDs = list_IDs
        self.n_channels = n_channels
        self.n_classes = n_classes
        self.shuffle = shuffle
        self.on_epoch_end()

    def __len__(self):
        'Denotes the number of batches per epoch'
        return int(np.floor(len(self.list_IDs) / self.batch_size))

    def __getitem__(self, index):
        'Generate one batch of data'
        # Generate indexes of the batch
        indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]

        # Find list of IDs
        list_IDs_temp = [self.list_IDs[k] for k in indexes]

        # Generate data
        X, y = self.__data_generation(list_IDs_temp)

        return X, y

    def on_epoch_end(self):
        'Updates indexes after each epoch'
        self.indexes = np.arange(len(self.list_IDs))
        if self.shuffle == True:
            np.random.shuffle(self.indexes)

    def __data_generation(self, list_IDs_temp):
        'Generates data containing batch_size samples' # X : (n_samples, *dim, n_channels)
        # Initialization
        X = np.empty((self.batch_size, *self.dim, self.n_channels))
        y = np.empty((self.batch_size), dtype=int)

        # Generate data
        for i, ID in enumerate(list_IDs_temp):
            # Store sample
            X[i,] = np.load('data/' + ID + '.npy')

            # Store class
            y[i] = self.labels[ID]

        return X, keras.utils.to_categorical(y, num_classes=self.n_classes)

Not sure where the problem could be


undefined method `name' for nil:NilClass

kata: https://www.codewars.com/kata/561e9c843a2ef5a40c0000a4/translation/new
Unable to re-publish the Scala translation because it always times out, I deleted it and wanted to redo it. All I get is:
Looks like there was a server error! (500)
We have taken note of the situation and will be looking into it.

If you would like to supply us with some details as to how this happened please visit our Github issues list. This will help us get things fixed faster.

undefined method `name' for nil:NilClass

It seems that it is impossible to re-publish Scala translations maybe because CW Scala is very very slow.
Similar : #851 #1090

Unable to re-publish the Scala translation because it always times out, I deleted it and wanted to redo it.

This is a duplicate of #1268, so I'm closing this one.


Add Summary Statistics

Ideas:

[ ] How long the crawl has been running
[ ] How many pages have been fetched so far
[ ] Average harvest rate

Closed in https://github.com/ContinuumIO/memex-explorer/pull/200/files


Use TcpStream instead of Websockets for sending logs to Cockpit in Common/logging

Because Cockpit now has a backend written in rust, we can use a simpler TcpStream to receive the logs. This should improve performance a bit, but mostly removing the ws crate will help with some build times, as it has lots of dependencies.

Doesn't seem relevant for now.


Namespaces and imports for PIL

In order to unify powdr-asm and powdr-pil, we need pil statements to be allowed outside machines (declarations only, of course, no constraints) and the namespace mechanism to work properly.
The idea is that we can run PIL reference resolution and type checking (type inference) already on the PIL parts in the assembly code. This way, we can remove the macros by a proper concept. We are also not "running blind" until after machine to pil translation. We would do PIL reference resolution again on the PIL translated from ASM, but it might even be optional.
We should probably change the current namespace separator from . to :: in the same way it is done in many languages: A::B is a namespace access, which means it can be resolved before type checking and A is the name of a module or a type, not a variable, while A.B is a member access, which can only be resolved once the type of A is clear and A is a variable.
Here is some example code that does not even make use of use or namespace access, since I think that is kind of clear.
let difference_to_next = |x| x' - x;
let every_k = |k| (|i| if i % k == 0 { 1 } else { 0 });

machine SimpleStatic(latch, operation_id) {
        degree 8;

        operation power_4<0> x -> y;

        col fixed operation_id = [0]*;
        let latch = every_k(4);

        let x;
        let y;

        (1 - latch) * difference_to_next(x) = 0;
}

The bigger problem is how we translate the machine to PIL.
If I understand the machine module system correctly, the machine does not create a namespace, it just creates a scope. This is totally fine when you parse the example above and resolve PIL references.
It is a also kind clear what happens to difference_to_next and every_k: Nothing.
The big open question is what happens to x and y when we instantiate SimpleStatic. We could create a PIL namespace (more the abstract concept than the existing syntactical concept we already have) - then it might work, but I don't really understand the machine instantiation concept yet well enough to judge.

Illustrate what? Function declarations are still pil statements. I still don't fully see how the pil namespaces work with asm machine instances, but maybe we can discuss this IRL at some point.

In the module system, symbols are machines or modules. We can add a new type of symbol for PIL functions and make your example work. Or is there something preventing that in your opinion?

So I think I see what you mean: basically how do you "remove" the machine. If you just remove it, you may have an x in two different machines which would end up colliding.
The general intuition I ended up with last time we talked about this is that we should not "translate the machine to PIL". We should instead keep the machine tree all the way down. I do not have a complete plan yet though.

Ah, so machines are actually namespaces? That's actually much better!
But it's a bit inconsistent because modules do not import the super namespace by default but machines do, right?

Ah, so machines are actually namespaces? That's actually much better!

No, they are not. By "we use PIL namespaces" I mean that each machine instance ends up in a different PIL namespace in the final PIL, see here.

I was referring to "symbols are machines or modules". So in the end machines are more like functions, right? They create their own scope but are not a namespace and all the symbols from the namespace the machine is defined in, are available inside the machine.
And this is the problematic part: If we want pil namespaces to behave like powdr modules, then the symbols outside of the namespace are not automatically visible inside the namespace, unless you import them. So we cannot just use a pil namespace for a powdr machine. We could add a use super:* at the beginning, though.

Implemented in https://github.com/powdr-labs/powdr/pull/821


[TAB-8127] Can't pass the licence system property to the start script

Provided fix to accept license key on command line Cleaned up batch code to be more consistent

Created PR incorrectly


⚠ Bofrost.be has degraded performance

In 366323c, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 2644 ms

Resolved: Bofrost.be performance has improved in adb4012.


Fix reference leak when adding a new function overload

Noticed this while reading through nb_func.cpp. I verified that the new test fails if run without the new Py_CLEAR.

Nice catch, that's nasty. Seems like your check doesn't play well with PyPy's cpyext layer.

Updated the test to not check the numeric refcount on PyPy.

Fair enough.


I didn't find dropout in the paper. May I ask if llama has used dropout?

I don't think llama used dropout during pretraining

thank you ~


Add Easing Presets Selector to Story Anim Tool

Add selector for frequently used bezier curves to story anim tool found here:
https://easings.net/
But still allow for custom input for easing

https://github.com/google/web-stories-wp/pull/2284


apacheHttpd: 2.4.61 -> 2.4.62

Description of changes

CVE fallout from the last update... oof
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Upstream changes:
                                                         -*- coding: utf-8 -*-
Changes with Apache 2.4.62

  *) SECURITY: CVE-2024-40898: Apache HTTP Server: SSRF with
     mod_rewrite in server/vhost context on Windows (cve.mitre.org)
     SSRF in Apache HTTP Server on Windows with mod_rewrite in
     server/vhost context, allows to potentially leak NTML hashes to
     a malicious server via SSRF and malicious requests.
     Users are recommended to upgrade to version 2.4.62 which fixes
     this issue.
     Credits: Smi1e (DBAPPSecurity Ltd.)

  *) SECURITY: CVE-2024-40725: Apache HTTP Server: source code
     disclosure with handlers configured via AddType (cve.mitre.org)
     A partial fix for  CVE-2024-39884 in the core of Apache HTTP
     Server 2.4.61 ignores some use of the legacy content-type based
     configuration of handlers. "AddType" and similar configuration,
     under some circumstances where files are requested indirectly,
     result in source code disclosure of local content. For example,
     PHP scripts may be served instead of interpreted.
     Users are recommended to upgrade to version 2.4.62, which fixes
     this issue.

  *) mod_proxy: Fix canonicalisation and FCGI env (PATH_INFO, SCRIPT_NAME) for
     "balancer:" URLs set via SetHandler, also allowing for "unix:" sockets
     with BalancerMember(s).  PR 69168.  [Yann Ylavic]

  *) mod_proxy: Avoid AH01059 parsing error for SetHandler "unix:" URLs.
     PR 69160 [Yann Ylavic]

  *) mod_ssl: Fix crashes in PKCS#11 ENGINE support with OpenSSL 3.2.
     [Joe Orton]

  *) mod_ssl: Add support for loading certs/keys from pkcs11: URIs
     via OpenSSL 3.x providers.  [Ingo Franzki <ifranzki linux.ibm.com>]

  *) mod_ssl: Restore SSL dumping on trace7 loglevel with OpenSSL >= 3.0.
     [Ruediger Pluem, Yann Ylavic]

  *) mpm_worker: Fix possible warning (AH00045) about children processes not
     terminating timely.  [Yann Ylavic]

https://dlcdn.apache.org/httpd/CHANGES_2.4.62

thanks!


Re-open Atom File Not Exists (but actually exists) & can't Reopen Project

Prerequisites

[x] Put an X between the brackets on this line if you have done all of the following:

Reproduced the problem in Safe Mode: https://flight-manual.atom.io/hacking-atom/sections/debugging/#using-safe-mode
Followed all applicable steps in the debugging guide: https://flight-manual.atom.io/hacking-atom/sections/debugging/
Checked the FAQs on the message board for common solutions: https://discuss.atom.io/c/faq
Checked that your issue isn't already filed: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aatom
Checked that there is not already an Atom package that provides the described functionality: https://atom.io/packages

Description
Atom won't show previously unclosed file when reopen Atom. It says unable to open project folder (even though the actual file is exists)
This is not happen when previous unclosed thing is project folder. Looks like Atom treat file as a folder, not as a single file.
Also, can't "Reopen Project" from Atom Menus > File > Reopen Project. (Nothing happen)

Steps to Reproduce

Open a file to Atom
Quit atom without closing opened file
Re-open Atom
Previous opened file doesn't appear (it say directory not found)

Expected behavior:
Previous opened file should be appear.
Reopen Project should be working.
Actual behavior:
Atom show warning message.
Reopen Project feature from Atom Menus should be working.
Reproduces how often:
Everytime
Versions
Atom Version
Atom    : 1.36.0
Electron: 2.0.18
Chrome  : 61.0.3163.100
Node    : 8.9.3

apm Version
apm  2.1.3
npm  6.2.0
node 8.9.3 x64
atom 1.36.0
python 2.7.10
git 2.21.0

OS MacOS Mojave 10.14.4 (18E226)
Additional Information
This happen after updating to version 1.36.0.
Trying launch atom --save still produce the same problems.
Trying to reinstall, but still no luck.

There's a good chance this is causing files to open in new windows, too, right? Seems like a recent update screwed a few things up.

🤔 Looks like we're attempting to use a previously opened file as a root directory. Two questions:

How exactly did you open the file in Atom? Did you run atom path/to/file from the command line, or open Atom from the dock and open the file with Ctrl-O, or some other sequence of events?
Can you share the contents of the file ${ATOM_HOME}/.atom/application.json between steps 2 and 3?

There's a good chance this is causing files to open in new windows, too, right? Seems like a recent update screwed a few things up.

I think that's unrelated, other than both being consequences of touching some ancient AtomApplication code. Can you open a separate issue for that one and fill out the template? I'm especially interested in your OS and the exact steps that result in an unexpected new window. We caught an issue that sounds very similar to that just before release and while we tested it out on all three platforms it's quite likely that we missed an execution path somewhere. (Fun fact: different methods get called when if you use the Ctrl-O shortcut, application:open-file command from the command palette, or run "File -> Open File..." from the system menu 🙃 )

🤔 Looks like we're attempting to use a previously opened file as a root directory. Two questions:

How exactly did you open the file in Atom? Did you run atom path/to/file from the command line, or open Atom from the dock and open the file with Ctrl-O, or some other sequence of events?

I've try these steps:

Both method from CLI  atom path/to/file.txt or GUI (by opening Atom from dock or right click file > Open With > Atom) produce error warning when I reopen Atom.
Opening file with Ctrl-O or "File -> Open File..."  or application:open-file command from the command palette. I quit Atom and reopen again > Atom still looking for the previous file I open from CLI  atom path/to/file.txt or GUI (and say not found)
Reopen previous closed project from CLI / GUI work. But reopen another project from Atom Menu > File > Reopen Project doesn't work at all.  (The same issue with #19149)

Can you share the contents of the file ${ATOM_HOME}/.atom/application.json between steps 2 and 3?

No application.json under ~/.atom directory between step 2 and 3.
The strange thing is when I run :
cat ${ATOM_HOME}/.atom/application.json
cat: /.atom/application.json: No such file or directory

when I do echo ${ATOM_HOME} it's empty and it's like Atom act their home dir is root of my OS directory ( / ) instead of my home user directory.
Note : The output of echo ${ATOM_HOME} is same both on my zsh or bash shells.

There's a good chance this is causing files to open in new windows, too, right? Seems like a recent update screwed a few things up.

I think that's unrelated, other than both being consequences of touching some ancient AtomApplication code. Can you open a separate issue for that one and fill out the template? I'm especially interested in your OS and the exact steps that result in an unexpected new window. We caught an issue that sounds very similar to that just before release and while we tested it out on all three platforms it's quite likely that we missed an execution path somewhere. (Fun fact: different methods get called when if you use the Ctrl-O shortcut, application:open-file command from the command palette, or run "File -> Open File..." from the system menu 🙃 )

Yes I've same issue about window behaviour like @joshmaines the others #19147

No application.json under ~/.atom directory between step 2 and 3.

Oh! Sorry, that was a mistype. It should be ${ATOM_HOME}/storage/application.json, and ATOM_HOME defaults to ${HOME}/.atom if unset. So the real path for you should be ${HOME}/.atom/storage/application.json.

Using the repro steps in the issue body (for step (1) I opened a file from the command line with atom README.md), I was able to reproduce and I see the file I opened from the command line listed when I check my application.json:
cat ~/.atom/storage/application.json
[{"initialPaths":["/Users/rsese/sandbox/test-open/README.md"]}]

This as after step (2) and it's the same if I close/restart Atom.

No application.json under ~/.atom directory between step 2 and 3.

Oh! Sorry, that was a mistype. It should be ${ATOM_HOME}/storage/application.json, and ATOM_HOME defaults to ${HOME}/.atom if unset. So the real path for you should be ${HOME}/.atom/storage/application.json.

Here is my ~/.atom/storage/application.json
[{"initialPaths":["/Users/ditatompel/atom-test/another-file-1.txt"]}]

Figuring that my ${ATOM_HOME} is empty, I've try to set ATOM_HOME to ${HOME}/.atom and try the same procedure, but still throw the same warning message.
Note : Try to Downgrade to Atom 1.35.1 and everything works normal as expected.

I'd just like to chime and say I'm having the same issue with File > Reopen Project where nothing happens regardless of if I have an atom window open or not on MacOS High Sierra 10.13.6. Downgrading to v1.35.1 like ditatompel suggested works as a temporary fix.

I reinstalled the atom before search in the issues, now, it seems that is a bug. looking forward the patch

Happening for me as well, except I can't even "Add Project Folder" for projects that are failing to reopen automatically or via menu.

Atom 1.36.1 and 1.37.0-beta1 have been released and include a fix for this issue.
I've had the same problem with 1.38.2 - basically, it never worked right on opening. Just a blank home window. Reopen Project didn't work either. And Reopen Last Item doesn't work.
 You may consider this an official warning.


Difference in Sage.jar between Windows and Linux builds

I have been working on rebuilding the Unraid Docker solution and was running into an issue with the Java8 docker as SageTV would run but a placeshifter would not connect.  The issue was related to a change in Java between 8 and 9 and Byte(ByteBuffer).
The odd thing is I often test using Java 8 on windows and had no issues connection a placeshifter...both builds were the latest 9.2.6 releases.
So I copied the Sage.jar from the windows release into the linux docker and restarted SageTV and now I was able to connect the placeshifter to this Java 8 docker.
In my research I find that you can build a jar using JDK9 with the --release=8 option and it will resolve the Byte/ByteBuffer compatibility issue.
So are we using this --release option for the Windows jar build and not the Linux build or what is different in the 2 build processes that makes the Linux build fail on Java 8 but the Windows build works?
Thoughts?

I obviously cannot speak for everyone, but this would allow the older docker container to work with Java 8 again which would address some compatibility issues with older plugins. On a side note, when I was building to fix an issue on Windows, I had to upgrade to Java 9 to be able to successfully compile the current code as it is on Github. I wasn't expecting that and spent a lot of time troubleshooting assuming I created the issue.

I’m pretty sure I built the last windows version with JDK 8 which would explain the difference. JDK9 was a result of needing to update the distribution for the automated build on GitHub.

I currently do not have a build system for windows nor Linux but as time permits I will set one up again.  If someone doesn't beat me to it I will test out the release=8 option under jdk9 and see what happens.
I think if we can support Java 8 as long as possible the better as we have many older in use plugins that could be affected.   The Jetty issue may be the biggest but I am close to a jetty 9 release that works well on Java 8+

I found a resolution to this.  I could not make it work with a build system that uses JDK9 as the --release option under Gradle is broken and Gradle does not plan to resolve it.  So I built a build system as follows...

JDK 11 and ubuntu 20 (I actually used "adoptopenjdk/openjdk11:jdk-11.0.11_9" docker as my base image).
Gradle 7.3.3 (needed as it supports --release)
Had to change the gradle.build a bit to support Gradle 7+ as compile has been removed and now needed to use implementation

I manually uploaded the linux installer to my jusjoken/sagetv-linux forked repo and setup a Java 8 docker to use that as the sagetv install file and the docker works well.
If anyone want to try out the docker it is the one available in unraid and dockerhub under "sagetvopen/sagetv-server-java8"
I can create a PR for these changes but it would also force the github workflow process to be updated as in the end that is what is building the linux releases.  I am not sure how to test those changes although in theory it should just be a matter of changing it to use ubuntu 20 and JDK 11.  The --release flag is in the updated build.gradle file
Let me know your thoughts and I will clean up the changes from my testing and create a PR.
Ken

For record purposes on this issue I updated the PR to include the workflow changes needed to test build on ubuntu 20.04 and use JDK 11 and it passed the PR tests....good news.  If the PR is accepted then the 9.2.7 build should run again on a java 8 docker (I guess I still need to change the version and push that too)

I did another PR to move the build process to use ubuntu 18.04 so the SageTV build would run on 18.04+. Although we could just expect users to update the linux os (that is what I did on the dockers), as 18.04 is under support for at least more than 1 more year I thought it better to also support 18.04.


Staffchat doesnt pop up in disc

i have discord srv setup and working but when i did "Staff chat" part staff doesnt work
Help when u can thank you :)

You'll have to provide a lot more information than just "it doesn't work".


[lit-next] Add dev mode warnings

Description: Potential warnings
Updating/LitElement

shadowed reactive properties
ShadyDOM use without platform-support
(optional?) changing properties after update
use of removed APIs like:

static render
static getStyles
adoptStyles
_getUpdateComplete
initialize
requestUpdateInternal
undefined behavior for attributes?
warning when requestUpdate is used and noting that it no longer returns a promise (ideally this would happen only if you await or then it but that may not be possible?)

lit-html

Warn on using directive in unsupported part type
Warn on bad HTML (e.g. scanner ends in unexpected state). This is already included but needs a test.

Acceptance criteria
PR landed.

Fixed via https://github.com/Polymer/lit-html/pull/1375.


AttesterSlashing with 0 indices and out of bounds indices

Based on this bug find in Nimbus, which was found thanks to the work by the fuzzing team (@zedt3ster and others), it showed how the attester slashing tests missed the high-index case. And cases for 0 attesting indices were missing as well.
We may want the spec to be more explicit about these cases, as the current approach is too implicit (in my opinion). For now, these tests help catch the case in other clients that may not have been given this input through fuzzing yet.
BLS is forced to be enabled in these tests, as that should catch it if not with anything else. But the high index may still cause a panic before that in a client. And the 0 indices case with BLS is important too, since the move away from empty aggregation was based on client BLS issues with empty aggregation there.

lgtm


Update FinRL.md (resolve: move to projects/python/FinRL.md)

combine with https://github.com/ColumbiaOSS/project-proposals-s2023/pull/279


Matrix Beta

In TEP-0090: Matrix, we introduced Matrix to support fanning out a PipelineTask with different combinations of Parameters.
This is an overarching issue to track feature requests and work we need to do before promoting Matrix to beta.

[x] #5925
[x] #6602
[ ] #5266
[x] #5267
[ ] https://github.com/tektoncd/pipeline/issues/7056

Optional, can be explored in the future if there are use cases:

[ ] #5268

/lifecycle frozen

We do not have any use case right now for Excluding combinations in a matrix so this will be excluded  https://github.com/tektoncd/pipeline/issues/5268
We are waiting to see if there are any use cases right now for consuming results from a matrix. See https://github.com/tektoncd/pipeline/issues/5266
cc: @jerop @pritidesai

@lbernick a humble request to not add something which is not proven as a blocker for this promotion.

Adding a recently discovered bug which has to be fixed before the promotion. A pipeline with matrix randomly fails with an error documented in https://github.com/tektoncd/pipeline/issues/7070.


stroke data submission

The dataset submitted into scholarsphere https://scholarsphere.psu.edu/concern/generic_works/3n583xt26b needs some curation.

The depositor replied giving me the go ahead to get a doi and update their license.
They will work on the README and data dictionary.


32-bit x86 version?

This is the kind I currently have at hand so you must trans-pile the code into an x86 version.

Nobody "trans-piles" x64 into x86. They just compile the source into x86. Also, no.


Can't get this to work with

Hey! I'd love to use this for picking and showing videos, but I'm currently stuck.
I can use the API to get a list of video assets, but trying to use react-native-video (master or your fork) crashes my app.
Any help or tips would be amazing!! :)
P.S. I tried to get the example app in this repo running, but failed to do so
package.json
	"dependencies": {
		"react": "~15.4.0",
		"react-native": "0.41.2",
		"react-native-photos-framework": "^0.1.1",
		"react-native-video": "git://github.com/olofd/react-native-video.git#react-native-photos-framework",
		...
	}

App
import RNPhotosFramework from 'react-native-photos-framework'
import Video from 'react-native-video'

...

  async componentDidMount() {
    const status = await RNPhotosFramework.requestAuthorization()

    if (status.isAuthorized) {
      const {assets} = await RNPhotosFramework.getAssets({
        startIndex: 0,
        endIndex: 1,
        includeMetadata: true,
        fetchOptions : {
          mediaTypes: ['video'],
          sourceTypes: ['userLibrary', 'cloudShared', 'itunesSynced'],
        }
      })
      this.setState({videos: assets})
    }
  }

...
  render() {
...

            {this.state.videos.map((video, i) => {
              console.log(video.video) // THIS IS UNDEFINED
              // return <Image key={i} source={{uri: video.uri}} style={{width : 100, height : 100}}></Image> // THIS WORKS
              return <Video  //THIS CRASHES
                source={{uri: video.uri}}
                key={i}
                resizeMode='cover'
                muted={false}
              />
            })}

Here's the crash:

Will take a look at this later today.
Skickat från min iPhone

24 feb. 2017 kl. 05:32 skrev Brandon Bayer<EMAIL_ADDRESS>Hey! I'd love to use this for picking and showing videos, but I'm currently stuck.
I can use the API to get a list of video assets, but trying to use react-native-video (master or your fork) crashes my app.
Any help or tips would be amazing!! :)
package.json
"dependencies": {
"react": "~15.4.0",
"react-native": "0.41.2",
"react-native-photos-framework": "^0.1.1",
"react-native-video": "git://github.com/olofd/react-native-video.git#react-native-photos-framework",
...
}
App
import RNPhotosFramework from 'react-native-photos-framework'
import Video from 'react-native-video'
...
async componentDidMount() {
const status = await RNPhotosFramework.requestAuthorization()
if (status.isAuthorized) {
  const {assets} = await RNPhotosFramework.getAssets({
    startIndex: 0,
    endIndex: 1,
    includeMetadata: true,
    fetchOptions : {
      mediaTypes: ['video'],
      sourceTypes: ['userLibrary', 'cloudShared', 'itunesSynced'],
    }
  })
  this.setState({videos: assets})
}

}
...
render() {
...
        {this.state.videos.map((video, i) => {
          console.log(video.video) // THIS IS UNDEFINED
          // return <Image key={i} source={{uri: video.uri}} style={{width : 100, height : 100}}></Image> // THIS WORKS
          return <Video  //THIS CRASHES
            source={{uri: video.uri}}
            key={i}
            resizeMode='cover'
            muted={false}
          />
        })}

Here's the crash:
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Thanks! So does any other library's that do video processing (like trimming) have to support the Photos framework to use it with this library? Or is there an easy way to "convert" the pk:// into asset-library:// ? Or some other means?
Thanks so much for your help!!!

hmm.. If asset.video is undefined, you must be using the wrong version of the library. The latest one is 0.1.2 make sure you are using that one and also reinstall my fork/branch of react-native video. Just tested this code in a clean react-native project with no issue, videos played:
npm i react-native-photos-framework --save && react-native link react-native-photos-framework
npm install git://github.com/olofd/react-native-video.git#react-native-photos-framework --save && react-native link
/**
 * Sample React Native App
 * https://github.com/facebook/react-native
 * @flow
 */

import React, { Component } from 'react';
import { AppRegistry, StyleSheet, Text, View, Image } from 'react-native';
import RNPhotosFramework from 'react-native-photos-framework';
import Video from 'react-native-video'

export default class test extends Component {

  constructor() {
    super();
    this.state = {
      images: []
    };
  }

  componentDidMount() {
    RNPhotosFramework.requestAuthorization().then((statusObj) => {
      if (statusObj.isAuthorized) {
        RNPhotosFramework.getAlbums({
          type: 'smartAlbum',
          subType: 'smartAlbumUserLibrary',
          assetCount: 'exact',
          fetchOptions: {
            sortDescriptors: [
              {
                key: 'title',
                ascending: true
              }
            ],
            includeHiddenAssets: false,
            includeAllBurstAssets: false
          },
          //When you say 'trackInsertsAndDeletes or trackChanges' for an albums query result,
          //They will be cached and tracking will start.
          //Call queryResult.stopTracking() to stop this. ex. on componentDidUnmount
          trackInsertsAndDeletes: true,
          trackChanges: false

        }).then((queryResult) => {
          const album = queryResult.albums[0];
          return album.getAssets({
            //The fetch-options from the outer query will apply here, if we get
            startIndex: 0,
            endIndex: 10,
            //When you say 'trackInsertsAndDeletes or trackAssetsChange' for an albums assets,
            //They will be cached and tracking will start.
            //Call album.stopTracking() to stop this. ex. on componentDidUnmount
            trackInsertsAndDeletes: true,
            trackChanges: false
          }).then((response) => {
            this.setState({
              images: response.assets
            });
          });
        });
      }
    });
  }

  renderVideo(asset, index) {
    return (<Video  //THIS CRASHES
      source={asset.video}
      style={styles.thumbVideo}
      key={index}
      resizeMode='contain'
      muted={false}
    />);
  }

  renderImage(asset, index) {
    if (asset.mediaType === 'video') {
      return this.renderVideo(asset, index);
    }
    return (
      <Image key={index} source={asset.image} style={{ width: 100, height: 100 }}></Image>
    );
  }

  render() {
    return (
      <View style={styles.container}>
        {this.state.images.map(this.renderImage.bind(this))}
      </View>
    );
  }
}

const styles = StyleSheet.create({
  container: {
    flex: 1,
    justifyContent: 'center',
    alignItems: 'center',
    backgroundColor: '#F5FCFF'
  },
  welcome: {
    fontSize: 20,
    textAlign: 'center',
    margin: 10
  },
  instructions: {
    textAlign: 'center',
    color: '#333333',
    marginBottom: 5
  },
  thumbVideo: {
    position: 'absolute',
    top: 0,
    left: 0,
    right: 0,
    bottom: 0,
    backgroundColor: 'transparent'
  }
});

AppRegistry.registerComponent('test', () => test);

You did write that you use master-branch of my fork of react-native-video. That would crash like this. You need to use the branch react-native-photos-framework of that fork.

Yay, 0.1.2 fixed it!
Thanks for your help!!


Keyboard shortcuts sometimes do not work

Describe the bug
When I press Ctrl+ V shortcut on my keyboard, nothing happens although I have files copied. I have to use paste button in context menu to actually paste my files.
To Reproduce
Steps to reproduce the behavior:

Copy any files
Go to destination folder
Press Ctrl + V
See error

Expected behavior
My files should be pasted by using shortcut.
Screenshots
None
Desktop (please complete the following information):

OS Version: Windows 10 20H2 19042.804
App version: v<IP_ADDRESS> (downloaded from MS Store)

Additional context
None
Log file
debug.txt

EDIT: I just realized that all shortcuts don't work in this app.

They do.. sometimes :(

Does closing/reopening the app adds times solve this issue?
Does "F5" shortcut work to refresh the folder?
Does "F2" to rename a file work?

I opened app one more time and all of them work!
So reopening app solved the problem (for now)

I found out the sequence of this bug occurs!!
When you open Files in another folder and closes it, reopening Files and navigate it into another folder will make keyboard shortcuts (except up and down) don't work. Furthermore, if you run the keyboard shortcut and navigate into another folder again, it will cause Files to crashes.
Proof:
https://user-images.githubusercontent.com/61906684/110288087-0f5a1900-801a-11eb-8ecf-7b9c3b378a75.mp4


Remove duplicated text

Learn Build status updates of commit 455afb5:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

articles/machine-learning/how-to-create-component-pipeline-python.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

@likebupt
Can you review the proposed changes?
When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

#sign-off


Combine routes, metadata, layouts into exports var on client side

As suggested in https://github.com/MoOx/statinamic/pull/145#issuecomment-184228487 , we already doing this in server side (#145)

Not a fan of this one. We are supposed to remove code, not add more.

Current coverage is 65.68%

Merging #162 into next will decrease coverage by -0.09% as of e2535d5

@@             next    #162   diff @@
=====================================
  Files          34      34
  Stmts         409     408     -1
  Branches        0       0
  Methods         0       0
=====================================
- Hit           269     268     -1
  Partial         0       0
  Missed        140     140

Review entire Coverage Diff as of e2535d5

Powered by Codecov. Updated on successful CI builds.

Can't think of a better way. Webpack will complaint if using require(require.resolve(...))

I am just going to revert the thing I asked you in the first place (I will drop exports.js file).

Your call. Feel free to close this PR


error: selectr need and element id to work

2.2 *  seletr need and element id to work
this is when using exactly the documentation suggested, due the right way its document.getelementById in novice cases, althrought i dont nkow if its the right way
documentation said that i need to do:
var mySelect = document.querySelector('select');
var mySelectr = new Selectr(mySelect);

but the document.querySelector('select') returns the firts input selector and this confuses novice users when try to use in and multi html where the id and name input select are same, and are multiple selects inputs
so documentation must be more clear, or are incorrect

As noted in my previous reply, the documentation has been updated.

i'll lose due all i posted in #2


License?

Hello,
I'm really interested on this project and also want to contribute here, but first I have to know if you are planning to release it open sourced (as Vue Storefront is), are you?
Thanks!

Hey!
I'm really glad you'd iike to help us. We are still preparing the repo for external contributions but it should be ready on Monday/Tuesday.
It'll be MIT licensed as Vue Storefront is :)
Can you please ping me on VS slack so we can discuss your possible contributions?

Perfect!
Sure, I'll call you on slack.