finalize() should not be overridden

Overriding Object.finalize() method is very bad for GC. Instead, sun.misc.Cleaner, or intermediate that automatically detects the presence of java.lang.ref.Cleaner and uses it if present (something like this: https://github.com/OpenHFT/Chronicle-Core/blob/067e963a0b79775d9dee33cb13b5fa5b7e49683e/src/main/java/net/openhft/chronicle/core/cleaner/impl/reflect/ReflectionBasedByteBufferCleanerService.java, for example) should be used.

Looks like a good improvement to have. I don't have much experience with the Cleaner-s. Any help here will be very welcome.

Are cleaners needed in this scenario? The finalize method calls close(). What's the use case? Can we just leave finalize out? Was there an actual bug? The user should close things themselves so finalize is just extra work that the GC has to do.

I have done what's Gzip streams are doing - if you forgot to close the stream the underlying file descriptor will be closed on the next GC cycle when the steam object is collected. I had a bug (https://github.com/luben/zstd-jni/issues/26) that leaked memory and FDs by not closing the streams automatically and users were expecting it.

BTW, calling .close() is recommended - I have noticed an edge case that may lead to leaked file descriptors: if the file was meanwhile deleted, the fd is not released. It's not specific to the library and can be observed without any compression on top if InputStream

Gzip stream use Cleaners as mentioned earlier in this post. Finalizers are deprecated. Maybe it'd just be similar to add a shutdown hook to close the resources on closure or similar.

Yes, it makes sense. May be we were looking before the Inflater was ported to Cleaners.

Just looked, Inflater and Deflater are still using finalize():
http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/java/util/zip/Inflater.java#l382
http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/java/util/zip/Deflater.java#l548
and Zip/Gzip streams rely on them for closing.
BTW, I am open to PR moving us to java.lang.ref.Cleaner but how do we still keep compatibility with JDKs < 9?

The deprecation of finalizers and cleaners happened after JDK8.

Yes, but we still support back to JDK6. Is there a way to implement bi-modal behavior depending on which JDK we are loaded in?

If you look at the Java 9 source code for cleaner it's a Weak/Soft/Phantom list of objects + a background thread or similar that executes the action. You can create a simple daemon thread that does something similar. Make it run on shutdown and/or after quite a long time (so it doesn't cause performance impact).

I think creating daemon threads and/or using shutdown hooks is wrong in a "stateless" library like zstd-jni.
@luben link in the first message of this issue leads to an example of bi-modal code.

Is it possible too use the fork/join pool to do the cleanup?  Probably not, WeakReferenceQueue blocks.
There are some other ways to do the bi-modal behavior, you can have both implementations in their own private static inner classes that hold static members of an interface you need to use to clean,  and try to touch each of these values from static context in turn until one of them works, catching the appropriate class/method/security error.

@scottcarey this is pretty much what does the code linked in the first message in this thread does. It just probes "sun.misc.Cleaner" and "jdk.internal.ref.Cleaner" directly. "java.lang.ref.Cleaner" can be added to the mix too.

@leventov Agree with you.  I have encountered this problem with zstd-jni.
Finalizers are inherently problematic and their use can lead to performance issues,
deadlocks, hangs, and other problematic behavior.
Furthermore the timing of finalization is unpredictable with no guarantee that a finalizer will be called.
See https://bugs.openjdk.java.net/browse/JDK-8165641
Use of finalizers is yet another source of potential memory leak issues. Whenever a class’ finalize() method is overridden, then objects of that class aren’t instantly garbage collected. Instead, the GC queues them for finalization, which occurs at a later point in time.
Additionally, if the code written in finalize() method is not optimal and if the finalizer queue cannot keep up with the Java garbage collector, then sooner or later, our application is destined to meet an OutOfMemoryError.

Was hit by this issue in production, so it's not a theoretic problem.
We use zstd as a compression format for a large batch of data files using Apache Flink (version 1.9 running on adoptopenjdk 8u222). Sometimes on a job restart (due to crash somewhere in our java code) we experienced JDK crashes with the following backtrace:
Stack: [0x00007f53a4e7d000,0x00007f53a4f7e000],  sp=0x00007f53a4f7c560,  free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libc.so.6+0x408f0]  abort+0x230
C  [libc.so.6+0x9090a]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  com.github.luben.zstd.ZstdInputStream.freeDStream(J)I+0
j  com.github.luben.zstd.ZstdInputStream.close()V+12
j  com.github.luben.zstd.ZstdInputStream.finalize()V+1
J 5784 C2 java.lang.ref.Finalizer.access$100(Ljava/lang/ref/Finalizer;Lsun/misc/JavaLangAccess;)V (6 bytes) @ 0x00007f5391919838 [0x00007f5391919660+0x1d8]
j  java.lang.ref.Finalizer$FinalizerThread.run()V+45
v  ~StubRoutines::call_stub

Flink creates a separate ClassLoader for each job on a separate thread. When job is done, the classloader is closed and all classes (and JNI libraries) are unloaded. The issue is that (I guess) on jvm8 there is no guarantee that finalizers will be run before the JNI library unload happening.
Reproducer code (sorry for scala), which tries to do something similar to Flink:
import java.io.{ByteArrayInputStream,ByteArrayOutputStream,File,FileInputStream,InputStream}
import java.net.URLClassLoader

import org.apache.commons.io.IOUtils

object Main {
  val url = new File("/home/shutty/Downloads/zstd-jni-1.4.3-1.jar").toURL
  val in = IOUtils.toByteArray(new FileInputStream(new File("/home/shutty/Downloads/demo.zst")))

  def main(args: Array[String]): Unit = {
      (0 to 100000).toList.par.foreach(openclose)
  }

  def openclose(id: Int) = {
    var cl = new URLClassLoader(Array(url))
    val result = read(cl)
    cl = null
    System.gc()
  }

  def read(cl: ClassLoader): String = {
    val zstdClass = cl.loadClass("com.github.luben.zstd.ZstdInputStream")
    val constructor = zstdClass.getConstructor(classOf[InputStream])
    val zstd = constructor.newInstance(new ByteArrayInputStream(in)).asInstanceOf[InputStream]
    val buf2 = new String(IOUtils.toByteArray(zstd))
    // yes, we did not close the ZstdInputStream here intentionally
    buf2
  }
}

On jvm11 everything works fine. The hack we did to solve this issue is to wrap the ZstdInputStream and override its finalizer with empty method.

Scala is fine.
@shuttie, what if we add constructor param (or setter) to make the finalizer noop, something similar with the wrapper you created?
In your production code, are you closing the ZstdInputStream manually? If not it may leak the memory allocated in the zstd native code.

@shuttie , in v1.4.4-3 I have added options to all classes no make finalizers noop, e.g. object.setFinalize(false) that may be simple for you than using wrapper.

If out.write(dst, 0, (int) dstPos); will throw an exception due to underlying OutputStream which happens frequently , /* Opaque pointer to Zstd context object */ private final long stream; is still left unclosed and creates Memory Leaks in Big Data Systems and Long Running application like Spark / Hadoop etc,   where the underlying filesystem switchover / Bad Data Nodes / Lease Expired etc will give exception on underlying OutputStream Frequently.

Finalizers (and other GC magics like phantom refs) still have GC overhead even for users that don't depend on them for cleanup, i.e users that always close their streams.
An alternative proposal: factor out the existing streams behavior minus finalize() into new base classes, and leave the finalize in the existing streams classes which extend the new ones. This lets users who always close and want to avoid finalizer overhead opt-in, and doesn't break existing users relying on GC magics for cleanup.

@raifo , that seems sane approach. I can put it in place shortly.

@raifo , I have added NoFinalizer base classes for the Input/Output streams in: https://github.com/luben/zstd-jni/commit/be9be47fae23b5510344b67f9c0f7b258c2f5890 . Any feedback is welcome. I intend to release it soon with the RecyclingBufferPool optimizations in the later commits. Any recommendation there will be also appreciated.

Just pushed v1.4.8-4 with the new classes without finalizers.

FWIW, by having ZstdOutputStream extend ZstdOutputStreamNoFinalizer a number of breaking changes were introduced.  Code that compiled on 1.4.8-3 no longer compiles with 1.4.8-4.
e.g.
ZstdOutputStream wrapper = new ZstdOutputStream(o).setCloseFrameOnFlush(true);

Results in this compiler error:
... error: incompatible types: ZstdOutputStreamNoFinalizer cannot be converted to ZstdOutputStream

@Logic-32 , that's a good point. There is pro/cons of keeping the API stability vs tying us to upstream versions. Regarding the runtime issue, most likely they would no break but will not run finalizers.
I will try some F-bounded generics magic this weekend to see if it applies and solves the problem, but if you have any other suggestion how to remove the finalizers in backward compatible way, please chime.

In part, it'd be nice to at least see a major version bump if you pursue the current change.  That way, consumers know to expect some breakage.
Semantic versioning aside, a good solution would depend on how backwards compatible you want to be.  If you don't care about JRE's < 9 then you could remove the finalize() override, add the Cleaner, and then have ZstdOutputStreamNoFinalizer extend ZstdOutputStream instead.
That way, all existing implementations continue to work (thanks to proper inheritance) and then the constructor for ZstdOutputStreamNoFinalizer can just skip the Cleaner call.

Yes, we still hugely care about JVM8 though we still compile and work on JVM6.
Regarding major version bump - it will stop correspond to the upstream Zstd version and I a little but hesitant - I would prefer backwards compatible solution.

Actually looking in the runtime case - it will not be a problem. It still breaks the compilation but I think I can fix this.

@Logic-32 , I  pushed a commit that I thing it should fix it. Can you try it by building it locally and compiling your software against it?

Thanks for the quick turn around!  Just reviewing the latest commit I believe you still need to add generics to ZstdOutputStream before that will work.  Otherwise, T will still be a ZstdOutputStreamNoFinalizer.
But, I wouldn't do that yet as it makes working with ZstdOutputStreamNoFinalizer a bit awkward.  A good static-code analyzer (or even IDE) would recommend declaring any instance of ZstdOutputStreamNoFinalizer as ZstdOutputStreamNoFinalizer<ZstdOutputStreamNoFinalizer> while ZstdOutputStream would remain generic-less.  Which just makes working with it a bit unpleasant.
However, I think you can get away with what you're going for, and still be backwards compatible, by overriding each setter in ZstdOutputStream and changing the return type to ZstdOutputStream without relying on the generics.  It's not as pleasant to maintain but better from a consumer standpoint.
e.g.
    @Override
    public synchronized ZstdOutputStream setCloseFrameOnFlush(boolean closeOnFlush) throws IOException {
        return (ZstdOutputStream) super.setCloseFrameOnFlush(closeOnFlush);
    }

Also, while we're on the topic of better for consumers, here's something to debate: how would you feel about moving ZstdOutputStreamNoFinalizer to a different package than ZstdOutputStream and keeping the name ZstdOutputStream?  Deprecating the current ZstdOutputStream in favor of the new one should be considered important as relying on the GC to close resources is strongly discouraged since you never know when GC will occur.  Since most developers are lazy however, they'll be more inclined to use ZstdOutputStream over ZstdOutputStreamNoFinalizer simply because the name is shorter.  The only problem with this approach is that you not only have to come up with a name for the new package but also hope that no one complains about requiring a package name to disambiguate the two.  Though, why you'd ever import both in any case is beyond me.
I'll try and check in over the weekend but will otherwise probably not be available until next week.  Let me know what your thoughts are!

Hmm, that's interesting idea to use the packages for disambiguation. Though I could not come with better name than zstd.
Yes, working with the NoFinaliser classes becomes a little bit more cumbersome. Now thinking, we can introduce an empty subclass for the NoFinalizer classes that just fill the generics - another wart on the implementation side.

I went ahead with creating *Base classes that have all the logic and 2 facades - with or without the finalizer that fill the generic param - this should be compatible with both 1.4.8-3 and previous, and with 1.4.8-4.

Thanks for the updates @luben.  I think those should work out nicely for us.
FWIW, we're a java shop that likes to be on the latest version of things.

@Logic-32 , I have pushed v1.4.8-5 with the latest changes.

I will keep the issue open to track adding Cleaners and further changes.

@Logic-32, as you suspected there are also runtime API problems with v1.4.8-4 and 5. The Avro/Parquet/Spark ecosystem noticed when trying to update dependencies (#161). I have released v1.4.8-6 that achieves the same result but keeping the runtime API compatibility - now the ZstdInputStream/ZstdOutputStream compose/delegate to the NoFinalizer classes: removing the inheritance restored the runtime contract.

Hi @luben, we saw some finalizer issue from ZstdDecompressCtx. I noticed there's "setFinalize" to disable finalizer for AutoCloseBase. But from our code path
  [ 0] java.lang.ref.Finalizer
  [ 1] java.lang.ref.Finalizer.register
  [ 2] com.github.luben.zstd.AutoCloseBase.<init>
  [ 3] com.github.luben.zstd.ZstdDecompressCtx.<init>
  [ 4] com.github.luben.zstd.Zstd.decompressByteArray

there seems no way to set it without touching (https://github.com/luben/zstd-jni/blob/3df7edc8e7cce6d0fd195dbb7cdfe3edbf23fb58/src/main/java/com/github/luben/zstd/Zstd.java#L407). Is there any workaround we can try? Thanks!

Yes, setFinalize(false) will just make the finalizer noop bit will not remove it completely. Recently I started adding a new hierarchy for the streams classes that don't override the finalizer. May be I can continue with the Compress/Decompress contexts.

Thanks @luben!
BTW, I think our issue occurred since (https://github.com/luben/zstd-jni/commit/5209351fc8e071084a3e6df13c0f60bd13f916f6), which led to more 'Finalizer' in heap and longer GC pause.

The idea of the Compress/Decompress Ctx is that you can create an instance and use them to compress/decompress multiple independent messages. This lowers the overhead. The particular commit forwards the Zstd methods to the new classes as we don't want to maintain duplicate code-paths.

Gotcha! Thanks for the explanation @luben!

BTW, is it feasible to add an option like "decompressByteArrayWithoutCtx" in Zstd class if we want to avoid Ctx? Thanks! @luben

Internally it will create and discard the context - either we do it on the java side, or it will happen on the C side.

i was thinking more of having an extra java api to directly access the old C code
JNIEXPORT jlong JNICALL Java_com_github_luben_zstd_Zstd_decompressByteArray
  (JNIEnv *env, jclass obj, jbyteArray dst, jint dst_offset, jint dst_size, jbyteArray src, jint src_offset, jint src_size) {
    size_t size = (size_t)(0-ZSTD_error_memory_allocation);
    if (dst_offset + dst_size > (*env)->GetArrayLength(env, dst)) return ERROR(dstSize_tooSmall);
    if (src_offset + src_size > (*env)->GetArrayLength(env, src)) return ERROR(srcSize_wrong);
    void *dst_buff = (*env)->GetPrimitiveArrayCritical(env, dst, NULL);
    if (dst_buff == NULL) goto E1;
    void *src_buff = (*env)->GetPrimitiveArrayCritical(env, src, NULL);
    if (src_buff == NULL) goto E2;
    size = ZSTD_decompress(dst_buff + dst_offset, (size_t) dst_size, src_buff + src_offset, (size_t) src_size);
    (*env)->ReleasePrimitiveArrayCritical(env, src, src_buff, JNI_ABORT);
E2: (*env)->ReleasePrimitiveArrayCritical(env, dst, dst_buff, 0);
E1: return size;

Do we have any estimation on perf penalty for not using ZSTD_decompressDCtx?

I think they will perform the same. The penalty is maintaining duplicate code.

I think they will perform the same. The penalty is maintaining duplicate code.

I see. It might be cleaner to just add Ctx without finalizer then.
I took a look at the code, and finalizer in Ctx was from AutoCloseBase. Maybe we can extract "void finalize()" from AutoCloseBase to its subclasses, and then make two versions of Ctx like ZstdInputStream/ZstdOutputStream (the one with finalizer wraps non-finalizer one)?
BTW, the issue we have is on presto (https://github.com/prestodb/presto/pull/15028).

Yes, I am thinking we may not even need the finalizer. I just have to make sure we manually close the contexts everywhere we use them.

Yes, I am thinking we may not even need the finalizer. I just have to make sure we manually close the contexts everywhere we use them.

Thanks! Let me know if something I could help!

@lic9, try v1.4.9-2 : it does not have finalizers in the Base class

@lic9, try v1.4.9-2 : it does not have finalizers in the Base class

Thanks! Just tested it and the issue seemed gone. Will run more tests.

@luben we have upgraded to v1.4.9-2 in presto: https://github.com/prestodb/presto/pull/15966
thanks for the fix!

Just do let you know, finalization will likely be deprecated for removal, see JEP 421.

Just do let you know, finalization will likely be deprecated for removal, see JEP 421.

yes, we will need some kind of major version bump when that comes in effect. And we can simplify a bit


Set up a calendly account

[x] create account
[x] customize
[x] Add three morning time blocks
[x] Add link to welcome email
[ ] Integrate Zoom (set up PS Zoom account)

@lukefretwell there's a Zoom integration. Should we use ProudCity's zoom account or mine? Or set up a free account for Proudly Serving?

@rebeccawoodbury done with customizing.
These workflows might be helpful to set up. I add the 24-hour prior automated reminder one.

@rebeccawoodbury disregard workflow comment. Just seeing that it's a 14-day trial.


Navbar alignment classes link is broken in v4 docs

The "alignment classes" link in http://v4-alpha.getbootstrap.com/components/navbar/#basics is broken

+1

The question is, do these still exist? One point down .pull-{left,right} are mentioned to "quickly align sub-components".

This was fixed in #17389.


⚠ API - Lens has degraded performance

In 99c172f, API - Lens ($LENS_API_URL) experienced degraded performance:

HTTP code: 503
Response time: 349 ms

Resolved: API - Lens performance has improved in 7bcb5b6 after 6 minutes.


removing master custom script extension from scaling templates

What this PR does / why we need it:
Currently old clusters that are scaled with a new acs-engine will lose their azure.json and parts of their kubelet config. Causing the cluster to stop working. this is caused by the switch from positional arguments to named variables in our cse

thanks for quick review and yes lets get E2E


No Alt-F3 (Windows or Linux) or Command+Ctrl+G (Mac OS X) to highlight every instance of the word in the document?

Any plan to introduce 'highlighting every instance of the word in the document'?

hi terribly sorry for missing this completely!!
i am surprised this doesn't work out of the box, it's part of what's available in the default keymap in codemirror's sublime keymap.

Still not supported?

just tested it myself after installing my plugin again just for this (i don't use sublime keybinds anymore)
it works for me. there must be something on your system that's intercepting the Alt-F3 combo

Any news? If not I would like to close this issue :smile:


Getting Unhandled Exception

Unhandled Exception : java.lang.RuntimeException: /app/.local/share/sfdx/client/7.176.1-458b658/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.28' not found (required by /app/.local/share/sfdx/client/7.176.1-458b658/bin/node)`
Getting above exception while trying to deploy to a developer org using deploy to salesforce button.

I'm also experiencing this issue.
Unhandled Exception : java.lang.RuntimeException: /app/.local/share/sfdx/client/7.176.1-458b658/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.28' not found (required by /app/.local/share/sfdx/client/7.176.1-458b658/bin/node)
`

me too

Hoping for an update on this issue soon, this is causing a ton of disruption for us.

I am facing the same issue.

Hoping we see a fix soon, this has actually caused a pretty big disruption for our business

Looking at this now

Just worked for me...

@justin-lyon i just tried with your repo and also works fine...while things seem to be fine above also - i have also redeployed apone a new Heroku stack - so curious to see if anyone can still repo this error?

@afawcett thanks i just confirmed the same. The gremlin is no longer there. i wonder what it was. nothing free tier heroku related?

Good news. @justin-lyon - nope nothing free tier releated

@afawcett It is resolved now. Thanks


Extend support for Python 3.10

Description
Python 3.10 is due to release on October 2021 and Fatiando libraries should support it too.
There's no much work to do so, we would need to expand the automated tests on CIs to Python 3.10, add 3.10 to the setup.py and check if every test pass.
This issue was inspired on @hugovk contribution to Pooch: https://github.com/fatiando/pooch/pull/260
Apply changes to
Libraries:

[ ] boule
[ ] erizo
[ ] harmonica
[x] pooch
[ ] rockhound
[ ] verde

Further instructions
In order to solve this issue we need to start opening Pull Requests on each
repository listed above. Optionally, we can open Issues on each repository if
further discussion specific to that repository is needed.
Remember to mention this Issue on every Issue or Pull Request opened on each
repository for keeping a record by adding something like:
Related to fatiando/maintenance#15

A quick checklist:

Open an Issue on each repository listed above (optional). Mention this Issue
on them.
Open a Pull Request on the same repository to tackle the Issue. Mention this
Issue on them.
Check-off the repository on the list above once the Pull Request is merged.
Close this issue when all items are checked-off.

We want your help
We know that maintenance tasks are very demanding, so we don't expect a single
person to tackle this issue by themselves. Any help is very welcomed, so please
comment below that you want to take care of the changes on any repository and
we will assign it to you.

boule can be checked off the list: https://github.com/fatiando/boule/pull/87


BUG: stats.weibull_min: poor fit results on shape parameters

Describe your issue.
I'm trying to fit frequency data. Since the use of frequency data is not supported I expand the data (this should not be an issue, apart from performances). The problem is that I get a poor result on fitting. I know location is close to 0.7. This is what I get with my attached code example:
(0.8959964805260656, 0.7499999999999998, 1.6443551275182964)
While in MATLAB using (I don't have a 3 parameters weibull in MATLAB so I subtract 0.7 from data):
wblfit(H-0.7,[],[],nrep);
I get:
scale=1.63052680315750 shape=1.16994972402310
I tried to use stats.fit with brute force optimizer and bounds, results gets a little better, but still far from MATLAB. I also tried to simulate a a 2-parameters weibull with floc=0 and expanded-0.7 results are better but not as good as MATLAB.
Reproducing Code Example
import numpy as np
import scipy.stats as stats

H = [15.25, 13.75, 12.75, 12.25, 11.75, 11.25, 10.75, 10.25, 9.75, 9.25, 8.75, 8.25, 7.75, 7.25, 6.75, 6.25, 5.75, 5.25, 4.75, 4.25, 3.75, 3.25, 2.75, 2.25, 1.75, 1.25, 0.75]

nrep = [1, 1, 1, 2, 5, 20, 17, 18, 37, 66, 112, 151, 215, 352, 525, 750, 1130, 1731, 2649, 4002, 6083, 9610, 14990, 21362, 26898, 26783, 13482]

H = np.sort(H)[::-1]
expanded = np.repeat(H, nrep)

parameters = stats.weibull_min.fit(expanded, loc=0.7, method='mle')

print(parameters)

Error message
No error message, but results are not good.

SciPy/NumPy/Python version and system information
1.13.0 1.26.2 sys.version_info(major=3, minor=10, micro=11, releaselevel='final', serial=0)
{
  "Compilers": {
    "c": {
      "name": "gcc",
      "linker": "ld.bfd",
      "version": "10.3.0",
      "commands": "cc"
    },
    "cython": {
      "name": "cython",
      "linker": "cython",
      "version": "3.0.10",
      "commands": "cython"
    },
    "c++": {
      "name": "gcc",
      "linker": "ld.bfd",
      "version": "10.3.0",
      "commands": "c++"
    },
    "fortran": {
      "name": "gcc",
      "linker": "ld.bfd",
      "version": "10.3.0",
      "commands": "gfortran"
    },
    "pythran": {
      "version": "0.15.0",
      "include directory": "C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pip-build-env-3bfntp2g\\overlay\\Lib\\site-packages/pythran"
    }
  },
  "Machine Information": {
    "host": {
      "cpu": "x86_64",
      "family": "x86_64",
      "endian": "little",
      "system": "windows"
    },
    "build": {
      "cpu": "x86_64",
      "family": "x86_64",
      "endian": "little",
      "system": "windows"
    },
    "cross-compiled": false
  },
  "Build Dependencies": {
    "blas": {
      "name": "openblas",
      "found": true,
      "version": "0.3.26.dev",
      "detection method": "pkgconfig",
      "include directory": "/c/opt/64/include",
      "lib directory": "/c/opt/64/lib",
      "openblas configuration": "USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS= NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= ZEN MAX_THREADS=24",
      "pc file directory": "c:/opt/64/lib/pkgconfig"
    },
    "lapack": {
      "name": "openblas",
      "found": true,
      "version": "0.3.26.dev",
      "detection method": "pkgconfig",
      "include directory": "/c/opt/64/include",
      "lib directory": "/c/opt/64/lib",
      "openblas configuration": "USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS= NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= ZEN MAX_THREADS=24",
      "pc file directory": "c:/opt/64/lib/pkgconfig"
    },
    "pybind11": {
      "name": "pybind11",
      "version": "2.12.0",
      "detection method": "config-tool",
      "include directory": "unknown"
    }
  },
  "Python Information": {
    "path": "C:\\Users\\runneradmin\\AppData\\Local\\Temp\\cibw-run-ohobiy3m\\cp310-win_amd64\\build\\venv\\Scripts\\python.exe",
    "version": "3.10"
  }
}

Also - when you use fit with method='mle', fit performs maximum likelihood estimation, so the objective is to maximize the likehood function. In this case, the log-likelihood function (the negative of weibull_min.nnlf) is higher for the the parameters returned by SciPy than the parameters returned by Matlab (whether you fix the location or not).
parameters = stats.weibull_min.fit(expanded, loc=0.7, method='mle')  # *guess* location = 0.7
-stats.weibull_min.nnlf(parameters, expanded)  # -142876.61696993175

parameters = stats.weibull_min.fit(expanded, floc=0.7, method='mle')  # *fix* location = 0.7
-stats.weibull_min.nnlf(parameters, expanded)  # -186163.09044160385

parameters = 1.16994972402310, 0.7, 1.63052680315750  # Matlab
-stats.weibull_min.nnlf(parameters, expanded)  # -186180.68880717325

Since there is not a bug here, I'll go ahead and close.


zappa package fails with zappa from git and slim_handler

Context
If you're trying to run a zappa as an editable from a git repo, it works, except if you have slim_handler enabled.
Expected Behavior
It should build a zip with zappa inside.
Actual Behavior
It fails with a traceback like this when building the handler_venv:
Traceback (most recent call last):
  File "/Users/joao/testes/zappa-1/1/ve/src/zappa/zappa/cli.py", line 2693, in handle
    sys.exit(cli.handle())
  File "/Users/joao/testes/zappa-1/1/ve/src/zappa/zappa/cli.py", line 504, in handle
    self.dispatch_command(self.command, stage)
  File "/Users/joao/testes/zappa-1/1/ve/src/zappa/zappa/cli.py", line 543, in dispatch_command
    self.package(self.vargs['output'])
  File "/Users/joao/testes/zappa-1/1/ve/src/zappa/zappa/cli.py", line 633, in package
    self.create_package(output)
  File "/Users/joao/testes/zappa-1/1/ve/src/zappa/zappa/cli.py", line 2171, in create_package
    venv=self.zappa.create_handler_venv(),
  File "/Users/joao/testes/zappa-1/1/ve/src/zappa/zappa/core.py", line 403, in create_handler_venv
    copytree(os.path.join(current_site_packages_dir, z), os.path.join(venv_site_packages_dir, z))
  File "/Users/joao/testes/zappa-1/1/ve/src/zappa/zappa/utilities.py", line 37, in copytree
    lst = os.listdir(src)
NotADirectoryError: [Errno 20] Not a directory: '/Users/joao/testes/zappa-1/1/ve/lib/python3.6/site-packages/zappa.egg-link

Possible Fix
Use the function for copying editables as when slim_handler is false.
Steps to Reproduce
With the following zappa_settings.json:
{
    "dev": {
        "project_name": "1",
        "runtime": "python3.6",
	"slim_handler": true
    }
}

Run on a virtualenv: pip install -e git+https://github.com/miserlou/zappa#egg=zappa
and then zappa package.
Your Environment

Zappa version used: 0.46.1 from git
Operating System and Python version: Mac OS X 10.12.6, python 3.6.4 (from home brew)
The output of pip freeze:

argcomplete==1.9.3
base58==1.0.0
boto3==1.7.45
botocore==1.10.45
certifi==2018.4.16
cfn-flip==1.0.3
chardet==3.0.4
click==6.7
docutils==0.14
durationpy==0.5
first==2.0.1
future==0.16.0
hjson==3.0.1
idna==2.7
jmespath==0.9.3
kappa==0.6.0
lambda-packages==0.20.0
pip-tools==2.0.2
placebo==0.8.1
python-dateutil==2.6.1
python-slugify==1.2.4
PyYAML==3.12
requests==2.19.1
s3transfer==0.1.13
six==1.11.0
toml==0.9.4
tqdm==4.19.1
troposphere==2.3.0
Unidecode==1.0.22
urllib3==1.23
Werkzeug==0.14.1
wsgi-request-logger==0.4.6
-e git+https://github.com/miserlou/zappa@d99c193e32733946fb52a4f9b2bdfd1d2929ba49#egg=zappa

Your zappa_settings.py: already above.

I was able to solve this by cloning the repo, navigating to it in my virtual environment and using python setup.py install. This installs it in a readable format.

There is another solution. Just remove the "-e" flag in the last line of the requirements.txt file and then reinstall the zappa package. If you have already installed zappa as editable package, you'll remove the editable package first.

having this same issue with 0.47.1, with or without slim_handler
"dev": {
        "app_function": "app.init.app",
        "profile_name": "...",
        "manage_roles": false,
        "role_name":"...",
        "role_arn": "...",
        "project_name": "...",
        "runtime": "python3.6",
        "s3_bucket": "..."
    },

Traceback (most recent call last):
  File "/Users/chrisbolman/.pyenv/versions/3.6.5/lib/python3.6/site-packages/zappa/cli.py", line 2712, in handle
    sys.exit(cli.handle())
  File "/Users/chrisbolman/.pyenv/versions/3.6.5/lib/python3.6/site-packages/zappa/cli.py", line 509, in handle
    self.dispatch_command(self.command, stage)
  File "/Users/chrisbolman/.pyenv/versions/3.6.5/lib/python3.6/site-packages/zappa/cli.py", line 546, in dispatch_command
    self.deploy(self.vargs['zip'])
  File "/Users/chrisbolman/.pyenv/versions/3.6.5/lib/python3.6/site-packages/zappa/cli.py", line 717, in deploy
    self.create_package()
  File "/Users/chrisbolman/.pyenv/versions/3.6.5/lib/python3.6/site-packages/zappa/cli.py", line 2227, in create_package
    disable_progress=self.disable_progress
  File "/Users/chrisbolman/.pyenv/versions/3.6.5/lib/python3.6/site-packages/zappa/core.py", line 603, in create_lambda_zip
    copytree(site_packages, temp_package_path, metadata=False, symlinks=False, ignore=shutil.ignore_patterns(*excludes))
  File "/Users/chrisbolman/.pyenv/versions/3.6.5/lib/python3.6/site-packages/zappa/utilities.py", line 41, in copytree
    lst = os.listdir(src)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/chrisbolman/Dropbox/br/venv-slack/lib/python3.6/site-packages'

@ChrisBolman were you able to get it working on python 3.6 ? I am having a similar problem as yours. I've tried downgrading Zappa to 0.45.1 but didn't solve it.

@sa77 unfortunately no, by default, even upgrading to zappa 0.51.0 by default I still get PATH issue between .pyenv/versions/3.6.5/lib/python3.6/site-packages/zappa/utilities.py and /Users/.../lib/python3.6/site-packages (outside pyenv). reinstalling python 3.6.5, creating a new virtualenv, and/or uninstalling + reinstalling zappa don't automatically resolve.


Exit emit doesnt working

I called exit emit like below.
<VOnboardingWrapper ref="wrapper" :steps="steps" :options="options" @exit="onExit"/>
const onExit = () => {
console.log('onexit');
};
But emit doesnt call onExit method.

Hi @denizzeybek thank you for catching this bug. It is fixed in the new release but please make sure you are using the lowercase @exit not the @EXIT.


Update how-to-configure-api-proxy-module.md

Changed default port for Azure Blob Storage on IoT Edge.

Learn Build status updates of commit 9649c9c:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

articles/iot-edge/how-to-configure-api-proxy-module.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

Hi @auyidi1 ,
Thanks for the fix.  Looks good.
Thanks!
Pat
#sign-off

ready to merge after Ignite content freeze


Duplicate KeyPair/Roles on AWS

What happened:
Shoot gets stuck during creation with the error message that a key pair and two roles (one with "nodes" and one with "bastion") belonging to the shoot are already present on the infrastructure and thus can't be created.
(Currently trying to reproduce the problem, will add exact error message afterwards)
What you expected to happen:
The shoot is created.
How to reproduce it (as minimally and precisely as possible):
Not completely sure, but it seems that this happens when the seed needs to scale up during shoot creation.
Anything else we need to know?:
Has been observed for AWS shoots only. Seems to happen when the shoot is created, but the seed is full and during shoot creation, the auto-scaling of the seed cluster kicks in and scales up.
The error can be resolved by manually removing the duplicate key pair and roles from the infrastructure.
Environment:

Gardener version: 0.26.2
Cloud provider or hardware configuration: AWS

@schrodit


Remove godfather image

This godfather image was cluttering the readme a bit, so I think maybe we should just keep the lego one for now.
Would bring it back if there was an good place to put it.
/cc @damncabbage

:chart_with_downwards_trend: rip


Ecto 0.13 Sandbox breaks Sqlite.Ecto integration tests

Using the ecto 0.13.0 hex release, I get the following test failures:
  6) Ecto.Integration.TypeTest: failure on setup_all callback, tests invalidated
     ** (RuntimeError) cannot begin test transaction with pool Ecto.Adapters.Poolboy.
     In order to use test transactions with Ecto SQL, you need to
     configure your repository to use Ecto.Adapters.SQL.Sandbox:

         pool: Ecto.Adapters.SQL.Sandbox

     stacktrace:
       (ecto) lib/ecto/adapters/sql.ex:321: Ecto.Adapters.SQL.test_transaction/3
       integration/sqlite/test_helper.exs:44: Ecto.Integration.Case.__ex_unit_setup_all_0/1
       integration/sqlite/test_helper.exs:40: Ecto.Integration.Case.__ex_unit__/2
       deps/ecto/integration_test/cases/type.exs:3: Ecto.Integration.TypeTest.__ex_unit__/2

I noticed that this new sandbox code seems to have been added around #721.  I updated my test_helper.exs file to use the Sandbox and added the Process.flag(:trap_exit, true) line to the end of the file as well.  When I do that, I get only the following error:
** (EXIT from #PID<0.48.0>) already started: #PID<0.167.0>

How do I get the integration tests to work again?
My branch with the above failure:  https://github.com/jazzyb/sqlite_ecto/tree/sandbox-failures

Yes, it is a bug in the sandbox pool, I will fix it. :)

Can you try Ecto master? If it works, I will release v0.13.1. :)

Different errors:
  7) Ecto.Integration.RepoTest: failure on setup_all callback, tests invalidated
     ** (exit) exited in: GenServer.call(Ecto.Integration.TestRepo.Pool, {:query, :begin, &Ecto.Integration.TestRepo.log/1, [timeout: 5000]}, 5000)
         ** (EXIT) no process
     stacktrace:
       (elixir) lib/gen_server.ex:356: GenServer.call/3
       (ecto) lib/ecto/adapters/sql.ex:335: Ecto.Adapters.SQL.test_transaction/4
       integration/sqlite/test_helper.exs:44: Ecto.Integration.Case.__ex_unit_setup_all_0/1
       integration/sqlite/test_helper.exs:40: Ecto.Integration.Case.__ex_unit__/2
       deps/ecto/integration_test/cases/repo.exs:1: Ecto.Integration.RepoTest.__ex_unit__/2

You need to give us more information on how to reproduce this issue, otherwise there is nothing we can do. Please read CONTRIBUTING.md file for more information about creating bug reports. Thanks!

Please ignore my message, I clicked the wrong button. :D

The root cause is this. We are getting {:error, {:sqlite_error, 'cannot start a transaction within a transaction'}} as reply and Ecto requires your query to return {:error, Exception.t}, so we can raise those errors.

Fixed in master. Also, you may want to integrate all of your tests together or set build_per_environment: false in your mix.exs so you don't compile everything twice all the time. :)

Ah.  Okay.  Thank you.  I'll look into that.

Will there be a 0.13.1 release?

Yes, are you ready? :)

Yep :)  I didn't change anything in Sqlite.Ecto, but it's working with master.

v0.13.1 is out. :)

Thanks for your help, José!


[FEATURE REQUEST] Graph visualizer

Description
Import the graph visualizer deveoped by Alessio into the main trunk of LiSA
Motivation
It allows to easily navigate and inspect the results.
Suggested implementation
https://github.com/Alessio-Campa/lisa/commits/master

Related to #128. Most of the work done in the fork will be rewritten.


Ci

When I build locally I am currently getting these test errors:
lightning-knd> failures:
lightning-knd> ---- integration::cli::test_cli_new_address stdout ----
lightning-knd> Error executing command: error sending request for url (https://<IP_ADDRESS>:2244/v1/newaddr): error trying to connect: tcp connect error: Connection refused (os error 111)
lightning-knd> thread 'integration::cli::test_cli_new_address' panicked at 'expected value at line 1 column 1', tests/integration/cli.rs:82:13
lightning-knd> ---- integration::cli::test_cli_disconnect_peer stdout ----
lightning-knd> Error executing command: error sending request for url (https://<IP_ADDRESS>:2244/v1/peer/disconnect): error trying to connect: tcp connect error: Connection refused (os error 111)
lightning-knd> thread 'integration::cli::test_cli_disconnect_peer' panicked at 'expected value at line 1 column 1', tests/integration/cli.rs:82:13
lightning-knd> note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
lightning-knd> ---- integration::cli::test_cli_get_info stdout ----
lightning-knd> Error executing command: error sending request for url (https://<IP_ADDRESS>:2244/v1/getinfo): error trying to connect: tcp connect error: Connection refused (os error 111)
lightning-knd> thread 'integration::cli::test_cli_get_info' panicked at 'expected value at line 1 column 1', tests/integration/cli.rs:82:13
lightning-knd> ---- integration::cli::test_cli_withdraw stdout ----
lightning-knd> Error executing command: error sending request for url (https://<IP_ADDRESS>:2244/v1/withdraw): error trying to connect: tcp connect error: Connection refused (os error 111)
lightning-knd> thread 'integration::cli::test_cli_withdraw' panicked at 'expected value at line 1 column 1', tests/integration/cli.rs:82:13
lightning-knd> ---- integration::cli::test_cli_get_balance stdout ----
lightning-knd> Error executing command: error sending request for url (https://<IP_ADDRESS>:2244/v1/getbalance): error trying to connect: tcp connect error: Connection refused (os error 111)
lightning-knd> thread 'integration::cli::test_cli_get_balance' panicked at 'expected value at line 1 column 1', tests/integration/cli.rs:82:13
lightning-knd> ---- integration::cli::test_cli_list_peers stdout ----
lightning-knd> Error executing command: error sending request for url (https://<IP_ADDRESS>:2244/v1/peer/listPeers): error trying to connect: tcp connect error: Connection refused (os error 111)
lightning-knd> thread 'integration::cli::test_cli_list_peers' panicked at 'expected value at line 1 column 1', tests/integration/cli.rs:82:13
lightning-knd> ---- integration::cli::test_cli_connect_peer stdout ----
lightning-knd> Error executing command: error sending request for url (https://<IP_ADDRESS>:2244/v1/peer/connect): error trying to connect: tcp connect error: Connection refused (os error 111)
lightning-knd> thread 'integration::cli::test_cli_connect_peer' panicked at 'expected value at line 1 column 1', tests/integration/cli.rs:82:13
lightning-knd> failures:
lightning-knd>     integration::cli::test_cli_connect_peer
lightning-knd>     integration::cli::test_cli_disconnect_peer
lightning-knd>     integration::cli::test_cli_get_balance
lightning-knd>     integration::cli::test_cli_get_info
lightning-knd>     integration::cli::test_cli_list_peers
lightning-knd>     integration::cli::test_cli_new_address
lightning-knd>     integration::cli::test_cli_withdraw
lightning-knd> test result: FAILED. 29 passed; 7 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.47s
lightning-knd> error: test failed, to rerun pass `--test main`

This is on a 64-Cores machines. Is this maybe related to concurrency?


Working with Literate?

Thanks for the awesome package!
How is one supposed to work with Literate? The link to the docs in the README is broken

Hello @gdalle, yeah I need to fix the docs + link..
try this please:
julia> using LiveServer, Documenter, Literate
julia> LiveServer.servedocs_literate_example()
julia> cd("servedocs_literate_example")
julia> servedocs(literate=joinpath("docs", "literate"))

this will create a folder servedocs_literate_example with a simple folder structure; if you modify scripts in docs/literate the changes will be reflected in your browser.
For now there's the assumption that Literate generates a md file next to the .jl script, if this bothers you, keep an eye on the ongoing discussion in #165 .

Thanks!!

(doc link fixed)


feat: debug log global registrations and logger overwrites

Currently, there's no visibility to global registration which is a relatively important operation.
For example, there's no way to know if a logger has been overwritten or not which might mean the successive log messages/errors from other parts of the API might not be appearing without the developer's knowledge.
To relieve that, a few log statements were added:

There can be only one global logger. If an existing one is overwritten, both of the loggers get a warn log line about the event.
Any time a global is (un)registered, debug log line is created with the type and version.

It relieves the immediate pain described in https://github.com/open-telemetry/opentelemetry-js-api/issues/56

Really like this idea. Can be frustrating for users when they think they are loading a diag logger but it gets overwritten. Maybe it would be even worth getting the stack when it is overwritten?
// use empty string as fallback in case stack can't be generated
const stack = new Error().stack ?? ""

oldLogger.warn(`A new DiagLogger was registered which overwrites this logger from ${stack}`);

Having a stack is a really good idea as well. I'll add that here for now to keep the focus of the PR.
Although... whenever I see variables in log messages I see a lost opportunity to structure the log entry:

Thought's on using structured logging everywhere instead? An API similar to pino perhaps:

log(object, string);
log(object | string);

What do you think of making it an option in the logger itself? logStackTrace: <LEVEL>/boolean.

@dyladan, @obecny. Reverted the changes to test structure. Everything should now be addressed.


[FEAT] UI Fix for Client's Home Page

Description:
The client's homepage requires a UI fix to improve its appearance and usability. The current UI needs to be updated based on the screenshot provided below. Contributors are welcome to enhance the design or fix any layout issues present.
Requirements:

Fix the UI issues on the client homepage to make it visually appealing and user-friendly.
Consider updating layout, font, color schemes, and overall page styling to enhance the user experience.

Steps to Reproduce:

Clone the project and set it up locally.
Navigate to the client's homepage: http://localhost/alimento/home.php
Use the following credentials for access if needed:

Email<EMAIL_ADDRESS>Password: Test@1234

Once logged in, review the homepage UI and take note of areas needing improvement.

Screenshot:

Files Involved:
The relevant files for the client’s homepage UI are located at:
alimento/home.php
Notes:

If required, contributors can add new restaurants by signing up at http://localhost/alimento/vendor/vendor_signup.php and using the vendor panel to add restaurants.
Alternatively, you can log in with the same credentials at http://localhost/alimento/vendor/vendor_login.php to add new restaurants.
Feel free to suggest and implement any further UI/UX improvements.

I like to work on this project, can u assign an issue to me

Hey @Web-Dev-Learner , Thank you for your interest in contributing to this project! Before you get started, could you please share the design you're aiming for? It would be great to see your vision before you begin working on it.
Looking forward to your ideas!

@Web-Dev-Learner, What's the progress ?

Hii @Vimall03 may I work on it?

Closing this issue as duplicate of #43


bug:  from "react-router-dom" in Navbar breaks everything

What version of daisyUI are you using?
v4.0.3
Which browsers are you seeing the problem on?
Firefox
Reproduction URL
https://stackblitz.com/edit/daisyui-react-vite-hlgbgp?file=src%2Fmain.tsx
Describe your issue
Placing a <Link ... > in the navbar, everything breaks.

This is not from daisyUI. As you can see even if you disable daisyUI, you still get that error.
daisyUI is a CSS library. <Link> is a React component from react-router
Ask your question here:
https://github.com/remix-run/react-router


How to install zstack with offline model

I want build a test enviroment , but i cant access the internet . Is there any way that i can install zstack with offline.
Thanks!!

Hi, if you can't access internet, you should setup intranet with yum
repository, and modify your system yum config files in /etc/yum.repo.d/*
and add local epel.repo there. ZStack installation will rely on some system
libs, which will be installed from yum repository.
And you should download zstack virtualrouter image and put it in private
http server, which could be accessed from your management node.
On Fri, Sep 11, 2015 at 10:22 AM, 251661412<EMAIL_ADDRESS>wrote:

I want build a test enviroment , but i cant access the internet . Is there
any way that i can install zstack with offline.
Thanks!!
—
Reply to this email directly or view it on GitHub
https://github.com/zstackorg/zstack/issues/124.

--
Yongkang You

Hi,  Yongkang You:
Appreciate for you reply , by the way,  where can i download the zstack virtualrouter image ?
Thank you very much !

Hi, Yongkang You:
Appreciate for your reply !

Hi，
0.9 VR image URL could be found in http://zstack.org/blog/v0.9-rc2.html
On Fri, Sep 11, 2015 at 1:33 PM, 251661412<EMAIL_ADDRESS>wrote:

Hi, Yongkang You:
Appreciate for you reply , by the way, where can i download the zstack
virtualrouter image ?
Thank you very much !
—
Reply to this email directly or view it on GitHub
https://github.com/zstackorg/zstack/issues/124#issuecomment-139457966.

--
Yongkang You

the 1.0 version has support for offline installation.


[Android] 0x05d-Testing-Data-Storage [SQLite Databases]

Use command
adb shell sqlite3 " + $path + $database + " .dump | grep #{plaintext} | wc -l

Another command :wink:
grep "SENSITIVE_INFORMATION" /data/data/PACKAGE_NAME/databases/*

returns nothing if no match is found
returns Binary file /data/data/PACKAGE_NAME/databases/DATABASE_FILE matches if a match is found

This also works without knowing the names of the databases in die databases directory.


how could I save audio to wavfile?

I want to use streamlit-webrtc to record wav and save to disk (like temp.wav'), but I have some problem, after I click stopbutton, audio_buffer becomeNone` instead of audio from microphone
my code
    webrtc_ctx = webrtc_streamer(
        key="sendonly-audio",
        mode=WebRtcMode.SENDONLY,
        audio_receiver_size=256,
        client_settings=WEBRTC_CLIENT_SETTINGS,
    )

    sound_window_len = 1  # 1ms

    audio_buffer = pydub.AudioSegment.silent(
        duration=sound_window_len
    )
    status_indicator = st.empty()

    while True:
        if webrtc_ctx.audio_receiver:
            try:
                audio_frames = webrtc_ctx.audio_receiver.get_frames(timeout=1)
            except queue.Empty:
                status_indicator.write("No frame arrived.")
                continue

            status_indicator.write("Running. Say something!")

            sound_chunk = pydub.AudioSegment.empty()
            for audio_frame in audio_frames:
                st.info('get audio frame')
                sound = pydub.AudioSegment(
                    data=audio_frame.to_ndarray().tobytes(),
                    sample_width=audio_frame.format.bytes,
                    frame_rate=audio_frame.sample_rate,
                    channels=len(audio_frame.layout.channels),
                )
                sound_chunk += sound

            if len(sound_chunk) > 0:
                audio_buffer += sound_chunk
        else:
            status_indicator.write("AudioReciver is not set. Abort.")
            break

    st.info("Writing wav to disk")
    audio_buffer.export('temp.wav', format='wav')

Hi, please try below.
The reason is that your app script is re-executed every time component state (e.g. playing or not) changes then audio_buffer object differs in the execution where the audio recording loop is running and the next execution where .export() is called.
This re-execution is Streamlit's fundamental execution model (See https://docs.streamlit.io/en/stable/main_concepts.html#app-model),
and it provides some ways to handle problems like this case, and the example below uses st.session_state to keep the same sound object over executions.
import queue

import pydub

import streamlit as st
from streamlit_webrtc import webrtc_streamer, WebRtcMode, ClientSettings

def main():
    webrtc_ctx = webrtc_streamer(
        key="sendonly-audio",
        mode=WebRtcMode.SENDONLY,
        audio_receiver_size=256,
        client_settings=ClientSettings(
            rtc_configuration={
                "iceServers": [{"urls": ["stun:stun.l.google.com:19302"]}]
            },
            media_stream_constraints={
                "audio": True,
            },
        ),
    )

    if "audio_buffer" not in st.session_state:
        st.session_state["audio_buffer"] = pydub.AudioSegment.empty()

    status_indicator = st.empty()

    while True:
        if webrtc_ctx.audio_receiver:
            try:
                audio_frames = webrtc_ctx.audio_receiver.get_frames(timeout=1)
            except queue.Empty:
                status_indicator.write("No frame arrived.")
                continue

            status_indicator.write("Running. Say something!")

            sound_chunk = pydub.AudioSegment.empty()
            for audio_frame in audio_frames:
                sound = pydub.AudioSegment(
                    data=audio_frame.to_ndarray().tobytes(),
                    sample_width=audio_frame.format.bytes,
                    frame_rate=audio_frame.sample_rate,
                    channels=len(audio_frame.layout.channels),
                )
                sound_chunk += sound

            if len(sound_chunk) > 0:
                st.session_state["audio_buffer"] += sound_chunk
        else:
            status_indicator.write("AudioReciver is not set. Abort.")
            break

    audio_buffer = st.session_state["audio_buffer"]

    if not webrtc_ctx.state.playing and len(audio_buffer) > 0:
        st.info("Writing wav to disk")
        audio_buffer.export("temp.wav", format="wav")

        # Reset
        st.session_state["audio_buffer"] = pydub.AudioSegment.empty()

if __name__ == "__main__":
    main()

If your purpose is only to record the sound to a file without any processing with pydub, recorder option is the simplest way:
from streamlit_webrtc import webrtc_streamer, WebRtcMode, ClientSettings
from aiortc.contrib.media import MediaRecorder

def main():
    def recorder_factory():
        return MediaRecorder("record.wav")

    webrtc_streamer(
        key="sendonly-audio",
        mode=WebRtcMode.SENDONLY,
        in_recorder_factory=recorder_factory,
        client_settings=ClientSettings(
            rtc_configuration={
                "iceServers": [{"urls": ["stun:stun.l.google.com:19302"]}]
            },
            media_stream_constraints={
                "audio": True,
                "video": False,
            },
        ),
    )

if __name__ == "__main__":
    main()

thx very much for your help, I will dive into webRTC and streamlit for details, thxs

just find out that recv docstring,
When running in async mode, only the latest frame is provided
and other frames are dropped which have arrived after
the previous recv() call and before the latest one.

So you should use recv_queued() or MediaRecorder instead of recv() to avoid missing audio frame, am I right?

If your purpose is only to record the sound to a file without any processing with pydub, recorder option is the simplest way:
from streamlit_webrtc import webrtc_streamer, WebRtcMode, ClientSettings
from aiortc.contrib.media import MediaRecorder

def main():
    def recorder_factory():
        return MediaRecorder("record.wav")

    webrtc_streamer(
        key="sendonly-audio",
        mode=WebRtcMode.SENDONLY,
        in_recorder_factory=recorder_factory,
        client_settings=ClientSettings(
            rtc_configuration={
                "iceServers": [{"urls": ["stun:stun.l.google.com:19302"]}]
            },
            media_stream_constraints={
                "audio": True,
                "video": False,
            },
        ),
    )

if __name__ == "__main__":
    main()

I use sample code as above, but it doesn't work as expected if you play it (seems it missing half of audio frame), it used to work before.

if you put wavfile in audition, you get ( by the way, record wav using webrtc_ctx.audio_receiver.get_frames(timeout=1) works fine

mm, while I'm not familiar with sound processing and cannot understand what the figures mean,
I wonder if setting some options to the MediaRecorder works.
MediaRecorder passes the options parameter to FFmpeg (via the underlying PyAV library). So, what if you set options like options={"sample_rate": ..., "channels": ...}?

set options doesn't work, but if I change mode to mode=WebRtcMode.SENDRECV instead of SENDONLY, MediaRecorder works fine

@whitphx @Jackiexiao I just did in your @Jackiexiao's gist. I played it right after I recorded it, and it seems the speed of the .wav file is two times faster. How could I solve this problem?

@hihunjin use this function to record https://gist.github.com/Jackiexiao/9c45d5ad9caf524471b5ed966c57b5b9#file-streamlit_recorder-py-L131

@Jackiexiao thank you for your investigation and problem-solving.
The fact that SENDRECV instead SENDONLY works fine is interesting hint. Thanks!

Hi! Im using the code that you post there  https://gist.github.com/Jackiexiao/9c45d5ad9caf524471b5ed966c57b5b9#file-streamlit_recorder-py-L131. When i click the START button it works just fine but when i stop it the audio file doesn't save it anywhere. I don't know if a need to change some parameter or something like that. Sorry im new at streamlit :)

Im getting this log
`2022-01-02 01:40:07.369 Connection(0) Check CandidatePair(('<IP_ADDRESS>', 52397) -> ('<IP_ADDRESS>', 54070)) State.FROZEN -> State.WAITING
2022-01-02 01:40:07.369 ICE connection state is checking
2022-01-02 01:40:07.369 Connection(0) Check CandidatePair(('<IP_ADDRESS>', 52397) -> ('<IP_ADDRESS>', 54070)) State.WAITING -> State.IN_PROGRESS
2022-01-02 01:40:21.245 ICE connection state is closed
2022-01-02 01:40:22.912 Exception in callback Transaction.__retry()
handle: <TimerHandle when=472955.485895703 Transaction.__retry()>
Traceback (most recent call last):
File "/usr/local/lib/python3.7/asyncio/selector_events.py", line 1002, in sendto
self._sock.sendto(data, addr)

AttributeError: 'NoneType' object has no attribute 'sendto'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/asyncio/events.py", line 88, in _run
self._context.run(self._callback, *self._args)

File "/home/appuser/venv/lib/python3.7/site-packages/aioice/stun.py", line 309, in __retry
self.__protocol.send_stun(self.__request, self.__addr)

File "/home/appuser/venv/lib/python3.7/site-packages/aioice/ice.py", line 242, in send_stun
self.transport.sendto(bytes(message), addr)

File "/usr/local/lib/python3.7/asyncio/selector_events.py", line 1011, in sendto
exc, 'Fatal write error on datagram transport')

File "/usr/local/lib/python3.7/asyncio/selector_events.py", line 677, in _fatal_error
self._loop.call_exception_handler({

AttributeError: 'NoneType' object has no attribute 'call_exception_handler'`

Im getting this log
`
2022-01-02 01:40:07.369 Connection(0) Check CandidatePair(('<IP_ADDRESS>', 52397) -> ('<IP_ADDRESS>', 54070)) State.FROZEN -> State.WAITING
2022-01-02 01:40:07.369 ICE connection state is checking
2022-01-02 01:40:07.369 Connection(0) Check CandidatePair(('<IP_ADDRESS>', 52397) -> ('<IP_ADDRESS>', 54070)) State.WAITING -> State.IN_PROGRESS
2022-01-02 01:40:21.245 ICE connection state is closed
2022-01-02 01:40:22.912 Exception in callback Transaction.__retry()
handle: <TimerHandle when=472955.485895703 Transaction.__retry()>
Traceback (most recent call last):
File "/usr/local/lib/python3.7/asyncio/selector_events.py", line 1002, in sendto
self._sock.sendto(data, addr)

AttributeError: 'NoneType' object has no attribute 'sendto'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/asyncio/events.py", line 88, in _run
self._context.run(self._callback, *self._args)

File "/home/appuser/venv/lib/python3.7/site-packages/aioice/stun.py", line 309, in __retry
self.__protocol.send_stun(self.__request, self.__addr)

File "/home/appuser/venv/lib/python3.7/site-packages/aioice/ice.py", line 242, in send_stun
self.transport.sendto(bytes(message), addr)

File "/usr/local/lib/python3.7/asyncio/selector_events.py", line 1011, in sendto
exc, 'Fatal write error on datagram transport')

File "/usr/local/lib/python3.7/asyncio/selector_events.py", line 677, in _fatal_error
self._loop.call_exception_handler({

AttributeError: 'NoneType' object has no attribute 'call_exception_handler'
`

Actually im getting this log https://github.com/whitphx/streamlit-webrtc/issues/552
Any idea on how to solve it :/?

Hi @whitphx and @Jackiexiao, this code is very helpful. Thanks a lot!
I wonder how to, by default, turn off the speaker while recording. The goal is to prevent audio feedback when not wearing headphones.

You can control the HTML attributes of the <audio /> element through the audio_html_attrs of webrtc_streamer() component as below.
    webrtc_ctx: WebRtcStreamerContext = webrtc_streamer(
        key="sendonly-audio",
        mode=WebRtcMode.SENDRECV,
        in_recorder_factory=recorder_factory,
        media_stream_constraints=MEDIA_STREAM_CONSTRAINTS,
        audio_html_attrs={"muted": True} # THIS!
    )

When you set only muted=true, the audio component is completely hidden. So if you want to show the audio control and mute the audio at the same time, set controls=true too as below.
        audio_html_attrs={"muted": True, "controls": True}

You can find other available attributes: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/audio


Outdated libraries still in plugins, slowing down startup  time

Hi!
There are dozen of old/outdated libraries loaded at DBeaver startup, that are not needed and that slow down startup time.
After disabling them, DBeaver starts waays quicker that before.
List so far:

thanks for the bug report

It seems to be Linux-specific behavior.
Workaround: delete ~/.eclipse folder. It will be recreated on DBeaver startup.

Is this still actually against the recent version?

There have been no updates for a long time. if issue is still actual - feel free to reopen it


Update index.yml (Typo "Azure OpenAI service"→"～ Service")

https://learn.microsoft.com/en-us/azure/cognitive-services/
https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/cognitive-services/index.yml
#PingMSFTDocs

@PatrickFarley

Can you review this PR?
IMPORTANT: When this content is ready to merge, you must add #sign-off in a comment or the approval may get overlooked.

#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

#sign-off

this text has been changed by another PR
#please-close


No re-sort after mass name change

What system are you running Yazi on?
Linux Wayland
What terminal are you running Yazi in?
kitty 32.3
yazi --debug output

##### ↓↓↓ Paste the output here: ↓↓↓ #####

Did you try the latest nightly build to see if the problem got fixed?
Yes, and I updated the debug information above (yazi --debug) to the nightly that I tried
Describe the bug
When I select several filenames and change them. I then save and return to yazi. Unfortunately, the display does not re-sort to reflect new file order. I need to leave the directory and enter again to get a re-sort.
Minimal reproducer
Select several files and select "r" to change the names. Change the names in a way that would change the sort order. Save and return.
Anything else?
I suggest forcing a re-sort after a mass name change. Or at least providing a command to refresh/reload a directory.

I was wondering, we can just have "refresh" in the keybinding in the keybinding for re-sort. I tested and it works just fine.
    { on = "r", run = ["rename --cursor=before_ext", "refresh"], desc = "Rename selected file(s)" },

This won't require any generalization of refreshing for every operation in src code. If the person wants, they can just adjust the keybindings.

The issue here i feel like is, after renaming, adding content to an empty file (which shows "Empty File") just after exiting EDITOR, etc. are simply that it is not reloaded, it takes on the previous value. Once you move away and see again, it recalculates the order/preview content, etc. and we see it fine.

Please try https://github.com/sxyazi/yazi/pull/1691


Add missing BSD license file

Upsteam license can be found here.
You must respect the following condition:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

not sure why UUID is in the project. Looks like it's getting set for a request ID but that doesn't seem to get used anywhere. Merging for now, but opened #3515 to track if we should remove UUID completely.


Accessing IrcBot from a script

Hi,
more of a question really, but how would I go about if I want to write a custom script and want to access the bot variable defined in the irc.coffee? Is that possible?

@nmcclain, you are a hero.


Fix Concurrency issues with StateChange hooks and add/removing channels

There are still a few ConcurrentModificationExceptions occurring. These are caused by clients adding a callback hook inside of a callback hook or modifying the channels while handling an error, etc.
the solution is to make lists as read-only and then whenever adding or removing hooks and channels, use only concurrently safe operations by converting the read-only list to a new mutable list, adding the value, and then returning the new list.
This allows for looping over an existing list and adding a value to that list at the same time without modifying the underlying list while it's being looped over.

@AdamGrzybkowski

Codecov Report

Merging #70 into master will increase coverage by 0.06%.
The diff coverage is 95.83%.

@@             Coverage Diff              @@
##             master      #70      +/-   ##
============================================
+ Coverage     91.36%   91.43%   +0.06%
  Complexity      178      178
============================================
  Files             9        9
  Lines           533      537       +4
  Branches         72       71       -1
============================================
+ Hits            487      491       +4
+ Misses           13       12       -1
- Partials         33       34       +1

Impacted Files
Coverage Δ
Complexity Δ

src/main/kotlin/org/phoenixframework/Socket.kt
92.35% <100%> (-0.33%)
71 <5> (ø)

src/main/kotlin/org/phoenixframework/Push.kt
83.33% <50%> (-0.67%)
25 <0> (ø)

.../main/kotlin/org/phoenixframework/DispatchQueue.kt
66.66% <0%> (+11.11%)
0% <0%> (ø)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2d5c52f...ec430fb. Read the comment docs.

@dsrees I don't think this extension copyAndAdd is needed

converting the read-only list to a new mutable list, adding the value, and then returning the new list

this could be achieved with kotlin plus operator. I've left a comment in the commit to master.


ImageJ2Options: add log level selection

Add selection of the SciJava log level to the ImageJ2 options dialog.
Known issues:

[x] Doesn't set the drop down's default value to the currently active log level. Is there a possibility to access the generated widget from an initialize() method?
Could LogService.WARN, etc, be implemented as an enum?

Is there a possibility to access the generated widget from an initialize() method?

Yes, although you shouldn't need to access the widget. Just give the field an initializer method in its @Parameter declaration, then change the value of that field to match what the LogService reports.

Could LogService.WARN, etc, be implemented as an enum?

Not without breaking backwards API compatibility in SciJava Common. I guess we could do this for SJC3. But what would be the advantage?

Just give the field an initializer method in its @Parameter declaration, then change the value of that field to match what the LogService reports.

That does not set the selection of the combo box. That's why I was asking about the widget. Should the selection of the combo box be set according to the value of the annotated field?

when you call LogService.getLevel() you'll have to convert the int back into the level using case logic, which is unfortunate...

Exactly...

Should the selection of the combo box be set according to the value of the annotated field?

Yes, that is what is supposed to happen. The initial value of the field should determine the selected item of the combo box, according to the equals method.

The initial value of the field should determine the initially selected item of the combo box.

That does not work for me.

I just tried to debug into this and could not reproduce. It then occurred to me what the problem is: the parameter value was persisted by the SaveInputsPreprocessor and is being restored the same. Please add persist = false to the logLevel parameter, and I bet you the problem will disappear for you.

We could add a hack that the SaveInputsPreprocessor ignores inputs that have an initializer callback. I'm not sure if that would break any code anywhere, but it would prevent others from stumbling over this in the future (I think I have stumbled over it at least twice before, in my dim memory).

and I bet you the problem will disappear for you.

It did :clap: Thanks, @ctrueden!

We could add a hack that the SaveInputsPreprocessor ignores inputs that have an initializer callback.

I would appreciate that hack since debugging such issues repeatedly is a pain (I thought the issue was somewhere with the DefaultValuePreprocessor because ERROR is the first choice and might be the fallback). This doesn't seem to be an urgent issue, so I could give it a shot next week?

I could give it a shot next week?

Sure. Here is where you would add the following code:
// NB: Do not persist values which are computed via an initializer.
if (item.getInitializer() != null) return;

Again: this breaks the (theoretical) use case where a command uses the initializer to specify the initial initial value (i.e.: the initial value the first time the command ever runs), but then wants to prefer the last-chosen value of the user for the initial value upon subsequent executions. I am not sure whether that is a big deal. But without this hack, you can have the behavior work either way, by setting persist=false or not.

Sure. Here is where you would add the following code

Feel free to do it yourself the next time (if it is just one line). I think writing that comment has taken you longer than actually integrating that line :smile:

Feel free to do it yourself the next time (if it is just one line). I think writing that comment has taken you longer than actually integrating that line :smile:

Nope, I didn't test it. And untested code is nonworking code. Besides: I am trying to train you. Training always takes longer than doing it yourself. But hopefully not globally over all time.

@stelfrich Can you take a couple of minutes to clean up these commits as mentioned in my earlier comments? Then we can merge it!

Thanks for the comments @ctrueden. I took the liberty to reformat both @Parameter annotations after realizing that they first one was not formatted according to the code style (or was it?). I have reformatted with Eclipse Mars and the current version of the ImageJ preferences.

I finally found time to rewrite/clean these commits. Merging. Thanks again @stelfrich!

I finally found time to rewrite/clean these commits. Merging. Thanks again @stelfrich!


A C interface to lager [JIRA: RIAK-2537]

@Vagabond I heard you were at some point contemplating a C layer to route messages from things like leveldb into a lager sink. I was wondering if you had any suggestions about how that should be constructed (port driver/NIF/something else?)
I have seen plenty of libraries that show how to send messages out of Erlang into C land, but almost none showing how to send messages from C land into Erlang.  Thanks for any pointers or suggestions you may have.

I don't remember this. I suspect linked-in-drivers and NIFs would have a way to send messages to an erlang process, but I haven't poked around in those APIs recently.


Compatibility upgrade and refactor

refactor into ES classes.

some promises and fat arrows, too.

result is copied to clipboard.
sample demo traces are available to try out.
fix the canary breaking issue: https://github.com/arthurvr/timeline-url/issues/9#issue-93343568
uses nice https://frontend.chrome-dev.tools URLs if CORS is detected https://github.com/arthurvr/timeline-url/issues/9#issuecomment-120666064
UI polish and stuff.

fixes #9

Lovely work man. :heart: :star: :100:

Gonna push some super tiny code style cleanup myself. Super neat work!

Gonna push some super tiny code style cleanup myself. Super neat work!

awesome. please do.
i got excited and did some fat arrow stuff.
however it's brand new so i think it breaks anyone on chrome & beta. :/
might have to bring that back. haha.
but yeah please feel free to hack up that stuff any which way you want.


Add TopologicalInventory API

Copy over the API from insights-api-topological_inventory
cc @bdunne

Closing in favor of using an API engine https://github.com/agrare/insights-topological_inventory-api


[JOSS REVIEW] Improve Summary

According to the Review Criteria the paper should give A summary describing the high-level functionality and purpose of the software for a diverse, non-specialist audience. My opinion is that the current summary is a bit too short, and require a descent level of expertise. I think it could be improved by showing some examples of where relevant applications arises and and what a phase field model is. Are there some relevant equations? How does this compare to "regular" solid mechanics.
This issue is related to the review here: https://github.com/openjournals/joss-reviews/issues/7307

Thank you for the constructive feedback. I’ll expand the summary to make it more accessible, including examples of applications for phase field models and key equations to clarify its functionality. I plan to complete this by the end of the week.

Expanded summary with examples of phase-field model applications and key equations for clarity. Ready for review
https://github.com/CastillonMiguel/phasefieldx/commit/487b67af6c257d0bf032dd8f01234dbc184ac4e1


Build pipeline

The configuration builds the application and produces builds for both Debug and Release configurations.

Testing Git Pull feature.