Noise is being added to generated speech in Python E2E flow (TFLite Models)

Description
I am currently working on a project that is built in Unity where I am modulating voices (e.g. source speech → voice modulator → target speech (elf)). I currently have an E2E flow with the TFLite models, but there is a decent amount of noise being added to the speech generation. It sounds almost like a clipping noise.  I'm currently using the TFLite models from the repo and I have split the quantizer into a QuantizerEncoder & QuantizerDecoder. I'm not sure if a better solution is to attempt to convert Lyra into a DLL and just run that in Unity vs the models, but this is what I have so far.
E2E Flow

Load a wav file via librosa
Pad the data to meet data_length % 320 == 0
Feed the data through the 4 models: Encoder, QuantizerEncoder, QuantizerDecoder, & Decoder
Store the waveform data as I go as:

One singular array of data
A series of audio clips

Save the singular array of waveform data as a wav file
Play back the file

Code
!wget -q https://huggingface.co/rocca/lyra-v2-soundstream/resolve/main/tflite/1.3.0/soundstream_encoder.tflite
!wget -q https://huggingface.co/rocca/lyra-v2-soundstream/resolve/main/tflite/1.3.0/lyragan.tflite
!wget -q https://huggingface.co/rocca/lyra-v2-soundstream/resolve/main/tflite/1.3.0/quantizer_encoder.tflite
!wget -q https://huggingface.co/rocca/lyra-v2-soundstream/resolve/main/tflite/1.3.0/quantizer_decoder.tflite

import tensorflow as tf
import numpy as np

import librosa

def getAudioData(audio_file, verbose=False):
    data, sr = librosa.load(audio_file, sr=None)

    if verbose:
        print(len(y))

    batch_size = 320
    padding_length = batch_size - (len(data) % batch_size)
    padded_data = np.pad(data, (0, padding_length), mode='constant', constant_values=0)

    return padded_data, sr

# Encoder:
def runEncoderInference(input_data, verbose=False):
    interpreter = tf.lite.Interpreter(model_path="encoder.tflite")
    interpreter.allocate_tensors()

    input_details = interpreter.get_input_details()
    output_details = interpreter.get_output_details()

    # Extract the first 320 samples
    input_data = np.array(input_data, dtype=input_details[0]['dtype'])
    input_data = np.reshape(input_data, (1, 320))

    interpreter.set_tensor(input_details[0]['index'], input_data)

    interpreter.invoke()
    output_data = interpreter.get_tensor(output_details[0]['index'])

    if verbose:
        print(output_data)

    return output_data

# Quantizer Encoder:
def runQuantizerEncoderInference(input_data2, verbose=False):
    interpreter = tf.lite.Interpreter(model_path="quantizer_encoder.tflite")
    interpreter.allocate_tensors()

    input_details = interpreter.get_input_details()
    output_details = interpreter.get_output_details()

    input_data1 = np.array(46, dtype=np.int32)
    interpreter.set_tensor(input_details[0]['index'], input_data1)

    # input_data2 = np.ones(input_details[1]['shape'], dtype=input_details[1]['dtype'])
    interpreter.set_tensor(input_details[1]['index'], input_data2)

    interpreter.invoke()
    output_data = interpreter.get_tensor(output_details[0]['index'])

    if verbose:
        print(output_data)

    return output_data

# Quantizer Decoder:
def runQuantizerDecoderInference(input_data, verbose=False):
    interpreter = tf.lite.Interpreter(model_path="quantizer_decoder.tflite")
    interpreter.allocate_tensors()

    input_details = interpreter.get_input_details()
    output_details = interpreter.get_output_details()

    # input_data = np.ones(input_details[0]['shape'], dtype=input_details[0]['dtype'])
    interpreter.set_tensor(input_details[0]['index'], input_data)

    interpreter.invoke()
    output_data = interpreter.get_tensor(output_details[0]['index'])

    if verbose:
        print(output_data)

    return output_data

# Decoder:
def runDecoderInference(input_data, verbose=False):
    interpreter = tf.lite.Interpreter(model_path="lyragan.tflite")
    interpreter.allocate_tensors()

    input_details = interpreter.get_input_details()
    output_details = interpreter.get_output_details()

    # input_data = np.ones(input_details[0]['shape'], dtype=input_details[0]['dtype'])
    interpreter.set_tensor(input_details[0]['index'], input_data)

    interpreter.invoke()
    output_data = interpreter.get_tensor(output_details[0]['index'])

    if verbose:
        print(output_data)

    return output_data

audio_file = "<wavfile_path>.wav"
data, sr = getAudioData(audio_file)

# Check length and pad with zeroes so that the length % 320 = 0
import numpy as np

batch_size = 320
num_batches = len(data) // batch_size
waveform_data = None
audio_clips = None

for i in range(num_batches):
    start_idx = i * batch_size
    end_idx = (i + 1) * batch_size
    batch_data = data[start_idx:end_idx]

    enc_output = runEncoderInference(batch_data)
    qe_output = runQuantizerEncoderInference(enc_output)
    qd_output = runQuantizerDecoderInference(qe_output)
    dec_output = runDecoderInference(qd_output)

    if i == 0:
        waveform_data = dec_output[0] # Concatenates all waveform data
        audio_clips = dec_output # Stores waveform data as clips
    else:
        waveform_data = np.concatenate((waveform_data, dec_output[0]))
        audio_clips = np.concatenate((audio_clips, dec_output))

import torchaudio
import torch

audio_tensor = torch.tensor([waveform_data])
output_file = "<your_output_path>.wav"
torchaudio.save(output_file, audio_tensor, sr)

Questions

Is the better solution to create a DLL and run that in Unity?
Do the models encompass all of the pre & post-processing needed to provide a clean output signal that the cpp implementation provides (e.g. Integration Test Example)
Have I made an error in my implementation? I haven't been able to find a python implementation yet that runs the data and tests, so this is what I've come up with so far.
Is the noise possibly due to the fact that I am concatenating all of the data to test vs playing each clip back iteratively? I attempted to playback the output iteratively as plain waveform data in a Jupyter Notebook vs stored wavfiles, but no sound was produced.

Resources
Sound samples.zip

Same happens to me... Did you solve it?

The output of the data had a prefix that was being added with values that were really high or low. I don't remember the exact cause of the issue, but I do remember the fix was managing how we were staging and creating the DLL. One of the methods we added to the DLL was incorrectly prefixing those out of range data points to that data.
Hope this helps!


Feat omniparse

Features

config新增omniparse配置
基于omniparse的API封装成SDK
RAG在解析pdf文件时指定使用omniparse，而不是llamaindex默认的PyPDF

Feature Docs

Influence

Result

Other

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
Attention: Patch coverage is 48.90110% with 93 lines in your changes missing coverage. Please review.

Project coverage is 30.82%. Comparing base (a369912) to head (340e148).
Report is 2 commits behind head on main.

Files
Patch %
Lines

metagpt/utils/omniparse_client.py
34.61%
51 Missing :warning:

metagpt/rag/parsers/omniparse.py
42.85%
32 Missing :warning:

metagpt/rag/engines/simple.py
41.66%
7 Missing :warning:

metagpt/rag/schema.py
89.28%
3 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1408      +/-   ##
==========================================
+ Coverage   30.65%   30.82%   +0.16%
==========================================
  Files         320      324       +4
  Lines       19423    19603     +180
==========================================
+ Hits         5954     6042      +88
- Misses      13469    13561      +92

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

omniparse已有SDK，可以考虑直接使用

The SDK written by omniparse is not yet complete, and many interface functions will directly report errors without being used.

lgtm


aws-c-sdkutils: fix cmake options

The -DCMAKE_MODULE_PATH will only pick up the latter one.
The $D/usr is actually expanded to '/usr', which is not correct for CMAKE_PREFIX_PATH, as CMAKE_PREFIX_PATH are prefixes used by find_xxx() commands. Specifying a host path is not correct.
To make cmake to find aws-c-common-config.cmake, pass ${STAGING_LIBDIR} to CMAKE_PREFIX_PATH so that find_package can correctly locate the file.
Fixes https://github.com/aws4embeddedlinux/meta-aws/issues/5032

/run

Thanks for these PRs and the issues. I will take a look shortly.


cmd/compile: go1.7beta2 performance regressions

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go 1.6.2, go1.7beta2

What operating system and processor architecture are you using (go env)?

linux/arm64, linux/arm

What did you do?

run the standard go1 benchmarks

What did you expect to see?

little to no performance regression

What did you see instead?

see:
https://gist.github.com/ajstarks/fa05919377de73a552870ce9e45e4844  for linux/arm64 (pine64)
https://gist.github.com/ajstarks/c937a1a80e76b4692cb52c1d57a4494d for linux/arm (CHIP)
regressions occur on GobEncode, GobDecode, JSONEncode, and JSONDecode

I'm away from my arm(64) machines for a couple of weeks. Help bisecting would be lovely.

benchviz for Pine64
pine64.pdf

CC @randall77

@quentinmit and I are looking at this. We found that it also slows down on linux/386, and linux/amd64 with -ssa=0, with GobDecode, JSONDecode and Template being the most significant. Gzip is getting faster, though. So, probably it is not the compiler?

Could be the runtime, or it could be front-end compiler changes. I think there were near zero 386 or arm-specific fixes this cycle, so it might be worth comparing the generated code to rule out the compiler (perhaps after some profiling to pinpoint the change).

Al those packages use reflection. Could it be related to the type reworking?

Reflection is definitely part of the puzzle. Here's the state of our investigation so far:

Template: ~22% performance hit on arm32, arm64, and 386. Largest hit due to http://golang.org/cl/20968. @rlh is trying to use VTune to pin down the slow code.
Revcomp: ~30% performance hit on 386 only. Entirely due to http://golang.org/cl/20901. @dr2chase is looking at this.
GobDecode: ~10% performance hit on arm32, arm64, and 386. Very noisy, no obvious culprit.
JsonDecode: ~30% performance hit on arm32, arm64, ~15% on 386. Less noisy, still no obvious culprit.
Gzip: ~20% performance hit on arm32, arm64. Still need to bisect (not as easy as bisecting on 386).

The slow version contains a call to this routine that the faster version
does not contain. This accounts for aroutne 3% of the total time. It is the
only routine that stood out in the top 20 routines.
inst retired| cpu_clk unhalted
reflect.(*name).name        3.3% 2.9% type.go 0x8166a40 go1-template-bad
reflect.(*name).name
On Tue, Jun 21, 2016 at 2:50 PM, Quentin Smith<EMAIL_ADDRESS>wrote:

Reflection is definitely part of the puzzle. Here's the state of our
investigation so far:

Template: ~22% performance hit on arm32, arm64, and 386. Largest hit
due to http://golang.org/cl/20968. @rlh https://github.com/rlh is
trying to use VTune to pin down the slow code.
Revcomp: ~30% performance hit on 386 only. Entirely due to
http://golang.org/cl/20901. @dr2chase https://github.com/dr2chase is
looking at this.
GobDecode: ~10% performance hit on arm32, arm64, and 386. Very
noisy, no obvious culprit.
JsonDecode: ~30% performance hit on arm32, arm64, ~15% on 386. Less
noisy, still no obvious culprit.
Gzip: ~20% performance hit on arm32, arm64. Still need to bisect
(not as easy as bisecting on 386).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/16117#issuecomment-227535551, or mute
the thread
https://github.com/notifications/unsubscribe/AA7Wn6TQ1oaw8NNBTuIsKaDeSjMWq_SXks5qODKDgaJpZM4I5NAI
.

CC @crawshaw

@RLH Is it easy for you to find the significant callers of reflect.(*name).name?
It's possible that the significant callers are from code that is looking for a specific name, which we could probably speed up somewhat by doing things like checking the length before building the string we return.

I have an existence proof that code alignment can cost you 10% on Revcomp.

@ianlancetaylor If I'm wielding pprof correctly, the callers are:
                                             1.64s 84.97% |   reflect.(*structType).FieldByName /usr/local/google/home/quentin/go-386/src/reflect/type.go
                                             0.29s 15.03% |   reflect.(*structType).Field /usr/local/google/home/quentin/go-386/src/reflect/type.go
     1.93s  2.05%  2.05%      1.93s  2.05%                | reflect.(*name).name /usr/local/google/home/quentin/go-386/src/reflect/type.go

Like @rlh, I only see 3% attributed to this function, though, not 10-20%.

Apologies for the kibitzing, but if you rearrange the name method so that it gets inlined, does that help? (Glancing at the code on my phone, it appears that this discussion through data could be simplified to possibly achieve this.) Just because only 3% is spent in the function doesn't mean that's the only cost—there's the call itself and the impact on optimization that comes from having a call in the middle of an otherwise basic block.

I fooled around with name.name as called by FieldByName a bit but only got about a 1% improvement on 386.

I don't think it's worth recovering the performance of FieldByName in BenchmarkTemplate. It has always been a linear scan through field names, so it has never been as fast as it could be.
We could either build a map in the reflect package to make FieldByName fast, or better we could modify the template package so that, like encoders and decoders, on first encounter with a type it builds a contraption for reading type data. That way FieldByName is only called once per type instead of once per template execution.

Using the simple cache in https://golang.org/cl/24322 I see about a 10% speedup in the go1 template test.
Whether this is worth doing for 1.7 I don't know.

For Gzip on arm, commit 53efe1e (http://golang.org/cl/20929) causes 20% slow-down on arm64, and 5% slow-down on arm32.

On arm64, JSONDecode is slowed down 6% by CL http://golang.org/cl/19694, and slowed down 11% by http://golang.org/cl/19695. @quentinmit confirmed that they also slow down JSONDecode on 386.

CL https://golang.org/cl/24452 is also for this issue.
With it and the above, the JSON decoding performance hit on 386 has gone from ~15% to ~4%.:
name             old time/op    new time/op    delta
CodeDecoder-8      48.4ms ± 0%    50.3ms ± 0%  +3.89%  (p=0.000 n=10+10)

name             old speed      new speed      delta
CodeDecoder-8    40.1MB/s ± 0%  38.6MB/s ± 0%  -3.75%  (p=0.000 n=10+10)

(On AMD64 with SSA, decoding is 9% faster than it was in 1.6.)
I don't see any other easy fixes, but I'll spend another day looking. @cherrymui could you please compare JSON decoding on arm64 between 1.6 and tip again?

With the current tip (33fa855), JSONDecode has been improved from ~30% slowdown to ~11%.
JSONDecode-8                190ms ± 0%     212ms ± 1%  +11.53%   (p=0.000 n=8+10)

JSONDecode-8             10.2MB/s ± 0%   9.1MB/s ± 1%  -10.34%   (p=0.000 n=8+10)

It's that right that, in Go1.7, quite some reflect performance is traded for more compact compiled binary size?
Especially Field() and FieldByName(), which call reflect.(*name).name, and something use Name().
Because of better codegen, the amd64 ssa backend can cover that in other place, but not for other archs.
Is there any plan to gain those performance back in later release?

@noblehng, in later releases, all architectures will use SSA. That's a major theme for Go 1.8.

@bradfitz, I'm looking at this CL https://go-review.googlesource.com/#/c/20968/.
To get a method name or struct field name, it is just a struct field access in Go1.6, but basically needs to unmarshal from a binary encoding in Go1.7. Sure adding a global cache will help, but a struct field access will still be faster even for SSA.
I'm not familiar with the runtime, but I'm thinking something like whether using the compact form for compiled binary but a pre-unmarshalled form for runtime reflection is doable in a later release.

@noblehng, yes, that's what this bug is tracking (the across-the-board performance loss, largely from reflect changes). @crawshaw is working on it. I was merely pointing out that more SSA is coming, which is basically unrelated (except that it covered up the loss on amd64, as you noted).

I can take a look at the gzip slowdown for arm and 386. There was much optimization work done on gzip, but primarily targeted towards amd64, some of which completely neglected other architecture or even consciously chose to favor amd64 at the expense of other architectures.
If there's nothing easily actionable, we can punt of the optimizations until Go1.8.

Sameer pointed this bug out to me. Regressions on microbenchmarks may be very misleading since they are just busy loops around tiny amounts of code. More interesting would be to run the x/bench subrepo benchmarks on these systems and see if there is anything significant there. If not, waiting for SSA on these other systems seems like a decent plan. I haven't looked at any of the CLs, but please remember to make very small, very high confidence changes. Thanks.

I compared the JSON target in x/benchmarks with JSONDecode from test/bench/go1 and got similar results: a slowdown from 1.6 to tip on 386, and https://golang.org/cl/24521 recovers it.
I'm just about ready to call gob and json done. If @dsnet has anything for gzip, great, but I think everything else can wait for 1.8.

For json benchmark in golang.org/x/benchmarks on ARM64,
go1.6.2
GOPERF-METRIC:allocated=7871382
GOPERF-METRIC:allocs=105419
GOPERF-METRIC:cputime=293700000
GOPERF-METRIC:gc-pause-one=671156
GOPERF-METRIC:gc-pause-total=151010
GOPERF-METRIC:rss=183361536
GOPERF-METRIC:sys-gc=5376000
GOPERF-METRIC:sys-heap=168591360
GOPERF-METRIC:sys-other=7205176
GOPERF-METRIC:sys-stack=753664
GOPERF-METRIC:sys-total=181926200
GOPERF-METRIC:time=36993340
GOPERF-METRIC:virtual-mem=165421056

go1.7beta2
GOPERF-METRIC:allocated=7871557
GOPERF-METRIC:allocs=105417
GOPERF-METRIC:cputime=355650000
GOPERF-METRIC:gc-pause-one=539134
GOPERF-METRIC:gc-pause-total=121305
GOPERF-METRIC:rss=174964736
GOPERF-METRIC:sys-gc=6182912
GOPERF-METRIC:sys-heap=162234368
GOPERF-METRIC:sys-other=7548456
GOPERF-METRIC:sys-stack=819200
GOPERF-METRIC:sys-total=176784936
GOPERF-METRIC:time=44925700
GOPERF-METRIC:virtual-mem=162672640

tip (a2a4db7)
GOPERF-METRIC:allocated=7870908
GOPERF-METRIC:allocs=105404
GOPERF-METRIC:cputime=280480000
GOPERF-METRIC:gc-pause-one=567936
GOPERF-METRIC:gc-pause-total=122674
GOPERF-METRIC:rss=183672832
GOPERF-METRIC:sys-gc=6170624
GOPERF-METRIC:sys-heap=169902080
GOPERF-METRIC:sys-other=7459128
GOPERF-METRIC:sys-stack=819200
GOPERF-METRIC:sys-total=184351032
GOPERF-METRIC:time=35300696
GOPERF-METRIC:virtual-mem=150028288

Looking at time metric (is it right?), it seems go1.7beta2 is ~20% slower than go1.6.2 but tip gets it back.

With GOARCH=386 on an otherwise amd64 host, I'm also seeing a 35% slowdown in BinaryTree17 that looks like it happened when we merged dev.garbage: https://rawgit.com/aclements/74e66853d989adce240fadd62c554d0b/raw/491a81c5874e4f78d01133666a4ae916a166ceef/go1-1.7-386.svg (note that this isn't a complete run yet; it's only a sampling of the commits).
@RLH, could you look at the BinaryTree17 profiles before (d8d33514) and after (56b54912) the merge with GOARCH=386 and GOHOSTARCH=386?

What gzip benchmark is being run? When I run the compress/flate benchmarks on the architectures in question, I actually see improvement (as I would expect) and not regressions. I'm using the builders to run the tests.
 <= linux-arm64-buildlet

benchmark                              old MB/s     new MB/s     speedup
BenchmarkDecodeDigitsSpeed1e4-8        21.21        24.35        1.15x
BenchmarkDecodeDigitsSpeed1e5-8        22.18        26.06        1.17x
BenchmarkDecodeDigitsSpeed1e6-8        22.25        26.01        1.17x
BenchmarkDecodeDigitsDefault1e4-8      21.99        24.39        1.11x
BenchmarkDecodeDigitsDefault1e5-8      26.28        27.16        1.03x
BenchmarkDecodeDigitsDefault1e6-8      26.90        27.50        1.02x
BenchmarkDecodeDigitsCompress1e4-8     21.97        24.48        1.11x
BenchmarkDecodeDigitsCompress1e5-8     26.20        27.15        1.04x
BenchmarkDecodeDigitsCompress1e6-8     26.90        27.51        1.02x
BenchmarkDecodeTwainSpeed1e4-8         22.37        24.38        1.09x
BenchmarkDecodeTwainSpeed1e5-8         25.77        27.79        1.08x
BenchmarkDecodeTwainSpeed1e6-8         26.22        27.94        1.07x
BenchmarkDecodeTwainDefault1e4-8       24.05        25.91        1.08x
BenchmarkDecodeTwainDefault1e5-8       30.73        32.86        1.07x
BenchmarkDecodeTwainDefault1e6-8       31.73        33.98        1.07x
BenchmarkDecodeTwainCompress1e4-8      24.04        25.79        1.07x
BenchmarkDecodeTwainCompress1e5-8      30.83        32.86        1.07x
BenchmarkDecodeTwainCompress1e6-8      31.92        33.98        1.06x
BenchmarkEncodeDigitsSpeed1e4-8        7.46         12.11        1.62x
BenchmarkEncodeDigitsSpeed1e5-8        9.67         14.73        1.52x
BenchmarkEncodeDigitsSpeed1e6-8        9.90         15.62        1.58x
BenchmarkEncodeDigitsDefault1e4-8      5.67         7.11         1.25x
BenchmarkEncodeDigitsDefault1e5-8      2.46         4.96         2.02x
BenchmarkEncodeDigitsDefault1e6-8      2.13         4.66         2.19x
BenchmarkEncodeDigitsCompress1e4-8     5.66         7.14         1.26x
BenchmarkEncodeDigitsCompress1e5-8     2.46         4.98         2.02x
BenchmarkEncodeDigitsCompress1e6-8     2.15         4.66         2.17x
BenchmarkEncodeTwainSpeed1e4-8         7.35         11.99        1.63x
BenchmarkEncodeTwainSpeed1e5-8         10.28        16.26        1.58x
BenchmarkEncodeTwainSpeed1e6-8         10.63        16.69        1.57x
BenchmarkEncodeTwainDefault1e4-8       5.30         6.60         1.25x
BenchmarkEncodeTwainDefault1e5-8       3.03         4.83         1.59x
BenchmarkEncodeTwainDefault1e6-8       2.78         4.54         1.63x
BenchmarkEncodeTwainCompress1e4-8      5.25         6.50         1.24x
BenchmarkEncodeTwainCompress1e5-8      2.43         4.46         1.84x
BenchmarkEncodeTwainCompress1e6-8      2.18         4.16         1.91x

@dsnet We are running the Gzip benchmark as found in test/bench/go1/gzip_test.go

tl;dr, leave gzip as is for go1.7, we can address arm performance issues in go1.8
Alright, I think I've figured out what's causing the slow-down for ARM. While all of the go1.7 optimizations as a whole have helped arm and arm64 (as seen in my post above), there are some cases where performance regresses (as is the case for test/bench/go1/gzip_test.go)
One of major changes made to compress/flate was that we started using 4-byte hashing instead of 3-byte hashing. The new hash function is stronger and leads to less collisions and thus fewer searches, which is good for performance. On amd64, we take advantage of the fact that most CPUs can perform 4-byte unaligned loads pretty efficiently (#14267). However, this optimization is not possible on arm, so the arm targets have the pay extra cost to utilize the stronger hash function.
 We can think about better approaches in Go 1.8. I don't know enough about ARM to know if it will ever have unaligned reads/writes. So it may be worth investigating using the previous hash function over the new one for non x86 architectures.

If you use the dev.ssa branch and hack in config to make arm use ssa, how do things look? I.e. Will ssa for arm save us for free?

@dsnet, which function did you disassemble? Is it hash4 in compress/flate?
@josharian, SSA can get the performance back on ARM:
Gzip-4                      3.68s ± 0%      2.59s ± 1%  -29.76%

Gzip-4                   5.27MB/s ± 0%   7.50MB/s ± 1%  +42.36%

This is tip of dev.ssa plus my pending CLs, comparing with SSA off vs on. Not sure about the hash function specifically (as I don't know which function it is...)

Correct, it is hash4.  Currently it does not do combined unaligned load. It seems CSE helps mostly.
It might be possible to do combined load. Newer ARM support unaligned load. But currently the compiler (both backends) generates same instructions, only the assembler rewrite some instructions based on GOARM.

This issue has headed in several different directions.
I don't think we currently plan to do any further work to speed up the reflect package, which is the core of the original report.  It would be nice to know what the performance losses are on tip.
I'm going to close this issue.  If there is further work to be done for 1.7 let's move it to more focused issues.


⚠ Shadow Does Streams has degraded performance

In 327c3d7, Shadow Does Streams (https://stream.everythingbagel.me) experienced degraded performance:

HTTP code: 200
Response time: 3630 ms

Resolved: Shadow Does Streams performance has improved in e55ee9a after 15 minutes.


beforeSave is not called when data type is incorrect

I have column in my collection with type of array.
But in some cases string is sent instead of array, so to take care of that I've created beforeSave callback that is checking exactly for that and if this is string, it'll be converted to array with that string
if ( !object.get("weekday")  instanceOf Array ) object.set("weekday", [object.get("weekday")]);
The point is, that if string is set, beforeSave is not even called, instead it instantly returns response error I've get error:
schema mismatch for Goal.weekday; expected [object Object] but got String'
I've tried to console.log something in the beginning of my beforeSave callback code but it's not called at all
Is there any way to have beforeSave callback that will be called BEFORE types of data are validated?
I think it's weird, because my beforeSave callback is made to fix some data types issues, but it's not allowed to be fired before data types are validated.

This is a bit of a weird edge case, but if it worked in Parse.com we should make it work in Parse Server. We'll look into it.

OK, I tried this on Parse.com and it didn't work. This is my Cloud Code on Parse.com:
Parse.Cloud.beforeSave('Object', function(request, response) {
  request.object.set('shouldBeString', 'string' + request.object.get('shouldBeString'));
  response.success();
});
```

And my response when I try to save a number:
```json
{"code":111,"error":"invalid type for key shouldBeString, expected string, but got number"}
```

Can you post the the exact code you are using on Parse.com, that works?

My code was never tested on parse.com
Example you've posted shows exactly the same issue as I've got. You've got beforeSave hook that would convert some value to string, but it's not allowed to be fired, because at first your value was number. I'm pretty sure such validation should be made just before data is saved to database, not before "beforeSave" hooks are called.

We are aiming to not have any changed behaviour from Parse.com, so I don't think we are going to make this change. Imagine if someone ports their existing cloud code that relies on the data format being correct before it reaches the beforeSave, they could start getting corrupt data and their whole app could break. You'll have to fix your client code to send the right data type.

Thats quite limiting in my opinion. I think there should be some kind of flag, that would be disabled by default that would enable "breaking" changes.
If the flag would be disabled by default, then migration would not break any app from parse.com.
I feel what I've just wrote looks like overkill, so I'll just say again - thats quite limiting. Especially when it's open source now.

Coming a from a background of using strongly typed languages, I'm not really a fan of the "take whatever you get and coerce it to the right type" pattern anyway. Also, if you only have one field that you have to muck with the types of, and you enable this flag, you now have to check the types of every field, in case any of them are wrong too.
Besides, Is it really that hard to just do the coercion at the call site? I can't imagine it being more than 1-2 lines of code.

My case is converting natural language duration like "4 hours 20 minutes" to minutes count integer. In such case it would be very useful to have such beforeSave hook so I don't need to worry about converting it everywhere where it's saved.

What i Would do in that case, is store the string in a param and use the beforeSave to process it. You can delete the string of you don't need it. You can also store in in the target object, as 'originalString' and do your processing


Add integration to Azure Batch

To follow the reference architecture, add code integrate with Azure Batch.
Azure Batch provides input files in environment variables that need to be read.

Assign to me @msmarioo


1.33 regression: Ssh fails with tmt-reproducer.sh

Running manually it seems the issue is that the private key doesn't work:
$ ssh -v -p10025 -i /var/tmp/tmt/run-018/plans/examples/provision/default-0/id_ecdsa<EMAIL_ADDRESS>debug1: Offering public key: /var/tmp/tmt/run-018/plans/examples/provision/default-0/id_ecdsa ECDSA SHA256:Y1t8uXs0xVMzhrfIzGN/DULtuwH0faoW6NCl5Xaz9ZQ explicit
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password

tmt reproducer
$ ./tmt-reproducer.sh
fatal: destination path 'testcode' already exists and is not an empty directory.
fatal: refusing to fetch into branch 'refs/heads/gluetool/d4a1eab381278f020310bca9e58ca92707057ba2' checked out at '/home/lecris/tmp/tmt/testcode'
Already on 'gluetool/d4a1eab381278f020310bca9e58ca92707057ba2'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   727  100   727    0     0   1914      0 --:--:-- --:--:-- --:--:--  1918
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   274  100   274    0     0    758      0 --:--:-- --:--:-- --:--:--   759
/var/tmp/tmt/run-016
Found 1 plan.

/plans/examples
summary: Documentation examples
    discover
        how: fmf
        directory: docs
        warn: Sources will not be extracted, prepare step is not enabled.
        Tests will be discovered after dist-git patching in prepare.
        summary: 0 tests selected
    provision
        queued provision.provision task #1: default-0

        provision.provision task #1: default-0
        how: virtual
        image: Fedora-Rawhide
        memory: 2048 MB
        disk: 40 GB
        qcow: Fedora-Cloud-Base-Generic.x86_64-Rawhide-20240514.n.0.qcow2
        effective hardware: {}
        name: tmt-016-BlkwxnTL
        key: /var/tmp/tmt/run-016/plans/examples/provision/default-0/id_ecdsa
        progress: booting...
        primary address: <IP_ADDRESS>
        topology address: <IP_ADDRESS>
        port: 10024
        fail: Failed to connect in 120s.

plan failed

The exception was caused by 1 earlier exceptions

Cause number 1:

    provision step failed

    The exception was caused by 1 earlier exceptions

    Cause number 1:

        Failed to connect in 120s.
/var/tmp/tmt/run-016

/plans/examples
    discover
        status: done
        summary: 0 tests selected
        warning: No tests found, finishing plan.
/var/tmp/tmt/run-016

/plans/examples
    prepare

        summary: 0 preparations applied
    execute
    finish
        warn: Nothing to finish, no guests provisioned.

plan failed

The exception was caused by 1 earlier exceptions

Cause number 1:

    No guests available for execution.

@LecrisUT Can you check with:

F39 compose just to confirm it is not Rawhide related
Run tmt with -dddvvv to see the full output or upload the tmt log.txt

Hmm it seems to be rawhide specific, F39 worked. I have cleaned the /var/tmp/tmt/testcloud and tried again with all 3 f39, f40 and rawhide. Only rawhide was failing. Another annoyance is that virsh is not cleaned at the end of tmt run --since prepare (the other 2 manage to get cleaned)

Seems to be resolved in 1.34


[C++][Parquet] Add WriteRecordBatchAsync to parquet writer

Describe the enhancement requested
This PR enhances the parquet::arrow::FileWriter to write columns in parallel by setting ArrowWriterProperties::use_threads to true and optionally providing an executor. As the review comment explains, a nested parallelism deadlock may occur if the file writer itself is running in the same executor.
To provide better usability, we can adopt the suggestion from @westonpace to add a WriteRecordBatchAsync method that calls ParallelForAsync and returns the future. This can then be safely called in parallel, even by thread pool threads (assuming they don't block on that future but wrap it up into a higher level AllComplete call later). The WriteRecordBatch method could then just return WriteRecordBatchAsync(...).status().
Component(s)
C++, Parquet

I have checked this again and found that it is non-trivial to make the change.
In short, WriteRecordBatch is subject to max number of rows allowed in a row group. So it may slice the input record batch and write the sliced batches into different row groups in order. Please check here for reference: https://github.com/apache/arrow/blob/master/cpp/src/parquet/arrow/writer.cc#L454
To implement WriteRecordBatchAsync, we need to call ParallelForAsync for each sliced batch and chain the returned futures so that they can be executed in order. CMIW, we don't have the utility to support this yet.
Any suggestion? @westonpace @lidavidm

In short, WriteRecordBatch is subject to max number of rows allowed in a row group. So it may slice the input record batch and write the sliced batches into different row groups in order.

I'm personally not too worried about that feature as a lot of that behavior can be obtained with the dataset writer which has max_rows_per_file and max_rows_per_group and is independent of format.  It handles multiple parallel writes across multiple files.

CMIW, we don't have the utility to support this yet.

We have decent utilities for working with async tasks.  For example, you could use a throttled async task scheduler if you want to execute them in order.


The GUI wallet on Ubuntu 18.04 can NOT work (hanging on "Create" step)

The gui wallet (Coda.Wallet.0.1.1.AppImage) is used. It seems that the daemon is crashed. Logs and screen captures are attached.

coda-wallet.log

Can you share what OS version you're using? Thanks!

Ubuntu 18.04.2 LTS \n \l

Thanks @StarLI-Trapdoor we'll play with this to see if we can reproduce and make a fix!

Closing this issue as it is stale for a long time.


Footer position changes on load, onresize script does not fire

When using a mini-footer, on page load the footer changes position - the footer moves form just below to the content to the bottom of the browser window. The fact the footer is at the bottom of the browser window is welcome, however it is causing issue with my resize script. My script is used to detect changes in size of the 'main' tag, and if such a change occurs, fires another script. The initial position of the footer causes the 'main' tag to have a certain height, however when the position of the footer is changed, this does not cause my script to be fired. If I then change the window size manually, the script fires as expected.
This is what the window looks like on initial load (the console shows the 'main' tag height):

This is what the window looks like after manually changing the size:

In the first screenshot you'll see the 'editor' does not push up against the footer; that is because it has not been resized to fit the 'main' tag. In the second screenshot, after manually changing the window size the 'main' tag height changed triggering a resize, which can be seen as it caused the 'editor' to resize and push up against the footer.

@noilly Would you mind setting up a codepen that reproduces this issue? (Maybe a resize handler is enough to show of a discrepancy?)

@surma I think this covers it! https://codepen.io/anon/pen/MadZpR

P.S. I am no longer sure it is the footer that causes the issue. Taking out the footer didn't solve the problem (though initially I was convinced that the problem didn't start happening until after I had inserted the footer) - the problem persists without the footer.

Thanks for the codepen, @noilly. Luckily, the solution is quite simple. You run your resizeAce() before our JS has run so it actually does the correct thing, just way before layout has actually adjusted to the screen.
document.querySelector('.mdl-layout').addEventListener('mdl-componentupgraded', function() {
  resizeAce();
});

That should fix it :)

Thanks, that fixed it! :)

FYI, I believe our mdl-componentupgraded events bubble. So, within that handler you may want to just check the event target and verify it is indeed the layout. That way it isn't running on every other component in your page being upgraded.


Bedtools coverage

Is there a function to calculate the coverage or fraction of intersect of two or more pyranges?
Similar to https://bedtools.readthedocs.io/en/latest/content/tools/coverage.html

https://biocore-ntnu.github.io/pyranges/statistics-similarity-between-pyranges.html
and
https://biocore-ntnu.github.io/pyranges/statistics-spatial-correlations-between-pyranges.html
Will answer more when I’m back in front of a computer. (These stats-methods
are not as rigorously tested yet, because my method of testing is comparing
my result against the result of running bedtools on the same random data
and I found bugs in bedtools.)
Endre
On Thursday, July 11, 2019, Thiago Britto Borges<EMAIL_ADDRESS>wrote:

Is there a function to calculate the coverage or fraction of intersect of
two or more pyranges?
Similar to https://github.com/biocore-ntnu/pyranges/issues/new
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/biocore-ntnu/pyranges/issues/16?email_source=notifications&email_token=AEHURUUTWTVPAIXYFKR3ML3P63TB3A5CNFSM4IAZZFVKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4G6RYUJA,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEHURUS34PCXRR7BAFOV3ADP63TB3ANCNFSM4IAZZFVA
.

There is currently no equivalent to the coverage-function (pr.coverage gives an RLE). I do not think it should be hard to write though given the tools that currently exist in PyRanges though. I will add it to my TODO-list, but I have a lot I should finish before I add it though :)

It's feasible to do it with a mix of pyranges.intersect + some computations with the pands.DataFrames. I was just askig because we want to replace some of ours pybedtools calls, and a robust coverage function would be necessary.

I have added coverage to 0.0.48. Thanks for the request :)
import pyranges as pr
exons, cpg = pr.data.exons(), pr.data.cpg()
r = cpg.coverage(exons)
r[r.FractionOverlaps > 0]
+--------------+-----------+-----------+-----------+------------------+----------------------+
| Chromosome   | Start     | End       | CpG       | NumberOverlaps   | FractionOverlaps     |
| (category)   | (int32)   | (int32)   | (int64)   | (int64)          | (float64)            |
|--------------+-----------+-----------+-----------+------------------+----------------------|
| chrX         | 584563    | 585326    | 66        | 1                | 0.32503276539973786  |
| chrX         | 1510501   | 1511838   | 173       | 1                | 0.1854899027673897   |
| chrX         | 1553851   | 1554115   | 20        | 1                | 0.23484848484848486  |
| chrX         | 2846195   | 2847511   | 92        | 1                | 0.1094224924012158   |
| ...          | ...       | ...       | ...       | ...              | ...                  |
| chrY         | 14532115  | 14533600  | 126       | 1                | 0.027609427609427608 |
| chrY         | 15591259  | 15591720  | 33        | 4                | 0.7093275488069414   |
| chrY         | 16941822  | 16942188  | 32        | 1                | 1.0                  |
| chrY         | 26979889  | 26980116  | 21        | 1                | 0.6607929515418502   |
+--------------+-----------+-----------+-----------+------------------+----------------------+
Unstranded PyRanges object has 72 rows and 6 columns from 2 chromosomes.


libnvvm.so not found

I'm following the add example, and I get to the last step, but when I build, I get this:
cuda_test/cuda_runner/target/debug/build/cuda_runner-170008ea0ebbe57c/build-script-build: error while loading shared libraries: libnvvm.so.4: cannot open shared object file: No such file or directory
As far as I can tell, libnvvm.so is in cuda, which should be found by rustcuda. Is there any env variable I need to set, or any place I need to copy libnvvm.so?

Nevermind, I fixed it by adding the folder containing libnvvm.so to my LD_LIBRARY_PATH


Noticed the namespace is incorrect here

Context
Small fix to the class namespace for plugins.

@Yuuki77 please review this

@thoiberg please review this


⚠ Bofrost.lu has degraded performance

In b82a5bf, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 2906 ms

Resolved: Bofrost.lu performance has improved in 767568a after 27 minutes.


Hello Word

Challenge Say Hello to HTML Elements has an issue.
User Agent is: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36.
Please describe how to reproduce this issue, and include links to screenshots if possible.
My code:

"Hello word"

Hello word

Hello @CMexpressIT thanks for reporting this issue.
You have an error in your code that's preventing you to proceed.
The challenge asks for:

your h1 element's text to say "Hello World"

While the text you have implemented is not. (Mind the spelling).

Please consider that Github is a place for reporting bug and issues, next time please check in one of the official chat room and ask for assistance first.
happy coding 👍

Sorry. ✅
On 30 May 2017 13:11, "Claudio Restifo"<EMAIL_ADDRESS>wrote:

Hello @CMexpressIT https://github.com/cmexpressit thanks for reporting
this issue.
You have an error in your code that's preventing you to proceed.
The challenge asks for:
your h1 element's text to say "Hello World"
While the text you have implemented is not. (Mind the spelling).
Please consider that Github is a place for reporting bug and issues, next
time please check in one of the official chat room
https://forum.freecodecamp.com/t/free-code-camp-official-chat-rooms/19390
and ask for assistance first.
happy coding 👍
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/freeCodeCamp/freeCodeCamp/issues/15120#issuecomment-304764623,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ATdxjJI_2YIdwdUpcVncgvz83WwR2Nqbks5r-4jtgaJpZM4NpxuO
.

No need to apologize @CMexpressIT .
Hope you fixed your code, and remember for the future the flow:
Read Search Ask
Happy coding journey 👍

Closing as resolved. Happy coding!


Adds small modifier to rb-select-control

Adds a small modifier to rb-select-control-directive
Updates the demo to display the small select control
Adds a test for the feature

TP: https://rockabox.tpondemand.com/entity/9491

Really good commit messages! You don't need to separate out the test commit, but it doesn't hurt either way.
One thing... Write your commit message in the imperative: "Fix bug" and not "Fixed bug" or "Fixes bug. See new guidelines here.


Prebuild coverage table

The coverage tool of Lasair is very slow because of a big query. Would be better to run the query on ingestion and make a coverage table, so that access would be quick.

Done, pushed and pulled.


Issue with batch operation on room loading process

Recently we've encountered problems related to https://github.com/wechaty/wechaty-puppet-hostie/issues/31
From the tests we've done on the issue above, we know that large batch operations in a short period of time will trigger it, which will cause wechaty-puppet-hostie to restart.
Based on our test, comparing with Room.findAll(), Contact.findAll()  seldom triggers the problem. Then I dig into the code and found that:
// src/user/contact.ts L224-L245
      const BATCH_SIZE = 16
      let   batchIndex = 0

      const invalidDict: { [id: string]: true } = {}

      while (batchIndex * BATCH_SIZE < contactList.length) {
        const batchContactList = contactList.slice(
          BATCH_SIZE * batchIndex,
          BATCH_SIZE * (batchIndex + 1),
        )
        await Promise.all(
          batchContactList.map(
            c => c.ready()
              .catch(e => {
                log.error('Contact', 'findAll() contact.ready() exception: %s', e.message)
                invalidDict[c.id] = true
              }),
          ),
        )

        batchIndex++
      }

The code in the method Contact.findAll() is doing a pagination call, which will reduce the count of requests made simultaneously, thus avoid the wechaty-puppet-hostie issue.
But the code in Room.findAll() does not have the pagination mechanism. So I think we should add pagination in the method.
We've tested the pagination in our scoped npm package @windmemory/wechaty, and seems all working as expected.

Nice catch!
It's related to https://github.com/wechaty/wechaty-puppet/issues/6 and https://github.com/wechaty/wechaty/issues/1450
I hope this will reduce the number of rejections of https://github.com/wechaty/wechaty-puppet-hostie/issues/31 , let's merge it to see.


Resolved performance problems with large datasets.

I noticed that with large datasets, this library breaks down really bad. After some investigation, I realized that you are "re"-rendering the entire list for EVERY item that is added to the list. This creates a runaway exponential time to render for each new item that is added to the list.
The solution is simple. Add a debounce around the render() method and performance goes up about 10x for large datasets (> 1000 records).
JSFiddle Before Fix  ~1600 records takes more than 30 seconds to render
JSFiddle After Fix  ~1600 records takes about 2 seconds

I feel like requestAnimationFrame would be more suitable here instead of setTimeout. I'm concerned there will still be the issues mentioned here with this change...

That concern is still a problem. I gave a response to that thread if you want to read through it. I will try to find some time to debug through the code, but this problem could be solved by passing an aync flag to the addChoice method, which will trigger a debounced render.  You could then add a new method called renderAsync and the code would be changed as follows.
/**
   * Render DOM with values asynchronously with a debounce.
   */
  renderAsync() {
    if (this.renderDebounce) {
      clearTimeout(this.renderDebounce);
    }

    this.renderDebounce = setTimeout(() => this.render(), 100);
  }

On a separate note, I had to add this to our library to get through this problem.
https://github.com/formio/formio.js/blob/master/src/components/select/Select.js#L13
This is for the JSON form renderer @ http://formio.github.io/formio.js/.  So far with the change I made (which is to duck-punch the render method to make it debounced), we have not seen any negative effects.

This is a major improvement and could do with being merged asap, however I noticed the setValueByChoice no longer works: https://jsfiddle.net/od1fL1kg/

@evu The latter is why it's not being merged. I'm currently working on getting decent test coverage so I can validate what breaks with this change.

@jshjohnson Just realised it's because the develop branch changed setValueByChoice to setChoiceByValue compared to the live branch.


ChatQnA Example - Unable to run due to Issues

I am trying to run the ChatQNA application

Able to run all the microservices using docker compose file.

Getting this error in the tgi-service, What is the correct way to provide the external IP/Hostname ?

Due to the aforementioned error, I am not able to run the curl example from the README,
curl http://${host_ip}:9000/v1/chat/completions\ -X POST \ -d '{"query":"What is Deep Learning?","max_new_tokens":17,"top_k":10,"top_p":0.95,"typical_p":0.95,"temperature":0.01,"repetition_penalty":1.03,"streaming":false}' \ -H 'Content-Type: application/json'

I think the  llm-tgi-server service is not able to connect to the tgi-service due to this hostname issue

Logs

`llm-tgi-server             | The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well.
llm-tgi-server             | Token is valid (permission: read).
llm-tgi-server             | Your token has been saved to /home/user/.cache/huggingface/token
llm-tgi-server             | Login successful
llm-tgi-server             | INFO:     <IP>:37050 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
llm-tgi-server             | ERROR:    Exception in ASGI application
llm-tgi-server             | Traceback (most recent call last):
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/huggingface_hub/utils/_errors.py", line 304, in hf_raise_for_status
llm-tgi-server             |     response.raise_for_status()
llm-tgi-server             |   File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
llm-tgi-server             |     raise HTTPError(http_error_msg, response=self)
llm-tgi-server             | requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: http://<IP>:9009/
llm-tgi-server             |
llm-tgi-server             | The above exception was the direct cause of the following exception:
llm-tgi-server             |
llm-tgi-server             | Traceback (most recent call last):
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/huggingface_hub/inference/_client.py", line 273, in post
llm-tgi-server             |     hf_raise_for_status(response)
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/huggingface_hub/utils/_errors.py", line 371, in hf_raise_for_status
llm-tgi-server             |     raise HfHubHTTPError(str(e), response=response) from e
llm-tgi-server             | huggingface_hub.utils._errors.HfHubHTTPError: 503 Server Error: Service Unavailable for url: http://<IP>:9009/
llm-tgi-server             |
llm-tgi-server             | The above exception was the direct cause of the following exception:
llm-tgi-server             |
llm-tgi-server             | Traceback (most recent call last):
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 411, in run_asgi
llm-tgi-server             |     result = await app(  # type: ignore[func-returns-value]
llm-tgi-server             |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
llm-tgi-server             |     return await self.app(scope, receive, send)
llm-tgi-server             |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
llm-tgi-server             |     await super().__call__(scope, receive, send)
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
llm-tgi-server             |     await self.middleware_stack(scope, receive, send)
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
llm-tgi-server             |     raise exc
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
llm-tgi-server             |     await self.app(scope, receive, _send)
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
llm-tgi-server             |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
llm-tgi-server             |     raise exc
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
llm-tgi-server             |     await app(scope, receive, sender)
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
llm-tgi-server             |     await self.middleware_stack(scope, receive, send)
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
llm-tgi-server             |     await route.handle(scope, receive, send)
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
llm-tgi-server             |     await self.app(scope, receive, send)
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
llm-tgi-server             |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
llm-tgi-server             |     raise exc
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
llm-tgi-server             |     await app(scope, receive, sender)
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
llm-tgi-server             |     response = await func(request)
llm-tgi-server             |                ^^^^^^^^^^^^^^^^^^^
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
llm-tgi-server             |     raw_response = await run_endpoint_function(
llm-tgi-server             |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/fastapi/routing.py", line 193, in run_endpoint_function
llm-tgi-server             |     return await run_in_threadpool(dependant.call, **values)
llm-tgi-server             |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/starlette/concurrency.py", line 42, in run_in_threadpool
llm-tgi-server             |     return await anyio.to_thread.run_sync(func, *args)
llm-tgi-server             |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-tgi-server             |   File "/usr/local/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
llm-tgi-server             |     return await get_async_backend().run_sync_in_worker_thread(
llm-tgi-server             |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-tgi-server             |   File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2134, in run_sync_in_worker_thread
llm-tgi-server             |     return await future
llm-tgi-server             |            ^^^^^^^^^^^^
llm-tgi-server             |   File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 851, in run
llm-tgi-server             |     result = context.run(func, *args)
llm-tgi-server             |              ^^^^^^^^^^^^^^^^^^^^^^^^
llm-tgi-server             |   File "/home/user/comps/llms/langchain/llm_tgi.py", line 73, in llm_generate
llm-tgi-server             |     response = llm.invoke(input.query)
llm-tgi-server             |                ^^^^^^^^^^^^^^^^^^^^^^^
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 276, in invoke
llm-tgi-server             |     self.generate_prompt(
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 633, in generate_prompt
llm-tgi-server             |     return self.generate(prompt_strings, stop=stop, callbacks=callbacks, **kwargs)
llm-tgi-server             |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 803, in generate
llm-tgi-server             |     output = self._generate_helper(
llm-tgi-server             |              ^^^^^^^^^^^^^^^^^^^^^^
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 670, in _generate_helper
llm-tgi-server             |     raise e
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 657, in _generate_helper
llm-tgi-server             |     self._generate(
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 1317, in _generate
llm-tgi-server             |     self._call(prompt, stop=stop, run_manager=run_manager, **kwargs)
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/langchain_community/llms/huggingface_endpoint.py", line 256, in _call
llm-tgi-server             |     response = self.client.post(
llm-tgi-server             |                ^^^^^^^^^^^^^^^^^
llm-tgi-server             |   File "/home/user/.local/lib/python3.11/site-packages/huggingface_hub/inference/_client.py", line 283, in post
llm-tgi-server             |     raise InferenceTimeoutError(
llm-tgi-server             | huggingface_hub.errors.InferenceTimeoutError: Model not loaded on the server: http://<IP>:9009. Please retry with a higher timeout (current: 120).`

Can someone please help in resolving this issue ?

@yogeshmpandey,
Thanks for raising the issues. We're currently in the process of actively refactoring the GenAIExamples to adhere to a microservice-based architecture. Please refer to the latest version of the README for updated instructions.
Service Unavailable for url: http://<IP>:9009/ this error is because you didn't export IP as a real IP address. Please follow the readme.

@yogeshmpandey  The GenAIExamples ChatQaA  are ready.
You can try it know with latest code of GenAIExample and GenAIComps.

closed as no more action required.


fix(license): use valid spdx identifier

There is an "X11" identifier per https://spdx.org/licenses/ .

Even if this repo is no longer actively maintained, @substack , it'd be great to get this addressed for the sake of those checking against any of the many requiring libraries that aren't inclined to make the change. Thanks!

There is an "X11" identifier per https://spdx.org/licenses/ .

The MIT and X11 licenses are basically the same. If the purpose was to release it under both licenses the spdx identifier should be (MIT OR X11).


Introduce MarkdownDocumentReader

Motivation
@markpollack and @tzolov, maybe you are interested in the new MarkdownDocumentReader, which can read structured Markdown documents. As @markpollack wrote in #105, it could be valuable. I agree.
So, I've prepared a simple implementation of that DocumentReader.
Description
For parsing Markdown documents, I've used the commonmark/commonmark-java library.
Document dividing
By default, all documents are divided by headers. This includes all header types from 1 to 6. For a simple document like:
# AAA
content 1

## BBB
content 2

### CCC
content 3

#### DDD
content 4

##### EEE
content 5

###### FFF
content 6

Six documents will be generated. Each of these documents will have entries in the metadata as follows:

category => header_X, where X is the number of the header
title => <header title>, e.g.: BBB from the example

There is also an option to divide the Markdown document by horizontal lines. This is not the default option, but it can be turned on through configuration.
Blockquotes and Code Blocks support
All blockquotes and code blocks are treated as separate documents. For code blocks where we the language can be determined, it is included in the lang metadata entry.
This behavior can be changed by setting options.
Additional metadata
The Markdown reader configuration also provides support for additional metadata, which may be set for all processed documents. It contains fixed values that offer more context about the created document, such as the service name that provides the document, or the environment in which it was created.
TODO

[ ] Add MarkdownDocumentReader to the documentation
[ ] Update a ETL Class Diagram

Hi, I'm super happy to see this.  I'll review asap.  I have a version of this on my machine from way back when i started the project.  Markdown is a great "lingua franca" for document ETL.

@markpollack great to hear it!
Any ideas or enhancements are more than appreciated.
I also have some future ideas about handling tables, but first things first. Baby steps.

I've added docs. I haven't tried it in anger yet but it looks great.  Merged in 56e678c487e64b9731d4f16e9bbaafe4a729b7f3


Ionic 3 - Slow load tabs first time

Hi,
I have problems when I loaded my TabPage for first time, the page stay on blank for 10 seconds! Some can help me?
Thx

Hello! Thank you for opening an issue with us!
The Ionic Framework issues list is designated for the reporting of bugs with the Ionic Framework or for requesting new features. As this seems like more of a support question I am going to close this issue.
I urge that you ask this question on our forum or on our slack channel.
Thanks for using Ionic!

Sure? I saw that it's pass in a lot of device. Maybe is issue of ionic
2018-02-14 23:35 GMT+01:00 Ken Sodemann<EMAIL_ADDRESS>
Closed #14021 https://github.com/ionic-team/ionic/issues/14021.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ionic-team/ionic/issues/14021#event-1474841395, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AfL3TtRgzebNlG-3cgHnykkyskim0_Ttks5tU1-5gaJpZM4SFWkX
.

Same issue for me on android. Only android, not ios device


Code update : Reboot BMC during BMC activation in progress

[x] Upload BMC image
[x] Activation in progress and reset BMC
[x] Verify the failed BMC image

I think this is going to be the same as https://github.com/openbmc/openbmc-test-automation/issues/993. Activating the image is the same thing as a code update.


chore(release): automate versions.json

This workflow update runs whenever something is pushed to main or any release/* branch. It does the same as before, when there are changesets, open a release PR, if there are no changesets, do a release.
If there are changesets and it's the main branch, it will add an updated versions.json to the release PR.
If there are no changesets and it's not the main branch, it will open a PR against main with an updated versions.json file.

[ ] Current version should always go to https://design-system.post.ch and not to https://design-system.post.ch/vCURRENTMAJOR
[ ] Insert date created and date updated

[ ] Updating versions.json from an old version needs to run on the main branch, because the old documentation does not know where the versions.json is on main, but the old documentation needs to provide it's package.json as input to the update workflow

For the moment, it's easier to do major version changes by hand. Instructions will follow.


[Bug]: Cannot decrypt existing secrets once the RSA key pair has been changed

NetBox Secrets plugin version
1.7.6
NetBox version
3.4.10
Steps to Reproduce

Create a first secret using one RSA key pair
Change the user's RSA key pair to another key pair
Try to decrypt the first secret
Reswitch to the initial RSA key pair
Try to decrypt the first secret

The steps can be seen here
Expected Behavior
For step 3, a popup saying that this secret has been encrypted with another key.
For step 5, the decrypted secret
Observed Behavior
Can't decode byte error

The reported issue is not a bug as the form clearly states that once a key is changed another administrator activate it. Your steps do not include that this was done.

I saw the video and the issue is that re-saving the form marked the key as active which should not have happened. In any case, a new bug should be filed to fix this issue as the one reported is incorrect.


[Dev] Build script improvements

Refactored BuildAll to load from Projects.json, which should make this a bit easier to expand upon in the future
Added Run command. Example: .\Development\Run.ps1 -ProjectsToRun Bizhawk,RTCV will launch Bizhawk and RTCV.

I like it.


Fixed negative timestamp tests

Pull Request Checklist
Thank you for taking the time to improve Arrow! Before submitting your pull request, please check all appropriate boxes:

[ ] 🧪 Added tests for changed code.
[x] 🛠 All tests pass when run locally (run tox or make test to find out!).
[ ] 📚 Updated documentation for changed code.
[x] ⏩ Code is up-to-date with the master branch.

If you have any questions about your code changes or any of the points above, please submit your questions along with the pull request and we will try our best to help!
Description of Changes
Fixed negative timestamp tests on macOS and Linux.

Codecov Report

Merging #743 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff          @@
##           master   #743   +/-   ##
=====================================
  Coverage     100%   100%
=====================================
  Files           9      9
  Lines        1678   1678
  Branches      283    283
=====================================
  Hits         1678   1678

Impacted Files
Coverage Δ

arrow/util.py
100% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 70fb905...1d22592. Read the comment docs.

Closing because of https://github.com/crsmithdev/arrow/pull/745.


Texture init fails when default data pointer is not null

I'm referring to this line. This is what I expect to work:
C3D_Tex tex;
C3D_TexInit(&tex, 64, 64, GPU_RGB8);

However it will fail depending on the initial value of the data pointer. At the moment, I just use tex.data = NULL; before it. Perhaps it's better to trust the user won't initialize multiple times instead of requiring the struct to be zero'd?

Noted. Currently you are supposed to memset C3D_Tex structures on the stack before initializing them.

See 033c600.


Mention that you have to enable systemd-networkd

I am new to using OpenVPN, and was following the guide on the Arch Linux wiki that mentioned this repository. Everything worked pretty smoothly, so thanks for that :)
After rebooting, I tried systemctl start<EMAIL_ADDRESS>but got the following error message:
ERROR: Cannot set tx queue length on tun0-unrooted: Operation not permitted (errno=1)

It took me some time to figure out where the problem was: in order for the /etc/systemd/network/tun0-unrooted.netdev profile to be activated, I needed to start systemd-networkd. I have no idea if this service is started by default for most users and I somehow disabled it, but if that is not the case, maybe you can mention somewhere in your readme that users have to enable that service for things to keep working through reboots.

Hi @Neftas

I am new to using OpenVPN, and was following the guide on the Arch Linux wiki that mentioned this repository. Everything worked pretty smoothly, so thanks for that :)

I'm glad to hear that :)

After rebooting I tried systemctl start<EMAIL_ADDRESS>but got the following error message:
ERROR: Cannot set tx queue length on tun0-unrooted: Operation not permitted (errno=1)

The error occurs because openvpn, running as an unprivileged user, doesn't have permission to set the transmit queue length for the tun0-unrooted device. This is not a fatal error (which is not to say it's irrelevant).
When openvpn-unroot creates a persistent tun device, it uses openvpn --mktun, which sets the txqueuelen to OpenVPN's default - 100. When starting, OpenVPN will blindly try to set the value again (even though it's already what it wants) and fail, but the error has no practical relevance.
However, when you reboot the system and the device is created by systemd (based on the generated .netdev file), the txqueuelen may, and probably will be, different. That's the case where the error is relevant.
So how do we resolve the problem ?
Systemd should allow us to set txqueuelen in the .netdev, or .link file, but it simply doesn't. The feature is unsupported apparently.
Passing --user/--group to openvpn --mktun, or setting User/Group in a .netdev file also doesn't allow the specified user to manipulate the setting.
This leaves us will only one option - set the value as root after the device is created, by invoking iproute2 from an init script, or an openvpn --up script:
ip link set dev tun0-unrooted txqueuelen 100

If it could be handled in systemd, or OpenVPN configs, I'd add the functionality to openvpn-unroot, but since that's impossible, it's left for the user to handle (or not), according to their preferences. In any case, OpenVPN will work correctly, though the different txqueuelen settings may impact network latency.

It took me some time to figure out where the problem was: in order for the /etc/systemd/network/tun0-unrooted.netdev profile to be activated, I needed to start systemd-networkd. I have no idea if this service is started by default for most users and I somehow disabled it, but if that is not the case, maybe you can mention somewhere in your readme that users have to enable that service for things to keep working through reboots.

This is weird. If systemd-networkd was not enabled, how was your networking being set up after boot ? I'm not sure what happened there, but the assumption is that either a) you are using systemd (including systemd-networkd) and the generated .netdev file will be enough to bring up the tunnel, or b) you're not using systemd and you'll handle bringing up the interface yourself.
I will keep your report in mind the next time I update the README file. I haven't had much time to work on openvpn-unroot recently, but I do plan to get back to it in the near future.
Thanks for taking the time to report the issue. I hope you find this explanation satisfactory.
Cheers.

Thanks for your explanation. I'm not sure I understand everything completely, but your reply gives plenty of openings for me to investigate myself. Thanks again for creating and maintaining this package, it is much appreciated.
Take care.


Export result set optimization

When exporting tens of thousands of pieces of data, it takes more than 5 minutes. Whether it is exported from the result set or directly, can the time be optimized?

It is not clear - what exactly you propose?
You can change fetch size parameter (in some cases it increases performance dramatically).

目前尚不清楚 - 你究竟提出了什么建议？
您可以更改提取大小参数（在某些情况下，它会显着提高性能）。

提高 导出效率


[FR] Provide a choice during install for Allowed IPs/Ranges

It would be really nice if the script gave us a choice for the Allowed IPs of each client to be <IP_ADDRESS> ::0 (ALL/full VPN) or the internal range provided by the VPN (<IP_ADDRESS>/24) this way it would allow those of us who use this with pihole to easily have split tunnel only for DNS traffic.
Perhaps the script could run a little check to see if pihole is present on the system and then offer this up as a choice during installation.

I don't really like having too much options, PiVPN is supposed to be an installer to help people get started with a fully functional VPN, not an an abstraction for every functionality, expecially when the change is very easy: https://github.com/pivpn/pivpn/wiki/WireGuard#how-do-i-allow-clients-access-to-my-home-network-but-not-the-internet-through-my-pivpn

Thanks for the response.
Whilst I understand what your saying this is a fundamental choice when implementing any VPN:
Do I want a split tunnel or not?
Adding this into the script won't add clutter but allow the installer to think of this fundamental question right from the get go and stop any confusion when deploying it as well as removing the need to edit every single file by hand which let's face it can be tedious with a large amount of clients.
You could also adding this as a command switch or part of the process for when we add a client with pivpn -a, Once it has asked you to input a title for the client it could simply ask if this is split tunnel or not, if yes just add the subnet used by wireguard instead of <IP_ADDRESS>/0 ::0
Personally I think it would be cleaner, Don't you?

this seems to me an edge case and a specific usage for power users ... those who want that kind of setup can do it themselves ... I kind of agree with @orazioedoardo ... the priority here is the common user who knows pretty much nothing!
I'd like to have certain kind of features (like this one) out of reach of common users because ... well ... users will be users and able to do the most weird things happen .. and who has to deal with their recurring issues?
I know it can be tedious for you ... but well ... PiVPN is not really meant to Manage a big number of users either ... otherwise it would be an enterprise solution :)

With a name like pivpn many users will come from pihole meaning all we want
is DNS going down the VPN.
If that's an enterprise use case I'll shut up and continue to tell them
their use case is not catered for as it's far too advanced for a noob.
Cheers
On Thu, 7 May 2020, 17:40 4s3ti<EMAIL_ADDRESS>wrote:

this seems to me an edge case and a specific usage for power users ...
those who want that kind of setup can do it themselves ... I kind of agree
with @orazioedoardo https://github.com/orazioedoardo ... the priority
here is the common user who knows pretty much nothing!
I'd like to have certain kind of features (like this one) out of reach of
common users because ... well ... users will be users and able to do the
most weird things happen .. and who has to deal with their recurring issues?
I know it can be tedious for you ... but well ... PiVPN is not really
meant to Manage a big number of users either ... otherwise it would be an
enterprise solution :)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/pivpn/pivpn/issues/1041#issuecomment-625366450, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAMF7VYVXR4MRAA7V2ED343RQLQA7ANCNFSM4MYTHQUA
.

@t0m5k1 No, split tunnel is not too advanced for a noob, however I think the user should be encouraged to learn the inner workings instead of having an option for everything.

Lately I was reconsidering making openvpn and wireguard steps equal, while providing "advanced options" for these additional features.

@t0m5k1 Sorry if i wasn't clear enough ... when i mentioned the enterprise solution was regarding the amount of users to manage with PiVPN. not about the feature itself :)

i am grateful for this repo and do not have anything to tilt scales in this (stale) conversation.
but i think it's worth framing our notions of "noobs" and "common users" vs "power users" among pivpn users as people who are nonetheless on the command line, making them a rare creatures at large. they have already taken the plunge into tedious complexity.
an uncommon need for control paired with a high tolerance for problem solving is the very reason that an otherwise rational person finds themselves cloning an open source repo onto a linux machine in order to leverage the functionality of their the personal dns service.
top to bottom, these people are not lambs; i think "encouraged to learn the inner workings" brought them here in the first place.

commit b5ec995 all but closes this?


Question about the shape of  X_train

X_train = tf.placeholder(tf.int32, [n_batch_train, 2, n_ctx, 2])
xmb[:, :, :, 1] = np.arange(n_vocab+n_special, n_vocab+n_special+n_ctx)
why there is a channel of additional tokens?

Problem solved! This part of the xmb is used for the learned positional encoding. https://github.com/huggingface/pytorch-openai-transformer-lm/issues/12#issuecomment-401770634


Market_basket

In this project I will be analyzing different items purchased using Apriori Algorithm.
Issue:#286

@Jayshah6699 this PR consists of 2 issues. So I request you to give it a level 5.


RISC-V Control Flow Graph and Candidate Execution Paths

We need to compute the CFG of a given binary, in particular the in-edges of all instructions. We also need to compute the shortest CFG paths between the entry point and all exit points as candidates for computing reachability. Once this is done we will also need to lengthen shortest CFG paths incrementally to longer paths by unrolling loops and recursion.

@cksystemsgroup/monsters I think there is a fundamental problem in CFG computation for modern programming languages.
Our engine relies on the fact, that we somehow generate a small amount of actuall (possible) execution paths. But modern programming languages have constructs like "function pointers" (C,..) and "virtual method dispatch" (C++, Rust,..), which makes it kind of impossible to generate viable execution paths statically. That is, because you can not decide how control flow will go on, without actually (partially) executing the code. And these languages features are widely used and therefore would drastically limit the amount of target programs.
A "standard" symbolic execution engine (like the old Monster) handles this problem quit well, because alltough a function pointers is inherently dynamic, they should be concrete in allmost all cases during symbolic execution. So I think that is a problem inherent to our approach.
One possible solution could be to build a candidate path on source code level by tracking the symbols used (Every function pointer has an function symbol assigned at some point in the code). But also with that approach, I don't think we will get a small set of candidate paths.

There is no source code for us, except for the links back to line numbers. Sure, a binary may compute the target address of a jump at runtime but let's ignore that case for now, meaning just terminate and say: can't do.

Well, for RISC-U that is not a problem at all. So we currently totally ignore that case anyway. Okay so we will figure that out, once we have to implement that.

Hey! To clarify what we want to do:
You can use programming languages like e.g. Rust or C to create the new
monster engine, but your "target" would not be C, C++, Rust or anything
like that, but the compiled Selfie binary, that we want to flash to a
real RISC-V-Board (like e.g. HiFive Unleashed). You can disassemble this
binary into RISC-V assembler instructions. For starters we need to find
the User Code within this binary/assembler and analyze it using the
presented new method ("propagation based local search"). Afterwards my
job (PHD) will be to use this implementation of yours to not only verify
the User Code, but also all possible traps (interrupts + kernel calls)
of the Selfie-OS (Marcell and Martin are currently working on this). The
new part of our project is that we are the first people, who will ever
implement this method on a binary level. The long-term perspective is to
make Selfie self-verifying on a binary level and allow Selfie to act as
a safety-critical application in a real-life scenario without any
additional Software like e.g. an external OS.
So for the beginning your projects "input" or "target" to analyze is the
dissassembled User Code Area of the machine-branch compiled
Selfie-Binary. Therefore no fancy Source Code Features involved. Does
that make it more clear or just more weird? ;)
Greetings, Mat
Am 14.08.2020 um 14:28 schrieb Christian Mösl:

@cksystemsgroup/monsters
https://github.com/orgs/cksystemsgroup/teams/monsters I think there
is a fundamental problem in CFG computation for modern programming
languages.
Our engine relies on the fact, that we somehow generate a small amount
of actuall (possible) execution paths. But modern programming
languages have constructs like "function pointers" (C,..) and "virtual
method dispatch" (C++, Rust,..), which makes it kind of impossible to
generate viable execution paths statically. That is, because you can
not decide how control flow will go on, without actually (partially)
executing the code. And these languages features are widely used and
therefore would drastically limit the amount of target programs.
A "standard" symbolic execution engine (like the old Monster) handles
this problem quit well, because alltough a function pointers is
inherently dynamic, they should be concrete in allmost all cases
during symbolic execution. So I think that is a problem inherent to
our approach.
One possible solution could be to build a candidate path on source
code level by tracking the symbols used (Every function pointer has an
function symbol assigned at some point in the code). But also with
that approach, I don't think we will get a small set of candidate paths.
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
https://github.com/cksystemsgroup/monster/issues/2#issuecomment-674050398,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ALOJJ7A637V4H2EKNIMAGSTSAUUYHANCNFSM4O4PT35A.

Our first target will be RISC-U (Selfie generated binaries). But as @ckirsch mentioned, Monster will be developed with the intention to extend that to handle a realistic RISC-V instruction set.
Maybe it was a bit confusing, how I explained the problem. This is not a problem of a fancy source code feature, but rather a problem of dynamic control flow in general. In RISC-U we have a instruction, were we can not generate a useful CFG for the general case (=> jalr). But as we know, that our binaries are generated by selfie (atm), we can make assumptions on how jalr is used and therefore can generate a useful CFG.
So to sum that up. I was worried, that our solver can work for Selfie binaries only.

The input of monster is a RISC-U binary (not assembly), for now. It will only eventually be a RISC-V binary. Monster should terminate if it encounters an instruction (invocation) that it cannot handle.


[Feature] Protection against specific file reads

Is your feature request related to a problem? Please describe.
A malicious script might try to harvest tokens from a machine.
Describe the feature you'd like
A wrapper around the nodejs fs api when plugins require which first checks to see if the file being requested contains the users discord token. This could slow down some plugins slightly, but would only happen on file reads, which can usually be cached.

Additional security features like this are planned.


Updates from aks-engine spike

What this PR does / why we need it: During our experimental spike to aks-engine, several forward-looking functional changes were made that we want here. From the commit history of the branch that this PR derives from:

Fix prow set up
e2e changes
removing openshift artifacts
accelerated networking rationalization, with tests
remove additional sed statements for ip-masq addons
Update go-dev tools image for go 1.11.2
remove unused azconst methods
add support PB6 vm skus
update azure_const unit test
update tiller versions in the recent versions of kubernetes
VSTS VHD pipeline hosted ubuntu pool
azureconst cruft
scale: persist scale down in api model
Add support for Kubernetes 1.11.5
Fix docker-engine install in VHD pipeline
remove IsOpenShift from E2E

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Special notes for your reviewer:
If applicable:

[ ] documentation
[ ] unit tests
[ ] tested backward compatibility (ie. deploy with previous version, upgrade with this branch)

Release note:

Updates from aks-engine spike

/lgtm


Support DNS queries from darknets

[ ] CJDNS
[ ] I2P
[ ] Onion

Superseeded by #20 and #19

Superseeded by https://github.com/virtu/darkseed/pull/20 and https://github.com/virtu/darkseed/pull/19


Fixes A problem I am facing with a header in the response containing …

…content-encoding in lowercase letters. Now I first check for the format Content-Encoding and if that is not found I also check the lowercase version.
Next to that there was an issue checking if a request is made for gzip content. If you ask for more than gzip, ie gzip,deflate,sdch. Now I check if what is asked contains gzip instead of equals to.
Fixed gh-662

@spencergibb I have added the HttpHeaders, though I do not really like the implementation. I tried to combine the two loops in the method but I got unit tests errors in: ServletPathZuulProxyApplicationTests. Could not figure out why there was a problem. Had to do with the response not being a gzip response while the headers indicate it was. Curious to know if this is what you meant.

@jettro have you signed the contributor's agreement?

Hi @spencergibb, yes I did sign the contributor's agreement.

Squashed and merged with some polish via a00fd2c3488cffa6ff9a25c197c719895d16473f


Missing C linkage in pico/util/datetime.h

Similar to issue #249 but in pico/util/datetime.h, the C linkage specification 'extern "C"' is missing so giving rise to linking failure when developing in cpp.
SDK v1.2.0

fixed


Completion refactoring

Just a usual refactoring, turning a rather ad-hoc completion infra into something extensible

bors r+


READ-WRITE SPLIT not work with SHARDING in SHARDINGJDBC

server:
  port: 12601

spring:
    main:
      allow-bean-definition-overriding: true
    shardingsphere:
      props:
        sql:
          show: true
      sharding:
        tables:
          test_sharding:
            actual-data-nodes: ds$->{0..1}.test_sharding
            table-strategy:
              inline:
                sharding-column: id
                algorithm-expression: test_sharding
        default-database-strategy:
          inline:
            sharding-column: id
            algorithm-expression: master$->{id % 2}
        master-slave-rules:
          ds1:
            master-data-source-name: master1
            slave-data-source-names: slave1
            load-balance-algorithm-type: round_robin
          ds0:
            master-data-source-name: master0
            slave-data-source-names: slave0
            load-balance-algorithm-type: round_robin
      datasource:
        names: master1,slave1,master0,slave0
        master1:
          name: master1
          url: jdbc:mysql://localhost:30001/TestReadWrite?useSSL=false&serverTimezone=GMT%2B8
          username: root
          password: 123456
          #
          type: com.alibaba.druid.pool.DruidDataSource
          driver-class-name: com.mysql.jdbc.Driver
        master0:
          name: master0
          url: jdbc:mysql://localhost:30003/TestReadWrite?useSSL=false&serverTimezone=GMT%2B8
          username: root
          password: 123456
          #
          type: com.alibaba.druid.pool.DruidDataSource
          driver-class-name: com.mysql.jdbc.Driver
        slave1:
          name: slave1
          url: jdbc:mysql://localhost:30002/TestReadWrite?useSSL=false&serverTimezone=GMT%2B8
          username: root
          password: 123456
          #
          type: com.alibaba.druid.pool.DruidDataSource
          driver-class-name: com.mysql.jdbc.Driver
        slave0:
          name: slave0
          url: jdbc:mysql://localhost:30004/TestReadWrite?useSSL=false&serverTimezone=GMT%2B8
          username: root
          password: 123456
          #
          type: com.alibaba.druid.pool.DruidDataSource
          driver-class-name: com.mysql.jdbc.Driver

---------------------------------------------------------------------------------------------------
springboot+mybatisplus
application.yml is shown above

sharding is available but
READ-WRITE SPLIT NOT WORKING
always read master,never read slave

version : 4.0.0-RC1

Please ref example first, all demos can run correctly.
https://github.com/apache/incubator-shardingsphere-example/

Submit issue for common feature, please make sure example are incorrect first.


[builder] cellxgene_ontology_guide integration

Integrate cellxgene_ontology_guide in place of the old TissueMapper.

COG should also be able to replace ontology_mappy.py

This is lower priority as it's not part of the core builder.

COG should also be able to replace ontology_mappy.py

Are its functionalities covered by COG? This is only used by Geneformer, which isn't part of core Census, so I wouldn't be too worried about it.


CDS-94151

What Type of PR is This?

[x] Issue
[ ] Feature
[ ] Maintenance/Chore

If tied to an Issue, list the Issue(s) here:

Issue(s)

@puthrayaharness should I add an extensive list like the one in FirstGen, https://developer.harness.io/docs/first-gen/firstgen-platform/techref-category/variables/built-in-variables-list/#artifact-rollback-variables?

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://66068a8c54c2bccc41598c11--harness-developer.netlify.app


Jest reports run complete but not exited only when ran with npm script

🐛 Bug Report
Jest reports "Jest did not exit one second after the test run has completed." when tests are executed with NPM scripts but does not report the same issue if ran with directly with the node process.
More details in https://github.com/aws/aws-sdk-js-v3/issues/2389
To Reproduce
Steps to reproduce the behavior:

clone https://github.com/clcpolevaulter/sts-example
Run docker run -w /workspace -v $(pwd):/workspace node:12-alpine npm run test and it will hang
Run docker run -w /workspace -v $(pwd):/workspace node:12-alpine npm i && node node_modules/.bin/jest --testEnvironment=node --testTimeout=60000 and it will NOT hang

Note: the test will also fail in these steps. To get the test to pass, you need to run them on an AWS EC2. This is likely not required to debug
Steps to reproduce the behavior (with passing test):

clone https://github.com/clcpolevaulter/sts-example to EC2 with Docker installed
Modify test/sample.spec.js to have an accountNumber and roleName that the EC2 Instance Profile can assume here
Run docker run -w /workspace -v $(pwd):/workspace node:12-alpine npm run test and it will hang
Run docker run -w /workspace -v $(pwd):/workspace node:12-alpine npm i && node node_modules/.bin/jest --testEnvironment=node --testTimeout=60000 and it will NOT hang

Expected behavior
Jest test to exit with a status code of 0.
machine:sts-example bob$ npm run test

><EMAIL_ADDRESS>test /Users/bob/Developer/sts-example
> npm i && jest --testEnvironment=node --testTimeout=60000

npm WARN<EMAIL_ADDRESS>No repository field.

audited 580 packages in 2.503s

25 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

  console.log
    <redacted>

      at Object.<anonymous> (test/sample.spec.js:10:17)

 PASS  test/sample.spec.js
  STS Example
    ✓ This will hang in docker... (267 ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        0.876 s, estimated 2 s
Ran all test suites.
  console.log
    {
      AssumedRoleId: 'redacted',
      Arn: 'arn:aws:sts::<redacted>:assumed-role/<redacted>/axios'
    }

      at Object.<anonymous> (test/sample.spec.js:17:17)

Actual Behavior
Jest test doesn't exit.
><EMAIL_ADDRESS>test /workspace
> npm ci && jest --testEnvironment=node --testTimeout=60000

added 580 packages in 7.106s
  console.log
    <redacted>

      at Object.<anonymous> (test/sample.spec.js:10:17)

PASS test/sample.spec.js
  STS Example
    ✓ This will hang in docker... (1092 ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        1.781 s
Ran all test suites.
  console.log
    {
      AssumedRoleId: '<redacted>',
      Arn: 'arn:aws:sts::<redacted>:assumed-role/<redacted>/axios'
    }

      at Object.<anonymous> (test/sample.spec.js:17:17)

Jest did not exit one second after the test run has completed.

This usually means that there are asynchronous operations that weren't stopped in your tests. Consider running Jest with `--detectOpenHandles` to troubleshoot this issue.

Link to repl or repo (highly encouraged)
https://github.com/clcpolevaulter/sts-example
envinfo
WDXAB51793VCG:sts-example ab51793$ npx envinfo --preset jest
npx: installed 1 in 1.659s

  System:
    OS: macOS 10.15.7
    CPU: (16) x64 Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
  Binaries:
    Node: 12.19.0 - ~/.nvm/versions/node/v12.19.0/bin/node
    Yarn: 1.17.3 - /usr/local/bin/yarn
    npm: 6.14.8 - ~/.nvm/versions/node/v12.19.0/bin/npm
  npmPackages:
    jest: 26.6.3 => 26.6.3

I also observe this behaviour with this code being called:
/**
 * hasSTSSession
 * @returns bool
 */
const hasSTSSession = (_) => {
  let hasSession = false;

  const sts = new STS();
  sts.getCallerIdentity({}, function(err, data) {
    if (err) {
      hasSession = false;
    } else {
      hasSession = true;
    }
  });

  return hasSession;
};

If I comment the middle part jest stop complaining.


Web-console: Add continue to spec view

If the user has a spec in local host, upon return to  the load data view they will see this. Reseting the spec will prompt a modal then the user will be taken back to the welcome view, else they will go to connect view.

Thank you for the fixes! 👍


Create new stacking context for tiles

Refs: #8098
To create a new stacking context we need to isolate the container div (https://developer.mozilla.org/en-US/docs/Web/Guide/CSS/Understanding_z_index/The_stacking_context).
This fixes a visual glitch on the wrap-guide whenever there are more than 5 tiles (i.e. the wrap guide gets hidden). I am sorry for not catching this early, but apparently I overlooked it while testing #8098.
/cc: @nathansobo

Wow. I have never seen that CSS directive.

Wow. I have never seen that CSS directive.

Yeah, I had never used it as well but apparently its job is exactly to create a new stacking context. From https://developer.mozilla.org/en-US/docs/Web/CSS/isolation:

The isolation CSS property defines if the element must create a new stacking context.

position: absolute could have done the trick as well, but it feels more like an hack given that we don't really care about that <div> positioning.

:+1: Merge at will.


Implement entityCount for subgraphs so that it affects subscriptions

Do you want to request a feature or report a bug?
Enhancement (somewhere in the middle)
What is the current behavior?
The entityCount field is implemented as a computed field. Removing entities or adding new entities however does not trigger subscription updates, which we'll want.
What is the expected behavior?
The expected behavior is that, as the number of entities change, subscriptions for the "subgraph of subgraphs" are updated.
A potential solution would be to re-count the entities of a subgraph when applying entity operations—which should be fairly cheap—and then write the new value to an entityCount field in the subgraph entity.

Seems we opened the same issue twice, so I'll close the other one and move the report here:
In PR https://github.com/graphprotocol/graph-node/pull/602 we introduced the concept of computed fields, which are fields in a schema that are not stored, for the subgraph of subgraphs so we could implement entityCount. However because the field is not stored, it cannot be subscribed to.
Supporting subscriptions would require a special listener to the store. If this turns out to be difficult or not a priority, we could at least return an error if someone tries to subscribe to it.

Moving to approach were we store entityCount is an option, but we'd have to do that with database triggers because that changes not only when a mapping applies an operation but also when we handle reversions.

The changes we made to subscriptions fixed this.


Pause Rendering

Would it be possible to use SDL_WINDOWEVENT_HIDDEN and SDL_WINDOWEVENT_EXPOSED/SDL_WINDOWEVENT_SHOWN to pause rendering when the background isn't in view, and so save same cycles/battery?
I think it might work during the poll loop watching for the quit event, at a glance.

I had a look, but not much luck. Seems like the background is always visible according to SDL2

Hi,
Maybe you can take a similar approach to what Anipaper does: instead of dealing with SDL2 for this, use X11:
From time to time (in a separate thread or not), get the position of all non-minimized windows and calculate the area (with overlap, the Line Sweep algorithm does this) of all of them: if the occupied area reaches a certain threshold (say, 70% of the entire screen), the program pauses, otherwise resumes execution.
The main idea is that if the wallpaper is only partially visible (either through multiple windows covering the screen or through a full-screen window), the wallpaper does not need to be animated.
You can check the polling thread here and the calculation of the area used here.

Sorry to bother you on a relatively old issue, but I like paperview a lot, so I feel obligated to say something when I have something to contribute.
Anyway, paperview's performance is already amazing as it is, so I don't know if it's worth adding more complexity to the code.