The netopeer-server service is terminated

hi
Last night I had netopeer2 running on the server, but today I found that it was terminated.

I checked the coredump file and it was not caused by a crash，and I checked the history, it was not manually killed。Does netopeer2 provide any way to check how it was terminated?

On any abnormal termination you should see some additional messages, this seems like a standard termination, when the server receives SIGTERM.

Yes, nothing wrong happened, and I suspect it was accidentally killed by another process


[BUG] Exception ticking world : updateTimeLightAndEntities : updateWeather

[Please fill out the form below and delete the sections in square brackets after reading them]
Version of Advanced Rocketry
AdvancedRocketry-1.12.2-1.6.0-155-universal

Why is this important?
Knowing the version of Advanced Rocketry helps us narrow down an offending piece of code faster because we can easily determine what code is added and when.]

Have you verified this is an issue in the latest unstable build

[x] Y

Why is this important?
If you're not testing with the latest build, you may be encountering a bug that's already been fixed.
Reporting bugs that have already been fixed wastes everyone's time.  (please look in open issues too, somebody else could have already reported it!)

Version of LibVulpes
LibVulpes-1.12.2-0.4.1-60-universal

Why is this important?
Advanced rocketry depends on a library mod (LibVulpes), some errors can really be in that mod.  So knowing the verison helps us not only start looking for fixes there without you needing to anwer more questions, it also helps us know if you have a version mismatch.

Version of Minecraft
1.12.2

Why is this important?
Different versions of minecraft have different codebases for Advanced Rocketry.  Knowing the version of minecraft (and thus AR) helps us pinpoint issues quicker!

Does this occur without other mods installed

[x] N
Likely not, but each mod added for the test could take 20+ minutes to test, as the bug is not instant. Could be helpful to describe what types of mods could cause something like this.

Why is this important?
Sometimes interactions between mods cause issues.  While it can be a problem with Advanced Rocketry, sometimes it's not.  When there are a lot of mods, it can take bugtesters HOURS to go through the list to determine the root cause.  When there are a lot of bug reports, that can add DAYS to the time it takes to fix a large set of issues.

Crash report or log or visualVM (if applicable)
https://gist.github.com/Rsslone/79485674e66fb153b2d9b475dda5999c

If it's an issue involving lag, please submit a visualVM sampler snapshot.  Mods like lag goggles often do not provide the kind of information needed to debug lag.
For more information on how to create a snapshot, please see "How to Run CPU Sampling" here.  Then export the snapshot as an .nps.  You may have to zip it since github is weird, then please upload it here.

Why is this important?
Full crashlogs contain valuable information like where exactly in the code the error occurred, what other mods may be running and what version of forge you have.  This helps us replicate and diagnose the issue FAR faster.

Description of the problem
The problem occurs after the server is running for about 20 mins to 2 hours.

Why is this important?
If we cannot reproduce the problem, we probably cannot fix it at all.

crash-2019-07-22_14.06.06-server.txt
same

@cybmp3 and @Rsslone Have you tried it without sponge?  Sponge tends to mess with a lot of internals so a crash report with sponge in it tends to not be all that helpful.

No I have not @zmaster587 but I might give it a test. I was able to get v1.5.0 to work however if that helps.

Having the same problem on my forge (sponge) server on the latest version (161), all mods are working apart from this one

I think it is very important to be compatible with sponge.

Tested version 1.5.0 and, just like Rsslone, I'm no longer having issues.

@cybmp3 It's unfortunately not easy to be compatible with sponge when sponge changes a good deal of minecraft's internals compared to just forge alone.  A mod like Advanced Rocketry unfortunately relies on some internal things other mods don't often use due to the complex and unconventional nature of the mod.
Things like rearranging weather to have different effects in different dimensions beyond just normal rain and snow for instance.
Keeping the mod compatible with forge and forge alone, is a higher priority than sponge compat due to the fact it is more difficult and time consuming to guarantee support multiple platforms at once and time is unfortunately something I have been very short on the last few years.  If you're willing to help or know somebody willing to help maintain AR for sponge, by all means support is very welcome.  But at least while it's just me working on this project, bugs that affect the forge version or things that make the game difficult to play will have priority over sponge support.
@janutemi @Rsslone The reason this cropped up in 1.6 is because the weather system was not working properly prior to 1.6.  Rain and snow would occur in places it shouldn't because the code to disable the rain/snow on airless planets wasn't being run.  Unforunately, sponge has changed how the world its data in worldinfo.  worldinfo is responsible for controlling weather.  If I fix it for sponge, there's a good chance it'll break it for people NOT using sponge unless I start implementing a different solution for sponge and forge, effectively doubling the amount of code to maintain for this feature.

AdvancedRocketry-1.12.2-1.6.0-universal.jar.zip
Can @janutemi or @Rsslone report whether or not it still crashes with this build?

Alright I'm letting it run with the sent version, will reply in a few hours if it functions properly (or soon if it crashes).

Haven't experienced a crash or any issues, seems like the sent build works fine.


Speed up inference (and ensure ergodicity) in the presence of ENSURE DEPENDENT

by block proposing the dependent column cliques.
Apparently the current implementation strategy for ENSURE DEPENDENT is to still propose column moves one at a time, but zero out the probability of any that violate the constraints.  This means that a column that is DEPENDENT on another can never change views.

If there is only one connected component of such columns, this merely slows convergence (I think)
If there is more than one, inference becomes non-ergodic, because the two components will get stuck either dependent or independent during initialization, and never be able to change.

The better proposal mechanism is conceptually simple: just propose moving such a collection of columns as a group.  Actual implementation difficulty is unknown.

Looking into this issue. Thoughts:
Adjusting the CRP probabilities

For ENSURE DEPENDENT: the CRP probabilities associated with each view should be based on the number of constrained "column cliques" in the view, and not the raw number of columns. One can formalize this choice by considering each column clique to e a single "multivariate" column, which receives a single table assignment in the CRP prior

or ENSURE INDEPENDENT: the CRP prior is not longer exchangeable, and bad things start to happen using the vanilla CRP probabilities, zeroing-out any tables with a customer that violates an independence constraint, and then making sure the table probabilities are normalized.

Consider the following example with three customer c1, c2, and c3 with the constraint ensure independent (c1,c2):
Scenario A: sample in the order (c1, c2, c3)
Pr[c1=1] = 1
sample -> [[c1 =1]]
Pr[c2=1 | c1] = 0 (by independence constraint)
Pr[c2=2 | c1] = 1 (probabilities are normalized)
sample --> [[c2=2]]
Pr[c3=1 | c1,c2] = 1/(2+a)
Pr[c3=2 | c1,c2] = 1/(2+a)
Pr[c3=3 | c1, c2] = a/(2+a)
sample --> [[c3=3]]
==> Implies that the probability all customers are on separate tables is a/(2+a).
Scenario B: sample in the order (c3, c1, c2)
Pr[c3=1] = 1
sample --> [[c3=1]]
Pr[c2=1 | c3] = 1/(1+a)
Pr[c2=2 | c3] = a/(1+a)
sample --> [[c2=2]]
Pr[c1=1 | c2, c3] \propto 1
Pr[c1=2 | c2, c3] = 0 (by independence constraint)
Pr[c1=3 | c2, c3] \propto a
sample --> [[c1=3]]
==> Implies the probability all customers on separate tables = a/(1+a) * a/(1+a)
Someone interested in modeling can develop a ``constrained CRP'' model which tries to formalize independence constraints, but for the time being (and due to the title of this ticket) it makes sense to worry about the ENSURE DEPENDENT case which has a straightforward resolution.
Patching the inference kernel for block proposals
The heavy lifting will happen in State.cpp.
https://github.com/probcomp/crosscat/blob/25fad43cc08422314068c26e44c2c087bec07c17/cpp_code/src/State.cpp#L236-L244

Resolved by https://github.com/probcomp/crosscat/commit/3873eee2f3d32fddfe63e6fd01417b7f44fe38b9


CUDA accelerated propagation

This is a proof of concept addition of to see if CUDA FFTs would significantly accelerate the angular spectrum code. While that is the case, in the course of implementation and benchmarking (profiling with ipython %prun) I discovered significant bottlenecks were np.fft.fftshift() and np.exp().  These are also addressed in this branch, the former with a numba gpu implementation fftshift and the latter using the NumExpr library.
This obviously adds a lot of optional dependencies, but my philosophy was follow the fftw approach and gracefully revert to numpy everywhere, so each addition gives partial improvement. On my machine tests are passing with and w/o numba or cuda installed.
While further optimization is possible, for a toy case with 3 optical surfaces (below) generating a PSF with a fresnel wavefront class is accelerated by as much as a factor of 3:

Benchmark Optical system:

wavelen = 770e-9

D_prim = 2.37 * u.m
fr_pri = 7.8
fl_pri = D_prim * fr_pri

D_relay = 20 * u.mm
fl_m2 = fl_pri * D_relay / D_prim
fr_m3 = 20.
fl_m3 = fr_m3 * D_relay

def wfirst_sys(npix=1024,ratio=0.25):
    wfirst_optsys = poppy.FresnelOpticalSystem(pupil_diameter=D_prim, npix=npix, beam_ratio=ratio)

    m1 = poppy.QuadraticLens(fl_pri, name='Primary')
    m2 = poppy.QuadraticLens(fl_m2, name='M2')
    m3 = poppy.QuadraticLens(fl_m3, name='M3')
    m4 = poppy.QuadraticLens(fl_m3, name='M4')

    wfirst_optsys.add_optic(poppy.CircularAperture(radius=D_prim.value/2))
    wfirst_optsys.add_optic(m1)
    wfirst_optsys.add_optic(m2, distance = fl_pri + fl_m2)
    wfirst_optsys.add_optic(m3, distance = fl_m2 + fl_m3)

    #wfirst_optsys.add_optic(poppy.ScalarTransmission(planetype=poppy.poppy_core.PlaneType.image, name='focus'),
    #                        distance=fl_m3)

    wfirst_optsys.add_optic(poppy.ScalarTransmission(planetype=poppy.poppy_core.PlaneType.intermediate,
                                                 name='focus'), distance=fl_m3)
    return wfirst_optsys
wfirst_optsys4096=wfirst_sys(npix=1024,ratio=0.25)
%timeit wfirst_optsys4096.calcPSF(wavelength=wavelen, display_intermediates=False, return_intermediates=False)

Benchmark computer:
Google Compute Engine:

Machine type
n1-standard-8 (8 vCPUs, 30 GB memory)
CPU platform
Intel Haswell
GPUs
1 x NVIDIA Tesla K80

also, h/t to @neilzim for the benchmark prescription.
I will investigate the travis failure to build.

This is spectacular! I've been wanting to dig into all these performance optimization toolkits but just haven't had time. In addition to cuda and numexpr did you do any tests with numba?   As I recall, @josePhoenix had done some tests with Cython but wasn't able to get performance better than using a well-optimized numpy for the array operations.  That was in the Fraunhofer code rather than the Fresnel though.   I'm excited to see you got a 3x speedup using cuda!

Awesome stuff, @douglase! I remember last time I looked at this I was trying to target OpenCL, which alas seems to be dead in the water, and numba's CUDA support was still proprietary/paid-license only. Exciting to see that support has matured so much!

(much of the benchmarking for this project is recreated in this notebookhttps://gist.github.com/douglase/3846e8f105cd9baec96706681d0b8ee5

Wow, those benchmarks are really striking. Numexpr is way out in front, and for much less programming effort than the more manual numba vectorize and cuda versions.
I've tried a bit here in benchmarking and adding numexpr calls to parts of the Fraunhofer pathway too, and gotten some nice speedups there. But the real heavy lifting is still the calls to the numpy dot product, which doesn't fit easily into what numexpr can help with.


Applying improper list should throw error

BiwaScheme have same error that I just fix in LIPS:
(let ((x '(1 2))) (+ 1 . x))

this should throw error but it evaluate to 1. this is syntax error, you can have improper list only inside data, not inside code.

Another example that should fail to evaluate is this:
(let ((x '(1 2 . 3))) (apply + x))

this evaluates to 3.


[Android][Coverage]Can i use istanbul/nyc collect weex android project code coverage ?

Describe the bug
this may not be a bug
i am new to Weex.
Recently I used Istanbul to collect code coverage of weex android project.
like thie
fetch('http://<IP_ADDRESS>:8889/coverage/client', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
      },
      body: JSON.stringify(window.__coverage__)
    })
    .then(function () {
      native_logMessage('[Coverage] upload success')
    })

on Web side, it is worked.
but window is not used on android.
I would like to ask Weex if there are other ways for me to obtain coverage data

To Reproduce

Environment
2.0.0-beta.31 (Core 2.0.0-beta.31)

@weex-cli/generator : v2.0.0-beta.16
@weex-cli/run : v2.0.0-beta.21
@weex-cli/device : v2.0.0-beta.2

Expected behavior

Screenshots

Additional context

You can use weex instead of window . Or use storage.

storage

you mean, weex.coverage and storage.coverage ?

weex is a global variable.
storage is a weex module.


Misalligned Variable Declaration Bug !!

It still shows misalligned variable declaration with red color even though you fixed it. Had to run norminette manually to see its okay.

This is a known problem, but sadly we haven't been able to find its origin.
A workaround for this is closing and reopening the file.


Set 5000 addresses as default for calling eth_getLogs

0 (no limit) was set by mistake and had to be modified on the deployment via envvar

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
2 unchanged lines in 2 files lost coverage.
Overall coverage remained the same at 88.564%

Files with Coverage Reduction
New Missed Lines
%

safe_transaction_service/history/indexers/erc20_events_indexer.py
1
91.49%

safe_transaction_service/history/indexers/events_indexer.py
1
93.1%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
5909

Relevant Lines:
6672

💛  - Coveralls


Feat/move view type and drm in source

Summary
Refactor: move viewType & drm inside source prop.
This is an internal change, but API change are also exposed to UI.
Motivation
Avoid real time issue when drm and source are not changed in the same time.
As ViewType also need to be known when we start playback, It also make sense to move it in source
Changes
many refactor
change default surfaceType with SurfaceView
Test plan
Can be tested in sample.

@freeboub
I think ViewType is independent of source prop. Sure, DRM is enforcing Surface view, but I don't think it's a reason to go into the source prop.
I was wondering if there was a specific reason it was put in source prop.

@freeboub The unified prop make sense However I think viewType is independent of source prop. Sure, DRM is enforcing Surface view, but I don't think it's a reason to go into the source prop. I was wondering if there was a specific reason it was put in source prop.

The issue here is the following.

First keep in mind that all prop set in native part come in an asychronious way. It means that we can receive the source and isUseTextureView after. (also true for drm prop)
We need to know the surface type requested and if drm is used to start the playback. (I don't think it make sense to configure the surface before receiving src and drm props).
In order to configure the surface only once a playback start, it is a lot easier to merge this prop in the source.

In worst case currently we can have following behavior:

receive the source -> start playback (first View init as texture)
receive useTextureView = false or drm  -> (second view change as surface view)
receive useSecureView => switch the surface to secure (Notice that current the secure flag is never set to false ...)

I hope it is clarifying why I did that.
`But while i am writing, I am wondering:

if it could make sens to add another prop, something
preferedViewType={ViewType.SURFACE}
to define which should be the default viewType.
if a ViewType.NONE can make sense for audio only content ?

Let me know your throught please !
Thank you

@freeboub

In worst case currently we can have following behavior:
receive the source -> start playback (first View init as texture)
receive useTextureView = false or drm -> (second view change as surface view)
receive useSecureView => switch the surface to secure (Notice that current the secure flag is never set to false ...)
I hope it is clarifying why I did that.

I understand current situation, but it's not just a problem with viewType, it's a problem with bridge in react-native, so I think this will continue to generate a lot of keys in the source prop.
I think users will expect props to be passed in batches on a single render and won't understand why viewType depends on source.
I think it's more a matter of improving the tricky code in native, which is the current approach, So I propose 3 approaches.

keep counting the number of renders in JS, make single object and inject as prop with every single render (within interface layer that is not exposed to the user).

get all the props and render counts from JS side, queue them as a stream and process them (currently native does this with some delay, which is very buggy).

change the interface to only handle callbacks after the source is inserted and loaded (like expo-video or shaka-player)

It's not up to me to decide, but if you have any questions about my suggestions, please let me know.

if it could make sens to add another prop, something "preferedViewType={ViewType.SURFACE}" to define which should be the default viewType.

I prefer clear naming, expecting expected behavior to work, and getting an error if there's a problem.

if a ViewType.NONE can make sense for audio only content ? (I will try to add a none value, just to test the feasibility)

Does audio only content really need a view? If it doesn't, I think it should be quietly ignored. I think users with video content might misunderstand that if he insert ViewType.None, it automatically becomes audio only.

@freeboub

In worst case currently we can have following behavior:
receive the source -> start playback (first View init as texture)
receive useTextureView = false or drm -> (second view change as surface view)
receive useSecureView => switch the surface to secure (Notice that current the secure flag is never set to false ...)
I hope it is clarifying why I did that.

I understand current situation, but it's not just a problem with viewType, it's a problem with bridge in react-native, so I think this will continue to generate a lot of keys in the source prop. I think users will expect props to be passed in batches on a single render and won't understand why viewType depends on source.

Yes you are right, I identified following props which are necessary to start playback: https://github.com/TheWidlarzGroup/react-native-video/issues/2693#issuecomment-1291154625
Notice that I put some props on root as deprecated in root of <Video to let time for user to update there code.
But If you think it is not good, we can keep api as is.

I think it's more a matter of improving the tricky code in native, which is the current approach, So I propose 3 approaches.

keep counting the number of renders in JS, make single object and inject as prop with every single render (within interface layer that is not exposed to the user).

I don't understand 'keep counting the number of renders in JS'. but 'make single object and inject as prop with every single render (within interface layer that is not exposed to the user).' This is more or less what I did here. I moved original props inside the source. and only source will provide the values to the native component

get all the props and render counts from JS side, queue them as a stream and process them (currently native does this with some delay, which is very buggy).

Yes, this is buggy, and native code doesn't necessary know when it receives all props update...

change the interface to only handle callbacks after the source is inserted and loaded (like expo-video or shaka-player)

Yes it make sense, but it means we should first support every action as function calls. It will also be a big change for applications (I mean it make sense for V7 be cannot be done for V6). I think this is 2 different paths to progress.
Originally, the current api was make to match <Video tag in HTML5, that's why a lot of thing is handled with props... (It is my understanding)

It's not up to me to decide, but if you have any questions about my suggestions, please let me know.

We are here to discuss, It's good for me to have reviews, Thank you again !

if it could make sens to add another prop, something "preferedViewType={ViewType.SURFACE}" to define which should be the default viewType.

I prefer clear naming, expecting expected behavior to work, and getting an error if there's a problem.

let's forget this point :)

if a ViewType.NONE can make sense for audio only content ? (I will try to add a none value, just to test the feasibility)

Does audio only content really need a view? If it doesn't, I think it should be quietly ignored. I think users with video content might misunderstand that if he insert ViewType.None, it automatically becomes audio only.

The view is allocated for exoplayer by app. The aim of allowing to use none type is:

avoid creating unnecessary view when playing audio only content (decrease memory footprint of the app)
allow to play only audio of a video without video decoding (I hope so).

This none mode is not really necessary I think, but can be usefull for user how really optimization.

@YangJonghun thank you for the new review! I fix some of the comment.
But originally I didn't expect doing this big PR so soon.
Can we please focus on: https://github.com/TheWidlarzGroup/react-native-video/pull/3846/files please ?

@YangJonghun thank you for the new review! I fix some of the comment. But originally I didn't expect doing this big PR so soon. Can we please focus on: https://github.com/TheWidlarzGroup/react-native-video/pull/3846/files please ?

@YangJonghun I merged the other Pr, thank you for the review. I will splitt this PR for easier review (it is too big now I think)

@YangJonghun Just for information, I tested to change useTextureView during playback and it works fine. I will then split the PR in 2: 1 for the surface type fix, one other for the DRM object management. Thank you for the feedbacks anyway !

@freeboub
I've had experience capturing DRM-free video and needing the view type to be dynamic. However, this is not currently a feature of this library, so I don't think it's a problem.
Thanks for great work :)

@KrzysztofMoch @YangJonghun I clean up this PR, so it is ready to be merged on my side.
(no more viewType change)
I also add multiDrm flag to support keyrotation

Finally fix ios issue ...


Upgrade Log4j2 from 2.10.0 to 2.18.0

Fixes #2
Applied Backports:

https://github.com/apache/hive/commit/caf7ac0099645ac8500d824556941447e66e25e3
https://github.com/apache/hive/commit/c9e7f5dd6191636232921279acc1a5dd5a6fcaff

OK (cannot review)


bug: resource api

Bug Report
Ionic version:

[x] 4.x
Current behavior:
Hi, there is something get wrong with the resource api:

$ ionic cordova resources

Thanks for the issue. Please update your Ionic CLI. Very old versions of the CLI used an online version of resource generation. Newer versions of the CLI still have this, but the generation is all done on your machine.
npm uninstall -g ionic
npm i -g @ionic/cli

@liamdebeasi tks a lot


Truncating of resource gallery cards splits strings mid-word

The current string splitting method in the _extensions/yaml_gallery_generator.py cuts strings off at a certain length (with a recent fix in #147 to not break any html). This is done when the description for a resource is longer than a certain length, and a "...more" modal is added.
Ideally this string-splitting would be a little more sophisticated and not split strings mid-word, but either before or after the closest word to our desired length.

This is done with the merging of  #147.


Send confirmation modal UI issues

Send confirmation modal is too big.
Title for send confirmation always says " Sent successfully to contact" even when sending to an address or to own account.

-Verified this issue on dev app v1.5.0(243)


Corrected the project homepage

Looks like this was a copy-paste from loofah?

I've already merged #20 that corrected this! Thank you for submitting this patch, though!


es6 promises and .catch

.eventually.throw seems to not work right for es6 promises (Firefox 35?+).
var thrower = function () { throw new Error("oops") };
expect(Promise.resolve(true).then(thrower).eventually.throw()   // never called

Firefox's ES6 catchs all errors in then clauses to .catch on the promise.

That's how all promises work.
Can you show a full example I can run in the console (or better yet, on requirebin.com), preferably without any syntax errors?

Requirebin wasn't working well for me.
var thrower = function () { throw new Error("oops") };
expect(Promise.resolve(true).then(thrower)).eventually.throw()

I see the problem.
Chai's .throw() only works on functions. Thus, .eventually.throw() only works on promises for functions (i.e., promises fulfilled with functions).
You have instead a promise rejected with an Error. Since that isn't a promise fulfilled with a function, .throw() will not work.

So what is the right work around?

var thrower = function () { throw new Error("oops") };
expect(Promise.resolve(true).then(thrower)).eventually.rejected


Fails on GitHub Enterprise

No idea what the issue is. Does this create a git branch for you locally? Do you need one? Some error message would be helpful.
❯ npm run storybook:deploy

><EMAIL_ADDRESS>storybook:deploy /Users/kevinSuttle/Code/platform-ui-primitives
> storybook-to-ghpages

=> Building storybook
/Users/kevinSuttle/Code/platform-ui-primitives/node_modules/@kadira/storybook-deployer/src/utils.js:15
  throw new Error(message);
  ^

Error: Exec code(1) on executing: node ./node_modules/.bin/build-storybook -o .out
undefined
    at Object.exec (/Users/kevinSuttle/Code/platform-ui-primitives/node_modules/@kadira/storybook-deployer/src/utils.js:15:9)
    at Object.<anonymous> (/Users/kevinSuttle/Code/platform-ui-primitives/node_modules/@kadira/storybook-deployer/bin/storybook_to_ghpages:24:16)
    at Module._compile (module.js:541:32)
    at Object.Module._extensions..js (module.js:550:10)
    at Module.load (module.js:458:32)
    at tryModuleLoad (module.js:417:12)
    at Function.Module._load (module.js:409:3)
    at Module.runMain (module.js:575:10)
    at run (node.js:348:7)
    at startup (node.js:140:9)

npm ERR! Darwin 15.5.0
npm ERR! argv "/Users/kevinSuttle/.nvm/versions/node/v6.2.2/bin/node" "/Users/kevinSuttle/.nvm/versions/node/v6.2.2/bin/npm" "run" "storybook:deploy"
npm ERR! node v6.2.2
npm ERR! npm  v3.9.5
npm ERR! code ELIFECYCLE
npm ERR<EMAIL_ADDRESS>storybook:deploy: `storybook-to-ghpages`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>storybook:deploy script 'storybook-to-ghpages'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the @watson-iot/platform-ui-primitives package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     storybook-to-ghpages
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs @watson-iot/platform-ui-primitives
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls @watson-iot/platform-ui-primitives
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/kevinSuttle/Code/platform-ui-primitives/npm-debug.log

Weird. Are you trying to deploy into the public gh-pages?

May be check your git project is commitable. No conflicts, etc.

No, into a GHE server.

I'm getting an error on the same line, reproducible with https://github.com/lorensr/segmented-control
macbook:segmented-control me$  git:(upstream ⚡ master) 1A npm ls @kadira/storybook-deployer
<EMAIL_ADDRESS>/Users/me/gh/segmented-control
└──<EMAIL_ADDRESS>
macbook:segmented-control me$  git:(upstream ⚡ master) nr deploy-storybook

><EMAIL_ADDRESS>deploy-storybook /Users/me/gh/segmented-control
> storybook-to-ghpages

=> Building storybook
/Users/me/gh/segmented-control/node_modules/@kadira/storybook-deployer/src/utils.js:15
  throw new Error(message);
  ^

Error: Exec code(1) on executing: git commit -m "Deploy Storybook to GitHub Pages"
undefined
    at Object.exec (/Users/me/gh/segmented-control/node_modules/@kadira/storybook-deployer/src/utils.js:15:9)
    at Object.<anonymous> (/Users/me/gh/segmented-control/node_modules/@kadira/storybook-deployer/bin/storybook_to_ghpages:38:14)
    at Module._compile (module.js:397:26)
    at Object.Module._extensions..js (module.js:404:10)
    at Module.load (module.js:343:32)
    at Function.Module._load (module.js:300:12)
    at Function.Module.runMain (module.js:429:10)
    at startup (node.js:139:18)
    at node.js:999:3

npm ERR! Darwin 15.6.0
npm ERR! argv "/usr/local/Cellar/node/5.4.1_1/bin/node" "/usr/local/bin/npm" "run" "deploy-storybook"
npm ERR! node v5.4.1
npm ERR! npm  v3.5.3
npm ERR! code ELIFECYCLE
npm ERR<EMAIL_ADDRESS>deploy-storybook: `storybook-to-ghpages`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>deploy-storybook script 'storybook-to-ghpages'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the segmented-control package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     storybook-to-ghpages
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs segmented-control
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls segmented-control
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/me/gh/segmented-control/npm-debug.log

@lorensr let me check this now.

I just fixed this with a new version v1.1.0. Try to reopen or (asked me) if this is not yet fixed.

http://lorensr.me/segmented-control/
works now, thanks arunoda! 😊

Hey @kevinSuttle be sure that storybook is located in .storybook folder otherwise it will fail.
@arunoda I think it should show a more helpful error message. Thanks

Thanks @gusdaud. @arunoda is it possible to run this as a webhook on the repo?

@kevinSuttle Not sure. If you are using CI I think it's possible.
May be you need to make sure, CI can push to GitHub.

I have faced the same issue, and I found the solution.

Note: you can require the root babel config using babel.config.js file.
For more info please check https://storybook.js.org/docs/configurations/custom-babel-config/#docs-content


Update glslang to 219b025 and use preinstalled glslangValidator if available

This updates glslang to https://github.com/KhronosGroup/glslang/commit/219b025d7eae1383770c89247e6c5eb1909ac188 to fix a compile error.
In addition the build.rs will use a preinstalled glslangValidator if found. This part might not be portable to android? I unfortunately have no experience there.

There are two reasons why I'm hesitating about this:

The PR doesn't include the precompiled windows binary, which I included because it can be tedious to build. But I'm hesitating about what to do with that binary.
glslangValidator is supposed to be an implementation detail, so I'm not sure it's a good idea to try to pick the program from the PATH. For example it could be annoying if shaders compiled differently depending on whether or not you have glslangValidator in your PATH.

No sweat :)
It's just what I did to prevent the lengthy and erroneous glslang build as I (with nixos) hat it in my $PATH. Your second reason sound good to just deny including the $PATH logic.

glsl-to-spirv is no more. https://github.com/vulkano-rs/vulkano/pull/947


Fix Info for composer 1

Q
A

Is bugfix?
✔

New feature?
❌

Breaks BC?
❌

Fixed issues
-

👍

👍


Service AnsibleTower and EmbeddedAnsible UI parity

Make the service layout same for AnsibleTower as it is for EmbeddedAnsible - Core part

Extract :raw_stdout_via_worker to shared Ansible codebase
Make the :job signature the same in service_ansible_tower.rb and service_ansible_playbook.rb

Depends on: https://github.com/ManageIQ/manageiq-providers-ansible_tower/pull/108
Other related PR: https://github.com/ManageIQ/manageiq-ui-classic/pull/4299
Fixes: BUG 1590840 BUG 1590844

@miq-bot add_label enhancement, automate, providers/ansible_tower

@bzwei since you reviewed the related https://github.com/ManageIQ/manageiq-providers-ansible_tower/pull/108, can you take a look at this as well?

@gmcculloug the dependent https://github.com/ManageIQ/manageiq-providers-ansible_tower/pull/108 is merged.


[HigherOrLower] Cog Rework

Hello for anyone who's wondering. I will be moving HigherOrLower to a seperate repository. This is because I still think people can use it, but it does need a rework in the code itself. This links in with #39, too.

Please move over to my separate repository for further information on higherorlower, thankyou.


Reload config(via USR1) has no effect after upgrading(via USR2)

1. What version of Caddy are you using (caddy -version)?
Caddy 0.11.0 (+1f7b5ab Thu Jul 26 10:48:36 UTC 2018) (unofficial)
2. What are you trying to do?
testing USR1 feature after USR2, then watch the log output
pkill -usr1 caddy
pkill -usr2 caddy
pkill -usr1 caddy

3. What is your entire Caddyfile?
it's empty

4.  Please paste any relevant HTTP request(s) here.
no request
6. What did you expect to see?
2018/07/26 18:54:31 [INFO] SIGUSR1: Reloading
2018/07/26 18:54:31 [INFO] Reloading
2018/07/26 18:54:31 [INFO] Reloading complete
7. What did you see instead (give full error messages and/or log)?
2018/07/26 18:55:33 [INFO] SIGUSR1: Reloading
2018/07/26 18:55:33 [ERROR] SIGUSR1: no Caddyfile loader with which to reload Caddyfile
8. How can someone who is starting from scratch reproduce the bug as minimally as possible?
I dig a little into the source code, it seems that before upgrade, the caddy use Caddyfile to trigger reload action, however, in the upgrading process, the caddy config will be piped(via pipe()) to the child(in the memory), then the Caddyfile is useless after that.(the loaderUsed.loader is no longer take effect after then)

Well... turns out, config reloads are totally rearchitected in Caddy 2, making this a non-issue.


How to determine if the query was served by a specific shard using AQL Explain?

Hi,
We have a sharded deployment of our ArangoDB and wanted to know how do I determine if a query utilized the shard key to direct the query to a specific shard using AQL explain in case of a) collection b) ArangoSearch view ?

For a simple query like FOR doc IN s FILTER doc._key == "12345" RETURN doc, the profiling output looks like this:
Execution plan:
 Id   NodeType                    Site  Calls   Par   Items   Filtered   Runtime [s]   Comment
  1   SingletonNode               COOR      1     -       1          0       0.00000   * ROOT
  7   SingleRemoteOperationNode   COOR      1     -       1          0       0.00072     - FOR doc IN s FILTER doc.`_key` == "12345" /* primary index scan */
  5   ReturnNode                  COOR      1     -       1          0       0.00000     - RETURN doc

Indexes used:
 By   Name      Type      Collection   Unique   Sparse   Cache   Selectivity   Fields       Stored values   Ranges
  7   primary   primary   s            true     false    false      100.00 %   [ `_key` ]   [  ]            "12345"

Optimization rules applied:
 Id   Rule Name                                           Id   Rule Name
  1   use-indexes                                          3   remove-unnecessary-calculations-2
  2   remove-filter-covered-by-index                       4   optimize-cluster-single-document-operations

Query Statistics:
 Writes Exec      Writes Ign      Doc. Lookups      Scan Full      Scan Index      Cache Hits/Misses      Filtered      Requests      Peak Mem [b]      Exec Time [s]
           0               0                 0              0               1                  0 / 0             0             0                 0            0.00106

The optimize-cluster-single-document-operations rule is active and you don't find any nodes in the execution plan with DBS in the Site column, respectively no REMOTE/GATHER operations. Calls and Items are 1. The Requests count under Query Statistics is 0.
The explain output doesn't include Calls, Items, and Requests, but it does show you which optimizer rules are active and you can see whether the execution plan contains REMOTE/GATHER nodes.
The optimizer rule is not eligible if you filter for two different documents, like FOR doc IN s FILTER doc._key == "12345" OR doc._key == "54321" RETURN doc (not a single document operation). The profile shows REMOTE/GATHER, several Calls, and Requests is e.g. 9 in case of a collection with 3 shards, which tells you that all DB-Servers were at least contacted once.
Execution plan:
 Id   NodeType        Site  Calls   Par   Items   Filtered   Runtime [s]   Comment
  1   SingletonNode   DBS       3     -       3          0       0.00001   * ROOT
  7   IndexNode       DBS       3     0       2          0       0.00033     - FOR doc IN s   /* primary index scan, index scan + document lookup, 3 shard(s) */
 10   RemoteNode      COOR      9     -       2          0       0.00005       - REMOTE
 11   GatherNode      COOR      4     -       2          0       0.00005       - GATHER   /* parallel, unsorted */
  5   ReturnNode      COOR      4     -       2          0       0.00001       - RETURN doc

Indexes used:
 By   Name      Type      Collection   Unique   Sparse   Cache   Selectivity   Fields       Stored values   Ranges
  7   primary   primary   s            true     false    false      100.00 %   [ `_key` ]   [  ]            (doc.`_key` IN [ "12345", "54324" ])

Optimization rules applied:
 Id   Rule Name                                 Id   Rule Name                                 Id   Rule Name
  1   replace-or-with-in                         4   remove-unnecessary-calculations-2          7   parallelize-gather
  2   use-indexes                                5   scatter-in-cluster                         8   async-prefetch
  3   remove-filter-covered-by-index             6   remove-unnecessary-remote-scatter

Query Statistics:
 Writes Exec      Writes Ign      Doc. Lookups      Scan Full      Scan Index      Cache Hits/Misses      Filtered      Requests      Peak Mem [b]      Exec Time [s]
           0               0                 2              0               2                  0 / 0             0             9                 0            0.00330

The rule is also not applicable when you query a View, even when asking only for a single document.

When using custom shard keys (not _key but let's say sk), you can observe that the query explain output tells you the shard if you ask for a single shard key like FOR doc IN t FILTER doc.sk == 4 RETURN doc (shard: s8010016):
Execution plan:
 Id   NodeType                  Site  Par     Est.   Comment
  1   SingletonNode             DBS              1   * ROOT
  2   EnumerateCollectionNode   DBS     ✓   100000     - FOR doc IN t   /* full collection scan, shard: s8010016  */   FILTER (doc.`sk` == 4)   /* early pruning */
  8   RemoteNode                COOR        100000       - REMOTE
  9   GatherNode                COOR        100000       - GATHER   /* parallel, unsorted */
  5   ReturnNode                COOR        100000       - RETURN doc

When asking for multiple shards, like FILTER doc.sk IN [1, 4], all shards are contacted, however, even if the shard keys belong to a single shard (3 shard(s)):
Execution plan:
 Id   NodeType                  Site  Par     Est.   Comment
  1   SingletonNode             DBS              1   * ROOT
  2   EnumerateCollectionNode   DBS     ✓   100000     - FOR doc IN t   /* full collection scan, 3 shard(s)  */   FILTER (doc.`sk` IN [ 1, 4 ])   /* early pruning */
  8   RemoteNode                COOR        100000       - REMOTE
  9   GatherNode                COOR        100000       - GATHER   /* parallel, unsorted */
  5   ReturnNode                COOR        100000       - RETURN doc

You can also observe the difference in the profiling output by looking at the Calls and Requests, although you should add a LIMIT 1 to either get 3 or 9 requests (without, you get some higher number that is dependent on the number of documents, possibly also the batchSize, etc. and you would need to compare the stats).
Using an arangosearch View, search-alias View, or an inverted index, the profiling output shows similar numbers for requests even if asking only for a single shard key and document, so all shards appear to be contacted (for the inverted index, it does show 3 shard(s)).


Have SSDPDiscover generic again

Reapply changes overwritten by #602 to have SSDPDiscovery functionality generic and not for hyperion only.

I removed it because of the log spam which is probably irritating to a user, in consider the specific search target more as debug.
You can rename it to just 'server'
This does not affect custom search targets as implementation.
Q: This is part of Hyperion, what do you mean by "other" search targets? For LedDevices?

Or a custom log string is possible to modify output more user friendly

Q: This is part of Hyperion, what do you mean by "other" search targets? For LedDevices?

Yes, I reused the SSDPDiscover functionality to discover a network LedDevice (here Nanoleaf) and therefore made it more generic.
[hyperiond SSDPDISCOVER] <INFO> Search for Service [nanoleaf:nl29]
...
[hyperiond SSDPDISCOVER] <INFO> Found service [nanoleaf:nl29] at: <IP_ADDRESS>:16021
[hyperiond LEDDEVICE] <INFO> Nanoleaf device discovered at [<IP_ADDRESS>:16021]

I agree to your point that a ssdp service name is not necessarily too user friendly for a standard INFO log.. Shouldn't we change the INFO message to DEBUG messages, as the discovery is quite technical?
The calling function may then decide, if an INFO message is relevant or not (see above).

Ready to merge?

Yes. Please go ahead.


Improvement improve java class loader usage

Proposed changes

Fix create function failure when two functions with the same jar
Avoid to add classpath each time for the same jar.

Problem summary
Describe your changes.
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[ ] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[ ] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[ ] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[ ] No

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 33.64 seconds
load time: 452 seconds
storage size:<PHONE_NUMBER>2 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221104054936_clickbench_pr_39809.html


Add Resque.

What's this PR do?
This adds minimal Resque metrics integration to Datadog.
Motivation
Internal tools asked!
Notes
There's more we could do here, but I only did the minimum that was easy since that's what @preinheimer asked for. We can add more and iterate if we like!
How To Deploy

Merge
Add to our internal puppet, etc.

How to Measure Success
Look for resque.worker_count!
How to Rollback
Revert, merge, repeat How To Deploy
cc @preinheimer

It works!
2016-05-26 20:47:34,341 | WARNING | dd.collector | checks.disk(disk.py:76) | Using `use_mount` in datadog.conf has been deprecated in favor of `use_mount` in disk.yaml
2016-05-26 20:47:34,357 | INFO | dd.collector | config(config.py:899) | initialized checks.d checks: ['network', 'stripe_consul', 'process', 'ntp', 'resque', 'linux_proc_extras', 'dns_check', 'linux_vm_extras', 'file', 'redisdb', 'disk']
2016-05-26 20:47:34,357 | INFO | dd.collector | config(config.py:900) | initialization failed checks.d checks: []
2016-05-26 20:47:34,357 | INFO | dd.collector | checks.collector(collector.py:533) | Running check resque
Metrics:
[('resque.jobs_failed',
<PHONE_NUMBER>,
  '5',
  {'hostname': 'qa-supportbox1.northwest.stripe.io', 'type': 'gauge'}),
 ('resque.queues_count',
<PHONE_NUMBER>,
  1,
  {'hostname': 'qa-supportbox1.northwest.stripe.io', 'type': 'gauge'}),
 ('resque.worker_count',
<PHONE_NUMBER>,
  1,
  {'hostname': 'qa-supportbox1.northwest.stripe.io', 'type': 'gauge'}),
 ('resque.jobs_processed',
<PHONE_NUMBER>,
  '20',
  {'hostname': 'qa-supportbox1.northwest.stripe.io', 'type': 'gauge'})]
Events:
[]
Service Checks:
[]
Service Metadata:
[{}]
    resque
    ------
      - instance #0 [OK]
      - Collected 4 metrics, 0 events & 0 service checks

Huzzah!
:shipit:

Looks great to me - I think we'll want to add that service check, but
LGTM
As a quick win!

Agreed. I'll make sure and follow up with service checks, tests and other iterative improvements.


Add release description to README.

Removed documentation that is available on docs.asciidoctor.org from the README.
Thank you for opening a pull request and contributing to AsciidoctorJ!
Please take a bit of time giving some details about your pull request:
Kind of change

[ ] Bug fix
[ ] New non-breaking feature
[ ] New breaking feature
[X] Documentation update
[ ] Build improvement

Description
What is the goal of this pull request?
Adding the steps to build and publish a new release to Maven central.
Additionaly I removed content from the README that is also available on the Antora site.
Imo this is too detailed for a README anyway, and duplication rather results in diverging information.
I also updated the docs for how to build in the Antora module, I wonder though if this rather only belongs in the README.
@abelsromero @mojavelinux Would be great if you could skim over this.
How does it achieve that?
Are there any alternative ways to implement this?
Are there any implications of this pull request? Anything a user must know?
Issue
If this PR fixes an open issue, please add a line of the form:
Fixes #Issue
Release notes
Please add a corresponding entry to the file CHANGELOG.adoc

Additionaly I removed content from the README that is also available on the Antora site.
Imo this is too detailed for a README anyway, and duplication rather results in diverging information.

Yes, that is the direction we want to head. We haven't yet completed a style and structure guide for projects, but when we do it would cover this topic. And we can always adjust if necessary when the time comes. So I vote to proceed!


How to make it work without using RxSwift?

Thanks again and sorry to ask simple question. how to make it work without RxSwift?
https://gist.github.com/steve21124/2de1d70850014075dc518d5b3af5835b

after checking it out again, finally able to make it work without RxSwft
https://gist.github.com/steve21124/4ef36081242b1f96d4d00ea6c0188f58


Use of s inside  causes undesirable formatting.

Bug description:
Recent changes seem to have rendered all <spans> inside a <ui-match> at 100% causing the first such span to push everything way to the right (out of the container).  See plunker below.  There do not seem to be any quick and easy work-around.  Seems to have been caused by PR #1680 in response to #1576 .  See also note added to 01055c5b
Link to minimally-working plunker that reproduces the issue:
https://plnkr.co/edit/yrOYc0w54E60zSk0w7HE?p=preview
Version of Angular, UI-Select, and Bootstrap/Select2/Selectize CSS
Angular: 1.5.x
UI-Select: 0.19.3
Bootstrap/Select2/Selectize CSS (if applicable): 3.3.6

I take it this hasn't been fixed yet?

@starf  no it is not, maybe you could have a look? Happy to review your PR.

@Jefiozie Thanks for the response but i'm afraid i'm not knowlegeable enough to touch that code. Have a good day.

Any news for this issue?
This prevents us from upgrading and currently each version has a bug that impacted us:

#1609 that prevents us to use 1.17.1 (seems fixed in 1.19.8)
#1822 and #1832 that prevents us to use 1.19.4 -> 1.19.? (seems fixed in 1.19.8)
this one (#1826) that prevents us to use 1.19.3  -> not fixed


Could not resize files undefined on Safari 5.1

I have some issues on Safari 5.1 with single file upload.
<input accept="image/*" class="btn" name="foto" ng-model="picFile" ng-required="!user.fotos.length > 0" ngf-fix-orientation="true" ngf-min-height="300" ngf-min-width="300" ngf-resize-if="$width > 1200" ngf-resize="{width: 1200, quality:0.9}" ngf-select="uploadMedia($file, 'fotos')" tabindex="1" type="button" value="+ Add photo">

I get the following error;
Could not resize files undefined
As far as I can see, Safari 5.1 supports HTML5 Form data and should not need shim.js or am I wrong?

Post on stack overflow. Hard to get answers on this stuff.

Does the same thing work on other browsers?

Yes, it does. The problem exists on both Windows and Mac Safari.

There is another similar issue with a workaround.

Could you verify if there is more info on this error with version 12.0.3?
I made it so that it returns the original error so we might have a clue what is going wrong.

Hi Daniel,
I just checked it out and I don't see any additional info.

However, if there is no easy fix for it, let's just close this issue :) Safari 5.1 was a long time ago..

I could probably add it to the criteria for not supported resize. Does it fail for all images?

Yes, it does.


Exante Brokerage initial setup

Description
Exante initial setup
Requires Documentation Change
No documentation changes require
How Has This Been Tested?
existing tests pass
Types of changes

[ ]  Bug fix (non-breaking change which fixes an issue)
[ ]  Refactor (non-breaking change which improves implementation)
[ ]  Performance (non-breaking change which improves performance. Please add associated performance test and results)
[x]  New feature (non-breaking change which adds functionality)
[ ]  Breaking change (fix or feature that would cause existing functionality to change)
[ ]  Non-functional change (xml comments/documentation/etc)

Checklist:

[x]  My code follows the code style of this project.
[x]  I have read the CONTRIBUTING document.
[x]  I have added tests to cover my changes.
[x]  All new and existing tests passed.
[x]  My branch follows the naming convention bug-<issue#>-<description> or feature-<issue#>-<description>

@Martin-Molinero Hi Martin!
Please, check the updates.

@Martin-Molinero Martin, please also merge the commit https://github.com/alexander-myltsev/Lean/commit/c27bb50cc7c11d94bb71966e6172e4a2d8e85623 at the same branch


Display 'F1' as the shortcut key for Toggle Developer Tools

Storage Explorer Version: 1.13.0
Build: 20200423.5
Branch: rel/1.13.0
Platform/OS: macOS Catalina (10.15.2)
Architecture: x64
Regression From: Not a regression
Steps to reproduce:

Launch Storage Explorer -> Go to Help.
Check the shortcut key for Toggle Developer Tools.

Expect Experience:
Display ’F12‘ as the shortcut key for Toggle Developer Tools.

Actual Experience:
Display 'F1' as the shortcut key for Toggle Developer Tools.

More Info:

This issue only reproduces in English and Dutch(Netherlands).
This issue doesn't reproduce on Mac OS Mojave (10.14.6).

Possibly an electron bug. I'll see if this repro's in electron 8.

@MRayermannMSFT
Verified this issue on build master/ 20200512.2. Not reproduce. So we close it.


Remaining Centering Issues

https://svgjs.com/docs/3.0/shape-elements/#gradient-constructor
https://alligator.io/svg/radial-gradients/

Only shows up on uneven levels
Maybe rotation or move can help?


Let users join confs as voice or video

Vector web shows in the banner: Ongoing conference call. Join as voice or video.

as per https://github.com/matrix-org/matrix-react-sdk/commit/fdfd6ecfd9d6890e90c4931ac0afb90627f286b6


Gfeitosa/increase coverage include int

Contributing to the Azure SDK
Please see our CONTRIBUTING.md if you are not familiar with contributing to this repository or have questions.
For specific information about pull request etiquette and best practices, see this section.

/azp run net - Azure.Communication.Chat - tests

/azp run net - Azure.Communication.Sms - tests

API change check
API changes are not detected in this pull request.

/azp run net - Azure.Communication.Sms - tests

/azp run net - Azure.Communication.Sms - tests

/azp run net - Azure.Communication.Sms - tests

/azp run net - Azure.Communication.Sms - tests

/azp run net - Azure.Communication.Sms - tests

/azp run net - Azure.Communication.Sms - tests

/azp run net - Azure.Communication.Sms - tests

/azp run net - Azure.Communication.Sms - tests


refactor: remove root

removes the root module entirely.

/rebase

ugh autorebase doesn't work for forked :/


update breakingchange label config

Packages impacted by this PR
Issues associated with this PR
Describe the problem that is addressed by this PR
What are the possible designs available to address the problem? If there are more than one possible design, why was the one in this PR chosen?
Are there test cases added in this PR? (If not, why?)
Provide a list of related PRs (if any)
Command used to generate this PR:**(Applicable only to SDK release request PRs)
Checklists

[ ] Added impacted package name to the issue description
[ ] Does this PR needs any fixes in the SDK Generator?** (If so, create an Issue in the Autorest/typescript repository and link it here)
[ ] Added a changelog (if necessary)

See this gist for the details of label definitions.
https://gist.github.com/raych1/353949d19371b69fb82a10dd70032a51

/check-enforcer override


Bazel: It should be possible to disable the disk cache

The user can configure the usage of a disk cache in the .bazelrc file:
common --disk_cache=/home/dockeruser/.cache/bazel_disk_cache/

The pointed directory can be outside of the repository due to a specific environment.
When the pointed directory does not exist, ORT crashes on any bazel command, even bazel --version:
Exception in thread "main" java.io.IOException: Running 'bazel version' in '/test/' failed with exit code 2:
ERROR: /home/dockeruser/.cache/bazel_disk_cache (Permission denied)
ERROR: Error initializing RemoteModule

The workaround (taken here) is to pass --disk_cache= as extra parameter to the Bazel commands run by ORT.
TBD if it should be configurable or passed every time.
@haikoschol FYI

Instead of adding a parameter, would it be an option that ORT simply takes care of creating the directory in case it does not exist?

I've just tried to reproduce the issue, by deleting .cache/bazel_disk_cache inside docker and then run the bazel fun test. The tests succeeds. So, I wonder if the issue description is correct.
Could the problem be a permission issue rather than the non-existance of the dir?

and then run the bazel fun test.

But did you test with the .bazelrc configuration described above ? I am not sure the disk cache is enabled by default.

would it be an option that ORT simply takes care of creating the directory in case it does not exist

Creating this directory works but I don't see how ORT can get the name of this directory? By parsing the .bazelrc file?
Don't forget that some projects can use a custom Bazel RC e.g. bazel --bazelrc=my_custom_bazelrc.

But did you test with the .bazelrc configuration described above ? I am not sure the disk cache is enabled by default.

Oh no, I missed that this step is required.
In my view, it could be a good solution to always overwrite any disk cache directory configured in .bazelrc.
If tested this and it avoid running into the error. For example, the following would work even if the
disk cache dir specified in .bazelrc does not exist:
mkdir -p ~/.cache/ort/bazel/disk-cache
bazel mod graph --disk_cache ~/.cache/ort/bazel/disk-cache

Maybe a better directory could be choosen.

One additional question I just came across is whether the content of the cache dir should be re-used across scans on CI.
We have ~/.ort/cache/analyzer which I believe is normally re-used. In this case (as its a build cache) it may be better to create a new empty temp directory, and use it only for a single ORT analysis process. What do you think ?

Yes I agree: better to use an empty temp directory for disk cache.
However I am not sure the commands run by ORT will fill the cache: these are not doing "a real build", no ?

However I am not sure the commands run by ORT will fill the cache: these are not doing "a real build", no ?

I'm not into this implementation deeply, but I do recall from the code review that there may be a build involved.

Thanks for the fix @fviernau!


awi-ciroh: proposing a node sharing setup for awi-ciroh

awi-ciroh now have a UI with small / medium / large / huge in the profile list, just like the legacy 1:1 user:node profile list options.
But, as of https://github.com/2i2c-org/infrastructure/pull/2539, they will just start on the n2-highmem-16 machine no matter what with a pod requesting memory based on the legacy 1:1 user:node profile list options.
I'll reach out to suggest aligning their setup to a typical node sharing setup, perhaps with the small + medium options.
Looking at the events memory/cpu usage in grafana.

I'm observing a max use of 14 GB memory, and for many users high as ~12 GB, and a notable CPU usage over time - not just 10 second burst but 30 minutes ~100% CPU use etc.
Based on typical usage, it seems that there is very often ~1-4 users on the hub:

With that in mind, I think it would make sense to use a small node by default (4 CPU, 32 GB), but since memory use may go beyond 32 GB, also provide a medium node (16 CPU, 128GB).
I think the small node should default on a request at ~8 GB of memory, allowing 4 users to share it. Like this, we probably arrive at a good user experience that is cost efficient and reliably, also helping users get quick startup times etc.

They are node sharing atm, so closing.


adjustable kmer size

I am using quite long illumina-quality like reads for a plant genome as the short reads input. However the correction on whole genome scale does not work accurate enough to really make a difference. My long reads are corrected only partially. One thing I am thinking is that since my short reads are quite large, upto a kilobase, and therefore the debruijn graph could be build using larger kmer sizes that de default 31/63 (?) combo. I think the 63 if is fine for mapping the long reads back, but the illumina kmer size for building the graph could be bigger. Any chance this can be increased? (either by me changing the source code or via a parameter). Thanks so far for this great software package! A question related, can I read from the logs also something on the unitig size that is constructed? That might give me a hint already. I can image that for human genome this is quite a bit bigger than for a repetitive plant.

Hi @HenrivdGeest,
Providing advanced options to modify k1 and k2 individually would not be an issue. Something to consider is that the size of the k-mer used to build the graph is also the size of the matches between the graph and the long reads. So if you would pick k1>31, this would definitely increase the contiguity of the graph on the first correction pass but it would decrease the probability of finding shared k1-mers between the graph and the long reads. It is not necessarily a problem though: You would get less matches for the first correction pass but the better graph contiguity would probably make up for it.
You could also consider 2-3 runs of Ratatosk where the output corrected reads of run x are the input long reads of run x+1. You could have k1/k2=31/63 on the first run, then k1/k2=95/121 on the second run, etc.
Finally, something to consider is the insert size. By default, Ratatosk uses 500bp as short read insert size but given the length of your reads, you might want to increase that to 1kbp. I can also add this as an advanced parameter.
Let me now if any of this is of interest to you.

Hi, thanks, I think both options would be nice. I do like your suggestion on doing it with a 2 pass, to circumvent that reads might be missed in the first pass. I would disable trimming for the first pass(es) and enable it for the last pass. I need to enable it because I see many chimeric long reads in my set, which really hamper the assembler (hifiasm)
Another thing, my short read data is single end. I already cut the reads in 300bp reads, but that gave worse performance. So I am using the reads as single reads of around ~1kb.
I am also wondering what happens with a part of a read which is highly repetitive. Will it be polished by the consensus of the repeat or will it not be polished?

You might be interested in this: https://github.com/chhylp123/hifiasm/issues/33
Also, BBMap/BBTools (https://sourceforge.net/projects/bbmap/) has a chimeric read detector called icecreamfinder.sh that might help (not sure).

Alright, I'm going to work on that today and hopefully, it will be available by the end of the day. Regarding the length of your "short" reads, Ratatosk considers paired-end reads as well as single end reads so I would definitely keep your short reads the way they are for better contiguity and color information in the graph. In the README file, I'm writing that Ratatosk works best with paired-end reads only for the general case where it is expected the length of the Illumina reads are 150bp +/- 100bp.
Regarding your repeat question, it all depends on the number of matches found between the graph and the long read in the repeat region. A repeat is going to create cycles and/or very branching subgraphs so Ratatosk needs enough k-mer matches (exact or inexact) in that region to get the length of the repeat right. That's something that we want to improve for the next release by giving the repeats special treatment.

Hi @HenrivdGeest and @jelber2 ,
I have pushed a new version of Ratatosk that comes with a few new advanced options. As we discussed, there are 3 options of interest for you:
   -i, --insert_sz                 Insert size of the input short reads (default: 500)
   -k, --k1                        Length of short k-mers for 1st pass correction (default: 31)
   -K, --k2                        Length of long k-mers for 2nd pass correction (default: 63)

In your case, I would use -i 1000 and set -k/-K with whatever lengths you want to play with. If you want to use  -k/-K larger than 63, you need to recompile Ratatosk with an extra compile option. It's very easy and all described here. Note that if you decide to do so, I would advise to make a new fresh clone of Bifrost (delete your local version and pull the new one) because my cmake install could use a bit of work.
Finally, I haven't tested changing any of those advanced parameters so let me know if it works and if you see any improvements.

@GuillaumeHolley great! I look forward to comparing --k1 31 and --k1 41 and its impact on assemblies!

Let me know how it goes :)

Comparison of simulated PacBio CLR reads corrected with Illumina for a Drosophila melanogaster genome assembly (haploid assembly "converted" to diploid with ~2% heterozygosity).
Using Ratatosk commit 9e247317
# Use Drosophila melongaster PacBio assembly
cd /genetics/elbers/test/fly2
wget https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/003/401/745/GCA_003401745.1_ASM340174v1/GCA_003401745.1_ASM340174v1_genomic.fna.gz

# Use BCFtools 1.10.2 and SAMtools 1.10
conda activate bcftools1.10.2

# Use Seqtk to convert soft-masked bases to upper-case bases, also compress with bgzip
# https://github.com/lh3/seqtk
/genetics/elbers/bin/seqtk/seqtk seq -U \
GCA_003401745.1_ASM340174v1_genomic.fna.gz | \
bgzip -@75 > GCA_003401745.1_ASM340174v1_genomic.fna_upper.fasta.gz

# Convert to diploid with approximately 2% heterozygosity rate, max indels=20bp
# mutate.sh part of BBTools/BBMap https://sourceforge.net/projects/bbmap/
/genetics/elbers/bbmap-38.86/mutate.sh \
in=GCA_003401745.1_ASM340174v1_genomic.fna_upper.fasta.gz \
ow=t \
vcf=GCA_003401745.1_ASM340174v1_genomic.fna_upper.diploid.vcf.gz \
out=GCA_003401745.1_ASM340174v1_genomic.fna_upper.diploid.fasta.gz \
ziplevel=6 \
ploidy=2 \
subrate=0.0192 \
indelrate=0.001 \
maxindel=20 \
nohomopolymers=t \
hetrate=1 2> GCA_003401745.1_ASM340174v1_genomic.fna_upper.diploid.fasta.log.txt

# genome size
bgzip -@75 -cd GCA_003401745.1_ASM340174v1_genomic.fna_upper.fasta.gz | \
grep -v ">"|wc -m
# 140687135

# 2% heterozygosity is how many bases
calc 0.02\*140687135
# 0.02*140687135 = 2813742.700000

# number of mutations added
bgzip -@75 -cd GCA_003401745.1_ASM340174v1_genomic.fna_upper.diploid.vcf.gz| \
grep -v "^#"|wc -l
# 2830139

# ~2% het rate

# get only 1 haplotype from the "diploid" reference
bgzip -@75 -dc GCA_003401745.1_ASM340174v1_genomic.fna_upper.diploid.fasta.gz|\
/genetics/elbers/bin/seqtk/seqtk seq -L0|paste - - |grep -P "haplo_0\t"| \
tr '\t' '\n' |\
/genetics/elbers/bin/seqtk/seqtk seq -L60 |\
bgzip -@75 \
> GCA_003401745.1_ASM340174v1_genomic.fna_upper.diploid.haplotype0.fasta.gz

# make reference for randomreads.sh
# randomreads.sh part of BBTools/BBMap https://sourceforge.net/projects/bbmap/
/genetics/elbers/bbmap-38.86/randomreads.sh build=1 \
ref=GCA_003401745.1_ASM340174v1_genomic.fna_upper.diploid.fasta.gz \
illuminanames=t addslash=t \
pacbio=t pbmin=0.13 pbmax=0.17 \
reads=100 paired=f \
gaussianlength=t \
minlength=1000 midlength=20000 maxlength=100000 \
out=/dev/null

# make 60x haploid coverage for Illumina reads
/genetics/elbers/bbmap-38.86/randomreads.sh build=1 \
ref=GCA_003401745.1_ASM340174v1_genomic.fna_upper.diploid.fasta.gz \
illuminanames=t addslash=t \
coverage=30 paired=t maxinsert=550 mininsert=450 \
out1=illumina1.fastq.gz out2=illumina2.fastq.gz > random_reads_illumina.log 2>&1

# make 30x haploid coverage for PacBio CLR reads
# error rate from 13 - 15 % minimum 1000bp midlength 20000bp maximum 30000bp
cd /genetics/elbers/test/fly2

/genetics/elbers/bbmap-38.86/randomreads.sh build=1 \
ow=t seed=1 \
ref=GCA_003401745.1_ASM340174v1_genomic.fna_upper.diploid.fasta.gz \
illuminanames=t addslash=t \
pacbio=t pbmin=0.13 pbmax=0.15 \
coverage=15 paired=f \
gaussianlength=t \
minlength=1000 midlength=20000 maxlength=30000 \
out=pacbio.fastq.gz > random_reads_pacbio.log 2>&1

# Ratatosk correction of raw PacBio CLR reads with initial k=41
/genetics/elbers/bin/Ratatosk/build/src/Ratatosk -v --k1 41 -c 75 \
-s illumina1.fastq.gz -s illumina2.fastq.gz -l pacbio.fastq.gz \
-o ratatosk.k41.pacbio > ratatosk.k41.log 2>&1

# De novo assemble Ratatosk corrected k=41 reads with Flye 2.8.1
# using --pacbio-corr and --keep-haplotypes options
# https://github.com/fenderglass/Flye/archive/2.8.1.tar.gz
rm -fr /genetics/elbers/test/fly2/flye
mkdir -p /genetics/elbers/test/fly2/flye
cd /genetics/elbers/test/fly2/flye
/genetics/elbers/Flye-2.8-1/bin/flye --pacbio-corr \
../ratatosk.k41.pacbio.fasta --keep-haplotypes --out-dir ./ \
--threads 75 > flye.log 2>&1

# this failed to assemble because median overlap rate was > 3%

# De novo assemble Ratatosk corrected k=41 reads with Flye --pacbio-raw
# and --keep-haplotypes options
cd /genetics/elbers/test/fly2/flye
/genetics/elbers/Flye-2.8-1/bin/flye --pacbio-raw \
../ratatosk.k41.pacbio.fasta --keep-haplotypes --out-dir ./ \
--threads 75 > flye-raw.log 2>&1

# Ratatosk correction of raw PacBio CLR reads with initial k=31
/genetics/elbers/bin/Ratatosk/build/src/Ratatosk -v --k1 31 -c 75 \
-s illumina1.fastq.gz -s illumina2.fastq.gz -l pacbio.fastq.gz \
-o ratatosk.k31.pacbio > ratatosk.k31.log 2>&1

# De novo assemble Ratatosk corrected k=31 reads with Flye --pacbio-corr
# and --keep-haplotypes options
rm -fr /genetics/elbers/test/fly2/flye2
mkdir -p /genetics/elbers/test/fly2/flye2
cd /genetics/elbers/test/fly2/flye2
/genetics/elbers/Flye-2.8-1/bin/flye --pacbio-corr \
../ratatosk.k31.pacbio.fasta --keep-haplotypes --out-dir ./ \
--threads 75 > flye.log 2>&1

# Ratatosk correction of Ratatosk corrected k=31 reads with initial k=41
/genetics/elbers/bin/Ratatosk/build/src/Ratatosk -v --k1 41 -c 75 \
-s illumina1.fastq.gz -s illumina2.fastq.gz -l ratatosk.k31.pacbio.fasta \
-o ratatosk.k31.k41.pacbio > ratatosk.k31.k41.log 2>&1

# De novo assemble Ratatosk corrected k=31 reads recorrected with k=41
# with Flye --pacbio-corr and --keep-haplotypes options
rm -fr /genetics/elbers/test/fly2/flye3
mkdir -p /genetics/elbers/test/fly2/flye3
cd /genetics/elbers/test/fly2/flye3
/genetics/elbers/Flye-2.8-1/bin/flye --pacbio-corr \
../ratatosk.k31.k41.pacbio.fasta --keep-haplotypes --out-dir ./ \
--threads 75 > flye.log 2>&1

# De novo assemble raw PacBio reads
# with Flye --pacbio-raw and --keep-haplotypes options
rm -fr /genetics/elbers/test/fly2/flye4
mkdir -p /genetics/elbers/test/fly2/flye4
cd /genetics/elbers/test/fly2/flye4
/genetics/elbers/Flye-2.8-1/bin/flye --pacbio-raw \
../pacbio.fastq.gz --keep-haplotypes --out-dir ./ \
--threads 75 > flye-run.log 2>&1

# Make copies of Flye assemblies
cd /genetics/elbers/test/fly2
cp /genetics/elbers/test/fly2/flye/assembly.fasta flye-pacbio-raw-keep-haplo-ratatosk.k41.fasta
cp /genetics/elbers/test/fly2/flye2/assembly.fasta flye-pacbio-corr-keep-haplo-ratatosk.k31.fasta
cp /genetics/elbers/test/fly2/flye3/assembly.fasta flye-pacbio-corr-keep-haplo-ratatosk.k31.k41.fasta
cp /genetics/elbers/test/fly2/flye4/assembly.fasta flye-pacbio-raw-keep-haplo.fasta

# Run Quast 5.1.0rc1 on haplotype0 of reference and flye results
# https://github.com/ablab/quast
# Remember that I converted GCA_003401745.1 to "diploid" with ~2% het rate
mkdir -p /genetics/elbers/test/fly2/quast5
cd /genetics/elbers/test/fly2/quast5
conda activate quast5.0.2
# still using QUAST 5.1.0rc1, just dependencies from quast 5.0.2 conda install
/genetics/elbers/quast-5.1.0rc1/quast.py --fragmented --fast --threads 75 \
-r ../GCA_003401745.1_ASM340174v1_genomic.fna_upper.diploid.haplotype0.fasta.gz \
--eukaryote -o ./ ../flye-pacbio-corr-keep-haplo-ratatosk.k31.fasta \
../flye-pacbio-raw-keep-haplo-ratatosk.k41.fasta \
../flye-pacbio-corr-keep-haplo-ratatosk.k31.k41.fasta \
../flye-pacbio-raw-keep-haplo.fasta \
> quast.log 2>&1 &
conda deactivate

# QUAST results but rename assemblies
# so flye-pacbio-raw-keep-haplo.fasta=flye_raw
# so flye-pacbio-corr-keep-haplo-ratatosk.k31.fasta=k31
# so flye-pacbio-raw-keep-haplo-ratatosk.k41.fasta=k41
# so flye-pacbio-corr-keep-haplo-ratatosk.k31.k41.fasta=k31.k41
perl -pe "s/flye_pacbio_corr_keep_haplo_ratatosk.//g" \
/genetics/elbers/test/fly2/quast5/report.tsv | \
perl -pe "s/flye_pacbio_raw_keep_haplo_ratatosk.//g" | \
perl -pe "s/flye_pacbio_raw_keep_haplo/flye_raw/g" | \
column -ts $'\t'

Assembly                     k31         k41          k31.k41      flye_raw
# contigs (>= 0 bp)          132         352          158          301
# contigs (>= 1000 bp)       131         333          156          277
# contigs (>= 5000 bp)       126         238          148          198
# contigs (>= 10000 bp)      117         192          126          155
# contigs (>= 25000 bp)      105         157          109          123
# contigs (>= 50000 bp)      88          127          89           91
Total length (>= 0 bp)       120546892   133934135    131609107    134023395
Total length (>= 1000 bp)    120545982   133920513    131607527    134006576
Total length (>= 5000 bp)    120526866   133606513    131586885    133769536
Total length (>= 10000 bp)   120459333   133290286    131426311    133480122
Total length (>= 25000 bp)   120256840   132670162    131129782    132966899
Total length (>= 50000 bp)   119630296   131555713    130371773    131725453
# contigs                    132         352          158          301
Largest contig               9942397     12326193     17064634     17041207
Total length                 120546892   133934135    131609107    134023395
Reference length             140694204   140694204    140694204    140694204
Reference GC (%)             42.05       42.05        42.05        42.05
N50                          4056694     3316413      7365893      8571016
NG50                         3701606     3127792      6147490      6120163
N90                          723206      363243       592870       679437
NG90                         -           179968       203468       218073
L50                          10          11           6            5
LG50                         13          12           7            6
L90                          31          54           26           24
LG90                         -           80           50           41
# misassemblies              26          56           44           38
# misassembled contigs       18          42           32           30
Misassembled contigs length  18435008    34001255     42829080     63273702
# local misassemblies        19          35           22           10
# scaffold gap ext. mis.     0           0            0            0
# scaffold gap loc. mis.     0           0            0            0
# unaligned mis. contigs     0           0            0            2
# unaligned contigs          0 + 9 part  2 + 12 part  0 + 12 part  0 + 12 part
Unaligned length             21597       77357        29046        88282
Genome fraction (%)          85.487      94.620       93.299       95.007
Duplication ratio            1.002       1.005        1.002        1.001
# N's per 100 kbp            0.00        0.00         0.00         0.00
# mismatches per 100 kbp     942.95      943.53       942.39       913.91
# indels per 100 kbp         63.97       93.40        63.93        172.23
Largest alignment            9942397     9013642      14839624     14841058
Total aligned length         120481525   133783449    131515984    133819905
NA50                         3701606     2886908      6147488      6120163
NGA50                        3343034     2768541      5509015      6120163
NA90                         668779      285834       453663       487312
NGA90                        -           174809       129793       170560
LA50                         11          13           6            7
LGA50                        13          14           7            7
LA90                         36          60           30           31
LGA90                        -           87           62           53

# Analyze quality scores of reads and assemblies using
# YAK version r55 - yet another k-mer analyzer
# https://github.com/lh3/yak
cd /genetics/elbers/test/fly2

/genetics/elbers/yak/yak count -b37 -t75 -o raw.yak \
<(bgzip -@75 -cd illumina1.fastq.gz) \
<(bgzip -@75 -cd illumina2.fastq.gz) > sr.raw.yak.log 2>&1

cd /genetics/elbers/test/fly2

rm -f test2
for ref in `ls -1 flye-pacbio-corr-keep-haplo-ratatosk.k31.fasta \
flye-pacbio-raw-keep-haplo-ratatosk.k41.fasta \
flye-pacbio-corr-keep-haplo-ratatosk.k31.k41.fasta \
ratatosk.k31.k41.pacbio.fasta \
ratatosk.k31.pacbio.fasta \
flye-pacbio-raw-keep-haplo.fasta \
ratatosk.k41.pacbio.fasta`;do
/genetics/elbers/yak/yak qv -t75 /genetics/elbers/test/fly2/raw.yak \
${ref} 2>/dev/null |tail -n +1034 >> test2
done

rm -f test3
for ref in `ls -1 flye-pacbio-corr-keep-haplo-ratatosk.k31.fasta \
flye-pacbio-raw-keep-haplo-ratatosk.k41.fasta \
flye-pacbio-corr-keep-haplo-ratatosk.k31.k41.fasta \
ratatosk.k31.k41.pacbio.fasta \
ratatosk.k31.pacbio.fasta \
flye-pacbio-raw-keep-haplo.fasta \
ratatosk.k41.pacbio.fasta`;do
echo $ref >> test3
done

paste test3 <(cut -f 2-3 test2) |sort -k2,2 | \
sed '1i Sequences_or_assembly\traw_quality_value\tadjusted_quality_value' |column -tx

Sequences_or_assembly                               raw_quality_value  adjusted_quality_value
ratatosk.k41.pacbio.fasta                           17.614             17.616
ratatosk.k31.pacbio.fasta                           26.866             26.883
flye-pacbio-raw-keep-haplo.fasta                    26.961             26.967
flye-pacbio-raw-keep-haplo-ratatosk.k41.fasta       30.019             30.033
ratatosk.k31.k41.pacbio.fasta                       30.413             30.483
flye-pacbio-corr-keep-haplo-ratatosk.k31.fasta      32.008             32.029
flye-pacbio-corr-keep-haplo-ratatosk.k31.k41.fasta  32.606             32.629

Thanks for your experiments @jelber2!

Here is another iteration of polishing with first --k1 31, then --k1 41, and finally --k1 45
# Ratatosk correction of Ratatosk corrected k=31 then Ratatosk corrected with
# k=41 and now initial k=45
/genetics/elbers/bin/Ratatosk/build/src/Ratatosk -v --k1 45 -c 75 \
-s illumina1.fastq.gz -s illumina2.fastq.gz -l ratatosk.k31.k41.pacbio.fasta \
-o ratatosk.k31.k41.k45.pacbio > ratatosk.k31.k41.k45.log 2>&1

# De novo assemble Ratatosk corrected k=31 reads recorrected with k=41
# with Flye --pacbio-corr and --keep-haplotypes options
rm -fr /genetics/elbers/test/fly2/flye10
mkdir -p /genetics/elbers/test/fly2/flye10
cd /genetics/elbers/test/fly2/flye10
/genetics/elbers/Flye-2.8-1/bin/flye --pacbio-corr \
../ratatosk.k31.k41.k45.pacbio.fasta --keep-haplotypes --out-dir ./ \
--threads 75 > flye.log 2>&1

cd /genetics/elbers/test/fly2
cp flye10/assembly.fasta flye-pacbio-corr-keep-haplo-ratatosk.k31.k41.k45.fasta
# Run Quast 5.1.0rc1 on haplotype0 of reference and flye results
# https://github.com/ablab/quast
# Remember that I converted GCA_003401745.1 to "diploid" with ~2% het rate
mkdir -p /genetics/elbers/test/fly2/quast5
cd /genetics/elbers/test/fly2/quast5
conda activate quast5.0.2
# still using QUAST 5.1.0rc1, just dependencies from quast 5.0.2 conda install
/genetics/elbers/quast-5.1.0rc1/quast.py --fragmented --fast --threads 75 \
-r ../GCA_003401745.1_ASM340174v1_genomic.fna_upper.diploid.haplotype0.fasta.gz \
--eukaryote -o ./ ../flye-pacbio-corr-keep-haplo-ratatosk.k31.fasta \
../flye-pacbio-raw-keep-haplo-ratatosk.k41.fasta \
../flye-pacbio-corr-keep-haplo-ratatosk.k31.k41.fasta \
../flye-pacbio-corr-keep-haplo-ratatosk.k31.k41.k45.fasta \
../flye-pacbio-raw-keep-haplo.fasta \
> quast.log 2>&1 &
conda deactivate

# QUAST results but rename assemblies
# so flye-pacbio-raw-keep-haplo.fasta=flye_raw
# so flye-pacbio-corr-keep-haplo-ratatosk.k31.fasta=k31
# so flye-pacbio-raw-keep-haplo-ratatosk.k41.fasta=k41
# so flye-pacbio-corr-keep-haplo-ratatosk.k31.k41.fasta=k31.k41
# so flye-pacbio-corr-keep-haplo-ratatosk.k31.k41.k45.fasta=k31.k41.k45
perl -pe "s/flye_pacbio_corr_keep_haplo_ratatosk.//g" \
/genetics/elbers/test/fly2/quast5/report.tsv | \
perl -pe "s/flye_pacbio_raw_keep_haplo_ratatosk.//g" | \
perl -pe "s/flye_pacbio_raw_keep_haplo/flye_raw/g" | \
column -ts $'\t'

Assembly                     k31         k41          k31.k41      k31.k41.k45  flye_raw
# contigs (>= 0 bp)          132         352          158          158          301
# contigs (>= 1000 bp)       131         333          156          154          277
# contigs (>= 5000 bp)       126         238          148          140          198
# contigs (>= 10000 bp)      117         192          126          125          155
# contigs (>= 25000 bp)      105         157          109          104          123
# contigs (>= 50000 bp)      88          127          89           89           91
Total length (>= 0 bp)       120546892   133934135    131609107    132172368    134023395
Total length (>= 1000 bp)    120545982   133920513    131607527    132169717    134006576
Total length (>= 5000 bp)    120526866   133606513    131586885    132125079    133769536
Total length (>= 10000 bp)   120459333   133290286    131426311    132017287    133480122
Total length (>= 25000 bp)   120256840   132670162    131129782    131632757    132966899
Total length (>= 50000 bp)   119630296   131555713    130371773    131072498    131725453
# contigs                    132         352          158          158          301
Largest contig               9942397     12326193     17064634     19032407     17041207
Total length                 120546892   133934135    131609107    132172368    134023395
Reference length             140694204   140694204    140694204    140694204    140694204
Reference GC (%)             42.05       42.05        42.05        42.05        42.05
N50                          4056694     3316413      7365893      8573362      8571016
NG50                         3701606     3127792      6147490      7619130      6120163
N90                          723206      363243       592870       549863       679437
NG90                         -           179968       203468       203799       218073
L50                          10          11           6            5            5
LG50                         13          12           7            6            6
L90                          31          54           26           25           24
LG90                         -           80           50           48           41
# misassemblies              26          56           44           38           38
# misassembled contigs       18          42           32           28           30
Misassembled contigs length  18435008    34001255     42829080     61981261     63273702
# local misassemblies        19          35           22           26           10
# scaffold gap ext. mis.     0           0            0            0            0
# scaffold gap loc. mis.     0           0            0            0            0
# unaligned mis. contigs     0           0            0            0            2
# unaligned contigs          0 + 9 part  2 + 12 part  0 + 12 part  0 + 12 part  0 + 12 part
Unaligned length             21597       77357        29046        22786        88282
Genome fraction (%)          85.487      94.620       93.299       93.713       95.007
Duplication ratio            1.002       1.005        1.002        1.002        1.001
# N's per 100 kbp            0.00        0.00         0.00         0.00         0.00
# mismatches per 100 kbp     942.95      943.53       942.39       943.76       913.91
# indels per 100 kbp         63.97       93.40        63.93        64.43        172.23
Largest alignment            9942397     9013642      14839624     17337786     14841058
Total aligned length         120481525   133783449    131515984    132121947    133819905
NA50                         3701606     2886908      6147488      6147347      6120163
NGA50                        3343034     2768541      5509015      6147347      6120163
NA90                         668779      285834       453663       442583       487312
NGA90                        -           174809       129793       145149       170560
LA50                         11          13           6            6            7
LGA50                        13          14           7            6            7
LA90                         36          60           30           31           31
LGA90                        -           87           62           60           53

cd /genetics/elbers/test/fly2

rm -f test2
for ref in `ls -1 flye-pacbio-corr-keep-haplo-ratatosk.k31.fasta \
flye-pacbio-raw-keep-haplo-ratatosk.k41.fasta \
flye-pacbio-corr-keep-haplo-ratatosk.k31.k41.fasta \
flye-pacbio-corr-keep-haplo-ratatosk.k31.k41.k45.fasta \
ratatosk.k31.k41.k45.pacbio.fasta \
ratatosk.k31.k41.pacbio.fasta \
ratatosk.k31.pacbio.fasta \
flye-pacbio-raw-keep-haplo.fasta \
ratatosk.k41.pacbio.fasta`;do
/genetics/elbers/yak/yak qv -t75 /genetics/elbers/test/fly2/raw.yak \
${ref} 2>/dev/null |tail -n +1034 >> test2
done

rm -f test3
for ref in `ls -1 flye-pacbio-corr-keep-haplo-ratatosk.k31.fasta \
flye-pacbio-raw-keep-haplo-ratatosk.k41.fasta \
flye-pacbio-corr-keep-haplo-ratatosk.k31.k41.fasta \
flye-pacbio-corr-keep-haplo-ratatosk.k31.k41.k45.fasta \
ratatosk.k31.k41.k45.pacbio.fasta \
ratatosk.k31.k41.pacbio.fasta \
ratatosk.k31.pacbio.fasta \
flye-pacbio-raw-keep-haplo.fasta \
ratatosk.k41.pacbio.fasta`;do
echo $ref >> test3
done

paste test3 <(cut -f 2-3 test2) |sort -k2,2 | \
sed '1i Sequences_or_assembly\traw_quality_value\tadjusted_quality_value' |column -tx

Sequences_or_assembly                                   raw_quality_value  adjusted_quality_value
ratatosk.k41.pacbio.fasta                               17.614             17.616
ratatosk.k31.pacbio.fasta                               26.866             26.883
flye-pacbio-raw-keep-haplo.fasta                        26.961             26.967
flye-pacbio-raw-keep-haplo-ratatosk.k41.fasta           30.019             30.033
ratatosk.k31.k41.pacbio.fasta                           30.413             30.483
flye-pacbio-corr-keep-haplo-ratatosk.k31.fasta          32.008             32.029
ratatosk.k31.k41.k45.pacbio.fasta                       32.332             32.442
flye-pacbio-corr-keep-haplo-ratatosk.k31.k41.fasta      32.606             32.629
flye-pacbio-corr-keep-haplo-ratatosk.k31.k41.k45.fasta  32.655             32.679

Thanks @jelber2. Seems like the 3-rounds Ratatosk correction (k=31/41/45) is doing well.

Yeah, it seems like (and this likely assembly/individual-specific), that one could find some optimum choice of iterative --k1 values to approximate something like HiFi reads.

I also played with different kmer sizes. I found that for bigger than 63 kmers I need to use a different binary, it cannot run both the small and big kmers. So for k31/k64 I use the default binary of Ratatosk, and for a k61/k95 I use te recompiled Ratatosk version.
After 4 iterations of increasing kmer lengths, I can see that it does improve a bit on quality. Although in my case it seems a minor gain, maybe a few percent more accurate.

Hi @HenrivdGeest,
As mentioned earlier, you indeed need to recompile Ratatosk to use a k-mer size larger than 63. However, you shouldn't have to use two different binaries: if you recompile Ratatosk for k=95, all k-mer length smaller or equal to 95 should be available in the binary. Regarding the error rate, Ratatosk is rather conservative in its correction: if the subsequences to correct are too long, if there are too many similar paths possible, if the traversed subgraph is too branching or there is only poor coloring information (among other reasons), Ratatosk will prefer not to correct rather than making the wrong correction call.
In the future, we are thinking about integrating phasing information in the correction to improve the quality (using WhatsHap for example) so that might be something of interest to you.
Do not hesitate to post here any benchmark you have about the read data set after/before correction or the assembly of the reads. I do not work with plant genomes but I would be happy to provide any help if I can.

Maybe one last thing to try @HenrivdGeest would be smaller k1/k2-mers for the initial Ratatosk run: maybe 25/45? That would increase the number of matches between the graph and the long reads but decrease the contiguity of the graph. It might actually be beneficial in your case.

This is what I tried:
phase=4;
/home/geest/bin/Ratatosk_k64 -s short.fasta -l long_with_errors.fasta a -c 128 -o phase1${phase} --verbose --insert_sz 1000 --k1 21 --k2 51;
/home/geest/bin/Ratatosk_k64 -s short -l phase1${phase}.fasta -c 64 -o phase2${phase} --verbose --insert_sz 1000 --k1 51 --k2 63 ;
/home/geest/bin/Ratatosk_k96 -s short  -l phase2${phase}.fasta -c 32 -o phase3${phase} --verbose --insert_sz 1000 --k1 61 --k2 95;
/home/geest/bin/Ratatosk_k96 -s short -l phase3${phase}.fasta -c 32 -o phase4${phase} --verbose --insert_sz 1000 --k1 71 --k2 95 ;
And these are the mapping numbers, measured with Alfred QC ( I took reads from a 4Mbase plant contig as a test)
uncorrected:
MismatchRate    0.0338654
DeletionRate    0.0270461
InsertionRate   0.0353234
ErrorRate       0.096282
phase14.fasta.bam
MismatchRate    0.00425731
DeletionRate    0.000926536
InsertionRate   0.0019838
ErrorRate       0.0071946
phase24.fasta.bam
MismatchRate    0.0035878
DeletionRate    0.000633081
InsertionRate   0.00134147
ErrorRate       0.00558711
phase34.fasta.bam
MismatchRate    0.00293067
DeletionRate    0.000530815
InsertionRate   0.000975063
ErrorRate       0.004459
phase44.fasta.bam
MismatchRate    0.00290499
DeletionRate    0.000523178
InsertionRate   0.001924023
ErrorRate       0.00437431
to the error rate went from ~10% to 0,7% in the first round, and got to 0,4% after 4 iterations.
Although especially the insertion rate got a nice boost from 0,2% to 0.1% from iteration 1 to iteration 4.
Still I am impressed by the error rate just after 1 round.