titancna with vrn_file not enough input data

Hi I am trying titancna on cohort with precalled mutation. I am using the ensemble-annotated.vcf as a vrn_file. bcbio find the vcd and generate a .tbi file but TitanCNA is complaining about not enough input data even if each vcf is plenty of variants. Do you have any idea what I am doing wrong?
Thanks a lot!
2018-03-18T15:19Z] ls04: Skipping TitanCNA; not enough input data: MMRF_1598_1_BM_CD138pos_T1_KHS5U_K14088_msk MMRF_1598_1_PB_CD3pos_C2_KHS5U_K14069_msk [2018-03-18T15:23Z] ls11: Skipping TitanCNA; not enough input data: MMRF_1890_1_BM_CD138pos_T1_KBS5U_L05558_msk MMRF_1890_2_PB_CD3pos_C1_KBS5U_L05562_msk [2018-03-18T15:40Z] lt12: Skipping TitanCNA; not enough input data: MMRF_1881_1_PB_Whole_C3_KBS5U_L05808_msk MMRF_1881_1_BM_CD138pos_T1_KBS5U_L05787_msk [2018-03-18T15:40Z] lt12: Unexpected error Traceback (most recent call last):   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/ipythontasks.py", line 51, in _setup_logging     yield config   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/ipythontasks.py", line 429, in detect_sv     return ipython.zip_args(apply(structural.detect_sv, *args))   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/structural/__init__.py", line 197, in detect_sv     for svdata in caller_fn(items):   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/structural/titancna.py", line 28, in run     het_file = _titan_het_file(heterogeneity.get_variants(paired.tumor_data), work_dir, paired)   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/structural/titancna.py", line 127, in _titan_het_file     work_dir, paired, OutWriter)   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/heterogeneity/bubbletree.py", line 147, in prep_vrn_file     stats = _is_possible_loh(rec, bcf_in, params, somatic_info)   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/heterogeneity/bubbletree.py", line 309, in _is_possible_loh     stats = _tumor_normal_stats(rec, somatic_info)   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/heterogeneity/bubbletree.py", line 292, in _tumor_normal_stats     alt, depth, freq = sample_alt_and_depth(rec, sample)   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/heterogeneity/bubbletree.py", line 350, in sample_alt_and_depth     all_counts = [int(x) for x in sample["AD"]] TypeError: int() argument must be a string or a number, not 'NoneType' [2018-03-18T15:41Z] lt18: Unexpected error Traceback (most recent call last):   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/ipythontasks.py", line 51, in _setup_logging     yield config   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/ipythontasks.py", line 429, in detect_sv     return ipython.zip_args(apply(structural.detect_sv, *args))   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/structural/__init__.py", line 197, in detect_sv     for svdata in caller_fn(items):   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/structural/titancna.py", line 28, in run     het_file = _titan_het_file(heterogeneity.get_variants(paired.tumor_data), work_dir, paired)   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/structural/titancna.py", line 127, in _titan_het_file     work_dir, paired, OutWriter)   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/heterogeneity/bubbletree.py", line 147, in prep_vrn_file     stats = _is_possible_loh(rec, bcf_in, params, somatic_info)   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/heterogeneity/bubbletree.py", line 309, in _is_possible_loh     stats = _tumor_normal_stats(rec, somatic_info)   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/heterogeneity/bubbletree.py", line 292, in _tumor_normal_stats     alt, depth, freq = sample_alt_and_depth(rec, sample)   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/heterogeneity/bubbletree.py", line 350, in sample_alt_and_depth     all_counts = [int(x) for x in sample["AD"]] TypeError: int() argument must be a string or a number, not 'NoneType' [2018-03-18T15:44Z] lt18: Unexpected error Traceback (most recent call last):   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/ipythontasks.py", line 51, in _setup_logging     yield config   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/ipythontasks.py", line 429, in detect_sv     return ipython.zip_args(apply(structural.detect_sv, *args))   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/structural/__init__.py", line 197, in detect_sv     for svdata in caller_fn(items):   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/structural/titancna.py", line 28, in run     het_file = _titan_het_file(heterogeneity.get_variants(paired.tumor_data), work_dir, paired)   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/structural/titancna.py", line 127, in _titan_het_file     work_dir, paired, OutWriter)   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/heterogeneity/bubbletree.py", line 147, in prep_vrn_file     stats = _is_possible_loh(rec, bcf_in, params, somatic_info)   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/heterogeneity/bubbletree.py", line 309, in _is_possible_loh     stats = _tumor_normal_stats(rec, somatic_info)   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/heterogeneity/bubbletree.py", line 292, in _tumor_normal_stats     alt, depth, freq = sample_alt_and_depth(rec, sample)   File "/home/pastore/data/bcbio/anaconda/lib/python2.7/site-packages/bcbio/heterogeneity/bubbletree.py", line 350, in sample_alt_and_depth     all_counts = [int(x) for x in sample["AD"]] TypeError: int() argument must be a string or a number, not 'NoneType'

Thanks for testing this out and sorry about the issues. We've not yet tried to run TitanCNA and heterogeneity analysis from pre-called VCF inputs, so we're likely running into issues with converting the data into the right formats. In this case it appears as if the ensemble inputs you're inputting don't have some of the expected keys like allele depth homogenized:
https://github.com/bcbio/bcbio-nextgen/blob/8a76c9e546cb79621707082fd763bd643e0e9652/bcbio/heterogeneity/bubbletree.py#L346
If you compare your input to what the code there expects it might give us some idea of what is missing. ensemble is going to be especially problematic for this because it's a combination of multiple callers, so you might want to explore using a single somatic caller.
We'll work on generalizing this but in the short term if you need to get an analysis done might need to recall with a somatic callers in the analysis to avoid having to do too much debugging. Hope this helps.

Thanks a lot! I am rerunning with vardict. Let you know if i will get other issues.
Thanks a lot!
Sent from my iPhone

On Mar 23, 2018, at 3:05 PM, Brad Chapman<EMAIL_ADDRESS>wrote:
Thanks for testing this out and sorry about the issues. We've not yet tried to run TitanCNA and heterogeneity analysis from pre-called VCF inputs, so we're likely running into issues with converting the data into the right formats. In this case it appears as if the ensemble inputs you're inputting don't have some of the expected keys like allele depth homogenized:
https://github.com/bcbio/bcbio-nextgen/blob/8a76c9e546cb79621707082fd763bd643e0e9652/bcbio/heterogeneity/bubbletree.py#L346
If you compare your input to what the code there expects it might give us some idea of what is missing. ensemble is going to be especially problematic for this because it's a combination of multiple callers, so you might want to explore using a single somatic caller.
We'll work on generalizing this but in the short term if you need to get an analysis done might need to recall with a somatic callers in the analysis to avoid having to do too much debugging. Hope this helps.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Thanks, closing this as a reasonable workaround was suggested which seemed to do the trick. Please re-open if this is a recurring issue.


Remove all deprecated functions

We need to update Wallaby's internal test suite and remove the deprecated functions.

Hey @keathley I saw all the deprecations while working on the JS dialogs. I already thought about making a PR to rewrite the tests with the new API. Could you use some help here?
What about tests that test the deprecated functions themselves, should they stay or be removed altogether?

What we already wanna remove them all in 0.17? That's cool, wasn't sure what your deprecation policy would be :D
I'd be super glad to remove them. Lots of them are not covered and get the test coverage down but of course writing tests for them doesn't make much sense :)

@padde Any help here would be great! Some of the old tests should be updated to use the new api. But a lot of them can just be deleted. For instance, all the tests like, "fill_in/3 finds elements by id" and "fill_in/3 finds elements by label text" can be deleted.
At some point I'm going to wrap the entire external api with property tests so we don't have to write all of these permutations by hand any more.

@keathley sounds great, I'd be glad to help. Just to be clear: do you want to keep the deprecated methods themselves in Browser or should they be removed as well? If they're kept, should there be at least one test for each deprecated method left in the test suite?
For the deprecation policy, SemVer might give some guidance here (if you should decide to stick with this). It generally forbids backwards incompatible changes in minor releases, but:

Major version zero (0.y.z) is for initial development. Anything may change at any time. The public API should not be considered stable.

If its easier you can just remove it all. But if you just want to work on tests that fine.
It was an interesting decision to deprecate things before we hit 1.0 but I knew that there were enough people using it and the api was a big enough change that I wanted to provide the deprecations as a way to help people migrate. Now that we have a good direction for the api I'm fine with removing the old functionality. Plus it gets us closer to a 1.0 release 😉

Very pragmatic decision, I might get onto removing some of them during the weekend (I'd try to do it in smaller chunks)

Hey @PragTob didn't want to steal this from you, please go ahead! I think it makes more sense to  remove deprecated functions along with their tests and also replace them in other tests in one go. The compiler will basically tell you exactly what to do once you've removed the function, and we'd probably trip over each others feet all the time when merging.

@padde we can also work together and coordinate, I'll see what I get to this weekend because next week I probably don't have that much time :) I'm just sometimes super excited by these "stupid" tasks because all the warning messages annoy the hell out of me :D

On another note, I doubt I'll get to any more than #181 at least this weekend so feel free to work away ( @padde )

As a working thing, I'd say if whoever starts working on one of this just post here the name of the function you started working on removing all the deprecations (like I dunno, find or whatever). That way we can synchronize, save for find itself there also shouldn't be too much overlap :)

Removing choose now

Work on this seems to have slowed down a lot. Frustrated by trying to look through browser.ex with all the deprecated functions in place, I deleted them all. No tests failed as a result, but that is because someone thoughtfully removed the failing tests ;-)
The removal of the tests has been pushed, but not the removal of the actual deprecated functions.
Would my changes (i.e. the deletions) make a useful pull request? Does someone already have this and waiting for a suitable moment to release it? Have I misunderstood all the foregoing comments completely?
Graeme
PS
A bit off-topic, but I have raised a couple of "issues" which were really just calls for help. Is there a better place for such discussions?

Hi there @graeme-defty - the tests aren't removed yet. I suspect you ran mix test but since a recent test you need to run mix test.all for which lots of tests should be failing then, at least I get lots of deprecation warnings.
Work has slowed down a bit here, true. I'm busy with non wallaby stuff and a lot of work also went into the selenium stuff - I might get back to some function removals though.
I don't see any issues from you. As we don't have a mailing list, this is a good place to raise issues imo. Other places can be #wallaby on slack or elixirforum.

Yes, exactly what I was doing. I have plenty of errors now, thanks.
Happy to lend a hand with this, if it helps.
g
On 7 May 2017 at 19:17, Tobias Pfeiffer<EMAIL_ADDRESS>wrote:

Hi there @graeme-defty https://github.com/graeme-defty - the tests
aren't removed yet. I suspect you ran mix test but since a recent test
you need to run mix test.all for which lots of tests should be failing
then, at least I get lots of deprecation warnings.
Work has slowed down a bit here, true. I'm busy with non wallaby stuff and
a lot of work also went into the selenium stuff - I might get back to some
function removals though.
I don't see any issues from you. As we don't have a mailing list, this is
a good place to raise issues imo. Other places can be #wallaby on slack
or elixirforum.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/keathley/wallaby/issues/177#issuecomment-299702585,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAK80KwrfNIoG4CsQEKtqyN_lyiltcKEks5r3bZogaJpZM4MvwL1
.

OK, so I looked at the very first deprecated function in browser.ex. This turned out to be check/n. My concern here is that it seems that removing check and uncheck removes functionality from the API.
As it stands, I can set (check), unset (uncheck) or toggle (click) the value of an option. Removing check and uncheck is a reduction in capability since now I can only toggle the value. Of course I can test the value first, but that puts the onus on the tester to do extra work that the framework can easily handle.
Not only that, this API is the tester's main way to specify testing operations, and should be as rich as possible, meaning as close as possible to how a user views his interactions with the browser screen. This makes tests much more readable.
My apologies if I have missed the point here. Also, please bear in mind that I am new to the project, so if this has been discussed ad nauseam elsewhere please point me to the conversation and I will try to alleviate my ignorance.
Thanks,
graeme

Hi there,
your master seems to be out of date. check was already removed in 894b72d0cb5
I can see your concern about the removed functionality, this could best be discussed in a separate issue that you could open and then also tag @keathley in there as he first defined those functions as deprecated and I'm sure he had an idea about it :)
I'm not sure about them. You are right though, click only is a real replacement for toggle. One might argue that the tester should know from the test setup which fields are checked and which aren't so that it doesn't make a difference and in the end what the user does is clicking.
But as I said, that's probably easier done in a separate issue and easy to restore :)
Tobi

Hi Tobi
Thanks for the advice and guidance.
I will (a) make sure I am working off the latest master and then (b) open
another issue making the case for another review of the interface. I am
still pretty new to Wallaby, so I hope I am not asking folks to go over old
ground, but I feel this is important as this API is a big part of how the
user views the framework.
Thanks again.
graeme
On 7 May 2017 at 23:09, Tobias Pfeiffer<EMAIL_ADDRESS>wrote:

Hi there,
your master seems to be out of date. check was already removed in 894b72d
https://github.com/keathley/wallaby/commit/894b72d0cb50de14e7b238f5b1735662aa50709a
I can see your concern about the removed functionality, this could best be
discussed in a separate issue that you could open and then also tag
@keathley https://github.com/keathley in there as he first defined
those functions as deprecated and I'm sure he had an idea about it :)
I'm not sure about them. You are right though, click only is a real
replacement for toggle. One might argue that the tester should know from
the test setup which fields are checked and which aren't so that it doesn't
make a difference and in the end what the user does is clicking.
But as I said, that's probably easier done in a separate issue and easy to
restore :)
Tobi
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/keathley/wallaby/issues/177#issuecomment-299716196,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAK80F8xevniKjUl4zNs5DSeGbw-CwwFks5r3eylgaJpZM4MvwL1
.

@graeme-defty don't worry about making us go over old ground, "worst" case there is already an issue/comment where the thinking is explained and we'll just link to it :) Thanks for contributing to wallaby!

@graeme-defty Thanks for the comments! I agree that we're loosing some functionality by removing check and uncheck. But the bigger idea behind these deprecations is to separate querying from actions. I'll use click_link and click_button as an example for a moment and then talk about why I feel like removing check and uncheck is a good idea.
The only reason to have click_link and click_button is to make it easier to specify the element that you actually want to click. We can achieve the same semantics by simply saying, click(link("Some Link")) or click(button("Some Button")). A great side benefit of this approach is that the end user no longer has to learn Wallaby's api rules. They get a few simple actions and can compose them any way that they want. This moves the declarative power out of the action api and into the query api. This movement is a huge benefit to myself as a maintainer because its easier to build stateless transformations. I also get a benefit as a user of the api because I can specify exactly what I want to interact with as a query.
With check and uncheck I agree that we're loosing some semantic power from the "Actions API". The goal is to move that power to the Query api. check and uncheck is very similar to click_link and click_button in that you're trying to specify the state of the element as part of the action. In other words, check and uncheck are trying to specify query logic as part of the click action.
My goal is to add a new option to the query api that would allow you to specify the "selected" state of the element. This would be equivalent to an uncheck action: click(checkbox("My checkbox", checked: true)).
Hopefully all of that makes sense. I'm happy to discuss more of these design decisions if you have thoughts on them. I appreciate you thinking and asking about this stuff :+1:

That makes sense to me at least, thanks for taking the time to explain @keathley ! :)

@keathley OK – let me try to outline my thinking in the most compelling way (experience tells me that I quite probably have only one shot at this, so it had better be good! :-) )
Apologies in advance for the length of this post, but I want to make sure I express myself clearly and there are a number of facets I want to explore.
First let me say thank you for taking time to listen. I am not being (intentionally) difficult. I am taking an interest because I think Wallaby has a great future. The expressiveness allowed by Elixir is perfect for this kind of framework, and I really like the direction in which Wallaby is heading.
Second, with respect to your email, and in particular the second paragraph (The only reason … ) rest assured that I absolutely do get the introduction of Query as the encapsulation of the act of locating an element. And an important benefit of this is that it reduces the number of unit tests dramatically even in those functions which remain. No longer do we have click_element_found_by_css, click_element_found_by_name, click_element_ … and endless other combinations. The act of finding the element is separated from the action taken on it, and the two can be tested independently. In fact I have some suggestions on further encapsulation of Query that you may like to consider, but I will save that for another discussion. I want to get to the main course.
The next 2 paragraphs in your mail highlight where we are looking at this from different perspectives.
When it comes to click_link, click_button, et al I could not agree with you more. All of these elements are simple (i.e. single-level) structures that have no state. You click them and that’s it. It really makes no sense to have multiple actions here.
The checkbox, conversely DOES have state. And a simple click reverses that state.  This is very very different from a link or a button. The action you propose click(checkbox("My checkbox", checked: true)). is different from uncheck in one important regard. The query in the former will fail if the checkbox is not checked. And as a tester, there will be many cases where I really don’t care what value it has initially,  I simply want to ensure that after my action, it will have the desired state.
Now Tobi raised this point in his response - a tester has the ability to know exactly the state of his test environment, and can in principle know how a checkbox is initially set, but that may depend on a number of factors, and this is an added complexity that I do not believe the tester should be forced to handle.
Let me give a concrete example.
Say I have a screen which adds a new user to my system and there is a group of personal interests/hobbies represented by checkboxes, some of which are set on by default. Now if I want to add a user for some test. I need the user to have a specific set of interests, but that is not the focus of the test. In such a situation I do not want to concern myself here with what the default values are. (I have another test to check the defaults are correct.) At some future time the default values may change. Then my test will fail and will need to be changed for what seems to me to be no good reason. The defaults may even come from a config file. Or perhaps the defaults may be generated based on other information we have already entered about this user. Granted this is all still under my control, but it leads to brittle tests.
Stepping back a moment, there is a certain elegance in the ‘everything is a find or a click’ approach. A minimal number of basic concepts which can be composed to build anything the tester needs. However, although this can be done, that does not necessarily mean that we should do it. A good analogy here is the Elixir language itself, which has a very small number of basic constructs and a very, very cool macro facility. So cool in fact that many of the basic language constructs we take for granted are actually macros. (It is a while since I read about this, so forgive me if I have mis-remembered, but I believe that ‘if’ and ‘case’ are examples.) Now Jose could have given us just the basics and the macros and left us to construct the rest of Elixir ourselves, but I do not think we would have thanked him for it. A minimal set of orthogonal concepts is great for under the covers, but the bones are better if we join them up and put a bit of meat on them.
So it is a judgement call. We have to pitch somewhere on the scale of “bare minimum of composable features and you can build the rest yourself” to “here are so many concepts and ways to do everything that it will make your head spin”.
Usually, the truth lies somewhere in the middle. But where?
For me, a good target for the language of describing tests (i.e. our API) is the language we would use if we were describing to a user over the phone how to perform the tests manually. I could of course say to my remote user “look at this checkbox and if it is checked then click it” but I am much more likely to simply say “make sure this checkbox is unchecked”. The telephone call analogy appeals to me because it is almost exactly the situation we are in with respect to the communication we have with someone who is reading our tests down the track. It is much easier for him if we speak to him (i.e. write our tests) in the language we would use in talking to a user sitting in front of our browser interface.
I totally agree with your point about minimising the learning curve for the tester (your point about having to learn Wallaby rules). We have the same goal here. But if we go the minimalist approach he has to learn our set of minimal constructs and the rules for composing them. But the best way to minimise his learning curve in IMHO is to use take our API closer to his language - i.e. the language we would use in the telephone call situation.
Well, apologies again for the length of this missive, and again thanks for listening to my views. I hope I have done my job well enough, but I hope that it has at least been useful.
g

OR ...
Maybe the way to look at it is that some DOM elements have a value and others do not.
Things without value would be links and buttons. You can click them and that's it.
Things with value include check boxes, radio button groups, drop-down lists and text boxes (any others?).  The  set_value functino could be retained to set the value of these elements. Depending on the type of the element set_value operates appropriately, setting a checkbox to true or false, selecting a particular radio button from a group, selecting one option from a drop-down list or performing a send_keys operation for a text box. That gives us Query, click and set_value as the primitive concepts. A fairly short list, and while not quite as obvious as check and uncheck, I think still fairly clear and readable.
What do you think?

@graeme-defty No worries about all of the text. I really do appreciate you taking the time to thinking about this stuff. :heart:
I think keeping set_value around makes a lot of sense. Its not as clear as check or uncheck but it allows users to compose their own actions in a more meaningful way. I need to do some more thinking about this stuff. We are going to have to make exceptions for other actions like attach_file. I just need to think about where we draw the line.

Sounds good.
Sorry I did not think of the that suggestion first. I could have saved myself a lot of typing and you a lot of reading!

I just posted a pull request for removing deprecated attr/2. (Stayed with a baby one :-) )
I am looking now at what needs to be done to set_value.

I have created a PR for the extension to set_value so it now works with check boxes, and incidentally, radio buttons and drop-down list options.

All deprecated functions were removed with v0.17. :tada:


Use udev rules/hwdb to match mice to drivers and SVGs

This adds udev rules/hwdb that sets RATBAG_DRIVER and RATBAG_SVG
properties to the event devices. We then use this property to select
the correct driver instead of checking the hardcoded list in the
drivers. Having this in a hwdb makes it easier for users to test
if drivers match their mouse with a system installed libratbag.
Each driver now has an "id" which is used to match the driver set
in RATBAG_DRIVER. A SVG can optionally be set as well.
All entries in the hwdb are currently without a name as that
information was not available in the hardcoded lists.
Note that this patch breaks the test-driver.

Thanks for the patch. I made a quick fix for make check to be happy so I'll just merge it and apply my patch on top.


Reset signal duration

I'm curious how the reset signal duration value of 60us was calculated?
The timing seems very tight to the tolerances of the chip (>50us) and I can't find any code which accommodates any flush of the 8-word FIFO buffer. Ws2812::write adds 60us onto Ws2812Direct::write but that only blocks until the write to the FIFO buffer is complete, not until the PIO has processed it. In the worst case I think this lag could be -
8 (word tx only FIFO buffer) x  3 (colours) x 8 (bits) x 1.25e-6 (800kHz) = 2.4e-4 = 2.4ms

The original example from the RP Pico datasheet uses a value of 10ms. This seems large enough to cover any required flush and the reset signal itself. However, 60us does not.
Please note I haven't taken a probe to the actual output or observed any real-world problems. I'm just trying to understand it from the specifications and the code written. A perfectly reasonable answer would be that someone attached a scope to the pin and observed the real-world timings (assuming sufficient data to fill the buffer)!

It is indeed taken from the 50μs specified in the ws2812's datasheet with a little bit more just in case.
However, you're right that before waiting for that delay we should also wait for the FIFO to be emptied.


How to make the ID field editable in the add/edit dialog form?

I have some old table, the ID field is string.
Something like "01","02","03" etc.
The admin can specify the ID string while adding. How to fullfill this?

I've tried out the solution: manually add the ID field in the Form.cs
Thanks a lot.


METS from edu.harvard.hul has undefined license

To resolve this, we should investigate if this functionality is still needed (METS support). If not, remove due to confusion on license. (One page lists this dependency as LGPL.) If used, we should replace with one of the following:
An alternative is https://sourceforge.net/projects/mets-api/ which is under Apache2 license, and http://sourceforge.net/projects/seventrain which is under BSD.

Working to pull this code into it's own repo.  I checked with @doolse who didn't conceptually see any major issues, and I've sent a follow up note to a few folks to confirm user stories.

Code changes and smoke tests complete.  Test binaries with and without METS are built.  Requested some advice on a more complete test suite for METS.  Plan to run the tests using the binaries tomorrow.
We will have to put the hul mets external dep into Maven local for now, and then publish ourselves.
@drewwills - please create an Equella-METS repo similar to #55 .

METS code published as an 'optional functionality' here:  https://github.com/equella/Equella-METS

I've used the way back machine to find the original license - it's LGPL, which means we can publish it yeah?
https://web.archive.org/web/20100413195423/http://hul.harvard.edu:80/mets/download.html

Confirmed with @drewwills and @bjagg yesterday - LGPL is valid to publish the dependency for Equella in the core base.

Removed the 'optional functionality' and 'migration testing' documentation.
Closed the pull request #91 and deleted the issues/78 branch
Added this dependency into #79
@drewwills - please remove the Equella-METS repo.  Then we can close this issue.

[ ] Done


[C++][Parquet] Add support for BYTE_STREAM_SPLIT encoding

From the Parquet issue ( https://issues.apache.org/jira/browse/PARQUET-1622 ):
Apache Parquet does not have any encodings suitable for FP data and the available text compressors (zstd, gzip, etc) do not handle FP data very well.
It is possible to apply a simple data transformation named "stream splitting". Such could be "byte stream splitting" which creates K streams of length N where K is the number of bytes in the data type (4 for floats, 8 for doubles) and N is the number of elements in the sequence.
The transformed data compresses significantly better on average than the original data and for some cases there is a performance improvement in compression and decompression speed.
You can read a more detailed report here:
[https://drive.google.com/file/d/1wfLQyO2G5nofYFkS7pVbUW0-oJkQqBvv/view
Apache Arrow can benefit from the reduced requirements for storing FP parquet column data and improvements in decompression speed.
Reporter: Martin Radev / @martinradev
Assignee: Martin Radev](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=martinradev) / @martinradev
Related issues:

[Format] Support lossy compression (is related to)

PRs and other links:

GitHub Pull Request #6005

Note: This issue was originally created as PARQUET-1716. Please see the migration documentation for further details.

Wes McKinney / @wesm:
Issue resolved by pull request 6005
https://github.com/apache/arrow/pull/6005


In addition to .nvmrc, read engines field from package.json

Since it's trivial in PowerShell to parse JSON it would be so awesome if I could just run Set-NodeVersion on any cloned open source repo and it directly uses the needed version as declared in package.json. A lot of packages have the engines field specified but only few have a .nvmrc

Do people actually use that part of the package.json? I'll admit to having almost never set it 😛.
But yeah, it could be a fallback.

Totally!
https://github.com/expressjs/express/blob/351396f971280ab79faddcf9782ea50f4e88358d/package.json#L82
https://github.com/sequelize/sequelize/blob/34d075e2a5575108a037debcd66e814fd6dd5c4c/package.json#L141
It's what powers these little awesome badges in READMEs, super useful:

Today's pretty hectic for me so I probably can't get to it, but I won't say no to a PR!

It would depend on #23 because the versions in there are semver ranges.

I don't think the version type supports ranges, so might be regex still!

@aaronpowell semver ranges are not a regular grammar unfortunately. Can PowerShell modules have NuGet dependencies? https://github.com/adamreeve/semver.net

.... kind of.
Publishing to the gallery is really quite simplistic, it just scans the directory you're publishing and creates a NuGet package that contains every file and pushes that up to the gallery.
So if you install the contents of that NuGet package into the folder it'll be included (it's how one of the releases shipped with a bunch of node releases by accident!).
Just thinking through the practicalities of it - you'd need to parse the range from out of the package.json, then download all the known versions and find out which one matches. I'm concerned about the overall performance of it, could result in a lot of calls to Node.js's website.

Got it. I don't think requesting https://nodejs.org/dist/index.json before the install to figure out the best version would be a big deal, compared to the download itself.
Would it be a better approach to have PowerShell cmdlets for semver.net and define that as a dependency?

I'm wondering if it wouldn't be smarter to wrap all the version detection logic up into a .NET library, then you could add the stuff for #23 as well.

How about I convert you into a full contributor so you can work on this without relying on a fork?

Sounds good!

So we're calling this as done?

Yup


How to handle network failure with the SDK?

Hi,
I'm wondering how to handle network failure when we send telemetry with the aspnetCore SDK.
Let's say the SDK failed to send the telemetry to AI. How can I handle that? Can I listen to the failure and log that locally?
Thanks

Channel implementation will make the best effort to send the telemetry. Make sure you are using latest versions of SDK so ServerTelemetryChannel will be initialized. Implementation of server telemetry channel is here: https://github.com/Microsoft/ApplicationInsights-dotnet
If you need to definitely make sure that item is delivered - you'd need to implement your own channel like this http://apmtips.com/blog/2016/11/10/sync-channel/ as Application Insights SDK do not guarantee delivery - it's designed to make a best effort in delivering.

Thanks for the quick reply. I tested the example of the sync-channel. It worked good for .NET framework (4.6.1) but not for dotnetCore 2.0. The Transmission SendAsync() and Wait() does not throw an exception. Should I open a bug?

Yes, please open a bug. Transmission class is not very extensive so you can just re-implement it yourself. And add any error handling you need.


remove awesome-token-sale

awesome-token-sale is deleted. Related #206

Thanks! Also there is a fork as a possible replacement: https://github.com/AronVanAmmers/awesome-token-sale
But I guess I'll let this one go.


Add Paragraph Tabbing ability

DOES NOT INCLUDE LIST TABBING DESCRIBED HERE:
https://streamable.com/lpl4mv

Hey @charles01pd2018 can you describe what this is adding - a bit confused. I saw the video on what it's not adding but is this just for indenting?

Hey @charles01pd2018 can you describe what this is adding - a bit confused. I saw the video on what it's not adding but is this just for indenting?

this is just adding the ability to tab regular paragraphs -> so like in google docs when you hit tab it indents

Hey @charles01pd2018 can you describe what this is adding - a bit confused. I saw the video on what it's not adding but is this just for indenting?

So it just inserts four spaces basically -> https://streamable.com/8joqc6

Hey @charles01pd2018 can you describe what this is adding - a bit confused. I saw the video on what it's not adding but is this just for indenting?

So it just inserts four spaces basically -> https://streamable.com/8joqc6

Oh got it!


Singularity not spawning with radiation enabled

Description
Looks like singulo are not spawning with the radiation component enabled.
Reproduction

Showradiation
spawn singulo
check radiation component.

Screenshots

Additional context
Someone mentioned its just started happening last several rounds. Current workaround is to turn it on in component via an admin.

Looks like its already fixed just not deployed from #22259


Use SavedStateRegistry for persisting state

This PR demonstrates how we could use the SavedStateRegistry to persist viewModel state due to process recreation.
I believe this would allow us to remove the concept of the MvRxViewModelStoreOwner as we no longer need to hook into the activity, or fragment's onSaveInstanceState and onCreate methods to directly save and restore the viewModels. Instead, we can register a SavedStateProvider that directly hooks in to the lifecycle when we create the viewModel. When we go to recreate the viewModel, we check the saved state registry to see if there is a persisted state we can restore.
The restoreViewModel method in MvRxViewModelStoreOwner helped ensure that all existingViewModels and parentViewModels were created before being accessed after system-initiated process death. Without the MvRxViewModelStoreOwner now, existingViewModel and parentViewModel could potentially access the viewModel before they have been recreated. This requires both these extensions to call into the modified MvRxViewModelProvider.get method and provide their own viewModel factories instead of using ViewModelProviders.of
Would love to get your thoughts on this @elihart @BenSchwab
Thanks!

@elihart I currently have it so that we're always persisting both the state and the initial args of the viewModel when saving to the bundle. This removes having to special case for saving fragment args in activityViewModels.

@elihart updated with your feedback! Sorry for the delay, I restructured my code a bit to check the saved state registry in the create method of the MvRxFactory so that we are registering/consuming the state in one place that is attached to the lifecycle of the viewModel. I think in my initial implementation I was registering it only when the viewModel was created, but consuming it each time MvRxViewModelProvider.get was called which is tied to the fragment/activity creation instead. I also updated/added some tests.

Thanks for reviewing! I've removed the BaseMvRxActivity and addressed PR feedback, but other than that I didn't have anything else to add to this PR.

New versions of Lifecycle (2.2.0-alpha03) and Lifecycle ViewModel SavedState (1.0.0-alpha03) were just released, this may be relevant to this PR.

I don't think we've decided completely, but imo we a lifecycle beta release would be a reasonable time to merge this. I think that alpha would be too early though.

@elihart @mradzinski Agreed. We might want to consider beta but I would be cautious to require that for any users of MvRx.
There is an official definition of alpha/beta/rc here: https://developer.android.com/jetpack/androidx/versions

Checking this again, is the latest alpha version of Lifecycle truly needed or it was updated per request? I think changes contained in this PR are available in the latest stable version of Google's library as well, so I would actually revert to stable and leave updating that library for another PR.

A newer version of androidx.fragment (stable release is currently 1.0.0) is needed to reference the SavedStateRegistry from the fragment/activity. androidx.fragment:fragment:1.1.0-rc04 requires a lifecycle version of 2.2.0-alpha02 which is why I bumped this version in this PR.

A stable release of androidx.fragment 1.1.0 is out - can we update to that and then be able to merge this? https://developer.android.com/jetpack/androidx/releases/fragment?authuser=2#1.1.0
It seems strange that a stable release of androidx.fragment would require an alpha version of the lifecycle library

I agree with @elihart. Here's what it looks to be the build.gradle file of the latest stable version (1.1.0) of androidx.fragments: https://android.googlesource.com/platform/frameworks/support/+/refs/heads/androidx-master-dev/fragment/fragment/build.gradle

@mradzinski @elihart I wonder if it was an issue with the code not syncing. I upgraded in this pr and this is the compile classpath
https://gist.github.com/gpeal/ef77f349eb90bc69b7518a94390609a0

@mmcwong you can now rebase on https://github.com/airbnb/MvRx/pull/285

@BenSchwab could you take another look at this before I merge this change? Namely the restriction around accessing the view model after the super.onCreate method


Create config section for snapshotter

Currently, the snapshotter is just a string. This should probably take more configuration, such as the type and root.
@AkihiroSuda

Implemented in #809
e.g.
# config.toml
...
# the first one is the default snapshotter
[[snapshotters]]
  name = "overlay"
  plugin = "snapshot-overlay"
# empty root -> system default
  root = ""

[[snapshotters]]
  name = "btrfs"
  plugin = "snapshot-btrfs"
  root = ""

[[snapshotters]]
  name = "overlay-alt"
  plugin = "snapshot-overlay"
  root = "/tmp/overlay-alt"

Now have support for this through multiple snapshotters, also each snapshotter can add its own custom configuration.


Welling/add publication ancillary

This PR adds the publication_ancillary 'assay type' to assay_types.yaml, on the road to support for Publications.

Drat, should have been merged against dev-integrate


parse with FormatItem::First fails early on UnexpectedTrailingCharacters

With the following FormatItem::First setup, an UnexpectedTrailingCharacters is returned due to the 2nd item parsing successfully but having remaining characters, even though the 3rd item would parse successfully.
use time::{format_description::FormatItem, macros::format_description, PrimitiveDateTime};

const TIMESTAMP_FORMAT_1: &[FormatItem] =
    format_description!("[year]-[month]-[day] [hour padding:none]:[minute]:[second]");

const TIMESTAMP_FORMAT_2: &[FormatItem] =
    format_description!("[day]/[month]/[year] [hour padding:none]:[minute]:[second]");

const TIMESTAMP_FORMAT_3: &[FormatItem] = format_description!(
    "[month padding:none]/[day padding:none]/[year] [hour padding:none repr:12]:[minute]:[second] [period]"
);

const TIMESTAMP_PARSER: FormatItem = FormatItem::First(&[
    FormatItem::Compound(TIMESTAMP_FORMAT_1),
    FormatItem::Compound(TIMESTAMP_FORMAT_2),
    FormatItem::Compound(TIMESTAMP_FORMAT_3),
]);

fn main() {
    let input = "12/12/2022 10:20:29 AM";
    dbg!(PrimitiveDateTime::parse(input, &TIMESTAMP_PARSER));
}

Link to playground
My expectation using this would use the 3rd item and ignore the UnexpectedTrailingCharacters error unless the 3rd item also failed to parse.
The simple solution for my specific case is to flip the 2nd and 3rd items, since they're unique enough for my use case. But I thought it worth reporting in case it is a bug and not the expected behaviour, or if i'm doing something wrong/unexpected with the api.

If I add a modifier try_complete:true/try_complete:false, with the default being false, would you view that as a reasonable solution?
The current behavior is false — the first match is returned no matter what. If it's true instead, then the parser would try to find the first match that has no input remaining, falling back to the first overall match otherwise.

e.g. the well_known items?

That's actually impossible, as they are their own types.

This would mean that making <Parsable>::parse work as (at least I would assume) intended

The current behavior is what was intended. Parsing is commonly performed on input that has text both before and after the value, and that is what I intended to facilitate.

This is also already true for the other FormatItem variants, though it's more subtle for them.

Only to the extent that the behavior is specified for formatting/parsing. There is no change of behavior depending on which method is used to invoke it.

Otherwise it might be enough to simply add some documentation about the limitations of using FormatItem::First with <Parsable>::parse so that it's obvious for anyone stumbling across it (like i did) that ordering the items to remove the potential for premature UnexpectedTrailingCharacters is important.

This can be done regardless of any changes elsewhere.

My original thinking was to have it be a modifier on [first]. Thinking about it again, I see an alternate path forward that's more flexible.
All that has to be done is introduce a component that indicates it's the end of the input. That, by itself, would permit the various use cases. The primary question I have is what it should be named. It will be quite easy to implement — it needs a new variant on error::ParseFromDescription and a declaration of the component itself.

I'm nowhere invested or opinionated enough to really push for one direction or the other :sweat_smile:. I wouldn't have had anything to say at all had I not been the one who posted the issue. And I'm certainly not familiar enough with all the possible use cases, so you wont get any flak from me for doing what you think is best.  but I'll try give a view from a casual user of the API (i.e. someone who generally just doing datetime parsing of plain strings).

The current behavior is what was intended. Parsing is commonly performed on input that has text both before and after the value, and that is what I intended to facilitate.

I struggle a bit with this in the context of <Parsable>::parse (or maybe I should be more specific and say <parsable::sealed::Sealed>::parse, since there is not actually a direct <Parsable>::parse, or even PrimitiveDateTime::parse or OffsetDateTime::parse since those are my usual entry points to the API, but i'll continue to use <Parsable>::parse for consistency and brevity) . For example, if I take something like:
PrimitiveDateTime::parse(
    "2021-01-02 00:00",
    &FormatItem::Compound(&fd::parse("[year]-[month]-[day] [hour]:[minute]")?),
);

and try to add anything to either the front or the back of the input string, I'll get a Result::Err, with no way to receive a potential result contained within. And as far as I can tell (based on some not so thorough grepping), the current implementation of <Parsable>::parse will always fail if there is extra text on either side of the input. I'm probably missing something, as these DateTime types use <Parsable>::parse_date_time which might have different expected behaviour with respect to extra input? though in that case i might expect the UnexpectedTrailingCharacters to come from that function, or the try_into() for the specific type, rather than the <Parsable>::parse function (but that also difficult since the information about how many characters have been consumed is not exposed in the <Parsable>::parse api.
But from a user perspective, things still work as expected in most cases PrimitiveDateTime::parse (and the other types that have a parse method) are failing if there is unexpected characters in the input, which fits with the experience of other APIs from other languages. It's only the case with FormatItem::First, which is already quite a unique API to have and maybe wasn't originally designed to be used in contexts like i've used (it was quite by chance I found it and decided to try use it vs manually looping over my possible cases to find the first one that matched), where it's not quite behaving as I expected. If an extra modifier to specify that a format description must consume the whole input existed when I came across the issue, I probably would have used it and moved on. But that is also assuming I found reference to the modifier before I found that rearranging the options also fixed it. I wouldn't immediately think that there would exist something to mark that a format description should consume the whole input, since with my simplistic use case, and my experience with other datetime parsing libraries in other languages I always expect the whole string to match.

An [end] component has been introduced, signaling the end of input. It will fail if there is any input remaining. For the example in your original issue, if you append [end] to each of the format_description! invocations, it will output what you're looking for.


Switch JWT library implementations

After reviewing the different libraries for constructing JWT authentication tokens, for the OnBehalfOf feature should use Nimbus JOSE + JWT
Exit Criteria

[x] Add a checkstyle rule to detect including references to org.apache.cxf.rs.security.jose
[x] Remove all references to org.apache.cxf.rs.security.jose [15] files impacted
[x] Replace all unfulfilled references with com.nimbusds.jose.* components
[ ] Add backwards combability tests for any features that have previously shipped

Additional Context

Prototype https://github.com/opensearch-project/security/compare/main...peternied:security:nimbus-jose-jwt

[Triage] This issue seems to have clearly outlined closure criteria and is the result of guidance around how to improve our overall security posture. We will need this to be backwards compatible so that the a token made with the CXF implementation can then go into the Nimbus version.
Given the prototype Peter made, this seems like something we can mark good first issue.

Started looking into it

Hi, here's the branch and draft PR I'm pushing my changes to: https://github.com/opensearch-project/security/pull/3421
So far so good although there are more differences between library implementations then I hoped for. Also, I noticed that nimbus library doesn't accept shorter keys for signing JWT, seems like cxf was adding padding if secret key was too short, nimbus on the other hand forces user to use proper length secret (512 bits for HMAC512 etc)

Hi, for visibility I'll write here instead of PR.
There are few differences between how both libraries dealt with implementing specification which caused issues.

For HMAC keys it's suggested that the key is at least as long as defined by the hash function. Shorter keys make it significantly to break hashing. The specification says that shorter keys should be padded: https://www.rfc-editor.org/rfc/rfc2104#section-2
CXF was apparently padding the keys, nibus on the other hand throws an error if key is too short. I added util method that pads key with "\0" which seems to do the trick. One thing I'm thinking about is introducing a utility class and moving the method there. https://github.com/MaciejMierzwa/security/blob/7e2c6ca4b9a4fa755e4ca9c81ff156e7f2eab3f1/src/main/java/org/opensearch/security/authtoken/jwt/JwtVendor.java#L108C26-L108C35
There is similar difference in implementation for audience claims. Specification suggests that audience claims is array of strings, in special cases it can be single string. Nimbus is parsing audience claims as array, no way around it: relevant part of library code. There are a lot of changes in tests because of that.
I removed cxf-rt-rs-security-jose package from dependencies. There are 3 other cxf packages that would be ok to remove. Was intention to remove all of those?

@cwperks
Seems like backward-compatibility tests are pretty flaky and are throwing different type of errors. Here's one that mentions build errors. Is the code compiled here different then the one compiled for unit testing?
https://github.com/opensearch-project/security/actions/runs/6420709692/job/17433407075?pr=3421
A couple commits before it actually passed: https://github.com/opensearch-project/security/actions/runs/6419670321/job/17430002039
Here is run that caused errors during serialization/deserialization communication between nodes., while no major changes in the code were made (style, changes in tests) https://github.com/opensearch-project/security/actions/runs/6407647266/job/17394880689
Overall seems like build artifacts from recent changes in the opensearch/os-security custom serialization might be causing it.

@MaciejMierzwa I've checked off a number of items from the issues description - there is only 1 outstanding item which is making sure that previous configurations are compatible with this change.  I believe the changes you made around how padding is handled should verify this - but I'd ask that we add a new tests that generates a token with CXF - save it into a string, then read it with the new system to ensure backwards compability - that should close out this issue,
Feel free to do that as part of your current PR [1] or open a new one afterward.

[1] https://github.com/opensearch-project/security/pull/3421

Hi @peternied, please take a look at this: https://github.com/MaciejMierzwa/security/tree/nimbus-jose-jwt-cxf-test
In commit: https://github.com/opensearch-project/security/commit/dd02d0c13995e30cfd6692ca86c02c5dbb3e23c0 I added proof of concept tests that are dealing with cxf/nimbus compability, how padding affects generated jwts etc. Since cxf library is supposed to be gone I guess there is no need to merge it to the main repo so I created separate branch for it.
Basically I tested 2 scenarios: Create jwt using cxf, encode it, parse it using nimbus or do it other way around. Only challenge is the padding that's added for case of too short secret - nimbus library wouldn't accept it, cxf would.

Basically I tested 2 scenarios: Create jwt using cxf, encode it, parse it using nimbus or do it other way around. Only challenge is the padding that's added for case of too short secret - nimbus library wouldn't accept it, cxf would.

@MaciejMierzwa In my mind, the test is to create a JWT with a key using the old system, then use the new system with the same key can still be able to process read + validate the contents of that JWT - is that possible?

Resolve by the merge of https://github.com/opensearch-project/security/pull/3421


Auto-detect on upload

Allow users to configure thumbor to automatically detect features/faces on uploading of new images.

hmm thats interesting :)


Passing a list of image files does not work and provide output in hocr

Environment

Tesseract Version: 20190526
Commit Number: N/A
Platform: Windows 32 & 64 bit

Current Behavior:
if a list of image files is passed to Tesseract, it is unable to process the list. Windows throws an "tesseract has stopped working" error.
Also, when providing a filelist, the output cannot be set to hocr.
Example command: tesseract listoffiles.txt output -l eng
Expected Behavior:
Tesseract should process the provided list of images.
Note: tesseract 4.0.0-beta.3 does not have this issue and processes the same command correctly.
Both versions however are not able to produce hocr files when provided a filelist.
Suggested Fix:
Unknown

Thank you for reporting this. I can reproduce this issue and will try to fix it.

What was the last installer which did not show this issue? I cannot find an installer for 4.0.0-beta.3.

Hi Stefan,
The version information shows the following:
[cid:53d04a40-8486-4c63-be18-8151bd57d8e9]
I suspect this version is downloaded from https://github.com/tdhintz/tesseract4win64
[https://avatars0.githubusercontent.com/u/10895586?s=400&v=4]https://github.com/tdhintz/tesseract4win64
tdhintz/tesseract4win64: charlesw/tesseract 4.0 build for x64 Windows using C++ run-time 141. - The world's leading software development platform · GitHubhttps://github.com/tdhintz/tesseract4win64
charlesw/tesseract 4.0 build for x64 and x86 Windows. This is a build of tesseract 4.0.0 beta, leptonica 1.77.0, and charlesw/tesseract .Net wrapper from September 10th sources. Tesseract 4.0.0 was built with OpenMP as follows ...
github.com
Regards,
Frank

Van: Stefan Weil<EMAIL_ADDRESS>Verzonden: zondag 23 juni 2019 15:39
Aan: UB-Mannheim/tesseract
CC: FrkBo; Author
Onderwerp: Re: [UB-Mannheim/tesseract] Passing a list of image files does not work and provide output in hocr (#13)
What was the last installer which did not show this issue? I cannot find an installer for 4.0.0-beta.3.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/UB-Mannheim/tesseract/issues/13?email_source=notifications&email_token=AKJYNNUMDGW4DDIEN4SKW2LP36KLJA5CNFSM4HYQSD6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYLBEKQ#issuecomment-504762922, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKJYNNSQTZCHKMEAIDBOBQLP36KLJANCNFSM4HYQSD6A.

I just released a new installer. My tests with it were successful. It also can create a hOCR file from a filelist.

Hi Stefan,
I can confirm the new installer works with filelists and hocr output! Thank you for time and effort in resolving this issue 🙂
Regards,
Frank

Van: Stefan Weil<EMAIL_ADDRESS>Verzonden: zondag 23 juni 2019 19:46
Aan: UB-Mannheim/tesseract
CC: FrkBo; Author
Onderwerp: Re: [UB-Mannheim/tesseract] Passing a list of image files does not work and provide output in hocr (#13)
I just released a new installer. My tests with it were successful. It also can create a hOCR file from a filelist.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/UB-Mannheim/tesseract/issues/13?email_source=notifications&email_token=AKJYNNS3DSQJPWFDV3E3XBTP37HJPA5CNFSM4HYQSD6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYLFTNA#issuecomment-504781236, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKJYNNX4XX2PRDPPQMUK2DLP37HJPANCNFSM4HYQSD6A.


Feature request: read from an abnf file

I would like to parse all the rules found in  https://github.com/dhall-lang/dhall-lang/blob/master/standard/dhall.abnf at once. My first attempt was with something like
with open("dhall.abnf", newline='') as f:
    contents = f.read()
    rules = re.split(r'\r\n\r\n', contents)

for rule in enumerate(rules):
    processed_rule = ... # strip comments and ignore subsequently empty rules
    Rule.create(processed_rule)

This naive approach didn't work, as there are several rules not separated by blank lines.
Would it be possible to either add a from_file class method, or generalize create to support something like
Rule.create(r'''; NOTE: There are many line endings in the wild
;
; See: https://en.wikipedia.org/wiki/Newline
;
; For simplicity this supports Unix and Windows line-endings, which are the most
; common
end-of-line =
      %x0A     ; "\n"
    / %x0D.0A  ; "\r\n"

; This rule matches all characters that are not:
;
; * not ASCII
; * not part of a surrogate pair
; * not a "non-character"
valid-non-ascii =
      %x80-D7FF
    ; %xD800-DFFF = surrogate pairs
    / %xE000-FFFD
    ; %xFFFE-FFFF = non-characters
    / %x10000-1FFFD''')

which would add two rules end-of-line and valid-non-ascii at once. I realize the above is missing the \r\n line endings dictated by ABNF; I was hoping that create could add them back similar to how it will append a \r\n if the string doesn't already end with one.
The end result would be the creation of a separate Rule instance for each grammar rule found in the given ABNF file.
A related issue would be the need to handle comment-only lines.

Actually, since my anticipated use would be something like
with open("dhall.abnf") as f:
    Rule.create(f.read())  # CRLF line endings are present in the argument

there's less need to supply CRLF if LF alone is used. For testing or simple grammars, being explicit
Rule.create('''; my grammar\r
foo = "1"\r
bar= "2"''')

would suffice.

Your request is an obvious hole in the interface, and you've offered a good example, as the contents of dhall.abnf can be parsed as an ABNF rulelist.
So I suggest a class method from_file(path: typing.Union(str, pathlib.Path)) -> None which supposes that the contents of path is an ABNF rulelist.  It would parse the rulelist and add the rules therein to the class.
I don't know that we want to attempt to outguess file creators with regard to line breaks.
I should also note that dhall.abnf mentions various expectations of the parser that mine may or may not meet.  These might end up driving another feature request.

As it turns out, dhall.abnf is not quite valid ABNF; unicode > 127 characters in comments are a problem.

I did miss that rulelist was a rule defined by ABNFGrammarRule, and was able to use that to define something very similar to create:
class Dhall(Rule):
    pass

with open("dhall.abnf", newline='') as f:
    contents = f.read()
parse_tree, start = ABNFGrammarRule("rulelist").parse(contents)
visitor = ABNFGrammarRuleNodeVisitor(Dhall)
visitor.visit(parse_tree)
for c in parse_tree.children:  # I think this loop would be vist_rulelist
    visitor.visit(c)

Having done that, I can write
node, _ = Dhall('complete-expression').parse('let x = 1 in x')

which produces a syntax tree, though not the correct one, likely due to the unmet expectations you mentioned.

I have an implementation of from_file working.  I worked my way down through the grammar, and may see what is happening here.  Dhall('let-binding').parse('let x = 1 in x') consumes the entire source.  This appears to happen because the parser is not recognizing 'in' as a keyword.

I believe I can deal with this situation by extending the parser to support negation and first-match alternation.  By adding this in, I was able to successfully parse the dhall examples I tried.

Rule.from_file is implemented.


feat: Add support for fvm

Is your feature request related to a problem? Please describe.
When I have been creating projects I have been using fvm to keep the flutter version set on a project level rather than on an environment level
Describe the solution you'd like
I was thinking of adding a flag to support the version of flutter you require on the project. This would then create the required .fvm folder with the required version. e.g.,
very_good create my_app --fvm stable
very_good create my_app --fvm 2.0.1
it could also check if fvm is installed or check that the version specified is installed and install it.
Describe alternatives you've considered
The alternative is to manually do this step
Additional context
This may have been something already considered and not appropriate for this tool. i.e., not wanting to take a dependency on fvm. Or not within the scope of this CLI tool - setting up the environment.
This will have a knock-on effect on the other package dependencies, so maybe too much as we will always have the latest Flutter version. In which case it could just set up the fvm folder for the support version of flutter of the bloc version. Sorry that may have sounded more complicated

Hi @b099l3 👋
Thanks for opening an issue!
As stated in our contributing guidelines we are currently not accepting feature requests.

At this time, we welcome bug tickets but will not be accepting feature requests because the roadmap and scope of this project is still being defined. In addition, we understand that Very Good Core is basic and lacks many features that might be desirable in a starter application but this is a conscious decision made to keep the scope of Very Good Core focused to just foundational items. In the future, we may support additional, more feature-rich templates but at this time we would ask that issues be reserved for bugs only.

Sorry for any inconvenience!

No problem, sorry this is my fault for not reading the contributing guidelines.

No problem, sorry this is my fault for not reading the contributing guidelines.

No worries! Thanks for opening an issue and we will keep this in mind for once the roadmap for very good cli has been finalized 👍


Is it possible to create a bar chart with different colored columns?

BTW I'm loving this library - it's so easy to use and when you do a Wicket AJAX update the charts re-animate like they're supposed to without me entering the el-weirdo world of JavaScript animation!
I've looked at the source code and BarDataSet's base class seems to only have a single set of colors that apply to all columns.
I've seen raw Chart JS bar charts that do have different colored columns. Is there a way to achieve this through Wicket Chart JS?

Ah... unless we're supposed to add multiple DataSets... yeah that probably makes sense! :)

@chrisco484 Thanks a lot first of all! So, is using multiple data sets an option that does solve your issue?

@pingunaut  Yes, I ended up using multiple datasets to get the different colors - but doing that is what made getting the Legend working important :)
My solution in the end was to just render my own legend in a simple HTML table that I place above the Bar chart in my main panel - works well but not perfect :)


Feature: use already existing SOCKS proxy

Hi,
first of all thanks for this great plugin :-)
It would be awesome if there is a setting like
knife[:bastion_skip_proxy_creation] = true

Most of the time I already have a project ssh config setting up various SOCKS proxies over SSH and keep them open for various other tools.

Hello,
I am attempting to run "knife bastion start" and continue to receive an error that the publickey is incorrect. We have existing bastion hosts in public subnets on AWS and are trying to reach the web servers, which are located in private subnets. I am able to ssh directly into the bastions but knife-bastion is not working.
I have setup the knife.rb file as dictated and set the bastion_users to the appropriate usernames. In this case, it would be ubuntu. The error output says that it is attempting to connect via<EMAIL_ADDRESS>rather than<EMAIL_ADDRESS>No matter what I set the port to or how I update the security groups, when I run knife bastion status, I receive an error that says tunnel is not open. (Please note that we have had no issues bootstrapping our servers manually using knife-bastion)
Here is the knife.rb config:
knife[:bastion_host] = "bastion-dns.amazonaws.com"
knife[:bastion_user] = ENV["ubuntu"] || ENV["ubuntu"] || ENV["ubuntu"]

# If you have multiple networks, that require different MFA tokens, specify
# each network name here. (This configuration is referenced to clarify the
# token a user should employ.)
#knife[:bastion_network] = "mynetwork"

# By default, the proxy server is created on port 4443. You may configure the
# local bastion port here:
knife[:bastion_local_port] = 4443

require "knife-bastion/activate"

and here is the error that I am receiving when running knife bastion start:
Creating a tunnel to Chef server:
  * Bastion host<EMAIL_ADDRESS>  *    Chef host: correct chef fqdn
  *   Local port: 443
  *      Timeout: 600 seconds
Establishing connection to bastiondns
<EMAIL_ADDRESS>Permission denied (publickey).


doc(README.md): Add some aesthetical styling

Hi @pbkwee !
Thanks for this nice script which already brought me invaluable time saver to confinue my mid-inflight 32bit to 64bit conversion.
In this pull request I added some basic Markdown styling to please my typographic nerd in me.

Merging.  I'll subsequently make a change to get rid of backticks leaking into the rendered doc.

You've got eagle eyes. My bad, i didn't spot that unwanted change as result of my regex search & replace. Thanks!


Look into proxying inventory and host cluster APIs through Express to avoid SSL/CORS errors

Based on a discussion with @fdupont-redhat, it seems like making our inventory and cluster API requests from the NodeJS side instead of the browser side could circumvent the SSL cert errors we have to show the user and prevent them from having to accept self-signed certs themselves, etc. We could just proxy /api/inventory/* and /api/cluster/* or whatever through Express to point to the real endpoints and then the browser would do everything through the single origin.

In order to prevent the same self-signed cert issues from blocking the proxied requests in Node, we'll need the operator to generate a CA cert file that we can point Node to. There are multiple ways to do this but I think the simplest would be to use the NODE_EXTRA_CA_CERTS environment variable. https://medium.com/trabe/monkey-patching-tls-in-node-js-to-support-self-signed-certificates-with-custom-root-cas-25c7396dfd2a

Already done in https://github.com/konveyor/virt-operator/pull/22. It provides the self-signed CA certificates for:

openshift-service-serving-signer@xxxxxxxxxx that signs the services internally
ingress-operator@xxxxxxxxxx that signs the routes when using OpenShift generated certs
*.apps.cluster-xxx.example.com, signed by ingress-operator for all the other applications exposed by a route

If Node proxies internal services, we don't even need these because our service are not using TLS internally. It is the ingress service that provides TLS termination, but then inside the namespace the traffic is unencrypted.
We can change that by using service serving certificates to provide a certificate to each service: inventory, xavier-analytics, UI, for added security. The concerned services will require the following changes:

pod: listen on 8443 and use cert/key provided in a secret
service: listen on 8443 and forward traffic to 8443
operator: annotate the service to generate service serving certificate secret
operator: update deployment to mount the service service certificate secret in the pod

I've created the following PRs to enable TLS for the inventory container, service:

https://github.com/konveyor/controller/pull/29
https://github.com/konveyor/virt-operator/pull/29

If you implement the proxy, you can point directly to the internal service: https://inventory:8443 and the TLS validation should work as the NODE_EXTRA_CA_CERTS variable already points to the CA certificates store.

Nice! yeah, we'd still need the proxy to take advantage of this, because I don't think we can ship that CA to the browser and inject it (nor should we, probably). This is great, it sounds like the simplest solution is just to proxy to internal services and not bother with TLS at that layer, but we can still provide Node with the certs in case we need them in the future, we may as well if they're there.

Removing the post-beta label, we should do this ASAP.

From the list of APIs in virtMeta.json, the following internal endpoints can be used by the proxy:

inventoryApi - http://inventory.openshift-migration.svc.cluster.local
inventoryPayloadApi - http://inventory-payload.openshift-migration.svc.cluster.local:8080
clusterApi - https://api.openshift-apiserver.svc.cluster.local

For HTTPS services, the CA certficates are in the /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt file.

Actually the inventoryPayloadApi is service is secured has it doesn't require authentication.
This leaves the payload open in the wild.
Although an overall solution would to secure the service it could work with this proxy solution by being restricted only virt-ui pod or at most to internal 'virt' network.

Mike and myself looked at the available packages to help implement this and we choose to use http-proxy-middleware because of its powered by http-proxy which makes it easily to integrate within express as a middleware.

@fdupont-redhat, @mturley, with the backend having now SSL free endpoints, we don't have to worry about NODE_EXTRA_CA_CERTS to point CA certificates.

The NODE_EXTRA_CA_CERTS environment is added by the operator, so you don't need to care about it in the Node configuration. And at least one internal service uses TLS: https://api.openshift-apiserver.svc.cluster.local. In https://github.com/konveyor/virt-ui/pull/214, I see that it sets secure: false, but it shoud be true in production.
I've also noticed in https://github.com/konveyor/virt-ui/pull/214 that the target is different in production and dev. Shouldn't it be set by virtMeta.json in every environment. In production, the value can be changed to the same values as the one that are hard coded. It's a small change in the operator to update the Config Map and konveyor/virt-operator#39 implements it.

@fdupont-redhat I was assuming since the local URLs would always be the same that they should be hard-coded. But it does make sense to let them be configured in virtMeta either way.
As for secure: false, sorry, I assumed we wouldn't use TLS internally at all in prod.

Well, the UI is actually configured to use TLS also internally and I've a PR opened to do the same for inventory. This will become the default pattern: HTTPS Everywhere.
Regarding, the hard coded URL, they can change based on the namespace where the operator is installed. It may not be openshift-migration.

Ok. Sounds good, we'll fix it. Thanks @fdupont-redhat

@fdupont-redhat, We'll use virtMeta since it becomes the reference and use secure: true (the default) for production.

We just need to also change the controller to put the .local URLs into virtMeta instead of the remote ones.

Per @fdupont-redhat , we should have the proxy middleware use logLevel: 'debug' if process.env.DEBUG is set, even in prod mode.

@mturley, good point. Just added it.


SCHEDULER : Publish current event

Hello,
thanks for providing this node, it works really well so far.
Is there a way to publish the current event?
Example:  I have configured

SUNRISE will define DAYLIGHT as payload
SUNSET will define NIGHT as payload

If I restart the system at 11am, NR will not now the current event .. which is of course DAYLIGHT
Please let me know if there is a possibility to publish the current event? .. or if this can be provided in a future release
thx a lot

Sorry. I was not clear enough before ... the daylight was just an example
I woul need an "UPDATE" trigger for the schedule node which is pushing thru the current event.
Another example:
I have a schedule defined from NOON time till SUNRISE start to run my Pool pump, as this is peak time for my solar panels.
In case I have to restart my stystem between NOON and SUNRISE, than my Pool pump will not start with your current scheduler. I would need to inject an UPDATE trigger to the scheduler to publish the current event and activate my Pool pump.
I have dozents of these events spcified and in normal operation the scheduler works fine, but sometimes it is necessary to publish the current event. It should be easily done internally as you already show the next event in the Node-Status.
Thanks for considering

I woul need an "UPDATE" trigger for the schedule node which is pushing thru the current event.

There is no "current" event existing. Events are single moments in time at which the node triggers something. What you request would be the last triggered event but the node does not remember events from the past, it triggers at certain moments and then forgets about the event.

I have a schedule defined from NOON time till SUNRISE start to run my Pool pump, as this is peak time for my solar panels.

As stated above, events are single moments in time and not time ranges, but I assume that you turn on your pump at noon and turn it off at sunrise (do you really mean sunrise or do you mean sunset?). So when you restart Node-RED within that time range, there shouldn't be a problem (your pump should continue to run), only if the down time of Node-RED would cover one of the time range borders.

It should be easily done internally as you already show the next event in the Node-Status.

Yes, the node knows about all upcoming events but not about past events. I don't see the point how triggering the next event could help (although implementing this would indeed be an easy thing).

I had some more thoughts about the topic and I can confirm that implementing an "UPDATE" trigger for the next upcoming event is easy to implement (as stated above), so I can add this in the next version.
But what is more or less impossible is to provide an "UPDATE" trigger for the previous event (i.e., the most recently past event). This is because scheduler node does not persist the actual time of past events (as already stated above) and it's also not possible to calculate past events (for instance because triggers could have a random offset and we do not know, what that random offset was or could have been; additionally, for crontable triggers, there is no API existing to fetch events from the past, only events from the future).

The new release 1.21.0 hopefully solves your issue. On one side, there is the new command trigger:next for scheduler node which triggers the next upcoming event. On the other hand, I added a brand new node, the so called state node, which internally maintains a state and can therefore provide the state value at any time. Feel free to try it out!


Fix removal of custom check from rarfile dependency

setup.py should also be updated to require >3.1 - see #1028 1028

legendastv Provider was removed


Fixing State Data

We used the wrong state data. Importing and merging correct state data (with lat, long.)

Close issue #3


updates for padding and displaying fail/pass

Ah, what I meant to send over this morning. 😅

padding is fine now, but I have again (we got it previously but I don't remember when/how it was fixed; there is not this issue without this last commit) the weird issue when I click on the arrow:

not nested by default
I click on the arrow, the rule name becomes transparent (or white on white)
if I select it, black comes back
Happens only inside the software, not if exported and displayed by a browser.

That is a weird bug, I'll try to tidy up the CSS. The default nesting shouldn't be in this PR. Right now it's all running recursively and following the same rules so I have to break the first out of that pattern.

The default nesting shouldn't be in this PR

Misunderstanding: it is not nested by default, as before. when I click for nesting, the rule name text color changes to white.

Yeah that's weird and hard to replicate but I'll see if I can fix it. It seemed to be "magic" before, so I will perform "magic" again.

The nesting as you prefer should be added in there and the HTML is valid! 👍

Guess that setting the arrow as "transparent" may be throwing it off, although that seems to be supported widely in browsers, so I changed it to white. Let me know how that looks.

I still have the issue with the text disappearing.
Other items (type/run/fail/pass hidden etc) are OK.
Still missing:

if fail and actual value, the value in the main line (after "fail") as in previous version

bases policy outcome on @outcome rather than fail count, shows actual if available at top level fail

Confirmed to work.

Still the transparency issue :(


tagCycle defined shortcuts not recognized

They seem to be defined correctly, but upon triggering them, nothing happens
Reported by Ekuboh on slack
Steps to reproduce

Create tags in roam/js/tag-cycle

- CTRL+Enter HASH
    - TODO
    - DONE
- alt+shift+z
    - 1
    - 2
    - 3

Press defined keystroke

works on my end


Refactor: Dockerimage

This PR refactors the docker image to reduce its size dramatically. Dive analyzed it as only 464MB.
# TODO
- [ ] figure out how to get the other db drivers to work, there are some native dependencies missing
- [ ] test with all dbms

fixes #4924

Is git actually required in the image now? I would say yes, as some plugins are installed via git.

Is git actually required in the image now? I would say yes, as some plugins are installed via git.

To install via git we would need the git executable, yeah

Now the image weights about 629MB. I'd love if we could reduce the needed dependencies for the database drivers.

What's really interesting and what could be used to consider using an alpine image would be this Dockerfile. It's an other open source project that's also based on Django and also has to deal with lots of images (from your cooked meals) and also has ldap support already built in and weights out 350mb (uncompressed).

@wolflu05 two things immediately stick out in the linked image

Running purge step after install
Running all install steps on the one line

Might be worth applying that to our image

Running purge step after install

I already do this as much as I can.

Running all install steps on the one line

I tried squashing all image layers, but this also doesn't reduce the size of the image.

I just tried how alpine turns out. Here is a little comparison with the current  state of pushed files:

Name
Size
Build time *

Old
1.03GB
119.0s

New debian slim based
778MB
135.8s

New alpine based
556MB
141.8s

* all builds on MacBook Pro 2019 (docker cache disabled)

The alpine image is looking pretty appealing then!

FYI, adding LDAP needed packages adds about 5MB to the alpine image, so it weights: 561MB. And about 5MB to the Debian image so it weights: 781MB.

Interested to hear what @matmair has to say to this alpine approach.

It certainly saves a lot of bandwidth. And the LDAP addition seems minor, but I have no horse in that race :)

If it works why not. Have you verified that ldap actually resolved all objectTypes and both ldap and ldaps with the installed dependencies?
The size reduction is certainly impressive. Is there something significantly different to the linked project? They seem to be even smaller so I wonder how much we need to change to achieve that size. With the amount the image gets pulled this PR certainly has positive carbon-footprint implications.
The change from Debian to Alpin should to be marked breaking and be mentioned in the release notes, plugins will need to adapt their (native) install instructions to work on the docker as apt will not work.

If it works why not. Have you verified that ldap actually resolved all objectTypes and both ldap and ldaps with the installed dependencies?

Not fully, need some further testing.

The size reduction is very impressive. Is there something significantly different to the linked project? They seem to be even smaller so I wonder how much we need to change to achieve that size. With the amount the image gets pulled this PR certainly has positive carbon-footprint implications.

They don't use lots of things we need:

weasyprint
mariadb
git
gnupg
bash (maybe we rewrite the init script to use ash)

The change from Debian to Alpin should to be marked breaking and mentioned in the release notes, plugins will need to adapt their (native) install instructions to work on the docker as apt will not work.

Yes definitely it will break some things.
Idea: provide both alpine and Debian for the next release for a smooth transition period until plugins are adapted.
Further analysis needed: Does switching out the images work easily with an already installed InvenTree. Maybe the venv that's mounted via volume contains some files that are not compatible with alpine.

Further analysis needed: Does switching out the images work easily with an already installed InvenTree. Maybe the venv that's mounted via volume contains some files that are not compatible with alpine.

We definitely need to investigate this before we push this :)

Oh and I forgot the biggest point why our alpine image is bigger than tandoors. Our codebase weights about 90MB with all the translated templates and po files.

Oh and I forgot the biggest point why our alpine image is bigger than tandoors. Our codebase weights about 90MB with all the translated templates and po files.

That should be solved by #2789 as we will only ship messages

@wolflu05 I think we can remove gnupg - we do not ship the code that used it anymore.
Would be interested in your thoughts on https://github.com/inventree/InvenTree/pull/5017 - background

@wolflu05 just a friendly bump to see where you are up to with this :)

Need to remove gnupg and do some more testing with other db drivers and also with existing installs.

@wolflu05 do you want/need any support on this one? Would like to use this for #5011

I'll need to update a few things and then would be good if you could test it too so we don't oversee something

Should I replace the old dockerfile and put the dev thing into a Dockerfile.dev so we still use Debian for development?

Should I replace the old Dockerfile with the alpine one and put the dev thing into a Dockerfile.dev so we still use Debian for development?
should I remove bash from the alpine image and rewrite the init script to use ash so we even save some more mb?

No, I don't think we should have a different base image for dev or prod
I think so, I don't see any real reason to keep bash

So we are clear: We are fully switching to Alpine? I am implementing the build step for react rn and do not want to use the wrong image ;-)

I would think so from my side. What do you mean @SchrodingersGat .
But nevertheless, it's totally irrelevant for you with what base image we went later. You can use the FROM node:18 as frontend_builder image for building and then a COPY --from frontend_builder <...> <...> to copy in whatever our prod image is.

The question is in which file as there is Dockerfile and Dockerfile.alpine. I will do it in the current file for now.

I would plan to remove the .alpine extension (replace the old file), was first just for trying it out to keep both side by side for comparison.

Are we still want to support the dev target from our current image? I don't think it's necessary as we have devcontainer and native install.

I would think so from my side. What do you mean @SchrodingersGat .

I support moving entirely to alpine for base image

Are we still want to support the dev target from our current image? I don't think it's necessary as we have devcontainer and native install.

I suppose this is a fair point? But is there a downside to leaving it in, at least for a while, for anyone who would still be using this?

I suppose this is a fair point? But is there a downside to leaving it in, at least for a while, for anyone who would still be using this?

I plan to replace the current Dockerfile completely with the .alpine one. And in the alpine one is no dev target. There is just the production target. So it would be more effort to recreate a dev target with alpine smoothly.

@matmair @SchrodingersGat I'm done now with the changes I wanted to do. The image now weights 524MB. The biggest things are (estimated):

weasyprint, fonts, ... (~100MB)
mysql related stuff (~70MB)
translated js templates (~50MB)
git (~20MB)

🚧 Help needed
I already test a lot of scenarios with existing installs, other dbms, already installed plugins, ... and everything worked so far. Before this is ready to merge, I'd like to ask if you could test this too so we do not overlook something.

@wolflu05 awesome :) I'll take a look at this tonight

The development image works as expected on my end. @matmair any final thoughts before we merge this in?

LGTM; I take it that we are fully removing support for the Raspberry Pi architecture with this as the package resources are removed?

LGTM; I take it that we are fully removing support for the Raspberry Pi architecture with this as the package resources are removed?

@wolflu05 does removing RPi support gain us any space? I would imagine not? Perhaps it is beneficial to leave it in for anyone who wants to target raspbian

Not sure, haven't tested, because I have no access to one, should I add it and Someone of you test it?

Not sure, haven't tested, because I have no access to one, should I add it and Someone of you test it?

I don't have a way to test it really, but I think maybe we should just leave those lines in. Unless you are building to target raspbian, they don't do anything

@wolflu05 thanks for the fine work here :)

@wolflu05 this PR truly was a breaking change: https://github.com/inventree/InvenTree/actions/runs/5506646753/jobs/10035587151
Looks like we did not run through the complete set of unit tests


Make multiple "Matches" type methods

Something like
Matches, MatchesAny, Contains, MatchesAnyNamePart, NamePartContains
NamePartContainswhich is a shitty name would be FirstName: "Mary Elizabeth"  matches both "Mary" and "Elizabeth"
These would build off of the existing Comparers similar to the addition of Contains

Matches ❌ - Should ignore order
MatchesAny ✔
Contains ✔
MatchesIgnoreOrder ❌ - Needs to work how matches works now


added user testimonial and achievement page

#1137
one can access this page from

working demo:
https://www.loom.com/share/1e7a3387a8924b10a183a6148b3b0c27?sid=f1e99dab-a68c-46d6-aa54-db27ab332646
some screenshot

working demo:
https://www.loom.com/share/1e7a3387a8924b10a183a6148b3b0c27?sid=f1e99dab-a68c-46d6-aa54-db27ab332646

@sanjay-kv sir please add Gssoc label


remove christina

Docs Build status updates of commit 61335c8:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

advocates/index.html.yml
:white_check_mark:Succeeded
View

advocates/toc.yml
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


request: spectrograph~ add a flag so that if one clicks on a bicoeff that is on top of the spectrograph~ - the spectrograph~ does not act like you clicked the array

currently if i use a spectrograph~ underneath a bicoeff - if i click-and-drag around in the bicoeff, the spectrograph~ acts like i've clicked content inside the array.

is there a possibility of omitting this type of "drawing" from leaking into the spectrograph~ device? or being shown in the spectrograph~ device

why is it underneath?

why is it underneath?

because while it is underneath, it allows me to see what the results are

i.e. i'm using the spectrograph~ below the bicoeff to show me the results of the eq..

that looks cool

that looks cool

thanks! unfortunately, having 16 of them results in the updates getting in very slowly. when i delete all 16 of them and just have one running at 1ms and 128 fft chunk, it is "realtime-feeling", but compared to having 16 of them - 16 results in them being very, very, very sluggish (like in the gif).
yeah, this kind of "peak eq + spectrum analysis" is kinda really what i'm after, as it really helps with figuring out which bits to notch up and down


Patch docker-compose timing issues in export/rules-engine

Fix #1544
The primary issue is observed when one runs "docker-compose up" after
pruning local docker volumes.
The following problematic symptoms have been observed. The requisite
code changes in this PR are enumerated below. I hesitate to properly
call this a "fix" because I think more analysis is warranted for a long
term solution. However for the purpose of stabilizing the release, this
should suffice.

Observed that population of configuration data in Consul and
subsequent creation of DBs/Collections and credentials in Mongo were
delayed when creating volumes for the first time. This lead to timing
issues in the following areas
I observed conditions where the Config-Seed was populating Consul for
a given service at the same time that that service's configuration was
being read into Consul by the config-seed. Thus services would come
up and read their configuration from Consul when that config was only
partially populated. In Delhi this would not be a problem because the
listenForConfigChanges() function would listen for a change anywhere
in the service's config in Consul. This would cause the entirety of
the ConfigurationStruct to be populated when the last key was written.
In Edinburgh, we only listen for changes to the "Writable" section,
which is written first.
-- This is why initializeConfiguration() has been modified to look at
the last value of the service's configuration within Consul to see
if it has been populated. If it is not, then we throw an error --
forcing a retry one second later. If it is populated, the whole
config has been written to Consul and we can proceed.
All services write logging information when they are bootstrapping.
If the service is configured to log remotely, those calls go to
support-logging. Because of the observed latency (up to 5 seconds) in
populating configuration and Mongo with new volumes, any service trying
to send its startup messages to support-logging would throw an error.
This lead to many, mnay errors in the starup process. Once support-
logging came up, the remote logging would work. Understand that every
service must receive its configuration information from Consul and,
where applicable, establish DB connectivity before it activates its
API handlers. If a service calls support-logging (or any other service)
before the API is enabled, the caller will receive a "connection re-
fused".
-- This was also mitigated by the code change above.
In the case where a service is delayed in its startup, it will not
register its endpoint with Consul. Export-client does not require a
database connection and so it will come up before export-distro. It
was observed that the rules-engine registers with export-client in
order to receive events. As a result of that, export-client attempts
to POST a notification to export-distro. Because export-distro had not
registered itself with Consul at the time export-client was being
configured, the endpoint information would be blank, leading to an
error like this (some fields omitted):
level=ERROR app=edgex-export-client source=registration.go:310
msg="error from distro: Put http://:0/api/v1/notify/registrations:
dial tcp :0: connect: connection refused"
Noticee the the host and port.
-- The code change to address this can be found in
internal/pkg/startup/endpoint.go
-- This only works due to the non-HA mode in which we deploy from
docker-compose. Right now, what's in the docker configuration is
always the same as what is (or will be) in Consul.
-- Note also that there is a circular dependency between distro/client
w/r/t how they call each other. Because the result of a notification
sent to distro in the above case is that distro simply calls back
to export-client.

Signed-off-by: Trevor Conn<EMAIL_ADDRESS>

Codecov Report

Merging #1550 into edinburgh will decrease coverage by 0.02%.
The diff coverage is 4%.

@@              Coverage Diff              @@
##           edinburgh    #1550      +/-   ##
=============================================
- Coverage      17.05%   17.03%   -0.03%
=============================================
  Files             78       78
  Lines           8790     8807      +17
=============================================
+ Hits            1499     1500       +1
- Misses          7141     7157      +16
  Partials         150      150

Impacted Files
Coverage Δ

internal/support/logging/init.go
0% <0%> (ø)
:arrow_up:

internal/pkg/startup/endpoint.go
0% <0%> (ø)
:arrow_up:

internal/core/metadata/init.go
0% <0%> (ø)
:arrow_up:

internal/core/data/init.go
0% <0%> (ø)
:arrow_up:

internal/support/scheduler/init.go
0% <0%> (ø)
:arrow_up:

internal/export/distro/init.go
0% <0%> (ø)
:arrow_up:

internal/core/command/init.go
0% <0%> (ø)
:arrow_up:

internal/export/client/init.go
0% <0%> (ø)
:arrow_up:

internal/export/client/registration.go
77.48% <100%> (+0.11%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c745a94...3c94f29. Read the comment docs.

@lenny-intel I want to make sure you saw the email I sent and have gone through the verification instructions therein. We need to be sure. Right now, AFAIK I'm the only one that has actually run the fix.

I still get “unhealthy” on the logging service via Portainer. Did you intend to leave that new indication in the logging service?

@jpwhitemn You must still be using that original docker-compose.yaml I sent you when we were troubleshooting then.

@tsconn23 , I was able to test this yesterday. After clean start (i.e. volume prune) the data is flowing to Rules Engine properly and all services are healthy except scheduler. If I then restart without prune the scheduler is healthy. When unhealthy, the only thing I see in the logs is:
level=ERROR ts=2019-07-17T21:59:36.673704006Z app=edgex-support-scheduler source=main.go:80 msg="edgex-support-scheduler: Service bootstrap failed!"

I reran this scenario multiple times and get the same result each time.

@xmlviking - can you help take a look
On Thursday, July 18, 2019, 9:23:42 AM CDT, Lenny Goodell<EMAIL_ADDRESS>wrote:

@tsconn23 , I was able to test this yesterday. After clean start (i.e. volume prune) the data is flowing to Rules Engine properly and all services are healthy except scheduler. If I then restart without prune the scheduler is healthy. When unhealthy, the only thing I see in the logs is:
level=ERROR ts=2019-07-17T21:59:36.673704006Z app=edgex-support-scheduler source=main.go:80 msg="edgex-support-scheduler: Service bootstrap failed!"
I reran this scenario multiple times and get the same result each time.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

@tsconn23 , @jpwhitemn , the support schedule issue occurs every time do a clean start. I resolved the issue by adding restart: unless-stopped to scheduler in the compose file. Adding this to all the appropriate service would make bootstrap more robust.

I'm testing this on my PC as well with the Edinburgh release versions of the compose files.  Trying to to trouble shoot any timing related issues with  Scheduler or potentially any of  default interval/intervalActions  or configured interval, intervalActions it's loading at boot strap.


Feature - Delete primary scenario

By the way, it s a good improvement to share. can you add an entry in the release notes (relnotes.md file) for Core 1.2 community edition?