Class vs ABC vs metaclass vs decorator vs __init_subclass__

Should the HWAddress base class be changed to an abstract base class, a metaclass, or a decorator? (Edit: or get an __init_subclass__ method?)

I'm leaning more toward the metaclass option because:

Similar to the ipaddress, I don't expect users to create any subclass from any of the concrete classes provided by the library.
Even if they do, the possibility of replacing metaclass is very unlikely since the usage of metaclass is already rare.

Correct me if I'm wrong, but using metaclass would allow you to reuse the comparison implementation between concrete classes without worrying about entangling concrete classes that are meant to be differentiable. For example, if there is a concrete class called CHEESE and it has identical structure to MAC, but CHEESE and MAC are not meant to be compared at all. In this case, it would require using special attributes of each of them, which may or may not exist in other classes. That would be a hassle.
But again, this is only me speaking from a generic and designing point of view of a software developer who does not have a knack for networking. Maybe IEEE won't ever release anything else in another 30 years and it does not worth the effort to consider such extensibility and openess.

I just think that hardware address is a very broad term and the concrete classes should not be able to interchangeable between one and another. A very good example IPv4Address and IPv6Address. They are both IP addresses, but they certainly can not be interchangeable when doing some abstract programming (they both share the same parent _BaseAddress, but at least it's private and users are not encourage to use it for type hinting). You have to know specifically that you're dealing with IPv4 or IPv6 or both of them. It's tedious to manually handle them separately (not always, but often enough), but your code has clarity.

Abstract base class is totally fine, but I just prefer to have my object's "family tree" flat :)

Jotting down some notes on the functional differences:

Metaclass or decorator allows us to run logic at hardware address class definition time, and so we can help users catch errors much sooner.

Decorator vs metaclass locality: a decorator only effects the thing that you decorate; a metaclass effects every subclass. So a metaclass can have "non-local" action: you can write a subclass which uses the metaclass of the parent without explicitly mentioning (or knowing of) the metaclass anywhere in that code.

Metaclass breaks multiple inheritance, except not even universally but as an invisible surprising landmine: two different base classes with two different metaclasses is not allowed by Python, as I understand it. Which means you can add a metaclass and suddenly stuff breaks in some multiply-inheriting subclass. (Can think of two workarounds which I haven't tested, but those both start by creating a new metaclass which inherits from both of the metaclasses.)

Metaclass ergonomics: class Bar(metaclass=Foo) vs class Bar(Foo). Relevant example is abc.ABC, which is a base class which has a metaclass which does all the actual adding of abstract base class logic.

So one good question to mix in: what kind of logic can we usefully do at class definition time, and for each of those, is it more useful to have that logic re-run or not re-run at subclass definition time?

For a while I thought of maybe adding size_in_bytes and size_in_nibbles (and maybe a more explicit size_in_bits alias for size).
And then I thought... why not let people define a class with their choice of size unit? size_in_bytes = 12 can be a lot more mentally ergonomic than size_in_bits = 96, especially when reading the code.
If I wanted to allow users the choice of defining hardware address classes in different size units, then a metaclass or decorator could do that, while an abstract base class couldn't.
But!
Nowadays I think all of this is better solved with either:

a separate library that provides numerical types with units, so people could do size = Bytes(8) where Bytes is defined somewhere as a unit with a factor of 8 relative to the base unit of a bit, or

a couple conversion functions: size = bits_from_bytes(8) and bytes_from_bits(mac.size), or

some mix of those two.

Because that's obviously the more composable solution.
So I don't think this exact example possibility needs to influence anything, just part of my my whole design thought process.

Notably, abc.ABC is (indirectly) a metaclass, and so the metaclass subclassing could bite people if they want to express something like:
I have a couple different address types, but they all have a certain common behavior. So I'll put the common behavior into a subclass of HWAddress, and I'll indicate that it's an ABC by multiply inheriting from abc.ABC, and... oh metaclass conflict. :(
This ties into the metaclass ergonomics thing from a couple comments ago, because what happens if `class Foo(BarWhichHasMetaclass, metaclass=Qux)

Noting that decorator and meta class don't offer any optimization or functional benefit besides the fail fast error detection thing, as far as I can tell.
I say this because my mind sometimes reaches for decorators or meta classes as the way to compute some information once from the information that the user specified when defining the class, and then have that information ready to go whenever it is desired. But all that can be computed on use, and cached on the instance or the class after first use if needed.

@gytqby Thank you for your comments on this by the way! Sorry it took me like three months to get back around to commenting on this. I did read your comments and appreciated them - they're definitely being factored into this whole design thought in my mind, and I do benefit from hearing them.

The metaclass vs ABC problems are largely solved by having the metaclass inherit from abc.ABCMeta.
Then multiply inheriting from abc.ABC and HWAddress works, I think, but even if it doesn't the solution is to just inherit from HWAddress, and it would be pretty clearly documented (or at least obvious in an IDE or when looking at the source ) that HWAddress is already an ABC, so you can just inherit from it if you're making an abstract base (sub)class of hardware addresses.

I think it's also a good point that most users won't be subclassing concrete classes, just as ipaddress users won't be.
(The one caveat is that subclassing is currently the only way to parse an address with custom formats. And there's at least one piece of code I've seen on GitHub that actually uses this capability. But that use-case would still work fine even if I used a metaclass or whatever.)

One of the biggest arguments against definition-time checks is that size should be optional for intermediate abstract classes which just want to define some behavior, and formats should be optional for quick-and-dirty stuff.
One of the biggest arguments for definition-time checks is making sure that if size and formats are both specified, they should agree with each other: I would love it if a hardware address class with size = 8 and formats = ('x',) raised an error about how the size and format don't match in the number of bits they represent.

Continuing from the comment in #18 where I just mentioned this:
The biggest good I see for doing a metaclass is enforcing that the number of x characters in each entry of formats matches the number of nibbles as implied by size.
So now here's a question? Is there ever a good reason for a subclass to have non-matching format strings? No.
In fact it is actually really bad because this is an error that disappears silently right now. Imagine I messed up and implemented EUI48 like this:
class EUI48(_StartsWithOUI):
    """48-Bit Extended Unique Identifier (EUI-48).
    EUI-48 is also the modern official name for what
    many people are used to calling a "MAC address".
    """

    __slots__ = ()

    size = 48

    formats = (
        'xx-xx-xx-xx-xx-xx',
        'xx:xx:xx:xx:xx:xx',
        'xxxx.xxxx.xxxx',
        'xxxxxxxxxxx',
    )

Do you see the error?
I probably would never spot it either:
>>> EUI48('01-23-45-67-89-ab')
EUI48('01-23-45-67-89-AB')
>>> EUI48('01:23:45:67:89:ab')
EUI48('01-23-45-67-89-AB')
>>> EUI48('0123.4567.89ab')
EUI48('01-23-45-67-89-AB')
>>> EUI48('0123456789ab')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/python3/lib/python3.10/site-packages/macaddress.py", line 100, in __init__
    self._address, _ = _parse(address, type(self))
  File "/home/user/python3/lib/python3.10/site-packages/macaddress.py", line 399, in _parse
    raise _value_error(string, 'cannot be parsed as', *classes)
ValueError: '0123456789ab' cannot be parsed as EUI48

Surprise!
>>> EUI48('0123456789a')
EUI48('00-12-34-56-78-9A')

Bad string parses, good string doesn't parse, and you don't know until it either suddenly and non-obviously errors out on good input or worse, silently wrongly parses wrong input.

I don't see any good reason to ever have formats that don't match the nibble size (the closest to a good reason which I can see looks more like a situation where it would be better if this library supported bitwise operators and/or prefix-cast-construction).
So I'm tempted to say that having a metaclass automatically check this is actually a great idea.

One abstract base class use-case that's been rattling around in my mind is to programmatically "document"/enforce clearly that a concrete hardware address class needs to provide a size.
Of course I can do this anyway in the __init__ provided by HWAddress, by raising a TypeError with a message like "hardware address class without size cannot be initialized" (although that's a little longer than I'd like it to be).
But ultimately the whole point of an ABC is precisely to express stuff like that. I could do it myself, but the Python ecosystem has standard language for it, and that language is an ABC.

Although the default error text is somewhat annoying:
TypeError: Can't instantiate abstract class Foo with abstract method size
It's not an abstract method, you know? It's an abstract property.
And if I'm going to be wrangling with abc.ABC to get it to use a slightly better error message, I almost might as well just skip using abc.ABC entirely.

Then again caring so much about the error text is probably the wrong move. In principle, it's probably better to delegate/abstract that to the abstract property implementation - trust that one day, Python will use a more appropriate adjusted error message in abstract property cases.

So if I was doing an abstract base class, I'd also do a metaclass, and I think my ideal looks something like this:
from abc import ABCMeta as _ABCMeta
from abc import abstractproperty as _abstractproperty

class HWAddressMeta(_ABCMeta):
    def __new__(metaclass, name, bases, namespace):
        dummy_class = type.__new__(metaclass, name, bases, namespace)
        if isinstance(dummy_class.size, _abstractproperty):
            formats = dummy_class.formats
            if len(formats):
                nibbles = sum(c == 'x' for c in formats[0])
                namespace['size'] = nibbles * 4
        cls = super().__new__(metaclass, name, bases, namespace)
        if isinstance(dummy_class.size, _abstractproperty):
            return cls
        nibbles_in_address = (cls.size + 3) >> 2
        bad_formats = []
        for format_ in cls.formats:
            nibbles_in_format = sum(c == 'x' for c in format_)
            if nibbles_in_format != nibbles_in_address:
                raise ValueError('bad format')
        return cls

and then update HWAddress like so:
class HWAddress(metaclass=HWAddressMeta):
     ...
    @_abstractproperty
    @classmethod
    def size(cls):
        """The size, in bits, of the hardware address class."""
        return None
    ...

What a mess! (Particularly the first half of __new__ in the metaclass.)

(I also haven't decided how I feel about abstractproperty vs the composition of property and abstractmethod - I feel like the first one has a more appropriate name, and makes way more sense in an isinstance check. Which, while we're at it, is a great example of why it's sometimes very useful to use non-flat class hierachies to communicate distinctions which might be useful in some cases.)

So out of all that dense and convoluted and hacky metaclass business, what do we actually get?

If any of the formats strings has inconsistent size, error.
If there is no size but there is formats, infer size from formats (if there are formats of different size, that will be caught by the previous point).
HWAddress subclasses with neither size nor formats refuse to instantiate.

Worth it? I don't know.

Notably, as of Python 3.6, we can use __init_subclass__ instead of metaclasses, to achieve all of the metaclass stuff.
With macaddress, I'm pretty fine with having features like these as progressive-enhancement non-EOL'ed Pythons, so I'm actually pretty okay with using __init_subclass__.

If we remove the "derive size from formats" convenience feature from the metaclass, it simplifies a lot:
class HWAddressMeta(_ABCMeta):
    def __new__(metaclass, name, bases, namespace):
        cls = super().__new__(metaclass, name, bases, namespace)
        if isinstance(cls.size, _abstractproperty):
            return cls
        nibbles_in_address = (cls.size + 3) >> 2
        for format_ in cls.formats:
            nibbles_in_format = sum(c == 'x' for c in format_)
            if nibbles_in_format != nibbles_in_address:
                raise ValueError('formats size mismatch')
        return cls

One gimmicky thing you can do if I keep allowing formats and size to mismatch:
class EUI48LowHalf(EUI48):
    formats = ('xx-xx-xx', 'xx:xx:xx', 'xxxxxx')

suffix = EUI48LowHalf('78-90-AB')

When would this ever be useful? Approximately never I think.
But imagine I also add bitwise operators, which is something I have been thinking about doing for at least half a year:
prefix = OUI('12-34-56')

eui48 = prefix | suffix
eui48 == EUI48('12-34-56-78-90-AB')

Boom, absolutely useless for the vast majority of developers using this library, but it's a neat emergent property, and I can imagine this being useful in a program managing MAC address assignments within a company or whatever.
Notably this scales for incremental parsing:
class IAB(HWAddress):
    size = 36
class IABOwner(IAB):
    formats = ('-xx-x',)
class DeviceID(EUI48):
    formats = ('x-xx',)

string = '01-23-45-67-89-AB'

oui = OUI(string[:8])
iab_owner = IABOwner(string[8:13])
device_id = DeviceID(string[13:])

eui48 = oui | iab_owner | device_id

But of course, first of all, why? When is this a thing that anyone would want to do?
Especially when we can just do this, and avoid two spurious weird classes which only exist to play weird formatting tricks:
string = '01-23-45-67-89-AB'
eui48 = EUI48(string)
iab = IAB(eui48)
oui = OUI(eui48)
iab_owner = iab ^ oui
device_id = eui48 ^ iab

(This is all assuming #8 is implemented, and bitwise operator overloads are implemented and cast their operands to the wider of the two classes as if the smaller one is a prefix of the larger one, mirroring how #8 works.)

In other news, looks like Python 3.11 has decided to break classmethod and property interaction.
This was already an only-recently-working sort-of-thing (in 3.9, classmethod finally learned how to play nice and transparently with the descriptor protocol - prior to that I don't even know what you could do, if anything), and now it's going away again.
Actually I had forgotten that classmethod-ness, abstract-ness, and property-ness only became composable so recently.
But in any case, I'm starting to feel that CPython is just not a stable language on the edges like it used to be. This isn't the first thing they've jiggled back and forth, in a way that just punishes early adoption trust. (The other one I can think of off the top of my head is type annotations, which changed in breaking ways in 3.10, and then there came lots of discussion of changing it yet again.) This is a super concerning feeling to have.

Okay, so I think anything involving class properties is out.
I could write my own custom read-only class property descriptor implementation - that isn't too hard:
class _classproperty:
    __slots__ = ('_method',)
    def __init__(self, method):
        self._method = method
    def __get__(self, obj, objtype=None):
        return self._method(objtype)

But then there's probably other downsides I'm not seeing yet, like static typing tools getting confused and all that.

This rules out something I was thinking about, which basically looked like making HWAddress.size an actually useful class property.
But that's probably for the best, because the only version of that which I'm happy with also included still using a metaclass like above to enforce some consistency checks on formats, and if I did use a class property for HWAddress.size then there would be non-local interplay to ensure correctness between that and HWAddressMeta.__new__.

My verdict so far is: either it stays a class, or I do __init_subclass__.
My emotional temptation is to just bury this whole idea for another year. Leave it as-is, with HWAddress as a normal class, and move on with my life.
This is currently holding me up pretty severely.

Here's the final little spurt of design exploration I'm willing to give this:
The most minimal version that achieves just the one biggest improvement - if formats has a format that doesn't match size, error out:
class HWAddress:
    def __init_subclass__(cls):
        errors = []
        for format_ in cls.formats:
            nibbles_in_address = (cls.size + 3) >> 2
            nibbles_in_format = sum(c == 'x' for c in format_)
            if nibbles_in_format < nibbles_in_address:
                errors.append(repr(format_) + ' is too short')
            elif nibbles_in_format > nibbles_in_address:
                errors.append(repr(format_) + ' is too long')
        if errors:
            name = cls.__name__
            details = '; '.join(errors)
            raise ValueError('bad formats for ' + name + ': ' + details)

Tentative verdict: that's going in.

Notably, this logic in __str__ is coupled with .formats[0] having the same number of x characters as the .size implies the address has nibbles:
...
offset = (4 - type(self).size) & 3
unconsumed_address_value = self._address << offset
for character in reversed(formats[0]):
   ...
...

If .formats[0] has fewer nibbles than the address, this would truncate the value; if has more nibbles, this would add trailing zeroes.
Seems like the sort of thing that's more likely to mask an error by silently doing the wrong thing than to be what the user wanted.
So having a "do formats match size?" check is valuable in that way too.

Reaffirming I'm fairly confidently settled into:

use __init_subclass__ for nice-to-have checks for developer errors (defining subclasses badly),
keep HWAddress as an abstract base class (it has always been semantically abstract, and in effect it can't be instantiated because __init__ always raises an error for it, or used much even if someone bypasses that, because it doesn't define a size), and
give up on metaclasses and the abc machinery.

With that settled, there's two things I currently have in mind for __init_subclass__:

verify that the number of "x" characters in formats matches the number of nibbles that are needed for size bits (example implementation in earlier comment), and
verify that once a HWAddress subclass defines with a size, subclasses of that subclass can't change size to another number.

I remember having three things in mind but I can't remember or reconstruct the third.

Actually, I feel enough design pressure to overcome the "oh no __init_subclass__ doesn't work on [ancient or obscure and incomplete Python]" hangup.
__init_subclass__ was added in 3.6, which came out in 2016! so long ago that already 3.7 is the oldest supported CPython. And I never supported Python 2 in this library.
I need to really ask myself if my portability obsession is a good reason to have a worse API, because __init_subclass__ is basically ideal otherwise.
It's not unreasonable to ask the handful of people who might use this library on Pythons with non-working __init_subclass__ support to deal with that problem. I can even help them deal with that problem by giving them a decorator, so that if they want that compatibility, they literally just need to slap the decorator on top of their subclasses. The README section for defining a subclass can just say "if you need to be portable to Pythons where __init_subclass__ does not work, decorate the subclass with @macaddress.hwaddress_init_subclass". In the rare case that portability matters that much, this is enough.

But it's still fair to say that that's foo's fault.


[Feature Request] Make it available from addons.mozilla.org

Hi
This addon works OK for me with Android Firefox-Beta using a "Custom Add-on collection".
Please fix it to work with addons.mozilla.org (AMO).
Info is here ---> https://blog.mozilla.org/addons/2023/08/10/prepare-your-firefox-desktop-extension-for-the-upcoming-android-release/

Thank you for submitting - will have a look!

Let's keep this issue open and monitor it. Checklist from the link:

Update your manifest.json background key and add “persistent”: false -> not using this property
Ensure listeners are registered synchronously at the top-level -> not sure about this one? Since we're using addEventListener() which I think is async
Record global state in the storage API, for example storage.session -> not using storage
Change timers to alarms -> not using any timers
Switch from using extension.getBackgroundPage for calling a function from the background page, to extension messaging or runtime.getBackgroundPage -> not using .getBackgroundPage

I think there is one thing I need to do: when updating extension, include "For Android" option. I'll remind myself to do it in near future!

I think there is one thing I need to do: when updating extension, include "For Android" option. I'll remind myself to do it in near future!

I'll have a look at this soon. Quite busy with work and life right now.

[ ] Investigate how to test web extension on Android device and verify that it works
[ ] Update web extension so it can be released it on Android devices (config)


knime: Update to version 4.7.2 and fix autoupdate

[X] I have read the Contributing Guide.

/verify


Use QUDT QuantityKinds directly

We should use QUDT directly where possible; if there are quantities in Brick w/o a QUDT equivalent then these should be added to QUDT where possible.
We should discuss making this change for 1.4; maybe we can add QUDT annotations directly in 1.3.1 or a subsequent release?

This is an excellent idea.


font-feature-settings

I am urgently missing the possibility to define font-feture-settings for open type. Even a plain csv of strings would be ok, like so:
font-feature-settings: 'tnum' on, 'lnum' on, 'cv01' on, 'cv02' on, 'cv03' on, 'cv04' on;
Figma supports this. Import ignores this. No manual setting for open type features are possible yet.
To define this as a token under "other" lets you define the token, but "other"-tokens cannot be applied to typography-tokens nor to compositions.

Unfortunately Figma doesn't support this yet in their Plugin API.
Marking as blocked by Figma and putting to Backlog

Unfortunately Figma doesn't support this yet in their Plugin API.
Marking as blocked by Figma and putting to Backlog

This explains why the import fails. But there is also no way to define it manually. Typography-tokens don't have font-feature-settings options. And if I define it under "others" I cannot apply it to a composition or typography.

We dont allow to define it because Figma has no support for it yet.

We dont allow to define it because Figma has no support for it yet.

Well, Figma does support OTF font-feature-settings. Inter is a good example for that. Here some of the settings you can set:

The settings also show up in the inspect panel.
Maybe the import is not possible. But there should be a way to define it in the plugin via UI or directly in JSON to have a complete style library to hand over.

As stated initially Figma's Plugin API does not support that, we can only build support for it once Figma adds this.

As stated initially Figma's Plugin API does not support that, we can only build support for it once Figma adds this.

But it may be worth considering even if not supported by figmas API for exchange, to be able to manually define it, so that at least the repository of tokens is is holding it for a workflow outside of figma.

Moved to featurebase for typography improvements as Figma enhances the plugin API options avail to us.

Added to Featurebase Request-	Enhanced typography support


Fixed feature allowing to change tabs programmatically.

Also added possibility to change tabs by their indexes.
For some reason "value" prop wasn't working so I've changed it to "tabsValue".
Besides I've added some dev code for testing.

Thanks a lot @thatside. Unfortunately I'm on vacation right now and can review this only by the end of this week. Hope that is fine

@thatside I looked at the PR, and indeed found that there was an issue. The main issue was that the value function was not placed inside watch 😄
I did a fix which is a bit simpler though but still inspired from your PR.
let indexToActivate = this.tabs.findIndex((tab, index) => tab.title === tabNameOrIndex || index === tabNameOrIndex)

which basically is enough to make it work both with v-model based on title and index without any other changes

Seems to be working OK with numeric values. Thank you!


Clean up library trim warning article

There was a lot of extra information between the article start
and the main action we want the user to do (set IsTrimmable), so
I tried to move some stuff around and get rid of info that's either
not necessary or out of date.

@adegao Anything else for me?

@IEvangelist Thanks for the read, sorry for the delay. Yes, I think the main approach here is to simplify by keeping the primary scenario (trimming libraries) front and center. There's a lot of other information here that is already present in other articles (like the Trim Warnings section) that's duplicated and distracting from the overall goal.


Call getrandom via syscall directly on Linux.

Doing this because getrandom wasn't added to glibc until 2.25.

Ran into one more issue with getrandom; it was added to glibc in version 2.25, but the chromium builders are using an older version.
Also, looking at the oldest supported versions of popular distros, half are using a version older than 2.25.
Introducing a HAVE_GETRANDOM define as I did in the earlier PR would also work for me.

This doesn't help in the way you are hoping, as those (notably supported) distributions (which includes a number that I'm actively forced to support myself) that are using old versions of glibc are also usually using commensurately old versions of the kernel that then also don't support this syscall. (To be "more honest" about this when doing builds, you can cross-compile to an old distribution to see what it actually has in its kernel headers.) Can you maybe add a !defined() check on __NR_getrandom to the block that is used for old versions of Android and Emscripten, so there's a fallback to /dev/urandom? (In the mean-time I've added a hack locally to use that /dev/urandom block on Linux.)


Write only successful TXs to the ledger

This PR changes the system so it adds only successful
TXs to the ledger. The TXs where chaincode execution was erroneous
are not persisted.
Description
See the summary.
Motivation and Context
Writing erroneous TXs (where the chaincode execution resulted
in an exception etc.) to the ledger makes it possible to flood the
ledger easily. On the other hand, Bitcoin follows the same approach
as this commit, it does not store unsuccessful transactions.
How Has This Been Tested?
Unit tests were run.
Checklist:

[x] I have added a Signed-off-by
[x] Either no new documentation is required by this change, OR I added new documentation
[x] Either no new tests are required by this change, OR I added new tests
[x] I have run goimports, go vet, and golint. I have cleaned up all valid errors and warnings in code I have added or modified. These tools may generate false positives.  The goal is clean, consistent, and readable code.

Signed-off-by: Gabor Hosszu<EMAIL_ADDRESS>

This PR is much more than it says in the comment.

I created a new PR containing only the last commit that adds the specified functionality.


Automatically install pptrees.so into /usr/local/share/yosys/plugins if available

This will bring back functionality that was previously removed, as I was not aware of the -i flag.

I think you probably want to use pkg-config to detect the Yosys install's plugin storage location?

I think this is done in https://github.com/chipsalliance/yosys-f4pga-plugins/blob/main/Makefile_plugin.common ?

Maybe @mkurc-ant can help?


[Bug] Modality unit Bugs

Describe the Bug
The happened after the recent getModalityUnit change.
The issue is couple of things

First the getModalityUnit is called on the targetId which can be the CT and we are checking isSUVCalculated on CT and it does not have scaling.PT
Also the referenceImageId might be the CT if drawn on CT first, then it means that later in the

const petSeriesModule = metaData.get('petSeriesModule', imageId);
return petSeriesModule?.units || 'unitless';
which is wrong, since a CT image ID will not have petSeriesModule
Steps to Reproduce
yarn run example petCt
add probe tool to all toogroups
The current behavior
It wrongly show the modality raw
The expected behavior
should say suv
OS
macos
Node version
18
Browser
chrome

Why the function 'getModalityUnit' sometimes return 'raw' and sometimes 'SUV' when the modality is PT?

Fixed in main


Lack of transparency on used training data. - Does finetuning make sense?

I'm aware the translation models are trained on training data from the OPUS corpus.
But for me it's very unclear on how much data exactly they have trained these models and whether they have used ALL available data from the OPUS corpus given the language directions.
Does it make sense to download OPUS data and further finetune these models?
Does it make sense to find other data sources and finetune the models? If so, how much sentence pairs (approximately) do I need to see an improvement?
I'm particularly interested in finetuning "Helsinki-NLP/opus-mt-nl-en" and "Helsinki-NLP/opus-mt-en-nl"
.

Yes, more or less all data in OPUS at that time of training. I am not sure about fine-tuning. It may also forget about previously learned information. You could continue training with some larger data set but then you may need some longer warm-up time as well to get the optimizer back on track.


Feature Request: Update application window with Workspace name prepended

Version: 1.15.0-insider
Commit SHA: 56a5bf1724f3b882e149a3e7a24e6105a7b837c2
Date: 2017-07-13T06:24:22.570Z
Shell 1.6.6
Renderer 56.0.2924.87
Node 7.4.0
Architecture x64
Ubuntu 16.04
When I select a file the application's toolbar is updated with the filename followed by an emdash Visual Studio (in this case - Insider ).  This is great so I can reference the window easily when I have multiple windows open.  When I'm in a Workspace and have multiple Workspaces open in different windows, it's hard to tell which window is what at a glance since I usually am looking at that area.

Isn't it already there? I see fileName - Workspace Root Directory. or I might be missing something silly.

https://github.com/Microsoft/vscode/commit/4d8a487b95be4c9aa7b35f3ea8c8ce93da9333fb


How to export/import types declared in commonjs modules?

I'm trying to import a type declared in a commonjs module (though not exported as part of its module.exports) in another file, and use it.
Say I have a file called sauce.js:
class Sauce { }

function getFavoriteSauce(){
	return new Sauce();
}

module.exports = { getFavoriteSauce };

And another file called index.js (where I want to refer to the class type Sauce):
const { getFavoriteSauce } = require("./sauce");

/**
 * @param {Sauce} sauceA // doesn't work
 * @param {Sauce} sauceB // doesn't work
 */
function mixSauces(sauceA, sauceB){
	sauceA.mixWith(sauceB);
};

While the type definition for getFavoriteSauce includes Sauce (() => Sauce) I can't mention Sauce as a type directly.
There doesn't appear to be any way to export the Sauce class and refer to it from another file. I've tried everything — @typedef, namely, with @static, @global and others thrown in out of desperation.
Is there no way to directly refer to Sauce?

Facing same issue. Is there any solution or work around?

You can declare types declared via @typedef as @global. Then jsdoc will create links in the HTML output.
If your quesiton isn't about jsdoc, — this tool right here — but about type-stuff in your IDE this is the wrong place.

If your quesiton isn't about jsdoc, — this tool right here — but about type-stuff in your IDE this is the wrong place.

Thank you for clearing that up. In this case the problem seems to come from the fact that Visual Studio Code doesn't support @global.


Get client IP address with serverless functions

I tried to get the client IP with a function like the one below but got 10.128.x.x in return. The 'x-forwarded-for' was not appended with the real client IP.
I also went through the req object but can't see a way to get the correct IP
Is there a way to get the client IP address?
export default (req, res) => {

    console.log('<< req.headers >>', req.headers);

    let ipaddress;
    if (req.headers['x-real-ip']) {
        ipaddress = req.headers['x-real-ip'];
    } else {
        ipaddress = req.headers['x-forwarded-for'].split(',')[0];
    }

    res.status(200).json(
        {
            ipaddress,
            timestamp: Date.now()
        }
    );
}

Hello,
Indeed, both above headers are sending an internal private IP.
@nunopato will be able to say more about it

I was about to open an issue, but I see this already existed and described the same thing but in a slightly different way.
const { host, hostname, ip, ips } = req;
Trying to destructure some properties from the req object that is passed into the function.
host and hostname are undefined
ip is empty string
ips is empty array
when running locally they do have values when I console.log them
request:  {"host":"localhost","hostname":"localhost","ip":"::ffff:<IP_ADDRESS>","ips":[]
I am trying to get the ip address of the source/origin that made the request.
After a fix from @dbarrosop and redeploying my Nhost project,
I am getting this when I console.log
{ "host":"{Subdomain}.functions.eu-central-1.nhost.run", "hostname":"{Subdomain}.functions.eu-central-1.nhost.run", "ip":"", "ips":["<IP_ADDRESS>"] }
was expecting to see the ip of the computer making the http request as well.

Could you verify if this is fixed now? Thanks!

Yes now it working.
👍🏽
"ips" comes back with the ip address of the compute making the http request to the nhost function.
but "ip":"" is still empty string (but maybe that is expected behavior)

that may be due to some proxy thing on the client side. Can you log the content of x-forwarded-for?

@dbarrosop
these are the logs with a single ip address, which is my current ip. so it is correct.
"X-Forwarded-For": "XXX.XX.XX.XXX"
the same ip address also appears in ips as a single entry
"ips":["XXX.XX.XX.XXX"]
but "ip" is still empty string ""


Gemini config not found when using ipython scheduler

Hi Brad
We got the iPython scheduler to work on our local install, but there still seems to be something amiss when using Gemini. I'm getting an error message (included below) that it isn't properly configured and missing the config for it's annotations. However, this configuration file is installed in the bcbio install directory, but gemini seems to be unaware of this.
The curious thing about this is when I run bcbio in local mode, there are no issues and everything runs smoothly.
I've already tried updating everything using bcbio-nextgen.py upgrade --tools --data --datatarget gemini, but the installer just skips over everything, since it's already up to date.
Any idea what might be the issue?
Thank a bunch!
M
[2016-08-10T06:23Z] pbsnode-1: Create gemini database for /home/projects/genome_in_a_bottle/bcbio/D1106193/work/gemini/D1106193-gatk-haplotype-decompose-vepeffects.vcf.gz : D1106193
[2016-08-10T06:24Z] pbsnode-1: CADD scores are being loaded (to skip use:--skip-cadd).
[2016-08-10T06:25Z] pbsnode-1: Traceback (most recent call last):
[2016-08-10T06:25Z] pbsnode-1: Traceback (most recent call last):
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/bin/gemini", line 6, in <module>
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/bin/gemini", line 6, in <module>
[2016-08-10T06:25Z] pbsnode-1: Traceback (most recent call last):
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/bin/gemini", line 6, in <module>
[2016-08-10T06:25Z] pbsnode-1:     gemini.gemini_main.main()
[2016-08-10T06:25Z] pbsnode-1: Traceback (most recent call last):
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/bin/gemini", line 6, in <module>
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_main.py", line 1227, in main
[2016-08-10T06:25Z] pbsnode-1:     gemini.gemini_main.main()
[2016-08-10T06:25Z] pbsnode-1:     gemini.gemini_main.main()
[2016-08-10T06:25Z] pbsnode-1:     gemini.gemini_main.main()
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_main.py", line 1227, in main
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_main.py", line 1227, in main
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_main.py", line 1227, in main
[2016-08-10T06:25Z] pbsnode-1:     args.func(parser, args)
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_main.py", line 298, in loadchunk_fn
[2016-08-10T06:25Z] pbsnode-1:     args.func(parser, args)
[2016-08-10T06:25Z] pbsnode-1:     args.func(parser, args)
[2016-08-10T06:25Z] pbsnode-1:     args.func(parser, args)
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_main.py", line 298, in loadchunk_fn
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_main.py", line 298, in loadchunk_fn
[2016-08-10T06:25Z] pbsnode-1:     gemini_load_chunk.load(parser, args)
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_main.py", line 298, in loadchunk_fn
[2016-08-10T06:25Z] pbsnode-1:     gemini_load_chunk.load(parser, args)
[2016-08-10T06:25Z] pbsnode-1:     gemini_load_chunk.load(parser, args)
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_load_chunk.py", line 828, in load
[2016-08-10T06:25Z] pbsnode-1:     gemini_load_chunk.load(parser, args)
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_load_chunk.py", line 828, in load
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_load_chunk.py", line 828, in load
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_load_chunk.py", line 828, in load
[2016-08-10T06:25Z] pbsnode-1:     annotations.load_annos(args)
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/annotations.py", line 181, in load_annos
[2016-08-10T06:25Z] pbsnode-1:     annotations.load_annos(args)
[2016-08-10T06:25Z] pbsnode-1:     annotations.load_annos(args)
[2016-08-10T06:25Z] pbsnode-1:     annotations.load_annos(args)
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/annotations.py", line 181, in load_annos
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/annotations.py", line 181, in load_annos
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/annotations.py", line 181, in load_annos
[2016-08-10T06:25Z] pbsnode-1:     anno_files = get_anno_files(args)
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/annotations.py", line 14, in get_anno_files
[2016-08-10T06:25Z] pbsnode-1:     anno_files = get_anno_files(args)
[2016-08-10T06:25Z] pbsnode-1:     anno_files = get_anno_files(args)
[2016-08-10T06:25Z] pbsnode-1:     anno_files = get_anno_files(args)
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/annotations.py", line 14, in get_anno_files
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/annotations.py", line 14, in get_anno_files
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/annotations.py", line 14, in get_anno_files
[2016-08-10T06:25Z] pbsnode-1:     config = read_gemini_config(args=args)
[2016-08-10T06:25Z] pbsnode-1:     config = read_gemini_config(args=args)
[2016-08-10T06:25Z] pbsnode-1:     config = read_gemini_config(args=args)
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/config.py", line 52, in read_gemini_config
[2016-08-10T06:25Z] pbsnode-1:     config = read_gemini_config(args=args)
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/config.py", line 52, in read_gemini_config
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/config.py", line 52, in read_gemini_config
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/config.py", line 52, in read_gemini_config
[2016-08-10T06:25Z] pbsnode-1:     fname = _get_config_file(dirs, use_globals=use_globals)
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/config.py", line 47, in _get_config_file
[2016-08-10T06:25Z] pbsnode-1:     fname = _get_config_file(dirs, use_globals=use_globals)
[2016-08-10T06:25Z] pbsnode-1:     fname = _get_config_file(dirs, use_globals=use_globals)
[2016-08-10T06:25Z] pbsnode-1:     fname = _get_config_file(dirs, use_globals=use_globals)
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/config.py", line 47, in _get_config_file
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/config.py", line 47, in _get_config_file
[2016-08-10T06:25Z] pbsnode-1:   File "/Shared/Software/lib/python2.7/site-packages/gemini/config.py", line 47, in _get_config_file
[2016-08-10T06:25Z] pbsnode-1:     .format(CONFIG_FILE, dnames))
[2016-08-10T06:25Z] pbsnode-1:     .format(CONFIG_FILE, dnames))
[2016-08-10T06:25Z] pbsnode-1:     .format(CONFIG_FILE, dnames))
[2016-08-10T06:25Z] pbsnode-1: ValueError: GEMINI configuration file gemini-config.yaml not found in ['/usr/local/share/gemini', '/home/galaxy/.gemini'].
[2016-08-10T06:25Z] pbsnode-1: Please ensure the GEMINI data is installed using the install-data.py script
[2016-08-10T06:25Z] pbsnode-1: http://gemini.readthedocs.org/en/latest/content/installation.html
[2016-08-10T06:25Z] pbsnode-1:     .format(CONFIG_FILE, dnames))
[2016-08-10T06:25Z] pbsnode-1: ValueError: GEMINI configuration file gemini-config.yaml not found in ['/usr/local/share/gemini', '/home/galaxy/.gemini'].
[2016-08-10T06:25Z] pbsnode-1: Please ensure the GEMINI data is installed using the install-data.py script
[2016-08-10T06:25Z] pbsnode-1: http://gemini.readthedocs.org/en/latest/content/installation.html
[2016-08-10T06:25Z] pbsnode-1: ValueError: GEMINI configuration file gemini-config.yaml not found in ['/usr/local/share/gemini', '/home/galaxy/.gemini'].
[2016-08-10T06:25Z] pbsnode-1: Please ensure the GEMINI data is installed using the install-data.py script
[2016-08-10T06:25Z] pbsnode-1: http://gemini.readthedocs.org/en/latest/content/installation.html
[2016-08-10T06:25Z] pbsnode-1: ValueError: GEMINI configuration file gemini-config.yaml not found in ['/usr/local/share/gemini', '/home/galaxy/.gemini'].
[2016-08-10T06:25Z] pbsnode-1: Please ensure the GEMINI data is installed using the install-data.py script
[2016-08-10T06:25Z] pbsnode-1: http://gemini.readthedocs.org/en/latest/content/installation.html
[2016-08-10T06:25Z] pbsnode-1: Indexing /home/projects/genome_in_a_bottle/bcbio/D1106193/work/gemini/D1106193-gatk-haplotype-decompose-vepeffects.vcf.gz with grabix.
[2016-08-10T06:25Z] pbsnode-1: Loading 164376 variants.
[2016-08-10T06:25Z] pbsnode-1: Breaking /home/projects/genome_in_a_bottle/bcbio/D1106193/work/gemini/D1106193-gatk-haplotype-decompose-vepeffects.vcf.gz into 4 chunks.
[2016-08-10T06:25Z] pbsnode-1: Loading chunk 0.
[2016-08-10T06:25Z] pbsnode-1: Loading chunk 1.
[2016-08-10T06:25Z] pbsnode-1: Loading chunk 2.
[2016-08-10T06:25Z] pbsnode-1: Loading chunk 3.
[2016-08-10T06:25Z] pbsnode-1: Traceback (most recent call last):
[2016-08-10T06:25Z] pbsnode-1:   File "/home/galaxy/bcbio/anaconda/bin/gemini", line 6, in <module>
[2016-08-10T06:25Z] pbsnode-1:     gemini.gemini_main.main()
[2016-08-10T06:25Z] pbsnode-1:   File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 1227, in main
[2016-08-10T06:25Z] pbsnode-1:     args.func(parser, args)
[2016-08-10T06:25Z] pbsnode-1:   File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 198, in load_fn
[2016-08-10T06:25Z] pbsnode-1:     gemini_load.load(parser, args)
[2016-08-10T06:25Z] pbsnode-1:   File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/gemini/gemini_load.py", line 48, in load
[2016-08-10T06:25Z] pbsnode-1:     load_multicore(args)
[2016-08-10T06:25Z] pbsnode-1:   File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/gemini/gemini_load.py", line 92, in load_multicore
[2016-08-10T06:25Z] pbsnode-1:     chunks = load_chunks_multicore(grabix_file, args)
[2016-08-10T06:25Z] pbsnode-1:   File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/gemini/gemini_load.py", line 258, in load_chunks_multicore
[2016-08-10T06:25Z] pbsnode-1:     wait_until_finished(procs)
[2016-08-10T06:25Z] pbsnode-1:   File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/gemini/gemini_load.py", line 349, in wait_until_finished
[2016-08-10T06:25Z] pbsnode-1:     raise ValueError("Processing failed on GEMINI chunk load")
[2016-08-10T06:25Z] pbsnode-1: ValueError: Processing failed on GEMINI chunk load
[2016-08-10T06:25Z] pbsnode-1: Uncaught exception occurred
Traceback (most recent call last):
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 21, in run
    _do_run(cmd, checks, log_stdout)
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 95, in _do_run
    raise subprocess.CalledProcessError(exitcode, error_msg)
CalledProcessError: Command 'set -o pipefail; /home/galaxy/bcbio/anaconda/bin/gemini  load  --passonly --skip-gerp-bp  -v /home/projects/genome_in_a_bottle/bcbio/D1106193/work/gemini/D1106193-gatk-haplotype-decompose-vepeffects.vcf.gz -t VEP --cores 4 --tempdir /tmp/D1106193/bcbiotx/669e0178-4ee5-4162-8dfe-58ccce39ef49/tmpt34k8f /tmp/D1106193/bcbiotx/669e0178-4ee5-4162-8dfe-58ccce39ef49/tmpt34k8f/D1106193-gatk-haplotype.db
  File "/Shared/Software/bin/gemini", line 6, in <module>
Traceback (most recent call last):
  File "/Shared/Software/bin/gemini", line 6, in <module>
    gemini.gemini_main.main()
Traceback (most recent call last):
  File "/Shared/Software/bin/gemini", line 6, in <module>
  File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_main.py", line 1227, in main
    gemini.gemini_main.main()
    gemini.gemini_main.main()
    gemini.gemini_main.main()
  File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_main.py", line 1227, in main
  File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_main.py", line 1227, in main
  File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_main.py", line 1227, in main
    args.func(parser, args)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_main.py", line 298, in loadchunk_fn
    args.func(parser, args)
    args.func(parser, args)
    args.func(parser, args)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_main.py", line 298, in loadchunk_fn
  File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_main.py", line 298, in loadchunk_fn
    gemini_load_chunk.load(parser, args)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_main.py", line 298, in loadchunk_fn
    gemini_load_chunk.load(parser, args)
    gemini_load_chunk.load(parser, args)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_load_chunk.py", line 828, in load
    gemini_load_chunk.load(parser, args)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_load_chunk.py", line 828, in load
  File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_load_chunk.py", line 828, in load
  File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_load_chunk.py", line 828, in load
    annotations.load_annos(args)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/annotations.py", line 181, in load_annos
    annotations.load_annos(args)
    annotations.load_annos(args)
    annotations.load_annos(args)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/annotations.py", line 181, in load_annos
  File "/Shared/Software/lib/python2.7/site-packages/gemini/annotations.py", line 181, in load_annos
  File "/Shared/Software/lib/python2.7/site-packages/gemini/annotations.py", line 181, in load_annos
    anno_files = get_anno_files(args)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/annotations.py", line 14, in get_anno_files
    anno_files = get_anno_files(args)
    anno_files = get_anno_files(args)
    anno_files = get_anno_files(args)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/annotations.py", line 14, in get_anno_files
  File "/Shared/Software/lib/python2.7/site-packages/gemini/annotations.py", line 14, in get_anno_files
  File "/Shared/Software/lib/python2.7/site-packages/gemini/annotations.py", line 14, in get_anno_files
    config = read_gemini_config(args=args)
    config = read_gemini_config(args=args)
    config = read_gemini_config(args=args)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/config.py", line 52, in read_gemini_config
    config = read_gemini_config(args=args)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/config.py", line 52, in read_gemini_config
  File "/Shared/Software/lib/python2.7/site-packages/gemini/config.py", line 52, in read_gemini_config
  File "/Shared/Software/lib/python2.7/site-packages/gemini/config.py", line 52, in read_gemini_config
    fname = _get_config_file(dirs, use_globals=use_globals)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/config.py", line 47, in _get_config_file
    fname = _get_config_file(dirs, use_globals=use_globals)
    fname = _get_config_file(dirs, use_globals=use_globals)
    fname = _get_config_file(dirs, use_globals=use_globals)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/config.py", line 47, in _get_config_file
  File "/Shared/Software/lib/python2.7/site-packages/gemini/config.py", line 47, in _get_config_file
  File "/Shared/Software/lib/python2.7/site-packages/gemini/config.py", line 47, in _get_config_file
    .format(CONFIG_FILE, dnames))
    .format(CONFIG_FILE, dnames))
    .format(CONFIG_FILE, dnames))
ValueError: GEMINI configuration file gemini-config.yaml not found in ['/usr/local/share/gemini', '/home/galaxy/.gemini'].
Please ensure the GEMINI data is installed using the install-data.py script
http://gemini.readthedocs.org/en/latest/content/installation.html
    .format(CONFIG_FILE, dnames))
ValueError: GEMINI configuration file gemini-config.yaml not found in ['/usr/local/share/gemini', '/home/galaxy/.gemini'].
Please ensure the GEMINI data is installed using the install-data.py script
http://gemini.readthedocs.org/en/latest/content/installation.html
ValueError: GEMINI configuration file gemini-config.yaml not found in ['/usr/local/share/gemini', '/home/galaxy/.gemini'].
Please ensure the GEMINI data is installed using the install-data.py script
http://gemini.readthedocs.org/en/latest/content/installation.html
ValueError: GEMINI configuration file gemini-config.yaml not found in ['/usr/local/share/gemini', '/home/galaxy/.gemini'].
Please ensure the GEMINI data is installed using the install-data.py script
http://gemini.readthedocs.org/en/latest/content/installation.html
Indexing /home/projects/genome_in_a_bottle/bcbio/D1106193/work/gemini/D1106193-gatk-haplotype-decompose-vepeffects.vcf.gz with grabix.
Loading 164376 variants.
Breaking /home/projects/genome_in_a_bottle/bcbio/D1106193/work/gemini/D1106193-gatk-haplotype-decompose-vepeffects.vcf.gz into 4 chunks.
Loading chunk 0.
Loading chunk 1.
Loading chunk 2.
Loading chunk 3.
Traceback (most recent call last):
  File "/home/galaxy/bcbio/anaconda/bin/gemini", line 6, in <module>
    gemini.gemini_main.main()
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 1227, in main
    args.func(parser, args)
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 198, in load_fn
    gemini_load.load(parser, args)
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/gemini/gemini_load.py", line 48, in load
    load_multicore(args)
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/gemini/gemini_load.py", line 92, in load_multicore
    chunks = load_chunks_multicore(grabix_file, args)
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/gemini/gemini_load.py", line 258, in load_chunks_multicore
    wait_until_finished(procs)
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/gemini/gemini_load.py", line 349, in wait_until_finished
    raise ValueError("Processing failed on GEMINI chunk load")
ValueError: Processing failed on GEMINI chunk load
' returned non-zero exit status 1
[2016-08-10T06:25Z] pbsnode-1: Unexpected error
Traceback (most recent call last):
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/ipythontasks.py", line 50, in _setup_logging
    yield config
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/ipythontasks.py", line 344, in prep_gemini_db
    return ipython.zip_args(apply(population.prep_gemini_db, *args))
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/bcbio/variation/population.py", line 36, in prep_gemini_db
    gemini_db = create_gemini_db(gemini_vcf, data, gemini_db, ped_file)
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/bcbio/variation/population.py", line 79, in create_gemini_db
    do.run(cmd, "Create gemini database for %s" % gemini_vcf, data)
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 21, in run
    _do_run(cmd, checks, log_stdout)
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 95, in _do_run
    raise subprocess.CalledProcessError(exitcode, error_msg)
CalledProcessError: Command 'set -o pipefail; /home/galaxy/bcbio/anaconda/bin/gemini  load  --passonly --skip-gerp-bp  -v /home/projects/genome_in_a_bottle/bcbio/D1106193/work/gemini/D1106193-gatk-haplotype-decompose-vepeffects.vcf.gz -t VEP --cores 4 --tempdir /tmp/D1106193/bcbiotx/669e0178-4ee5-4162-8dfe-58ccce39ef49/tmpt34k8f /tmp/D1106193/bcbiotx/669e0178-4ee5-4162-8dfe-58ccce39ef49/tmpt34k8f/D1106193-gatk-haplotype.db
  File "/Shared/Software/bin/gemini", line 6, in <module>
Traceback (most recent call last):
  File "/Shared/Software/bin/gemini", line 6, in <module>
    gemini.gemini_main.main()
Traceback (most recent call last):
  File "/Shared/Software/bin/gemini", line 6, in <module>
  File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_main.py", line 1227, in main
    gemini.gemini_main.main()
    gemini.gemini_main.main()
    gemini.gemini_main.main()
  File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_main.py", line 1227, in main
  File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_main.py", line 1227, in main
  File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_main.py", line 1227, in main
    args.func(parser, args)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_main.py", line 298, in loadchunk_fn
    args.func(parser, args)
    args.func(parser, args)
    args.func(parser, args)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_main.py", line 298, in loadchunk_fn
  File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_main.py", line 298, in loadchunk_fn
    gemini_load_chunk.load(parser, args)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_main.py", line 298, in loadchunk_fn
    gemini_load_chunk.load(parser, args)
    gemini_load_chunk.load(parser, args)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_load_chunk.py", line 828, in load
    gemini_load_chunk.load(parser, args)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_load_chunk.py", line 828, in load
  File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_load_chunk.py", line 828, in load
  File "/Shared/Software/lib/python2.7/site-packages/gemini/gemini_load_chunk.py", line 828, in load
    annotations.load_annos(args)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/annotations.py", line 181, in load_annos
    annotations.load_annos(args)
    annotations.load_annos(args)
    annotations.load_annos(args)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/annotations.py", line 181, in load_annos
  File "/Shared/Software/lib/python2.7/site-packages/gemini/annotations.py", line 181, in load_annos
  File "/Shared/Software/lib/python2.7/site-packages/gemini/annotations.py", line 181, in load_annos
    anno_files = get_anno_files(args)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/annotations.py", line 14, in get_anno_files
    anno_files = get_anno_files(args)
    anno_files = get_anno_files(args)
    anno_files = get_anno_files(args)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/annotations.py", line 14, in get_anno_files
  File "/Shared/Software/lib/python2.7/site-packages/gemini/annotations.py", line 14, in get_anno_files
  File "/Shared/Software/lib/python2.7/site-packages/gemini/annotations.py", line 14, in get_anno_files
    config = read_gemini_config(args=args)
    config = read_gemini_config(args=args)
    config = read_gemini_config(args=args)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/config.py", line 52, in read_gemini_config
    config = read_gemini_config(args=args)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/config.py", line 52, in read_gemini_config
  File "/Shared/Software/lib/python2.7/site-packages/gemini/config.py", line 52, in read_gemini_config
  File "/Shared/Software/lib/python2.7/site-packages/gemini/config.py", line 52, in read_gemini_config
    fname = _get_config_file(dirs, use_globals=use_globals)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/config.py", line 47, in _get_config_file
    fname = _get_config_file(dirs, use_globals=use_globals)
    fname = _get_config_file(dirs, use_globals=use_globals)
    fname = _get_config_file(dirs, use_globals=use_globals)
  File "/Shared/Software/lib/python2.7/site-packages/gemini/config.py", line 47, in _get_config_file
  File "/Shared/Software/lib/python2.7/site-packages/gemini/config.py", line 47, in _get_config_file
  File "/Shared/Software/lib/python2.7/site-packages/gemini/config.py", line 47, in _get_config_file
    .format(CONFIG_FILE, dnames))
    .format(CONFIG_FILE, dnames))
    .format(CONFIG_FILE, dnames))
ValueError: GEMINI configuration file gemini-config.yaml not found in ['/usr/local/share/gemini', '/home/galaxy/.gemini'].
Please ensure the GEMINI data is installed using the install-data.py script
http://gemini.readthedocs.org/en/latest/content/installation.html
    .format(CONFIG_FILE, dnames))
ValueError: GEMINI configuration file gemini-config.yaml not found in ['/usr/local/share/gemini', '/home/galaxy/.gemini'].
Please ensure the GEMINI data is installed using the install-data.py script
http://gemini.readthedocs.org/en/latest/content/installation.html
ValueError: GEMINI configuration file gemini-config.yaml not found in ['/usr/local/share/gemini', '/home/galaxy/.gemini'].
Please ensure the GEMINI data is installed using the install-data.py script
http://gemini.readthedocs.org/en/latest/content/installation.html
ValueError: GEMINI configuration file gemini-config.yaml not found in ['/usr/local/share/gemini', '/home/galaxy/.gemini'].
Please ensure the GEMINI data is installed using the install-data.py script
http://gemini.readthedocs.org/en/latest/content/installation.html
Indexing /home/projects/genome_in_a_bottle/bcbio/D1106193/work/gemini/D1106193-gatk-haplotype-decompose-vepeffects.vcf.gz with grabix.
Loading 164376 variants.
Breaking /home/projects/genome_in_a_bottle/bcbio/D1106193/work/gemini/D1106193-gatk-haplotype-decompose-vepeffects.vcf.gz into 4 chunks.
Loading chunk 0.
Loading chunk 1.
Loading chunk 2.
Loading chunk 3.
Traceback (most recent call last):
  File "/home/galaxy/bcbio/anaconda/bin/gemini", line 6, in <module>
    gemini.gemini_main.main()
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 1227, in main
    args.func(parser, args)
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 198, in load_fn
    gemini_load.load(parser, args)
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/gemini/gemini_load.py", line 48, in load
    load_multicore(args)
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/gemini/gemini_load.py", line 92, in load_multicore
    chunks = load_chunks_multicore(grabix_file, args)
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/gemini/gemini_load.py", line 258, in load_chunks_multicore
    wait_until_finished(procs)
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/gemini/gemini_load.py", line 349, in wait_until_finished
    raise ValueError("Processing failed on GEMINI chunk load")
ValueError: Processing failed on GEMINI chunk load
' returned non-zero exit status 1
[2016-08-10T06:25Z] pbsnode-1: Did not stop IPython cluster correctly
Traceback (most recent call last):
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/ipython.py", line 91, in stop
    ipython_cluster.stop_from_view(view)
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/cluster_helper/cluster.py", line 849, in stop_from_view
    _stop(view.clusterhelper["profile"], view.clusterhelper["cluster_id"])
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/cluster_helper/cluster.py", line 855, in _stop
    subprocess.check_call(args)
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['/home/galaxy/bcbio/anaconda/bin/python', '-E', '-c', 'import resource; cur_proc, max_proc = resource.getrlimit(resource.RLIMIT_NPROC); target_proc = min(max_proc, 10240) if max_proc > 0 else 10240; resource.setrlimit(resource.RLIMIT_NPROC, (max(cur_proc, target_proc), max_proc)); cur_hdls, max_hdls = resource.getrlimit(resource.RLIMIT_NOFILE); target_hdls = min(max_hdls, 10240) if max_hdls > 0 else 10240; resource.setrlimit(resource.RLIMIT_NOFILE, (max(cur_hdls, target_hdls), max_hdls)); from ipyparallel.apps.ipclusterapp import launch_new_instance; launch_new_instance()', 'stop', '--cluster-id=5515fd51-f4f8-44b3-b883-4e002a969d73', '--profile-dir=/home/projects/genome_in_a_bottle/bcbio/D1106193/work/log/ipython']' returned non-zero exit status 11

Sorry about the issue, I'm not exactly sure what is going on here. bcbio doesn't do anything differently to resolve the GEMINI references locally or distributed. Is it possible there's a mount issue on the cluster where pbsnode-1 isn't able to access a drive where the GEMINI databases are available? Does the working run happen on the same worker node or a different machine in the cluster? Hope this is helpful.

Hi Brad,
Thanks for chiming in. All files are located on the same mountpoint (inputs, tools, reference files), so a mount issue is out of the question. All processing is done on the same node. So no issues there. Any other ideas?
Thanks
M

My best guess at this point is there is something strange about the GEMINI installation that is missing it looking in the right place for the gemini configuration file, which should be /home/galaxy/bcbio/gemini-config.yaml along with the data in /home/galaxy/bcbio/gemini_data. It's a bit hard to diagnose remotely and without understanding your install setup, but this could happen if your install machine has the GEMINI database configured elsewhere (maybe in your home directory as a ~/.gemini directory) that is not found when running as the galaxy user. Is there anything custom/different about the GEMINI installation which might cause this?
If you don't need GEMINI you can add tools_off: [gemini] to turn it off and get your analysis finished. Sorry about the problems and hope this helps.

Gemini is an essential part of the pipeline for us, so plainly turning it off is not an option. There also isn't anything custom about the installation. It was installed by the galaxy user in a publicly available directory. For the installation part, only the bcbio install script was used, so there shouldn't be any issues I suppose.
I'll try to pinpoint to issue a bit further. Thanks for the help!
M

Ok, so I checked out some more things, turns out the issue is caused by another installation of the gemini executable.
the bcbio run command is submitted to the cluster with a script. In this script the PATH environmental value is set to the correct value before starting bcbio. It seems however that bcbio tries to use the gemini installation on the default location, instead of the one installed by bcbio itself.
I think this can be fixed by simply removing the other gemini install from $PATH. I'll test this and get back to you if I get some results
cheers!
M

Hi Brad,
is there a possibility to set a custom environment for the iPython jobs spawned bij ipython parallel? It seems the cluster resets $PATH to it's default values, which doesn't include the custom path with the executables for bcbio. I have no idea how the rest of the system manages to work up untill that step, but I suppose we just have the same packages installed.
M

Thanks for diagnosing the issue more and tracing it back to a secondary GEMINI installation. Glad that got figured out.
We don't have a way to specify the PATH within the job scripts run by bcbio. It's generally assumed that this inherits from the parent process (where you can set it) which seems to be the case with most schedulers we've run into. What type of system are you running on that resets the PATH? Is it possible to adjust this on the cluster side? This seems like it would break a lot of analyses, not just bcbio, since it would remove anything not on the default path.

Those are exactly my thoughts too. As I said before, the bcbio startup command is wrapped in a jobscript which sets the PATHvariable. Currently we're running on a torque based scheduler. Do you think this could have anything to to with it? It's weird all other steps do work, except for gemini.
Do you suppose a force reinstall of the gemini executable in the bcbio conda env would be any good? or would this just break some more settings?

I just went through the error report again and it seems bcbio tries to use the correct version of gemini, but the other version then somehow get precendence.
[2016-08-10T06:25Z] pbsnode-1: Uncaught exception occurred
Traceback (most recent call last):
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 21, in run
    _do_run(cmd, checks, log_stdout)
  File "/home/galaxy/bcbio/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 95, in _do_run
    raise subprocess.CalledProcessError(exitcode, error_msg)
CalledProcessError: Command 'set -o pipefail; /home/galaxy/bcbio/anaconda/bin/gemini  load  --passonly --skip-gerp-bp  -v /home/projects/genome_in_a_bottle/bcbio/D1106193/work/gemini/D1106193-gatk-haplotype-decompose-vepeffects.vcf.gz -t VEP --cores 4 --tempdir /tmp/D1106193/bcbiotx/669e0178-4ee5-4162-8dfe-58ccce39ef49/tmpt34k8f /tmp/D1106193/bcbiotx/669e0178-4ee5-4162-8dfe-58ccce39ef49/tmpt34k8f/D1106193-gatk-haplotype.db
  File "/Shared/Software/bin/gemini", line 6, in <module>
Traceback (most recent call last):
  File "/Shared/Software/bin/gemini", line 6, in <module>
    gemini.gemini_main.main()

As you can see, bcbio tries to call the gemini executable in /home/galaxy/bcbio/anaconda/bin
but then executes the executable in /Shared/software/bin
weird..

Thanks for the additional debugging on this. We've used Torque in the past so I know it can pass through the PATH and not reset variables, but perhaps there are configuration options to do something different. Have you talked with the folks managing your cluster? Being able to pass through environmental variables like the PATH is a reasonable request you could make.
Practically to fix this specific problem, what I believe happens is that the original GEMINI occurs with the one installed by bcbio since we explicitly reference it, but the parallel calls use the first occurrence of GEMINI on the PATH:
https://github.com/arq5x/gemini/blob/7ce9da220384449df82dd40122473bfd72424bab/gemini/gemini_load.py#L358
which pulls in your broken version in /Shared/Software/bin/gemini. I pushed a fix which I hope will handle this if you update bcbio to the latest development version (bcbio_nextgen.py upgrade -u development). Hope that gets it running for you.

Great! thanks! I'll get back to you when I get the chance to test it. We currently have an analysis running, which I don't want to interrupt. Or is it possible to perform an upgrade of tools and data while an analysis is running?
M

Upgrades during runs don't typically cause issues but could be a problem in the care cases where a tool is being uninstalled/installed in the middle of a process running it. So it's typically safe to do but I would wait if you need your other runs to absolutely positively finish without any problems.

The fix seems to have cured the issue, thanks!


co19/LayoutTests/fast/forms/change-form-element-document-crash_t01 failing on firefox 40/windows

Might be due to the firefox update, or might be related to Dartium JsInterop changes.

@alan-knight  - I noticed the status file marks this test as failing, but this is making the dart2js-linux bots red. See for example: https://build.chromium.org/p/client.dart/builders/full-linux-2-4-be/builds/3228/steps/dart2js-ff tests --dart2js-batch/logs/stdio
Is this a windows only failure?

it appears that it is also passing in windows here https://build.chromium.org/p/client.dart/builders/full-win7-ie11-2-4-be/builds/6337/steps/dart2js-ff tests --dart2js-batch/logs/stdio

Unsuppressed in https://codereview.chromium.org/1496383002


Identity Access Management Implementations

Description
Implementing Identity and Access Management (IAM) is crucial for organizations to enhance cybersecurity and protect sensitive data. This is achieved by Integrating VEDA auth services with Keycloak APIs and models.

Closing this as this is a duplicating issue. Please refer https://github.com/NASA-IMPACT/veda-auth-central/issues/34


Move phony to be a dev-only dependency

I noticed an odd dependency when installing this package. I think the eloquent/phony should have been a dev requirement. But I think it's not really needed and can be solved with vanilla phpunit just as well.

Coverage remained the same at 93.182% when pulling d38e7c5362c8e6c981abd07d659f101be08920c1 on yannickl88:fix-elequent-phony into a8d30daa6d34e6e54dd96a65addbccdcd3b820d9 on thephpleague:master.

travis seems to fail because HHVM does not support the ubuntu version anymore. Rest of the tests do pass.

Coverage remained the same at 93.182% when pulling 66795bd6df0d7ad51bd92cc44fe3c5f28d150fd1 on yannickl88:fix-elequent-phony into a8d30daa6d34e6e54dd96a65addbccdcd3b820d9 on thephpleague:master.

Coverage remained the same at 93.182% when pulling 66795bd6df0d7ad51bd92cc44fe3c5f28d150fd1 on yannickl88:fix-elequent-phony into a8d30daa6d34e6e54dd96a65addbccdcd3b820d9 on thephpleague:master.

ugh, the travis matrix is always a pain to get right. Think it should be good now.

Coverage remained the same at 93.182% when pulling 1c100bcd1b99a236d1ee0a0dc561b663c7199ebf on yannickl88:fix-elequent-phony into a8d30daa6d34e6e54dd96a65addbccdcd3b820d9 on thephpleague:master.

Coverage remained the same at 93.182% when pulling 5ac108ccb3744486af5496ff469508bca25c041e on yannickl88:fix-elequent-phony into a8d30daa6d34e6e54dd96a65addbccdcd3b820d9 on thephpleague:master.

Coverage remained the same at 93.182% when pulling 5ac108ccb3744486af5496ff469508bca25c041e on yannickl88:fix-elequent-phony into a8d30daa6d34e6e54dd96a65addbccdcd3b820d9 on thephpleague:master.

Coverage remained the same at 93.182% when pulling 5ac108ccb3744486af5496ff469508bca25c041e on yannickl88:fix-elequent-phony into a8d30daa6d34e6e54dd96a65addbccdcd3b820d9 on thephpleague:master.

Coverage remained the same at 93.182% when pulling 5ac108ccb3744486af5496ff469508bca25c041e on yannickl88:fix-elequent-phony into a8d30daa6d34e6e54dd96a65addbccdcd3b820d9 on thephpleague:master.

Coverage remained the same at 93.182% when pulling 5ac108ccb3744486af5496ff469508bca25c041e on yannickl88:fix-elequent-phony into a8d30daa6d34e6e54dd96a65addbccdcd3b820d9 on thephpleague:master.

Thank you! This will be released in 2.0.1as soon as #41 is completed.

:+1: thanks!

Released in 2.0.1.


only response  MarkMonitor data

As far as I know，whois queries  should return information for multiple registries however python-whois only return MarkMonitor data

How do you know this? Is there an example you can provide ?


[ON HOLD] 1699 lint at 160

fixes # 1699
lint at 160 for flake8, yamllint, pylint
keep behaviour of  ansible-lint with its default value: [E204] Lines should be no longer than 120 chars
PR Type

Bugfix Pull Request

Can someone take responsibility to push this forward?
I can manage with or without this change but I want to see this issue resolved.

Okay, since we don't have a consensus and it stayd blocked for long, let's close this PR and reevaluate the situation later.


Is there a way to implement Stochastic RSI

I want to use stochastic RSI for my trading strategy. After going through the code, I cannot find any direct method to do that. I can modify and add a new calculate function but I don't want to do that unless required.Can someone guide me if it is there? If not, I will develop it.

Hello,
It may be something like that:
// Stoch RSI = (RSI - MinimumRSIn) / (MaximumRSIn - MinimumRSIn)
int n = 10;
RSIIndicator rsi = new RSIIndicator(new ClosePriceIndicator(data), n);
LowestValueIndicator minRsi = new LowestValueIndicator(rsi, n);
HighestValueIndicator maxRsi = new HighestValueIndicator(rsi, n);

int tickIndex = 12;// your index
Decimal value = rsi.getValue(tickIndex).minus(minRsi.getValue(tickIndex)).dividedBy(maxRsi.getValue(tickIndex).minus(minRsi.getValue(tickIndex)));

Hi!
Thank you @gcauchis :)
Feel free to make a PR if you want to get this added.


modules/keymaps: fix false-positive lua warning

It appears that submodule options for list-of submodules are considered "defined" when actually defined, while "plane" submodules (i.e. plugins.wtf.keymaps) seem to behave differently...
To work around the issue, I've made the (hidden and deprecated) lua keymap option "nullOr bool", so that we can assume "null" means "unset"...
This could cause an edge-case if a user is reading the lua value in their config and assuming it is a non-null boolean though.

To work around the issue, I've made the (hidden and deprecated) lua keymap option "nullOr bool", so that we can assume "null" means "unset"...

I'm OK with that.

This could cause an edge-case if a user is reading the lua value in their config and assuming it is a non-null boolean though.

Probably no-one, and certainly a very small amount of people. So I wouldn't care much. (Good to point it out nonetheless).


Compilable code

I've finetuned CodeT5 large on a small python dataset(~1700) data points. I see that the results are more or less correct but the code is not always compilable(due to inconsistent spacing and new line characters). Any idea on fixing this? And how CodeBLEU work if the code generated by the model isn't compilable? The model might generate non compilable code during initial phases of the training right?

Hi there, we cannot gaurantee the generated code is compilable for in a good format as we directly use the code files without normalization or refactoring for pretraining. You might consider to include another post-processing step to reformat the generated code from our models.

Hi thanks, can you direct us to necessary post processing steps that we might need to adopt?


PWM LED Driver on ESP32

This PR is a initial support for PWM LED Driver on ESP32.
There are some work I want to do yet:

Remove struct ledc_dev_t to improve the readability of the code;
Split some functions;
Add more configuration options on Kconfig;
Write a specific sample application;
Rewrite this driver to use this new API: https://github.com/zephyrproject-rtos/zephyr/pull/5342, ESP32 has two PWM controllers: one for led and another for servo motor, it can be confusing to have these two drivers in the same directory;

Issue:

For some reason the set frequency is divided by 2;

This PR implements this issue: https://github.com/zephyrproject-rtos/zephyr/issues/4699

Ops, sorry.


Feature Request - CheckDB Snapshots in other location

The use case is that in a Data drive with few space available, as CHECKDB operations create a snapshot of the databases in the drive and checks that instead, the problem is that there if there is not enough space available in the drive for the snapshot to generate operation will fail.
Could it be possible to add to generate DB snapshot manually in other location and perform the operation over it and then delete it as a feature to scripts? Maybe a param @SnapshotLocation to specify it needs to create it in other location and then execute the command.

This is a good idea. The risk is just that if the job is getting aborted for some reason, then the snapshot will not be deleted.
The best would be if Microsoft could add an option to the CHECKDB command, where the snapshots should be created. Please vote for this idea on UserVoice.
https://feedback.azure.com/forums/908035-sql-server/suggestions/36206044-option-to-specify-a-location-for-the-dbcc-checkdb
Then I would also add support for it in my scripts.


Floating Point to Fixed Point Converter Module

Description & Motivation

FloatToFixed is a new component for lossless conversion of a floating-point type to a fixed-point type.
Float8ToFixed is a new component for lossless conversion of a FP8 formats to a fixed-point type.

Related Issue(s)

Testing

Both components are tested exhaustively.
Backwards-compatibility

Is this a breaking change that will not be backwards-compatible? If yes, how so?

This PR is not a breaking change.
Documentation

Does the change require any updates to documentation? If so, where? Are they included?

Both components are added to the documentation.

@desmonddak , this PR is ready for review.


Rename of RefreshableCredentials is causing an issue

Describe the bug
I'm running dbt debug in a databricks job cluster.
The dbt cli using the databricks adapter is having trouble connecting to databricks. This is the error I'm seeing.
It cannot find the RefreshableCredentials class since it was recently renamed to SessionCredentials.
Steps To Reproduce
The easiest way to reproduce this is by running dbt debug in a databricks job while using the
Expected behavior
It should successfully retrieve the credentials and connect to the cluster.
Screenshots and log output
+ dbt debug
22:32:36  Running with dbt=1.5.0
22:32:36  dbt version: 1.5.0
22:32:36  python version: 3.9.5
22:32:36  python path: /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/python
22:32:36  os info: Linux-5.15.0-1035-aws-x86_64-with-glibc2.31
22:32:36  Using profiles.yml file at /tmp/tmp-dbt-run-96368-profiles/profiles.yml
22:32:36  Using dbt_project.yml file at /tmp/tmp-dbt-run-96368/dbt_project.yml
22:32:36  Configuration:
22:32:38  Encountered an error:
cannot import name 'RefreshableCredentials' from 'databricks.sdk.oauth' (/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/databricks/sdk/oauth.py)
22:32:38  Traceback (most recent call last):
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/cli/requires.py", line 86, in wrapper
    result, success = func(*args, **kwargs)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/cli/requires.py", line 71, in wrapper
    return func(*args, **kwargs)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/cli/main.py", line 417, in debug
    results = task.run()
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/task/debug.py", line 110, in run
    self.test_configuration()
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/task/debug.py", line 294, in test_configuration
    profile_status = self._load_profile()
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/task/debug.py", line 254, in _load_profile
    profile: Profile = Profile.render(
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/config/profile.py", line 436, in render
    return cls.from_raw_profiles(
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/config/profile.py", line 401, in from_raw_profiles
    return cls.from_raw_profile_info(
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/config/profile.py", line 355, in from_raw_profile_info
    credentials: Credentials = cls._credentials_from_profile(
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/config/profile.py", line 165, in _credentials_from_profile
    cls = load_plugin(typename)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/adapters/factory.py", line 202, in load_plugin
    return FACTORY.load_plugin(name)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/adapters/factory.py", line 57, in load_plugin
    mod: Any = import_module("." + name, "dbt.adapters")
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 855, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/adapters/databricks/__init__.py", line 1, in <module>
    from dbt.adapters.databricks.connections import DatabricksConnectionManager  # noqa
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/adapters/databricks/connections.py", line 54, in <module>
    from databricks.sdk.oauth import OAuthClient, RefreshableCredentials
ImportError: cannot import name 'RefreshableCredentials' from 'databricks.sdk.oauth' (/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/databricks/sdk/oauth.py)

System information
dbt version
22:32:36  dbt version: 1.5.0
The operating system you're using:
os info: Linux-5.15.0-1035-aws-x86_64-with-glibc2.31
The output of python --version:
22:32:36  python version: 3.9.5
Additional context
I believe this is caused due to the latest databricks-sdk-py release with the change that was pushed here:
https://github.com/databricks/databricks-sdk-py/pull/116
The best fix would be to just bump the version of the databricks-sdk to 0.1.7 in the  dbt-databricks repository.
I'm happy to raise that PR.

Maybe fix the version of databricks-sdk in requirements.txt until the package reaches major release?

I think this issue has been fixed by the merged PR above.  If not, please reopen.


Why so many size-specific hash codes?

There seem to be unique codes defined for a lot of algorithm-plus-size combinations, especially for blake2 and skein... why is this not handled using the length specifier?

Unfortunately, the length specifier is only used for truncation as that's a function common to all hash functions (well, except the identity "hash" function).
However, I'm not sure why we didn't just allow the hash function itself to handle the length (e.g., sha* would truncate, etc.). @daviddias, were you part of this discussion.

Unfortunately, fixing this at this point may not be an option.

blake2b-256 truncated to 160bits gives different hash then blake2b-160 not truncated, so they are different hash functions
you can try it for example in python
import hashlib

# blake2b-160 full
>>> hashlib.blake2b(b'hello', digest_size=20).digest().hex()
'b5531c7037f06c9f2947132a6a77202c308e8939'

# blake2b-256 truncated to 20bytes = 160bits
>>> hashlib.blake2b(b'hello', digest_size=32).digest()[:20].hex()
'324dcf027dd4a30a932c441f365a25e86b173def'

# blake2b-256 full
>>> hashlib.blake2b(b'hello', digest_size=32).digest().hex()
'324dcf027dd4a30a932c441f365a25e86b173defa4b8e58948253471b81b72cf'


[LiveComponent] onUpdated don't triggered when using ComponentWithFormTrait with LiveProp to edit

          Hello, I also have the same problem,

I also have the same problem. After some research, I noticed that when using a LiveProp assigned to a form that is instantiated with ComponentWithFormTrait, the variable name must not be the same as the form name.
example:
    #[LiveProp(writable: true, updateFromParent: true, onUpdated: 'onUpdatedComponent')]
    public Component $component ;

    protected function instantiateForm() : FormInterface
    {
        return $this->createForm(ComponentType::class, $this->component) ;
    }

error :
An exception was thrown when rendering a model ("The field name ‘component’ cannot be used by multiple LiveProp properties in a component. Currently, “component” and “formValues” are both trying to use it in “xxx”. Try adding LiveProp(fieldName=“somethingElse”) for the “component” property to avoid this.").
Consequence:
If we change our variable like this
    #[LiveProp(writable : true, updateFromParent : true, onUpdated : 'onUpdatedComponent')]]
    public Component $c ;

    protected function instantiateForm() : FormInterface
    {
        return $this->createForm(ComponentType::class, $this->c) ;
    }

The onUpdated function cannot be triggered because, on line 664 of ux-live-component/src/LiveComponentHydrator.php, the $frontendName variable is equal to “c”. However, what we want is to have “component”, since on line 673, we check whether we have “c” in $dehydratedUpdatedProps. But the problem is that in this last variable, the Live Component only updates the “component” variable, since it's the form variable and not the variable name passed to LiveProps.
I hope I've made myself clear :D
Thanks in advance
Originally posted by @akyoscommunication in https://github.com/symfony/ux/issues/2057#issuecomment-2406793711

Did you try doing what the error suggest ?
   #[LiveProp(fieldName="foo", ....)]
    public Component $component


[JN-980] save preferred language during registration

DESCRIPTION
This sets the preferredLanguage profile field upon registration. Previously, we did not honor a user's preferred language immediately after registration (they would have to re-select it to actually set the profile field). This resulted in some of the content being loaded in the default language (english) and they'd also receive the welcome email in english.
TO TEST:  (simple manual steps for confirming core behavior -- used for pre-release checks)

In the demo study, select a language other than English
Register a new user (try this twice: once with b2c enabled and once with it disabled - it should work for both)
Verify that when you're redirected to the dashboard, the content is in the language that you originally selected prior to the b2c registration screens
Also confirm that you receive the welcome email in the correct language

@connorlbark yeah the B2c verification code email is out of scope. that work is captured in https://broadworkbench.atlassian.net/browse/JN-959


[C# client generation] combination of form url encoded operation and UseSystemTextJson results in invalid code

Client.Class.liquid contains the following code:
{%         elseif operation.ConsumesFormUrlEncoded -%}
                var json_ = Newtonsoft.Json.JsonConvert.SerializeObject({{ operation.ContentParameter.VariableName }}, _settings.Value);
                var dictionary_ = Newtonsoft.Json.JsonConvert.DeserializeObject<System.Collections.Generic.Dictionary<string, string>>(json_, _settings.Value);
                var content_ = new System.Net.Http.FormUrlEncodedContent(dictionary_);
                content_.Headers.ContentType = System.Net.Http.Headers.MediaTypeHeaderValue.Parse("{{ operation.Consumes }}");
{%         else -%}

Newtonsoft is used unconditionally in this code to convert the payload from an arbitrary object into a dictionary. However, the code also passes _settings.Value to SerializeObject and DeserializeObject. When UseSystemTextJson is true, _settings is of type System.Lazy<System.Text.Json.JsonSerializerOptions> and its value cannot be passed to Newtonsoft's methods. Therefore the generated code cannot be compiled.
It seems as though there should be additional conditional code in this block to branch based on the value of UseSystemTextJson and use either Newtonsoft's or System.Text.Json's serializer/deserializer.

Above mentioned problem still occurs. The culprit is currently at line 209 and 210:
https://github.com/RicoSuter/NSwag/blob/d8cca1c915a9c164ea13dfc2b8947dede2804024/src/NSwag.CodeGeneration.CSharp/Templates/Client.Class.liquid#L209


cosmos kit address not updating when switching keplr address after closing tab

the address showing in the cosmos kit as connected does not update  in this particular flow:

connect your webApp to keplr using comos kit
close the webApp tab
go to keplr extension and switch the address to a different address
open the webApp and let cosmos kit auto load the keplr data
the connected address shown is still the old address and not the newly switched address.

I can't reproduce it with the latest version. are you using the latest CosmosKit?

@kombos  The CosmosKit docs site is using the latest version and it works fine. https://docs.cosmoskit.com/

@kombos is correct, there are some issues with useChain. Even the Osmosis frontend is broken.
Example:

Go to Osmosis
Connect
Switch Accounts -> All good
Disconnect
Connect
Switch Accounts -> broken

I know that for some reason the example is working, maybe some forceful refresh somewhere but it doesn't matter how they got it working, the hook should always provide the correct result


Updating docs for wildcard path search dataflow connector

Updates made

Changed screenshot for source options with correct wildcard path
Corrected wildcard path for recursive folder search
Added additional example for multi level search

@mithunkr90 : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#49912

#sign-off

@djpmsft: I'm sorry - only the author of this article, @linda33wj, can sign off on your changes. But we do have an exception process - if you are on the Microsoft content or product team for this product area, you can ask the PR review team to review and merge it by sending mail to the techdocprs alias.

@djpmsft Should "/data/sales/20??/**/ ... Gets all files in the 20th century" be "Gets all files in the 21st century"?

@djpmsft Should "/data/sales/20??/**/ ... Gets all files in the 20th century" be "Gets all files in the 21st century"?

Good catch. Overlooked it as it was part of the original doc.


Setting up MongoDB Integration

What
This PR seeks to integrate MongoDB with Vectorflow
Why
Currently Vectorflow does not have a mongo connector, and should address #34
How
MongoDB as a database let's users store vector embeddings but is different from other vector databases, where in Mongo, one needs to associate a Search Index to a collection. This means that we'll be uploading our embeddings to a Mongo databases' collection, and that users of Mongo are expected to create their own search indexes which is associated with the collection. Since we cannot directly store our data in the Search Index, we store data in a collection as:
...
def write_embeddings_to_mongodb(upsert_list, vector_db_metadata):
    mongo_conn_uri = os.getenv('VECTOR_DB_KEY')
    try:
        mongo_client = pymongo.MongoClient(mongo_conn_uri)
    except Exception as e:
        logging.error(e)
        return None
    db_name, collection = vector_db_metadata.index_name.split(".")
    db = mongo_client[db_name]

    if collection not in db.list_collection_names():
        logging.error(f"Index {vector_db_metadata.index_name} does not exist in environment {vector_db_metadata.environment}")
        return None

    index = db.get_collection(collection)
...

A few things to note:

To connect to Mongo, we use PyMongo's Driver mongo_client = pymongo.MongoClient("<mongo_conn_uri>"). This means for this implementation, the VECTOR_DB_KEY is the connection uri.
Since we cannot specify the search index directly and we have to store embeddings in a collection, we pass a string mongo_databasename.collection_name to the  vector_db_metadata.index_name. The tests below were conducted with databasename=vector_embeds and collection_name=test.
While adding pymongo certain dependencies in the requirements.txt changed.

Verification
Prior to data upload:

Request to vectorflow:

After data upload on MongoDB:

Unit Test Results:

and

The last test failure, as agreed, is due an existing issue not related to Mongo integration, so I'm considering that all test cases have passed.

@dgarnitz, let me know what you think of the changes, and the notes I made

@farhan0167 with regards to your comment:
"In the code block for connecting to Mongo, when user enters the a wrong VECTOR_DB_KEY, vectorflow seems to get stuck in BATCH_PROCESSING state even though the expected behavior is to exit with a None. Not sure if this is desirable"
This is not desirable. The system needs to mark the batch as failed. Did you determine if it is actually returning None and just failing to update, or is it not reaching the return None part of the code?
If its not reaching that part of the code, is it because the system hangs trying to connect to the DB over and over again? If so, I'm pretty sure the MongoDB client will have some configuration that can tell it to only retry X times before failing or some type of timeout that you can set to like 10 seconds.

@dgarnitz I'll get to it. Also can you clarify one thing as well. What are your thoughts on the implementation where we need to pass the mongodb connection uri to the mongo_conn_uri = os.getenv('VECTOR_DB_KEY') as a VECTOR_DB_KEY? I was looking at the documentation last night, and I noticed that you've added uri's in the vector_db_metadata.environment previously. What purpose would environment serve for this implementation since I believe there is no unique environment variable that needs to be passed for Mongo. When I was testing it, I arbitrarily passed an environment variable mongo to it, but it does not serve any purpose in the code.

@farhan0167 if the connection URI has a piece that is the database password, pass that as the VECTOR_DB_KEY and the rest of it as the environment variable. The reason is that the environment value is stored in the DB whereas the key is not, so its more secure

@dgarnitz, I've resolved the issues you brought up, and below are some notes on the changes:

Rebased with the most recent changes pushed.
Removed the dependency versions from the requested packages
Fixed the part where previously inserting a wrong connection string kept the system in BATCH PROCESSING status instead of FAILED. The issue was in the way MongoDB connections work as referenced here. This essentially means that, for us to validate whether the connection string we passed succeeded was by pinging a database within that account.
As for the VECTOR_DB_KEY issue, users will need to pass the connection string to the environment variable as

mongodb+srv://<username>:<password>@cluster0.zin0tgq.mongodb.net/?retryWrites=true&w=majority

The user needs to modify the <username> with their username in the environment field, and pass the <password> as VECTOR_DB_KEY. The connection string is usually retrieved from the Atlas Console->Databases->Connect->Drivers->Python as:
.
5. Reran all the tests with success:

Additionally, in terms of success with uploading to Mongo, everything works and uploads as expected like before.