Mark skipped tests as skipped

Attempting to fix #6 -- mark skipped tests as skipped (instead of only renaming them).
I've tested this on our local Jenkins. The test (the first in the list) is marked as skipped:

Thanks for the pull request! I hope to take a deeper look at it in the next few days, but feel free to poke me again if I forget (which I do too frequently).

This seems to be two separate feature changes:

Mark skipped tests as skipped.
Mark pending tests as failures.

As such, I think I'd like to see this as two separate PRs. This helps keep the history clean when looking back. How do you feel about that split? If you agree, we can repurpose this PR as whichever of the two you prefer. :palm_tree:

Sounds good. I've left the 'Mark skipeed tests as skipped' part in this PR.
The pending tests thing will go into a separate PR.

FYI: The pending=failure part live in #10 now.

/me pokes @shepmaster :)
(but don't feel pressured to work on a weekend) :sunny: :surfer:

@shepmaster could you please have a look?

@shepmaster is there something I could help you with making your decision about this (and the other) PR easier?
I know this is open source software and you have other things to do*, but some feedback about the PRs would be appreciated :green_heart:
*) I found you by accident on stackoverflow. Love your answers on the rust tag! :)

is there something I could help you with

Thank you for asking, and for being so understanding! I meant to get to this during the Thanksgiving weekend, but got caught up! This looks good to me.
Thanks for the contribution! On to the next PR!

Thanks you for your work! And I've seen what you did in #12 -- don't hesitate to ask for more tests in pull requests next time. :)
I'll happily provide those to make your life as a maintainer easier.

don't hesitate to ask for more tests in pull requests

Normally I would, but after letting the PR dangle for so long, I felt that I might as well make the tiny changes I wanted without wasting any more of your time. :+1:


I hope you add some for JAVA devs

I noticed i could only see nmarulo in your list specifically for JAVA.
I hope there is more.

Do you know of any? You could submit some you've found too!  :)
I'll go look for some too. I've been searching around a bunch to try and find some more in the Java/Enterprisey realms.

Just found this repo, so will be submitting my Java stream (twitch.tv/jitterted).

I'll welcome any Java streamers that people manage to find. With adding Ted to the list, I'm going to close this one.


chore(deps): upgrade dependencies

Upgrades project dependencies. See details in workflow run.

Automatically created by projen via the "upgrade-k8s-23-main" workflow

⚪ Backport skipped
The pull request was not backported as there were no branches to backport to. If this is a mistake, please apply the desired version labels or run the backport tool manually.
Manual backport
To create the backport manually run:
backport --pr 1048

Questions ?
Please refer to the Backport tool documentation


feat(content-banner): Fix card margin inside content-banner. - OEL-2288

Tested adding the new rule on UCPKN and it fixes the issue reported in #517


useDrag() Hook Changes

The useDrag hook has two modes of operation: building out return items with spec.item and with spec.begin(). When you use spec.begin(), objects are expected to be of the type in spec.item.type, which muddles concepts and can lead to some confusion. The deepest change here is separating the concerns of draggable type from draggable items.
This change is a major semver change that refactors the useDrag() ergonomics.

spec.type is required. This should be the same value that was in spec.item.type. This simplifies use cases that only use drag-types and no further drag-object details.
spec.item can be an object or function. The object does not need to contain the type property. The function may return an item object or null. null cancels the drag operation. The function form of item replaces begin, which is no longer present.
end() provides the dragObject and dropResult as arguments. (e.g. end(item, result, monitor))

The typings for monitors has been tightened. The interfaces for DragSourceMonitor and DropTargetMonitor now have type parameters for the DragObject and DropResult which default to unknown. Furthermore the getItem() and getResult() methods can be parameterized with the type expected (default is monitor typings).
Finally, dnd-core has been updated so that if a dragHandler returns null or undefined, the system will cancel the drag operation. This allows clients using useDrag({ item: () => condition ? null: validItem }) to cancel drag operations with client-side logic.
Bugfixes include correcting liveness issues with hooks that were evident in the Dustbin stress test and fixing infinite render loops from shallowEqual being used when deepEqual is necessary.
Fixes #3110

@darthtrevino It might be good idea to put deprecated warning when someone tries to usebegin function. Right now it's bit difficult to trace what went wrong after update, especially when this is not listed as breaking change in changelog.

Right - I struggled a bit with whether to deprecate begin or just nuke it with this release. At the very least we can add a friendly developer message if someone defines spec.begin to let them know to switch to spec.item

I'm adding a developer message here - https://github.com/react-dnd/react-dnd/pull/3151


Hydra-router request timeout is not passed to Hydra Core

The value for request timeout can be set in the config file, but when the call is created using Hydra core the options.timeout is not present so the default of 30 secs is used. However there are some processes that can take longer than 30 secs.
Am I interpreting correctly this parameter? if that is the case then when this call is done:
hydra.makeAPIRequest(msg, {timeout: this.requestTimeout})
The msg will be passed to
makeAPIRequest(message) {
return super._makeAPIRequest(message);
}
But not the timeout, so the actual value in Hydra core will be NaN therefore the value selected will be:
REQUEST_TIMEOUT = 30000;
socket.setTimeout(options.timeout * 1000 || REQUEST_TIMEOUT, () => {

@edcjrh You're absolutely correct. Would you like to issue a pull request or would you like us to update the makeAPIRequest(message) function signature to allow for sendOpts to be passed?  This is already present in the _makeAPIRequest(message, sendOpts = { }) signature.

Hi Carlos,
It would be ok if you update the function to allow fro sendOpts to be passed.
Thank you!!
Regards,
Juan

From: Carlos Justiniano<EMAIL_ADDRESS>Sent: Wednesday, May 23, 2018 4:34 PM
To: flywheelsports/hydra-router
Cc: Juan Huertas; Mention
Subject: Re: [flywheelsports/hydra-router] Hydra-router request timeout is not passed to Hydra Core (#178)
@edcjrhhttps://github.com/edcjrh You're absolutely correct. Would you like to issue a pull request or would you like us to update the makeAPIRequest(message) function signature to allow for sendOpts to be passed? This is already present in the _makeAPIRequest(message, sendOpts = { }) signature.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/flywheelsports/hydra-router/issues/178#issuecomment-391414749, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ASudVqQBkubOh8RWJ2g0j7uk3Kp1Gnabks5t1Y-zgaJpZM4UKVI1.

@edcjrh the fix is now available in hydra:1.6.8 and hydra-express:1.6.9 . I haven't updated Hydra-Router to use hydra.1.6.8 as we have a new hydra-router currently in test.  Feel free to update your hydra-router package.json in the meantime.  Thanks for pointing out this issue.

Sorry for the really late comment.
The fix works perfectly, thanks!


no hacky overwriting of typeinf_local

And the changes also remove the dirty overwriting of typeinf_local

okay, it turns out that enabling optimization (and thus integrating error reports with internal code cache) needs some work.
I would like to make this PR only remove the hacky overwriting of typeinf_local. Getting rid of that has one drawback that now we can't get "as much errors as possible" in a single gf call (e.g. fib("1000") in demo.jl) because native interpreter's typeinf_local early escape as soon as it uncovers Union{}. So for example now we only get 4 error reports for demo.jl, but it's acceptable imho because at least one error point would be reported for one toplevel gf call anyway. Apart from that, it is clearly better and preferable in terms of maintenance and performance.


[CodeCompletion] Fast completion for top-level code in single file script

(based on #28904 )
e.g. Playground.
A single file script is like a single function body; the interface of the file does not affect any other files. So when a completion happens in a single file script, re-parse the whole file. But we are still be able to reuse imported modules.
rdar://problem/58378157

@swift-ci Please smoke test

@swift-ci Please smoke test

Can one of the admins verify this patch?

@swift-ci Please smoke test

@benlangmuir Could you take another look?

@swift-ci Please smoke test Linux

@swift-ci Please smoke test macOS

@swift-ci Please smoke test macOS

@swift-ci Please smoke test OS X platform


click latest release as a dependency to typer causes error

Hi - I use typer and it has click as a dependency.  The latest click release 8.1.0 does not work and gives the error below - 8.0.4 works so my workaround is to explicitly specify 8.0.4.
Traceback (most recent call last):
  File  line 158, in <module>
    import typer
  File "/lib/python3.7/site-packages/typer/__init__.py", line 12, in <module>
    from click.termui import get_terminal_size as get_terminal_size
    ImportError: cannot import name 'get_terminal_size' from 'click.termui' (/lib/python3.7/site-packages/click/termui.py)

I know it's not a lot of information but maybe it means something to developers here.  Not sure if this is more appropriate to post as a typer issue.

This is already addressed in the typer repo.

This was already issuing a deprecation warning, documented as such in the changelog, and then documented as removed in the changelog.
Use a tool like pip-tools to pin your dependency tree and control when you get updates. Be sure to run your tests with deprecation warnings treated as errors so that you get notified of these types of changes early.

The typer maintainer is currently at a conference, I'm sure they'll make a release when they get back and have the ability to.

Hello! I just released Typer 0.4.1 that should handle this. :rocket: :nerd_face:
Thanks for the conversation and for understanding!

FWIW MO is that this and https://github.com/pallets/click/issues/2232 and https://github.com/psf/black/issues/2964  and https://github.com/pallets/click/issues/2228caused roughly 1/2 of the world's Python CI jobs to break ;)

Use a tool like pip-tools to pin your dependency tree and control when you get updates. Be sure to run your tests with deprecation warnings treated as errors so that you get notified of these types of changes early.

This is usually not an option as various libraries would pin on different versions and this will make things fail even more.
May I suggest  this should have been a version 9 rather than a version 8.1?


Add Facebook icon linked to openESTdata group

https://www.facebook.com/groups/1792458637715167
Next to "Lisainfo" on the menu bar.

Are there any other social media icons we should add?

Are there any other social media icons we should add?

Twitter also: https://twitter.com/openESTdata

It is absolutely fine! Great idea!


feat: Add variants metrics to SDK

This PR adds support for variant metrics to the Rust SDK. It's about 95 lines of implementation and 150 lines of tests.
Variant metrics should be counted whenever get_variant is called:

if a feature doesn't have variants, register the disabled variant
if a feature is disabled, register the disabled variant
if a feature doesn't exist, it should still register the disabled variant.
otherwise, register the variant that was returned

Discussion points
I've implemented the variant metrics on each feature instance as a DashMap (and thus also added a new crate).
Currently, each feature will get a new DashMap initialized. It gets initialized with a capacity of 0.
It may be worth making the DashMap an Option and only initializing it when it's requested, but that might require a fair
bit of trickery (and suggestions on how to do it would be very welcome).
Tests
I've added tests to ensure that metrics are being counted and that the conversion from CachedFeature to ToggleMetrics works as expected. I've added new tests for these, but the tests for variant counting could be added to the existing get_variant tests. However, I felt that would muddy the tests, so I preferred creating new tests.
I have not tested the serde serialization output, because I don't think that would add much value. But I'd be happy to put in a test for that too if you think it would be useful.
Benches
Of course, benchmarks are a fickle thing, but I figured they'd be useful to make sure there's no giant blunders. Running it locally on my machine against main, I found that most benchmarks have no change or it is within the noise threshold. Three benches indicated performance improved, and three indicated that it had regressed.
Most of the improvements and regressions are =< ~6%, so I don't know how much stock to put in them. However, one regression, batch/parallel unknown-features(str) reported a throughput decrease of 24%, so that may be significant.
Performance improvements
While I don't see how this would improve performance in any way, the following benches were reported as improved:
single_call/single_call(str)
                        time:   [253.58 ns 254.46 ns 255.51 ns]
                        thrpt:  [3.9137 Melem/s 3.9299 Melem/s 3.9435 Melem/s]
                 change:
                        time:   [-3.5225% -2.9598% -2.3842%] (p = 0.00 < 0.05)
                        thrpt:  [+2.4425% +3.0501% +3.6511%]
                        Performance has improved.

batch/parallel distinct-features(enum)
                        time:   [24.728 ms 24.762 ms 24.838 ms]
                        thrpt:  [16.104 Melem/s 16.154 Melem/s 16.176 Melem/s]
                 change:
                        time:   [-6.5260% -5.8273% -5.1205%] (p = 0.00 < 0.05)
                        thrpt:  [+5.3969% +6.1878% +6.9816%]
                        Performance has improved.

batch/parallel unknown-features(enum)
                        time:   [2.9772 ms 2.9899 ms 3.0029 ms]
                        thrpt:  [133.20 Melem/s 133.79 Melem/s 134.35 Melem/s]
                 change:
                        time:   [-6.5641% -6.0584% -5.5674%] (p = 0.00 < 0.05)
                        thrpt:  [+5.8956% +6.4491% +7.0252%]
                        Performance has improved.

Performance regressions
The following regressions were reported:
batch/single thread(str)
                        time:   [12.815 ms 12.892 ms 12.974 ms]
                        thrpt:  [3.8537 Melem/s 3.8783 Melem/s 3.9016 Melem/s]
                 change:
                        time:   [+1.3055% +2.0763% +2.8471%] (p = 0.00 < 0.05)
                        thrpt:  [-2.7683% -2.0341% -1.2887%]
                        Performance has regressed.

batch/parallel same-feature(str)
                        time:   [28.019 ms 28.085 ms 28.170 ms]
                        thrpt:  [14.199 Melem/s 14.242 Melem/s 14.276 Melem/s]
                 change:
                        time:   [+1.2655% +1.8355% +2.4361%] (p = 0.00 < 0.05)
                        thrpt:  [-2.3782% -1.8025% -1.2497%]
                        Performance has regressed.

batch/parallel unknown-features(str)
                        time:   [4.2516 ms 4.2622 ms 4.2794 ms]
                        thrpt:  [93.470 Melem/s 93.849 Melem/s 94.082 Melem/s]
                 change:
                        time:   [+30.189% +31.927% +33.541%] (p = 0.00 < 0.05)
                        thrpt:  [-25.116% -24.200% -23.188%]
                        Performance has regressed.

@sighphyre As mentioned in the comment, I  do think that the one significant regression may be related to dashmap initialization. I'll see if I can figure a way around it and if it changes anything.
As for clippy: yeah, I got those lints locally as well, but I didn't want to touch them because those lines were unrelated to the PR. Happy to fix them up, though!

Edit: I was slightly off earlier. When you check is_enabled_str for a feature that doesn't exist, that's when we initialize a dashmap. That's gonna be harder to get around.

Hey, so a few thoughts.
Firstly, great to see work coming in! Forgive my commenting just on the description, its all the time I have today.
I presume that this is not running conformance tests - we kind of need to fix up https://github.com/Unleash/unleash-client-rust/pull/42 and merge it and then repeat for the other commits to the conformance test suite.
Being up to date with that suite is really important to me because it lets us see that the official API works with new features, and I'd really encourage work on that before new features - because we have to manually test all new features and can't lean on the conformance test suite as much as we might otherwise.
Secondly, on dashmap - I need to look at the code I think. I think a rwlock in every get_variant call is a bit much - the basic ideal I think we're aiming for is 'smart if's', which means (to me) that it shouldn't matter how often unleash is called into, its not meaningfully worse than using 'if some_manual_feature_constant'. In particular the cold path concerns me - if the API is down, or someone deletes the toggle from the API, we shouldn't suddenly slow things down. Admittedly this SDK has a lot of headroom, but thats only because we've paid attention to overheads all the way :>.
In terms of code alternatives, I think generating an efficient lock-free structure from the API document, putting in the RCU ARC, and then indexing into it should be very efficient, lock tree, and quite easy to code.

Closing this in favor of #60


update tools

solves #20

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.githubsands seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


[QUESTION] How to set a parameter from Body OR Query

First check

[x] I used the GitHub search to find a similar issue and didn't find it.
[x] I searched the FastAPI documentation, with the integrated search.
[x] I already searched in Google "How to X in FastAPI" and didn't find any information.

Description
I am trying to setup a dependency that will accept a paramter from a Body or Query (it will be used as a dependency for POST and GET requests.
I need to be able to look for the env parameter in either the Body or Query. Something like this maybe:
def testFactory(env: str = Body(...) or Query(...)):
	testDict = {
		'testenv1' = TestObject1(),
		'testenv2' = TestObject2()
	}
	return testDict.get(env)

I'm an amateur programmer, so please let me know if I'm just not doing it right!

This is my current workaround, but it feels like it could be better.  Basically looking for this functionality in a single dependency function or class instead of 3
def testDict():
	return {
		'testenv1' = TestObject1,
		'testenv2' = TestObject2
	}

def qTestFactory(env: str = Query(...), testObject = Depends(testDict)):
	return testObject.get(env)()

def bTestFactory(env: str = Body(...), testObject = Depends(testDict)):
	return testObject.get(env)()

@taoofshawn here's an example of doing what I think you want to do:
from typing import Optional

import uvicorn
from fastapi import FastAPI, Depends, Query, Body, HTTPException
from starlette.status import HTTP_400_BAD_REQUEST

app = FastAPI()

def get_env(
    query_env: Optional[str] = Query(None, alias="env"),
    body_env: Optional[str] = Body(None, alias="env"),
) -> str:
    if query_env is not None:
        return query_env
    if body_env is not None:
        return body_env
    raise HTTPException(
        HTTP_400_BAD_REQUEST, detail="You must provide 'env' in body or query"
    )

@app.get("/")
def test(env: str = Depends(get_env)):
    return env

@app.post("/")
def test(env: str = Depends(get_env)):
    return env

if __name__ == "__main__":
    uvicorn.run(app)

However, I strongly recommend that you do not take this approach. There are a few problems here:

It's unclear the users which env will be used if they provide both.
It's not clear that env is required because we have the make the parameters optional in order to be able to choose a location.

I recommend that you either just put this option in query (which you can do for both POST and GET) or have two different dependencies and select one per endpoint.  Something like this:
import uvicorn
from fastapi import FastAPI, Depends, Query, Body

app = FastAPI()

def query_env(env: str = Query(...)) -> str:
    return env

def body_env(env: str = Body(...)) -> str:
    return env

@app.get("/")
def test(env: str = Depends(query_env)):
    return env

@app.post("/")
def test(env: str = Depends(body_env)):
    return env

if __name__ == "__main__":
    uvicorn.run(app)

At that point it's less code to just include the params directly in your route rather than use a dependency. I am assuming you need a dependency so you can do something else with the env.

You're right, I think it would be better to keep "env" as a query parameter for all requests and just also accept a body for the other params for posts.  I think it got (incorrectly) burned into my head somewhere to always use query for gets and form or body for posts.
thanks @dbanty !

Thanks for the help here @dbanty ! :bow:
Thanks for reporting back and closing the issue @taoofshawn :+1:

It would be convenient if fastapi did support this type of usage without having to create two separate duplicate path operators (just decorated differently).  Something equivalent to "get_arguments" in tornado.  Nevertheless, it's helpful to hear that the best approach currently is to define two separate path operators.
Maybe one could make a single pydantic class to represent the common input args and somehow reuse it for GET and POST's?

@falkben if you are needing to put that in many places, yeah, you could just create a class dependency to get and store the alternative parameters.


InvalidArgumentError (see above for traceback): Incompatible shapes: [0,4] vs. [16,4]

When I run luminoth training  with fasterrcnn + vgg16 and coco datdasets，I got this error infomation
InvalidArgumentError (see above for traceback): Incompatible shapes: [0,4] vs. [16,4]
         [[{{node losses/RCNNLoss/sub_1}} = Sub[_device="/job:localhost/replica:0/task:0/device:GPU:0"](losses/RCNNLoss/bbox_offset_cleaned/GatherV2, losses/RCNNLoss/bbox_offsets_target_labeled/GatherV2)]]
         [[{{node fasterrcnn/rcnn/rcnn_proposal_1/BoundingBoxTransform/change_order_19/stack/_1589}} = _Recv[_device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

below is my config file , is there something wrong with it ?
train:
  # Run on debug mode (which enables more logging).
  debug: True
  # Seed for random operations.
  seed:
  # Training batch size for images. FasterRCNN currently only supports 1.
  batch_size: 1
  # Base directory in which model checkpoints & summaries (for Tensorboard) will
  # be saved.
  job_dir: jobs/
  # Ignore scope when loading from checkpoint (useful when training RPN first
  # and then RPN + RCNN).
  ignore_scope:
  # Enables TensorFlow debug mode, which stops and lets you analyze Tensors
  # after each `Session.run`.
  tf_debug: False
  # Name used to identify the run. Data inside `job_dir` will be stored under
  # `run_name`.
  run_name:
  # Disables logging and saving checkpoints.
  no_log: False
  # Displays debugging images with results every N steps. Debug mode must be
  # enabled.
  display_every_steps:
  # Display debugging images every N seconds.
  display_every_secs: 300
  # Shuffle the dataset. It should only be disabled when trying to reproduce
  # some problem on some sample.
  random_shuffle: True
  # Save Tensorboard timeline.
  save_timeline: False
  # The frequency, in seconds, that a checkpoint is saved.
  save_checkpoint_secs: 600
  # The frequency, in number of global steps, that the summaries are written to
  # disk.
  save_summaries_steps:
  # The frequency, in seconds, that the summaries are written to disk.  If both
  # save_summaries_steps and save_summaries_secs are set to empty, then the
  # default summary saver isn't used.
  save_summaries_secs: 30
  # Run TensorFlow using full_trace mode for memory and running time logging
  # Debug mode must be enabled.
  full_trace: False
  # Clip gradients by norm, making sure the maximum value is 10.
  clip_by_norm: False
  # Learning rate config.
  learning_rate:
    # Because we're using kwargs, we want the learning_rate dict to be replaced
    # as a whole.
    _replace: True
    # Learning rate decay method; can be: ((empty), 'none', piecewise_constant,
    # exponential_decay, polynomial_decay) You can define different decay
    # methods using `decay_method` and defining all the necessary arguments.
    decay_method:
    learning_rate: 0.0003

  # Optimizer configuration.
  optimizer:
    # Because we're using kwargs, we want the optimizer dict to be replaced as a
    # whole.
    _replace: True
    # Type of optimizer to use (momentum, adam, gradient_descent, rmsprop).
    type: momentum
    # Any options are passed directly to the optimizer as kwarg.
    momentum: 0.9

  # Number of epochs (complete dataset batches) to run.
  num_epochs: 1000

  # Image visualization mode, options = train, eval, debug, (empty).
  # Default=(empty).
  image_vis: train
  # Variable summary visualization mode, options = full, reduced, (empty).
  var_vis:

eval:
  # Image visualization mode, options = train, eval, debug,
  # (empty). Default=(empty).
  image_vis: eval

dataset:
  type: object_detection
  # From which directory to read the dataset.
  dir: '/data/Datasets/coco/tf'
  # Which split of tfrecords to look for.
  split: train
  # Resize image according to min_size and max_size.
  image_preprocessing:
    min_size: 600
    max_size: 1024
  # Data augmentation techniques.
  data_augmentation:
    - flip:
        left_right: True
        up_down: False
        prob: 0.5
    # Also available:
    # # If you resize to too small images, you may end up not having any anchors
    # # that aren't partially outside the image.
    # - resize:
    #     min_size: 600
    #     max_size: 1024
    #     prob: 0.2
    # - patch:
    #     min_height: 600
    #     min_width: 600
    #     prob: 0.2
    # - distortion:
    #     brightness:
    #       max_delta: 0.2
    #     hue:
    #       max_delta: 0.2
    #     saturation:
    #       lower: 0.5
    #       upper: 1.5
    #     prob: 0.3

model:
  type: fasterrcnn
  network:
    # Total number of classes to predict.
    num_classes: 20
    # Use RCNN or just RPN.
    with_rcnn: True

  # Whether to use batch normalization in the model.
  batch_norm: False

  base_network:
    # Which type of pretrained network to use.
    architecture: vgg_16
    # Should we train the pretrained network.
    trainable: True
    # From which file to load the weights.
    weights:
    # Should we download weights if not available.
    download: False
    # Which endpoint layer to use as feature map for network.
    endpoint: conv5/conv5_1
    # Starting point after which all the variables in the base network will be
    # trainable. If not specified, then all the variables in the network will be
    # trainable.
    fine_tune_from: conv4/conv4_2
    # Whether to train the ResNet's batch norm layers.
    train_batch_norm: False
    # Whether to use the base network's tail in the RCNN.
    use_tail: True
    # Whether to freeze the base network's tail.
    freeze_tail: False
    # Output stride for ResNet.
    output_stride: 16
    arg_scope:
      # Regularization.
      weight_decay: 0.0005

  loss:
    # Loss weights for calculating the total loss.
    rpn_cls_loss_weight: 1.0
    rpn_reg_loss_weights: 2.0
    rcnn_cls_loss_weight: 1.0
    rcnn_reg_loss_weights: 2.0

  anchors:
    # Base size to use for anchors.
    base_size: 256
    # Scale used for generating anchor sizes.
    scales: [0.5, 1, 2]
    # Aspect ratios used for generating anchors.
    ratios: [0.5, 1, 2]
    # Stride depending on feature map size (of pretrained).
    stride: 16

  rpn:
    activation_function: relu6
    l2_regularization_scale: 0.0005  # Disable using 0.
    # Sigma for the smooth L1 regression loss.
    l1_sigma: 3.0
    # Number of filters for the RPN conv layer.
    num_channels: 512
    # Kernel shape for the RPN conv layer.
    kernel_shape: [3, 3]
    # Initializers for RPN weights.
    rpn_initializer:
      _replace: True
      type: variance_scaling_initializer
      #mean: 0.0
      #stddev: 0.01
      factor: 1.0
      uniform: True
      mode: FAN_AVG

    cls_initializer:
      _replace: True
      type: truncated_normal_initializer
      mean: 0.0
      stddev: 0.01
    bbox_initializer:
      _replace: True
      type: truncated_normal_initializer
      mean: 0.0
      stddev: 0.001

    proposals:
      # Total proposals to use before running NMS (sorted by score).
      pre_nms_top_n: 12000
      # Total proposals to use after NMS (sorted by score).
      post_nms_top_n: 2000
      # Option to apply NMS.
      apply_nms: True
      # NMS threshold used when removing "almost duplicates".
      nms_threshold: 0.7
      min_size: 0  # Disable using 0.
      # Run clipping of proposals after running NMS.
      clip_after_nms: False
      # Filter proposals from anchors partially outside the image.
      filter_outside_anchors: False
      # Minimum probability to be used as proposed object.
      min_prob_threshold: 0.0

    target:
      # Margin to crop proposals to close to the border.
      allowed_border: 0
      # Overwrite positives with negative if threshold is too low.
      clobber_positives: False
      # How much IoU with GT proposals must have to be marked as positive.
      foreground_threshold: 0.6
      # High and low thresholds with GT to be considered background.
      background_threshold_high: 0.3
      background_threshold_low: 0.0
      foreground_fraction: 0.5
      # Ration between background and foreground in minibatch.
      minibatch_size: 256
      # Assign to get consistent "random" selection in batch.
      random_seed:  # Only to be used for debugging.

  rcnn:
    layer_sizes: []  # Could be e.g. `[4096, 4096]`.
    dropout_keep_prob: 1.0
    activation_function: relu6
    l2_regularization_scale: 0.0005
    # Sigma for the smooth L1 regression loss.
    l1_sigma: 1.0
    # Use average pooling before the last fully-connected layer.
    use_mean: True
    # Variances to normalize encoded targets with.
    target_normalization_variances: [0.1, 0.2]

    rcnn_initializer:
      _replace: True
      type: variance_scaling_initializer
      factor: 1.0
      uniform: True
      mode: FAN_AVG
    cls_initializer:
      _replace: True
      type: random_normal_initializer
      mean: 0.0
      stddev: 0.01
    bbox_initializer:
      _replace: True
      type: random_normal_initializer
      mean: 0.0
      stddev: 0.001

    roi:
      pooling_mode: crop
      pooled_width: 7
      pooled_height: 7
      padding: VALID

    proposals:
      # Maximum number of detections for each class.
      class_max_detections: 100
      # NMS threshold used to remove "almost duplicate" of the same class.
      class_nms_threshold: 0.6
      # Maximum total detections for an image (sorted by score).
      total_max_detections: 300
      # Minimum prob to be used as proposed object.
      min_prob_threshold: 0.0

    target:
      # Ratio between foreground and background samples in minibatch.
      foreground_fraction: 0.25
      minibatch_size: 64
      # Threshold with GT to be considered positive.
      foreground_threshold: 0.5
      # High and low threshold with GT to be considered negative.
      background_threshold_high: 0.5
      background_threshold_low: 0.1

Using the VGG as the base network isn't working currently. You should use the ResNet101 instead, as you'll get much better results. In fact, we're planning on removing the rest of the base networks in a future release, except maybe for a smaller ResNet.
If you want a faster model, you might try using SSD instead.

I Use the ResNet101， but I still got the same error，the config file I use is ../models/fasterrcnn/base_config.yml， with coco datasets
root@G560-ubuntu1:/home/p00306424/luminoth/luminoth/models/fasterrcnn# lumi train -c base_config.yml
INFO:tensorflow:Training 279 vars from pretrained module; from "truncated_base_network/resnet_v1_101/block2/unit_1/bottleneck_v1/shortcut/weights:0" to "truncated_base_network/resnet_v1_101/block4/unit_3/bottleneck_v1/conv3/BatchNorm/beta:0".
/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gradients_impl.py:108: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
  "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
INFO:tensorflow:Starting training for <luminoth.models.fasterrcnn.fasterrcnn.FasterRCNN object at 0x7f0793b67fd0>
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/util/tf_should_use.py:118: initialize_local_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use `tf.local_variables_initializer` instead.
INFO:tensorflow:ImageVisHook was created with mode = "train"
INFO:tensorflow:Create CheckpointSaverHook.

Traceback (most recent call last):
  File "/usr/local/bin/lumi", line 11, in <module>
    load_entry_point('luminoth==0.2.1.dev0', 'console_scripts', 'lumi')()
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/luminoth-0.2.1.dev0-py2.7.egg/luminoth/train.py", line 307, in train
    config, environment=environment
  File "/usr/local/lib/python2.7/dist-packages/luminoth-0.2.1.dev0-py2.7.egg/luminoth/train.py", line 239, in run
    ], options=run_options)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 583, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1059, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1150, in run
    raise six.reraise(*original_exc_info)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1135, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1207, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 987, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 887, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1110, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1286, in _do_run
    run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1308, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [0,4] vs. [3,4]
         [[{{node losses/RCNNLoss/sub_1}} = Sub[_device="/job:localhost/replica:0/task:0/device:GPU:0"](losses/RCNNLoss/bbox_offset_cleaned/GatherV2, losses/RCNNLoss/bbox_offsets_target_labeled/GatherV2)]]
         [[{{node losses/RCNNLoss/strided_slice_1/_4893}} = _Recv[_device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Caused by op u'losses/RCNNLoss/sub_1', defined at:
  File "/usr/local/bin/lumi", line 11, in <module>
    load_entry_point('luminoth==0.2.1.dev0', 'console_scripts', 'lumi')()
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/luminoth-0.2.1.dev0-py2.7.egg/luminoth/train.py", line 307, in train
    config, environment=environment
  File "/usr/local/lib/python2.7/dist-packages/luminoth-0.2.1.dev0-py2.7.egg/luminoth/train.py", line 67, in run
    total_loss = model.loss(prediction_dict)
  File "/usr/local/lib/python2.7/dist-packages/luminoth-0.2.1.dev0-py2.7.egg/luminoth/models/fasterrcnn/fasterrcnn.py", line 192, in loss
    prediction_dict['classification_prediction']
  File "/usr/local/lib/python2.7/dist-packages/luminoth-0.2.1.dev0-py2.7.egg/luminoth/models/fasterrcnn/rcnn.py", line 391, in loss
    sigma=self._l1_sigma
  File "/usr/local/lib/python2.7/dist-packages/luminoth-0.2.1.dev0-py2.7.egg/luminoth/utils/losses.py", line 22, in smooth_l1_loss
    diff = bbox_prediction - bbox_target
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py", line 851, in binary_op_wrapper
    return func(x, y, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_math_ops.py", line 8246, in sub
    "Sub", x=x, y=y, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3260, in create_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Incompatible shapes: [0,4] vs. [3,4]
         [[{{node losses/RCNNLoss/sub_1}} = Sub[_device="/job:localhost/replica:0/task:0/device:GPU:0"](losses/RCNNLoss/bbox_offset_cleaned/GatherV2, losses/RCNNLoss/bbox_offsets_target_labeled/GatherV2)]]
         [[{{node losses/RCNNLoss/strided_slice_1/_4893}} = _Recv[_device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

root@G560-ubuntu1:/home/p00306424/luminoth/luminoth/models/fasterrcnn#


Possible bug in Attitude Error

When the m_quaternionGain is increased it makes one of the motors spin faster while the rest remain still.
By increasing the PITCH_MAX_RAD in config.hpp it made it more noticeable.
I've tried to do some debugging over the last few days but its still present.
First I checked the control actions to the motor equations (end of main.cpp) and they where correct.
When the referenceQuaternion is set to (0,0,0) it seems to go away.
I checked the quaternion.cpp file and the quaternion from NED Euler angles is correct.
I then checked the mapRadioToAngle function in the radio.cpp and the formula correct therefore I can't see anything wrong with the  referenceQuaternion.
I also printed the relativeQuaternion and it seemed to be behaving correctly so I'm not sure what the issue is, was thinking maybe its something to do with subtracting one quaternion from the other, but not sure.

Issue resolved
Issue was mapRadioToAngle function in radio.cpp file


Update index.vue

Please makes sure the items are checked before submitting your PR, thank you!

[ ] Rebase before creating a PR to keep commit history clear.
[ ] Only One commit
[ ] No eslint errors

x-input right slot 没有正确结束

Thanks


Gem doesn't work with S3

Using the following configs I get uploaded file with 0 byte file uploaded to s3 bucket.
#db/migrate/20170216113435_add_p_to_user.rb
class AddPToUser < ActiveRecord::Migration[5.0]
  def change
    add_column :users, :avatar_processing, :boolean
  end
end

# app/models/user.rb
class User < ApplicationRecord
  has_attached_file :avatar, styles: {thumb: "100x100#"}
  validates_attachment_content_type :avatar, content_type:["image/jpg", "image/jpeg", "image/png", "image/gif"]

  process_in_background :avatar
end

# config/application.rb
#....

  class Application < Rails::Application
    config.active_job.queue_adapter = :sidekiq

    config.paperclip_defaults = {
      storage: :s3,
      s3_region: 'us-west-2',
      s3_host_name: "s3-us-west-2.amazonaws.com",
      s3_credentials: {
        bucket: 'bucket_name',
        access_key_id: 'xxx',
        secret_access_key: 'xxx',
        s3_region: 'us-west-2',
      }
    }
  end

Extra info
  rails (~> 5.0.1)
  delayed_paperclip (~> 3.0, >= 3.0.1)
  paperclip (5.1.0)
  delayed_paperclip (3.0.1)
  sidekiq (4.2.9)
  paperclip-av-transcoder (0.6.4)

#sidekiq log
2017-02-16T12:01:29.951Z 7471 TID-rm9ck DelayedPaperclip::ProcessJob JID-e16b95988df375eab98f7b99 INFO: start
2017-02-16T12:01:33.730Z 7471 TID-rm9ck DelayedPaperclip::ProcessJob JID-e16b95988df375eab98f7b99 INFO: done: 3.779 sec

The same problem occurs when I use the gem with rails (= <IP_ADDRESS>) and paperclip-av-transcoder postprocessor.

I'm having this same issue, my uploads work fine without the gem enabled but as soon as I place the setting for processing in background nothing gets uploaded besides the original image.

I would suggest trying Paperclip v5.0.0. I was having problems with 5.1.0 and I'm using s3, but using v5.0.0 seems to fix most of my problems.

same issue and fixed by downgrading aws-sdk from 3.0.1 to 2.10.39

On my case the DelayedPaperclip worker doesn't even appear on sidekiq log. And It stills upload on normal way to S3.
I have downgraded
aws-sdk from 3.0.1 to 2.10.39
Paperclip 5.1.0 to 5.0.0
And the same problem happens.


NTR: LSM1-7 & LSM2-8 complex

Neither of these complexes have a defined function (yet). But they are highly conserved from yeast to humans to plants so I thought it might warrant composition-specific terms here.

Name: LSM1-7 complex
Def: A protein complex essential for P-body formation and deadenylation-dependent decapping of mRNA thereby regulating mRNA decay and subsequent degradation by the 5' to 3' pathway.
xrefs:
GOC:bhm
IntAct:EBI-16419327
PMID:23221597, 23620288, 26764377
Relationships:
is_a GO:0043234 protein complex
part_of GO:0000932 P-body
capable_of_part_of GO:NEW positive regulation of GO:0000290 deadenylation-dependent decapping of nuclear-transcribed mRNA (https://github.com/geneontology/go-ontology/issues/14410)

Name: LSM2-8 complex
Def: A protein complex that is part of the U6 snRNP spliceosome machinery. Stabilises U6 small nuclear RNA to ensure the efficiency and accuracy of constitutive and alternative splicing of selected pre-mRNAs depending on the environmental conditions.
xrefs:
GOC:bhm
IntAct:EBI-16419597
PMID:23221597, 23620288, 28482101
Relationships:
is_a GO:0043234 protein complex
part_of GO:0005634 nucleus
part_of GO:0005688 U6 snRNP
[or maybe GO:0005688 U6 snRNP has_part LSM2-8 complex? I always find this relationship confusing!]
capable_of_part_of GO:0000398 mRNA splicing, via spliceosome
capable_of GO:0017070 U6 snRNA binding
Thanks,
Birgit

GO:0005688 U6 snRNP
is  probably equivalent to : LSM2-8 complex
it shouldn't be a parent of  Name: LSM1-7 complex-  because it is specifically the complex which binds to small nuclear RNA U6  and it is vey well known by this name.

yes they seem to be the dame compositionally.
please make LSM2-8 complex a synonym of  U6 snRNP instead

But:
Def: A ribonucleoprotein complex that contains small nuclear RNA U6. The LSM2-8 does not contain the RNA, acc to the papers we read.

it shouldn't be a parent of Name: LSM1-7 complex- because it is specifically the complex which binds to small nuclear RNA U6 and it is vey well known by this name.

I don't think I suggested that, the 2 complexes are different.

but doesn't it bind to the snRNA to exert its function?
https://en.wikipedia.org/wiki/U6_spliceosomal_RNA
@krchristie  will know more.

I see its already assigned to @krchristie

binding to snRNA doesn't necessarily mean it's part of it.

GO:0005688 U6 snRNP
is probably equivalent to : LSM2-8 complex

No, these are not equivalent. LSM2-8 complex is specifically the heptameric LSM ring, which is part_of the U6 snRNP.

Well, it's definitely accurate to say that the U6 snRNP always has_part the LSM2-8 ring. I'd have to check if we can say that the LSM2-8 ring is always part_of the U6 snRNP...

Seems if we are going to have the Lsm1-7 and Lsm2-8 complexes, we should also have the Sm protein complex. Perhaps we should also have a parent term called something like "Sm-like protein family complex" as a grouping term for all of them. There looks to be precedent in the ontology for "____ family complex" terms.
Tharun S. Roles of eukaryotic Lsm proteins in the regulation of mRNA function.
Int Rev Cell Mol Biol. 2009;272:149-89. doi: 10.1016/S1937-6448(08)01604-3.
Review. PMID: 19121818

The Sm proteins are the founding members of the Sm-like protein family. There are seven Sm proteins that are conserved in all eukaryotes. They are B/B', D1, D2, D3, E, F, and G. These proteins assemble into a ring-shaped heteroheptamer on the U snRNAs of the major (U1, U2, U4, and U5 snRNAs) and minor (U11, U12, and U4atac snRNAs) spliceosomes forming the core of the corresponding snRNPs (Lerner and Steitz, 1979, 1981; Luhrmann et al, 1990; Will et al, 1999). The Sm proteins are required for the stability of the snRNAs and biogenesis of the snRNPs and therefore are very important in pre-mRNA splicing (Will and Luhrmann, 2001). All eukaryotes have an Lsm protein that is homologous to each of the six Sm proteins D1, D2, D3, E, F, and G named Lsm2, Lsm3, Lsm4, Lsm5, Lsm6, and Lsm7, respectively, and two more Lsm proteins, Lsm1 and Lsm8 that are weakly related to the Sm-B protein (Achsel et al, 1999; Salgado-Garrido et al, 1999; Seraphin, 1995).

Regarding use of relationships with part_of or has_part, consider whether either of these statements is true, using Lsm2-8 complex as an example:

part_of
The Lsm2-8 complex is ALWAYS part_of the U6 snRNP complex
if ALWAYS true, then you can use the part_of relationship

has_part
The U6 snRNP complex ALWAYS has_part the Lsm2-8 complex
if ALWAYS true, then you can use the has_part relationship

Name: LSM1-7 complex
Def: A **heteroheptameric protein complex composed of Lsm1, Lsm2, Lsm3, Lsm4, Lsm5, Lsm6, and Lsm7, or orthologs thereof, that complex selectively binds to mRNPs targeted for decay. It is ** essential for deadenylation-dependent decapping of mRNA and P-body formation thereby regulating mRNA decay and subsequent degradation by the 5' to 3' pathway. [I think we need to include composition.]
xrefs:
GOC:bhm
IntAct:EBI-16419327
PMID:23221597, 23620288, 26764377 [not sure I'm keen to put three A. thaliana refs on defs for a highly conserved complex, would prefer to use the review I cited above.]
Relationships:
is_a GO:0043234 protein complex
part_of GO:0000932 P-body [I'm not sure we can say this complex is ALWAYS part_of the P-body. I kind of thought that it was involved in bringing mRNAs subject to decay TO the P-body, implying that it binds them elsewhere and helps to localize them there]
capable_of_part_of GO:NEW positive regulation of GO:0000290 deadenylation-dependent decapping of nuclear-transcribed mRNA (#14410)
Also note that we already have a term for the Lsm1-7-Pat 1 complex. It seems like most of the functions you ascribe to the Lsm1-7 complex are when it is bound to Pat1, so I wonder if it would be more appropriate to put the description of biological role in mRNA decay and P-body formation into that def, and put only the RNA binding part in the def of the Lsm1-7 complex term.
Name: LSM2-8 complex
Def: A protein complex that is part of the U6 snRNP spliceosome machinery. This is not all it does Stabilises U6 small nuclear RNA to ensure the efficiency and accuracy of constitutive and alternative splicing of selected pre-mRNAs depending on the environmental conditions.
Revised Def: A **heteroheptameric protein complex composed of Lsm2, Lsm3, Lsm4, Lsm5, Lsm6,  Lsm7, and Lsm8, or orthologs thereof, that complex selectively binds to the U6 snRNA and also to
xrefs:
GOC:bhm
IntAct:EBI-16419597
PMID:23221597, 23620288, 28482101 [again, not sure I'm keen to put three A. thaliana refs on defs for a highly conserved complex, would prefer to use the review I cited above.]
Relationships:
is_a GO:0043234 protein complex
part_of GO:0005634 nucleus
part_of GO:0005688 U6 snRNP
[or maybe GO:0005688 U6 snRNP has_part LSM2-8 complex? I always find this relationship confusing!]
capable_of_part_of GO:0000398 mRNA splicing, via spliceosome
capable_of GO:0017070 U6 snRNA binding
saving now, but will come back and revise this comment

Ah OK, so how does  LSM2-8 complex differ from   "GO:0005688 U6 snRNP"?
I only have Lsm2-8 annotated to this.

The Lsm2-8 complex is a heteroheptameric ring-shaped complex that does not contain the U6 snRNA.
It is also worth noting that the Lsm2-8 complex also binds to unspliced transcripts within the nucleus and is  involved in degradation, so I think it is entirely appropriate to have a term specifically for the Lsm2-8 ring/complex since it is present in multiple other complexes.

Thanks, Karen. I'm only work 2 days this week due to school holidays and they were jampacked! I'll get back to your questions above next week.

Useful quote in thinking of a definition for grouping term "Sm-like protein family complex"

An interesting feature of the Sm-like proteins is their ability to assemble into complexes of multiple combinations of varied functions and RNA binding specificities (Beggs, 2005; Wilusz and Wilusz, 2005).

Here's info on the Sm/Lsm ring present in the core U7 snRNP complex:
from: Marzluff WF, Koreski KP. Birth and Death of Histone mRNAs. Trends Genet. 2017
Oct;33(10):745-759. doi: 10.1016/j.tig.2017.07.014. Epub 2017 Aug 31. Review.  Like the spliceosomal snRNPs, U7 snRNP can exist as a core snRNP and a larger holo-snRNP which contains factors essential for processing. The core U7 snRNP consists of U7 snRNA bound to five Sm proteins, B, D3, E, F, and G, that are found in spliceosomal snRNAs, and to Lsm10 and Lsm11 which replace SmD1 and SmD2 [18,19]. Lsm11 is much larger than the other Sm proteins, 360 amino acids (aa) in mammals and 256 aa in Drosophila, and the N terminus of Lsm11 plays a crucial role in histone pre-mRNA processing [20].

The Tharun review also discusses some other Lsm family complexes:

Sm complex (SmB/B', SmD1, SmD2, SmD3, SmE, SmF, and SmG), binds to all the snRNAs EXCEPT U6
U8 snoRNA associated Lsm complex, possibly only six membered (Lsm8, Lsm2, Lsm3, Lsm4, Lsm6, Lsm7 [no known homolog of Lsm5]
snR5 associated Lsm complex, possibly only six membered ([no known homolog of Lsm 1 or Lsm8] but containing Lsm2, Lsm3, Lsm4, Lsm5, Lsm6, Lsm7

@bmeldal - No worries about not getting to this till next week. I've been putting my thoughts into various comments, both suggested revisions to the two terms you proposed, as well as others that might be merited, perhaps with a structure like I've put below. Defs of these should focus on RNA binding function, especially for those that are present in more than one complex. Many of the processes you mentioned in your initially proposed defs more likely belong in the defs of the larger complexes with additional components that actually have those roles, e.g. "Lsm1-7-Pat1 complex" and "U6 snRNP".
-- Lsm family ring complex
--- Lsm1-7 complex
--- Lsm2-8 complex
--- Sm complex
--- U8 snoRNA associated Lsm complex
--- snR5 associated Lsm complex

I think we should have a discussion about functional classifications of complexes.  I would suggest that we ANNOTATE complexes with functions, complete with evidence code and reference, rather than defining their functions in the ontology.  The GO terms could reflect complex membership.

Thanks you @krchristie, here my comments:
In principle I agree with everything so let's get this finalised:

Well, it's definitely accurate to say that the U6 snRNP always has_part the LSM2-8 ring. I'd have to check if we can say that the LSM2-8 ring is always part_of the U6 snRNP...

LSM2-8 is not always part of U6 snRNP:
PMID:22446625
Tang et al (2012). Telomerase RNA biogenesis involves sequential binding by Sm and Lsm complexes. Nature. 2012 Mar 25;484(7393):260-4. doi: 10.1038/nature10924.

"Here we demonstrate that the canonical Sm ring and the Lsm2–8 complex sequentially associate with fission yeast TER1. The Sm ring binds to the TER1 precursor, stimulates spliceosomal cleavage and promotes the hypermethylation of the 5′-cap by Tgs1. Sm proteins are then replaced by the Lsm2–8 complex, which promotes the association with the catalytic subunit and protects the mature 3′-end of TER1 from exonucleolytic degradation."
Lots of experimental evidence.

Seems if we are going to have the Lsm1-7 and Lsm2-8 complexes, we should also have the Sm protein complex. Perhaps we should also have a parent term called something like "Sm-like protein family complex" as a grouping term for all of them. There looks to be precedent in the ontology for "____ family complex" terms.

I think "Sm-like protein family complex" is a good idea as there are already a number of other Sm-X complexes in GO.

Tharun S. Roles of eukaryotic Lsm proteins in the regulation of mRNA function.
Int Rev Cell Mol Biol. 2009;272:149-89. doi: 10.1016/S1937-6448(08)01604-3.
Review. PMID: 19121818

I have no access to the fulltext for this one but I have found similar statements elsewhere.
Essentially, Sm-related complexes stablise snRNAs and provide the basis for bigger snRNP complexes. These occur in a manifold of combinations with fewer or more associated proteins and other complexes depending on cellular circumstances (see review PMID:27627834)
A case in point in PMID:28768202 (LSM1-7 and LSM2-8 both associate with Pat1B but are cytoplasmic- and nuclear-specific, resp.)
Also: Sm complexes don't assemble without the snRNA while LSM complexe assemble on their own. This may need to go in the def or a comment.
Term requests review:
Name: LSM1-7 complex
Def: Def: A protein complex essential for P-body formation and deadenylation-dependent decapping of mRNA thereby regulating mRNA decay and subsequent degradation by the 5' to 3' pathway.
Revised Def: A heteroheptameric, cytoplasmic protein complex composed of Lsm1, Lsm2, Lsm3, Lsm4, Lsm5, Lsm6, and Lsm7, or orthologs thereof, that shows a strong binding preference for oligoadenylated RNAs over polyadenylated RNAs. Binds associated protein, especially Pat1, in the P-body where it facilitates the deadenylation-dependent decapping of mRNA thereby regulating mRNA decay and subsequent degradation by the 5' to 3' pathway.

'I think we need to include composition and the specific function of the Lsm1-7 complex, which is to bind RNA, maybe less on the process. Note that we already have a term for the "Lsm1-7-Pat 1 complex". It seems like most of the functions you ascribe to the Lsm1-7 complex are what it does when it is bound to Pat1, so I wonder if it would be more appropriate to put the description of biological role in mRNA decay and P-body formation into that def, and put only the RNA binding part in the def of the Lsm1-7 complex term.'

Yes, looks like LSM1-7 always binds Pat1 in the P-body and then becomes functionally.
xrefs:
GOC:bhm, GOC:krc
IntAct:EBI-16419327
PMID:23221597, 23620288, 26764377

'not sure I'm keen to put three A. thaliana refs on defs for a highly conserved complex, would prefer to use the review I cited above.'

Yes, and add review PMID:27627834.
Relationships:
is_a GO:0043234 protein complex
part_of GO:0000932 P-body

'I'm not sure we can say this complex is ALWAYS part_of the P-body. I kind of thought that it was involved in bringing mRNAs subject to decay TO the P-body, implying that it binds them elsewhere and helps to localize them there'

Yes, you are right. So it's part_of cytoplasm.
capable_of_part_of GO:NEW positive regulation of GO:0000290 deadenylation-dependent decapping of nuclear-transcribed mRNA (#14410)

'seems this might be more appropriate to say for the existing term for the "Lsm1-7-Pat1 complex"'

and then make LSM1-7 complex part_of Lsm1-7-Pat1 complex? At least it's Lsm1-7-Pat1 complex has_part LSM1-7 complex.
Name: LSM2-8 complex
Def: A protein complex that is part of the U6 snRNP spliceosome machinery.
Revised Def: A heteroheptameric, nuclear protein complex composed of Lsm2, Lsm3, Lsm4, Lsm5, Lsm6, Lsm7, and Lsm8, or orthologs thereof, that selectively binds to snRNAs, in particular the U6 snRNA, and also to unspliced transcripts localized within the nucleus.
xrefs:
GOC:bhm, GOC:krc
IntAct:EBI-16419597
PMID:23221597, 23620288, 28482101

'again, not sure I'm keen to put three A. thaliana refs on defs for a highly conserved complex, would prefer to use the review I cited above.'

Yes add other review, too, see above.
Relationships:
is_a GO:0043234 protein complex
part_of GO:0005634 nucleus
GO:0005688 U6 snRNP has_part LSM2-8 complex
capable_of_part_of GO:0000398 mRNA splicing,

'via spliceosome seems like this belongs on the U6 snRNP complex'

Yes, I agree.
capable_of GO:0017070 U6 snRNA binding

The Tharun et al review uses the phrase "Sm-core of U6 snRNP" to refer to the Sm ring that binds to U7, with the composition listed in the first quote from the Marzluff & Koreski paper.

"core" is so contentious: is it just the Sm-like ring protein complex or the Sm-like snRNP complex? The snRNP complex associates with more proteins and complexes to make larger machineries!

-- Lsm family ring complex
--- Lsm1-7 complex
--- Lsm2-8 complex
--- Sm complex
--- U8 snoRNA associated Lsm complex
--- snR5 associated Lsm complex

I'd call it "Sm-like family complex" - with your suggested term as exact synonym?
Plus the existing X-Sm complexes in GO, e.g.
GO:0034715 pICln-Sm protein complex
GO:0034719 SMN-Sm protein complex
GO:0034731 Lsm-containing SMN-Sm protein complex
GO:0034730 SmD-containing SMN-Sm protein complex
GO:0032797 SMN complex
GO:0034718 SMN-Gemin2 complex
GO:0034731 Lsm-containing SMN-Sm protein complex
GO:1990726 Lsm1-7-Pat1 complex
No idea if I found all as I just did a text search.
Phuh, I hope I commented on everything that needed commenting on!

@thomaspd

I think we should have a discussion about functional classifications of complexes. I would suggest that we ANNOTATE complexes with functions, complete with evidence code and reference, rather than defining their functions in the ontology. The GO terms could reflect complex membership.

This is in contrast to what we've done over the last few years. Making new terms only based on composition was getting exponentially too big so we were trying to stick to grouping terms one level above composition and aimed for functional relationships.
We are also using the functional grouping terms to create structure in the ontology. The structure below 'protein complex' was really flat before we started our project.
I was advised to add capable_of and capable_of_part_of relationships so the reasoner could then bring existing terms into new grouping terms using the inference. Otherwise we had to find all relevant exisiting terms by hand. @dosumis was instrumental in facilitating this.
This should be discussed in the WG and also get a view from the users as to what they are using.

@bmeldal - It sounds like we're in agreement about the "Lsm1-7 complex" and "Lsm2-8 complex" terms.
However, regarding a grouping term, I was thinking of something specific for the Sm/Lsm rings, though that might not make sense if some/all of these rings are never "free" but only form on RNA, as you mentioned for the Sm rings that interact with most of the spliceosomal snRNAs (except U6 which binds Lsm2-8).
It seems that you are including complexes that include additional components. If we are going to include everything that has one of these Sm/Lsm ring (sub)complexes as a part, then we should  probably also include the terms for the snRNPs that contain U1, U6, etc.
So, we should decide whether either or both of these grouping terms is appropriate. Let me know what you think.

Well, the question which type of grouping therm is useful? We did something similar for the class of  GO:0070603 SWI/SNF superfamily-type complex recently:
https://github.com/geneontology/go-ontology/issues/14106
We could use one overarching term for "Sm-like protein family complex" that include the possibility of snRNAs as members and have a child of LSM-like ring complexes that don't include the snRNAs and add a comment that the complex binds the snRNAs after assembly.

So to confirm I am understanding @bmeldal, you would find it useful to have BOTH a term for:

"Sm-like protein family complex" for all complexes that contain any of these Sm or Lsm rings and also other non-Lsm/Sm components
"LSM-like type? ring complexes" - What would this term be for? Just the LSM complexes that assemble on their own without requiring the RNA to assemble onto?, so therefore NOT the Sm ring that binds to snRNAs other than U6? Also, seems odd to put the word "like" right after an acronym, LSM, that already includes the word "like". Perhaps "LSM-type" would be better.

We've discussed these complexes further:
The Sm/Lsm rings need associated proteins, such as Pat1 for Lsm1-7, to function but these differ dep on the specific ring and the RNA substrate. Therefore, the simple rings structure aren't really functional. Having said that, we are not planning on making all potential variants right now.
Lsm1-7:

seems Pat1 (which comes in different flavours!) is always required.

[ ] Keep existing GO:1990726 Lsm1-7-Pat1 complex and don't make new Lsm1-7 complex term.

We will add Pat1 (or its variants) to our complexes in the CP.

[ ] update def:
-Def: A conserved protein complex that plays an important role in coupling deadenylation and decapping in the 5'-to-3' mRNA decay pathway. An example is found in S. cerevisiae.
+Def, as discussed above for Lsm1-7: A conserved, heteroheptameric, cytoplasmic protein complex composed of Lsm1, Lsm2, Lsm3, Lsm4, Lsm5, Lsm6, Lsm7, and Pat1, or orthologs thereof, that shows a strong binding preference for oligoadenylated RNAs over polyadenylated RNAs. May bind further associated proteins.  Facilitates the deadenylation-dependent decapping of mRNA in the P-body thereby regulating mRNA decay and subsequent degradation by the 5' to 3' pathway.

xrefs:
GOC:bhm, GOC:krc
IntAct:EBI-16419327
PMID:23221597, 27627834, 28768202, 19121818
Relationships:
part_of cytoplasm.
capable_of_part_of GO:0000290 deadenylation-dependent decapping of nuclear-transcribed mRNA (replace current capable_of_part_of term?)
Lsm2-8:
I haven't identified a full list of associated proteins and it might assemble and function directly on the snRNA so I'm thinking here the 'simple' ring complex term would be needed

[ ]  create new term

Def: A heteroheptameric, nuclear protein complex composed of Lsm2, Lsm3, Lsm4, Lsm5, Lsm6, Lsm7, and Lsm8, or orthologs thereof, that selectively binds to snRNAs, in particular the U6 snRNA, and also to unspliced transcripts localized within the nucleus.
xrefs:
GOC:bhm, GOC:krc
IntAct:EBI-16419597
PMID:23221597, 27627834, 28768202, 19121818
Relationships:
is_a GO:0043234 protein complex
part_of GO:0005634 nucleus
capable_of_part_of GO:0000398 mRNA splicing,
capable_of GO:0017070 U6 snRNA binding

[ ] update GO:0005688 U6 snRNP has_part LSM2-8 complex

New grouping term?

If we go with the above we can't have a term for Lsm ring protein complexes as some also have other proteins (e.g. Pat1), so the only suitable grouping term would be for the "Sm-like protein family complex". Discuss!

@bmeldal - I like your suggestions for

just updating the definition for the existing term for the Lsm1-7-Pat1 complex
new term for the Lsm2-8 complex

I agree with your conclusion that the only suitable grouping term would be the broader one for "Sm-like protein family complex" that includes complexes that contain other proteins in addition to the Sm/Lsm rings. So my question for you is do you want this term. If you think it is useful, I'm happy to make it.

[ ] GO:NEW Sm-like protein family complex

Def: A protein complex involved in either spliceosomal snRNP assembly in the cytoplasm, pre-mRNA splicing in the nucleus and/or decapping and the 5' to 3' mRNA degradation pathway in the cytoplasm. May consist of a large assemblies of Sm proteins (Small nuclear ribonucleoproteins), SMN proteins (survival motor neuron proteins) and/or Lsm proteins (Sm-like proteins), associated factors like Gemin proteins and small-non-coding RNAs (snRNA), pre-mRNAs or mRNAs. May form supercomplexes of the SMN core complex with other Sm and associated proteins. Subcomplexes are often involved in more than one pathway and specificity is determined by their associated proteins/complexes.
xrefs:
GOC:bhm
PMID:27627834, 19121818
Relationships:
is_a GO:0032991 macromolecular complex

I would prefer a definition more focused on the composition, and mentioning the roles more generally, something like this:
Term: GO:NEW Sm-like protein family complex
Def: A protein complex containing members of the Like-Sm family of proteins, which includes both the Sm proteins and the Lsm proteins, and which generally form hexameric or heptameric ring structures which bind to RNA. While some of these ring complexes may form independently of RNA, many only form in association with their target RNA. In addition to Lsm-family proteins, many of these complexes contain additional protein members. Members of this family of complexes include the snRNPs which comprise the majority of the spliceosome. Others are involved in the 5' to 3' degradation pathways of mRNAs in the cytoplasm and of unspliced transcripts in the nucleus, as well as other diverse roles.
Def xrefs: GOC:bhm, GOC:krc, PMID:27627834, PMID:19121818
Relationships: is_a GO:0032991 macromolecular complex

I agree that the GO classes should focus on composition, not function.  As I'd suggested earlier, IntAct can annotate complexes with GO MF and BP terms, using evidence codes and references.  These annotations could then be used on the IntAct site to browse or search by function.  Birgit, would that work for you?

@thomaspd - While I am in favor of focusing on the composition as the definition of what is common in this family of complexes, I am not in favor of dropping function out of definitions entirely and leaving the functions to be defined only by annotation, especially not for specific complexes.
Not only is this inconsistent long defined GOC practice, I think it will be detrimental to consistency in annotation. For curators to find the right terms, definitions have to be sufficiently explanatory for us to identify which is the correct term.

Yes, it's fine to have reference to function in the text definition, just not in the logical definition.

I think whether or not it is OK to have it in the logical definition depends on the specificity of the complex. In this grouping term, there is clearly no logical definition that is appropriate.

I have

put in the new terms for Lsm2-8 and the Sm-like protein family complex grouping term
modified the definition of the existing Lsm1-7-Pat1 complex term
given the various existing terms that contain Sm or Lsm proteins is_a relationship to the new grouping term

@bmeldal - For future, it would be really helpful if you would list the IntAct database cross references separately from the definition references, since the IntAct references belong in a different place.
+id: GO:0120115
+name: Lsm2-8 complex
+namespace: cellular_component
+def: "A heteroheptameric, nuclear protein complex composed of Lsm2, Lsm3, Lsm4, Lsm5, Lsm6, Lsm7, and Lsm8, or orthologs thereof, that selectively binds to snRNAs, in particular the U6 snRNA, and also to unspliced transcripts localized within the nucleus." [GOC:bhm, GOC:krc, PMID:19121818, PMID:23221597, PMID:27627834, PMID:28768202]
+xref: IntAct:EBI-16419597
+is_a: GO:0120114 ! Sm-like protein family complex
+created_by: kchris
+creation_date: 2017-11-06T18:56:22Z

Thanks @krchristie
Ok, will separate the def xref and dbxrefs in future.
As for the def: did you drop the SMN proteins (survival motor neuron proteins) on purpose? To me they seemed like a third group within the Sm-like family of proteins.

Hi @bmeldal - Regarding the def, I wasn't aware that the SMN proteins were considered to be members of the Lsm family. Looking back through the Tharun review:

Tharun S. Roles of eukaryotic Lsm proteins in the regulation of mRNA function.
Int Rev Cell Mol Biol. 2009;272:149-89. doi: 10.1016/S1937-6448(08)01604-3.
Review.  PMID:19121818

to see if I missed anything, I don't see any mention of the SMN proteins as a member of the Lsm family. The only occurance of "SMN" at all in the Tharun review is this citation:

R.S. Pillai, M. Grimmler, G. Meister, C.L. Will, R. Luhrmann, U. Fischer, D. Schumperli
Unique Sm core structure of U7 snRNPs: Assembly by a specialized SMN complex and the role of a new component, Lsm11, in histone RNA processing
Genes Dev., 17 (2003), pp. 2321-2333

The Pillai et al. paper talks about the SMN complex as an assembly factor for Sm proteins, but does not refer to SMN as an Sm protein itself.
In addition, this review about SMN:

Raimer AC, Gray KM, Matera AG. SMN - A chaperone for nuclear RNP social
occasions? RNA Biol. 2017 Jun 3;14(6):701-711. doi:
10.1080/15476286.2016.1236168. Epub 2016 Sep 20. PubMed PMID: 27648855

also only talks about SMN as a chaperone or assembly factor for Sm proteins, but not as an Sm protein itself.
So, while I agree that the SMN complex terms belong as types of the 'Sm-like protein family complex' term, I do not see any indication that SMN is itself a member of the Lsm family.
Do you have papers indicating that SMN is a member of the Lsm family?

Sorry, I went by the rather short GO term defs and somehow assumed they were all part of the same family. I just looked at SMN directly and you are right, they are a chaperones/assembly proteins and the SMN complex is responsible for the assembly of the Sm/Lsm rings. The SMN complex itself shouldn't even be part of this complex group.
Sorry, @krchristie I didn't read up on the SMN separately.

@bmeldal - Thanks for checking on this. To be clear before I fix this, I'm thinking it's OK for the SMN-Sm protein complex to stay an is_a child of the grouping term. Here are the current child terms of the grouping term:
NOT a is_a 'Sm-like protein family complex'

GO:0032797 SMN complex
GO:0034718 SMN-Gemin2 complex

?? Are these is_a 'Sm-like protein family complex??' - they do contain Lsm-family proteins

GO:0034719 SMN-Sm protein complex
and two child terms:

GO:0034731 Lsm-containing SMN-Sm protein complex

GO:0034730 SmD-containing SMN-Sm protein complex

Definitely is_a 'Sm-like protein family complex'

GO:1990726 Lsm1-7-Pat1 complex
GO:0034715 pICln-Sm protein complex
Also, need to add this parentage for all the snRNPs that become part of the spliceosome, except for U6 snRNP which I've already done

Hi @krchristie yes, I think you are right with the assessment above, plus of course the new Lsm2-8 term but that is already correctly assigned.

Also, need to add this parentage [Sm-like protein family complex] for all the snRNPs that become part of the spliceosome.

Wow, I hadn't thought of this, but yes, there are more children!

I have removed this relationship:
GO:0034719 SMN-Sm protein complex
is_a 'Sm-like protein family complex'
and added this one:
'small nuclear ribonucleoprotein complex'
is_a 'Sm-like protein family complex'
This term is the parent term of all the splicesosomal snRNPs and also the U7 snRNP, which is also a Sm-family complex according to the Tharun review:

Another Sm-like protein complex conserved in higher eukaryotes and involved in a specialized mechanism through which metazoan histone mRNA 3'-ends are generated is the unique Sm-core
complex of the U7 snRNP, which is made up of five Sm proteins and two Lsm proteins.

I think this covers everything we've discussed, but let me know if you spot anything we missed, @bmeldal

Thank you, Karen, your efforts are much appreciated! Complexes will be in the next release (just missed the one from yesterday).


manage-order-c21-changes [DO NOT MERGE]

Before creating a pull request make sure that:

[ ] commit messages are meaningful and follow good commit message guidelines
[ ] README and other documentation has been updated / added (if needed)
[ ] tests have been updated / new tests has been added (if needed)
[ ] the enable_keep_helm label has been added, if the helm release should be persisted after a successful build

Please remove this line and everything above and fill the following sections:
JIRA link (if applicable)
https://tools.hmcts.net/jira/browse/PRL-4814
Change description
Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[ ] No

No longer needed already merged, hence closing it


Add Github username and profile link to CONTRIBUTORS.md

@RyanWong25 Hi Ryan, thanks for your PR!
Don't forget to follow me and this organization on GitHub to staytuned to the latest updates!
Thanks :)


HIVE-25149: Support parallel load for Fast HT implementations

What changes were proposed in this pull request?

Introducing new HiveConf controlling the number of threads used in parallel HT load for MJ
Extending VectorMapJoinFastHashTableLoader to support parallel HT loading

Why are the changes needed?

Support parallel HT loading for Fast HT implementations
Does this PR introduce any user-facing change?

NO
How was this patch tested?

Existing tests cover functionality and correctness.
Performance tests will be added as part of HIVE-25146

cc @ramesh0201

+1. Looks good to me.


Graph Explorer Unable to Connect

Following the instruction on the Graph Explorer GitHub page, I'm presented with this error when trying to connect to my TinkerPop Graph DB

I know I can connect to my JanusGraph/TinkerPop Gremlin Server running at http://localhost:8182 as I can remotely connect to it through the gremlin console and perform queries.
Is there a step I might have missed or is there a configuration issue between the latest graph-explorer and invana-engine?

Hey @Donny3000 , please check here https://github.com/invanalabs/graph-explorer/issues/73 for new instructions. The project is in the middle of design transition phase. Apologies for the poor documentation. Will fix it soon.

Thanks, @rrmerugu for your prompt reply and incredible work on this great project! I took look at #73 and it all made sense to me. But, I seem to be getting the same error message/pop-up. To be clear these are the steps I'm performing:

docker run -p 8000:8000 -d -e GREMLIN_SERVER_URL=ws://localhost:8182/gremlin --name invana-engine invanalabs/invana-engine
docker run -p 8888:8888 -d --name graph-explorer invanalabs/graph-explorer
Then, point my browser to http://localhost:8888

Is this the correct sequence/steps to perform?

I see, in this case, the first thing you shall debug is whether Invana Engine GraphQL is configured correctly or not.   Do a simple query by going to http://localhost:8000/graphql. If that failing with error, there is some issue with configuring invana-engine. All the above steps assume you are running a janusgraph instance using docker run -it -p 8182:8182 janusgraph/janusgraph or other methods, and janusgraph is available at 8182 port

Thanks again for the assistance. I'm am indeed running a JanusGraph instance with the docker command you showed. I can confirm that the JanusGraph instance is up and running correctly because I can start a gremlin console client to remotely connect to the server and issue the "g.V().limit(2).toList()" command and get results back. However, if I issue the command in your screenshot using the GraphQL interface, this is the error I get

Do you have any additional insight as to why this would be happening? Thanks again for your assistance!

Looks like this is a docker issue, can you check the logs of the janusgraph and invana-engine container using docker logs <container ID>. That might lead you to some hints.

It seems invana_engine us running into connection issues when I issue the g.V().limit(5).toList() GraphQL query. It's still not clear as to why this "connection refused" error is happening since I can indeed connect to my JanusGraph server instance remotely through the gremlin client console.

So, I was able to get it working by using the following docker run command to start the invana-engine
docker run -p 8000:8000 -d -h invana_engine -e GREMLIN_SERVER_URL=ws://<IP_ADDRESS>:8182/gremlin --name invana-engine invanalabs/invana-engine
where the IP address of <IP_ADDRESS> IP address assigned by the docker network to the JanusGraph server. Now, although I do have it working, this seems very suboptimal.
What is the correct way to solve this problem without having to resort to using the container's internal docker IP address for the GREMLIN_SERVER_URL?

Glad it's working. I generally do something like this docker run --rm -d --network host --name my_nginx nginx , where I map the ports of the docker to host network directly. So I dont have to access the docker container over the docker network IP(<IP_ADDRESS>:8182), instead I can connect to localhost:8182 directly.
Hope this helps, https://docs.docker.com/network/network-tutorial-host/

Thanks, @rrmerugu! I'll read through that documentation to see if that will help me.


CkEditor Link Popup crashes in French

Description of bug
With a French locale, clicking the link icon in the toolbar

does not work as expected. The screen gets a white overlay and there's an error in the console regarding DNN Pages.
This is not the case in the other languages I tested:
en, nl & it
Steps to reproduce
List the precise steps to reproduce the bug:

Make sure you are on a "French" page.
Place the HTML editor,
Select a text and Click the Link icon
See error :-)

Current behavior
No link popup
Expected behavior
The link popup should work..
Error information
Uncaught TypeError: editor.lang.dnnpages is undefined
Affected version

[ ] 10.00.00 alpha build
[x] 09.10.02 release candidate
[x] 09.10.01 latest supported release

ok already logged and fixed..
https://github.com/dnnsoftware/Dnn.Platform/issues/4801
While logging this I was surprised @valadas did not noticed this, but is looks like he did :-)
Must be something with the upgrade that I see this @ the clients site.

Yeah, it was fixed in 9.10.2

Right, I think I missed that because the issue was closed on aug 30 :-)

I just compared the two zips :-)


Fix broken link for examples

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


pynetbox greater than 7.0.0 fails to insert complex custom_fields

pynetbox version
v7.2.0
NetBox version
v3.5.9
Python version
3.11
Steps to Reproduce
Hello,
in our setup we introduced a complex custom_field in each device that is of the following form, in python notation:
{
  "metadata_config" : [
    {"a" : "b", "c": "d"},
    {"a":  "e", "c": "k"}
  ]
}

I found out that any version over 7.0.0, essentially anything containing the #518 fix that rewrote flatten_custom, is breaking updating this custom_field via pynetbox. update() returns fine, but all dicts are set as None into netbox.
The issue seems to lie with a cornercase in flatten_custom.
Below you can find a way to test this with 7.0.0, 7.2.0 and a candidate fix that I'll open a PR with.
Thanks!
to run it, just execute ./run.sh, after writing adding these two files into some directory.
filename: run.sh
#!/bin/bash

pip install pytest

echo "v7.0.0 works"
pip install pynetbox==7.0.0
pytest

echo "v7.2.0 fails"
pip install pynetbox==7.2.0
pytest

echo "candidate fix works"
pip install git+https://github.com/costasd/pynetbox@6fe8494f74cbe7779c98cb949af2ee8d95e74d0e
pytest

filename: test_custom_fields.py
import pynetbox

class TestComplexCustomFields:
    def test_delete_custom_field(self):
        nb = pynetbox.api('http://localhost:8080', token='0123456789abcdef0123456789abcdef01234567')
        field = nb.extras.custom_fields.get(name="metadata_config")
        nb.extras.custom_fields.delete([field])

    def test_create_custom_field(self):
        nb = pynetbox.api('http://localhost:8080', token='0123456789abcdef0123456789abcdef01234567')
        field = nb.extras.custom_fields.create(
            name="metadata_config",
            content_types=["dcim.device"],
            type="json",
            description="Format: [{\"key1\": \"value1\", \"key2\": \"value2\"}]",
            data_type="object",
            default=[])
        assert field

    def test_complex_structures_to_custom_field(self):
        nb = pynetbox.api('http://localhost:8080', token='0123456789abcdef0123456789abcdef01234567')

        device = nb.dcim.devices.get(1)
        dev_id = device.id

        # A list containing a dict
        data = [
            [],
            [{"a": "b"}],
            [{"a": "b", "c": "d"}],
            [{"a": "b", "c": "d"}, {"a": "e", "c": "k"}],
        ]
        for d in data:
            assert device.update({"custom_fields": {"metadata_config": d}})

            device = nb.dcim.devices.get(id=dev_id)
            assert device.custom_fields['metadata_config'] == d

Expected Behavior
Netbox should contain the structure we push.
ie. for:
{
  "metadata_config" : [
    {"a" : "b", "c": "d"},
    {"a":  "e", "c": "k"}
  ]
}

we should get the same back, either by browsing netbox or querying the relevant custom_field through pynetbox
Observed Behavior
Instead, the following gets stored into netbox and gets retrieved:
{
  "metadata_config" : [
    None,
    None
  ]
}

@abhi1693 @arthanson Any chance of a merge or a comment here?  I just lost a few hours of debugging eventually figuring out that pynetbox simply wasn't doing what I was looking for.  It's specifically impactful for custom fields of type JSON.

Closed as this has been merged.


Add importer helpers to support OS-specific importers

Usage
Importer: importer.TryAll(
	importer.TryAllEnvVars(fieldname.Token, "EXAMPLE_ACCESS_TOKEN"),
	importer.MacOnly(
		TryExampleConfigFile("~/Library/Application Support/example/config.yml"),
	),
	importer.LinuxOnly(
		TryExampleConfigFile("$XDG_CONFIG_HOME/example/config.yml"),
	),
),

I am not sure if this is the right way to handle this.
It would make more sense to prefix the path with https://pkg.go.dev/os#UserConfigDir and be then platform independent. There is already an abstraction in Go we should not built it by own 😄
Also you can go into weird edge cases like, you have Linux but not defined env $XDG_CONFIG_HOME for some reason

The pattern used in Go's UserConfigDir is not widely spread enough to base this feature around it, unfortunately. There's many tools that use ~/.config or XDG_CONFIG_HOME on Mac as well, instead of Application Support. That said, we can always still introduce a platform-independent shorthand later.
Also, the code snippet in this PR description was just one example use case for the MacOnly and LinuxOnly helpers introduced in this PR.

we can always still introduce a platform-independent shorthand later.

++
I will be merging this since it's treated as blocker for DigitalOcean config file importer on macOS


Unknown image format error when adding PNG ArrayBuffer

I'm using the prebuilt version of PDFKit in the browser, and when attempting to add a PNG I get the following error:
Uncaught Error: Unknown image
    PDFImage.open                                               format.util.js:546
    module.exports.image                                        deflate.js:773
    img.onload                                                  PDFrenderer.js:195

I'm converting my images to PNG on the server (to crop them into circles) and the mime type of the images returned by the server is 'image/png'.  I'm unsure whether the method I'm using to convert the PNG into an ArrayBuffer is incorrect.
Here is the code I use to fetch the PNG and convert it  into an ArrayBuffer:
var img = new Image, ctxData;

img.onError = function () {
    throw new Error('Cannot load image: "' + url + '"');
}

img.onload = function () {
    var canvas = document.createElement('canvas');
    document.body.appendChild(canvas);
    canvas.width = img.width;
    canvas.height = img.height;

    var ctx = canvas.getContext('2d');
    ctx.drawImage(img, 0, 0);
    ctxData = canvas.toDataURL('image/png').slice('data:image/png;base64,'.length);
    ctxData = atob(ctxData);
    document.body.removeChild(canvas);

    var buffer = [];

    for (var i = 0, l = ctxData.length; i < l; i++) {
        buffer.push(ctxData.charCodeAt(i));
        buffer._isBuffer = true;
        buffer.readUInt16BE = function (offset, noAssert) {
            var len = this.length;
            if (offset >= len) return;

            var val = this[offset] << 8;
            if (offset + 1 < len)
                val |= this[offset + 1];
            return val;
        }
    }
    pdf.image(buffer);
}

img.src = url;

This works fine for JPEGs when this line is changed from
ctxData = canvas.toDataURL('image/png').slice('data:image/png;base64,'.length);

to
ctxData = canvas.toDataURL('image/jpeg').slice('data:image/jpeg;base64,'.length);

however I need to be able to pass in PNGs so that I can place I can place round images over any background.
I've also tried passing in the full url (e.g. 'http://mysite.dev/userimages/1234/roundavatar.png'), however I'm then presented with the following error:
Uncaught TypeError: undefined is not a function
    PDFImage.open                 util.js:535
    module.exports.image          deflate.js:773

Hope this is just something I've been doing wrong, and thanks for the great library!

You don't need to draw it to a canvas at all. Just do an AJAX request to load the image, and pass the result as an arraybuffer into pdfkit. Something like this:
var xhr = new XMLHttpRequest;
xhr.onload = function() {
  // set the font using the arraybuffer returned from the xhr
  doc.image(xhr.response);
};

xhr.responseType = 'arraybuffer';
xhr.open('GET', url, true);
xhr.send();

Hello, thanks for your response!

Hello, thanks for your quick response!
I've tried your method, but sadly it has not worked for me. The script fails below the comment in the snippet below, complaining that undefined is not a function:
PDFImage.open = function(src, label) {
    var data, match;
    if (Buffer.isBuffer(src)) {
      data = src;
    } else {
      if (match = /^data:.+;base64,(.*)$/.exec(src)) {
        data = new Buffer(match[1], 'base64');
      } else {
        /* ======================================================
           Script fails here
           ====================================================== */
        data = fs.readFileSync(src);
        if (!data) {
          return;
        }
      }
    }
    if (data[0] === 0xff && data[1] === 0xd8) {
      return new JPEG(data, label);
    } else if (data[0] === 0x89 && data.toString('ascii', 1, 4) === 'PNG') {
      return new PNG(data, label);
    } else {
      throw new Error('Unknown image format.');
    }
  };

However I have managed to get it working, but by loading the image using the method I originally used (base64 encoding using the canvas). I noticed that even though when passing through a PNG
data[0] === 0x89

would return true, however
data.toString('ascii', 1, 4) === 'PNG'

would always return false. By commenting out the second condition it now gets to
return new PNG(data, label);

fine and renders in the PNG.
This probably isn't the best practice, as I'm sure that second condition was put in for a reason, but it works as I need, and I thought I would log this here incase someone else encounters a similar problem and needs a quick fudge to get it working.
Again thanks for your help, and thanks for the great library!

I also ran into the problem of the 'readFileSync is not a function' error, turns out the pdfkit.js needs a little tweaking because an array buffer is not directly a compatible Buffer object. This is not so hard to deal with though.
So get an image like suggested:
var myImage;
var oReq = new XMLHttpRequest();
oReq.open("GET", "myimage.jpg", true);
oReq.responseType = "arraybuffer";
oReq.onload = function(oEvent)
{
	myImage = oReq.response; //image as an arraybuffer
	makePDF();
};
oReq.send(null)

//then use myImage like normal:
doc.image(myImage);

And then as I said you need to tweak the pdfkit.js file. At about line 2888:
PDFImage.open = function(src, label) {
	var data, match;
	if (Buffer.isBuffer(src)) {
		data = src;
	} else {
		//START NEW
		if (src instanceof ArrayBuffer) {
			data = new Buffer(new Uint8Array(src), 'object');
		} else
		//END NEW
		if (match = /^data:.+;base64,(.*)$/.exec(src)) {
			data = new Buffer(match[1], 'base64');
		} else {
			data = fs.readFileSync(src);
			if (!data) {
				return;
			}
		}
	}

Make sure you also include blob-stream.js. I added an extra option after //START NEW that takes care of array buffers that come from XMLHttpRequests.
I don't know if this is the best solution, but it works. I didn't test all browsers though.

@Janneman84 I think this workaround causes an issue when trying to use multiple images, due to how it is pulling from the ImageRegistry in image() when isBuffer returns false.

@Janneman84 I used your workaround to be able to embed  PNGs on the browser. I'm not 100% sure what I am about to say is related to this, but I couldn't find a ticket about it so I will go ahead and mention. If I try to embed multiple images with doc.image, it always uses the first one. I think this has to do with the way it pulls from imageRegistry when isBuffer returns false.

I tried adding a couple of different images on one page with my fix. Jpg, png's and svg, big and small. It all works fine no problem at all. I wonder what the problem is that you're experiencing.
By the way you can also go the base64 way by setting a DataURL as the image.src instead of a filename. This works out of the box without my fix. You can convert images to DataURL's on many websites. If you do that, do you still have the same problem?
Also are you sure all images are actually loaded before starting to draw? I did something like this:
var img1done = false;
var img2done = false;
var img3done = false;

var img1 = new Image();
img1.onload = function() {
    img1done = true;
    if(img2done && img3done)
        draw(doc,img1,img2,img3);
}
img1.src = 'images/image1.png';

var img2 = new Image();
img2.onload = function() {
    img2done = true;
    if(img1done && img3done)
        draw(doc,img1,img2,img3) ;
}
img2.src = 'images/image2.png';

var img3 = new Image();
img3.onload = function() {
    img3done = true;
    if(img1done && img2done)
        draw(doc,img1,img2,img3);
}
img3.src = 'images/image3.png'

@Janneman84 I'm using promises, so once they resolve I call doc.image using the response.data. Without the second fix I mentioned, the first image added with doc.image is used for all calls to doc.image. I'll see if I can reproduce it and post the code when I get off work.
Here is how I am using the response data, which is an ArrayBuffer:
doc.image(response.data, 508, 231, {
    width: 230,
    height: 150
});

@devongovett Do you have any thoughts regarding opening up additional options for passing in image data on the browser, such as adding the clause for an ArrayBuffer like @Janneman84 did in his tweak?

Would love to see this functionality make it in. Using @mattblang 's bower fork at the moment.

There is a simple workaround when using browserify.
require('buffer');
var arraybuffer = 'image from server';
doc.image(new Buffer(arraybuffer));

It worked fine for me.

+1 for @simonbernard solution. This works like a charm:
require('buffer')

var xhr = new XMLHttpRequest
xhr.responseType = 'arraybuffer'
xhr.onload = function() {
  doc.image(new Buffer(xhr.response))
}
xhr.open('GET', 'images/sample.png', true)
xhr.send()

Many thanks for the xhr+Buffer solution.
@blvz, there's a () missing after   new XMLHttpRequest in your third line of code.

@sruf Nope. It's correct. ;)

Sorry, I must have corrected a mistake at the same time and wrongly assumed correlation == causation... Thanks again for the code!

quick fix for those who don't want to add module loading to a project in order to use Buffer: consider converting images to base64 and adding them to doc.image() as simple variables.


Seek for suggestions about Business/Application Data

Hi,
In this artical, it suggests not to store the business/application data as script: https://dzone.com/articles/continuous-integration-and-continuous-delivery-for-2.
Is there any best practice to handle Business/Application Data with flyway? For example, there's a table named person with columns:
id=>int, primary key
name=>string.
there's one record (1, 'jack') and we want to update the name to be 'Jack'. the sql script'd be like this
update public.person set name='Jack' where id=1. but it might have several environments and the id'd be different between them. what's the right way to handle this kind of operations to bussiness/appliction data?
Thanks,
Jack

This scenario is ultimately outside the scope of Flyway
Flyway is about your schema and can also be used with static data. Dynamic data doesn't fit into the current process since your migrations aren't always going to be reproducible
If this data is static, you can fix this with a new migration. If not, then it's outside the scope of Flyway
I'll be closing this for now, but at the same time, if you have any thoughts or suggestions about how Flyway could deal with these scenarios we'd love to hear it!


Exception: Datafile is not a LiteDB database.

I think i found one of the reasons for the error: LiteDB.LiteException: Datafile is not a LiteDB database.
As i saw on other issues referecing this problem, is the database file that become kinda corrupt, but only when the system tryes to read, on the studio everything goes fine.
Im implementing the LiteDB in a very small project (.net core 2.2 console app as a windows service) and im using it just to keep track of the e-mails that im sending.
I faced this exception a bunch of times, then delete the db file and start again, but i noticed now, that i only have this problem when i read the file on the LiteDB Studio.
If i create a new file, everything goes fine until i open the database on the studio(dont tried on the shell), after i open the database on the studio i get this problem. Maybe the error is caused by it.
I dont have the time yet to investigate the code by my own, but im trying to help.
Cheers

Hello @MestreDosMagros (kkkk, nick muito engraçado!!)
I will try to reproduce this steps to check this. v5 use lock FileOpen concepts to not allow 2 process open same database at same time (only if both are using Mode=Shared). Are you using Mode=Embeded?

kkkk Valeu!
I'm using shared mode, but the problem exists even when i got just my console app running after i open the db file externally.
For example, my service created a database an run ok, then i make a copy of that db file, and open one copy externally on the studio, with that file i get the error running the app, when i swap the file and use the one i dont opened externally it runs fine.

I'm having the same issue. I am using the latest 4.x version of LiteDB in my project, and the latest LiteDB Studio exe from github.

Same here

Same here, exact same.

+1

We get it without even using studio. In very rare cases Store seems to fail, and after few failures to insert data, it starts to receive the error.

+1

+1 Opening db file with LiteDB Studio, app that created data file cannot open it again even when disconnected.

+1

+1

+1

+1


Competitive programming

Description 📜
In the Competitive Programming Domain, Kadane's Algorithm is add in the Array section with detailed explanation documentation and python program.
Fixes # 2206

Type of change 📝

[ ] Au#dio (Should be in mp3 format Includes speech clarity, Concise ,Low distortion)
[ ] Vi#deo (Animations, screen-recordings, presentations and regular explanatory films are all possibilties etc)
[ ] Documentation (Content Creation in the form of codes or tutorials)
[ ] Other (If you choose other, Please mention changes below)

Domain of Contribution 📊

[ ] Android Dev #(Flutter)
[ ] Android Dev #(Java)
[ ] Android Dev #(Kotlin)
[ ] Backend Dev #(Java)
[ ] Backend Dev #(.NET)
[ ] Backend Dev #(PHP)
[ ] Backend Dev #(Python)
[ ] C/#CPP
[ ] Competitive Programming
[ ] Cyber #Security
[ ] DS#A
[ ] Data#base
[ ] Datascience with #Python
[ ] Datascience with #R
[ ] Frontend Dev #HTML/CSS/JS
[ ] Frontend Dev #React/Angular/Vue
[ ] Go#lang
[ ] Interview #Prep
[ ] Java_#Domain
[ ] Java#Script
[ ] ME#RN
[ ] Machine #Learning
[ ] Open #Source
[ ] Python_#Domain
[ ] Ru#st
[ ] Statis#tics
[ ] UI/#UX

Checklist ✅

[X] I follow Contributing Guidelines & Code of conduct of this project.
[X] I have performed a self-review of my own code or work.
[X] I have commented on my code, particularly in hard-to-understand areas.
[X] My changes generates no new warnings.
[X] I'm a GWOC'21 contributor

Screenshots / Gif (Optional) 📸

Please don't pull a request from your forked repo's Competitive_Programming branch to girlscript:main. It should be Sin-Sumit:Competitive_Programming -> girlscript:Competitive_Programming.

Thank You @atarax665 really good point.