Support of firebase database!

Type

[ ] Bug Fix
[ ] Functions: <Function Name>
[ ] Callbacks: <Callback Name>
[ ] Handlers: <Handler Type>
[X] Others: \ Support of firebase database using aoi.fb added!

Want a credit? Discord tag or other social media link: Luís Fernando#1581

Your PR changes are just allocating that aoi.js "supports" it, but we don't if the package does or not.
Since this package isn't by us, we can't fully state we support it.
It'll just be consider as a third party which is supported but not by default.
You can still use the package.

Your PR changes are just allocating that aoi.js "supports" it, but we don't if the package does or not. Since this package isn't by us, we can't fully state we support it. It'll just be consider as a third party which is supported but not by default. You can still use the package.

in the commit comment you had said it could still be used, however, it gives error when customizing the db and using aoi.fb

aoi.js main structure database is JSON, due to native support of other DB’s it won't support all unless that DB has the "right" methods for aoijs or aoijs adds those methods, I don’t use Firebase nor know much of it, but aoijs uses “tables” and other methods.
Hence that error is popping up if you read the src.
It'll actually have to be coded in aoijs to accept the other methods that DB uses.

aoi.js main structure database is JSON, due to native support of other DB’s it won't support all unless that DB has the "right" methods for aoijs or aoijs adds those methods, I don’t use Firebase nor know much of it, but aoijs uses “tables” and other methods.
Hence that error is popping up if you read the src.
It'll actually have to be coded in aoijs to accept the other methods that DB uses.

Good night, I've done tests and it's working normally with firebase, this was the error returned before the edits to the aoijs code were made!
We have some users, who are already using the modified source, because they needed to use the firebase database.

I'll explain in simple term.
Please read the edited files, ../Database.js
Understand aoijs using a class extend of CustomDB,
This can already be supported by using database methods.
What you're editing is to make accepted by default which shouldn't be.
Here's an example how to use other DB's.
database : {
type : "database type",
db : require('package here'),
tables : ['main']
}


Show VM migrations on a bar chart

Key points:

use stack chart to preserve space
use resize observer to adjust the chart size dynamically

TODO:

confirm stack chart as preferred option
create toggle to switch between supported time frames (1 month or 24h)
create custom hook to get data points for the chart - the points needs to be sorted, x part need to be human readable
fix problems with overview page layout - double scrollbar is visible after the migration card got bigger

confirm stack chart as preferred option

I like side be side (https://www.patternfly.org/charts/bar-chart#basic-with-right-aligned-legend)
@sgratch @ahadas can you way your opinions ?
p.s.
we can always change the chart type as needed ( other type of chart ? https://www.patternfly.org/charts/scatter-chart#basic )

fix problems with overview page layout

we can make it a separate card if it helps ...

note:
the chart view is a new view, it does not replace the table view, it's in addition to the table view.

confirm stack chart as preferred option

I like side be side (https://www.patternfly.org/charts/bar-chart#basic-with-right-aligned-legend)
@sgratch @ahadas can you way your opinions ?

The problem is limited width. The card needs to be narrow to fit into the layout. With full 30 data points and a legend this will be hard to achieve. Actually it would be better to reverse the chart - the page layout works better with data growing vertically (like tables)

in the 7-day view we should not have a problem it's 7 data points
in 24 hours view - we have 24 items, but we can do buckets of 6 hours, and then it's only 4 data items ...

create toggle to switch between supported time frames (1 month or 24h)

the issue is 7 days and 24 hours not 1 month and 24 hours

Hey guys, don't forget that the current patternfly.org site is running PF5 components now!  I believe that @patternfly/react-charts@^7 is tied to PF5.  It think it is still all victory charts underneath, but the PF styles around things are for sure going to be different.
A quick look at console on main show it is still running PF4 (with "@patternfly/react-charts": "6.94.19",) -- https://github.com/openshift/console/blob/master/frontend/package.json
Probably a good thing to be mindful of the release selected on the patternfly.org page!

Changes in the recent force push:

removed resize observer - the legend and number of data points will not depend on the size of the container - Victory charts follow responsive design so the charts scale automatically
moved charts to a new card - not sure how to name it - for now it's Last Migrations
optimized layout for 2 modes: last 7 days (bucket size: 1 day) and last 24h (bucket size: 4h )
add toggle group to switch between supported modes - since we have only 2 of them a dropdown seemed less user friendly to me

TODO:

data hook

24h view

7 day view

Nice :+1:
a. in the issue it say use kebab for the dropdown to choose time span, please use kebab
b. for now the chart card should be below the table chart (https://github.com/kubev2v/forklift-console-plugin/pull/702#issuecomment-1682489582)
c. please use vertical chart with 3 separate bars for started/failed/finished (https://github.com/kubev2v/forklift-console-plugin/pull/702#issuecomment-1682489582)

Changes in the recent force push:

improve resizing of the Overview page by introducing break points:
2.  2xl (the chart and settings are in the second column),
3.  xl (only settings are in the second column)
4. lg (only one column for all cards)
data hook - used standard useK8sWatchResource combined with transformation  logic (filtering and formatting depends on the period chosen by the user)

@yaacov

a. in the issue it say use kebab for the dropdown to choose time span, please use kebab

Done. However the kebab toggle seems to small here - see the screenshots below.

b. for now the chart card should be below the table chart (https://github.com/kubev2v/forklift-console-plugin/pull/702#issuecomment-1682489582)

Done.

c. please use vertical chart with 3 separate bars for started/failed/finished (https://github.com/kubev2v/forklift-console-plugin/pull/702#issuecomment-1682489582)

Done. Looks better if  there are many migrations on the cluster(i.e. generated mock data on the the recording below) but with small data sets it looks empty (i.e. mock data on screenshots below)

Screencast from 08-25-2023 06:51:19 PM.webm

Notes:
a. Need to always show the full time frame, for example, 7 days, need to show 7 days, days with 0 may show 0 but the need to be in the graph,
for example in this image we see only 3 days, we should see all 7:

b. time format should be a "usual" time format, it should not be a number ( again issue a. need all the time-slots feeled)
for example we see 06, we should see 06:00-07:00 for the timeslot between 6 and 7:

c. blue as the color for "failed" , hmm, I think red is more appropriate, in the table we use blue for started, green for finish successfully, and red for failed, we should use the same color scheme

Note III:
the order of the columns should be like in the table started | finished | failed
in the images, started is the last column instead of first

@yaacov

on my screen the spacings are not the same, the spacing between all cards should be the same:

Most likely we need another break point for the biggest screens - however it will be difficult to implement without testing on such screen.

Switched to interval based labels with hour granularity i.e. 14 - 18 which translates to 14:00 - 18:59 with minute granularity. See Luxon's helpers endOf and startOf

please use time format of HH:MM as suggested above,
so the labels can look like 14:00-16:00 or 14:00-15:59 but not "17" and "13-17" like in the image you attached

finished  !=  finished successfully - currently we count all migrations with status.completed regardless of the cause

if you look at the table you will see it say "successful" so we need to show the migrations that where finished successfully.
if you think we currently show the wrong value, it should be tracked in an issue and fixed.

PF4 recommends the same color family for the nested charts - https://www.patternfly.org/charts/colors-for-charts#chart-with-nested-grouped-variables

PF suggest using the PatternFly blue colors, but not when the value has meaning of "Danger" and "Success", in this cases "Status and state colors" it suggest red and green colors.
in our cases failed migrations fall into the category of "danger" and successful migration fall into the category of "success"
a. please use "successful" string for the finished migrations, and make sure it actually show the number of successful migrations (fix may be in a separate PR)
b. please update the colors to follow PF  "Status and state colors" color scheme
Ref: https://www.patternfly.org/design-foundations/colors#patternfly-blue
Ref: https://www.patternfly.org/design-foundations/colors#status-and-state-colors

Most likely we need another break point for the biggest screens - however it will be difficult to implement without testing on such screen.

best way to start is removing all the size modifiers and use clean PF GridItem with span and rowspan for better maintainability and readability, it will also have better results visually (no strange gaps)

LGTM
I have some issues with:
a. making the bucketing system more genral
b. making the cards layout work on large screens.
I will address this issues in a followup, great job :fire:


Install older gptq version

Attempts to add support for older GPTQ-For-LLama version. Unfortunately, not yet working as expected.

Cloned https://github.com/oobabooga/text-generation-webui into a copy repository: https://github.com/paolorechia/gptq-for-llama-old.
Added new requirements file run_server_old_gptq.sh to install the older version instead.

Make sure to run pip uninstall gptq_for_llama  when switching between the different library versions.
Server should run like this:
export USE_4BIT=true
export USE_13B_MODEL=true
export MODEL_PATH=vicuna-13B-1.1-GPTQ-4bit-128g
export MODEL_CHECKPOINT=vicuna-13B-1.1-GPTQ-4bit-128g.no-act-order.pt
uvicorn servers.vicuna_server:app

@alexandme, @unoriginalscreenname
I've tried addressing your issues by supporting a different GPTQ-For-LLama, forked from this repo: https://github.com/oobabooga/text-generation-webui
However, I could not get it to run properly, this code version does not seem compatible with https://huggingface.co/TheBloke/vicuna-AlekseyKorshuk-7B-GPTQ-4bit-128g
Do you guys have any idea which version of the library works for you? Commit, branch etc?

@alexandme, @unoriginalscreenname
I've tried addressing your issues by supporting a different GPTQ-For-LLama, forked from this repo: https://github.com/oobabooga/text-generation-webui
However, I could not get it to run properly, this code version does not seem compatible with https://huggingface.co/TheBloke/vicuna-AlekseyKorshuk-7B-GPTQ-4bit-128g
Do you guys have any idea which version of the library works for you? Commit, branch etc?

Hello, thank you for your time looking into this. Seems like with your version it does not work.
Just to make sure, the version of gptq-for-llama seen after running the installation from this PR is the following:
pip show gptq-for-llama
Name: gptq-for-llama
Version: 0.1.0
Summary:
Home-page: https://github.com/paolorechia/GPTQ-for-LLaMa

Not directly related to this, just to mention:
Following config does not work using both old and new version of gptq-for-llama
NVIDIA-SMI 510.73.05    Driver Version: 510.73.05    CUDA Version: 11.6
Quadro P5000

This config  does work with new version of gptq-for-llama
NVIDIA-SMI 510.73.05    Driver Version: 510.73.05    CUDA Version: 11.6
NVIDIA RTX A5000

Hi, @alexandme
thanks for trying it out but it’s not ready yet.
I couldn’t make it to work with oobagooba current version, which is why I’m
wondering if you know which branch or commit of GPTQ-For-LLama works with
your setup (not with my code, but using with oobagooba code).
If you install this PR’s version correctly you should see
Version: 0.1.1
You need to use a different requirements file directly with pip install to
change it. But like I mentioned it’s also not gonna work yet, as it’s not
ready.
Alexander @.***> schrieb am So. 30. Apr. 2023 um 13:37:

@alexandme https://github.com/alexandme, @unoriginalscreenname
https://github.com/unoriginalscreenname
I've tried addressing your issues by supporting a different
GPTQ-For-LLama, forked from this repo:
https://github.com/oobabooga/text-generation-webui
However, I could not get it to run properly, this code version does not
seem compatible with
https://huggingface.co/TheBloke/vicuna-AlekseyKorshuk-7B-GPTQ-4bit-128g
Do you guys have any idea which version of the library works for you?
Commit, branch etc?
Hello, thank you for your time looking into this. Seems like with your
version it does not work.
Just to make sure, the version of gptq-for-llama seen after running the
installation from this PR is the following:
pip show gptq-for-llama
Name: gptq-for-llama
Version: 0.1.0
Summary:
Home-page: https://github.com/paolorechia/GPTQ-for-LLaMa
Not directly related to this, just to mention:
Following config does not work using both old and new version of
gptq-for-llama
NVIDIA-SMI 510.73.05    Driver Version: 510.73.05    CUDA Version: 11.6
Quadro P5000
This config does work with new version of gptq-for-llama
NVIDIA-SMI 510.73.05    Driver Version: 510.73.05    CUDA Version: 11.6
NVIDIA RTX A5000
—
Reply to this email directly, view it on GitHub
https://github.com/paolorechia/learn-langchain/pull/12#issuecomment-1529003500,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABJDFZ65DKDLOPRGHGGO3TLXDZFJLANCNFSM6AAAAAAXQOGZEQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

I'm closing this, as I'm gonna try a different approach.


Documenting access, license and scope for US pvol data

@adokter or @CeciliaNilsson709 could you document the US pvol data for the DMP in this document?
For reference, here's what I wrote for the European pvol data: Rmd and webpage.
From what I know, it's:

[ ] Source: NEXRAD, but what is the access point
[ ] License: free to use (url for this info)
[ ] Format: odim?
[ ] Geographical scope: any good page with a list of all NEXRAD radars? How many of these generate data? See the European section
[ ] Temporal scope: 25 years, from x to x

Updated this issue with links to the relevant Rmd files (rather than wiki).


chore: add primary-actions to code-block and use it across the app

In the last feature we added, I tried to just go through and use what we had to make the feature.
Now we are coming back to cleanup/refactor and refine any bits which I held off on whilst building out the feature.
This PR adds a primary-actions slot to CodeBlock (I noticed it already has a secondary-actions slot, but no primary-)., and then uses that to add a [Refresh] button wherever we have that.
Lastly, the CodeBlock.vue file needs moving into the new code-block/ folder I added here, but I figured it might be easier to review the changes first, then move once we get an a-ok here (I'll probably also put EnvoyData and ResourceCodeBlock in the same folder)

Generally wondering: what’s the purpose of this PR? I feel like adding the Refresh button like we’re currently doing is good enough and more flexible (you can put the refresh button anywhere in the UI). Why do we need this to be part of the CodeBlock component?

Not totally sure if I understand you question, but:

Why do we need this to be part of the CodeBlock component?

The refresh button is not part of the CodeBlock component. You still add that from The Outside via a 'primary-actions' slot which then appears above the CodeBlock like a toolbar. We already had a configurable secondary-actions slot, but the current layout requires a separate area above the CodeBlock. I also didn't want to try adding a refresh button into secondary-actions as it covers up the text that I want to see.

Generally wondering: what’s the purpose of this PR?

Mainly, you now no longer need to repeat the styles for the 'toolbar' layout for whatever tooling buttons you need to add to the code block. Ideally I would have like to add this button to the actual KCodeBlock 'toolbar', but it doesn't look like you can configure those from The Outside.

Reading your comment again I do think I've understood you correctly, and I think you might have misunderstood what is happening here:

I feel like adding the Refresh button like we’re currently doing is good enough and more flexible (you can put the refresh button anywhere in the UI)

There is no Refresh button in the CodeBlock component, only a slot to put arbitrary buttons. As these buttons are added from the The Outside its easier to add actions/events to them without a tonne of extra plumbing (as would be the case if I'd hardcoded buttons inside the CodeBlock itself)
Does that help to answer your questions?

My question is rather why this is related to the CodeBlock via the introduction of primary actions at all: We use refresh buttons in other places, too, and they don’t always refresh just the contents of a code block like they do for the envoy data triplet. It looks like what’s being generalized here is the concept of an actions toolbar and more specifically the repetition of its styles is avoided. That could be done via a CSS rule set in src/assets/styles/_components.scss which we could use in other places, too.
Where the refresh button is located currently on the envoy data pages -- i.e. above the code block -- is rather coincidental.

My question is rather why this is related to the CodeBlock via the introduction of primary actions at all

CodeBlock has a secondary actions slot, but no primary actions slot. At the very least thats a little weird. This gives it a primary slot above the editor, where we commonly add more actions (a refresh action that refreshes the content of the codeblock)
I would add it to secondary actions but it puts the button in the wrong place. Ideally I would be able to add buttons to the KCodeBlock toolbar but I can't.

Is there a difference is here to when you added the secondary-actions block? I'm doing exactly the same thing here, but I need the positioning to be different so I don't think I can use the secondary actions block without some strange CSS.
A larger point is: We currently have 4 different components just to show code, this feels very weird.
Ideally we should have one and only one that is sufficiently configurable. This is the start of a step towards reducing those 4 components (which is essentially the same component) down to 1 or 2 components.

@kleinfreund As explained in https://github.com/kumahq/kuma-gui/pull/2042#discussion_r1458838858 I'm waiting on this to move on with this work, I would appreciate it if we can find a way to resolve this a little quicker. I'd be happy to catchup offline in order to get this resolved in one sitting.

Quick note: I just added a commit on the end here to do:

Lastly, the CodeBlock.vue file needs moving into the new code-block/ folder I added here, but I figured it might be easier to review the changes first, then move once we get an a-ok here (I'll probably also put EnvoyData and ResourceCodeBlock in the same folder)


[#17] Support for Nested APIBlueprint enhancement

Support for Nested APIBlueprint enhancement

todo: Update documentation about Nested APIBlueprint

Hey @luolingchun, I have been working with your patch for the most of the week, and it seems to work, the only thing I have found is the order of when one does what, but if the right order is followed it works :D
Maybe it makes sense to have a way to do call-backs. E.g. if I add an endpoint endpoint_1 to an APIBlueprint name child and when register that APIBlueprint at one name parent I cannot afterwords add another endpoint (endpoint_2) to the child
from flask_openapi3 import APIBlueprint

parent = APIBlueprint(name="parent", import_name=__name__, url_prefix="/parent")

child = APIBlueprint(name="child", import_name=__name__, url_prefix="/child")

@child.post(rule="endpoint_1", responses={"204": MsgResponse, "400": ErrorMsgResponse})
def endpoint_1(body: PostRequest):
    return {"msg": "test msg"}

parent.register_api(child)

@child.post(rule="endpoint_2", responses={"204": MsgResponse, "400": ErrorMsgResponse})
def endpoint_2(body: PostRequest):
    return {"msg": "test msg"}

@dvaerum There is the same problem in Flask, Therefore, I think we should solve this problem by ourselves when coding.
from flask import Flask, Blueprint

app = Flask(__name__)
bl = Blueprint('test', __name__)

@bl.route('/route1')
def route1():
    return 'Hello, World!'

app.register_blueprint(bl)

@bl.route('/route2')
def route2():
    return 'Hello, World!'

if __name__ == '__main__':
    print(app.url_map)

Map([<Rule '/route1' (GET, HEAD, OPTIONS) -> test.route1>,
 <Rule '/static/<filename>' (GET, HEAD, OPTIONS) -> static>])

Sounds completely fair

@luolingchun I have been using this patch of yours for some time now and still have not found any problems with it :+1:  Just want to give you an update :smiley:

@dvaerum I made a preview release v2.0.0rc1.


Html  attribute

Adding html links "target" attribute

Hi @commodusgr
It seems that similar changes have already been suggested in another PR before yours for this article you're editing, sorry about that. 😓
If you feel you have more to add, please feel free to open up a new PR.
Thanks again! 😊

If you have any questions, feel free to reach out through Gitter or by commenting below. 💬


Don't error on prepared statements containing unspecified arrays

Closes https://github.com/staabm/phpstan-dba/issues/156

Can confirm it's fixed for me now thanks

thx for the confirmation. feel free to report any issues you might have Jordi.


Create a CI Pipeline

Create a CI pipeline to run on any PR that checks:

If commits follow Conventional Commits
If the box is correctly build and provisioned

Also change main branch protection rules to validate the execution status of the pipeline before merging.

PR-29 closes this isse


allow go tests the option to be run via ginkgo instead of requiring g…

…o test
Signed-off-by: Chris Ahl<EMAIL_ADDRESS>

Oooh, that was a bit aggressive of openshift-merge-robot


No way to display pathlines with "draw mesh" option via Settings API

🔍 Before submitting the issue

[X] I have searched among the existing issues
[X] I am using a Python virtual environment

🐞 Description of the bug
The title explains the problem. With UI, you select "Draw Mesh" before "Save/Display" and you see both mesh and pathlines. With Settings API, pathlines dict entries do not seem to have any way of describing displayed mesh, and surfaces_list points to list of surfaces, from which pathlines are released. Another way would be to display mesh from solver.results.graphics.mesh, but this also does not work as previously displayed mesh is not retained when displaying pathlines afterwards.
A sensible option would be to either allow retaining previously displayed plot when showing pathlines, or add an option to pathlines dict which plot should be added alongside pathlines.
📝 Steps to reproduce
As written in the description, no steps as there is functionality missing.
💻 Which operating system are you using?
Windows
📀 Which ANSYS version are you using?
23R1
🐍 Which Python version are you using?
3.11
📦 Installed packages
ansys-api-fluent==0.3.12
ansys-api-platform-instancemanagement==1.0.0b3
ansys-fluent-core==0.15.0
ansys-platform-instancemanagement==1.1.1
appdirs==1.4.4
googleapis-common-protos==1.59.1
grpcio==1.56.0
h5py==3.9.0
importlib-metadata==6.7.0
lxml==4.9.2
markdown-it-py==3.0.0
mdurl==0.1.2
numpy==1.25.0
pandas==2.0.3
protobuf==3.20.3
protoc-gen-swagger==0.1.0
Pygments==2.15.1
python-dateutil==2.8.2
pytz==2023.3
PyYAML==6.0
rich==13.4.2
six==1.16.0
tzdata==2023.3
zipp==3.15.0

Does it even make the pathlines ?
I am a bit confused by the pathlines.surface_list = ['inlet']. You can't select the volume of the simulation here. I believe it tries to make pathlines inside the surfaces selected (which makes no sense)...
Here is what I use but no success:
graphics_session = Graphics(solver)
pathlines1 = graphics_session.Pathlines["pathlines-1"]
pathlines1.field = "velocity-magnitude"
pathlines1.surfaces_list = ['inlet']
pathlines1.display("window-0")

If it actually make the pathlines in the object backend, and if we can recover it back, it that would be fine by me. You can use pyvista to do the visual part.

I hope I didn't make a wrong description of the problem, but I am not using pyfluent-visualization, pyvista etc. Just core ansys functionality from pyfluent.
solver.results.graphics.pathline[plot-name] = {dict} then solver.results.graphics.pathline[plot-name].display()

Sorry, I was unaware you could use pyfluent core to make pathline directly. Could you share an example of what you would put in {dict} ?
If I manage to get the pathlines I will make a pyvista figure of the line + mesh. I could share that afterward if you're interested.

Here is what I used and what you can get by making the pathline plot via GUI and then accessing it with
solver.results.graphics.pathlines['pathlines-2']() :
https://gist.github.com/rafalszulejko/a6a5bbe41dfc1c2e7701b2bb065d32af
I think you can essentially ignore the axes and curves.
Regarding pyvista approach - if you know how to change camera angle from the default one I'd be very interesting to learn how, because lack of that info in the docs was the biggest motivator for me to ignore pyfluent-visualization altogether.

@h-krishnan @ajain-work (+"Gitesh") do we require a settings API change wrt this?


avoid creating log directory in package directory during testing

As per title.

Codecov Report

Merging #68 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #68   +/-   ##
=======================================
  Coverage   77.37%   77.37%
=======================================
  Files          49       49
  Lines        2617     2617
  Branches      423      423
=======================================
  Hits         2025     2025
  Misses        570      570
  Partials       22       22

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 87b0809...0959b83. Read the comment docs.

Can you describe what log directories I should be looking for and that don't get created with this PR ?
I tried building and testing workspaces with and without this change and didn't notice a difference.
After running `colcon build` using testing_log_dir:
$ find -name log
./install_debug_isolated/fastrtps/include/fastrtps/log
./log
./src/eProsima/Fast-RTPS/src/cpp/log
./src/eProsima/Fast-RTPS/include/fastrtps/log
./build_debug_isolated/fastrtps/src/cpp/CMakeFiles/fastrtps.dir/log

After running `colcon test` using testing_log_dir:
$ find -name log
./install_debug_isolated/fastrtps/include/fastrtps/log
./log
./src/eProsima/Fast-RTPS/src/cpp/log
./src/eProsima/Fast-RTPS/include/fastrtps/log
./build_debug_isolated/fastrtps/src/cpp/CMakeFiles/fastrtps.dir/log

After running `colcon test` using master:
./install_debug_isolated/fastrtps/include/fastrtps/log
./log
./src/eProsima/Fast-RTPS/src/cpp/log
./src/eProsima/Fast-RTPS/include/fastrtps/log
./build_debug_isolated/fastrtps/src/cpp/CMakeFiles/fastrtps.dir/log


Proxy-pallet: Add a storagemap to track drawdows by type

The Fronted team required a new storage map in order to track/filter the drawdowns by their type.

Backend done.


No module named 'core'

Hello, I have this error do you have any issues of this ?
Traceback (most recent call last):   File "C:\Users\xxxx\Desktop\pywinassistant-Pre-Database\core\assistant.py", line 8, in <module>     from core.voice import speaker, set_volume, set_subtitles ModuleNotFoundError: No module named 'core'

Hey I'm not one of the devs but you should probably try running it like this:
python3 -m core.assistant


[ngAria] Don't warn about missing aria-label when aria-labelledby is specified

ngAria requires "aria-label" even when a valid "aria-labelledby" attribute is defined.
Many of components in our templates that require an "aria-label" attribute are already labeled by blocks of text next to it. I would therefore like to use "aria-labelledby" instead of having to maintain a copy of the text in "aria-label".
I believe that many developers would see the logic in specifying "aria-labelledby" vs having to maintain an "aria-label" property when the interface already seems self-explanatory to them.
https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/ARIA_Techniques/Using_the_aria-labelledby_attribute

Sounds reasonable. Is this something you'd be interested to work on?

Seems I have been mistaken and this is actually an issue with angular-material. Sorry, I falsely assumed mdAria was using ngAria for this.


fix(postgres): use explicit schema in table identifier when altering comments

Closes #4108

Codecov Report
Patch coverage: 100.00% and project coverage change: -0.02 :warning:

Comparison is base (549beec) 99.68% compared to head (44aa537) 99.66%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4123      +/-   ##
==========================================
- Coverage   99.68%   99.66%   -0.02%
==========================================
  Files         214      214
  Lines       13799    13847      +48
  Branches     3237     3258      +21
==========================================
+ Hits        13755    13801      +46
- Misses         41       43       +2
  Partials        3        3

Impacted Files
Coverage Δ

packages/mariadb/src/MariaDbSchemaHelper.ts
100.00% <ø> (ø)

packages/knex/src/schema/SchemaGenerator.ts
100.00% <100.00%> (ø)

packages/knex/src/schema/SchemaHelper.ts
100.00% <100.00%> (ø)

packages/mysql/src/MySqlSchemaHelper.ts
100.00% <100.00%> (ø)

packages/postgresql/src/PostgreSqlSchemaHelper.ts
100.00% <100.00%> (ø)

... and 9 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


CAMEL-10716 Salesforce Composite API wrongly checks for minimum supported version

Fixes an issue when the Batch request and the configured componentversion is version 34.0.
Added unit test and refactored the check into separate method to make the unit test easier to write.

Thanks for the PR it has been merged. Do you mind closing this?

Thanks @davsclaus :+1:


fix: fix index on namespace table

The current version adds index on job table and not namespace: https://github.com/odpf/optimus/blob/main/store/postgres/migrations/000008_add_namespace.up.sql#L13

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 75.109%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
6044

Relevant Lines:
8047

💛  - Coveralls


import csv: add the ability to construct a timestamp from multiple columns

Here is an example CSV:
Date,Time,Tag,Value
2020-05-22,00:00:00,test,0
2020-05-22,00:05:00,test,1
2020-05-22,00:10:00,test,2

I would like a way to combine multiple time columns into a single timestamp to be used on the import. In this case, the resulting line protocol timestamps would combine the Date and the Time column to produce:
1590105600000000000
1590105900000000000
1590106400000000000

other example CSVs that should be able to be parsed with this solution:
Year,Month,Day,Hour,Minute,Second,Tag,Value
2020,05,22,00,00,00,test,0
2020,05,22,00,05,00,test,1
2020,05,22,00,10,00,test,2

the current behavior prints a warning:
2020-06-25T21:47:08.731185Z	info	WARNING: at most one dateTime column is expected, 'date' column is ignored

The same thoughts as in https://github.com/influxdata/influxdb/issues/18743#issuecomment-650491417 apply herein. I am thinking of adding another annotation that would add a computed column from existing columns (and constant values) as a solution. Something like
#computed,date|dateTime:dateTime:2006-01-02|"{{row ""Year""}}-{{row ""Month""}}-{{row ""Day""}}"

The original columns can be then ignored during csv import.


”Odoreu” is listed in the drop-down menu

Description:
"Odoreu" is listed in the drop-down menu alongside its corresponding county and commune, according to the law.
Precondition:
The website is up an running.

Step 1
Type "Odoreu" in the search bar.

Expected results
The location is listed in the drop-down menu as "Comuna Odoreu, SATU MARE".


pot_metric does not work with Booster

plot_metric works with LGBMModel, but not with lightgbm.basic.Booster.
This seems to be by design, but seems like a clear gap in functionality.
https://github.com/microsoft/LightGBM/blob/master/python-package/lightgbm/plotting.py
    if isinstance(booster, LGBMModel):
        eval_results = deepcopy(booster.evals_result_)
    elif isinstance(booster, dict):
        eval_results = deepcopy(booster)
    else:
        raise TypeError('booster must be dict or LGBMModel.')

I'm just getting started with lightgbm, is there an easy workaround?

After more hunting I found the workaround from this issue: https://github.com/microsoft/LightGBM/issues/1491 which points to this example
This took me a surprisingly long time to figure out. I would suggest updating the exception message from plot_metric to give instructions on how to use the method with Booster
something like
if isinstance(booster, Booster):
    raise Exception("to plot the metrics for a Booster type, first output the metrics using evals_result then pass that in to this method as `booster`")


Fix can't modify cell content issue

PR Details

Description

Related Issue

#1263
Motivation and Context

How Has This Been Tested

Types of changes

[ ] Docs change / refactoring / dependency upgrade
[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist

[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

LGTM, thanks for your great work.


WIP: Kafka Reader

Initial implementation still very much a work in progress.
Connection logic needs to move into a terafoundation connector.
If this can read a full size worth of results it will complete the slice. Otherwise it will automatically checkpoint the slice every wait ms even if there are 0 results.
The wait time and size set in the configuration appears very finicky. Still trying to understand what's going on. Setting a longer wait time results in elasticsearch errors from the worker when trying to save state.
    {
      "_op": "teraslice_kafka_reader",
      "size": 1000,
      "topic": "testing2",
      "group": "testing-group",
      "wait": 10000
    },

refs #1

This is roughly useable as a first pass so I'm going to go ahead and merge it.


CountMinSketch is not serializable

Could we make com.clearspring.analytics.stream.frequency.CountMinSketch implementing java.io.Serializable please?
I am getting this error:
org.apache.commons.lang3.SerializationException: java.io.NotSerializableException: com.clearspring.analytics.stream.frequency.CountMinSketch
at org.apache.commons.lang3.SerializationUtils.serialize(SerializationUtils.java:156)

CountMinSketch has serialize and deserialize in the current implementation.  It should be possible to extend the Serializable interface as well.  Have you looked to see if using the serialize/deserialize methods is viable for you?  Are you interested in submitting a pull request that makes the desired modification?

Using serialize and deserialize would make my code complex.
Best for me would be to implement Serializable as it requires only a simple change on your code, and no changes on my code.
I can create a pull request if it is fine for you guys.
Thanks

https://github.com/addthis/stream-lib/pull/110

Hi
Any news on this ticket?
Thanks

Sorry for the delay. The code looks fine. I'll do a final pass over.

Thanks
Let me know when can be merged and released :+1:

I close the issue as the code has been merged

It's released.

Thanks :+1:


The code is solved.

I have solved the java easy code.
I have added a scanner class in java to take input.
Whenever  s user will enter his/her name he will be greeted with a message
Hello, (name) Nice to meet you.

Merged

This is an automated message from Fork, Commit, Merge [BOT].

Thank you for your contribution! Your pull request has been merged. The files have been reset for the next contributor.
What's next?
If you're looking for more ways to contribute, I invite you to check out my other projects. Just click here to find more. These projects contain real issues that you can help resolve. You can also check out the Influences section in the README to find more projects similar to this one.
Also please leave a star to this project if you feel it helped you, i would really appreciate it.
I look forward to seeing your contributions!


Update ISSUE_TEMPLATE.md . npm cache clear throws error

Laravel Mix Version: 1.5.0
Node Version (node -v): v6.10.2
NPM Version (npm -v): 5.4.2
OS: Ubuntu

Description:
Steps provided by new issue template throws an error. Change npm cache clear with npm cache verify
Steps To Reproduce:
Execute:
rm -rf node_modules && npm cache clear && npm install
npm ERR! As of npm@5, the npm cache self-heals from corruption issues and data extracted from the cache is guaranteed to be valid. If you want to make sure everything is consistent, use 'npm cache verify' instead.
npm ERR!
npm ERR! If you're sure you want to delete the entire cache, rerun this command with --force.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/sergi/.npm/_logs/2017-10-11T15_41_26_165Z-debug.log

npm cache clear -f

Above command should work on npm 5

Yes but then there is a warning about not necessary to clear cache. Maybe verify is better?

From the npm docs

verify: Verify the contents of the cache folder, garbage collecting any unneeded data, and verifying the integrity of the cache index and all cached data.

But here we want user to install latest versions of all dependencies before reporting any issue.
Verify option does not fit here.


Change some improper translations.

And also remove junk code between each lifecycle method. Then add index number before each method to help the reader understand the running order better.

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the master branch of freeCodeCamp.
[x] None of my changes are plagiarized from another source without proper attribution.
[x] My article does not contain shortened URLs or affiliate links.

If your pull request closes a GitHub issue, replace the XXXXX below with the issue number.
Closes #XXXXX

@muraleo
Congrats on your first pull request (PR)! 🎉
Thank you for your contribution to the page! 👍 We're happy to accept these changes and look forward to future contributions.


⚠ Bofrost.lu has degraded performance

In 20949b4, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 982 ms

Resolved: Bofrost.lu performance has improved in cf0371a after 21 minutes.


getLongList returns integerlist?

Describe the bug
getLongList seems to return an integer list?
 List<Long> legacy = userdata.getLongList(profile.getPk() + ".followers");
When trying to loop through it i get
Exception in thread "main" java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Long
Here is an example of the list: https://paste.md-5.net/mitokajuni.json
Yes the list is getting returned as is, not empty.
To Reproduce
Steps to reproduce the behavior:
Create a List using #getLongList()
Loop through it using for (long entry : list), throws this error
Expected behavior
it should loop but i think it's casting Integer while creating the loop or something?
Screenshots
N/A
Desktop (please complete the following information):

OS: Windows 10
Browser Chrome?
Version dont know (simplixstorage version 3.2.2)

Additional context
N/A

Any further questions?

Nope, that would be all, just want to be notified when this is fixed :)

I am sorry for the late response - I am currently in my final exams in school.
I still don't exactly understand where this is an issue of SimplixStorage. What exactly do you want to change?
The exception you are currently getting is caused by the way the data is read by the background library SimplixStorage is using - It seems to be recognized as an Integer, not as a Long.
As stated above we would need to convert every value in the List manually which would require some computation time.
Regards,
Leonhard

No problem i am also doing finals soon and i dont actually fully remember what the issue is.
I think i was trying to use the longs you get from longList and loop through them but since they are read by integers it looks like its throwing that error.
And wouldn't this be a problem if it is an actual long but it recognizes it as an Integer. Because longs can be higher than integers so wouldn't that be somewhat an issue
Basically i am trying to get a list of longs but it looks like its returning a list of Integers instead. Trying to loop through them causes those errors.

Well it looks like its just of a different data type. getIntergerList should do the job.

Well it looks like its just of a different data type. getIntergerList should do the job.

not all longs can be an interger though? So storing a long that isn't an int when trying to retrieve it will result in an error..


bridge AST Creator for keyword?-like methods

Can you explain what you're doing here?

Can you explain what you're doing here?

@fabsx00 There is a change made in the parser production that a keyword can be used as a methodIdentifier. The AstCreator did not accommodate  those changes. This PR does that.

Closing the PR, @rahul-privado will post the fix.

Please do not merge this.


chore: add game id to server endpoints

Changes

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#1014  👈
#1013
#1006
#1005
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @NathanFlurry and the rest of your teammates on  Graphite

Presumably this is so game info can be checked even if a token isn't passed in?


HetznerBareMetalHost status is part of spec

/kind bug
What steps did you take and what happened:
If you describe a HetznerBareMetalHost you can see, that the .status fields are placed under .spec.status.
What did you expect to happen:
Placing status as .status and not as .spec.status is the "normal" way. Are there any reasons why this is placed under .spec.status here?
Anything else you would like to add:
You can see the same structure in the code here.
Environment:

cluster-api-provider-hetzner version: v1.0.0-beta.34
Kubernetes version: v1.30.0

Yes there is. The status in Kubernetes is something that can be lost and the reproduced. What you see in spec.status cannot be reproduced in some parts and therefore cannot be put into the Kubernetes status, at least parts.
This is why we had the options of putting it under spec (what we did) or in the annotations in some compact json string. We decided to put it in the spec because it is more stable than the annotations that have a size limit.
If you have another perspective how to achieve this (to not lose this data because we cannot just re-create it in any reconcile loop), I'm open for it!

Thank you for your quick and detailed reply @janiskemper!
I think the decision is understandable and I have no further perspective or questions.
The whole decision (or this issue) could be written/referenced as code comment to avoid future questions and to document the decision process.

makes sense! will do!


remove additional params separator

given current conventions/ patterns of norns script parameters, this separator creates an extra separator in the norns parameters.

i'm nearly done rewriting the whole thing. this will not be an issue in the next iteration.


Define gff3 specification

We need to write an explicit gff3 specification for our transcriptome gff3, with code to test this added to check_gff_fasta.py. Making transcriptome annotations compatible is the key to bringing in new organisms and to making data portable between different tools such as ORF finders.
I'm concerned that we're not following standard practice as described in gmod, in particular on IDs/parent, and on feature types such as UTR5 vs five_prime_utr.
We might want to add

start_codon
stop_codon
ORF (for upstream ORFs)

@ewallace will check with the Trips-viz team to see if we can make something compatible with their browser. This would make it easy to process output in Riboviz and visualize in Trips-viz.
Related to #25

tripvs-viz developers are willing to engage so we should do that.

Confirmed that

TRIPS-Viz developers are engaging. @ewallace to set up a call with them and @FlicAnderson to figure next steps.
official gff3 specification has sequence ontology terms five_prime_UTR and three_prime_UTR, so we need to change the names in our gffs accordingly.

Checker check_gff_fasta.py, currently has three constraints/tests:

the beginning of every CDS is a start codon (ATG; translates to M)
the stop of every CDS is a stop codon (TAG, TGA, TAA; translates to *)
there are no stop codons internal to the CDS.

We want to:

give UTR features as arguments, with default values from SO five_prime_UTR and three_prime_UTR
put each constraint in a separate function
add flexibility in start codon specification for non-ATG starts, so must check the codon not the translated amino acid
add flexibility in stop codon specification for non-TGA/TAG/TAA stops, ditto
add constraint to check UTRs (a) present and (b) longer than a user-specified length, default 13.
check uORFs in the same way as CDSs, if present
count how many transcripts have multiple CDSs
add constraint to check if CDSs or uORFs overlap
report out how many features pass each constraint
flags to allow filtering for each constraint
count all features and their lengths

We still have some questions:

do we want to list uORFs as uORFs or as CDSs with names e.g. YAL002W_uORF1? The issue is that generate_stats_figs.R currently only quantifies over CDSs, so uORF quantification requires change there.
how do we specify alternative in-frame starts, i.e. ORF extensions?
how do we test parent relationships, or feature names more generally. For example, should the five_prime_UTR of YAL002W be called YAL002W_five_prime_UTR?

Note that gffutils already has feature inspection code in inspection.py. Already, check_fasta_gff.py uses gffutils so this should help.

Made a start on this and replacing #23/#50 python scripts in new branch #gff-spec-74.
Some example files and a README.md are in data/annotationexamples in that branch. Incomplete as needs the transcript-centric gffs adding.
Would like to discuss with @mikej888 what else is needed in the spec. We could code this using Biopython and gffutils, which would then replace script_for_transcript_annotation.Rmd.

I'll explore the content in more detail when I've progressed #167, but to summarise my interpretation of the task to date....
Port script_for_transcript_annotation.Rmd into a Python script which:

Validates input genome GFF file (from #50, "First challenge will be to validate the input genome gff file. Our check_gff_fasta.py validates the transcriptome gff and fasta file. The upstream challenge in script_for_transcript_annotation.Rmd is to check that and then make valid transcriptome files.")
Creates transcriptome GFF/FASTA from genome GFF/FASTA files.
Uses Biopython and gffutils packages.

I agree with that interpretation.
Today I updated branch gff-spec-74 with transcript-centric gffs for examples YAL10 and YBL72ish in commit def937e. Those gffs and fasta files are generated from a mixture of R/tidyverse/unix bedtools code in a separate repository yeastutrgff. Some of the code in r/tidyverse is less clunky; but the intron removal we did quite separately using bedtools functions.
We should test gff and fasta compatibility with check_gfff_fasta before anything else; I tested transcript length from gff and fasta agreed by eye, but not that CDS were in the correct position.
Also, I will email the fungidb developer team to find out how they produce their transcript information for the fungidb website.

Updated and fixed some bugs in the two examples:

YAL10_transcriptsfixed_R64-2-1_left18_right15
YBL72ish_transcripts_withUTRs

Still did not test with check_fasta_gff because I don't currently have access to an installation where that is working. Or, I have not understood something about how to call the code from the command line. We can discuss at dev meeting.

The syntax is now
$ python -m riboviz.tools.check_fasta_gff -f <FASTA_FILE> -g <GFF_FILE>

For example:
$ python -m riboviz.tools.check_fasta_gff -f vignette/input/yeast_YAL_CDS_w_250utrs.fa -g vignette/input/yeast_YAL_CDS_w_250utrs.gff3
Created by: RiboViz
Date: 2020-04-20 02:26:42.265614
Command-line tool: /home/ubuntu/riboviz/riboviz/tools/check_fasta_gff.py
File: /home/ubuntu/riboviz/riboviz/tools/check_fasta_gff.py
Version: commit 0ecd69ec0786eaeee78c4ce0999f4a181c775044 date 2020-03-04 05:45:40-08:00

Checking fasta file vignette/input/yeast_YAL_CDS_w_250utrs.fa
with gff file vignette/input/yeast_YAL_CDS_w_250utrs.gff3
YAL001C doesn't start with ATG.
YAL001C doesn't stop at end.
YAL001C has internal STOP.

$ python -m riboviz.tools.check_fasta_gff -f data/yeast_CDS_w_250utrs.fa -g data/yeast_CDS_w_250utrs.gff3
Created by: RiboViz
Date: 2020-04-20 02:27:04.606550
Command-line tool: /home/ubuntu/riboviz/riboviz/tools/check_fasta_gff.py
File: /home/ubuntu/riboviz/riboviz/tools/check_fasta_gff.py
Version: commit 0ecd69ec0786eaeee78c4ce0999f4a181c775044 date 2020-03-04 05:45:40-08:00

Checking fasta file data/yeast_CDS_w_250utrs.fa
with gff file data/yeast_CDS_w_250utrs.gff3
Q0050 has internal STOP.
Q0055 has internal STOP.
Q0060 has internal STOP.
Q0065 has internal STOP.
Q0070 has internal STOP.
Q0045 has internal STOP.
Q0075 doesn't start with ATG.
Q0075 has internal STOP.
Q0085 has internal STOP.
Q0110 has internal STOP.
Q0115 has internal STOP.
Q0120 has internal STOP.
Q0105 has internal STOP.
Q0140 has internal STOP.
Q0160 has internal STOP.
Q0250 has internal STOP.
Q0255 has internal STOP.
Q0275 has internal STOP.

 Inconsistent line found in >YAL001C_CDS at line 3.
Line length of fasta file is not consistent! Inconsistent line found in >YAL001C_CDS at line 3.
Line length of fasta file is not consistent! Inconsistent line found in >YAL001C_CDS at line 3.
Line length of fasta file is not consistent! Inconsistent line found in >YAL001C_CDS at line 3.
Line length of fasta file is not consistent! Inconsistent line found in >YAL001C_CDS at line 3.
Line length of fasta file is not consistent! Inconsistent line found in >YAL001C_CDS at line 3.
Line length of fasta file is not consistent! Inconsistent line found in >YAL001C_CDS at line 3.
Line length of fasta file is not consistent! Inconsistent line found in >YAL001C_CDS at line 3.
Line length of fasta file is not consistent! Inconsistent line found in >YAL001C_CDS at line 3.
Line length of fasta file is not consistent! Inconsistent line found in >YAL001C_CDS at line 3.

YBL72ish_transcripts_withUTRs:
$ python -m riboviz.tools.check_fasta_gff -f data/annotationexamples/YBL72ish_transcripts_withUTRs.fasta -g data/annotationexamples/YBL72ish_transcripts_withUTRs.gff
Created by: RiboViz
Date: 2020-04-20 02:39:03.714980
Command-line tool: /home/ubuntu/riboviz/riboviz/tools/check_fasta_gff.py
File: /home/ubuntu/riboviz/riboviz/tools/check_fasta_gff.py
Version: commit 5ed5018113923c0110c86a6ae8f97d099cee0ca6 date 2020-04-18 10:16:21+01:00

Checking fasta file data/annotationexamples/YBL72ish_transcripts_withUTRs.fasta
with gff file data/annotationexamples/YBL72ish_transcripts_withUTRs.gff

Fix issue with YAL10_transcriptsfixed_R64-2-1_left18_right15:

From How can I join the wrap lines in a huge FASTA file?, which concatenates all the sequences for a record into a single wrapped line:

$ cat data/annotationexamples/YAL10_transcriptsfixed_R64-2-1_left18_right15.fasta | sed -e '/^>/s/$/@/' -e 's/^>/#/'|tr '\n' ' '|sed 's/ //g'|sed 's/@/\n/g'|sed 's/#/\n>/g'|sed '/^$/d' > YAL10_transcriptsfixed_R64-2-1_left18_right15_wrapped.fasta

$ python -m riboviz.tools.check_fasta_gff -f YAL10_transcriptsfixed_R64-2-1_left18_right15_wrapped.fasta -g data/annotationexamples/YAL10_transcriptsfixed_R64-2-1_left18_right15.gff
Created by: RiboViz
Date: 2020-04-20 03:07:59.642646
Command-line tool: /home/ubuntu/riboviz/riboviz/tools/check_fasta_gff.py
File: /home/ubuntu/riboviz/riboviz/tools/check_fasta_gff.py
Version: commit 5ed5018113923c0110c86a6ae8f97d099cee0ca6 date 2020-04-18 10:16:21+01:00

Checking fasta file YAL10_transcriptsfixed_R64-2-1_left18_right15_wrapped.fasta
with gff file data/annotationexamples/YAL10_transcriptsfixed_R64-2-1_left18_right15.gff
'YAL001C_mRNA not in YAL10_transcriptsfixed_R64-2-1_left18_right15_wrapped.fasta.'
'YAL002W_mRNA not in YAL10_transcriptsfixed_R64-2-1_left18_right15_wrapped.fasta.'
'YAL003W_mRNA not in YAL10_transcriptsfixed_R64-2-1_left18_right15_wrapped.fasta.'
'YAL005C_mRNA not in YAL10_transcriptsfixed_R64-2-1_left18_right15_wrapped.fasta.'
'YAL007C_mRNA not in YAL10_transcriptsfixed_R64-2-1_left18_right15_wrapped.fasta.'
'YAL008W_mRNA not in YAL10_transcriptsfixed_R64-2-1_left18_right15_wrapped.fasta.'
'YAL009W_mRNA not in YAL10_transcriptsfixed_R64-2-1_left18_right15_wrapped.fasta.'
'YAL010C_mRNA not in YAL10_transcriptsfixed_R64-2-1_left18_right15_wrapped.fasta.'
'YAL011W_mRNA not in YAL10_transcriptsfixed_R64-2-1_left18_right15_wrapped.fasta.'
'YAL012W_mRNA not in YAL10_transcriptsfixed_R64-2-1_left18_right15_wrapped.fasta.'

Preferable, in-Python approach, using format_fasta.py, which reads then writes the FASTA file. The output file has each sequence spread over multiple lines but all the lines have the same length, with the possible, but acceptable, exceptions of the final line for a sequence:

from Bio import SeqIO
import sys

in_file = sys.argv[1]
out_file = sys.argv[2]
with open(in_file, "rt") as f:
    records = list(SeqIO.parse(f, "fasta"))
SeqIO.write(records, out_file, "fasta")

$ python format_fasta.py data/annotationexamples/YAL10_transcriptsfixed_R64-2-1_left18_right15.fasta YAL10_transcriptsfixed_R64-2-1_left18_right15_formatted.fasta

$ python -m riboviz.tools.check_fasta_gff -f YAL10_transcriptsfixed_R64-2-1_left18_right15_formatted.fasta -g data/annotationexamples/YAL10_transcriptsfixed_R64-2-1_left18_right15.gff
Created by: RiboViz
Date: 2020-04-20 03:05:47.342654
Command-line tool: /home/ubuntu/riboviz/riboviz/tools/check_fasta_gff.py
File: /home/ubuntu/riboviz/riboviz/tools/check_fasta_gff.py
Version: commit 5ed5018113923c0110c86a6ae8f97d099cee0ca6 date 2020-04-18 10:16:21+01:00

Checking fasta file YAL10_transcriptsfixed_R64-2-1_left18_right15_formatted.fasta
with gff file data/annotationexamples/YAL10_transcriptsfixed_R64-2-1_left18_right15.gff
'YAL001C_mRNA not in YAL10_transcriptsfixed_R64-2-1_left18_right15_formatted.fasta.'
'YAL002W_mRNA not in YAL10_transcriptsfixed_R64-2-1_left18_right15_formatted.fasta.'
'YAL003W_mRNA not in YAL10_transcriptsfixed_R64-2-1_left18_right15_formatted.fasta.'
'YAL005C_mRNA not in YAL10_transcriptsfixed_R64-2-1_left18_right15_formatted.fasta.'
'YAL007C_mRNA not in YAL10_transcriptsfixed_R64-2-1_left18_right15_formatted.fasta.'
'YAL008W_mRNA not in YAL10_transcriptsfixed_R64-2-1_left18_right15_formatted.fasta.'
'YAL009W_mRNA not in YAL10_transcriptsfixed_R64-2-1_left18_right15_formatted.fasta.'
'YAL010C_mRNA not in YAL10_transcriptsfixed_R64-2-1_left18_right15_formatted.fasta.'
'YAL011W_mRNA not in YAL10_transcriptsfixed_R64-2-1_left18_right15_formatted.fasta.'
'YAL012W_mRNA not in YAL10_transcriptsfixed_R64-2-1_left18_right15_formatted.fasta.'

Thanks, that was my mistake on YAL10 examples, both the sequence names and the absence of line wrapping. Try the update in 48b2fb6?

$ git log -1
commit 48b2fb6b390774d37b2acb297c1f3d005e586299 (HEAD -> gff-spec-74, origin/gff-spec-74)
Author: Edward Wallace<EMAIL_ADDRESS>Date:   Mon Apr 20 17:36:37 2020 +0100

    YAL10 example, seqnames and text wrap

$ python -m riboviz.tools.check_fasta_gff -f data/annotationexamples/YAL10_transcriptsfixed_R64-2-1_left18_right15.fasta -g data/annotationexamples/YAL10_transcriptsfixed_R64-2-1_left18_right15.gff
Created by: RiboViz
Date: 2020-04-21 01:31:35.544253
Command-line tool: /home/ubuntu/riboviz/riboviz/tools/check_fasta_gff.py
File: /home/ubuntu/riboviz/riboviz/tools/check_fasta_gff.py
Version: commit 48b2fb6b390774d37b2acb297c1f3d005e586299 date 2020-04-20 17:36:37+01:00

Checking fasta file data/annotationexamples/YAL10_transcriptsfixed_R64-2-1_left18_right15.fasta
with gff file data/annotationexamples/YAL10_transcriptsfixed_R64-2-1_left18_right15.gff

Sorted!

We agreed that these files, plus script for transcript annotation (#23) are enough starting material to test the gff3 specification and annotation task.
We could easily also add other fungal examples from fungidb data. perhaps a small selection of genes from S. pombe 972h- which has UTR and intron annotations, example S. pombe act1 gene.
Later, we need to add more examples to cover:

multiple ORFs per transcript, #159 for bacteria.
drosophila examples covering bicistronic genes.

Today I discussed drosophila examples with Julie Aspden. Julie and her student Isabel will send us drosophila .fasta and .gff examples, hopefully in next week. From those we can obtain the bicistronic transcripts for testing.

@ewallace and @mikej888 discussed this work. Mike's initial task can be to write a script to calculate the positions of all codons in a gene. This would allow us to replace yeast_codon_pos_i200.RData (binary R file format with nested lists and which starts at position 201) with a plain-text three column table. For example, suppose we have coding sequences:
G1: ATG AAA TAA
G2: ATG GGG CCC TAG

ATG start codon, TAA, TAG end codons. Then the codon position table should read: (check column names for consistency):
Gene  Pos  Codon
G1    1    ATG
G1    2    AAA
G1    3    TAA
G2    1    ATG
G2    2    GGG
G2    3    CCC
G2    4    TAG

i.e. a table with all codons and their positions. generate_stats_figs.R can then be updated to read this and use (a subset of) the data within the table.
The script to produce this should take both FASTA and GFF files and output this table. Remember that GFF positions are 1-indexed.
Check to see if Biopython has any useful functions to help e.g. to chop a sequence into codons.
Complement with unit tests.
More generally consider how these, and related functions, can be collected together into a coherent library for use by ourselves and others.
https://github.com/xryanglab/RiboCode may provide inspiration.
RiboViz overloads names to refer to genes, transcripts and coding sequences. It would be desirable to give genes, transcripts and coding sequences different names. GFF files record transcript names in the leftmost sequence column. This also records the names of features. Example names could include: YAL001C_<transcript-plus-12-minus-15>, or, following FungiDB <GENE>_t0,1,2,3 for transcripts. Naming patterns should be configurable.

Relates to @FlicAnderson work on #194 which will read this data.

Merged current develop branch into gff-spec-74 and committed WIP version riboviz/extract_cds_codons.py which uses the GFF file CDS entries to get coding sequences then splits these into their codons and creates a DataFrame holding the data in the desired format. This DataFrame needs to be saved as a TSV file, the code refactored into more functions, the command-line part extracted out into riboviz.tools.extract_cds_codons.py (essentially copy check_fasta_gff.py) and tests need to be implemented.

Current state:

Refactored into functions, within riboviz/fasta_gff.py.
Moved riboviz/check_fasta_gff.py function into the above.
Added riboviz/tools/extract_cds_codons.py.
 Includes test for G1, G2 example above.

Questions:

It is CDS (coding sequence) codons not all sequence codons?
Should CDS start and stop codons themselves be included?
check_fasta_gff-related questions:

Should padding out sequence via N to length divisable by three be done? If not then how should such sequences be handled? Currently implemented. But, should it be padded at the end of the CDS, after the stop codon? Or before the stop codon?
What about the other check_fasta_gff checks? Not implemented.

Can multiple CDS arise in a single transcript/gene? If so then how should numbering scheme work? Will the NAME attribute be different for each in the GFF? Should we include a CDS index in the TSV file?
What name do you want for the Gene in the TSV file? At present, the sequence ID, from the GFF file, is used e.g. YAL001C_mRNA. Could use attributes["Name"] value i.e. YAL001C_CDS from the GFF file? What if there is no attribute?
Extend to support configurable naming pattern:

RiboViz overloads names to refer to genes, transcripts and coding sequences.
Give genes, transcripts and coding sequences different names.
GFF files record transcript names in the leftmost sequence column.
This also records the names of features.
Example names could include: YAL001C_<transcript-plus-12-minus-15>, or, following FungiDB <GENE>_t0,1,2,3 for transcripts.

Change data input file yeast_codon_pos_i200.RData to alternative file format #194, "the choice to ignore the first 200 codon positions, and genes shorter than 200nt, is hard-coded in both the data-creation and data-analysis code. This should be a parameter."

Does that comment relate to this?

It is CDS (coding sequence) codons not all sequence codons?

Yes.

Should CDS start and stop codons themselves be included?

start - yes.
stop - ideally, that should be a flag. default value yes I think?

check_fasta_gff-related questions:

Should padding out sequence via N to length divisable by three be done? If not then how should such sequences be handled? Currently implemented. But, should it be padded at the end of the CDS, after the stop codon? Or before the stop codon?
No, if there is an incomplete codon, we should not count it. Possibly that could be a flag, but I can't imagine a use-case, unless @shahpr or @lianafaye suggest otherwise?

What about the other check_fasta_gff checks? Not implemented.

Can multiple CDS arise in a single transcript/gene? If so then how should numbering scheme work? Will the NAME attribute be different for each in the GFF? Should we include a CDS index in the TSV file?

What name do you want for the Gene in the TSV file? At present, the sequence ID, from the GFF file, is used e.g. YAL001C_mRNA. Could use attributes["Name"] value i.e. YAL001C_CDS from the GFF file? What if there is no attribute?

Extend to support configurable naming pattern:

RiboViz overloads names to refer to genes, transcripts and coding sequences.
Give genes, transcripts and coding sequences different names.
GFF files record transcript names in the leftmost sequence column.
This also records the names of features.
Example names could include: YAL001C_<transcript-plus-12-minus-15>, or, following FungiDB <GENE>_t0,1,2,3 for transcripts.

Change data input file yeast_codon_pos_i200.RData to alternative file format #194, "the choice to ignore the first 200 codon positions, and genes shorter than 200nt, is hard-coded in both the data-creation and data-analysis code. This should be a parameter."

Does that comment relate to this?

Progress on refactoring check_fasta_gff functionality.
Reports on:

CDS has length not divisible by 3.
Beginning of a CDS does not have a start codon (ATG).
End codon of the CDS is not a stop codon (TAG, TGA, TAA).
Stop codons internal to the CDS.
CDS has no ID or Name attribute.
CDS has non-unique ID attribute.
Multiple CDS for a sequence.

If no feature ID attribute then Name is used in TSV file. If no Name attribute, then the token Undefined is used.
Outputs TSV file with Sequence ID, Feature ID, Issue columns.

Progress at commit f6ce9d0, refactoring check_fasta_gff functionality.

Additionally reports on Sequence defined in GFF file missing in FASTA file.
Added riboviz/test/data/ test files test_fasta_gff_check.gff and test_fasta_gff_check.fasta.
Added tests to riboviz.test.test_fasta_gff

I ran this and it produced the same output as Mike:
python -m riboviz.tools.get_cds_codons_file -f riboviz/test/data/test_fasta_gff_codons.fasta -g riboviz/test/data/test_fasta_gff_codons.gff -c test_cds.txt

head test_cds.txt -n 20
# Created by: RiboViz
# Date: 2020-10-14 14:46:26.835204
# Command-line tool: /homes/ewallac2/riboviz/riboviz/tools/get_cds_codons_file.py
# File: /homes/ewallac2/riboviz/riboviz/fasta_gff.py
# Version: commit f6ce9d043d030ac7a13a02651f34ed3d05b37c08 date 2020-09-16 09:19:35-07:00
Gene	Codon	Pos
YAL001C_CDS	ATG	1
YAL001C_CDS	GCC	2
YAL001C_CDS	CAC	3
YAL001C_CDS	TGT	4
YAL001C_CDS	TAA	5
YAL002C_CDS	ATG	1
YAL002C_CDS	GTA	2
YAL002C_CDS	TCA	3
YAL002C_CDS	GGA	4
YAL002C_CDS	TAG	5
YAL004CSingleCodonCDS_CDS	ATG	1
YAL004CSingleCodonCDS_CDS	AGA	2
YAL004CSingleCodonCDS_CDS	TGA	3
YAL005CMultiCDS_CDS_1	ATG	1

I could also run:
python -m riboviz.tools.check_fasta_gff -f riboviz/test/data/test_fasta_gff_codons.fasta -g riboviz/test/data/test_fasta_gff_codons.gff

This returns:
Created by: RiboViz
Date: 2020-10-14 14:51:29.662039
Command-line tool: /homes/ewallac2/riboviz/riboviz/tools/check_fasta_gff.py
File: /homes/ewallac2/riboviz/riboviz/tools/check_fasta_gff.py
Version: commit f6ce9d043d030ac7a13a02651f34ed3d05b37c08 date 2020-09-16 09:19:35-07:00

Sequence YAL003CMissingGene_mRNA missing in FASTA file
Sequence YAL005CMultiCDS_mRNA has multiple CDS
Sequence YAL007CBadLength_mRNA feature YAL007CBadLength_CDS has length not divisible by 3
Sequence YAL007CBadLength_mRNA feature YAL007CBadLength_CDS doesn't stop at end
Sequence YAL008CNoIdNameAttr_mRNA feature Undefined has no 'ID' or 'Name' attribute
Sequence YAL009CMultiDuplicateCDS_mRNA has multiple CDS
Sequence YAL009CMultiDuplicateCDS_mRNA has non-unique 'ID' attribute YAL009CMultiDuplicateCDS_CDS

TODOs after discussion with @ewallace to answer preceding questions:
get_cds_codons:

Provide feature_format as a parameter, default value {}_CDS.

check_fasta_gff:

Provide a flag to allow which of ID or Name are to be used in output reports if both are defined.
Provide feature_format as a parameter, default value {}_CDS, if a feature name is undefined (i.e. same behaviour as get_cds_codons replacing current Undefined parameter).
For non-unique feature IDs have one output report to cover all the non-unique feature IDs.
Add a Notes column to the output file with additional information e.g. number of duplicates, from the above.
Add the free-text printed to standard output to this Notes column too.
Iterate through FASTA file and check any sequences in FASTA file that have no entries in GFF file.

Get sequence column identifiers the from GFF file.
Get list of sequence names from the FASTA file.
Calculate FASTA - GFF names.

In light of Solve issue with and add e. coli daa example-datasets/12, allow specification of non-canonical start codons. check_fasta_gff needs extended to allow non-canonical start codons as arguments (e.g. GTG or TTG (V or L after translation). It would make the code clearer too if we checked for start codons directly rather than their translations.

TODOs as of 03/11/20:
get_cds_codons:

Provide feature_format as a parameter, default value {}_CDS.

check_fasta_gff:

In light of Solve issue with and add e. coli daa example-datasets/12, allow specification of non-canonical start codons. check_fasta_gff needs extended to allow non-canonical start codons as arguments (e.g. GTG or TTG (V or L after translation). It would make the code clearer too if we checked for start codons directly rather than their translations.
Provide a flag to allow which of ID or Name are to be used in output reports if both are defined.
Provide feature_format as a parameter, default value {}_CDS, if a feature name is undefined (i.e. same behaviour as get_cds_codons replacing current Undefined parameter).
For non-unique feature IDs have one output report to cover all the non-unique feature IDs.
Add a Notes column to the output file with additional information e.g. number of duplicates, from the above.
Add the free-text printed to standard output to this Notes column too.
Iterate through FASTA file and check any sequences in FASTA file that have no entries in GFF file.

Get sequence column identifiers the from GFF file.
Get list of sequence names from the FASTA file.
Calculate FASTA - GFF names.

Provide suitable error messages for missing/empty files.

Completed as of commit c49ff73:
get_cds_codons:

--cds-feature-format <FORMAT> is now supported (default {}_CDS) to format sequence ID if no feature ID or name can be found.

check_fasta_gff:

--feature-format <FORMAT> is now supported (default {}_CDS) to format sequence ID if no feature ID or name can be found.
riboviz.check_fasta_gff reports number of duplicated feature IDs.
TSV now has additional Data column which can be used for JSON-strings with additional data about an issue e.g. number of duplicated feature IDs. Updates to tests and printing output still to be done.

Completed as of commit d13fe54:
check_fasta_gff:

TSV Data column is used for arbitrary data.
Duplicate feature IDs (DuplicateFeatureId) reported for each sequence.
Summary issue (DuplicateFeatureIds) records count of all such duplications.
Sequences defined in the FASTA file that have no related features in the GFF file are reported.
Removed translation of sequences. Sequences are now split into codons and these used for start and stop codon-related checks.

Commit:  77a896b, added https://github.com/riboviz/riboviz/blob/gff-spec-74/docs/user/check-fasta-gff.md.

Completed as of commit 6958b97:

Renamed riboviz.tools.get_cds_codons_file => riboviz.tools.get_cds_codons.
riboviz.tools.get_cds_codons and riboviz.tools.check_fasta_gff allow user to specify which of feature ID or Name are to be used in output reporting if both are defined.

Completed as of commit 2b888b2...
Added missing and empty file handling to riboviz.get_cds_codons and riboviz.check_fasta_gff.
Updated riboviz.check_fasta_gff to:

Record supplementary data about additional issues

NO_START_CODON: actual codon found.
NO_STOP_CODON: actual codon found.
MULTIPLE_CDS:  count of the number of CDSs found.

Support list of allowed start codons, default ['ATG'].

Updated riboviz.tools.check_fasta_gff to support optional --start-codon flag to provide list of allowed start codons e.g. --start-codon ATG AAA. Defaults to ATG. @acope3 please try this for "Solve issue with and add e. coli data, https://github.com/riboviz/example-datasets/issues/12

@ewallace will ask his honours students to test this in the new year.

Reminder! We will ask @3mma-mack and @swinterbourne to test this, as part of adding S. pombe data example-datasets#21.

Reminder! We will ask @3mma-mack and @swinterbourne to test this, as part of adding S. pombe data example-datasets#21.

I'm testing this. It works.
python -m riboviz.tools.check_fasta_gff \
  -f ../example-datasets/fungi/cryptococcus/annotation/H99_10p_up12dwn9_CDS_with_120bputrs.fa \
  -g ../example-datasets/fungi/cryptococcus/annotation/H99_10p_up12dwn9_CDS_120bpL_120bpR.gff3

returns no issues
Running on the issue from https://github.com/riboviz/example-datasets/issues/12
python -m riboviz.tools.check_fasta_gff \
  -f ../example-datasets/bacteria/ecoli/annotations/Ecoli_REL606_CDS_w_25utrs.fa \
  -g ../example-datasets/bacteria/ecoli/annotations/Ecoli_REL606_CDS_w_25utrs.gff3

returns results:
Sequence ECB_02723 feature ECB_02723 has length not divisible by 3
Sequence ECB_02723 feature ECB_02723 doesn't end with a recognised stop codon but with ANN
Sequence ECB_02723 feature ECB_02723 has an internal stop codon
Sequence ECB_00018 feature ECB_00018 doesn't start with a recognised start codon but with GTG
Sequence ECB_00021 feature ECB_00021 doesn't start with a recognised start codon but with GTG
...

Specifying more start codons, ATG/TTG/GTG:
python -m riboviz.tools.check_fasta_gff \
  -f ../example-datasets/bacteria/ecoli/annotations/Ecoli_REL606_CDS_w_25utrs.fa \
  -g ../example-datasets/bacteria/ecoli/annotations/Ecoli_REL606_CDS_w_25utrs.gff3 \
  --start-codon ATG TTG GTG

results in (complete output here):
Created by: RiboViz
Date: 2021-01-16 09:23:19.720064
Command-line tool: /Users/edwardwallace/Repos/riboviz/riboviz/riboviz/tools/check_fasta_gff.py
File: /Users/edwardwallace/Repos/riboviz/riboviz/riboviz/tools/check_fasta_gff.py
Version: commit 6b4cfda7a250898f7d216d1a914d4b6ed9f01909 date 2021-01-12 07:25:44-08:00

Sequence ECB_02723 feature ECB_02723 has length not divisible by 3
Sequence ECB_02723 feature ECB_02723 doesn't end with a recognised stop codon but with ANN
Sequence ECB_02723 feature ECB_02723 has an internal stop codon
Sequence ECB_00526 feature ECB_00526 doesn't start with a recognised start codon but with CTG
Sequence ECB_00820 feature ECB_00820 doesn't start with a recognised start codon but with CTG
Sequence ECB_00835 feature ECB_00835 doesn't start with a recognised start codon but with CTG
Sequence ECB_01432 feature ECB_01432 has an internal stop codon
Sequence ECB_01627 feature ECB_01627 doesn't end with a recognised stop codon but with AGC
Sequence ECB_01907 feature ECB_01907 doesn't start with a recognised start codon but with CTG
Sequence ECB_01909 feature ECB_01909 doesn't start with a recognised start codon but with CTG
Sequence ECB_03779 feature ECB_03779 has an internal stop codon
Sequence ECB_03951 feature ECB_03951 has an internal stop codon

I spot-checked: ECB_01432/fdnG and ECB_03779/fdhF and ECB_03951/fdhF each have a UGA codon which encodes selenocysteine, not stop codons. So this is giving sensible results on real data.
Running on the tiny simulated dataset:
python -m riboviz.tools.check_fasta_gff \
  -f ../example-datasets/simulated/mok/annotation/tiny_2genes_20utrs.fa \
  -g ../example-datasets/simulated/mok/annotation/tiny_2genes_20utrs.gff3

detects no problems.
Running a biologically erroneous alternative on the same dataset:
python -m riboviz.tools.check_fasta_gff \
  -f ../example-datasets/simulated/mok/annotation/tiny_2genes_20utrs.fa \
  -g ../example-datasets/simulated/mok/annotation/tiny_2genes_20utrs.gff3 \
  --start-codon AAA

returns appropriate results (here in full):
Created by: RiboViz
Date: 2021-01-16 09:49:49.650282
Command-line tool: /Users/edwardwallace/Repos/riboviz/riboviz/riboviz/tools/check_fasta_gff.py
File: /Users/edwardwallace/Repos/riboviz/riboviz/riboviz/tools/check_fasta_gff.py
Version: commit 6b4cfda7a250898f7d216d1a914d4b6ed9f01909 date 2021-01-12 07:25:44-08:00

Sequence MAT feature MAT doesn't start with a recognised start codon but with ATG
Sequence MIKE feature MIKE doesn't start with a recognised start codon but with ATG

So the logic of the program seems great.
I have some requests on output formatting that should be easy?

Can we add the parameters to the header, both for the stout and the tsv header? So print

fasta file: FASTA
gff file: GFF
start codon: START_CODON

I'm not sure if any other params are needed?

Could we calculate summary information: number of issues of each type? Something like:

print("Issue\tCount\n")
for (issue_type, issue_data) in issues:
	if issue_type in ISSUE_FORMATS:
		n_issues_of_type = unique(issue_data) # count the number of issues of that type
		print(issue_type + "\t" + n_issues_of_type + "\n")

Then this summary info could be printed in the header in stdout and the .tsv. I am wondering if this could replace writing out features to stdout, or if it's worth wrapping in a verbose flag? The point is it's helpful to have a simple output for files with no issues, reminding the user "here's the issues we checked for and there aren't any".

Lastly, note that this has been refactored, so closing this issue will also close #25.

I'm testing this. It works.
python -m riboviz.tools.check_fasta_gff \
  -f ../example-datasets/fungi/cryptococcus/annotation/H99_10p_up12dwn9_CDS_with_120bputrs.fa \
  -g ../example-datasets/fungi/cryptococcus/annotation/H99_10p_up12dwn9_CDS_120bpL_120bpR.gff3

returns no issues
Running on the issue from https://github.com/riboviz/example-datasets/issues/12
python -m riboviz.tools.check_fasta_gff \
  -f ../example-datasets/bacteria/ecoli/annotations/Ecoli_REL606_CDS_w_25utrs.fa \
  -g ../example-datasets/bacteria/ecoli/annotations/Ecoli_REL606_CDS_w_25utrs.gff3

returns results:
Sequence ECB_02723 feature ECB_02723 has length not divisible by 3
Sequence ECB_02723 feature ECB_02723 doesn't end with a recognised stop codon but with ANN
Sequence ECB_02723 feature ECB_02723 has an internal stop codon
Sequence ECB_00018 feature ECB_00018 doesn't start with a recognised start codon but with GTG
Sequence ECB_00021 feature ECB_00021 doesn't start with a recognised start codon but with GTG
...

Specifying more start codons, ATG/TTG/GTG:
python -m riboviz.tools.check_fasta_gff \
  -f ../example-datasets/bacteria/ecoli/annotations/Ecoli_REL606_CDS_w_25utrs.fa \
  -g ../example-datasets/bacteria/ecoli/annotations/Ecoli_REL606_CDS_w_25utrs.gff3 \
  --start-codon ATG TTG GTG

results in (complete output here):
Created by: RiboViz
Date: 2021-01-16 09:23:19.720064
Command-line tool: /Users/edwardwallace/Repos/riboviz/riboviz/riboviz/tools/check_fasta_gff.py
File: /Users/edwardwallace/Repos/riboviz/riboviz/riboviz/tools/check_fasta_gff.py
Version: commit 6b4cfda7a250898f7d216d1a914d4b6ed9f01909 date 2021-01-12 07:25:44-08:00

Sequence ECB_02723 feature ECB_02723 has length not divisible by 3
Sequence ECB_02723 feature ECB_02723 doesn't end with a recognised stop codon but with ANN
Sequence ECB_02723 feature ECB_02723 has an internal stop codon
Sequence ECB_00526 feature ECB_00526 doesn't start with a recognised start codon but with CTG
Sequence ECB_00820 feature ECB_00820 doesn't start with a recognised start codon but with CTG
Sequence ECB_00835 feature ECB_00835 doesn't start with a recognised start codon but with CTG
Sequence ECB_01432 feature ECB_01432 has an internal stop codon
Sequence ECB_01627 feature ECB_01627 doesn't end with a recognised stop codon but with AGC
Sequence ECB_01907 feature ECB_01907 doesn't start with a recognised start codon but with CTG
Sequence ECB_01909 feature ECB_01909 doesn't start with a recognised start codon but with CTG
Sequence ECB_03779 feature ECB_03779 has an internal stop codon
Sequence ECB_03951 feature ECB_03951 has an internal stop codon

I spot-checked: ECB_01432/fdnG and ECB_03779/fdhF and ECB_03951/fdhF each have a UGA codon which encodes selenocysteine, not stop codons. So this is giving sensible results on real data.
Running on the tiny simulated dataset:
python -m riboviz.tools.check_fasta_gff \
  -f ../example-datasets/simulated/mok/annotation/tiny_2genes_20utrs.fa \
  -g ../example-datasets/simulated/mok/annotation/tiny_2genes_20utrs.gff3

detects no problems.
Running a biologically erroneous alternative on the same dataset:
python -m riboviz.tools.check_fasta_gff \
  -f ../example-datasets/simulated/mok/annotation/tiny_2genes_20utrs.fa \
  -g ../example-datasets/simulated/mok/annotation/tiny_2genes_20utrs.gff3 \
  --start-codon AAA

returns appropriate results (here in full):
Created by: RiboViz
Date: 2021-01-16 09:49:49.650282
Command-line tool: /Users/edwardwallace/Repos/riboviz/riboviz/riboviz/tools/check_fasta_gff.py
File: /Users/edwardwallace/Repos/riboviz/riboviz/riboviz/tools/check_fasta_gff.py
Version: commit 6b4cfda7a250898f7d216d1a914d4b6ed9f01909 date 2021-01-12 07:25:44-08:00

Sequence MAT feature MAT doesn't start with a recognised start codon but with ATG
Sequence MIKE feature MIKE doesn't start with a recognised start codon but with ATG

So the logic of the program seems great.
I have some requests on output formatting that should be easy?

Can we add the parameters to the header, both for the stout and the tsv header? So print

fasta file: FASTA
gff file: GFF
start codon: START_CODON

I'm not sure if any other params are needed?

Could we calculate summary information: number of issues of each type? Something like:

print("Issue\tCount\n")
for (issue_type, issue_data) in issues:
	if issue_type in ISSUE_FORMATS:
		n_issues_of_type = unique(issue_data) # count the number of issues of that type
		print(issue_type + "\t" + n_issues_of_type + "\n")

Then this summary info could be printed in the header in stdout and the .tsv. I am wondering if this could replace writing out features to stdout, or if it's worth wrapping in a verbose flag? The point is it's helpful to have a simple output for files with no issues, reminding the user "here's the issues we checked for and there aren't any".

Lastly, note that this has been refactored, so closing this issue will also close #25.

Changes as of commit a63c24c:

FASTA file, GFF file and start codons printed and added to file header.
Issue counts calculated, printed, added to file header.
Issue details only printed, if a -v verbose flag is provided.

Example with -v:
$ python -m riboviz.tools.check_fasta_gff \
    -f data/yeast_CDS_w_250utrs.fa \
    -g data/yeast_CDS_w_250utrs.gff3 \
    -o check_data_CDS.tsv -v
...

Issue counts calculated, printed, added to file header.
Issue details only printed, if a -v verbose flag is provided.

Example with -v:
$ python -m riboviz.tools.check_fasta_gff \
    -f data/yeast_CDS_w_250utrs.fa \
    -g data/yeast_CDS_w_250utrs.gff3 \
    -o check_data_CDS.tsv -v
...

Is it possible to include in the summary the issues that were checked for with zero counts?
For example, editing the current output:
$ python -m riboviz.tools.check_fasta_gff \
>   -f ../example-datasets/simulated/mok/annotation/tiny_2genes_20utrs.fa \
>   -g ../example-datasets/simulated/mok/annotation/tiny_2genes_20utrs.gff3
Created by: RiboViz
Date: 2021-01-19 18:20:53.048658
Command-line tool: /exports/eddie3_homes_local/ewallac2/riboviz/riboviz/riboviz/tools/check_fasta_gff.py
File: /exports/eddie3_homes_local/ewallac2/riboviz/riboviz/riboviz/tools/check_fasta_gff.py
Version: commit a63c24cfb46fd21e73cc4bbc29d5ea31b2f01fde date 2021-01-19 05:17:54-08:00

Configuration:
fasta_file	../example-datasets/simulated/mok/annotation/tiny_2genes_20utrs.fa
gff_file	../example-datasets/simulated/mok/annotation/tiny_2genes_20utrs.gff3
start_codons	['ATG']

Issue summary:
Issue	Count
NoStartCodon	0
OtherIssue	0

My point is that an output that says "we checked this and there are no problems of this type" is more informative that an output that says " ".

I tested this and it works delightfully.
Is it possible to include in the summary the issues that were checked for with zero counts?
For example, editing the current output:
$ python -m riboviz.tools.check_fasta_gff \
>   -f ../example-datasets/simulated/mok/annotation/tiny_2genes_20utrs.fa \
>   -g ../example-datasets/simulated/mok/annotation/tiny_2genes_20utrs.gff3
Created by: RiboViz
Date: 2021-01-19 18:20:53.048658
Command-line tool: /exports/eddie3_homes_local/ewallac2/riboviz/riboviz/riboviz/tools/check_fasta_gff.py
File: /exports/eddie3_homes_local/ewallac2/riboviz/riboviz/riboviz/tools/check_fasta_gff.py
Version: commit a63c24cfb46fd21e73cc4bbc29d5ea31b2f01fde date 2021-01-19 05:17:54-08:00

Configuration:
fasta_file	../example-datasets/simulated/mok/annotation/tiny_2genes_20utrs.fa
gff_file	../example-datasets/simulated/mok/annotation/tiny_2genes_20utrs.gff3
start_codons	['ATG']

Issue summary:
Issue	Count
NoStartCodon	0
OtherIssue	0

My point is that an output that says "we checked this and there are no problems of this type" is more informative that an output that says " ".

Commit: 666d1f7. Prints ordered list of counts for all issues, including those which were zero.
$ python -m riboviz.tools.check_fasta_gff \
    -f data/yeast_CDS_w_250utrs.fa \
    -g data/yeast_CDS_w_250utrs.gff3 \
    -o check_data_CDS.tsv -v
...
Configuration:
fasta_file	data/yeast_CDS_w_250utrs.fa
gff_file	data/yeast_CDS_w_250utrs.gff3
start_codons	['ATG']

Issue summary:
Issue	Count
InternalStopCodon	17
NoStartCodon	1
MultipleCDS	0
SequenceNotInGFF	0
IncompleteFeature	0
NoStopCodon	0
DuplicateFeatureId	0
NoIdName	0
DuplicateFeatureIds	0
SequenceNotInFASTA	0
...

Commit: 666d1f7. Prints ordered list of counts for all issues, including those which were zero.
$ python -m riboviz.tools.check_fasta_gff \
    -f data/yeast_CDS_w_250utrs.fa \
    -g data/yeast_CDS_w_250utrs.gff3 \
    -o check_data_CDS.tsv -v
...
Configuration:
fasta_file	data/yeast_CDS_w_250utrs.fa
gff_file	data/yeast_CDS_w_250utrs.gff3
start_codons	['ATG']

Issue summary:
Issue	Count
InternalStopCodon	17
NoStartCodon	1
MultipleCDS	0
SequenceNotInGFF	0
IncompleteFeature	0
NoStopCodon	0
DuplicateFeatureId	0
NoIdName	0
DuplicateFeatureIds	0
SequenceNotInFASTA	0
...

Beautiful!

Beautiful!


TDR-2292 Rename closed_for to closed and open_on_transfer to open

In the FilePropertyValues table, replace the PropertyValue content for rows where PropertyName is 'ClosureType'
'closed_for' --> 'Closed'
'open_on_transfer' --> 'Open'

Hi Emma, could you rename this PR and migration script name to something more descriptive like "rename closed_for to closed and open_on_transfer to open"? Just so that in the future we can easily find it if we need to

Good point. Script and pull request now renamed.

@juliuswelby @pashworth-tna All the commits are unsigned, so won't be able to merge this PR.
See instructions here about setting up signed commits: https://github.com/nationalarchives/tdr-dev-documentation/blob/master/manual/development-setup/signed-commits.md
Would suggest the easiest approach is to create new branch and recommit the changes to this new branch once signed commits is set up.


Mayo/Egg white/Cheese potentially too hard to craft

I thought it would be easier to implement the mayo & egg white system but with how it is now you can only craft one mayo/egg white at a time, and have to manually click in each item/out. The way it is right now is conceptually fun but practically annoying.

#8 has given me a bit of a brain-blast. 2 bottles + 1 egg = 1 raw egg yolk & 1 raw egg white (both bottled). Cook each of those separately in a furnace and you get mayo & meringue.
The egg yolk would be the primary outcome, and the egg white would stay behind in the crafting table. This is something that the milk bucket does when a cake is made. Should be easy to implement.

#8 has given me a bit of a brain-blast. 2 bottles + 1 egg = 1 raw egg yolk & 1 raw egg white (both bottled). Cook each of those separately in a furnace and you get mayo & meringue.
The egg yolk would be the primary outcome, and the egg white would stay behind in the crafting table. This is something that the milk bucket does when a cake is made. Should be easy to implement.

brain-blast for cheese - smelt with the bucket, cheese stays in bucket, get cheese in bucket. Cheese in bucket is both edible, craftable, and consistent with milk bucket in that either will return the bucket to an empty state. This also opens the door for things like cheesy nachos and stuff

brain-blast for cheese - smelt with the bucket, cheese stays in bucket, get cheese in bucket. Cheese in bucket is both edible, craftable, and consistent with milk bucket in that either will return the bucket to an empty state. This also opens the door for things like cheesy nachos and stuff

Meant to close #8. 🤦♂

Meant to close #8. 🤦♂


Got error while compiling.

I got this error. I have using vue 2.5.16 and vue-template-compiler 2.5.16
[Vue warn]: Failed to mount component: template or render function not defined.

found in

---> <Anonymous>
       <Root>

Your question is very vague. The reason why vue is warning you is because it can't find the root id in the html document, so what you need to do is the mount method e.g
import Index from '../Index.vue;
new Vue(Index).**mount**('#root');
I'm sure this project already have these things pre done, just follow the examples in pages folder

Try yarn, so it can use the tested and locked versions of all packages.
If you want to use the latest of everything you'll have to do some testing on your end.

After release of vue-loader v13.0.0 we need to add default in require so i have fixed by adding the default in
import Loading from './Loading'

// Load layout components dynamically.
const requireContext = require.context('~/layouts', false, /.*\.vue$/)

const layouts = requireContext.keys()
  .map(file =>
    [file.replace(/(^.\/)|(\.vue$)/g, ''), requireContext(file)]
  )
  .reduce((components, [name, component]) => {
    components[name] = component.default
    return components
  }, {})

Fixed with 4.1, always uses esModule: true


System.IdentityModel.Tokens.Jwt & Microsoft.IdentityModel.JsonWebTokens 5.5.0 are unsigned

I have dropped System.IdentityModel.Tokens.Jwt into a project I am working on, and the latest default version was 5.5.0. When I pushed to our company's main repo (Git on bitbucket) I was told that the build failed with and error, which included the following info:-
...
Installing Microsoft.IdentityModel.JsonWebTokens 5.5.0.
[12:04:35]  Installing System.IdentityModel.Tokens.Jwt 5.5.0.
[12:04:35]C:\BuildAgent\work...[proj path].csproj : error NU3004: Package 'Microsoft.IdentityModel.JsonWebTokens 5.5.0' from source 'https://api.nuget.org/v3/index.json': This repository indicated that all its packages are repository signed; however, this package is unsigned.
[12:04:35]C:\BuildAgent\work...[proj path].csproj : error NU3004: Package 'System.IdentityModel.Tokens.Jwt 5.5.0' from source 'https://api.nuget.org/v3/index.json': This repository indicated that all its packages are repository signed; however, this package is unsigned.
...
I looked in another company project, which referenced IdentityServer4.AspNetIdentity version 2.5.0, which didn't have this problem. I could see the following chain of dependency:
<EMAIL_ADDRESS>->
<EMAIL_ADDRESS>->
<EMAIL_ADDRESS>->
<EMAIL_ADDRESS>I replaced the nuget reference to System.IdentityModel.Tokens.Jwt to version 5.3.0 and this resolved the problem. Note that this also reverted Microsoft.IdentityModel.JsonWebTokens to 5.3.0.
This proves to me that the 5.5.0 version of these components do not have a valid signature if the git repo is set to declare that all nuget packages are signed.
Note that some related packages, Microsoft.IdentityModel.Logging and Microsoft.IdentityModel.Tokens appear to be OK at 5.5.0 without any error occurring.

Unfortunately I don't have admin rights to the setup of our company's GIT repo however as stated above, if it is stated in the repo that all packages are signed then 5.5.0 of these components are likely to break any project using them.

@williamBurgeson nuget.exe reports the signatures are fine.
E:\releases\Wilson\5.5.0\artifacts>\releases\nuget\nuget.exe verify -signature Microsoft.IdentityModel.JsonWebTokens.5.5.0.nupkg
Verifying Microsoft.IdentityModel.JsonWebTokens.5.5.0
E:\releases\Wilson\5.5.0\artifacts\Microsoft.IdentityModel.JsonWebTokens.5.5.0.nupkg
Signature Hash Algorithm: SHA256
Timestamp: 6/24/2019 1:10:48 PM
Verifying author primary signature's timestamp with timestamping service certificate:
Subject Name: CN=Symantec SHA256 TimeStamping Signer - G3, OU=Symantec Trust Network, O=Symantec Corporation, C=US
SHA1 hash: A9A4121063D71D48E8529A4681DE803E3E7954B0
SHA256 hash: C474CE76007D02394E0DA5E4DE7C14C680F9E282013CFEF653EF5DB71FDF61F8
Issued by: CN=Symantec SHA256 TimeStamping CA, OU=Symantec Trust Network, O=Symantec Corporation, C=US
Valid from: 12/22/2017 4:00:00 PM to 3/22/2029 4:59:59 PM
Signature type: Author
Verifying the author primary signature with certificate:
Subject Name: CN=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
SHA1 hash: F404000FB11E61F446529981C7059A76C061631E
SHA256 hash: 3F9001EA83C560D712C24CF213C3D312CB3BFF51EE89435D3430BD06B5D0EECE
Issued by: CN=DigiCert SHA2 Assured ID Code Signing CA, OU=www.digicert.com, O=DigiCert Inc, C=US
Valid from: 2/25/2018 4:00:00 PM to 1/27/2021 4:00:00 AM
Successfully verified package 'Microsoft.IdentityModel.JsonWebTokens.5.5.0'.
E:\releases\Wilson\5.5.0\artifacts>\releases\nuget\nuget.exe verify -signature System.IdentityModel.Tokens.Jwt.5.5.0.nupkg
Verifying System.IdentityModel.Tokens.Jwt.5.5.0
E:\releases\Wilson\5.5.0\artifacts\System.IdentityModel.Tokens.Jwt.5.5.0.nupkg
Signature Hash Algorithm: SHA256
Timestamp: 6/24/2019 1:10:42 PM
Verifying author primary signature's timestamp with timestamping service certificate:
Subject Name: CN=Symantec SHA256 TimeStamping Signer - G3, OU=Symantec Trust Network, O=Symantec Corporation, C=US
SHA1 hash: A9A4121063D71D48E8529A4681DE803E3E7954B0
SHA256 hash: C474CE76007D02394E0DA5E4DE7C14C680F9E282013CFEF653EF5DB71FDF61F8
Issued by: CN=Symantec SHA256 TimeStamping CA, OU=Symantec Trust Network, O=Symantec Corporation, C=US
Valid from: 12/22/2017 4:00:00 PM to 3/22/2029 4:59:59 PM
Signature type: Author
Verifying the author primary signature with certificate:
Subject Name: CN=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
SHA1 hash: F404000FB11E61F446529981C7059A76C061631E
SHA256 hash: 3F9001EA83C560D712C24CF213C3D312CB3BFF51EE89435D3430BD06B5D0EECE
Issued by: CN=DigiCert SHA2 Assured ID Code Signing CA, OU=www.digicert.com, O=DigiCert Inc, C=US
Valid from: 2/25/2018 4:00:00 PM to 1/27/2021 4:00:00 AM
Successfully verified package 'System.IdentityModel.Tokens.Jwt.5.5.0'.
E:\releases\Wilson\5.5.0\artifacts>\releases\nuget\nuget.exe verify -signature Microsoft.IdentityModel.Tokens.5.5.0.nupkg
Verifying Microsoft.IdentityModel.Tokens.5.5.0
E:\releases\Wilson\5.5.0\artifacts\Microsoft.IdentityModel.Tokens.5.5.0.nupkg
Signature Hash Algorithm: SHA256
Timestamp: 6/24/2019 1:10:43 PM
Verifying author primary signature's timestamp with timestamping service certificate:
Subject Name: CN=Symantec SHA256 TimeStamping Signer - G3, OU=Symantec Trust Network, O=Symantec Corporation, C=US
SHA1 hash: A9A4121063D71D48E8529A4681DE803E3E7954B0
SHA256 hash: C474CE76007D02394E0DA5E4DE7C14C680F9E282013CFEF653EF5DB71FDF61F8
Issued by: CN=Symantec SHA256 TimeStamping CA, OU=Symantec Trust Network, O=Symantec Corporation, C=US
Valid from: 12/22/2017 4:00:00 PM to 3/22/2029 4:59:59 PM
Signature type: Author
Verifying the author primary signature with certificate:
Subject Name: CN=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
SHA1 hash: F404000FB11E61F446529981C7059A76C061631E
SHA256 hash: 3F9001EA83C560D712C24CF213C3D312CB3BFF51EE89435D3430BD06B5D0EECE
Issued by: CN=DigiCert SHA2 Assured ID Code Signing CA, OU=www.digicert.com, O=DigiCert Inc, C=US
Valid from: 2/25/2018 4:00:00 PM to 1/27/2021 4:00:00 AM
Successfully verified package 'Microsoft.IdentityModel.Tokens.5.5.0'.
E:\releases\Wilson\5.5.0\artifacts>\releases\nuget\nuget.exe verify -signature Microsoft.IdentityModel.Logging.5.5.0.nupkg
Verifying Microsoft.IdentityModel.Logging.5.5.0
E:\releases\Wilson\5.5.0\artifacts\Microsoft.IdentityModel.Logging.5.5.0.nupkg
Signature Hash Algorithm: SHA256
Timestamp: 6/24/2019 1:10:43 PM
Verifying author primary signature's timestamp with timestamping service certificate:
Subject Name: CN=Symantec SHA256 TimeStamping Signer - G3, OU=Symantec Trust Network, O=Symantec Corporation, C=US
SHA1 hash: A9A4121063D71D48E8529A4681DE803E3E7954B0
SHA256 hash: C474CE76007D02394E0DA5E4DE7C14C680F9E282013CFEF653EF5DB71FDF61F8
Issued by: CN=Symantec SHA256 TimeStamping CA, OU=Symantec Trust Network, O=Symantec Corporation, C=US
Valid from: 12/22/2017 4:00:00 PM to 3/22/2029 4:59:59 PM
Signature type: Author
Verifying the author primary signature with certificate:
Subject Name: CN=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
SHA1 hash: F404000FB11E61F446529981C7059A76C061631E
SHA256 hash: 3F9001EA83C560D712C24CF213C3D312CB3BFF51EE89435D3430BD06B5D0EECE
Issued by: CN=DigiCert SHA2 Assured ID Code Signing CA, OU=www.digicert.com, O=DigiCert Inc, C=US
Valid from: 2/25/2018 4:00:00 PM to 1/27/2021 4:00:00 AM
Successfully verified package 'Microsoft.IdentityModel.Logging.5.5.0'.
Please reopen if you can see what the issue is.

I'll have a chat with my boss to see what exactly the offending settings would be at our end, maybe it's an issue with git, one of its associated buld dependencies or our build engine, or just that we have misconfigured something - but I can confirm that 5.5.0 broke the build and 5.3.0 was ok

@williamBurgeson I had reviewed the 5.3.0 packages and verified the signing certs were the same.
I encourage you to find the root cause we may have an issue that we need to fix.