Information disappeared

My preferred work title disappeared and had to be reentered even though it had appeared in "blue" before which would indicate that it had been successfully added.

can you list the steps you did, or give more information about when the preferred work title disappeared?

I typed in a title and hit Enter. The title appeared in blue in the light blue box. At a later point, after I had added further information to other fields, I noticed the title had disappeared, and I don't believe it was in the underlying rdf. This was also the case, by the way, with my original answer to question 1; it too had disappeared. In the current iteration, however, they have persisted. I should point out, however, that two of my additional contributors and their corresponding relationship designators entered last night have since disappeared from the interface but not from the underlying rdf: the imaginary editor and composer, "Passacaglia, Emilio" and the contributor, Jane Austen (entered as a URL: http://id.loc.gov/authorities/names/n79032879). In the rdf they are alive and well.
This was the rdf which has remained stable:
<> http://fakepropertyforusertesting1 "If it's working correctly, you should see a different button get highlighted every time you press the tab key."@en .
<> http://fakepropertyforusertesting1 "Oui, c'est Très  bien."@fr .
<> http://fakepropertyforusertesting2 "Thanks for testing!"@en .
<> http://id.loc.gov/ontologies/bibframe/contribution _:c14n16 .
<> http://id.loc.gov/ontologies/bibframe/contribution _:c14n17 .
<> http://id.loc.gov/ontologies/bibframe/contribution _:c14n3 .
<> http://id.loc.gov/ontologies/bibframe/hasInstance _:c14n14 .
<> http://id.loc.gov/ontologies/bibframe/subject http://id.loc.gov/authorities/subjects/sh85096148 .
<> http://id.loc.gov/ontologies/bibframe/title _:c14n21 .
<> http://sinopia.io/vocabulary/hasResourceTemplate "Harvard:rt:test:userfeedback" .
<> http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://id.loc.gov/ontologies/bibframe/Work .
_:c14n0 http://sinopia.io/vocabulary/hasResourceTemplate "sinopia:resourceTemplate:bf2:Extent" .
_:c14n0 http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://id.loc.gov/ontologies/bibframe/Extent .
_:c14n0 http://www.w3.org/2000/01/rdf-schema#label "50 pages"@en .
_:c14n1 http://id.loc.gov/ontologies/bibframe/heldBy http://id.loc.gov/vocabulary/organizations/cst .
_:c14n1 http://id.loc.gov/ontologies/bibframe/identifiedBy _:c14n22 .
_:c14n1 http://id.loc.gov/ontologies/bibframe/identifiedBy _:c14n5 .
_:c14n1 http://sinopia.io/vocabulary/hasResourceTemplate "stanford:RT:BF2:Monograph:Item" .
_:c14n1 http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://id.loc.gov/ontologies/bibframe/Item .
_:c14n10 http://id.loc.gov/ontologies/bibframe/source _:c14n13 .
_:c14n10 http://sinopia.io/vocabulary/hasResourceTemplate "stanford:RT:BF2:Identifiers:Local" .
_:c14n10 http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://id.loc.gov/ontologies/bibframe/Local .
_:c14n10 http://www.w3.org/1999/02/22-rdf-syntax-ns#value "1232498789"@en .
_:c14n11 http://id.loc.gov/ontologies/bibframe/qualifier "paperback"@en .
_:c14n11 http://id.loc.gov/ontologies/bibframe/status "incorrect"@en .
_:c14n11 http://sinopia.io/vocabulary/hasResourceTemplate "stanford:RT:BF2:Identifiers:ISBN" .
_:c14n11 http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://id.loc.gov/ontologies/bibframe/Isbn .
_:c14n11 http://www.w3.org/1999/02/22-rdf-syntax-ns#value "1628111011"@en .
_:c14n12 http://sinopia.io/vocabulary/hasResourceTemplate "lc:RT:bf2:Agent:Role" .
_:c14n12 http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://id.loc.gov/ontologies/bibframe/Role .
_:c14n12 http://www.w3.org/2000/01/rdf-schema#label "composer"@en .
_:c14n13 http://sinopia.io/vocabulary/hasResourceTemplate "stanford:RT:BF2:Identifiers:Source" .
_:c14n13 http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://id.loc.gov/ontologies/bibframe/Source .
_:c14n13 http://www.w3.org/2000/01/rdf-schema#label "Alma control number"@en .
_:c14n14 http://id.loc.gov/ontologies/bibframe/carrier http://id.loc.gov/vocabulary/carriers/nc .
_:c14n14 http://id.loc.gov/ontologies/bibframe/copyrightDate "2015"@en .
_:c14n14 http://id.loc.gov/ontologies/bibframe/dimensions "25 cm"@en .
_:c14n14 http://id.loc.gov/ontologies/bibframe/editionStatement "8th edition, revised"@en .
_:c14n14 http://id.loc.gov/ontologies/bibframe/extent _:c14n0 .
_:c14n14 http://id.loc.gov/ontologies/bibframe/hasItem _:c14n1 .
_:c14n14 http://id.loc.gov/ontologies/bibframe/identifiedBy _:c14n10 .
_:c14n14 http://id.loc.gov/ontologies/bibframe/identifiedBy _:c14n11 .
_:c14n14 http://id.loc.gov/ontologies/bibframe/identifiedBy _:c14n18 .
_:c14n14 http://id.loc.gov/ontologies/bibframe/identifiedBy _:c14n20 .
_:c14n14 http://id.loc.gov/ontologies/bibframe/issuance http://id.loc.gov/vocabulary/issuance/mono .
_:c14n14 http://id.loc.gov/ontologies/bibframe/media http://id.loc.gov/vocabulary/mediaTypes/n .
_:c14n14 http://id.loc.gov/ontologies/bibframe/provisionActivity _:c14n7 .
_:c14n14 http://id.loc.gov/ontologies/bibframe/provisionActivityStatement "Boston : PBS Press, 2015."@en .
_:c14n14 http://id.loc.gov/ontologies/bibframe/responsibilityStatement "Madonna."@en .
_:c14n14 http://id.loc.gov/ontologies/bibframe/title _:c14n15 .
_:c14n14 http://sinopia.io/vocabulary/hasResourceTemplate "Harvard:rt:test:userfeedback:instance" .
_:c14n14 http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://id.loc.gov/ontologies/bibframe/Instance .
_:c14n15 http://id.loc.gov/ontologies/bibframe/mainTitle "The wizard of OZ revisted"@en .
_:c14n15 http://id.loc.gov/ontologies/bibframe/note _:c14n2 .
_:c14n15 http://id.loc.gov/ontologies/bibframe/partName "Dorothy's shoes"@en .
_:c14n15 http://id.loc.gov/ontologies/bibframe/partNumber "part one"@en .
_:c14n15 http://id.loc.gov/ontologies/bibframe/subtitle "a series of compendia"@en .
_:c14n15 http://sinopia.io/vocabulary/hasResourceTemplate "sinopia:resourceTemplate:bf2:Title" .
_:c14n15 http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://id.loc.gov/ontologies/bibframe/Title .
_:c14n16 http://id.loc.gov/ontologies/bibframe/agent "Passacaglia, Emilio" .
_:c14n16 http://id.loc.gov/ontologies/bibframe/role _:c14n12 .
_:c14n16 http://id.loc.gov/ontologies/bibframe/role _:c14n4 .
_:c14n16 http://sinopia.io/vocabulary/hasResourceTemplate "sinopia:RT:userfeedback:Agents:Contribution" .
_:c14n16 http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://id.loc.gov/ontologies/bibframe/Contribution .
_:c14n17 http://id.loc.gov/ontologies/bibframe/agent http://id.loc.gov/authorities/names/n79089462 .
_:c14n17 http://id.loc.gov/ontologies/bibframe/role _:c14n19 .
_:c14n17 http://sinopia.io/vocabulary/hasResourceTemplate "sinopia:RT:userfeedback:Agents:Contribution" .
_:c14n17 http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://id.loc.gov/ontologies/bibframe/Contribution .
_:c14n18 http://id.loc.gov/ontologies/bibframe/qualifier "(paperback)"@en .
_:c14n18 http://sinopia.io/vocabulary/hasResourceTemplate "stanford:RT:BF2:Identifiers:ISBN" .
_:c14n18 http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://id.loc.gov/ontologies/bibframe/Isbn .
_:c14n18 http://www.w3.org/1999/02/22-rdf-syntax-ns#value "9781628111019"@en .
_:c14n19 http://sinopia.io/vocabulary/hasResourceTemplate "lc:RT:bf2:Agent:Role" .
_:c14n19 http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://id.loc.gov/ontologies/bibframe/Role .
_:c14n19 http://www.w3.org/2000/01/rdf-schema#label "cartographer"@en .
_:c14n2 http://sinopia.io/vocabulary/hasResourceTemplate "sinopia:resourceTemplate:bf2:Title:Note" .
_:c14n2 http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://id.loc.gov/ontologies/bibframe/Note .
_:c14n2 http://www.w3.org/2000/01/rdf-schema#label "Love me some Lollipop Guild!"@en .
_:c14n20 http://id.loc.gov/ontologies/bibframe/source _:c14n8 .
_:c14n20 http://sinopia.io/vocabulary/hasResourceTemplate "stanford:RT:BF2:Identifiers:Local" .
_:c14n20 http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://id.loc.gov/ontologies/bibframe/Local .
_:c14n20 http://www.w3.org/1999/02/22-rdf-syntax-ns#value "9380928309"@en .
_:c14n21 http://id.loc.gov/ontologies/bibframe/mainTitle "Test title qui est Très bien."@fr .
_:c14n21 http://sinopia.io/vocabulary/hasResourceTemplate "Harvard:rt:test:userfeedback:worktitle" .
_:c14n21 http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://id.loc.gov/ontologies/bibframe/Title .
_:c14n22 http://sinopia.io/vocabulary/hasResourceTemplate "stanford:RT:BF2:Identifiers:Barcode" .
_:c14n22 http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://id.loc.gov/ontologies/bibframe/Barcode .
_:c14n22 http://www.w3.org/1999/02/22-rdf-syntax-ns#value "alma19879"@en .
_:c14n3 http://id.loc.gov/ontologies/bibframe/agent http://id.loc.gov/authorities/names/n79032879 .
_:c14n3 http://sinopia.io/vocabulary/hasResourceTemplate "sinopia:RT:userfeedback:Agents:Contribution" .
_:c14n3 http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://id.loc.gov/ontologies/bibframe/Contribution .
_:c14n4 http://sinopia.io/vocabulary/hasResourceTemplate "lc:RT:bf2:Agent:Role" .
_:c14n4 http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://id.loc.gov/ontologies/bibframe/Role .
_:c14n4 http://www.w3.org/2000/01/rdf-schema#label "editor"@en .
_:c14n5 http://sinopia.io/vocabulary/hasResourceTemplate "stanford:RT:BF2:Identifiers:LC" .
_:c14n5 http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://id.loc.gov/ontologies/bibframe/ShelfMarkLcc .
_:c14n5 http://www.w3.org/1999/02/22-rdf-syntax-ns#value "PN1503 .B55 2015x"@en .
_:c14n6 http://sinopia.io/vocabulary/hasResourceTemplate "stanford:RT:BF2:ProvisionName" .
_:c14n6 http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://id.loc.gov/ontologies/bibframe/Agent .
_:c14n6 http://www.w3.org/2000/01/rdf-schema#label "PBS Press"@en .
_:c14n7 http://id.loc.gov/ontologies/bibframe/agent _:c14n6 .
_:c14n7 http://id.loc.gov/ontologies/bibframe/date "2015"@en .
_:c14n7 http://id.loc.gov/ontologies/bibframe/place _:c14n9 .
_:c14n7 http://sinopia.io/vocabulary/hasResourceTemplate "stanford:RT:BF2:PublicationInfo" .
_:c14n7 http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://id.loc.gov/ontologies/bibframe/Publication .
_:c14n8 http://sinopia.io/vocabulary/hasResourceTemplate "stanford:RT:BF2:Identifiers:Source" .
_:c14n8 http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://id.loc.gov/ontologies/bibframe/Source .
_:c14n8 http://www.w3.org/2000/01/rdf-schema#label "OCLC control number"@en .
_:c14n9 http://sinopia.io/vocabulary/hasResourceTemplate "stanford:RT:BF2:ProvisionPlace" .
_:c14n9 http://www.w3.org/1999/02/22-rdf-syntax-ns#type http://id.loc.gov/ontologies/bibframe/Place .
_:c14n9 http://www.w3.org/2000/01/rdf-schema#label "Boston"@en .

(Contributors opened). I have to say the interface is very confusing with regard to what contributions, agents, and roles, are open or closed (and can be removed) versus those that are empty and can be added. It's very easy to think that data has been lost when it hasn't been. We need to make sure that people don't add data a second time because of the interface.

thanks for the feedback. marking this as "Design Needed" so that UX designer can look at it if we work on this in the next work cycle. Maybe some indication that a collapsed field has data in it?

This will be addressed by #1300, so closing this ticket. If in the future, it is still a problem, please re-ticket.

I don't see where in #1300 there is a design change that would indicate that user-entered info is present in a collapsed section

duplicate of #1249


Cant install

I honestly tried everything but it always seem to fail to install dependencies or get from git. Any ideas?

Please explain what you're trying to accomplish. To use Marzipano as a library, you can download a pre-built version from our website, or install it with npm. You don't need to check out the git repository unless you want to make changes to the source code.

It was the not script it was my computer. No really sure but here are the
two ways I tried.

Download the files from git and unzip in a folder. Then run NPM install.
Lots of errors. Did not install.
Tried to pull from Git, but every time the same issues.

Finally it seems to install but I did the NPM install from inside the Git
CMD rather than normal CMD and that worked :)
Amazing plugin.
I am guessing the creator is not part of the GIt repository, am I
correct.
On Sat, Nov 12, 2016 at 11:02 PM, Tiago Quelhas<EMAIL_ADDRESS>wrote:

Please explain what you're trying to accomplish. To use Marzipano as a
library, you can download a pre-built version from our website, or install
it with npm. You don't need to check out the git repository unless you want
to make changes to the source code.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/google/marzipano/issues/51#issuecomment-260165641,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHZub6OIU6BeI4-MHz4ycs5HbCjPDiuMks5q9ou8gaJpZM4KwmgI
.

For future reference, the issues when running the scripts under cmd.exe may be related to bug #3.
The Marzipano Tool (I'm guessing that's what you meant by "creator") isn't part of the open source release, so you won't find the source code for it in the repository.


Change (id) to (instancetype)

This library is exactly what I was looking for. Nice job. I noticed that most of the initializers returned id though instead of the recommended instancetype. This PR just changes that. Tests passed locally.

Looks like there is a Travis config issue.
$ brew install xctool
Error: xctool-0.2.1 already installed
To install this version, first `brew unlink xctool'
The command "brew install xctool" failed and exited with 1 during .

Don't worry, I'll take a look. There's a long time since this was updated, maybe something is deprecated. Since you only changed the methods' contracts, there's no reason why these tests would fail.


How do you use hooks?

I see hooks being described in general terms under "Concepts" in the documentation, and I see particular hooks being described in detail under "API" in the documentation, but I see no example of how they would be used.  Do you have any examples of how they get "hooked" into a DAG?  Thanks.

A hook is basically a connection to some system. So an operator and a sensor would use a "hook"  to connect to hive to execute query "X", which effects some update (operator) or checks on the results of some rows (sensor). I just finished a pull request for a new http operator, which can talk to many HTTP-based systems. An operator would usually use POST/PUT or DELETE (through the hook) and the sensor would usually use a "GET" request.
Most hooks require a connection declaration. Rather than encoding that into every hook itself, there's a connections tab on the admin menu (of the webserver), where you can configure what ops and sensors connect to. So that way you keep username/pw and host configs out of the way of code. In the DAG you can then still refer to specific connection-id's.

It abstracts 2 components of any external system:

The connection information (hostname, login, password, default schema, ...)
The API, making a more pythonesque / intuitive / homogeneous API on top of external systems API

These hooks are used by operators to interact with the external system. For example, HiveToMySqlTransfer and PostgresToMySqlTransfer would both use the MySql hook function that allows for loading data in MySql.
You can also use these hooks in PythonOperator, or interactively, say in iPython or Juniper to interact with your external systems without having to log in and use their often too complex API.
Here's an example of how easy it is to get some data out of Presto and into a Pandas dataframe:
from airflow.hooks import PrestoHook
h = PrestoHook()
df = h.get_pandas_df("SELECT * FROM users LIMIT 100")
...

Cool, cool.  So, how would I go about using the S3Hook object and its load_file operation to load a local file up to an S3 bucket within a DAG?  The basic flow would be, use a BashOperator to fetch data from a remote-endpoint (using curl or an API client or what-have-you), copy that file up to AWS S3, and then use another BashOperator to bulk-COPY the data into AWS Redshift.

Well consider the fact that you may have multiple workers and that each task may take place on any worker. For that reason Airflow tasks should only use the local filesystem within the task execution scope and cleanup after themselves.
If you want to leverage the hooks, your best bet is to do it within a PythonOperator. You write a python function that say uses the requests library to fetch your file, push that to S3 using the S3Hook, then probably another task to run load from S3 to redshift (BashOperator? PostgresOperator?).

Thanks.  I'm not even sure if I want to leverage hooks.  I just know that I want to scribble a file onto a worker's local file-system, copy it to S3, and do a Redshift bulk COPY.  The first is already handled with a BashOperator.  The third can be handled either with a BashOperator or a PostgresOperator.  For copying the file to S3, I figured that since Airflow already has a "S3-thing" (i.e., an S3Hook) then I'd try to use it.  But, it seems I ought to be able to use a BashOperator for that as well, along with the AWS CLI.  That might be easier than using the PythonOperator in the way that you describe.

Step 1-2 need to happen atomically if you use the local file system and you want your DAG to work on a multi-worker environment, unless you use a mountpoint that all workers have mounted.
PythonOperator leveraging hooks leads to better looking, more reusable code than BashOperator in general, but whatever works for you!

@mistercrunch

I want to use the S3KeySensor to check if some files exist in the aws S3, when I config the connection of  S3KeySensor ,I don't know how should I config the host && schema && login && password since I needn't use these params when I access aws s3 using aws cli.
Can you give me an axample ?

@Thomasmisfit , you could use the following and use a config file. We support boto, s3cmd and aws credential files.

Or you could also enter the access and secret key in the extra field. The keys have to be named aws_access_key_id and aws_secret_access_key.


Updated K8s version

PR Title:
Update Kubernetes dependencies to v0.30.0
Description:
This PR updates the following Kubernetes dependencies to v0.30.0 to align with the latest K8s v1.30 release:

k8s.io/api from v0.29.1 to v0.30.0
k8s.io/apiextensions-apiserver from v0.29.1 to v0.30.0
k8s.io/apimachinery from v0.29.1 to v0.30.0
k8s.io/client-go from v0.29.1 to v0.30.0

These updates are necessary to leverage the latest Kubernetes features and improvements, ensure compatibility with the Kubernetes v1.30 cluster, and maintain stability and security in our project.
Related Issue:
Resolves #260
Checklist:

[x] Updated go.mod and go.sum files.
[x] Verified the project builds correctly with the updated dependencies.
[x] Tested the changes in a development environment to ensure compatibility.

@nabuskey could you please review this PR!

@AryanSharma9917 Looks like it's not building correctly. Would you please take a look? https://github.com/cnoe-io/idpbuilder/actions/runs/10776701846/job/29884087640?pr=384

@AryanSharma9917 Any progress on this? I'd like to do this before cutting the next release.

@nabuskey could you please look into it?


RPC JSON API - listtransactions command

I tried this command: listtransactions and I got this output: []
I made two deposits and two withdrawals so I expect another output. What is the problem?
PD: I'm running your precompiled wallet on Windows Server 2016
PD2: I tried with this one: listsinceblock and I get the expected output but, what is the difference between those commands?
Thanks in advance.

on linux
deviant-cli -version
Deviant Core RPC client version v<IP_ADDRESS>-8e06493
it works as expected (see below).
Such rpc call went from bitcoin core https://en.bitcoin.it/wiki/Original_Bitcoin_client/API_calls_list
based on the definition above, list is blank for every account if there is only one transaction, otherwise the last should be shown


Add option to flush the label cache

Label text is found from Uberon and Interlex identifiers using appropriate endpoints, respectively SciGraph and Interlex. The resulting text is held in a local cache, labels.sqlite, in order to minimise Internet access.
There should be some means to clear the cache so as to ensure label text is refreshed. Manually deleting the cache database is one way to do this -- having a say --refresh-labels command line option would be cleaner.

The label cache is now incorporated into a local knowledge base which we don't want to be easily removed/cleared; any maintenance can be done directly using SQL. Closing the issue.


vscode <IP_ADDRESS> Run and debug function buttons are not in the same style...

vscode <IP_ADDRESS> Run and debug function buttons are not in the same style, so it's easy for newbies to use it and not easy to understand what it really means, so it's difficult to use.

I hope this issue will be fixed in vscode 1.7.5. Thank you very much!
System: Windows 10
code: <IP_ADDRESS>

These are different buttons with different functions and meanings, I think this is appropriate


Restrict access to unassigned jobs

My actions before raising this issue

[x] Read/searched the docs
[x] Searched past issues

Expected Behaviour

User must not have access to unassigned jobs and see them.
Current Behaviour

Unassigned jobs now is presented in jobs list and users can see them, if they are assigned on some job in the task. But links to unassigned tasks follow to 403 error. It might be a little confusing.

Possible Solution

Filter jobs in a task only with assigned ones.
Steps to Reproduce (for bugs)

Create a task as admin with multiple jobs and assign some job to some user.
Log in CVAT with that user credentials.

Operating System and version (e.g. Linux, Windows, MacOS):
Code example or link to GitHub repo or gist to reproduce problem:
Other diagnostic information / logs:
Logs from `cvat` container

Next steps
You may join our Gitter channel for community support.

@azhavoro @ActiveChooN, Is there any improvement or any solution in this matter?


Routing - Route error handling only at Angular (client-side)

Hi meanjs developers,
I have been following 0.4.0 for quite awhile and I notice the latest commit includes route error handling at both server and client side as oppose to the previous version whereby route error handling is solely done on the server side and any invalid route at client side is simply redirected to the home page.
I was wondering would it be possible to solely rely on Angular to perform route error handling?
Would it be possible to simply perform a res.status(404).send(...) in the "renderNotFound" function which in the case would require Angular to catch all 404 errors and redirect them to a 404 error page at the client side.
What would be the reason for performing route error handling at both server and client side? Is the current committed approach more viable? Sorry for asking, but i am rather new to web development...
Appreciate your replies. thanks!

Hey, thanks! I wrote that solution.
Server side handling makes sense for /api/* paths which are supposed to return json answer (or html if client requests so) and for /public/* paths so that browsers can catch missing asset files with appropriate error at console. Any other wrong routes are passed for Angular (i.e. server just pushes index.html and Angular ui-router then checks if the route exists or not).
In case of Angular headers are already sent to the browser, so it can't produce proper 404 header error code, which is a bummer. Getting 404 at least for some of the routes is a good thing and very important if one is to build REST api clients outside main Mean app.
The problem with non-public/api route errors is that the server doesn't know about them since routes are defined only at frontend. It would need quite a hack to make it isomorphic from its current state, so it would need to be well thought to avoid duplicate route definitions. (Could a Gulp task handle it somehow?)
Does this answer to your question or did I miss something?

Hi simison, thanks for the explanation!
But i am facing a slight issue with this approach.
In my application, I am trying to utilize express parameters to search for a user profile by his/her username.
So in the server side "users.server.routes.js", i have the following route:
app.route('/api/:username').get(users.read);

app.param('username', users.userByUsername);

and in client side "users.client.routes.js", i have the following state:
state('profile', {
  url: '/:username',
  templateUrl: '/modules/users/views/profile/profile.client.view.html'
}).

and a service to handle the request:
angular.module('users').factory('UserProfile', ['$resource',
  function($resource) {
    return $resource('api/:username', {username: '@username'}, {});
  }
]);

As you can see, if a user were to enter "localhost:3000/rubbish", whereby considering the username "rubbish" does not exist, it will be considered a valid route unless the function "users.userByUsername" performs a response of http 404 error, back to the client side.
I tried to perform a redirect to "/not-found" of the server side, but angular will simply render the "profile.client.view.html" after rendering the 404 page.
So my concern is, how should i go about handling this route error?
It is, in a sense, a valid route, but invalid as the username does not exist...
Thanks for your help!

 :-) I really like how Medium.com solved this with their "medium.com/@username" urls (that @).
In this case I'd try to build something with route resolvers, which is anyway a way to go and future Meanjs versions are moving to that direction as well. Check this: https://github.com/johnpapa/angular-styleguide#route-resolve-promises and https://github.com/meanjs/mean/pull/636 for work-in-progress.
So that promise would resolve with failure. I'd return 404 from the API for not-found users — see, the route at server side can be valid, but you can still return 404 from controller when user/article/etc doesn't exist.
In case of promise failure at route resolver, you can either redirect to /not-found or show some not-found message at profile view. You can even redirect at profile-view-controller, but that's a bit dirty practise.

Hi simison, thanks for the information!
Didn't know what i was getting myself into, haha! I just wanted to mimic what github was doing for viewing of user profile.
Thanks for the advice! Will look into applying route resolvers in my application.

@C-YanHua if you got all the help you need from @simison I guess we can close this issue?

Hi, for those who might be interested in my approach on performing route error handling at Angular only. The following is my solution to my problem (It is neither exactly perfect, nor elegant):
So at the server side, i modified/added the following:
In "core.server.routes.js":
// Define HTTP error pages.
app.route('/500-server-error').get(core.sendServerError);
app.route('/404-page-not-found').get(core.sendPageNotFound);

// Return page not found error for all undefined api, module or lib routes.
app.route('/:url(api|modules|lib)/*').get(core.sendPageNotFound);

In "core.server.controller.js":
exports.sendServerError = function(req, res) {
  res.status(500).send();
};

exports.sendPageNotFound = function(req, res) {
  return res.status(404).send();
};

In users.authorization.server.controller.js:
exports.userByUsername = function(req, res, next, username) {
  User.findOne({username: username}).exec(function(err, user) {
    if (err || !user) {
      return res.redirect('/404-page-not-found');
    }

    req.profile = user;
    next();
  });
};

In users.client.config.js (personally I feel that this should move into core config):
case 404:
  // Transit to 404 not found page.
  $injector.get('$state').transitionTo('404-page-not-found');
  break;
case 500:
  // Transit to 500 internal server error page.
  $injector.get('$state').transitionTo('500-server-error');
break;

In core.client.routes.js:
state('404-page-not-found', {
  templateUrl: '/modules/core/views/404.client.view.html'
}).
state('500-server-error', {
  templateUrl: '/modules/core/views/500.client.view.html'
});

Add the necessary view pages.
Thanks @simison for all the advice and help provided. I chose to ride on the http interceptor that was utilize by mean to catch 404 errors, but it is still advisable to perform route resolver for the routing.


2 indexes are created on auth_users.email

These lines in the DDL create two indices on column auth_users.email: First we create a "unique" column email (which adds an index), then we create a "unique index" email_unique_idx on LOWER(email).
I doubt we will ever need both of them.
In practice I can see the index created for the unique column being used in getUserByEmail, but I can never where email_unique_idx is used.  Indeed I find it hard to imagine how it could be useful given that we anyway lowercase all values of auth_user.email.
Having multiple indexes is both confusing and less efficient.

KV is here.  auth_users is gone.


Switch to connection based typechecking

Work in progress, only the error detection and recovery uses it now, visualization still uses the expression tree types.
It's a big change thus early testing and comments welcome.
Also depends on #227.

Now also the visualization uses the new connection based types, thus time for a bug hunt.

I think it is all working now, please try and test.

Changes in connections within a lambda do not correctly propagate visually to the outside anymore.
In the example below, when attaching the wire in the top-right (highlighted in red), the ($) below will claim that the resulting type will be b instead of Num a:

Any further changes to the ($) will correct the type again. I don't see why this happens though, since any visual invalidation on the outside is (not?) set after the inside of the lambda is computed.

@Flumble that bug is fixed now, and hopefull is the last.


Fix sign in design, typo

:tada: This PR is included in version 1.17.0-dev.20 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.22.4 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


problem getting to public systems biology workspace

From home pages -> workspaces -> NIAID Systems Biology

related to
#809

Fixed. Will push to alpha shortly.


OLQ: Fixed missing parameter in DeployVM function

Description
Issue logs:
03/05/2019 10:18:55 : [INFO ] Validating XML Files from C:\GitHub\LISAv2 folder recursively...
03/05/2019 10:19:04 : [INFO ] Collected Test : BVT-VERIFY-DEPLOYMENT-PROVISION
03/05/2019 10:19:04 : [INFO ] 1 Test Cases have been collected
03/05/2019 10:19:04 : [INFO ] Starting the test
03/05/2019 10:19:04 : [INFO ] BVT-VERIFY-DEPLOYMENT-PROVISION started running.
03/05/2019 10:19:04 : [ERROR] EXCEPTION: Cannot find an overload for "DeployVMs" and the argument count: "6".
03/05/2019 10:19:04 : [ERROR] Source: Line 40 in script .\TestProviders\OLQProvider.psm1.
03/05/2019 10:19:04 : [INFO ] LISAv2 exit code: 1

PR information

[x] The title of the PR is clear and informative.
[x] There are a small number of commits, each of which has an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, see this page.
[x] Except for special cases involving multiple contributors, the PR is started from a fork of the main repository, not a branch.
[x] If applicable, the PR references the bug/issue that it fixes in the description.
[x] Verify PR checker results and fix any coding errors.
[x] Included LISAv2 sample test results to demonstrate code functionality.

Quality of Code and Contribution Guidelines

[x] I have read the contribution guidelines.

Execution is verified on OLQ platform.


Implement List File Share API Support

What type of PR is this?

/kind new feature

What this PR does / why we need it:
This PR is to support the List File Share API. This includes the following:

Define FileShare DataModel
DB framework Support
Add List FileShare API support for routing
List File Share for a Backend
List all File Shares

Which issue(s) this PR fixes:

Fixes #989 #1002 #1003
Test Report Added?:

/kind TESTED

Test Report:
The following testing is done for List File Share API:

[x] List Backend

[x] List File Share for Azure Backend

[x] List File Share for AWS Backend

[x] List all File Share

List Backends:

http://<IP_ADDRESS>:8089/v1/94b280022d0c4401bcf3b0ea85870519/backends

{
    "backends": [
        {
            "id": "5ee26198870f6c0001db05b8",
            "tenantId": "94b280022d0c4401bcf3b0ea85870519",
            "userId": "558057c4256545bd8a307c37464003c9",
            "name": "azure-backend",
            "type": "azure-s3",
            "region": "Southeast Asia",
            "endpoint": "s3.amazonaws.com",
            "bucketName": "opensds-him"
        },
        {
            "id": "5ee261a4870f6c0001db05b9",
            "tenantId": "94b280022d0c4401bcf3b0ea85870519",
            "userId": "558057c4256545bd8a307c37464003c9",
            "name": "aws-backend",
            "type": "aws-s3",
            "region": "ap-south-1",
            "endpoint": "s3.amazonaws.com",
            "bucketName": "opensds-him"
        }
    ],
    "next": 2
}

List File Share for Azure Backend:

http://<IP_ADDRESS>:8089/v1/94b280022d0c4401bcf3b0ea85870519/backends/5ee26198870f6c0001db05b8/file/shares

{
    "fileshares": [
        {
            "id": "5ee26313a4fb8b2510259399",
            "createdAt": "2020-06-11T01:12:32.662883Z",
            "updatedAt": "2020-06-11T01:12:34.682329Z",
            "name": "azure-fileshareDemo",
            "description": "Azure FileShare",
            "tenantId": "94b280022d0c4401bcf3b0ea85870519",
            "userId": "558057c4256545bd8a307c37464003c9",
            "backendId": "5ee26198870f6c0001db05b8",
            "backend": "azure-backend",
            "type": "generalPurpose",
            "region": "Southeast Asia",
            "availabilityZone": "Southeast Asia",
            "status": "success"
        }
    ],
    "next": 1
}

List File Share for AWS Backend:

http://<IP_ADDRESS>:8089/v1/94b280022d0c4401bcf3b0ea85870519/backends/5ee261a4870f6c0001db05b9/file/shares

{
    "fileshares": [
        {
            "id": "5ee26257a4fb8b2510259397",
            "createdAt": "2020-06-11T01:12:29.662883Z",
            "updatedAt": "2020-06-11T01:12:31.682329Z",
            "name": "aws-fileshareDemo",
            "description": "AWS FileShare",
            "tenantId": "94b280022d0c4401bcf3b0ea85870519",
            "userId": "558057c4256545bd8a307c37464003c9",
            "backendId": "5ee261a4870f6c0001db05b9",
            "backend": "aws-backend",
            "type": "bursting",
            "region": "ap-south-1",
            "availabilityZone": "ap-south-1a",
            "status": "success",
            "tags": [
                {
                    "key": "Name",
                    "value": "FSName"
                },
                {
                    "key": "Department",
                    "value": "Billing"
                }
            ]
        },
        {
            "id": "5ee262d3a4fb8b2510259398",
            "createdAt": "2020-06-11T01:12:29.662883Z",
            "updatedAt": "2020-06-11T01:12:31.682329Z",
            "name": "aws-fileshare-test",
            "description": "AWS FileShare",
            "tenantId": "94b280022d0c4401bcf3b0ea85870519",
            "userId": "558057c4256545bd8a307c37464003c9",
            "backendId": "5ee261a4870f6c0001db05b9",
            "backend": "aws-backend",
            "type": "generalPurpose",
            "region": "ap-south-1",
            "availabilityZone": "ap-south-1b",
            "status": "success",
            "tags": [
                {
                    "key": "Name",
                    "value": "TestFS"
                },
                {
                    "key": "Department",
                    "value": "Testing"
                }
            ]
        }
    ],
    "next": 2
}

List all File Share:

http://<IP_ADDRESS>:8089/v1/94b280022d0c4401bcf3b0ea85870519/file/shares

{
    "fileshares": [
        {
            "id": "5ee26257a4fb8b2510259397",
            "createdAt": "2020-06-11T01:12:29.662883Z",
            "updatedAt": "2020-06-11T01:12:31.682329Z",
            "name": "aws-fileshareDemo",
            "description": "AWS FileShare",
            "tenantId": "94b280022d0c4401bcf3b0ea85870519",
            "userId": "558057c4256545bd8a307c37464003c9",
            "backendId": "5ee261a4870f6c0001db05b9",
            "backend": "aws-backend",
            "type": "bursting",
            "region": "ap-south-1",
            "availabilityZone": "ap-south-1a",
            "status": "success",
            "tags": [
                {
                    "key": "Name",
                    "value": "FSName"
                },
                {
                    "key": "Department",
                    "value": "Billing"
                }
            ]
        },
        {
            "id": "5ee262d3a4fb8b2510259398",
            "createdAt": "2020-06-11T01:12:29.662883Z",
            "updatedAt": "2020-06-11T01:12:31.682329Z",
            "name": "aws-fileshare-test",
            "description": "AWS FileShare",
            "tenantId": "94b280022d0c4401bcf3b0ea85870519",
            "userId": "558057c4256545bd8a307c37464003c9",
            "backendId": "5ee261a4870f6c0001db05b9",
            "backend": "aws-backend",
            "type": "generalPurpose",
            "region": "ap-south-1",
            "availabilityZone": "ap-south-1b",
            "status": "success",
            "tags": [
                {
                    "key": "Name",
                    "value": "TestFS"
                },
                {
                    "key": "Department",
                    "value": "Testing"
                }
            ]
        },
        {
            "id": "5ee26313a4fb8b2510259399",
            "createdAt": "2020-06-11T01:12:32.662883Z",
            "updatedAt": "2020-06-11T01:12:34.682329Z",
            "name": "azure-fileshareDemo",
            "description": "Azure FileShare",
            "tenantId": "94b280022d0c4401bcf3b0ea85870519",
            "userId": "558057c4256545bd8a307c37464003c9",
            "backendId": "5ee26198870f6c0001db05b8",
            "backend": "azure-backend",
            "type": "generalPurpose",
            "region": "Southeast Asia",
            "availabilityZone": "Southeast Asia",
            "status": "success"
        }
    ],
    "next": 3
}

Special notes for your reviewer:

@himanshuvar In the list Fileshare for Azure backends there is no tags key but  in the AWS list there is a tags key which is an object array.
I think this should be an empty array in case of Azure fileshares instead of completely removing it from the output.

@himanshuvar @kumarashit Another suggestion:

@himanshuvar @kumarashit Another suggestion:
The response is currently
{
    "fileshares": [
        {
           ...
         },
        {
           ...
        }
      ]
}

Our other APIs all return an empty array or an array of items. If possible remove the fileshares key.
So the response will look lilke:
[
        {
           ...
         },
        {
           ...
        }
]

or
[]

in case of empty result.

@anvithks This API is designed/implemented based on the current multi-cloud APIs.
You can check the swagger. https://github.com/sodafoundation/multi-cloud/blob/development/openapi-spec/swagger.yaml

@himanshuvar In the list Fileshare for Azure backends there is no tags key but in the AWS list there is a tags key which is an object array.
I think this should be an empty array in case of Azure fileshares instead of completely removing it from the output.

Agree. Let's keep it consistent @himanshuvar

@himanshuvar @kumarashit Another suggestion:
The response is currently
{
    "fileshares": [
        {
           ...
         },
        {
           ...
        }
      ]
}

Our other APIs all return an empty array or an array of items. If possible remove the fileshares key.
So the response will look lilke:
[
        {
           ...
         },
        {
           ...
        }
]

or
[]

in case of empty result.

IMO, the "key" should be kept. This is how the cloud API response also follow:
AWS
{
        FileSystems: [{
        CreationTime: 2020-06-03 10:46:16 +0000 UTC,
        CreationToken: "tokenstring",
        Encrypted: false,
        FileSystemId: "fs-4ffe7d9e",
        LifeCycleState: "available",

<Shares>
    <Share>
      <Name>share-name</Name>
      <Snapshot>Date-Time Value</Snapshot>
      <Properties>
        <Last-Modified>date/time-value</Last-Modified>
        <Etag>etag</Etag>
        <Quota>max-share-size</Quota>
      </Properties>
      <Metadata>
        <metadata-name>value</metadata-name>
      </Metadata>
    </Share>
  </Shares>

And also, multcloud APIs follow similar format

Is it possible to change to master branch

Is it possible to change to master branch

Yes, The PR is raised to master.
https://github.com/sodafoundation/multi-cloud/pull/1014


Training tools dependencies

Hi, I've been trying to build the training tools, but still can not success.
I already installed the 3 dependencies mentioned at Training Tesseract :

libicu
libpango
libcairo

I also build leptonica  in the same machine.
All the native libraries are already added to $LD_LIBRARY_PATH and binaries to $PATH.
I am using gcc 4.8.4 whic meets c++11 needed standards.
Standing at root source of Tesseract, after running ./configure --prefix=path/to/tesseract I am still getting the message: You can not build training tools because of missing dependency.
  BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1966080
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for ar... ar
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for dlfcn.h... yes
checking whether we are using the GNU C++ compiler... (cached) yes
checking whether g++ accepts -g... (cached) yes
checking how to run the C++ preprocessor... g++ -E
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... yes
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for ld used by g++... /usr/bin/ld -m elf_x86_64
checking if the linker (/usr/bin/ld -m elf_x86_64) is GNU ld... yes
checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking for g++ option to produce PIC... -fPIC -DPIC
checking if g++ PIC flag -fPIC -DPIC works... yes
checking if g++ static flag -static works... yes
checking if g++ supports -c -o file.o... yes
checking if g++ supports -c -o file.o... (cached) yes
checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether byte ordering is bigendian... no
checking if compiling with clang... no
checking whether compiler supports C++11... yes
checking for snprintf... yes
checking for library containing sem_init... -lpthread
checking for ANSI C header files... (cached) yes
checking whether time.h and sys/time.h may both be included... yes
checking for sys/wait.h that is POSIX.1 compatible... yes
checking sys/ipc.h usability... yes
checking sys/ipc.h presence... yes
checking for sys/ipc.h... yes
checking sys/shm.h usability... yes
checking sys/shm.h presence... yes
checking for sys/shm.h... yes
checking limits.h usability... yes
checking limits.h presence... yes
checking for limits.h... yes
checking malloc.h usability... yes
checking malloc.h presence... yes
checking for malloc.h... yes
checking for stdbool.h that conforms to C99... yes
checking for _Bool... no
checking whether #! works in shell scripts... yes
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking for getline... yes
checking for wchar_t... yes
checking for long long int... yes
checking for off_t... yes
checking for mbstate_t... yes
checking for leptonica... yes
checking for l_generateCIDataForPdf in -llept... yes
checking leptonica headers version >= 1.71... yes
checking unicode/uchar.h usability... yes
checking unicode/uchar.h presence... yes
checking for unicode/uchar.h... yes
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for pango... yes
checking for cairo... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: creating tesseract.pc
config.status: creating api/Makefile
config.status: creating ccmain/Makefile
config.status: creating opencl/Makefile
config.status: creating ccstruct/Makefile
config.status: creating ccutil/Makefile
config.status: creating classify/Makefile
config.status: creating cube/Makefile
config.status: creating cutil/Makefile
config.status: creating dict/Makefile
config.status: creating neural_networks/runtime/Makefile
config.status: creating textord/Makefile
config.status: creating viewer/Makefile
config.status: creating wordrec/Makefile
config.status: creating tessdata/Makefile
config.status: creating tessdata/configs/Makefile
config.status: creating tessdata/tessconfigs/Makefile
config.status: creating testing/Makefile
config.status: creating java/Makefile
config.status: creating java/com/Makefile
config.status: creating java/com/google/Makefile
config.status: creating java/com/google/scrollview/Makefile
config.status: creating java/com/google/scrollview/events/Makefile
config.status: creating java/com/google/scrollview/ui/Makefile
config.status: creating doc/Makefile
config.status: creating config_auto.h
config.status: executing depfiles commands
config.status: executing libtool commands

Does any one can help me with this? I am stocked at this point.
Thank you so much!

What OS ?
Which branch, 3.05 or master (4.0)
For training tools you need Dev versions
sudo apt-get install libicu-dev
sudo apt-get install libpango1.0-dev
sudo apt-get install libcairo2-dev

@joshimar, if you updated from an older version of Tesseract, you have to re-run ./autogen.sh to create an updated configure script. The latest Tesseract version from Git always shows the reason why training tools cannot be built, so either you are using an old and buggy Tesseract code or you missed updating configure, too.

@Shreeshrii I downloaded it directly from master branch a month ago, if I type tesseract --version I get
tesseract 3.05.00dev
leptonica-1.73
libpng 1.2.49 : libtiff 3.9.4 : zlib 1.2.3

I am using Oracle Linux Server release 6.8 so, unfortunately I can not use apt-get. Also tried sudo yum update and then yum install any-of-3packages , but didn't work. That is why I decided to download every dependency and install them in a common prefix. The sites where I obtained these tools are:
http://packages.ubuntu.com/trusty/libicu-dev: icu_52.1.orig.tar.gz,
http://www.linuxfromscratch.org/blfs/view/svn/x/cairo.html: cairo-1.14.6.tar.xz, http://www.linuxfromscratch.org/blfs/view/svn/x/pango.html: pango-1.40.3.tar.xz
And all the dependencies specified in their websites.

@stweil  Actually I didn't update Tesseract, I downloaded the latest version from master a month ago and also use .autogen.sh, I will try to re build it again and will let you know if it works.
Thank you!!

Please try with the following packages and their dependencies
http://packages.ubuntu.com/trusty/libicu-dev
http://packages.ubuntu.com/trusty/libpango1.0-dev
http://packages.ubuntu.com/trusty/libcairo2-dev

It seems yum uses different packages and names are also different. Try pango_devel and cairo-devel

@stweil I am trying to re build Tesseract, but now I am getting the following:

./.libs/libtesseract.so: undefined reference to omp_get_thread_num' ./.libs/libtesseract.so: undefined reference to GOMP_sections_end_nowait'
./.libs/libtesseract.so: undefined reference to omp_get_num_threads' ./.libs/libtesseract.so: undefined reference to GOMP_parallel_start'
./.libs/libtesseract.so: undefined reference to GOMP_parallel_end' ./.libs/libtesseract.so: undefined reference to GOMP_sections_next'
./.libs/libtesseract.so: undefined reference to `GOMP_parallel_sections_start'

I set thevariables setenv CFLAGS /usr/include setenv LDFLAGS /usr/lib and tried setenv LDFLAGS /usr/local/lib setenv LDFLAGS /usr/local/lib
I see libgomp.so.1  libgomp.so.1.0.0 under /usr/lib and /usr/lib64
do you have any idea of what is happening? Becuase I also have installed Leptonica, but I don´t know if I am missing something during compilation process.
Thank you !!

@Shreeshrii I tried what you mentioned, using yum and downloading these packages, but still not working.
Do I have to set anything else rather than just adding the native libraries  paths to my LD_LIBRARY_PATH? Something that might cause my GCC crash because of possible missing dependencies?
Thanks a lot Shreeshrii!

@stweil  Those messages are appearing compiling master (4.0) using gcc 4.8.4, so I compiled version 3.5 instead. Still working on building training tools.

Can you upload your full build log (output from configure and make) somewhere, so I might have a look?

Sure @stweil !
  BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1966080
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for ar... ar
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for dlfcn.h... yes
checking whether we are using the GNU C++ compiler... (cached) yes
checking whether g++ accepts -g... (cached) yes
checking how to run the C++ preprocessor... g++ -E
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... yes
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for ld used by g++... /usr/bin/ld -m elf_x86_64
checking if the linker (/usr/bin/ld -m elf_x86_64) is GNU ld... yes
checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking for g++ option to produce PIC... -fPIC -DPIC
checking if g++ PIC flag -fPIC -DPIC works... yes
checking if g++ static flag -static works... yes
checking if g++ supports -c -o file.o... yes
checking if g++ supports -c -o file.o... (cached) yes
checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether byte ordering is bigendian... no
checking if compiling with clang... no
checking whether compiler supports C++11... yes
checking for snprintf... yes
checking for library containing sem_init... -lpthread
checking for ANSI C header files... (cached) yes
checking whether time.h and sys/time.h may both be included... yes
checking for sys/wait.h that is POSIX.1 compatible... yes
checking sys/ipc.h usability... yes
checking sys/ipc.h presence... yes
checking for sys/ipc.h... yes
checking sys/shm.h usability... yes
checking sys/shm.h presence... yes
checking for sys/shm.h... yes
checking limits.h usability... yes
checking limits.h presence... yes
checking for limits.h... yes
checking malloc.h usability... yes
checking malloc.h presence... yes
checking for malloc.h... yes
checking for stdbool.h that conforms to C99... yes
checking for _Bool... no
checking whether #! works in shell scripts... yes
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking for getline... yes
checking for wchar_t... yes
checking for long long int... yes
checking for off_t... yes
checking for mbstate_t... yes
checking for leptonica... yes
checking for l_generateCIDataForPdf in -llept... yes
checking leptonica headers version >= 1.71... yes
checking unicode/uchar.h usability... yes
checking unicode/uchar.h presence... yes
checking for unicode/uchar.h... yes
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for pango... yes
checking for cairo... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: creating tesseract.pc
config.status: creating api/Makefile
config.status: creating ccmain/Makefile
config.status: creating opencl/Makefile
config.status: creating ccstruct/Makefile
config.status: creating ccutil/Makefile
config.status: creating classify/Makefile
config.status: creating cube/Makefile
config.status: creating cutil/Makefile
config.status: creating dict/Makefile
config.status: creating neural_networks/runtime/Makefile
config.status: creating textord/Makefile
config.status: creating viewer/Makefile
config.status: creating wordrec/Makefile
config.status: creating tessdata/Makefile
config.status: creating tessdata/configs/Makefile
config.status: creating tessdata/tessconfigs/Makefile
config.status: creating testing/Makefile
config.status: creating java/Makefile
config.status: creating java/com/Makefile
config.status: creating java/com/google/Makefile
config.status: creating java/com/google/scrollview/Makefile
config.status: creating java/com/google/scrollview/events/Makefile
config.status: creating java/com/google/scrollview/ui/Makefile
config.status: creating doc/Makefile
config.status: creating config_auto.h
config.status: config_auto.h is unchanged
config.status: executing depfiles commands
config.status: executing libtool commands
Configuration is done.
You can now build and install tesseract by running:
$ make
$ sudo make install
You can not build training tools because of missing dependency.
Check configure output for details.

I need the make output as well. configure shows that OpenMP is supported. Your problem is caused by partial OpenMP support. I think that configure --disable-openmp will work for you (but result in slow binaries).

Thanks @stweil
The make output is pretty large, so it's attached at the end.
Also, after trying
./configure --prefix=path --disable-openmp
I'm still getting the message
"You can not build training tools because of missing dependency."
make-output.txt

Yes, you already said that training tools cannot be build with your 3.05 (is it the latest 3.05?).
configure --disable-openmp was meant to be used with 4.0.

Sure, actually I tried with both versions, but I'am getting the same message after ./configure.
When I do make:
3.05 finishes.
4.00 throws:
./.libs/libtesseract.so: undefined reference to `omp_get_thread_num'
./.libs/libtesseract.so: undefined reference to `GOMP_sections_end_nowait'
./.libs/libtesseract.so: undefined reference to `omp_get_num_threads'
./.libs/libtesseract.so: undefined reference to `GOMP_parallel_start'
./.libs/libtesseract.so: undefined reference to `GOMP_parallel_end'
./.libs/libtesseract.so: undefined reference to `GOMP_sections_next'
./.libs/libtesseract.so: undefined reference to `GOMP_parallel_sections_start'

This is caused by https://github.com/tesseract-ocr/tesseract/commit/6140be6a5575e9159e3678adf4ee9e673b3ff2cc. If I cherry-pick it to 3.05 build will failed with gcc...

OpenMP never worked for 3.05. Commit 6140be6 fixed OpenMP support partially. I'll send updates which fix the remaining issue.

@stweil:  Thanks. At this stage I am more concerned with build process than if feature works ;-)

@joshimar, please try latest versions from git (master and 3.05) and close this issue if it works now.

Thanks for your help guys, I will let you know!

Hi! I am sorry for the delay.
I could build tesseract now, but still can´t build training tools, here is what I did:
I downloaded the code from master just today, after doing
./autogen.sh
setenv LIBLEPT_HEADERSDIR $install/include
./configure --prefix=$install --with-extra-libraries=$install/lib

Now I see that there are different outputs related with Leptonica and one of them is not saying yes, I wonder what does this new param says because I have installed latest Leptonica's version (1.73) under $install. Also In installed libicu under this location and I am seeing an issue with this either:

checking for g++... g++
checking for C++ compiler default output file name... a.out
checking whether the C++ compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
fatal: Not a git repository: './.git'
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking for style of include used by make...  BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for ar... ar
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for dlfcn.h... yes
checking whether we are using the GNU C++ compiler... (cached) yes
checking whether g++ accepts -g... (cached) yes
checking how to run the C++ preprocessor... g++ -E
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... yes
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for ld used by g++... /usr/bin/ld -m elf_x86_64
checking if the linker (/usr/bin/ld -m elf_x86_64) is GNU ld... yes
checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking for g++ option to produce PIC... -fPIC -DPIC
checking if g++ PIC flag -fPIC -DPIC works... yes
checking if g++ static flag -static works... yes
checking if g++ supports -c -o file.o... yes
checking if g++ supports -c -o file.o... (cached) yes
checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether byte ordering is bigendian... no
checking if compiling with clang... no
checking whether compiler supports C++11... yes
checking for snprintf... yes
checking for library containing sem_init... -lpthread
checking for ANSI C header files... (cached) yes
checking whether time.h and sys/time.h may both be included... yes
checking for sys/wait.h that is POSIX.1 compatible... yes
checking sys/ipc.h usability... yes
checking sys/ipc.h presence... yes
checking for sys/ipc.h... yes
checking sys/shm.h usability... yes
checking sys/shm.h presence... yes
checking for sys/shm.h... yes
checking limits.h usability... yes
checking limits.h presence... yes
checking for limits.h... yes
checking malloc.h usability... yes
checking malloc.h presence... yes
checking for malloc.h... yes
checking for stdbool.h that conforms to C99... yes
checking for _Bool... no
checking whether #! works in shell scripts... yes
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking for getline... yes
checking for wchar_t... yes
checking for long long int... yes
checking for off_t... yes
checking for mbstate_t... yes
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for LEPTONICA... no
checking for leptonica... yes
checking for l_generateCIDataForPdf in -llept... yes
checking leptonica headers version >= 1.71... yes
checking for ICU_UC... no
checking for ICU_I18N... no
checking unicode/uchar.h usability... yes
checking unicode/uchar.h presence... yes
checking for unicode/uchar.h... yes
checking for pango... yes
checking for cairo... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: creating tesseract.pc
config.status: creating api/Makefile
config.status: creating arch/Makefile
config.status: creating ccmain/Makefile
config.status: creating opencl/Makefile
config.status: creating ccstruct/Makefile
config.status: creating ccutil/Makefile
config.status: creating classify/Makefile
config.status: creating cube/Makefile
config.status: creating cutil/Makefile
config.status: creating dict/Makefile
config.status: creating lstm/Makefile
config.status: creating neural_networks/runtime/Makefile
config.status: creating textord/Makefile
config.status: creating viewer/Makefile
config.status: creating wordrec/Makefile
config.status: creating tessdata/Makefile
config.status: creating tessdata/configs/Makefile
config.status: creating tessdata/tessconfigs/Makefile
config.status: creating testing/Makefile
config.status: creating java/Makefile
config.status: creating java/com/Makefile
config.status: creating java/com/google/Makefile
config.status: creating java/com/google/scrollview/Makefile
config.status: creating java/com/google/scrollview/events/Makefile
config.status: creating java/com/google/scrollview/ui/Makefile
config.status: creating doc/Makefile
config.status: creating config_auto.h
config.status: executing depfiles commands
config.status: executing libtool commands
Configuration is done.
You can now build and install tesseract by running:
$ make
$ sudo make install
You can not build training tools because of missing dependency.
Check configure output for details.

I can execute
make
make install

without any problem so I guess 6140be6 fixed it but still can not build training tools.
Here is what I have under $install :
lib
liblept.a  liblept.la liblept.so  liblept.so.1  liblept.so.1.73  liblept.so.5  liblept.so.5.0.0  libtesseract.a  libtesseract.la libtesseract.so  libtesseract.so.4  libtesseract.so.4.0.0  modules  pkgconfig  terminfo
include
leptonica  tesseract
bin
bash  convertfilestopdf  convertfilestops  convertformat  convertsegfilestopdf  convertsegfilestops  converttopdf  converttops  fileinfo  printimage  printsplitimage  printtiff  sh  splitimage2pdf  tesseract  xtractprotos
Thank you!

configure uses pkg-config to find Leptonica and libicu, so at least the *.pc files must be found by pkg-config. They are not found because you had chosen a different install path. Either copy the *.pc files to the correct location, or tell pkg-config how to find them (please look into the manual for that).

checking for LEPTONICA... no
checking for leptonica... yes

That's funny!

The first check is from pkg-config (failed), the second one is the old fallback check (which is removed in PR #577).

Alright.
Now, before running ./configure... I did:
setenv PKG_CONFIG_PATH $install/usr/lib64/pkgconfig:$install/lib/pkgconfig
Because this is where I also installed libicu-dev.   BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for ar... ar
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for dlfcn.h... yes
checking whether we are using the GNU C++ compiler... (cached) yes
checking whether g++ accepts -g... (cached) yes
checking how to run the C++ preprocessor... g++ -E
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... yes
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for ld used by g++... /usr/bin/ld -m elf_x86_64
checking if the linker (/usr/bin/ld -m elf_x86_64) is GNU ld... yes
checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking for g++ option to produce PIC... -fPIC -DPIC
checking if g++ PIC flag -fPIC -DPIC works... yes
checking if g++ static flag -static works... yes
checking if g++ supports -c -o file.o... yes
checking if g++ supports -c -o file.o... (cached) yes
checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether byte ordering is bigendian... no
checking if compiling with clang... no
checking whether compiler supports C++11... yes
checking for snprintf... yes
checking for library containing sem_init... -lpthread
checking for ANSI C header files... (cached) yes
checking whether time.h and sys/time.h may both be included... yes
checking for sys/wait.h that is POSIX.1 compatible... yes
checking sys/ipc.h usability... yes
checking sys/ipc.h presence... yes
checking for sys/ipc.h... yes
checking sys/shm.h usability... yes
checking sys/shm.h presence... yes
checking for sys/shm.h... yes
checking limits.h usability... yes
checking limits.h presence... yes
checking for limits.h... yes
checking malloc.h usability... yes
checking malloc.h presence... yes
checking for malloc.h... yes
checking for stdbool.h that conforms to C99... yes
checking for _Bool... no
checking whether #! works in shell scripts... yes
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking for getline... yes
checking for wchar_t... yes
checking for long long int... yes
checking for off_t... yes
checking for mbstate_t... yes
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for LEPTONICA... yes
checking for l_generateCIDataForPdf in -llept... yes
checking leptonica headers version >= 1.71... yes
checking for ICU_UC... yes
checking for ICU_I18N... yes
checking for pango... yes
checking for cairo... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: creating tesseract.pc
config.status: creating api/Makefile
config.status: creating arch/Makefile
config.status: creating ccmain/Makefile
config.status: creating opencl/Makefile
config.status: creating ccstruct/Makefile
config.status: creating ccutil/Makefile
config.status: creating classify/Makefile
config.status: creating cube/Makefile
config.status: creating cutil/Makefile
config.status: creating dict/Makefile
config.status: creating lstm/Makefile
config.status: creating neural_networks/runtime/Makefile
config.status: creating textord/Makefile
config.status: creating viewer/Makefile
config.status: creating wordrec/Makefile
config.status: creating tessdata/Makefile
config.status: creating tessdata/configs/Makefile
config.status: creating tessdata/tessconfigs/Makefile
config.status: creating testing/Makefile
config.status: creating java/Makefile
config.status: creating java/com/Makefile
config.status: creating java/com/google/Makefile
config.status: creating java/com/google/scrollview/Makefile
config.status: creating java/com/google/scrollview/events/Makefile
config.status: creating java/com/google/scrollview/ui/Makefile
config.status: creating doc/Makefile
config.status: creating config_auto.h
config.status: executing depfiles commands
config.status: executing libtool commands
Configuration is done.
You can now build and install tesseract by running:
$ make
$ sudo make install
You can not build training tools because of missing dependency.
Check configure output for details.

Now, after this check it seems that I have all the dependencies, but still can't go to the training tools building process. :s

That's strange. You can debug the problem like this:

Edit configure.am: remove the first line with AM_CONDITIONAL([ENABLE_TRAINING], false)
Run ./autogen.sh
Try ./configure
Repeat all three steps until you find which line caused the problem

checkout master. Should be fixed.

Hi guys.
Thank you very much for your help.
I could now build training tools using master (Tesseract 4.0).
I be able to build them I had to remove all the lines AM_CONDITIONAL([ENABLE_TRAINING], false)   in configure.ac, to be specific, they were 4.
After that, then trying to build training tools I saw an error related with libicu, basically the compiler was not finding some .h files. As I have libicu installed under $install and I also added $install/lib to $LD_LIBRARY_PATH, $install/bin to $PATH and also set $PKG_CONFIG_PATH=$install/usr/lib64/pkgconfig:$install/lib/pkgconfig I guess there was some other configuration to tell GCC where to look for the include files, but I could not find it, so to solve this I had to copy all the files under $install/include/unicode to /usr/include/unicode so GCC could find these files.  Do you have any suggestion when this happens to tell GCC specifically where to look for the .h files? (a non standard "include" directory) I think there should be another way so I don't really need to copy the include files to the standard location.
I have one final question: Do you guys know where can I ask questions to Leptonica's team? They have a mailing list in Google Code, but it seems that we can not open any other issue.
Once again, thanks for your help on this!

I think that the pango-devel and cairo-devel in RHEL/Centos 6 would work with Tesseract 4.

I have one final question: Do you guys know where can I ask questions to Leptonica's team? They have a mailing list in Google Code, but it seems that we can not open any other issue.

(I)
https://github.com/DanBloomberg/leptonica/issues
(II)
From http://leptonica.org/

Contact: Dan Bloomberg (bloomberg "at" ieee "dot" org) for questions and suggestions

Can we close this issue?

I am sorry!
Sure, now you can close this issue.
Once again, thank you for helping me to solve this.
I am now running training tools on Oracle Linux Server 6.8 successfully.
Merry Christmas and happy new year!


[FEATURE] Support to export data to google driver

问卷星 No.12

The auth secret of google driver is required, so export data to chome.storage.sync
https://developer.chrome.com/docs/extensions/reference/storage/#:~:text=StorageArea-,sync,-Items%20in%20the

Can't implement


Fix warnings in per thread query test

There are only some (more) warnings when building Boost that we can ignore.


Failover support for the siddhi operator

Purpose

The siddhi operator initial versions only support basic deployment type. Here in this new feature, we enable high available(failover) deployment using siddhi operator. For more details about the new functionalities refer this mail thread. The resolved issues in this PR this issue https://github.com/siddhi-io/siddhi-operator/issues/33

Goals

Add failover support to the siddhi operator

Approach

Change the parser
Upgrade siddhi operator CRD to v1aplha2
Change functionalities

Automation tests

Unit tests

Test each function that can be tested in go level.

Integration tests

Added E2E tests to simulate default and failover scenarios in the k8s cluster using a fake client.

Test environment

minikube version: v0.33.1
Google Kubernetes Engine

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Buddhi Kothalawala seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


rfrom(), etc, don't adjust location correctly.

A reverse iterator should be storing an actual iterator to the next (in document order) location; reverse iterators constructed from rfrom and such do not account for this.

Rendered obsolete by removing the reverse iterators in https://github.com/jpetrie/quip/commit/597f3662be7493f554ed11c4d4c50047b7f9bb95.


change tension of cardinal interpolation

cardinal interpolation has parameter "tensity" which allows to manipulate with length of tangens
as described here https://github.com/d3/d3-shape/blob/master/README.md#curves
it would be great be able to change it in angular2
thanks!

actually I need to prevent graph from falling below to zero


Zarr v3

Attempt to run against Zarr v3. There are still lots of failures - see #295

This is passing against the upstream Zarr v3 branch, so I think it can be merged. There will be follow-on improvements once https://github.com/zarr-developers/zarr-python/issues/1978 is fixed.


how to replicate the wmt16 ende translation result on transformer big model?

Hi, i try to replicate the WMT16 en-de translation result on transformer Big model.
i try to preprocess the data with this:
TEXT=./examples/translation/wmt16_en_de
fairseq-preprocess --source-lang en --target-lang de
--trainpref $TEXT/train.tok.clean.bpe.32000
--validpref $TEXT/newstest2013.tok.bpe.32000
--testpref $TEXT/newstest2014.tok.bpe.32000
--destdir data-bin/wmt16_en_de_bpe32k
--nwordssrc 32768 --nwordstgt 32768
--joined-dictionary --worker 8

Then I train the model with this:
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 fairseq-train
data-bin/wmt16_en_de_bpe32k
--arch transformer_vaswani_wmt_en_de_big --share-all-embeddings
--optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0
--lr-scheduler inverse_sqrt --warmup-init-lr 1e-07 --warmup-updates 4000
--lr 0.0005 --min-lr 1e-09
--dropout 0.3 --weight-decay 0.0
--criterion label_smoothed_cross_entropy --label-smoothing 0.1
--max-tokens 3584 --fp16
--no-progress-bar --save-interval-updates 1000 --keep-interval-updates 20
--save-dir checkpoints/transformer_vaswani_en_de_big --max-epoch 15

to generate the result and calculate the BLEU with this:
CUDA_VISIBLE_DEVICES="0" python ../generate.py
data-bin/wmt16_en_de_bpe32k
--path $1
--remove-bpe --beam 4 --batch-size 64 --lenpen 0.6
--max-len-a 1 --max-len-b 50 |tee ./tmp/gen.out

Compute BLEU score
grep ^H ./tmp/gen.out | cut -f3- > ./tmp/gen.out.sys
grep ^T ./tmp/gen.out | cut -f2- > ./tmp/gen.out.ref
fairseq-score --sys ./tmp/gen.out.sys --ref ./tmp/gen.out.ref

However, after about 15 epoches, I got :
| Translated 3003 sentences (86132 tokens) in 29.5s (101.93 sentences/s, 2923.70 tokens/s)
| Generate test with beam=4: BLEU4 = 26.59, 57.8/32.3/20.3/13.2 (BP=1.000, ratio=1.021, syslen=64408, reflen=63078)
Namespace(ignore_case=False, order=4, ref='./tmp/gen.out.ref', sacrebleu=False, sys='./tmp/gen.out.sys')
BLEU4 = 26.59, 57.8/32.3/20.3/13.2 (BP=1.000, ratio=1.021, syslen=64408, reflen=63078)
Far away from the result in "attention is all you need"(28.4), and the model training takes almost a full day. Can you think of any reasons for this?

How many updates did you complete? Also what GPU are you using and is --fp16 correctly speeding things up compared to FP32?
Another couple tricks to reproduce results from Attention is all you need:

checkpoint averaging: use the scripts/average_checkpoints.py script to average the last few (5?) checkpoints, making sure to save checkpoints more frequently near the end. You can adjust this with --save-interval-updates.
apply compound splitting, which increases BLEU by ~0.5: https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/utils/get_ende_bleu.sh

@myleott
I trained the transformer big model for 15 epoches and used 8 v100 GPUs.  Checkpoint averaging and compound splitting were not used, when i calculated the BLEU.

For the paper "Scaling neural machine translation", Which training set was used in Table 1 ? i try to replicate the result on transformer base model for WMT14 ende translation, and get a lower BLEU  as well
I train the model with this:
ARCH=transformer_wmt_en_de
# Train a model:
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 fairseq-train \
     data-bin/wmt14_en_de_bpe32k \
     --arch $ARCH --share-all-embeddings \
     --optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0 \
     --lr-scheduler inverse_sqrt --warmup-init-lr 1e-07 --warmup-updates 4000 \
     --min-lr 1e-09 \
     --dropout 0.3 --weight-decay 0.0 --criterion label_smoothed_cross_entropy \
     --label-smoothing 0.1 \
     --max-tokens 3584 \
     --fp16 \
     --lr 1e-03 \
     --update-freq 16 \
     --max-update 9600 \
     --no-progress-bar --save-interval-updates 1000 --keep-interval-updates 20 \
     --save-dir checkpoints/transformer_base \

Thank you !

We used WMT-16 En-De and Transformer big, following the instructions here: https://github.com/pytorch/fairseq/tree/master/examples/scaling_nmt

@myleott
all the experiments(+fp16、+cumul、+2x lr、+5k tkn/gpu) in "scaling neural machine translation" Table 1 were conducted on transformer big model and WMT16 ?

Yes

@myleott
Thank you !

@myleott
hi, i try to replicate the result of '+ 2x lr' on transformer big and wmt16. At the begining, the loss and nll_loss reduced as usual(updates:0->3057), but the loss and nll_loss increased constantly(updates:3058->9600).
| WARNING: overflow detected, setting loss scale to: 8.0
| epoch 006 | valid on 'valid' subset | loss 4.878 | nll_loss 3.103 | ppl 8.59 | num_updates 2000 | best_loss 4.87848
| saved checkpoint checkpoints/transformer_vaswani16_ende_big/checkpoint_6_2000.pt (epoch 6 @ 2000 updates) (writing took 7.748462915420532 seconds)
| epoch 006 | loss 5.375 | nll_loss 3.851 | ppl 14.43 | wps 105156 | ups 0 | wpb 402417.835 | bsz 13150.676 | num_updates 2036 | lr 0.000509049 | gnorm 0.475 | clip 0.000 | oom 0.000 | loss
| epoch 006 | valid on 'valid' subset | loss 4.821 | nll_loss 3.064 | ppl 8.36 | num_updates 2036 | best_loss 4.82093
| saved checkpoint checkpoints/transformer_vaswani16_ende_big/checkpoint6.pt (epoch 6 @ 2036 updates) (writing took 7.87086820602417 seconds)
| WARNING: overflow detected, setting loss scale to: 8.0
| epoch 007 | loss 5.070 | nll_loss 3.517 | ppl 11.45 | wps 105713 | ups 0 | wpb 402428.516 | bsz 13155.935 | num_updates 2377 | lr 0.000594291 | gnorm 0.387 | clip 0.000 | oom 0.000 | loss
| epoch 007 | valid on 'valid' subset | loss 4.629 | nll_loss 2.850 | ppl 7.21 | num_updates 2377 | best_loss 4.62924
| saved checkpoint checkpoints/transformer_vaswani16_ende_big/checkpoint7.pt (epoch 7 @ 2377 updates) (writing took 7.967353820800781 seconds)
| WARNING: overflow detected, setting loss scale to: 16.0
| WARNING: overflow detected, setting loss scale to: 8.0
| epoch 008 | loss 4.875 | nll_loss 3.306 | ppl 9.89 | wps 105795 | ups 0 | wpb 402477.000 | bsz 13161.029 | num_updates 2717 | lr 0.000679282 | gnorm 0.355 | clip 0.000 | oom 0.000 | loss_
| epoch 008 | valid on 'valid' subset | loss 4.449 | nll_loss 2.677 | ppl 6.39 | num_updates 2717 | best_loss 4.44949
| saved checkpoint checkpoints/transformer_vaswani16_ende_big/checkpoint8.pt (epoch 8 @ 2717 updates) (writing took 7.882242679595947 seconds)
| WARNING: overflow detected, setting loss scale to: 8.0
| WARNING: overflow detected, setting loss scale to: 8.0
| epoch 009 | valid on 'valid' subset | loss 4.338 | nll_loss 2.567 | ppl 5.93 | num_updates 3000 | best_loss 4.33773
| saved checkpoint checkpoints/transformer_vaswani16_ende_big/checkpoint_9_3000.pt (epoch 9 @ 3000 updates) (writing took 7.749082565307617 seconds)
| epoch 009 | loss 4.744 | nll_loss 3.164 | ppl 8.97 | wps 105280 | ups 0 | wpb 402417.374 | bsz 13154.794 | num_updates 3057 | lr 0.000764274 | gnorm 0.346 | clip 0.000 | oom 0.000 | loss_
| epoch 009 | valid on 'valid' subset | loss 4.329 | nll_loss 2.558 | ppl 5.89 | num_updates 3057 | best_loss 4.32936
| saved checkpoint checkpoints/transformer_vaswani16_ende_big/checkpoint9.pt (epoch 9 @ 3057 updates) (writing took 7.789773464202881 seconds)
| WARNING: overflow detected, setting loss scale to: 16.0
| WARNING: overflow detected, setting loss scale to: 8.0
| WARNING: overflow detected, setting loss scale to: 4.0
| WARNING: overflow detected, setting loss scale to: 2.0
| WARNING: overflow detected, setting loss scale to: 1.0
| WARNING: overflow detected, setting loss scale to: 0.5
| WARNING: overflow detected, setting loss scale to: 0.25
| epoch 010 | loss 7.143 | nll_loss 5.919 | ppl 60.49 | wps 103749 | ups 0 | wpb 402502.988 | bsz 13146.860 | num_updates 3392 | lr 0.000848015 | gnorm 0.726 | clip 0.000 | oom 0.000 | loss
| epoch 010 | valid on 'valid' subset | loss 11.788 | nll_loss 11.226 | ppl 2395.81 | num_updates 3392 | best_loss 4.32936
| saved checkpoint checkpoints/transformer_vaswani16_ende_big/checkpoint10.pt (epoch 10 @ 3392 updates) (writing took 4.796957731246948 seconds)
| epoch 011 | loss 11.626 | nll_loss 11.070 | ppl 2149.65 | wps 106200 | ups 0 | wpb 402421.772 | bsz 13160.719 | num_updates 3734 | lr 0.000933507 | gnorm 0.480 | clip 0.000 | oom 0.000 |
| epoch 011 | valid on 'valid' subset | loss 14.099 | nll_loss 13.758 | ppl 13858.17 | num_updates 3734 | best_loss 4.32936
| saved checkpoint checkpoints/transformer_vaswani16_ende_big/checkpoint11.pt (epoch 11 @ 3734 updates) (writing took 4.8482091426849365 seconds)
| WARNING: overflow detected, setting loss scale to: 2.0
| WARNING: overflow detected, setting loss scale to: 1.0
| WARNING: overflow detected, setting loss scale to: 0.5
| epoch 012 | valid on 'valid' subset | loss 14.852 | nll_loss 14.363 | ppl 21064.62 | num_updates 4000 | best_loss 4.32936
| saved checkpoint checkpoints/transformer_vaswani16_ende_big/checkpoint_12_4000.pt (epoch 12 @ 4000 updates) (writing took 4.745401382446289 seconds)
| epoch 012 | loss 11.545 | nll_loss 10.977 | ppl 2015.82 | wps 104868 | ups 0 | wpb 402440.389 | bsz 13164.761 | num_updates 4073 | lr 0.000990998 | gnorm 0.604 | clip 0.000 | oom 0.000 |
| epoch 012 | valid on 'valid' subset | loss 15.443 | nll_loss 15.009 | ppl 32969.28 | num_updates 4073 | best_loss 4.32936

I trained the model with this:
# Train a model:
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 fairseq-train \
     data-bin/wmt16_en_de_bpe32k \
     --arch transformer_vaswani_wmt_en_de_big --share-all-embeddings \
     --optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0 \
     --lr-scheduler inverse_sqrt --warmup-init-lr 1e-07 --warmup-updates 4000 \
     --min-lr 1e-09 \
     --dropout 0.3 --weight-decay 0.0 --criterion label_smoothed_cross_entropy \
     --label-smoothing 0.1 \
     --max-tokens 3584 \
     --fp16 \
     --lr 1e-03 \
     --update-freq 16 \
     --max-update 9600 \
     --no-progress-bar --save-interval-updates 1000 --keep-interval-updates 20 \
     --save-dir checkpoints/transformer_wmt16_ende_big \

Please help me handle this question, thank you !

Can you try --ddp-backend=no_c10d? This version may behave a bit better when combined with large update-freq settings.

@myleott
ok, i will try --ddp-backend=no_c10d, thank you !

Hi,
I got the same situation when running the big models. I am using the newest version of fairseq. What can be the reason?

@myleott
hi, i try to replicate the result of '+ 2x lr' on transformer big and wmt16. At the begining, the loss and nll_loss reduced as usual(updates:0->3057), but the loss and nll_loss increased constantly(updates:3058->9600).
| WARNING: overflow detected, setting loss scale to: 8.0
| epoch 006 | valid on 'valid' subset | loss 4.878 | nll_loss 3.103 | ppl 8.59 | num_updates 2000 | best_loss 4.87848
| saved checkpoint checkpoints/transformer_vaswani16_ende_big/checkpoint_6_2000.pt (epoch 6 @ 2000 updates) (writing took 7.748462915420532 seconds)
| epoch 006 | loss 5.375 | nll_loss 3.851 | ppl 14.43 | wps 105156 | ups 0 | wpb 402417.835 | bsz 13150.676 | num_updates 2036 | lr 0.000509049 | gnorm 0.475 | clip 0.000 | oom 0.000 | loss
| epoch 006 | valid on 'valid' subset | loss 4.821 | nll_loss 3.064 | ppl 8.36 | num_updates 2036 | best_loss 4.82093
| saved checkpoint checkpoints/transformer_vaswani16_ende_big/checkpoint6.pt (epoch 6 @ 2036 updates) (writing took 7.87086820602417 seconds)
| WARNING: overflow detected, setting loss scale to: 8.0
| epoch 007 | loss 5.070 | nll_loss 3.517 | ppl 11.45 | wps 105713 | ups 0 | wpb 402428.516 | bsz 13155.935 | num_updates 2377 | lr 0.000594291 | gnorm 0.387 | clip 0.000 | oom 0.000 | loss
| epoch 007 | valid on 'valid' subset | loss 4.629 | nll_loss 2.850 | ppl 7.21 | num_updates 2377 | best_loss 4.62924
| saved checkpoint checkpoints/transformer_vaswani16_ende_big/checkpoint7.pt (epoch 7 @ 2377 updates) (writing took 7.967353820800781 seconds)
| WARNING: overflow detected, setting loss scale to: 16.0
| WARNING: overflow detected, setting loss scale to: 8.0
| epoch 008 | loss 4.875 | nll_loss 3.306 | ppl 9.89 | wps 105795 | ups 0 | wpb 402477.000 | bsz 13161.029 | num_updates 2717 | lr 0.000679282 | gnorm 0.355 | clip 0.000 | oom 0.000 | loss_
| epoch 008 | valid on 'valid' subset | loss 4.449 | nll_loss 2.677 | ppl 6.39 | num_updates 2717 | best_loss 4.44949
| saved checkpoint checkpoints/transformer_vaswani16_ende_big/checkpoint8.pt (epoch 8 @ 2717 updates) (writing took 7.882242679595947 seconds)
| WARNING: overflow detected, setting loss scale to: 8.0
| WARNING: overflow detected, setting loss scale to: 8.0
| epoch 009 | valid on 'valid' subset | loss 4.338 | nll_loss 2.567 | ppl 5.93 | num_updates 3000 | best_loss 4.33773
| saved checkpoint checkpoints/transformer_vaswani16_ende_big/checkpoint_9_3000.pt (epoch 9 @ 3000 updates) (writing took 7.749082565307617 seconds)
| epoch 009 | loss 4.744 | nll_loss 3.164 | ppl 8.97 | wps 105280 | ups 0 | wpb 402417.374 | bsz 13154.794 | num_updates 3057 | lr 0.000764274 | gnorm 0.346 | clip 0.000 | oom 0.000 | loss_
| epoch 009 | valid on 'valid' subset | loss 4.329 | nll_loss 2.558 | ppl 5.89 | num_updates 3057 | best_loss 4.32936
| saved checkpoint checkpoints/transformer_vaswani16_ende_big/checkpoint9.pt (epoch 9 @ 3057 updates) (writing took 7.789773464202881 seconds)
| WARNING: overflow detected, setting loss scale to: 16.0
| WARNING: overflow detected, setting loss scale to: 8.0
| WARNING: overflow detected, setting loss scale to: 4.0
| WARNING: overflow detected, setting loss scale to: 2.0
| WARNING: overflow detected, setting loss scale to: 1.0
| WARNING: overflow detected, setting loss scale to: 0.5
| WARNING: overflow detected, setting loss scale to: 0.25
| epoch 010 | loss 7.143 | nll_loss 5.919 | ppl 60.49 | wps 103749 | ups 0 | wpb 402502.988 | bsz 13146.860 | num_updates 3392 | lr 0.000848015 | gnorm 0.726 | clip 0.000 | oom 0.000 | loss
| epoch 010 | valid on 'valid' subset | loss 11.788 | nll_loss 11.226 | ppl 2395.81 | num_updates 3392 | best_loss 4.32936
| saved checkpoint checkpoints/transformer_vaswani16_ende_big/checkpoint10.pt (epoch 10 @ 3392 updates) (writing took 4.796957731246948 seconds)
| epoch 011 | loss 11.626 | nll_loss 11.070 | ppl 2149.65 | wps 106200 | ups 0 | wpb 402421.772 | bsz 13160.719 | num_updates 3734 | lr 0.000933507 | gnorm 0.480 | clip 0.000 | oom 0.000 |
| epoch 011 | valid on 'valid' subset | loss 14.099 | nll_loss 13.758 | ppl 13858.17 | num_updates 3734 | best_loss 4.32936
| saved checkpoint checkpoints/transformer_vaswani16_ende_big/checkpoint11.pt (epoch 11 @ 3734 updates) (writing took 4.8482091426849365 seconds)
| WARNING: overflow detected, setting loss scale to: 2.0
| WARNING: overflow detected, setting loss scale to: 1.0
| WARNING: overflow detected, setting loss scale to: 0.5
| epoch 012 | valid on 'valid' subset | loss 14.852 | nll_loss 14.363 | ppl 21064.62 | num_updates 4000 | best_loss 4.32936
| saved checkpoint checkpoints/transformer_vaswani16_ende_big/checkpoint_12_4000.pt (epoch 12 @ 4000 updates) (writing took 4.745401382446289 seconds)
| epoch 012 | loss 11.545 | nll_loss 10.977 | ppl 2015.82 | wps 104868 | ups 0 | wpb 402440.389 | bsz 13164.761 | num_updates 4073 | lr 0.000990998 | gnorm 0.604 | clip 0.000 | oom 0.000 |
| epoch 012 | valid on 'valid' subset | loss 15.443 | nll_loss 15.009 | ppl 32969.28 | num_updates 4073 | best_loss 4.32936

I trained the model with this:
# Train a model:
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 fairseq-train \
     data-bin/wmt16_en_de_bpe32k \
     --arch transformer_vaswani_wmt_en_de_big --share-all-embeddings \
     --optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0 \
     --lr-scheduler inverse_sqrt --warmup-init-lr 1e-07 --warmup-updates 4000 \
     --min-lr 1e-09 \
     --dropout 0.3 --weight-decay 0.0 --criterion label_smoothed_cross_entropy \
     --label-smoothing 0.1 \
     --max-tokens 3584 \
     --fp16 \
     --lr 1e-03 \
     --update-freq 16 \
     --max-update 9600 \
     --no-progress-bar --save-interval-updates 1000 --keep-interval-updates 20 \
     --save-dir checkpoints/transformer_wmt16_ende_big \

Please help me handle this question, thank you !


Create installers

As part of attracting non-technician players I'm suggest to add installers in releases instead of zipped build, it can be done with electron-builder. I have some experience with it btw. What do you think about it, @kvnneff?

I'm down.  If you want to set that up go for it!


[Docs] Add maintenance of OpenAssetIO-Test-CMake

What
Add docs covering maintenance of the OpenAssetIO-Test-CMake repository as relevant things change in OpenAssetIO (e.g. additional C++ headers).

We've simplified the Test-CMake project to narrow its testing focus, and therefore require little maintenance. It is also run on CI as part of integrations.yml, so we should detect breaking changes.  Closing.


added entry for Web based call control

Web based call control, communicates to server using  web socket, wrapped into Android app using WebView, uses port to access contacts from device

It'll look nicer on the site if you can keep the image at 1000x800 but show only the app, not the browser UI as well.

Thanks, @prikhi . I have update the image as you suggested

Thanks, site should be updated soon.


"Cannot read property 'apply' of undefined" when clicking on a button with $meteor.loginWithFacebook()

Hi there,
since a couple of days the promise behind $meteor.loginWithFacebook and $meteor.loginWithGoogle don't work anymore in one of my project and I don't know why.
Simple Button with a simple ng-clicked controller call that does $meteor.loginWithFacebook() produces this error message :
TypeError: Cannot read property 'apply' of undefined
    at null.loginWithFacebook (http://localhost:3000/packages/angular.js?d2db9bfa9c7bfe58b5c47cd57d99ed5e3c86f7de:1558:20)
    at Scope.$scope.loginWithFacebook (http://localhost:3000/app/client/views/login/login.controller.js?d5174ac68dde8434571098a376bd3b5c7f2ca741:61:21)
    at fn (eval at <anonymous> (http://localhost:3000/packages/angular_angular.js?0a328c602667881bb749eae26bbfccca25285a42:13345:15), <anonymous>:4:242)
    at callback (http://localhost:3000/packages/angular_angular.js?0a328c602667881bb749eae26bbfccca25285a42:23572:17)
    at Scope.$eval (http://localhost:3000/packages/angular_angular.js?0a328c602667881bb749eae26bbfccca25285a42:16012:28)
    at Scope.$apply (http://localhost:3000/packages/angular_angular.js?0a328c602667881bb749eae26bbfccca25285a42:16112:25)
    at HTMLButtonElement.<anonymous> (http://localhost:3000/packages/angular_angular.js?0a328c602667881bb749eae26bbfccca25285a42:23577:23)
    at HTMLButtonElement.jQuery.event.dispatch (http://localhost:3000/packages/jquery.js?1015953f785c9b76503e2ecb391507dce965f357:4691:9)
    at HTMLButtonElement.elemData.handle (http://localhost:3000/packages/jquery.js?1015953f785c9b76503e2ecb391507dce965f357:4359:28)(anonymous function) @ angular.js:12477(anonymous function) @ angular.js:9246Scope.$apply @ angular.js:16094(anonymous function) @ angular.js:23554jQuery.event.dispatch @ jquery.js:4665elemData.handle @ jquery.js:4333

I am using v.1.0.3.

I got the same error for loginWithPassword function, v.1.0.3

I got this resolved. @maxfriedmann, install accounts-password-client-side then include accounts-password-client-side.bundle.js and accounts-pase-client-side.bundle.js

Thanks @piotrraczynski

Hi,
I am getting the exact same error while trying $meteor.loginWithFacebook()
    at null.loginWithFacebook```

I am using all latest versions as well as included the package/JS files mentioned.

regards,
Vasanth

+1 @piotrraczynski

With the new Angular Meteor 1.3 release the way we work with Meteor accounts has improved a lot.
No need for Angular wrappers anymore.
Check it out and let me know how did it worked for you here or in the post's comments

Hi,
I am getting the same error  TypeError: Cannot read property 'apply' of undefined at null.loginWithFacebook ,  i am using angular-meteor 1.3.6 version..

@neosekar You don't need to use $meteor anymore.
You can just use Meteor.loginWithFacebook as described in the official meteor docs.
If you need to send a callback to the function use $bindToContext to be able to update your data reactively. Check the usage example in ReactiveContext docs.

If i use Meteor method like this
** Meteor.loginWithFacebook({},
function (error) {
if (error) {
return console.log(error);
}
});**
 i am getting error like this,  **TypeError: Meteor.loginWithFacebook is not a function**.

Can you post any working sample code of loginWIthFacebook in angular-meteor.

@neosekar have you add the accounts-facebook package?

I already added the list of packages
accounts-base          1.2.2  A user account system
accounts-facebook      1.0.6  Login service for Facebook accounts
accounts-password      1.1.4  Password support for accounts
autopublish            1.0.4  (For prototyping only) Publish the entire ...
blaze-html-templates   1.0.1  Compile HTML templates into reactive UI wi...
ecmascript             0.1.6 Compiler plugin that supports ES2015+ in a...
es5-shim               4.1.14  Shims and polyfills to improve ECMAScript...
insecure               1.0.4  (For prototyping only) Allow all database ...
jquery                 1.11.4  Manipulate the DOM using CSS selectors
meteor-base            1.0.1  Packages that every Meteor app needs
mobile-experience      1.0.1  Packages for a great mobile user experience
mongo                  1.1.3  Adaptor for using MongoDB and Minimongo ov...
service-configuration  1.0.5  Manage the configuration for third-party s...
session                1.1.1  Session variable
standard-minifiers     1.0.2  Standard minifiers used with Meteor apps b...
tracker                1.0.9  Dependency tracker to allow reactive callb...*
But nothing works. Can you post any working example of loginWithFacebook  ?

@neosekar let's do the opposite, can you please provide an non working example I can look at?
in a separate repo.
Thanks

Here i attached the repository url  .

Hey, anyone have the working example of loginwithfacebook(); pls post it..

Hey, loginwithfacebook();  still not working..

@neosekar there's a need to publish accounts-facebook in the same way that @idanwe as done with accounts-password here:
https://github.com/idanwe/accounts-base-client-side
https://github.com/idanwe/accounts-password-client-side
@neosekar do you want to try to publish this one yourself?  we can help

@neosekar take a look at https://github.com/idanwe/accounts-password-client-side/issues/2#issuecomment-162036663

Hi,
I'm also getting this exception did any one resolve this ?