Show the environment ID at the environment list page

What this PR does / why we need it:

use table instead of the list
show env ID

Which issue(s) this PR fixes:
Fixes #1238
Does this PR introduce a user-facing change?:

Show the environment ID at the environment list page

Nice. Thank you.
/approve


Transceivers aren't automatically created when generating answer for Plan-B Offer

Your environment.

Version: v3.0.19
Tools: Janus 0.7.2
Other Information - Golang version go1.13.7 linux/amd64, Ubuntu 18.04 (5.4.0-70-generic #78~18.04.1-Ubuntu)

What did you do?
Target was to connect to a Janus videoroom plugin and start receiving RTP packets.
In this case there were only one publisher in the room.
From here using the pion webrtc (v3.0.19) whenever a "joinandconfigure" message is sent to Janus following SDP is received.
v=0
o=-<PHONE_NUMBER>056160 1 IN IP4 <IP_ADDRESS>
s=VideoRoom 77
t=0 0
a=group:BUNDLE audio video data
a=msid-semantic: WMS janus
m=audio 9 UDP/TLS/RTP/SAVPF 111
c=IN IP4 <IP_ADDRESS>
a=sendonly
a=mid:audio
a=rtcp-mux
a=ice-ufrag:L1lB
a=ice-pwd:7qBnV/D85G5gH/C8ajwP0T
a=ice-options:trickle
a=fingerprint:sha-256 49:1D:14:A2:74:46:A0:62:12:A6:28:66:FF:C7:FF:C1:7B:23:66:6F:2F:E0:BD:7E:8D:E5:5E:0E:7F:45:65:9D
a=setup:actpass
a=rtpmap:111 opus/48000/2
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=ssrc:673801478 cname:janus
a=ssrc:673801478 msid:janus janusa0
a=ssrc:673801478 mslabel:janus
a=ssrc:673801478 label:janusa0
a=candidate:1 1 udp<PHONE_NUMBER> <IP_ADDRESS> 34965 typ host
a=candidate:4 1 udp<PHONE_NUMBER> <IP_ADDRESS> 34965 typ srflx raddr <IP_ADDRESS> rport 34965
a=end-of-candidates
m=video 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 <IP_ADDRESS>
a=sendonly
a=mid:video
a=rtcp-mux
a=ice-ufrag:L1lB
a=ice-pwd:7qBnV/D85G5gH/C8ajwP0T
a=ice-options:trickle
a=fingerprint:sha-256 49:1D:14:A2:74:46:A0:62:12:A6:28:66:FF:C7:FF:C1:7B:23:66:6F:2F:E0:BD:7E:8D:E5:5E:0E:7F:45:65:9D
a=setup:actpass
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtcp-fb:96 goog-remb
a=extmap:13 urn:3gpp:video-orientation
a=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=ssrc:3838312341 cname:janus
a=ssrc:3838312341 msid:janus janusv0
a=ssrc:3838312341 mslabel:janus
a=ssrc:3838312341 label:janusv0
a=candidate:1 1 udp<PHONE_NUMBER> <IP_ADDRESS> 34965 typ host
a=candidate:4 1 udp<PHONE_NUMBER> <IP_ADDRESS> 34965 typ srflx raddr <IP_ADDRESS> rport 34965
a=end-of-candidates
m=application 9 DTLS/SCTP 5000
c=IN IP4 <IP_ADDRESS>
a=sendrecv
a=sctpmap:5000 webrtc-datachannel 16
a=mid:data
a=ice-ufrag:L1lB
a=ice-pwd:7qBnV/D85G5gH/C8ajwP0T
a=ice-options:trickle
a=fingerprint:sha-256 49:1D:14:A2:74:46:A0:62:12:A6:28:66:FF:C7:FF:C1:7B:23:66:6F:2F:E0:BD:7E:8D:E5:5E:0E:7F:45:65:9D
a=setup:actpass
a=candidate:1 1 udp<PHONE_NUMBER> <IP_ADDRESS> 34965 typ host
a=candidate:4 1 udp<PHONE_NUMBER> <IP_ADDRESS> 34965 typ srflx raddr <IP_ADDRESS> rport 34965
a=end-of-candidates

Please note that the audio and video both are marked as 'sendonly'.
After setting this as remote description (SetRemoteDescription) and trying to create and answer (CreateAnswer(nil)) following is observed on the answer,
{answer v=0
o=-<PHONE_NUMBER>12244114<PHONE_NUMBER> IN IP4 <IP_ADDRESS>
s=-
t=0 0
a=fingerprint:sha-256 30:C3:B9:DF:7C:6E:50:11:17:ED:E5:BD:E3:DC:F2:4D:84:35:17:C3:4B:D7:9C:3D:10:D3:D6:B9:44:E9:AF:C3
a=group:BUNDLE audio video data
m=audio 9 UDP/TLS/RTP/SAVPF 111
c=IN IP4 <IP_ADDRESS>
a=setup:active
a=mid:audio
a=ice-ufrag:jzWwrOsXiSYejpVj
a=ice-pwd:fCSXAuOpTwsTlNKogVQEEwOMsaRPoUcR
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:111 opus/48000/2
a=inactive
m=video 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 <IP_ADDRESS>
a=setup:active
a=mid:video
a=ice-ufrag:jzWwrOsXiSYejpVj
a=ice-pwd:fCSXAuOpTwsTlNKogVQEEwOMsaRPoUcR
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtcp-fb:96 goog-remb
a=inactive
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 <IP_ADDRESS>
a=setup:active
a=mid:data
a=sendrecv
a=sctp-port:5000
a=ice-ufrag:jzWwrOsXiSYejpVj
a=ice-pwd:fCSXAuOpTwsTlNKogVQEEwOMsaRPoUcR
 0xc0001c8c60}

Please note that in the answer audio and video both are marked as 'inactive' which results in not receiving RTP data.
A way around is found by manipulating the remote SDP before setting the remote descriptor like following,
    remoteSDP := resp.Jsep["sdp"].(string)
	manipulatedSDP := strings.ReplaceAll(remoteSDP, "a=sendonly", "a=sendrecv")

In short if the remote SDP contains a=sendonly, in the answer it's always generating a=inactive.
What did you expect?
The answer should have created a=recvonly or a=sendrecv
What happened?
In this case the answer was always creating a=inactive

Thanks for getting back and really appreciate your feedback.
Yes, I am using a relatively older version of Janus which only supports Plan-B. But I also tried to explicitly create transceivers (just after creating new peerconnection) and that did not work either.
	 	if _, err = peerConnection.AddTransceiverFromKind(webrtc.RTPCodecTypeAudio); err != nil { 		panic(err) 	} else if _, err = peerConnection.AddTransceiverFromKind(webrtc.RTPCodecTypeVideo); err != nil { 		panic(err) 	}
But I tried the video-room example with the same Janus endpoint and got the expected SDP.
The difference between the example and my code is,

The example is trying to publish and I'm trying to receive.
Could there be any possibility that it might have something to do with creating Answers ?

Hey @mashroor-ax
Sorry for the delay, work/life has kept me really busy lately. Newer version of Janus do support Unified Plan, is it possible to upgrade?
I am going to remove Plan-B support entirely in v4. I don't have the bandwidth to make any changes to then unfortunately. https://github.com/pion/webrtc/issues/1914


[zh] Use appropriate localization(s) for term “namespace”

This is a Bug Report

In reading the k8s documentation, I found that a large number of documents translate namespace to 名字空间
在阅读k8s文档中，发现大量的文档将namespace 翻译成 名字空间
but in practice, people are more used to using 命名空间
而大家的实际使用过程中，更加习惯使用命名空间
There is a dispute here that
在此发生了争议，
https://github.com/kubernetes/website/pull/37665
In my view k8s issue, there is also a large amount of controversy, everyone's point of contention are 名字空间
在我查看 k8s 问题，也出现了大量的争议，大家的争议点都是 名字空间
https://github.com/kubernetes/website/pull/33179
https://github.com/kubernetes/website/issues/28459
https://github.com/kubernetes/website/issues/35798
Would there be these controversies if it were a 命名空间？People who raise these controversies are feeling doubtful about 名字空间
Regarding the reason why it was modified to a namespace, I think it might be as follows
关于为什么修改成名字空间的原因，我觉得可能如下
https://github.com/kubernetes/website/issues/28459#issuecomment-863623927

We had such discussions previously when drafting and reviewing the https://kubernetes.io/zh/docs/contribute/localization_zh/#术语对照 page. I personally prefer "名字空间”，because that is what namespace for ... a space for names. There are other voices arguing that "命名空间” is very popular in many articles or books.

In the k8s ecosystem, they all actually use the 命名空间
在k8s生态系统中，它们实际上都使用了  命名空间
例如，他们（包括CNCF项目)
For instance, they (Includes CNCF projects):

rancher
istio
volcano
kubeEdge

In addition, when I took the CKA certificate, the topic descriptions in it also used 命名空间
https://github.com/kubernetes/website/pull/37665#issuecomment-1301839124

Those discussion history can serve a background why we don't want to invest time on this issue again. We had discussions on this and we articulated the reason why "名字空间" is acceptable, or even a better option.

When deciding how to translate a term, we consider whether a translation is good not by how it is used in other fields or contexts. We were and we are still trying to find the best Chinese word for an English word. In other words, we honor the true facts rather than how others are doing.

作为严谨的科技工作者，我们的职责不是人云亦云，不是根据事物的流行度来做判断，而是在工作中（包括社区工作中）去重新审视所有即成事实的、已被认定的事物。这一过程并不是轻松的、写意的，而是天人交战的、绞尽脑汁的。毕竟，Kubernetes 的官方文档会是很多下游社区的信息源；毕竟，大家在社区中所做的贡献都要在阳光下晾晒。

In my opinion, it is not a matter of people's clouds
在我认为，这不是人云亦云的问题
This is not academic research; the role of official documentation is to be able to teach users, and it is using statements that are more acceptable to users that will keep them from getting confused.
这并不是学术研究，官方文档的作用在于能够教授使用者，而使用使用者更加容易接受的说法，才会让使用者不会感到困惑。

Problem:
Proposed Solution:
Page to Update:
https://kubernetes.io/...

/retitle [zh] Use appropriate localization(s) for term “namespace”

名字空间 feels fine to me

+1 @wenerme
One of reasons that you discussed much before is that you also know 名字空间 refers to namespace. The reverse derivation is also correct, no other options.
namespace = 名字空间, 命名空间
名字空间, 命名空间 = namespace

It's not necessary to force everyone to follow the old way. 命名空间 is the legacy and old way in my opinion.

In my opinion there is no consideration of ecological issues, for those who know, they are a thing. But for learners, newbies, it can be confusing for them
The existing textbook materials, are written in namespace, in the search engine query, query 名字空间, the exact match is very rare, the use of 名字空间 is not very short time, I check the commit, he submitted also more than two years, but we are still more accustomed to 命名空间
If you don't care about ecology, then it doesn't matter
It's like the unification of China into Mandarin, not because it sounds good, but because Mandarin is more widely used by people

You are not forcing everyone to use it, but you are forcing everyone to accept your solution, can you find the documentation for using 命名空间 in the documentation?

I basically agree with @wenerme
In the Chinese dictionary 辞海, the meanings of  名字 and 名称 are as follows:
名字： 1.人的名与字。 2.指姓名。 3.名称;名号。 4.命名;称其名。 5.犹名誉;名声。 6.即名词。
名称： 1.名号称谓。亦指事物名。"

In daily practical use, 名字 is mainly used for people in general use, while 名称 is mainly used for affairs. I feel like this is where everyone gets confused about 名字空间.
在日常实际使用中，名字 在一般使用中多用于人或者拟人化的事务，而名称则多用于物品或事务。我感觉这也是大家对于名字空间的困惑所在，会下意识感到有点别扭。
So I tend to translate 名字空间 to 名称空间.
因此我倾向于将 名字空间 翻译为 名称空间。

Any context will create a namespace, but if you mean the Namespace in kube, I prefer 名字空间
https://github.com/wenerme/wener/blob/923ca381eb26fc7a9a917cf132ab4aa64e5436e5/notes/dev/dict.md#namespace-翻译-namespace

When we localize a reference to a Linux kernel process namespace, what Chinese word should we prefer?

Kubernetes product:
Azure Kubernetes Service (AKS) - 命名空间
https://learn.microsoft.com/zh-cn/azure/aks/concepts-clusters-workloads#namespaces
Amazon EKS - 命名空间
https://docs.aws.amazon.com/zh_cn/eks/latest/userguide/sample-deployment.html
Red Hat OpenShift - 项目和命名空间
https://access.redhat.com/documentation/zh-cn/openshift_container_platform/4.3/html/authentication/rbac-projects-namespaces_using-rbac#doc-wrapper
CNCF projects:
rancher
https://docs.ranchermanager.rancher.io/zh/how-to-guides/new-user-guides/manage-namespaces
istio
https://istio.io/latest/zh/docs/tasks/security/authentication/authn-policy/#namespace-wide-policy
volcano
https://volcano.sh/zh/docs/installation/#步骤-1
kubeEdge
https://kubeedge.io/zh/docs/setup/keadm_zh/#master
CNCF Ambassadors:
Jimmy Song - blog: 不同命名空间中的服务要绑定到同一个网关中的情况在实际情况下经常出现，而入口网关无法在多个命名空间中共享；
https://jimmysong.io/blog/why-gateway-api-is-the-future-of-ingress-and-mesh/
Lei Zhang - special column: 其解释了如何利用命名空间（Namespace）这一概念来组织 K8s 租户的资源。
https://www.infoq.cn/article/fsmwwgyknsgwpegjho4g
sig-docs-zh-reviews:
chenxuc
https://github.com/kubernetes/website/pull/37665
kubernetes Contributor:
yinyicao
https://github.com/kubernetes/website/pull/33179
linruikai
https://github.com/kubernetes/website/issues/35798
google search results(命名空间>名字空间):
k8s+"命名空间" - About 246,000 results
https://www.google.com/search?q=k8s+"命名空间"
k8s "名字空间" - About 20,300 results
https://www.google.com/search?q=k8s+"名字空间"

All discussions are based on references, rather than creating a vocabulary without any expert endorsement and then explaining it.
In official commercial Kubernetes product(Azure Kubernetes Service,Amazon EKS,Red Hat OpenShift), in CNCF projects(rancher,istio,volcano,kubeEdge), in K8S experts(CNCF Ambassadors:Jimmy Song,Lei Zhang),in sig-docs-zh-reviews(chenxuc) ,in enthusiastic contributors(yinyicao,linruikai,aimuz),All are used correct(命名空间).

/triage accepted

/language zh

I wasn't insisting that '名字空间' is the only acceptable translation. I also shared my reason why I myself believe '名字空间' could be a better choice. I'm NOT forcing others to accept this opinion, and I don't accept that '名字空间' is a wrong translation. With all those background information provided, we had achieved a consensus -- both are acceptable in the docs. In other words, WE AGREED TO DISAGREE.
Because this word translation doesn't hurt that much for people to learn Kubernetes, we always tried to redirect contributors to other more valuable fixes. That is the reason I closed the PR which prefers one word to another.
If we can achieve the consensus -- both are acceptable -- again, we can close this issue and add a comment to localization style guide.

To insist that the argument is correct, provide references, not unfounded explanations.
On the premise that there are reference materials, replace all '名字空间' with '命名空间', I believe no one will have an opinion.
You can even see that there is no PR that needs to change '名字空间' to '命名空间' only PR that replaces  '命名空间' with '名字空间'.
It can even be seen that there is no PR to change '名字空间' to '命名空间', only PR to replace  '命名空间' with '名字空间'.
No namespace('名字空间') is used in official commercial Kubernetes product(Azure Kubernetes Service,Amazon EKS,Red Hat OpenShift), in CNCF projects(rancher,istio,volcano,kubeEdge), in K8S experts(CNCF Ambassadors:Jimmy Song,Lei Zhang) , so it is a new vocabulary('名字空间') created by myself, without expert approval, and cannot guarantee its correctness.

This is not academic research; the role of official documentation is to be able to teach users, and it is using statements that are more acceptable to users that will keep them from getting confused.

In fact, this issue was raised only by readers who were confused.
I do not support random creation of new words to confuse readers,
If you can create new words at will, why not use '名空间','姓名空间','名称空间'.
The first focus of localization is correctness.
This is a wrong translation('名字空间'),please adjust it to be correct('命名空间').
Google Kubernetes Engine (GKE): 命名空间
https://cloud.google.com/kubernetes-engine/docs/how-to/restrict-resources-access-by-namespace?hl=zh-cn

Because this word translation doesn't hurt that much for people to learn Kubernetes, we always tried to redirect contributors to other more valuable fixes. That is the reason I closed the PR which prefers one word to another.

If you think this is of little value, why are namespaces being replaced with namespaces in the documentation
If both nouns can be used, why does only one usage exist in the documentation? So who modified him to merge in such worthless ones?
If it was just not worth much to begin with, then why recognize you would modify him? Change it to what you think is the right way?

I looked through the linux kernel documentation and found the following information
In the Linux kernel localization, the process namespace has not been translated for localization yet, but there is another namespace in Linux that I think can be used as a reference
Symbol Namespaces
https://www.kernel.org/doc/html/latest/core-api/symbol-namespaces.html
https://www.kernel.org/doc/html/latest/translations/zh_CN/core-api/symbol-namespaces.html

I searched for related Linux-related books

Professional Linux Kernel Architecture - 《深入Linux内核架构》 译者: 郭旭
In the Chinese version 1.3 内核的组成部分 5

The Linux Programming Interface: A Linux and UNIX System Programming Handbook - 《Linux/UNIX系统编程手册》译者: 孙剑 许从年 董健 / 孙余强 郭光伟 陈舸

These books are some of the highest rated Linux related books

In the course of reviewing the books, no relevant books were found that used the 名字空间

If both uses were actually allowed, then the document would not contain only 名字空间
Another quote from @windsonsea

I call my father as daddy, papa, old man, Mr.Y.

If both uses were really allowed, then PR should not be negated by way of priority.
When you change 命名空间 to 名字空间, don't negate other PRs in a worthless way. please don't double standard.

Use 名字空间 more, may attract more newer doc and book to use it if they feel better about it, how about just let this doc use 名字空间.

Agree with 名称空间

In the case of existing sources that are using 命名空间, I think there is no need to check your own correctness and should avoid causing more disturbance to the reader, especially for noun changes where you feel you are correct and you get satisfaction inside, but have not considered whether the reader buys it.
Whether the upstream or downstream material is using 命名空间, there should be no idiosyncrasy in translation.
Another word about the value of @tengqm , I think as long as there are contributors involved, it is valuable, it may not be valuable to you, but without value then contributors do not submit relevant questions.
In addition, I would like to share a quote that I think is good
https://www.figma.com/resources/learn-design/research/#firsthand-research

Stop looking at what other companies are doing. Start spending more time with existing/potential customers.


sparse: fix alist format by including padding

In #2 it was pointed that the alist format used by ldpc-toolbox is incorrect because it does not include zeros for padding of irregular codes, as required by MacKay's original definition.
This fixes the problem in the following way:

The SparseMatrix::from_alist function supports reading both the format with zero padding and the format without zero padding.

The SparseMatrix::write_alist and SparseMatrix::alist functions now include zero padding. There are variants of these functions called write_alist_no_padding and alist_no_padding which omit the zero padding.

Additionally, the whitespace at the end of each line generated by write_alist has been removed.

@page0egap let me know if this looks good.

line 315-316, the comments are wrong. alist_no_padding does not include padding.
I does not find other errors. For myself, it's able to parse irregular ldpc's alist format file, for example CCSDS(N=128, M=64). I think it solves my problem.

In file sparse.rs, in line 315-316, the comments are wrong

Good catch! A copy & paste mistake. I've just fixed this and force-pushed.


IndexOutOfRangeException while compiling big project after upgrading to Bridge 16.5

We recently upgraded our big project (around 1800 files) to bridge 16.5 (was 15.x).
Here is a part of output with trace level logging
  2017-11-24T19:22:13:731 	Info		Building syntax tree done
  2017-11-24T19:22:13:731 	Trace		Loaded assembly: ICSharpCode.NRefactory.Cecil, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=d4bfe873e7598c49
  2017-11-24T19:22:13:732 	Info		Assembly definition to references...
  2017-11-24T19:22:13:733 	Trace			Loading AssemblyDefinition Bridge ...
  2017-11-24T19:22:14:467 	Trace			Loading AssemblyDefinition done
  2017-11-24T19:22:14:467 	Trace			Loading AssemblyDefinition Bridge.Html5 ...
  2017-11-24T19:22:14:494 	Trace			Loading AssemblyDefinition done
  2017-11-24T19:22:14:494 	Trace			Loading AssemblyDefinition Bridge.WebGL ...
  2017-11-24T19:22:14:496 	Trace			Loading AssemblyDefinition done
  2017-11-24T19:22:14:496 	Trace			Loading AssemblyDefinition Bridge.jQuery2 ...
  2017-11-24T19:22:14:502 	Trace			Loading AssemblyDefinition done
  2017-11-24T19:22:14:502 	Trace			Loading AssemblyDefinition UnityEngine ...
  2017-11-24T19:22:14:529 	Trace			Loading AssemblyDefinition done
  2017-11-24T19:22:14:530 	Info		Assembly definition to references done
  2017-11-24T19:22:18:047 	Error		Bridge.NET Compiler error: System.IndexOutOfRangeException: Index was outside the bounds of the array.
    at Bridge.Translator.PreconverterFixer.VisitInvocationExpression (ICSharpCode.NRefactory.CSharp.InvocationExpression invocationExpression) [0x0007c] in <fdc967ae33394a39a77018cf85281675>:0
    at ICSharpCode.NRefactory.CSharp.InvocationExpression.AcceptVisitor[T] (ICSharpCode.NRefactory.CSharp.IAstVisitor`1[S] visitor) [0x00000] in <fe4e92d250774e63a7170b739d729067>:0
    at Bridge.Translator.PreconverterFixer.VisitChildren (ICSharpCode.NRefactory.CSharp.AstNode node) [0x00018] in <fdc967ae33394a39a77018cf85281675>:0
    at ICSharpCode.NRefactory.CSharp.DepthFirstAstVisitor`1[T].VisitIfElseStatement (ICSharpCode.NRefactory.CSharp.IfElseStatement ifElseStatement) [0x00000] in <fe4e92d250774e63a7170b739d729067>:0
    at ICSharpCode.NRefactory.CSharp.IfElseStatement.AcceptVisitor[T] (ICSharpCode.NRefactory.CSharp.IAstVisitor`1[S] visitor) [0x00000] in <fe4e92d250774e63a7170b739d729067>:0
    at Bridge.Translator.PreconverterFixer.VisitChildren (ICSharpCode.NRefactory.CSharp.AstNode node) [0x00018] in <fdc967ae33394a39a77018cf85281675>:0
    at ICSharpCode.NRefactory.CSharp.DepthFirstAstVisitor`1[T].VisitBlockStatement (ICSharpCode.NRefactory.CSharp.BlockStatement blockStatement) [0x00000] in <fe4e92d250774e63a7170b739d729067>:0
    at ICSharpCode.NRefactory.CSharp.BlockStatement.AcceptVisitor[T] (ICSharpCode.NRefactory.CSharp.IAstVisitor`1[S] visitor) [0x00000] in <fe4e92d250774e63a7170b739d729067>:0
    at Bridge.Translator.PreconverterFixer.VisitChildren (ICSharpCode.NRefactory.CSharp.AstNode node) [0x00018] in <fdc967ae33394a39a77018cf85281675>:0
    at ICSharpCode.NRefactory.CSharp.DepthFirstAstVisitor`1[T].VisitConstructorDeclaration (ICSharpCode.NRefactory.CSharp.ConstructorDeclaration constructorDeclaration) [0x00000] in <fe4e92d250774e63a7170b739d729067>:0
    at ICSharpCode.NRefactory.CSharp.ConstructorDeclaration.AcceptVisitor[T] (ICSharpCode.NRefactory.CSharp.IAstVisitor`1[S] visitor) [0x00000] in <fe4e92d250774e63a7170b739d729067>:0
    at Bridge.Translator.PreconverterFixer.VisitChildren (ICSharpCode.NRefactory.CSharp.AstNode node) [0x00018] in <fdc967ae33394a39a77018cf85281675>:0
    at ICSharpCode.NRefactory.CSharp.DepthFirstAstVisitor`1[T].VisitTypeDeclaration (ICSharpCode.NRefactory.CSharp.TypeDeclaration typeDeclaration) [0x00000] in <fe4e92d250774e63a7170b739d729067>:0
    at Bridge.Translator.PreconverterFixer.VisitTypeDeclaration (ICSharpCode.NRefactory.CSharp.TypeDeclaration typeDeclaration) [0x0003f] in <fdc967ae33394a39a77018cf85281675>:0
    at ICSharpCode.NRefactory.CSharp.TypeDeclaration.AcceptVisitor[T] (ICSharpCode.NRefactory.CSharp.IAstVisitor`1[S] visitor) [0x00000] in <fe4e92d250774e63a7170b739d729067>:0
    at Bridge.Translator.PreconverterFixer.VisitChildren (ICSharpCode.NRefactory.CSharp.AstNode node) [0x00018] in <fdc967ae33394a39a77018cf85281675>:0
    at ICSharpCode.NRefactory.CSharp.DepthFirstAstVisitor`1[T].VisitNamespaceDeclaration (ICSharpCode.NRefactory.CSharp.NamespaceDeclaration namespaceDeclaration) [0x00000] in <fe4e92d250774e63a7170b739d729067>:0
    at ICSharpCode.NRefactory.CSharp.NamespaceDeclaration.AcceptVisitor[T] (ICSharpCode.NRefactory.CSharp.IAstVisitor`1[S] visitor) [0x00000] in <fe4e92d250774e63a7170b739d729067>:0
    at Bridge.Translator.PreconverterFixer.VisitChildren (ICSharpCode.NRefactory.CSharp.AstNode node) [0x00018] in <fdc967ae33394a39a77018cf85281675>:0
    at ICSharpCode.NRefactory.CSharp.DepthFirstAstVisitor`1[T].VisitSyntaxTree (ICSharpCode.NRefactory.CSharp.SyntaxTree unit) [0x00000] in <fe4e92d250774e63a7170b739d729067>:0
    at ICSharpCode.NRefactory.CSharp.SyntaxTree.AcceptVisitor[T] (ICSharpCode.NRefactory.CSharp.IAstVisitor`1[S] visitor) [0x00000] in <fe4e92d250774e63a7170b739d729067>:0
    at Bridge.Translator.Translator.Preconvert (Bridge.Translator.MemberResolver resolver, Bridge.Contract.IAssemblyInfo config) [0x00051] in <fdc967ae33394a39a77018cf85281675>:0
    at Bridge.Translator.Translator.Translate () [0x001a3] in <fdc967ae33394a39a77018cf85281675>:0
    at Bridge.Translator.TranslatorProcessor.Process () [0x00006] in <fdc967ae33394a39a77018cf85281675>:0
    at Bridge.Builder.Program.Main (System.String[] args) [0x0007d] in <3e59cf4b0f4740a587c5d5476c539ab9>:0

Obviously not that much info here. Is there any way we can track down exact cs file that is causing the problem?

For simple test project output is:
  2017-11-24T19:38:21:323 	Info		Assembly definition to references done
  2017-11-24T19:38:22:237 	Info		Inspecting types...
  2017-11-24T19:38:22:238 	Trace		Visiting syntax tree /Users/stopiccot/...

So it looks like something is crashing in "Inspecting types..." step?

Hello, and yes, the information given by the output log is pretty lost in precision. Anyway, for you got lots of files building there, I would expect a needle in a huge haystack, but this is totally an use case we should review and support in bridge, an edge case that used to work in 15, and I wonder since which bridge version it started.
Is it feasible for you to try walking back the minor version releases of bridge and identify exactly between which versions this issue begun? That would help a lot in identifying the culprit change.
If pinpoint debugging is not a problem for you, I suggest you to build bridge from sources, building in Debug mode and replacing the DLLs within the project. You can get to instrument bridge builder if you have the translator's and all built in debug mode and add AttachDebugger to your Bridge.targets file.
Then with Bridge own project open, you can run the build from your big project, then once it runs, it will ask you for a Visual Studio instance to attach the debugger to -- then you choose the Bridge one, where it will (hopefully) stop right in the call that breaks during your build. Then you'll have a full-context stack trace with the exact file and sentence being translated at that time.
Well, but I would understand if you deem this low level debug too much. Just too hard for us to guess what's wrong with the information provided. From the point you identify the line in your code it fails it would probably be very easy to reproduce it in a deck which we can work on the fix.

We created pull request with changes which help us to identify the problem
https://github.com/bridgedotnet/Bridge/pull/3288
You can try the code form the branch or wait for next public release and paste Bridge log here if the problem will be reproducible still.

That's great. Where I can get instructions how to build Bridge with Visual Studio for Mac on macOS?

I've build Bridge.NET by cherry-picking a97633 to master so it would fail the same way as 16.5 packages from nuget by with additional trace logs. Here is what I get:
2017-11-27T16:34:17:771 	Warning		VisitInvocationExpression fails in /Users/.../Infrastructure/Facebook/FacebookException.cs ((Line 35, Col 17)): Index was outside the bounds of the array.
2017-11-27T16:34:17:772 	Warning		VisitInvocationExpression fails in /Users/.../Infrastructure/Facebook/FacebookException.cs ((Line 37, Col 1)): Index was outside the bounds of the array.
2017-11-27T16:34:17:773 	Warning		VisitInvocationExpression fails in /Users/.../Infrastructure/Facebook/FacebookException.cs ((Line 38, Col 1)): Index was outside the bounds of the array.
2017-11-27T16:34:17:773 	Warning		VisitInvocationExpression fails in /Users/.../Infrastructure/Facebook/FacebookException.cs ((Line 39, Col 1)): Index was outside the bounds of the array.

And here is a part of FacebookException.cs
using System;
using Facebook.Unity;
using System.Collections.Generic;
using System.Linq;

namespace Test.Infrastructure.Facebook
{
    public class FacebookException : Exception
    {
        private IResult _innerResponse;

        private string _type;
        public string Type { get { return _type; } }

        private string _facebookMessage;
        public string FacebookMessage { get { return _facebookMessage; } }

        private string _userMessage;
        public string UserMessage { get { return _userMessage; } }

        public FacebookAuthError AuthError { get; private set; }

        public FacebookApiError ApiError { get; private set; }

        public bool WasCancelled { get { return (_innerResponse != null) ? _innerResponse.Cancelled : false; } }

        public FacebookException(IResult result)
            : base(result.Error)
        {
            Log.Debug(this, "Facebook error result {0}", result.RawResult);

            _innerResponse = result;

            IDictionary<string, object> errorDictionary;
            if (_innerResponse.ResultDictionary.TryGetValue("error", out errorDictionary))
            {
                errorDictionary.TryGetValue("type", out _type);
                errorDictionary.TryGetValue("message", out _facebookMessage);
                errorDictionary.TryGetValue("error_user_msg", out _userMessage);

                object codeObject;
                int code;
                if (errorDictionary.TryGetValue("code", out codeObject))
                {
                    code = Convert.ToInt32(codeObject);
                    if (code >= 200 && code < 300)
                    {
                        ApiError = FacebookApiError.ApiPermissionError;
                    }
                    else
                    {
                        ApiError = ApiErrorMap.ContainsKey(code) ? ApiErrorMap[code] : FacebookApiError.Unknown;
                    }
                }

                if (errorDictionary.TryGetValue("error_subcode", out codeObject))
                {
                    code = Convert.ToInt32(codeObject);
                    AuthError = AuthErrorMap.ContainsKey(code) ? AuthErrorMap[code] : FacebookAuthError.Unknown;
                }
            }
        }

so all errors are related to TryGetValue calls, but not all of them. Lines 43 and 56 do not emit any errors. For now I don't understand why some calls are invalid and other ones are not. Any insights on this?

Maybe if you can stub out the references and draw this example in a simple deck we would be able to help you further.

Are you sure that your source code is valid? I see compilation errors 'cannot convert from out string to out object'
https://dotnetfiddle.net/PfSdNr

Oh... that makes complete sense now. The thing is it's a Unity3D code being converted and probably old mono runtime used by Unity3D allows such code but Bridge doesn't. On the other hand Bridge doesn't spot this error while parsing source code too)

Bridge relies on msbuild to validate source code. So, in your case compilation in Visual Studio should fails (before Bridge compilation). At least for me, Visual Studio shows that source code is not valid and Bridge compilation step is not invoked at all because it fails on msbuild step.

We are using Visual Studio for Mac for our Bridge project which probably explains it. But correct me if I'm wrong starting from Mono 5 also uses Roslyn to compile the code so it should also fail. Anyway for now it's easier to correct code a bit.

Hey, I think this issue is no longer an issue, is it? The error was identified so, what's originally reported here is no longer actual, right? Let me know if I'm wrong and we'll reopen the issue! :)


Typings Issue In Src Code

Error/Warning
import formValuesToCreateProductMapper
Argument of type '{ title: any; handle: any; status: any; description: any; thumbnail: any; collection_id: any; type: { id: any; value: any; } | undefined; images: any; options: { title: string; }[]; tags: any; variants: { title: any; ... 7 more ...; material: any; }[]; ... 8 more ...; discountable: boolean; }' is not assignable to parameter of type 'AdminPostProductsReq'.
  Types of property 'variants' are incompatible.
    Type '{ title: any; allow_backorder: any; manage_inventory: any; sku: any; ean: any; inventory_quantity: number | undefined; options: { value: string; }[]; prices: any; material: any; }[]' is not assignable to type 'ProductVariantReq[]'.
      Type '{ title: any; allow_backorder: any; manage_inventory: any; sku: any; ean: any; inventory_quantity: number | undefined; options: { value: string; }[]; prices: any; material: any; }' is not assignable to type 'ProductVariantReq'.
        Types of property 'inventory_quantity' are incompatible.
          Type 'number | undefined' is not assignable to type 'number'.
            Type 'undefined' is not assignable to type 'number'.ts(2345)
            ```

 Location: src/domain/products/new.tsx

 createProduct.mutate(formValuesToCreateProductMapper(newData), {...

Fixed in recent release of admin.


Complete transfer of ownership - Fixes #39

Pull Request (PR) description
This PR completes the transfer of ownership of the module to the DSC Community by updating all licenses and references.
This Pull Request (PR) fixes the following issues

Fixes #39

Task list

[x] Added an entry under the Unreleased section of the change log in the CHANGELOG.md.
Entry should say what was changed, and how that affects users (if applicable).
[x] Resource documentation added/updated in README.md in resource folder.
[ ] Resource parameter descriptions added/updated in schema.mof
and comment-based help.
[ ] Comment-based help added/updated.
[ ] Localization strings added/updated in all localization files as appropriate.
[ ] Examples appropriately added/updated.
[ ] Unit tests added/updated. See DSC Resource Testing Guidelines.
[ ] Integration tests added/updated (where possible). See DSC Resource Testing Guidelines.
[ ] New/changed code adheres to DSC Resource Style Guidelines and Best Practices.

@gaelcolas - would you mind reviewing?

This change is

@gaelcolas - can you review this one? I'd like to have it through before I migrate it to the new structure.


Exportable configuration and testing tools

Now that pipeable operators are available and becoming more popular, there is a lot of opportunity for third parties to create modules that export pipeable operators. I think this is desirable, and I think that a few of things can be done by rxjs to support this better.
Currently, if you want to create a pipeable operator module you will have to create your own TypeScript configuration, linting, building, and set up testing support for yourself if you want to do it. These are built into rxjs, but they could function as separate modules -- i.e. the compiling and testing scripts could be separate modules imported into rxjs and used by other libraries.
Right now if you want to create a pipeable operator module you can create a tsconfig like this:
{
    "extends": "./node_modules/rxjs/src/tsconfig.json",
    "files": ["index.ts"],
    "compilerOptions": {
        "outDir": "dist"
    }
}

This isn't bad, and actually it would be great to see more things like this that would simplify third party creation of pipeable operators as separate modules that would confirm to rxjs's own development standards and provide consistency across a lot of different modules.
Here is what I think would help:

Make tsconfig.json a bit easier to import
Export tslint.json or create a module for it.
Create a module for testing dependencies and scripts. Otherwise third party devs have to set this up themselves which creates a barrier for testing which makes it less likely.
Create a module for building dependencies and scripts. Same reasoning as point 2.
Documentation

A template would also help, but my imaginary project would be able to do something like
// package.json
{
  "dependencies": {
    "rxjs": "*"
  },
  "devDependencies": {
    "@reactivex-scripts/build": "*",
    "@reactivex-scripts/lint": "*",
    "@reactivex-scripts/test": "*"
  },
  "scripts": {
    "build": "rxjs-scripts-build src",
    "lint": "rxjs-scripts-lint src",
    "test": "rxjs-scripts-test spec",
  }
}

Then you would just need the source and tests without much additional setup.

3 is covered by https://github.com/ReactiveX/rxjs/issues/3231 (probably can include 5 as well), but except those I don't see reason for 1 / 2 should be handled by core. Core's configuration is highly tamed for producing multiple library bundle, not intended to be used for common source of third party. Moreover, it is good to share common compilation rules, it is not something enforced. Since this is JS anyway, any 3rd party can choose own rules or configurations.
There's some discussions around bootstrapping for custom and I think this is also falls into those category, while I think that's also territory of 3rd party modules or so. I don't think core lib itself will ship something included in library as ooto for these kind. One day we achieve monorepo config we could have those 3rd party pkg comes into single repo though.
I'm closing this issues for now, as we are targeting testing / other ergonomics enhancement in existing issues. Once we have those things ready, we can discuss remaning things separately.


Thoughts on refactoring pi3d_demos version

Hi @helgeerbe you've done some great stuff here. I've started to think about refactoring PictureFrame2020 (not just because of the bad 2020 karma) I had intended to break it into different functional classes very much along the lines you have but I'm not sure whether to just adopt your code as it is and add in the 'improvements' which have happened since you diverged. If I can keep it compatible with Home Assistant but still work without it - the main thing is that Wolfgang's setup will work (almost) unaltered; there's a lot of really good work in his picture frame blog and the less he has to re-write the better!
One thing has occurred to me (I've only just started to think about this so more will appear when I actually start to hack code):
make the file_list (and associated date, GPS, exif etc data) into an sqlite database see https://github.com/pi3d/pi3d_demos/discussions/41#discussioncomment-268838
PS you might want to enable github discussions and transfer this as it's not really an issue.

Moved to discussions https://github.com/helgeerbe/picture_frame/discussions/5 and closed.

Moved to discussions https://github.com/helgeerbe/picture_frame/discussions/5 and closed.


Add Javanese Script for jav-java

Originally posted in forum
https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/tesseract-ocr/8r8YOQgTBT4/xHpCTp9DAwAJ
From: Christopher Imantaka Halim

> Hi,
>
> I want to develop an OCR for Javanese Script / Aksara.
> https://en.wikipedia.org/wiki/Javanese_script
>
> Plan on using Tesseract version 4.0
> I've read the wiki but somehow got confused.
>
> What do I need to prepare, to start the bare minimum training process? (for Tesseract 4.0)
> In some other thread someone said that training using image files are not supported yet.
> Also found out that box file/tiff pairs are not supported also.
> (I did try making one box file, using this online tool: https://pp19dd.com/tesseract-ocr-chopper/)
>
> Do we have an example of the training "inputs" somewhere on the github projects?
>
> Sorry if this is a stupid question, I'm a newbie. :)
>
> Thanks before

Collect training text in Javanese script (Unicode). You will need large number of lines, 500,000 or so to train from scratch.  Or, if you can identify a current language/script supported by tesseract which is similar, then you can train by replacing layer, Try to get representative training text of about 50000 lines with 50 words each.

Collect unicode fonts which can correctly render the above text. The more fonts you have the better it will be.

Collect word frequency lists in Javanese script.

Preferably use the linux platform for doing training.

See https://github.com/tesseract-ocr/tesseract/wiki/TrainingTesseract-4.00#training-just-a-few-layers

You can try this as it will be faster than training from scratch.
Please post links to Javanese script related resources below.
If there is a transliterator which convertes Javanese in Latin script to Javanese script, that can be used for converting the files for lang jav as a start.

Do we have an example of the training "inputs" somewhere on the github projects?

See
https://github.com/tesseract-ocr/langdata/tree/master/jav
https://github.com/tesseract-ocr/langdata/blob/master/README.md

jav is Javanese language in Latin script.
zat kasebut lan kanthi Kategori:Tokoh ing user:OffsBlink para pedunung PL09Puryono| kaya désa
2006 90%; sisih wiwit dan papan wilayah Delengen 5 || ! Wétan, Cathetan € sawijining | saged
amarga Cathetan jaba saka Dominique jiwa. ingkang User:ZorroIII Indonesia 1] langkung NGC

He wants it in Javanese script

The Javanese script, natively known as Aksara Jawa (ꦲꦏ꧀ꦱꦫꦗꦮaksarajawa) and Hanacaraka (ꦲꦤꦕꦫꦏhanacaraka), is an abugida developed by the Javanese people to write several Austronesian languages spoken in Indonesia, primarily the Javanese language and an early form of Javanese called Kawi, as well as Sanskrit, an Indo-Aryan language used as a sacred language throughout Asia. The Javanese script is a descendant of the Brahmi script and therefore has many similarities with the modern scripts of South India and Southeast Asia. The Javanese script, along with the Balinese script, is considered the most elaborate and ornate among Brahmic scripts of Southeast Asia.[1]

This might be similar to Thai/Khmer - could try using that to train from.

http://unicode.org/udhr/d/udhr_jav_java.html
Universal Declaration of Human Rights - Javanese (Javanese)

https://jv.wikipedia.org/wiki/Parembugan:Joko_Widodo
Most of Javanese wikipedia seems to be in Latin script.

https://r12a.github.io/scripts/javanese/

Did you unpack jav from best/fast?

Hello, thanks a lot for your help. I appreciate it.
Thanks again for providing the links to javanese scripts.
Sorry, I always thought that we need images as training data, but it is not the case for Tesseract 4.0.
:)
Another question, do we have to collect all 500,000 text lines before begin the training?
Can I, lets say, collect only 100 lines, then start the training?
(also well aware that the result may not be good, like overfitting)

100 lines will work only for fine tuning. But you can give it a try to get familiar with training process.

Did you unpack jav from best/fast?

@amitdo I had only looked at langdata. I checked just now after your post. The unicharset in both is in Latin Script only. See below for tessdata_fast version
94
NULL 0 Common 0
Joined 7 0,255,0,255,0,0,0,0,0,0 Latin 1 0 1 Joined	# Joined [4a 6f 69 6e 65 64 ]a
|Broken|0|1 f 0,255,0,255,0,0,0,0,0,0 Common 2 10 2 |Broken|0|1	# Broken
J 5 0,255,0,255,0,0,0,0,0,0 Latin 73 0 3 J	# J [4a ]A
E 5 0,255,0,255,0,0,0,0,0,0 Latin 58 0 4 E	# E [45 ]A
N 5 0,255,0,255,0,0,0,0,0,0 Latin 61 0 5 N	# N [4e ]A
I 5 0,255,0,255,0,0,0,0,0,0 Latin 66 0 6 I	# I [49 ]A
F 5 0,255,0,255,0,0,0,0,0,0 Latin 84 0 7 F	# F [46 ]A
R 5 0,255,0,255,0,0,0,0,0,0 Latin 74 0 8 R	# R [52 ]A
M 5 0,255,0,255,0,0,0,0,0,0 Latin 63 0 9 M	# M [4d ]A
A 5 0,255,0,255,0,0,0,0,0,0 Latin 60 0 10 A	# A [41 ]A
G 5 0,255,0,255,0,0,0,0,0,0 Latin 65 0 11 G	# G [47 ]A
: 10 0,255,0,255,0,0,0,0,0,0 Common 12 6 12 :	# : [3a ]p
P 5 0,255,0,255,0,0,0,0,0,0 Latin 67 0 13 P	# P [50 ]A
L 5 0,255,0,255,0,0,0,0,0,0 Latin 72 0 14 L	# L [4c ]A
T 5 0,255,0,255,0,0,0,0,0,0 Latin 59 0 15 T	# T [54 ]A
U 5 0,255,0,255,0,0,0,0,0,0 Latin 68 0 16 U	# U [55 ]A
B 5 0,255,0,255,0,0,0,0,0,0 Latin 76 0 17 B	# B [42 ]A
, 10 0,255,0,255,0,0,0,0,0,0 Common 18 6 18 ,	# , [2c ]p
K 5 0,255,0,255,0,0,0,0,0,0 Latin 75 0 19 K	# K [4b ]A
H 5 0,255,0,255,0,0,0,0,0,0 Latin 62 0 20 H	# H [48 ]A
D 5 0,255,0,255,0,0,0,0,0,0 Latin 71 0 21 D	# D [44 ]A
S 5 0,255,0,255,0,0,0,0,0,0 Latin 64 0 22 S	# S [53 ]A
# 10 0,255,0,255,0,0,0,0,0,0 Common 23 4 23 #	# # [23 ]p
Ê 5 0,255,0,255,0,0,0,0,0,0 Latin 78 0 24 Ê	# Ê [ca ]A
- 10 0,255,0,255,0,0,0,0,0,0 Common 25 3 25 -	# - [2d ]p
. 10 0,255,0,255,0,0,0,0,0,0 Common 26 6 26 .	# . [2e ]p
Y 5 0,255,0,255,0,0,0,0,0,0 Latin 69 0 27 Y	# Y [59 ]A
W 5 0,255,0,255,0,0,0,0,0,0 Latin 70 0 28 W	# W [57 ]A
O 5 0,255,0,255,0,0,0,0,0,0 Latin 77 0 29 O	# O [4f ]A
' 10 0,255,0,255,0,0,0,0,0,0 Common 30 10 30 '	# ' [27 ]p
8 8 0,255,0,255,0,0,0,0,0,0 Common 31 2 31 8	# 8 [38 ]0
! 10 0,255,0,255,0,0,0,0,0,0 Common 32 10 32 !	# ! [21 ]p
” 10 0,255,0,255,0,0,0,0,0,0 Common 33 10 33 "	# ” [201d ]p
É 5 0,255,0,255,0,0,0,0,0,0 Latin 79 0 34 É	# É [c9 ]A
? 10 0,255,0,255,0,0,0,0,0,0 Common 35 10 35 ?	# ? [3f ]p
C 5 0,255,0,255,0,0,0,0,0,0 Latin 85 0 36 C	# C [43 ]A
È 5 0,255,0,255,0,0,0,0,0,0 Latin 80 0 37 È	# È [c8 ]A
2 8 0,255,0,255,0,0,0,0,0,0 Common 38 2 38 2	# 2 [32 ]0
; 10 0,255,0,255,0,0,0,0,0,0 Common 39 10 39 ;	# ; [3b ]p
/ 10 0,255,0,255,0,0,0,0,0,0 Common 40 6 40 /	# / [2f ]p
( 10 0,255,0,255,0,0,0,0,0,0 Common 41 10 43 (	# ( [28 ]p
" 10 0,255,0,255,0,0,0,0,0,0 Common 42 10 42 "	# " [22 ]p
) 10 0,255,0,255,0,0,0,0,0,0 Common 43 10 41 )	# ) [29 ]p
1 8 0,255,0,255,0,0,0,0,0,0 Common 44 2 44 1	# 1 [31 ]0
3 8 0,255,0,255,0,0,0,0,0,0 Common 45 2 45 3	# 3 [33 ]0
7 8 0,255,0,255,0,0,0,0,0,0 Common 46 2 46 7	# 7 [37 ]0
“ 10 0,255,0,255,0,0,0,0,0,0 Common 47 10 47 "	# “ [201c ]p
Z 5 0,255,0,255,0,0,0,0,0,0 Latin 87 0 48 Z	# Z [5a ]A
[ 10 0,255,0,255,0,0,0,0,0,0 Common 49 10 50 [	# [ [5b ]p
] 10 0,255,0,255,0,0,0,0,0,0 Common 50 10 49 ]	# ] [5d ]p
| 0 0,255,0,255,0,0,0,0,0,0 Common 51 10 51 |	# | [7c ]
V 5 0,255,0,255,0,0,0,0,0,0 Latin 86 0 52 V	# V [56 ]A
0 8 0,255,0,255,0,0,0,0,0,0 Common 53 2 53 0	# 0 [30 ]0
5 8 0,255,0,255,0,0,0,0,0,0 Common 54 2 54 5	# 5 [35 ]0
— 10 0,255,0,255,0,0,0,0,0,0 Common 55 10 55 -	# — [2014 ]p
_ 10 0,255,0,255,0,0,0,0,0,0 Common 56 10 56 _	# _ [5f ]p
€ 0 0,255,0,255,0,0,0,0,0,0 Common 57 4 57 €	# € [20ac ]
e 3 0,255,0,255,0,0,0,0,0,0 Latin 4 0 58 e	# e [65 ]a
t 3 0,255,0,255,0,0,0,0,0,0 Latin 15 0 59 t	# t [74 ]a
a 3 0,255,0,255,0,0,0,0,0,0 Latin 10 0 60 a	# a [61 ]a
n 3 0,255,0,255,0,0,0,0,0,0 Latin 5 0 61 n	# n [6e ]a
h 3 0,255,0,255,0,0,0,0,0,0 Latin 20 0 62 h	# h [68 ]a
m 3 0,255,0,255,0,0,0,0,0,0 Latin 9 0 63 m	# m [6d ]a
s 3 0,255,0,255,0,0,0,0,0,0 Latin 22 0 64 s	# s [73 ]a
g 3 0,255,0,255,0,0,0,0,0,0 Latin 11 0 65 g	# g [67 ]a
i 3 0,255,0,255,0,0,0,0,0,0 Latin 6 0 66 i	# i [69 ]a
p 3 0,255,0,255,0,0,0,0,0,0 Latin 13 0 67 p	# p [70 ]a
u 3 0,255,0,255,0,0,0,0,0,0 Latin 16 0 68 u	# u [75 ]a
y 3 0,255,0,255,0,0,0,0,0,0 Latin 27 0 69 y	# y [79 ]a
w 3 0,255,0,255,0,0,0,0,0,0 Latin 28 0 70 w	# w [77 ]a
d 3 0,255,0,255,0,0,0,0,0,0 Latin 21 0 71 d	# d [64 ]a
l 3 0,255,0,255,0,0,0,0,0,0 Latin 14 0 72 l	# l [6c ]a
j 3 0,255,0,255,0,0,0,0,0,0 Latin 3 0 73 j	# j [6a ]a
r 3 0,255,0,255,0,0,0,0,0,0 Latin 8 0 74 r	# r [72 ]a
k 3 0,255,0,255,0,0,0,0,0,0 Latin 19 0 75 k	# k [6b ]a
b 3 0,255,0,255,0,0,0,0,0,0 Latin 17 0 76 b	# b [62 ]a
o 3 0,255,0,255,0,0,0,0,0,0 Latin 29 0 77 o	# o [6f ]a
ê 3 0,255,0,255,0,0,0,0,0,0 Latin 24 0 78 ê	# ê [ea ]a
é 3 0,255,0,255,0,0,0,0,0,0 Latin 34 0 79 é	# é [e9 ]a
è 3 0,255,0,255,0,0,0,0,0,0 Latin 37 0 80 è	# è [e8 ]a
4 8 0,255,0,255,0,0,0,0,0,0 Common 81 2 81 4	# 4 [34 ]0
6 8 0,255,0,255,0,0,0,0,0,0 Common 82 2 82 6	# 6 [36 ]0
9 8 0,255,0,255,0,0,0,0,0,0 Common 83 2 83 9	# 9 [39 ]0
f 3 0,255,0,255,0,0,0,0,0,0 Latin 7 0 84 f	# f [66 ]a
c 3 0,255,0,255,0,0,0,0,0,0 Latin 36 0 85 c	# c [63 ]a
v 3 0,255,0,255,0,0,0,0,0,0 Latin 52 0 86 v	# v [76 ]a
z 3 0,255,0,255,0,0,0,0,0,0 Latin 48 0 87 z	# z [7a ]a
= 0 0,255,0,255,0,0,0,0,0,0 Common 88 10 88 =	# = [3d ]
< 0 0,255,0,255,0,0,0,0,0,0 Common 89 10 90 <	# < [3c ]
> 0 0,255,0,255,0,0,0,0,0,0 Common 90 10 89 >	# > [3e ]
@ 10 0,255,0,255,0,0,0,0,0,0 Common 91 10 91 @	# @ [40 ]p
$ 0 0,255,0,255,0,0,0,0,0,0 Common 92 4 92 $	# $ [24 ]
£ 0 0,255,0,255,0,0,0,0,0,0 Common 93 4 93 £	# £ [a3 ]

@topherseance Please see attached zip file which has a test training for Javanese including both Javanese and Latin script. Only trained (replace a layer)
upto about 7% accuracy on the small training data that I could gather.
Keep us updated on your progress with training.
jav-traineddatas.zip

generate javanese script training data compactible with tesseract 3.04/3.05

The requirements of training data for tesseract 3.0x are quite different from those for 4.0.0 LSTM training.
You can use jav-java text from UDHR or wikipedia as linked in posts above.

Hello, sorry for the hiatus, had other tasks to do.
Only found 2 Javanese fonts so far:

Noto Sans Javanese (by Google)
Tuladha Jejeg (by R.S. Wihananto)

I tried to create a starter traineddata for Noto Sans Javanese by using this command below, it works succesfully:
topher@topher-ubuntu:~$ ~/tesseract/src/training/tesstrain.sh   --fonts_dir ~/tess-javanese/fonts   --lang jav   --linedata_only   --noextract_font_properties   --langdata_dir ~/tesseract/langdata   --tessdata_dir ~/tesseract/tessdata   --fontlist "Noto Sans Javanese"   --output_dir ~/tess-javanese/jav01-train

=== Starting training for language 'jav'
[Sen Jul 9 14:37:14 WIB 2018] /usr/local/bin/text2image --fonts_dir=/home/topher/tess-javanese/fonts --font=Noto Sans Javanese --outputbase=/tmp/font_tmp.l81FA3YVZ2/sample_text.txt --text=/tmp/font_tmp.l81FA3YVZ2/sample_text.txt --fontconfig_tmpdir=/tmp/font_tmp.l81FA3YVZ2
Stripped 1 unrenderable words
Rendered page 0 to file /tmp/font_tmp.l81FA3YVZ2/sample_text.txt.tif

=== Phase I: Generating training images ===
Rendering using Noto Sans Javanese
[Sen Jul 9 14:37:16 WIB 2018] /usr/local/bin/text2image --fontconfig_tmpdir=/tmp/font_tmp.l81FA3YVZ2 --fonts_dir=/home/topher/tess-javanese/fonts --strip_unrenderable_words --leading=32 --xsize 2560 --char_spacing=0.0 --exposure=0 --outputbase=/tmp/tmp.TYPQCfx2ed/jav/jav.Noto_Sans_Javanese.exp0 --max_pages=0 --font=Noto Sans Javanese --text=/home/topher/tesseract/langdata/jav/jav.training_text
Rendered page 0 to file /tmp/tmp.TYPQCfx2ed/jav/jav.Noto_Sans_Javanese.exp0.tif

=== Phase UP: Generating unicharset and unichar properties files ===
[Sen Jul 9 14:37:17 WIB 2018] /usr/local/bin/unicharset_extractor --output_unicharset /tmp/tmp.TYPQCfx2ed/jav/jav.unicharset --norm_mode 1 /tmp/tmp.TYPQCfx2ed/jav/jav.Noto_Sans_Javanese.exp0.box
Extracting unicharset from box file /tmp/tmp.TYPQCfx2ed/jav/jav.Noto_Sans_Javanese.exp0.box
Word started with a combiner:0xa9b8
Normalization failed for string 'ꦸ'
Word started with a combiner:0xa9bc
Word started with a combiner:0xa981
Normalization failed for string 'ꦼꦁ'
Word started with a combiner:0xa9b8
Normalization failed for string 'ꦸ'
Word started with a combiner:0xa983
Normalization failed for string 'ꦃ'
Word started with a combiner:0xa9c0
Normalization failed for string '꧀ꦠ'
Word started with a combiner:0xa9bc
Normalization failed for string 'ꦼ'
Word started with a combiner:0xa9c0
Normalization failed for string '꧀ꦲ'
Word started with a combiner:0xa9b6
Word started with a combiner:0xa981
Normalization failed for string 'ꦶꦁ'
Word started with a combiner:0xa9b6
Normalization failed for string 'ꦶ'
Word started with a combiner:0xa9b6
Normalization failed for string 'ꦶ'
Word started with a combiner:0xa9b6
Normalization failed for string 'ꦶ'
Word started with a combiner:0xa983
Normalization failed for string 'ꦃ'
Word started with a combiner:0xa9b6
Normalization failed for string 'ꦶ'
Wrote unicharset file /tmp/tmp.TYPQCfx2ed/jav/jav.unicharset
[Sen Jul 9 14:37:17 WIB 2018] /usr/local/bin/set_unicharset_properties -U /tmp/tmp.TYPQCfx2ed/jav/jav.unicharset -O /tmp/tmp.TYPQCfx2ed/jav/jav.unicharset -X /tmp/tmp.TYPQCfx2ed/jav/jav.xheights --script_dir=/home/topher/tesseract/langdata
Loaded unicharset of size 16 from file /tmp/tmp.TYPQCfx2ed/jav/jav.unicharset
Setting unichar properties
Setting script properties
Failed to load script unicharset from:/home/topher/tesseract/langdata/Javanese.unicharset
Warning: properties incomplete for index 3 = ꧋
Warning: properties incomplete for index 4 = ꦱ
Warning: properties incomplete for index 5 = ꦒ
Warning: properties incomplete for index 6 = ꦫ
Warning: properties incomplete for index 7 = ꦮ
Warning: properties incomplete for index 8 = ꦮꦺ
Warning: properties incomplete for index 9 = ꦤ
Warning: properties incomplete for index 10 = ꦏ
Warning: properties incomplete for index 11 = ꦥꦺ
Warning: properties incomplete for index 12 = ꦝ
Warning: properties incomplete for index 13 = ꦪ
Warning: properties incomplete for index 14 = ꦗ
Warning: properties incomplete for index 15 = ꧉
Writing unicharset to file /tmp/tmp.TYPQCfx2ed/jav/jav.unicharset

=== Phase E: Generating lstmf files ===
Using TESSDATA_PREFIX=/home/topher/tesseract/tessdata
[Sen Jul 9 14:37:17 WIB 2018] /usr/local/bin/tesseract /tmp/tmp.TYPQCfx2ed/jav/jav.Noto_Sans_Javanese.exp0.tif /tmp/tmp.TYPQCfx2ed/jav/jav.Noto_Sans_Javanese.exp0 lstm.train
Tesseract Open Source OCR Engine v4.0.0-beta.2-342-g12f4 with Leptonica
Page 1

=== Constructing LSTM training data ===
[Sen Jul 9 14:37:17 WIB 2018] /usr/local/bin/combine_lang_model --input_unicharset /tmp/tmp.TYPQCfx2ed/jav/jav.unicharset --script_dir /home/topher/tesseract/langdata --words /home/topher/tesseract/langdata/jav/jav.wordlist --numbers /home/topher/tesseract/langdata/jav/jav.numbers --puncs /home/topher/tesseract/langdata/jav/jav.punc --output_dir /home/topher/tess-javanese/jav01-train --lang jav
Failed to read data from: /home/topher/tesseract/langdata/jav/jav.wordlist
Failed to read data from: /home/topher/tesseract/langdata/jav/jav.punc
Failed to read data from: /home/topher/tesseract/langdata/jav/jav.numbers
Loaded unicharset of size 16 from file /tmp/tmp.TYPQCfx2ed/jav/jav.unicharset
Setting unichar properties
Setting script properties
Failed to load script unicharset from:/home/topher/tesseract/langdata/Javanese.unicharset
Warning: properties incomplete for index 3 = ꧋
Warning: properties incomplete for index 4 = ꦱ
Warning: properties incomplete for index 5 = ꦒ
Warning: properties incomplete for index 6 = ꦫ
Warning: properties incomplete for index 7 = ꦮ
Warning: properties incomplete for index 8 = ꦮꦺ
Warning: properties incomplete for index 9 = ꦤ
Warning: properties incomplete for index 10 = ꦏ
Warning: properties incomplete for index 11 = ꦥꦺ
Warning: properties incomplete for index 12 = ꦝ
Warning: properties incomplete for index 13 = ꦪ
Warning: properties incomplete for index 14 = ꦗ
Warning: properties incomplete for index 15 = ꧉
Config file is optional, continuing...
Failed to read data from: /home/topher/tesseract/langdata/jav/jav.config
Null char=2
Moving /tmp/tmp.TYPQCfx2ed/jav/jav.Noto_Sans_Javanese.exp0.box to /home/topher/tess-javanese/jav01-train
Moving /tmp/tmp.TYPQCfx2ed/jav/jav.Noto_Sans_Javanese.exp0.tif to /home/topher/tess-javanese/jav01-train
Moving /tmp/tmp.TYPQCfx2ed/jav/jav.Noto_Sans_Javanese.exp0.lstmf to /home/topher/tess-javanese/jav01-train

Created starter traineddata for language 'jav'

Run lstmtraining to do the LSTM training for language 'jav'

But when I tried to do the same for Tuladha Jejeg font, it shows this error:
topher@topher-ubuntu:~$ ~/tesseract/src/training/tesstrain.sh   --fonts_dir ~/tess-javanese/fonts   --lang jav   --linedata_only   --noextract_font_properties   --langdata_dir ~/tesseract/langdata   --tessdata_dir ~/tesseract/tessdata   --fontlist "Tuladha Jejeg"   --output_dir ~/tess-javanese/jav02-train

=== Starting training for language 'jav'
[Sen Jul 9 14:45:14 WIB 2018] /usr/local/bin/text2image --fonts_dir=/home/topher/tess-javanese/fonts --font=Tuladha Jejeg --outputbase=/tmp/font_tmp.uAxequREYg/sample_text.txt --text=/tmp/font_tmp.uAxequREYg/sample_text.txt --fontconfig_tmpdir=/tmp/font_tmp.uAxequREYg
Rendered page 0 to file /tmp/font_tmp.uAxequREYg/sample_text.txt.tif

=== Phase I: Generating training images ===
Rendering using Tuladha Jejeg
[Sen Jul 9 14:45:16 WIB 2018] /usr/local/bin/text2image --fontconfig_tmpdir=/tmp/font_tmp.uAxequREYg --fonts_dir=/home/topher/tess-javanese/fonts --strip_unrenderable_words --leading=32 --xsize 2560 --char_spacing=0.0 --exposure=0 --outputbase=/tmp/tmp.k4Fb5CaR5k/jav/jav.Tuladha_Jejeg.exp0 --max_pages=0 --font=Tuladha Jejeg --text=/home/topher/tesseract/langdata/jav/jav.training_text
Rendered page 0 to file /tmp/tmp.k4Fb5CaR5k/jav/jav.Tuladha_Jejeg.exp0.tif

=== Phase UP: Generating unicharset and unichar properties files ===
[Sen Jul 9 14:45:17 WIB 2018] /usr/local/bin/unicharset_extractor --output_unicharset /tmp/tmp.k4Fb5CaR5k/jav/jav.unicharset --norm_mode 1 /tmp/tmp.k4Fb5CaR5k/jav/jav.Tuladha_Jejeg.exp0.box
Extracting unicharset from box file /tmp/tmp.k4Fb5CaR5k/jav/jav.Tuladha_Jejeg.exp0.box
Word started with a combiner:0xa9bc
Word started with a combiner:0xa981
Normalization failed for string 'ꦼꦁ'
Word started with a combiner:0xa983
Normalization failed for string 'ꦃ'
Word started with a combiner:0xa9c0
Normalization failed for string '꧀ꦠ'
Word started with a combiner:0xa9bc
Normalization failed for string 'ꦼ'
Word started with a combiner:0xa9c0
Normalization failed for string '꧀ꦲ'
Word started with a combiner:0xa9b6
Word started with a combiner:0xa981
Normalization failed for string 'ꦶꦁ'
Word started with a combiner:0xa9b6
Normalization failed for string 'ꦶ'
Word started with a combiner:0xa9b6
Normalization failed for string 'ꦶ'
Word started with a combiner:0xa9b6
Normalization failed for string 'ꦶ'
Word started with a combiner:0xa983
Normalization failed for string 'ꦃ'
Word started with a combiner:0xa9b6
Normalization failed for string 'ꦶ'
Wrote unicharset file /tmp/tmp.k4Fb5CaR5k/jav/jav.unicharset
[Sen Jul 9 14:45:17 WIB 2018] /usr/local/bin/set_unicharset_properties -U /tmp/tmp.k4Fb5CaR5k/jav/jav.unicharset -O /tmp/tmp.k4Fb5CaR5k/jav/jav.unicharset -X /tmp/tmp.k4Fb5CaR5k/jav/jav.xheights --script_dir=/home/topher/tesseract/langdata
Loaded unicharset of size 17 from file /tmp/tmp.k4Fb5CaR5k/jav/jav.unicharset
Setting unichar properties
Setting script properties
Failed to load script unicharset from:/home/topher/tesseract/langdata/Javanese.unicharset
Warning: properties incomplete for index 3 = ꧋
Warning: properties incomplete for index 4 = ꦱꦸ
Warning: properties incomplete for index 5 = ꦒ
Warning: properties incomplete for index 6 = ꦫ
Warning: properties incomplete for index 7 = ꦮꦸ
Warning: properties incomplete for index 8 = ꦮꦺ
Warning: properties incomplete for index 9 = ꦤ
Warning: properties incomplete for index 10 = ꦮ
Warning: properties incomplete for index 11 = ꦏ
Warning: properties incomplete for index 12 = ꦥꦺ
Warning: properties incomplete for index 13 = ꦝ
Warning: properties incomplete for index 14 = ꦪ
Warning: properties incomplete for index 15 = ꦗ
Warning: properties incomplete for index 16 = ꧉
Writing unicharset to file /tmp/tmp.k4Fb5CaR5k/jav/jav.unicharset

=== Phase E: Generating lstmf files ===
Using TESSDATA_PREFIX=/home/topher/tesseract/tessdata
[Sen Jul 9 14:45:17 WIB 2018] /usr/local/bin/tesseract /tmp/tmp.k4Fb5CaR5k/jav/jav.Tuladha_Jejeg.exp0.tif /tmp/tmp.k4Fb5CaR5k/jav/jav.Tuladha_Jejeg.exp0 lstm.train
Tesseract Open Source OCR Engine v4.0.0-beta.2-342-g12f4 with Leptonica
Page 1
Empty page!!
Empty page!!
ERROR: /tmp/tmp.k4Fb5CaR5k/jav/jav.Tuladha_Jejeg.exp0.lstmf does not exist or is not readable

The tesseract/langdata/jav contains only one file: jav.training_text, the contents are one line javanese text:
꧋ꦱꦸꦒꦼꦁꦫꦮꦸꦃꦮꦺꦤ꧀ꦠꦼꦤ꧀ꦲꦶꦁꦮꦶꦏꦶꦥꦺꦝꦶꦪꦃꦗꦮꦶ꧉

(taken from here)
Opened the /tmp/ folder, look after /tmp/tmp.k4Fb5CaR5k/jav/jav.Tuladha_Jejeg.exp0.tif, and I think it is rendered correctly. Here's the file: jav.Tuladha_Jejeg.exp0.zip
Did I do everything right? Sorry if it was a rookie mistake.
Another info, Javanese script, by Unicode standard, has glyph-combining letters. (See Pasangan)
Tuladha Jejeg uses SIL Graphite to do the combination, whereas Noto Sans Javanese uses OpenType's ligatures and anchors.
(I think OpenType has a wider range of compatibility and support than SIL Graphite, for example, Chrome browser doesn't support SIL Graphite, causing javanese scripts won't render correctly for that font)
Does it has anything to do with the error?
Thanks before

Text2image uses Pango for font rendering. It is possible that it does not support the SIL graphite fonts. I also get errors for Annapurna SIL devanagari font and do not use it.

We plan on using the OCR for old textbook scans written in javanese script.
So far, Tuladha Jejeg font is the most similar font with those found in old textbooks.
Noto Sans Javanese looks a bit more 'modern'.

Just tested with other text strings, some of them worked, some did not. here's what we found:
ꦲꦤꦕꦫꦏ simple phrase, no gylph-combining --> success
ꦲꦤꦕꦫꦏꦮꦺꦤ꧀ꦠꦼ uses glyph-combining: Pasangan --> success
ꦲꦤꦕꦫꦏꦮꦺꦤ꧀ꦠꦼꦮꦸ uses glyph-combining: Sandhangan --> failure

Word started with a combiner:0xa9bc
Word started with a combiner:0xa981
Normalization failed for string 'ꦼꦁ'
Word started with a combiner:0xa983
Normalization failed for string 'ꦃ'
Word started with a combiner:0xa9c0
Normalization failed for string '꧀ꦠ'
Word started with a combiner:0xa9bc
Normalization failed for string 'ꦼ'
Word started with a combiner:0xa9c0
Normalization failed for string '꧀ꦲ'
Word started with a combiner:0xa9b6
Word started with a combiner:0xa981
Normalization failed for string 'ꦶꦁ'
Word started with a combiner:0xa9b6
Normalization failed for string 'ꦶ'
Word started with a combiner:0xa9b6
Normalization failed for string 'ꦶ'
Word started with a combiner:0xa9b6
Normalization failed for string 'ꦶ'
Word started with a combiner:0xa983
Normalization failed for string 'ꦃ'
Word started with a combiner:0xa9b6
Normalization failed for string 'ꦶ'
Please look at the validation/normalization rules for Indic scripts in the
code. Something there maybe triggering these errors.
On Mon, Jul 9, 2018 at 5:20 PM topherseance<EMAIL_ADDRESS>wrote:

Just tested with other text strings, some of them worked, some did not.
here's what we found:
ꦲꦤꦕꦫꦏ simple phrase, no gylph-combining --> success
ꦲꦤꦕꦫꦏꦮꦺꦤ꧀ꦠꦼ uses glyph-combining: Pasangan
https://en.wikipedia.org/wiki/Javanese_script#Pasangan --> success
ꦲꦤꦕꦫꦏꦮꦺꦤ꧀ꦠꦼꦮꦸ uses glyph-combining: Sandhangan
https://en.wikipedia.org/wiki/Javanese_script#Sandhangan --> failure
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tesseract-ocr/langdata/issues/126#issuecomment-403452754,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE2_o_D04G4y8MeiheI5Yt1TlKO7fxFpks5uE0N6gaJpZM4Tfa_w
.

--

भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com

see https://github.com/tesseract-ocr/tesseract/issues/1038
There may not be any Javanese script related rules, but the generic rules
and unicode range may cover these.
On Mon, Jul 9, 2018 at 6:50 PM Shree Devi Kumar<EMAIL_ADDRESS>wrote:

Word started with a combiner:0xa9bc
Word started with a combiner:0xa981
Normalization failed for string 'ꦼꦁ'
Word started with a combiner:0xa983
Normalization failed for string 'ꦃ'
Word started with a combiner:0xa9c0
Normalization failed for string '꧀ꦠ'
Word started with a combiner:0xa9bc
Normalization failed for string 'ꦼ'
Word started with a combiner:0xa9c0
Normalization failed for string '꧀ꦲ'
Word started with a combiner:0xa9b6
Word started with a combiner:0xa981
Normalization failed for string 'ꦶꦁ'
Word started with a combiner:0xa9b6
Normalization failed for string 'ꦶ'
Word started with a combiner:0xa9b6
Normalization failed for string 'ꦶ'
Word started with a combiner:0xa9b6
Normalization failed for string 'ꦶ'
Word started with a combiner:0xa983
Normalization failed for string 'ꦃ'
Word started with a combiner:0xa9b6
Normalization failed for string 'ꦶ'
Please look at the validation/normalization rules for Indic scripts in the
code. Something there maybe triggering these errors.
On Mon, Jul 9, 2018 at 5:20 PM topherseance<EMAIL_ADDRESS>wrote:

Just tested with other text strings, some of them worked, some did not.
here's what we found:
ꦲꦤꦕꦫꦏ simple phrase, no gylph-combining --> success
ꦲꦤꦕꦫꦏꦮꦺꦤ꧀ꦠꦼ uses glyph-combining: Pasangan
https://en.wikipedia.org/wiki/Javanese_script#Pasangan --> success
ꦲꦤꦕꦫꦏꦮꦺꦤ꧀ꦠꦼꦮꦸ uses glyph-combining: Sandhangan
https://en.wikipedia.org/wiki/Javanese_script#Sandhangan --> failure
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tesseract-ocr/langdata/issues/126#issuecomment-403452754,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE2_o_D04G4y8MeiheI5Yt1TlKO7fxFpks5uE0N6gaJpZM4Tfa_w
.

--

भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com

--

भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com

I ignored the errors and continued with training using 5 fonts which seem to cover javanese code range.
eration 29986: ALIGNED TRUTH : ꦤ꧀ ꦄꦩꦺꦫꦶꦏ ꦒꦝꦃ ꦠꦁꦒꦺꦭ꧀ ꦏꦺꦕꦩꦠꦤ꧀ ꦥꦺꦏꦭꦺꦴꦔꦤ꧀ ꦧꦚ꧀ꦗꦸꦂ ꦭ
Iteration 29986: BEST OCR TEXT : ꦤ꧀ ꦄꦩꦺꦫꦶꦏ ꦒꦝꦃ ꦠꦁꦒꦺꦭ꧀ ꦏꦺꦕꦩꦠꦤ꧀ ꦥꦺꦏꦭꦺꦴꦔꦤ꧀ ꦧꦚ꧀ꦗꦂ ꦭ
File /tmp/tmp.hW6IvPk7gK/jav_java/jav_java.Noto_Sans_Javanese.exp2.lstmf page 100 :
Mean rms=1.605%, delta=4.39%, train=13.896%(52.571%), skip ratio=0.4%
Iteration 29987: ALIGNED TRUTH : ꦶꦠꦸꦠ꧀ ꦏꦺꦱꦺꦤꦶꦪꦤ꧀ ꦱꦩ꧀ꦥꦸꦤ꧀ ꦗꦁꦏꦺꦥ꧀ ꦫꦺꦏꦺꦴꦂ ꦥꦶꦪ ꦲꦺꦩ꧀ꦥ꧀ꦭꦺꦴꦏ꧀ ꦲꦺꦩ꧀ꦧꦃꦏꦏꦸꦁ ꦲꦺꦩ꧀ꦧꦃꦥꦸꦠꦿ
Iteration 29987: BEST OCR TEXT : ꦱꦶꦠꦸꦠ꧀ ꦏꦺꦱꦺꦤꦶꦪꦤ꧀ ꦱꦩ꧀ꦥꦸꦤ꧀ ꦗꦁꦏꦺꦥ꧀ ꦫꦺꦏꦺꦂ ꦥꦶꦪ ꦲꦺꦩꦺꦏ꧀ ꦲꦺꦩ꧀ꦧꦃꦏꦏꦸꦁ ꦲꦺꦩ꧀ꦧꦃ ꦥꦸꦠꦶ
File /tmp/tmp.hW6IvPk7gK/jav_java/jav_java.Tuladha_Jejeg.exp0.lstmf page 31 :
Mean rms=1.605%, delta=4.392%, train=13.901%(52.578%), skip ratio=0.4%
Iteration 29988: ALIGNED TRUTH : ꦤꦺꦴꦧꦺꦭ꧀ ꦱꦱ꧀ꦠꦿ ꦤꦺꦴꦧꦺꦭ꧀ ꦱ
Iteration 29988: BEST OCR TEXT : ꦏꦤꦺꦴꦧꦺꦭ꧀ ꦱꦱꦿ ꦤꦺꦴꦠꦺꦭ꧀ ꧈
File /tmp/tmp.hW6IvPk7gK/jav_java/jav_java.Tuladha_Jejeg.exp-1.lstmf page 499 :
Mean rms=1.605%, delta=4.392%, train=13.884%(52.578%), skip ratio=0.4%
Iteration 29989: ALIGNED TRUTH : ꦼꦱꦶꦲꦗꦶ ꦮꦼꦱꦶ ꦮꦼꦱ꧀ꦠ ꦮꦼꦱ꧀ꦥꦢ
Iteration 29989: BEST OCR TEXT : ꦼꦱꦶꦲꦗꦼ ꦮꦼꦱꦶ ꦮꦼꦱ꧀ꦠ ꦥꦼꦱ꧀ꦥꦢ
File /tmp/tmp.hW6IvPk7gK/jav_java/jav_java.Tuladha_Jejeg.exp1.lstmf page 123 :
Mean rms=1.605%, delta=4.386%, train=13.875%(52.559%), skip ratio=0.4%
Iteration 29990: ALIGNED TRUTH : ꦱ
Iteration 29990: BEST OCR TEXT : ꦕꦱ
File /tmp/tmp.hW6IvPk7gK/jav_java/jav_java.Tuladha_Jejeg.exp-2.lstmf page 362 :
Mean rms=1.606%, delta=4.395%, train=13.961%(52.617%), skip ratio=0.4%
Iteration 29991: ALIGNED TRUTH : ꦩꦪꦸꦫ ꦩꦫꦁ ꦩꦫꦏꦂꦩ ꦩꦫꦏꦠ
Iteration 29991: BEST OCR TEXT : ꦩꦥꦪꦸꦫ ꦩꦫꦁ ꦩꦫꦏꦂꦩ ꦩꦫꦏꦠ
File /tmp/tmp.hW6IvPk7gK/jav_java/jav_java.Tuladha_Jejeg.exp2.lstmf page 685 (Perfect):
Mean rms=1.605%, delta=4.391%, train=13.966%(52.592%), skip ratio=0.4%
Iteration 29992: ALIGNED TRUTH : ꦩꦺꦤ꧀ꦠ꧀ ꦩꦫꦺꦠ꧀ ꦠꦻꦴꦤ꧀ ꦝꦺꦮꦺꦏꦺ ꦲꦺꦤ꧀ꦠꦸꦏ꧀ ꦮꦶꦒꦠꦶ ꦱꦔꦺꦠ꧀ ꦱꦠꦸꦁ ꦒꦭ
Iteration 29992: BEST OCR TEXT : ꦩꦺꦤ꧀ꦠ꧀ꦩꦫꦺꦠ꧀ ꦠꦻꦴꦤ꧀ ꦝꦺꦮꦺꦏꦺꦲꦺꦤ꧀ꦠꦸꦏ꧀ꦮꦶꦒꦠꦶꦱ ꦔꦺꦠ꧀ ꦱꦠꦸꦁ ꦒꦭ
File /tmp/tmp.hW6IvPk7gK/jav_java/jav_java.Carakan_Anyar.exp0.lstmf page 604 :
Mean rms=1.605%, delta=4.391%, train=13.965%(52.612%), skip ratio=0.4%
Iteration 29993: ALIGNED TRUTH : ꦤ꧀ ꦱꦶꦗꦶ ꦏꦧꦸꦥꦠꦺꦤ꧀ ꦠꦥꦤꦸꦭꦶ ꦧꦒꦺꦪꦤ꧀ ꦱꦏꦶꦁ ꦆꦧꦸꦏꦸꦛ ꦏꦺꦕꦩ
Iteration 29993: BEST OCR TEXT : ꦤ꧀ ꦱꦶꦗꦶꦏꦧꦸꦥꦠꦺꦤ꧀ ꦠꦥꦤꦸꦭꦶꦧꦒꦺꦪꦤ꧀ ꦱꦏꦶꦁ ꦆꦧꦸꦤ꧀ꦛ ꦏꦺꦕꦩ
File /tmp/tmp.hW6IvPk7gK/jav_java/jav_java.Carakan_Anyar.exp-1.lstmf page 162 :
Mean rms=1.605%, delta=4.39%, train=13.965%(52.591%), skip ratio=0.4%
Iteration 29994: ALIGNED TRUTH : ꦱꦮꦶꦱꦺ ꦏꦧꦸꦥꦠꦺꦤ꧀ ꦕꦶꦪꦚ꧀ꦗꦸꦂ ꦱꦏ ꦠꦻꦴꦤ꧀ ꦥꦿꦺꦴꦮ꦳ꦶꦤ꧀ꦱꦶ ꦭꦶꦩ꧀ꦧꦸꦂꦒ꧀ ꦢꦶꦮ ꦩꦺꦤꦺꦲꦶ
Iteration 29994: BEST OCR TEXT : ꦱꦮꦶꦱꦺꦏꦧꦸꦥꦠꦺꦤ꧀ ꦕꦶꦪꦚ꧀ꦗꦸꦂ ꦱꦏ ꦠꦻꦴꦤ꧀ ꦥꦿꦺꦴꦮ꦳ꦶꦤ꧀ꦱꦶ ꦭꦶꦩ꧀ ꦧꦂꦒ꧀ ꦢꦶꦮ ꦩꦺꦤꦺꦲꦶ
File /tmp/tmp.hW6IvPk7gK/jav_java/jav_java.Carakan_Anyar.exp1.lstmf page 759 :
Mean rms=1.604%, delta=4.386%, train=13.948%(52.549%), skip ratio=0.4%
Iteration 29995: ALIGNED TRUTH : ꦝꦺ ꦝꦠꦺꦁ ꦏꦱꦸꦭ꧀ꦠꦤꦤ꧀ ꦢꦺꦩꦏ꧀ ꦏꦺꦕꦩꦠꦤ꧀ ꦏꦭꦶꦠꦶꦢꦸ ꦥꦂꦠꦻ ꦒꦺꦴꦭ꧀ꦏꦂ
Iteration 29995: BEST OCR TEXT : ꦝꦺꦝꦠꦺꦴꦁ ꦏꦱꦸꦭ꧀ꦠꦤꦤ꧀ ꦢꦺꦩꦏ꧀ ꦏꦺꦕꦩꦠꦤ꧀ ꦏꦭꦶꦠꦶꦢꦸꦥꦂ ꦠꦻ ꦒꦺꦴꦭ꧀ ꦏꦂ
File /tmp/tmp.hW6IvPk7gK/jav_java/jav_java.Carakan_Anyar.exp-2.lstmf page 160 :
Mean rms=1.603%, delta=4.37%, train=13.868%(52.512%), skip ratio=0.4%
Iteration 29996: ALIGNED TRUTH : ., ꦭꦶꦩ꧀ꦥꦢ꧀ ꦭꦶꦩ꧀ꦥꦸꦁ ꦭꦶꦁꦒꦶꦃ ꦭꦶꦁꦒ ꦭꦶꦁꦱꦁ ꦭꦶꦁꦱꦶꦂ ꦭꦶꦁꦱꦼꦩ꧀
Iteration 29996: BEST OCR TEXT : . , ꦭꦶꦩ꧀ꦥꦢ꧀ ꦭꦶꦩ꧀ꦥꦸꦁ ꦭꦶꦁꦒꦶꦃ ꦭꦶꦁꦒ ꦭꦶꦁꦱꦁ ꦭꦶꦁꦱꦶꦂ ꦭꦶꦱꦼꦩ꧀
File /tmp/tmp.hW6IvPk7gK/jav_java/jav_java.Carakan_Anyar.exp2.lstmf page 183 :
Mean rms=1.603%, delta=4.372%, train=13.869%(52.514%), skip ratio=0.4%
Iteration 29997: ALIGNED TRUTH : ꦥꦿꦏꦫꦤ ꦥꦿꦏꦫ ꦥꦿꦏꦮꦶꦱ꧀ ꦥꦿꦏꦱꦶꦠ ꦥꦿꦏꦱ ꦥꦿꦏꦫ ꦥꦿꦏꦮꦶꦱ꧀ ꦥꦿꦏꦱꦶꦠ ꦥꦿꦏꦱ ꦮꦶꦢꦸꦫ
Iteration 29997: BEST OCR TEXT : ꦥꦿꦏꦫꦤ ꦥꦿꦏꦫ ꦥꦿꦏꦮꦶꦱ꧀ ꦥꦿꦏꦱꦶꦠ ꦥꦿꦏꦱ ꦥꦿꦏꦫ ꦥꦿꦏꦮꦶꦱ꧀ ꦥꦿꦏꦱꦶꦠ ꦥꦿꦏꦱ ꦮꦶꦢꦸꦫ
File ./jav_java-layer_train/jav_java.Carakan-Unicode.exp0.lstmf page 321 (Perfect):
Mean rms=1.603%, delta=4.372%, train=13.869%(52.514%), skip ratio=0.4%
Iteration 29998: ALIGNED TRUTH : ꦤꦭꦶꦏ ꦏꦸꦮꦶ ꦒꦩ꧀ꦧꦂ:ꦥ꦳꧀ꦭꦒ꧀ ꦲꦺꦴꦥ꦳꧀ ꦲꦶꦁꦏꦁ ꦏꦺꦢꦃ ꦢꦤ꧀ ꦱꦺ
Iteration 29998: BEST OCR TEXT : ꦤꦭꦶꦏ ꦏꦸꦮꦶꦒꦩ꧀ꦧꦂꦥ꦳꧀ ꦭꦒ꧀ ꦲꦺꦴꦥ꦳꧀ ꦲꦶꦁꦏꦁ ꦏꦺꦢꦃ ꦢꦤ꧀ ꦱꦺ
File /tmp/tmp.hW6IvPk7gK/jav_java/jav_java.Carakan-Unicode.exp-1.lstmf page 3 :
Mean rms=1.602%, delta=4.368%, train=13.857%(52.475%), skip ratio=0.4%
Iteration 29999: ALIGNED TRUTH : ꦲꦱꦱ꧀ꦠ ꦲꦱꦶꦂ ꦲꦱꦶꦃ ꦲꦱꦶꦤ꧀ ꦲꦱꦶꦫꦤ꧀ ꦲꦱꦭ꧀ ꦲꦸꦱꦸꦭ꧀ - ꦲꦱꦭ꧀ ꦲꦱꦱ꧀ꦠ ꦲꦱꦶꦂ ꦲꦱꦶ
Iteration 29999: BEST OCR TEXT : ꦲꦱꦱ꧀ꦠ ꦲꦱꦶꦂ ꦲꦱꦶꦃ ꦲꦱꦶꦤꦏ꧀ ꦲꦱꦶꦫꦤ꧀ ꦲꦱꦭ꧀ꦲꦸꦱꦸꦁꦭ꧀ - ꦲꦱꦭ꧀ ꦲꦱꦱ꧀ꦠ ꦲꦱꦶꦂ ꦲꦱꦶ
File /tmp/tmp.hW6IvPk7gK/jav_java/jav_java.Carakan-Unicode.exp1.lstmf page 71 :
Mean rms=1.602%, delta=4.368%, train=13.852%(52.464%), skip ratio=0.4%
At iteration 28598/30000/30058, Mean rms=1.602%, delta=4.368%, char train=13.852%, word train=52.464%, skip ratio=0.4%,  New worst char error = 13.852 wrote checkpoint.

Finished! Error rate = 13.384

Can you please share the command you did for the above training?
I still can't get the training to work successfully. I used the "training from scratch" method. Again, sorry if it is a newbie mistake.
Also, I couldn't find a .unicharset file in the langdata/jav, should I need to create one?
Warning: given outputs 111 not equal to unicharset of 26.
Num outputs,weights in Series:
  1,36,0,1:1, 0
Num outputs,weights in Series:
  C3,3:9, 0
  Ft16:16, 160
Total weights = 160
  [C3,3Ft16]:16, 160
  Mp3,3:16, 0
  Lfys48:48, 12480
  Lfx96:96, 55680
  Lrx96:96, 74112
  Lfx256:256, 361472
  Fc26:26, 6682
Total weights = 510586
Built network:[1,36,0,1[C3,3Ft16]Mp3,3Lfys48Lfx96Lrx96Lfx256Fc26] from request [1,36,0,1 Ct3,3,16 Mp3,3 Lfys48 Lfx96 Lrx96 Lfx256 O1c111]
Training parameters:
  Debug interval = 100, weights = 0.1, learning rate = 0.002, momentum=0.5
null char=25
Loaded 16/16 pages (1-16) of document /home/topher/tess-javanese/jav03-train/jav.Prada.exp0.lstmf
Loaded 16/16 pages (1-16) of document /home/topher/tess-javanese/jav03-train/jav.Prada.exp0.lstmf
Encoding of string failed! Failure bytes: ffffffea ffffffa6 ffffff83 ffffffea ffffffa6 ffffffa9 ffffffea ffffffa6 ffffffb2 ffffffea ffffffa6 ffffffb8 ffffffea ffffffa7 ffffff88 20 ffffffea ffffffa6 ffffffb2 ffffffea ffffffa6 ffffffb6 ffffffea ffffffa6 ffffffaa ffffffea ffffffa6 ffffffba ffffffea ffffffa6 ffffffb4 ffffffea ffffffa6 ffffffb2 ffffffea ffffffa6 ffffff8f ffffffea ffffffa7 ffffff80 ffffffea ffffffa6 ffffffa9 ffffffea ffffffa6 ffffffb8 ffffffea ffffffa7 ffffff89 20 ffffffea ffffffa6 ffffff8f ffffffea ffffffa6 ffffffbc ffffffea ffffffa6 ffffffa4 ffffffea ffffffa6 ffffffad ffffffea ffffffa6 ffffffb6 ffffffea ffffffa6 ffffffad ffffffea ffffffa6 ffffffa4 ffffffea ffffffa7 ffffff80 ffffffea ffffffa6 ffffffa5 ffffffea ffffffa6 ffffffb2 ffffffea ffffffa6 ffffffa9 ffffffea ffffffa6 ffffffa4 ffffffea ffffffa6 ffffffb2 ffffffea ffffffa6 ffffff8f ffffffea ffffffa7 ffffff80 ffffffea ffffffa6 ffffffb2 ffffffea ffffffa6 ffffff8f ffffffea ffffffa7 ffffff80 ffffffea ffffffa6 ffffff8f ffffffea ffffffa6 ffffffb1 ffffffea ffffffa6 ffffffbc ffffffea ffffffa6 ffffffa7 ffffffea ffffffa6 ffffffb8 ffffffea ffffffa6 ffffffa0 ffffffea ffffffa7 ffffff80 ffffffea ffffffa7 ffffff88
Can't encode transcription: 'ꦏꦧꦺꦃꦩꦲꦸ꧈ ꦲꦶꦪꦺꦴꦲꦏ꧀ꦩꦸ꧉ ꦏꦼꦤꦭꦶꦭꦤ꧀ꦥꦲꦩꦤꦲꦏ꧀ꦲꦏ꧀ꦏꦱꦼꦧꦸꦠ꧀꧈' in language ''
Starting sh -c "trap 'kill %1' 0 1 2 ; java -Xms1024m -Xmx2048m -jar ./ScrollView.jar & wait"
Error: Unable to access jarfile ./ScrollView.jar
sh: 1: kill: No such process

I did run the unicharset_extractor with a .txt file containing javanese texts. Here's the resulting unicharset file:
jav.unicharset.txt
Each line contains 0,255,0,255,0,0,0,0,0,0, I guess it is some sort of coordinates, is it the correct value?
Or maybe I should use it anyway? The unicharset file you copy-pasted earlier in this thread also contains 0,255,0,255,0,0,0,0,0,0 for each line.

Found another font: Prada

Start of Javanese script unicode range may need to be added to
https://github.com/tesseract-ocr/tesseract/blob/master/src/training/validator.h
On Thu, Jul 19, 2018 at 3:29 AM topherseance<EMAIL_ADDRESS>wrote:

Found another font: Prada
https://sites.google.com/site/fontsundaprada/unduh-font-sunda-prada/prada.ttf
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tesseract-ocr/langdata/issues/126#issuecomment-406087613,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE2_o78VzTinzCsjGIABMgZmA9cOsczXks5uH6_WgaJpZM4Tfa_w
.

--

भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com

Encoding of string failed! Failure bytes: ffffffea ffffffa6 ffffff83 ffffffea ffffffa6 ffffffa9 ffffffea ffffffa6 ffffffb2 ffffffea ffffffa6 ffffffb8 ffffffea ffffffa7 ffffff88

The text is clearly not encoded in utf-8.

Can you please share the commands and steps you did for the above training?

Please see https://github.com/Shreeshrii/tessdata_jav_java

I collected few javanese aksara here, probably has about several thousand textlines:
https://github.com/topherseance/bible_javanese_aksara

@Shreeshrii when you run your scripts, layertrain.sh or plustrain.sh, did you receive the Encoding of string failed error?
I ran the script and still got this:
File /tmp/tmp.fAmoYPWBIL/jav_java/jav_java.Carakan_Anyar.exp1.lstmf page 569 :
Mean rms=5.024%, delta=42.402%, train=100.11%(100%), skip ratio=61.7%
Encoding of string failed! Failure bytes: ffffffe2 ffffff80 ffffff8b ffffffea ffffffa6 ffffffb1 ffffffea ffffffa6 ffffffb6 ffffffea ffffffa6 ffffff81 20 ffffffea ffffffa6 ffffffa2 ffffffea ffffffa6 ffffffb6 ffffffea ffffffa6 ffffffaa ffffffea ffffffa6 ffffffaa ffffffea ffffffa6 ffffffb2 ffffffea ffffffa6 ffffffb6 20 ffffffea ffffffa6 ffffffa2 ffffffea ffffffa6 ffffffb8 ffffffea ffffffa6 ffffffa9
Can't encode transcription: 'ꦮꦺꦠꦤ꧀ꦱꦶꦁ ꦢꦶꦪꦪꦲꦶ ꦢꦸꦩ' in language ''
Iteration 1171: ALIGNED TRUTH : ꦩꦭꦁꦲꦠꦺꦤꦶ ꦧꦸꦩꦶ ꦧꦸꦩ꧀ꦥꦼꦠ꧀
Iteration 1171: BEST OCR TEXT :
File /tmp/tmp.fAmoYPWBIL/jav_java/jav_java.Tuladha_Jejeg.exp0.lstmf page 480 :
Mean rms=5.024%, delta=42.397%, train=100.111%(100%), skip ratio=61.7%
Iteration 1172: ALIGNED TRUTH : ꦄꦢꦩ꧀ ꦩꦭꦶꦏ꧀ ꦮ꦳ꦶꦢꦺꦪꦺꦴ ꦒ
Iteration 1172: BEST OCR TEXT :
File /tmp/tmp.fAmoYPWBIL/jav_java/jav_java.Carakan_Anyar.exp-1.lstmf page 40 :
Mean rms=5.023%, delta=42.37%, train=100.113%(100%), skip ratio=61.7%
Iteration 1173: ALIGNED TRUTH : ꦏꦤ꧀ꦕ꧀ꦂꦶꦠ꧀ ꦏꦤ꧀ꦛꦶꦁ
Iteration 1173: BEST OCR TEXT :

Checked the encoding of jav.training_text, I guess it is encoded in UTF-8
topher@topher-ubuntu:~/tesseract/langdata/jav$ file -i jav.training_text
jav.training_text: text/plain; charset=utf-8

Are you getting the error on all lines of training text or just some lines?
I have had the error before but not with the current set of files.
What is your locale?
On Mon 6 Aug, 2018, 12:47 PM topherseance<EMAIL_ADDRESS>wrote:

@Shreeshrii https://github.com/Shreeshrii when you run your scripts,
layertrain.sh or plustrain.sh, did you receive the Encoding of string
failed error?
I ran the script and still got this:
File /tmp/tmp.fAmoYPWBIL/jav_java/jav_java.Carakan_Anyar.exp1.lstmf page 569 :
Mean rms=5.024%, delta=42.402%, train=100.11%(100%), skip ratio=61.7%
Encoding of string failed! Failure bytes: ffffffe2 ffffff80 ffffff8b ffffffea ffffffa6 ffffffb1 ffffffea ffffffa6 ffffffb6 ffffffea ffffffa6 ffffff81 20 ffffffea ffffffa6 ffffffa2 ffffffea ffffffa6 ffffffb6 ffffffea ffffffa6 ffffffaa ffffffea ffffffa6 ffffffaa ffffffea ffffffa6 ffffffb2 ffffffea ffffffa6 ffffffb6 20 ffffffea ffffffa6 ffffffa2 ffffffea ffffffa6 ffffffb8 ffffffea ffffffa6 ffffffa9
Can't encode transcription: 'ꦮꦺꦠꦤ꧀ꦱꦶꦁ ꦢꦶꦪꦪꦲꦶ ꦢꦸꦩ' in language ''
Iteration 1171: ALIGNED TRUTH : ꦩꦭꦁꦲꦠꦺꦤꦶ ꦧꦸꦩꦶ ꦧꦸꦩ꧀ꦥꦼꦠ꧀
Iteration 1171: BEST OCR TEXT :
File /tmp/tmp.fAmoYPWBIL/jav_java/jav_java.Tuladha_Jejeg.exp0.lstmf page 480 :
Mean rms=5.024%, delta=42.397%, train=100.111%(100%), skip ratio=61.7%
Iteration 1172: ALIGNED TRUTH : ꦄꦢꦩ꧀ ꦩꦭꦶꦏ꧀ ꦮ꦳ꦶꦢꦺꦪꦺꦴ ꦒ
Iteration 1172: BEST OCR TEXT :
File /tmp/tmp.fAmoYPWBIL/jav_java/jav_java.Carakan_Anyar.exp-1.lstmf page 40 :
Mean rms=5.023%, delta=42.37%, train=100.113%(100%), skip ratio=61.7%
Iteration 1173: ALIGNED TRUTH : ꦏꦤ꧀ꦕ꧀ꦂꦶꦠ꧀ ꦏꦤ꧀ꦛꦶꦁ
Iteration 1173: BEST OCR TEXT :
Checked the encoding of jav.training_text, I guess it is encoded in UTF-8
topher@topher-ubuntu:~/tesseract/langdata/jav$ file -i jav.training_text
jav.training_text: text/plain; charset=utf-8
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tesseract-ocr/langdata/issues/126#issuecomment-410611481,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE2_oyxLRSEm1PBGpSlEa_nM9rTjNJpDks5uN-1sgaJpZM4Tfa_w
.

Just some lines, I guess.
My locale is EN.

There is probably some invisible code or character that is not in
unicharset. You can try to identify it from the text and provided codes. If
it is only a few lines, you can ignore.
I have tried more training but still not getting much better results, error
rate is around 7% on training set.
On Mon 6 Aug, 2018, 10:31 PM topherseance<EMAIL_ADDRESS>wrote:

Just some lines, I guess.
My locale is EN.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tesseract-ocr/langdata/issues/126#issuecomment-410777703,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE2_o0-LjXqa5ywOofM-MQ7kq22rg_y7ks5uOHZpgaJpZM4Tfa_w
.

My locale is en_us.utf8
That might make some difference in the display of the codes.
On Tue 7 Aug, 2018, 9:18 AM Shree Devi Kumar<EMAIL_ADDRESS>wrote:

There is probably some invisible code or character that is not in
unicharset. You can try to identify it from the text and provided codes. If
it is only a few lines, you can ignore.
I have tried more training but still not getting much better results,
error rate is around 7% on training set.
On Mon 6 Aug, 2018, 10:31 PM topherseance<EMAIL_ADDRESS>wrote:

Just some lines, I guess.
My locale is EN.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tesseract-ocr/langdata/issues/126#issuecomment-410777703,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE2_o0-LjXqa5ywOofM-MQ7kq22rg_y7ks5uOHZpgaJpZM4Tfa_w
.

Ok, I think I got the reason for the error. It is related to the text not
passing the 'normalization' rules as setup for the script.
For Javanese, I copied rules from existing languages but these need to be
verified and corrected.
https://github.com/tesseract-ocr/tesseract/blob/master/src/training/validate_javanese.cpp
On Tue, Aug 7, 2018 at 9:56 AM Shree Devi Kumar<EMAIL_ADDRESS>wrote:

My locale is en_us.utf8
That might make some difference in the display of the codes.
On Tue 7 Aug, 2018, 9:18 AM Shree Devi Kumar<EMAIL_ADDRESS>wrote:

There is probably some invisible code or character that is not in
unicharset. You can try to identify it from the text and provided codes. If
it is only a few lines, you can ignore.
I have tried more training but still not getting much better results,
error rate is around 7% on training set.
On Mon 6 Aug, 2018, 10:31 PM topherseance<EMAIL_ADDRESS>wrote:

Just some lines, I guess.
My locale is EN.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tesseract-ocr/langdata/issues/126#issuecomment-410777703,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE2_o0-LjXqa5ywOofM-MQ7kq22rg_y7ks5uOHZpgaJpZM4Tfa_w
.

--

भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com

I am getting the following kind of errors. Please check whether the
Javanese text is valid.
Word started with a combiner:0xa9ba
Word started with a combiner:0xa9b4
Normalization failed for string 'ꦺꦴ'
Word started with a combiner:0xa9c0
Normalization failed for string '꧀ꦲꦶꦁ'
Invalid start of grapheme sequence:M=0xa9ba
Normalization failed for string '꧇ꦺ'
Invalid start of grapheme sequence:M=0xa9ba
Normalization failed for string 'ꦩ꧀ꦥꦸꦺ'
Invalid start of grapheme sequence:M=0xa9ba
Normalization failed for string 'ꦏ꧀ꦢꦴꦺ'
Invalid start of grapheme sequence:M=0xa9ba
Normalization failed for string '꧇ꦺ'
Invalid start of grapheme sequence:M=0xa9ba
Normalization failed for string 'ꦲꦶꦁꦺ'
Invalid start of grapheme sequence:M=0xa9ba
Normalization failed for string 'ꦗꦿꦴꦺ'
Invalid start of grapheme sequence:M=0xa9ba
Normalization failed for string 'ꦧꦼꦺ'
Invalid start of grapheme sequence:M=0xa9ba
Normalization failed for string 'ꦤ꧀ꦱꦴꦺ'
Invalid start of grapheme sequence:M=0xa9ba
Normalization failed for string 'ꦮꦶꦺ'
Invalid start of grapheme sequence:M=0xa9ba
Normalization failed for string 'ꦤ꧀ꦏꦁꦺ'
Invalid start of grapheme sequence:M=0xa9ba
Normalization failed for string 'ꦒꦴꦺ'
Invalid start of grapheme sequence:M=0xa9ba
Normalization failed for string 'ꦠ꧀ꦤꦶꦺ'
Invalid start of grapheme sequence:M=0xa9ba
On Tue, Aug 7, 2018 at 8:36 PM Shree Devi Kumar<EMAIL_ADDRESS>wrote:

Ok, I think I got the reason for the error. It is related to the text not
passing the 'normalization' rules as setup for the script.
For Javanese, I copied rules from existing languages but these need to be
verified and corrected.
https://github.com/tesseract-ocr/tesseract/blob/master/src/training/validate_javanese.cpp
On Tue, Aug 7, 2018 at 9:56 AM Shree Devi Kumar<EMAIL_ADDRESS>wrote:

My locale is en_us.utf8
That might make some difference in the display of the codes.
On Tue 7 Aug, 2018, 9:18 AM Shree Devi Kumar<EMAIL_ADDRESS>wrote:

There is probably some invisible code or character that is not in
unicharset. You can try to identify it from the text and provided codes. If
it is only a few lines, you can ignore.
I have tried more training but still not getting much better results,
error rate is around 7% on training set.
On Mon 6 Aug, 2018, 10:31 PM topherseance<EMAIL_ADDRESS>wrote:

Just some lines, I guess.
My locale is EN.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tesseract-ocr/langdata/issues/126#issuecomment-410777703,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE2_o0-LjXqa5ywOofM-MQ7kq22rg_y7ks5uOHZpgaJpZM4Tfa_w
.

--

भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com

--

भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com

@
topherseance
The LSTM training langdata for Javanese in Latin script is now available at
https://github.com/tesseract-ocr/langdata_lstm/tree/master/jav
You can convert it to aksara jawa to use for training.

Done converting this file to aksara jawa:
https://github.com/tesseract-ocr/langdata_lstm/blob/master/jav/jav.training_text
Result:
https://github.com/topherseance/javanese-aksara-training-text
But what about the other files?
For example, .numbers, .wordlist.
Is the .numbers file correct? Seems to contain random letters..

Thanks.
Please convert the wordlist also.
The numbers file should actually just show patterns where numbers are
found. Please look at the file for English as a sample.
On Tue, Aug 14, 2018 at 11:05 AM topherseance<EMAIL_ADDRESS>wrote:

Done converting this file to aksara jawa:
https://github.com/tesseract-ocr/langdata_lstm/blob/master/jav/jav.training_text
Result:
https://github.com/topherseance/javanese-aksara-training-text
But what about the other files?
For example, .numbers, .wordlist.
Is the .numbers file correct? Seems to contain random letters..
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tesseract-ocr/langdata/issues/126#issuecomment-412759720,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE2_o7yeAUKbXxt8zXaXa51o6IMYFIYQks5uQmGngaJpZM4Tfa_w
.

--

भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com

Updated my repo to include conversion result with whitespaces:
https://github.com/topherseance/javanese-aksara-training-text
https://github.com/topherseance/javanese-aksara-training-text/blob/master/with-whitespace-combined.txt

Thank you.
On Thu, Aug 16, 2018 at 12:22 AM, topherseance<EMAIL_ADDRESS>wrote:

Updated my repo to include conversion result with whitespaces:
https://github.com/topherseance/javanese-aksara-training-text
https://github.com/topherseance/javanese-aksara-training-text/blob/master/
with-whitespace-combined.txt
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tesseract-ocr/langdata/issues/126#issuecomment-413297999,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE2_o7AzrqkOhomn1NcrFJq12gcPfpg7ks5uRG3bgaJpZM4Tfa_w
.

--

भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com

Hi, first I'm sorry perhaps this is different topic but I think quite related. I've opened an issue at https://github.com/tesseract-ocr/langdata/issues/152 (Balinese script OCR) but stiil confuse (newbie syndrom) until finally landed here.
I'm ready for collecting training text but still on hold since same case like Javanese fonts, Balinese script has Bali SImbar Dwijendra font (see posted issue) most similar to ancient script but not yet tested for training (I'm afraid same incompatibility issue like Tuladha Jejeg, will check soon). At the other side Balinese script also has Noto Sans/Seri Balinese from Google.
Also, I've download https://github.com/Shreeshrii/tessdata_jav_java, and its README.md said "Source code changes will be needed in tesseract... "
Could you direct me how to use all material here since Javanese script has  big influence to  Balinese script. Geograhically, Bali and Java are also neighbor to each other.
Thank you very much before for your kind attention.

I had done aksara jawa training and created two traineddata files - see links given in https://github.com/Shreeshrii/tessdata_jav_java/blob/master/README.md
But I am not sure how accurate those are or whether @topherseance did further training on same.
The changes to tesseract codebase were made via:
https://github.com/tesseract-ocr/tesseract/commit/0eb7be1cd1707931abd77903793bf966a6640d58#diff-eaafd22a79065f5b8d28318d482e650d
https://github.com/tesseract-ocr/tesseract/commit/7957288fd5502551b6c7f073c5f4ecd1f0b11dd8#diff-eaafd22a79065f5b8d28318d482e650d
https://github.com/tesseract-ocr/tesseract/commit/b34cf9d424e88cd09aaa193697127c90ff76e0ce#diff-eaafd22a79065f5b8d28318d482e650d

Thanks for the quick response @Shreeshrii
Here is the update condition:

At the attachment, we have 2 fonts with Balinese-unicode, namely Vimala (most similar to the non-Balinese-unicode Bali SImbar Dwijendra) and Noto Sans Balinese (like Javanese that has Noto Sans Javanese).
I want to use https://github.com/Shreeshrii/tessdata_jav_java as a base for training with my Balinese training text. See the attachment for the Balinese version of  Article 1 of the Universal Declaration of Human Rights (https://en.wikipedia.org/wiki/Balinese_script). And about three code for that text, I don't know, jav for javanese, bal for balinese?

The question is how do I do that? For several hours try to learn and get the strategy but still far away..
bal.training_text.txt
balinese-unicode.zip

UDHR is a small text. You will need larger text for training.
LSTM training takes time, days and weeks.
On Mon, Mar 23, 2020, 10:17 gindrawan<EMAIL_ADDRESS>wrote:

Thanks for the quick response @Shreeshrii https://github.com/Shreeshrii
Here is the update condition:

At the attachment, we have 2 fonts with Balinese-unicode, namely
Vimala (most similar to the non-Balinese-unicode Bali SImbar Dwijendra) and
Noto Sans Balinese (like Javanese that has Noto Sans Javanese).
I want to use https://github.com/Shreeshrii/tessdata_jav_java as a
base for training with my Balinese training text. See the attachment for
the Balinese version of Article 1 of the Universal Declaration of Human
Rights (https://en.wikipedia.org/wiki/Balinese_script). And about
three code for that text, I don't know, jav for javanese, bal for balinese?

The question is how do I do that? For several hours try to learn and get
the strategy but still far away..
bal.training_text.txt
https://github.com/tesseract-ocr/langdata/files/4367222/bal.training_text.txt
balinese-unicode.zip
https://github.com/tesseract-ocr/langdata/files/4367183/balinese-unicode.zip
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tesseract-ocr/langdata/issues/126#issuecomment-602383280,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABG37I4MBIBCLURDZ7ZFML3RI3SXFANCNFSM4E35V7YA
.

UDHR is a small text. You will need larger text for training. LSTM training takes time, days and weeks.
…
On Mon, Mar 23, 2020, 10:17 gindrawan @.***> wrote: Thanks for the quick response @Shreeshrii https://github.com/Shreeshrii Here is the update condition: 1. At the attachment, we have 2 fonts with Balinese-unicode, namely Vimala (most similar to the non-Balinese-unicode Bali SImbar Dwijendra) and Noto Sans Balinese (like Javanese that has Noto Sans Javanese). 2. I want to use https://github.com/Shreeshrii/tessdata_jav_java as a base for training with my Balinese training text. See the attachment for the Balinese version of Article 1 of the Universal Declaration of Human Rights (https://en.wikipedia.org/wiki/Balinese_script). And about three code for that text, I don't know, jav for javanese, bal for balinese? The question is how do I do that? For several hours try to learn and get the strategy but still far away.. bal.training_text.txt https://github.com/tesseract-ocr/langdata/files/4367222/bal.training_text.txt balinese-unicode.zip https://github.com/tesseract-ocr/langdata/files/4367183/balinese-unicode.zip — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#126 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABG37I4MBIBCLURDZ7ZFML3RI3SXFANCNFSM4E35V7YA .

Yes, I know that.. I want to start from small training text first and incrementally add later (if possible) while get more understanding to the learning process. I have already had larger training text in Noto Sans Balinese (up to 30 thousand words and possibility doubling it for Vimala). More likely, the number continue to grow since there are other sources that still haven't processed yet. I dont know if that number is enough..

Language and script codes follow the assigned names as per standards bodies.
Balinese language three letter code is ban. Balinese script is bali. The
script can be used for a couple of other languages also.

I suggest moving discussion to issue at #152
https://github.com/tesseract-ocr/langdata/issues/152 (Balinese script OCR
On Mon, Mar 23, 2020, 17:59 Shree Devi Kumar<EMAIL_ADDRESS>wrote:

Language and script codes follow the assigned names as per standards
bodies.
Balinese language three letter code is ban. Balinese script is bali. The
script can be used for a couple of other languages also.

Ok, thanks. I'll post the update at https://github.com/tesseract-ocr/langdata/issues/152.

@topherseance: if you're still looking for the Javanese OCR, a team in UKDW is working on it.

https://trawaca.id/?lang=en
https://meta.wikimedia.org/wiki/WikiNusantara_2019/Acara/OCR_Aksara_Jawa (presentation on Indonesian Wikipedia Meetup)
https://commons.wikimedia.org/wiki/Early_Javanese_books - PDFs of Javanese script.

@topherseance: if you're still looking for the Javanese OCR, a team in UKDW is working on it.

https://trawaca.id/?lang=en
https://meta.wikimedia.org/wiki/WikiNusantara_2019/Acara/OCR_Aksara_Jawa (presentation on Indonesian Wikipedia Meetup)
https://commons.wikimedia.org/wiki/Early_Javanese_books - PDFs of Javanese script.

@shreeshrii & @topherseance: there are more than 20 Javanese script fonts available here:
https://bennylin.github.io/keyboards/jawa-fonts.html

@bennylin Are these Unicode fonts?

Yes

Are there any labelled datasets with scanned images and their Unicode groundtruth transcription that can be used for training/testing tesseract's jav-java traineddata?
What accuracy did the UKDW ocr achieve?

I'm not in the loop for the research. You might want to contact Dr. Lucia Krisnawati for that.


Auth Cookie Sharing topic updates

Addresses #5495
Fixes #12432
Internal Review Topic

No rush on review.
This would have been a 2.2 pass to get the samps updated, BUT...
Let's drop the samples! Plz! :smile: The long-range plan is to have the ones in engineering cross-linked in here anyway.  The upgrade of that samp when I did upgrades here for Core 2.0 to 2.1 went fine. However, it just broke again when trying to update it to 2.2 😝 ... like it did back when I did the 1.x to 2.0 round of updates. If you have the engineering link to shared cookie sample apps, I'll place it in here. Even if not, see if this version of the topic without the samples works right now, and we can put the engineering samples link in here later.
Let's go ahead and fix Confusion about "Use a common user database" for ASP.NET Core + ASP.NET 4.x case #12432 ... Can you provide the line for the Use a common user database section that makes sense? I think this came up once before, and the answer might have been no 🎲🎲 - not going to happen on a shared dB given the alternate Identity schemas. How would u like to address it?

Cookie interop samples/tests got dropped from 3.0 branches entirely, so yeah best to just remove them from now.  The 2.2 source should still work for the most part

@HaoK Sounds good ... much better than occasionally battling with the NETFX sample.
This can be reviewed anytime ... no rush. It's ready whenever u are.

The schemas for Core and 4.x should not be too different, but to share a database, the newest version of identity should 'own' the database, and all older versions of EF should add migrations to account for any schema changes.  @ajcvickers is that an apt description of what the recommendation approach for having multiple versions of identity target the same shared database?

Thx ... I'll wait for Arthur's response and then react to that before calling for a docs review + merging this.

Perhaps migrations aren't the right word to use. Here's a link to the old POCO classes that were needed to adapt the old 4.x identity to work against identity core's schema.  Basically the idea was adding properties to the POCO to match the identity core schema, but the older identity frameworks would just ignore these columns

adding properties to the POCO to match the identity core schema

That's what I was thinking (guessing). Something like this ...

DA or Fluent should be able to remap where needed if the names are merely different. The older-identity app needs an IdentityUser class in the app in order to apply annotations. (With Fluent tho, would that even be required?)
For any columns in the common dB that only support the newer-Identity app, the older-Identity app will ignore them.
For any columns in the common dB that only support the older-Identity app, the newer-Identity app ignores them.
WRT to the use of migrations on either side, migrations ignore columns not part of that's app's model schema.

I was just curious to see the deltas (Core 3.0.0-preview6.19307.2 vs. NETFX 2.2.2) ...
IdentityUser from Microsoft.Extensions.Identity.Stores ...
public virtual string SecurityStamp { get; set; }
public virtual bool PhoneNumberConfirmed { get; set; }
public virtual string PhoneNumber { get; set; }
public virtual string PasswordHash { get; set; }
public virtual string NormalizedUserName { get; set; }
public virtual string NormalizedEmail { get; set; }
public virtual DateTimeOffset? LockoutEnd { get; set; }
public virtual bool LockoutEnabled { get; set; }
public virtual TKey Id { get; set; }
public virtual bool EmailConfirmed { get; set; }
public virtual string Email { get; set; }
public virtual string ConcurrencyStamp { get; set; }
public virtual int AccessFailedCount { get; set; }
public virtual bool TwoFactorEnabled { get; set; }
public virtual string UserName { get; set; }

IdentityUser from Microsoft.AspNet.Identity.EntityFramework ...
public virtual string Email { get; set; }
public virtual bool EmailConfirmed { get; set; }
public virtual string PasswordHash { get; set; }
public virtual string SecurityStamp { get; set; }
public virtual string PhoneNumber { get; set; }
public virtual bool PhoneNumberConfirmed { get; set; }
public virtual bool TwoFactorEnabled { get; set; }
public virtual DateTime? LockoutEndDateUtc { get; set; }
public virtual bool LockoutEnabled { get; set; }
public virtual int AccessFailedCount { get; set; }
public virtual ICollection<TRole> Roles { get; }
public virtual ICollection<TClaim> Claims { get; }
public virtual ICollection<TLogin> Logins { get; }
public virtual TKey Id { get; set; }
public virtual string UserName { get; set; }

I can see why this gets a bit 😵 ... and roles and claims are handled differently, which makes it even more challenging.
The doc challenge here is that a few sentences that describe the overall scenario probably won't be all that helpful. Is it a viable remark to say that in many scenarios it's better to upgrade older apps and use the newer Identity everywhere? ... that long-term it's actually easier to do that than to build all of this infrastructure and maintain it.

Yeah I don't think we've gotten more than one or two requests asking about this either, so I don't think its something many people are trying to do

@Rick-Anderson Updated and ready to go.


basic working demo example

I'm fine with this, but it seems like you're not using the other libraries (in features, etc.) That should be the next on your list to do...just integrate with the code that we have.


Fix BC for trees

Fix for BC for trees

make test

make test

make test

make test

make test


Confusing UX for editable columns

I worked out by trial and error that if you make changes to fields in the grid column, you need to click the overall page's Save or Save & Publish button to commit them.
That works okay once you know how but I find it quite unintuitive as I associate those buttons as only making changes to the page, not any related objects shown in gridifields. This is especially true if the items in the gridfield aren't directly related to the page you're viewing. I think I'm not alone in finding this confusing judging by #65 and #147.
Think it would be a much better user experience to have save buttons that are localised to each individual row in a gridfield. Perhaps after making an edit a Save button could slide out underneath the row? Or there could be an always visible Save button in each row, but it's only semi-opaque until you actually make changes?
This issue aside, it's a very useful module so thanks for that :)

#32 is also related. If there is no Save button on the page you can't save changes currently. Having a Save button on each row would fix that.


[FLINK-11613][chinese-translation,Documentation]Translate the "Project Template for Scala" page into Chinese

What is the purpose of the change
This pull request completes the Chinese translation of "Project Template for Scala" page from official document.
Brief change log

Translate the "Project Template for Scala" page into Chinese

Verifying this change
This change is to add a new translated document.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no

need to modify commit log