Error in Training Tesseract LSTM 4.0

Hello All,
I want to train tesseract as , But I got this error, Any help
 tesstrain.sh --fonts_dir /usr/share/fonts --lang ara  --training_text ../../tesserac-ocr/langdata/ara.training_text   --langdata_dir ../langdata --tessdata_dir ./tessdata   --fontlist "Arial"   --output_dir ~/tesstutorial/aratest
=== Starting training for language 'ara'
[ر أبر 12 07:48:58 EET 2017] /usr/bin/text2image --fonts_dir=/usr/share/fonts --font=Arial --outputbase=/tmp/font_tmp.Rj3QkZFztb/sample_text.txt --text=/tmp/font_tmp.Rj3QkZFztb/sample_text.txt --fontconfig_tmpdir=/tmp/font_tmp.Rj3QkZFztb
Rendered page 0 to file /tmp/font_tmp.Rj3QkZFztb/sample_text.txt.tif

=== Phase I: Generating training images ===
Rendering using Arial
[ر أبر 12 07:49:17 EET 2017] /usr/bin/text2image --fontconfig_tmpdir=/tmp/font_tmp.Rj3QkZFztb --fonts_dir=/usr/share/fonts --strip_unrenderable_words --fontconfig_refresh_config_file=false --leading=32 --char_spacing=0.0 --exposure=0 --outputbase=/tmp/tmp.XBcy4TVQwb/ara/ara.Arial.exp0 --font=Arial --text=../../tesserac-ocr/langdata/ara.training_text
ERROR: Non-existent flag --fontconfig_refresh_config_file=false
ERROR: /tmp/tmp.XBcy4TVQwb/ara/ara.Arial.exp0.box does not exist or is not readable
ERROR: /tmp/tmp.XBcy4TVQwb/ara/ara.Arial.exp0.box does not exist or is not readable

--training_text ../../tesserac-ocr/langdata/ara.training_text
is there a typo in your folder name tesserac instead of tesseract?

$ training/tesstrain.sh  \
   --fonts_dir /usr/share/fonts/ \
  --lang ara   \
  --linedata_only \
  --noextract_font_properties \
  --exposures "0"    \
  --langdata_dir ../langdata \
  --tessdata_dir ../tessdata  \
  --output_dir ~/tesstutorial/ara  \
  --fontlist "Arial"

=== Starting training for language 'ara'
[Wed Apr 12 14:55:47 DST 2017] /usr/local/bin/text2image --fonts_dir=/usr/share/fonts/ --font=Arial --outputbase=/tmp/font_tmp.OIUrMxATnl/sample_text.txt --text=/tmp/f
ont_tmp.OIUrMxATnl/sample_text.txt --fontconfig_tmpdir=/tmp/font_tmp.OIUrMxATnl
Rendered page 0 to file /tmp/font_tmp.OIUrMxATnl/sample_text.txt.tif

=== Phase I: Generating training images ===
Rendering using Arial
[Wed Apr 12 14:56:35 DST 2017] /usr/local/bin/text2image --fontconfig_tmpdir=/tmp/font_tmp.OIUrMxATnl --fonts_dir=/usr/share/fonts/ --strip_unrenderable_words --leadin
g=32 --xsize=2550 --ptsize=16 --char_spacing=0.0 --exposure=0 --outputbase=/tmp/tmp.xPjs35P5oP/ara/ara.Arial.exp0 --font=Arial --text=../langdata/ara/ara.training_text
0
Rendered page 0 to file /tmp/tmp.xPjs35P5oP/ara/ara.Arial.exp0.tif
/tmp/tmp.xPjs35P5oP/ara/gt/ara.Arial.exp0.txt

=== Phase UP: Generating unicharset and unichar properties files ===
[Wed Apr 12 14:56:38 DST 2017] /usr/local/bin/unicharset_extractor -D /tmp/tmp.xPjs35P5oP/ara/ /tmp/tmp.xPjs35P5oP/ara/ara.Arial.exp0.box
Extracting unicharset from /tmp/tmp.xPjs35P5oP/ara/ara.Arial.exp0.box
Wrote unicharset file /tmp/tmp.xPjs35P5oP/ara//unicharset.
[Wed Apr 12 14:56:39 DST 2017] /usr/local/bin/set_unicharset_properties -U /tmp/tmp.xPjs35P5oP/ara/ara.unicharset -O /tmp/tmp.xPjs35P5oP/ara/ara.unicharset -X /tmp/tmp
.xPjs35P5oP/ara/ara.xheights --script_dir=../langdata
Loaded unicharset of size 187 from file /tmp/tmp.xPjs35P5oP/ara/ara.unicharset
Setting unichar properties
Writing unicharset to file /tmp/tmp.xPjs35P5oP/ara/ara.unicharset

=== Phase D: Generating Dawg files ===
Generating word Dawg
[Wed Apr 12 14:56:41 DST 2017] /usr/local/bin/wordlist2dawg -r 1 ../langdata/ara/ara.wordlist /tmp/tmp.xPjs35P5oP/ara/ara.word-dawg /tmp/tmp.xPjs35P5oP/ara/ara.unichar
set
Set reverse_policy to RRP_REVERSE_IF_HAS_RTL
Loading unicharset from '/tmp/tmp.xPjs35P5oP/ara/ara.unicharset'
Reading word list from '../langdata/ara/ara.wordlist'
Reducing Trie to SquishedDawg
Writing squished DAWG to '/tmp/tmp.xPjs35P5oP/ara/ara.word-dawg'
Generating frequent-word Dawg
[Wed Apr 12 14:56:46 DST 2017] /usr/local/bin/wordlist2dawg -r 1 /tmp/tmp.xPjs35P5oP/ara/ara.wordlist.clean.freq /tmp/tmp.xPjs35P5oP/ara/ara.freq-dawg /tmp/tmp.xPjs35P
5oP/ara/ara.unicharset
Set reverse_policy to RRP_REVERSE_IF_HAS_RTL
Loading unicharset from '/tmp/tmp.xPjs35P5oP/ara/ara.unicharset'
Reading word list from '/tmp/tmp.xPjs35P5oP/ara/ara.wordlist.clean.freq'
Reducing Trie to SquishedDawg
Writing squished DAWG to '/tmp/tmp.xPjs35P5oP/ara/ara.freq-dawg'
[Wed Apr 12 14:56:47 DST 2017] /usr/local/bin/wordlist2dawg -r 2 ../langdata/ara/ara.punc /tmp/tmp.xPjs35P5oP/ara/ara.punc-dawg /tmp/tmp.xPjs35P5oP/ara/ara.unicharset
Set reverse_policy to RRP_FORCE_REVERSE
Loading unicharset from '/tmp/tmp.xPjs35P5oP/ara/ara.unicharset'
Reading word list from '../langdata/ara/ara.punc'
Reducing Trie to SquishedDawg
Writing squished DAWG to '/tmp/tmp.xPjs35P5oP/ara/ara.punc-dawg'
[Wed Apr 12 14:56:47 DST 2017] /usr/local/bin/wordlist2dawg -r 0 ../langdata/ara/ara.numbers /tmp/tmp.xPjs35P5oP/ara/ara.number-dawg /tmp/tmp.xPjs35P5oP/ara/ara.unicha
rset
Set reverse_policy to RRP_DO_NO_REVERSE
Loading unicharset from '/tmp/tmp.xPjs35P5oP/ara/ara.unicharset'
Reading word list from '../langdata/ara/ara.numbers'
Reducing Trie to SquishedDawg
Writing squished DAWG to '/tmp/tmp.xPjs35P5oP/ara/ara.number-dawg'
[Wed Apr 12 14:56:47 DST 2017] /usr/local/bin/wordlist2dawg -r 1 ../langdata/ara/ara.word.bigrams /tmp/tmp.xPjs35P5oP/ara/ara.bigram-dawg /tmp/tmp.xPjs35P5oP/ara/ara.u
nicharset
Set reverse_policy to RRP_REVERSE_IF_HAS_RTL
Loading unicharset from '/tmp/tmp.xPjs35P5oP/ara/ara.unicharset'
Reading word list from '../langdata/ara/ara.word.bigrams'
Reducing Trie to SquishedDawg
Writing squished DAWG to '/tmp/tmp.xPjs35P5oP/ara/ara.bigram-dawg'

=== Phase E: Generating lstmf files ===
Using TESSDATA_PREFIX=../tessdata
[Wed Apr 12 14:57:00 DST 2017] /usr/local/bin/tesseract /tmp/tmp.xPjs35P5oP/ara/ara.Arial.exp0.tif /tmp/tmp.xPjs35P5oP/ara/ara.Arial.exp0 lstm.train ../langdata/ara/ar
a.config
Tesseract Open Source OCR Engine v4.00.00alpha-361-g1477e17 with Leptonica
Page 1

=== Constructing LSTM training data ===
Creating new directory /home/shree/tesstutorial/ara
Copying ../langdata/ara/ara.config to /home/shree/tesstutorial/ara
Moving /tmp/tmp.xPjs35P5oP/ara/ara.unicharset to /home/shree/tesstutorial/ara
Moving /tmp/tmp.xPjs35P5oP/ara/ara.number-dawg to /home/shree/tesstutorial/ara/ara.lstm-number-dawg
Moving /tmp/tmp.xPjs35P5oP/ara/ara.punc-dawg to /home/shree/tesstutorial/ara/ara.lstm-punc-dawg
Moving /tmp/tmp.xPjs35P5oP/ara/ara.word-dawg to /home/shree/tesstutorial/ara/ara.lstm-word-dawg
Moving /tmp/tmp.xPjs35P5oP/ara/ara.Arial.exp0.box to /home/shree/tesstutorial/ara
Moving /tmp/tmp.xPjs35P5oP/ara/ara.Arial.exp0.tif to /home/shree/tesstutorial/ara
Moving /tmp/tmp.xPjs35P5oP/ara/ara.Arial.exp0.lstmf to /home/shree/tesstutorial/ara
Moving /tmp/tmp.xPjs35P5oP/ara/gt/ara.Arial.exp0.txt to /home/shree/tesstutorial/ara

Completed training for language 'ara'

shree@ALL-IN-1-TOUCH:/mnt/c/Users/User/shree/tesseract-ocr$

@Shreeshrii Thanks a lot, one more question please to get full understanding of it --langdata_dir parameter is the directory of
again thanks for replying!

Put Arabic langdata files under ara folder in langdata - similar to
https://github.com/tesseract-ocr/langdata
so that you have
./langdata
./langdata/ara
./tessdata
./tesseract
./tesseract/tessdata
./tesseract/training
etc

I got it, Thanks @Shreeshrii

Shree, please redirect people that ask questions about the training process to the forum.
Thanks!

Hello,@Shreeshrii
I have the same problem like @am0awad .
` training/tesstrain.sh  \

--fonts_dir /Library/Fonts
--lang chi_sim
--linedata_only
--noextract_font_properties
--exposures "0"
--langdata_dir ./langdata
--tessdata_dir ./tessdata
--output_dir ./tesstutorial/chi_sim `

I have already download the https://github.com/tesseract-ocr/langdata to my folder, and I have all the folders:
./langdata
./langdata/chi_sim
./tessdata
but I still got this:
`=== Starting training for language 'chi_sim'
mktemp: illegal option -- -
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
mktemp [-d] [-q] [-u] -t prefix
training/tesstrain_utils.sh: line 189: /sample_text.txt: Permission denied
[2017年 8月 3日 星期四 15时23分24秒 CST] /usr/local/bin/text2image --fonts_dir=/Library/Fonts --font=AR PL UKai CN Light --outputbase=/sample_text.txt --text=/sample_text.txt --fontconfig_tmpdir=
=== Phase I: Generating training images ===
Rendering using Arial Unicode MS
Rendering using AR PL UKai CN Light
[2017年 8月 3日 星期四 15时23分25秒 CST] /usr/local/bin/text2image --fontconfig_tmpdir= --fonts_dir=/Library/Fonts --strip_unrenderable_words --leading=32 --char_spacing=0.0 --exposure=0 --outputbase=/var/folders/br/xq_2dfh91y1gdd90lny2501w0000gn/T/tmp.NTXG7RhZ/chi_sim/chi_sim.AR_PL_UKai_CN_Light.exp0 --font=AR PL UKai CN Light --text=./langdata/chi_sim/chi_sim.training_text
[2017年 8月 3日 星期四 15时23分25秒 CST] /usr/local/bin/text2image --fontconfig_tmpdir= --fonts_dir=/Library/Fonts --strip_unrenderable_words --leading=32 --char_spacing=0.0 --exposure=0 --outputbase=/var/folders/br/xq_2dfh91y1gdd90lny2501w0000gn/T/tmp.NTXG7RhZ/chi_sim/chi_sim.Arial_Unicode_MS.exp0 --font=Arial Unicode MS --text=./langdata/chi_sim/chi_sim.training_text
ERROR: /var/folders/br/xq_2dfh91y1gdd90lny2501w0000gn/T/tmp.NTXG7RhZ/chi_sim/chi_sim.Arial_Unicode_MS.exp0.box does not exist or is not readable
ERROR: /var/folders/br/xq_2dfh91y1gdd90lny2501w0000gn/T/tmp.NTXG7RhZ/chi_sim/chi_sim.AR_PL_UKai_CN_Light.exp0.box does not exist or is not readable
ERROR: /var/folders/br/xq_2dfh91y1gdd90lny2501w0000gn/T/tmp.NTXG7RhZ/chi_sim/chi_sim.AR_PL_UKai_CN_Light.exp0.box does not exist or is not readable`
would anyone like to help me?
Thank you very much!

Ray has updated the LSTM training process.
Please install the latest code from github.
Please read the wiki again for the new instructions and try.
ShreeDevi

भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com
On Thu, Aug 3, 2017 at 1:07 PM, yoyoshuang<EMAIL_ADDRESS>wrote:

Hello,@Shreeshrii https://github.com/shreeshrii
I have the same problem like @am0awad https://github.com/am0awad .
` training/tesstrain.sh \
--fonts_dir /Library/Fonts
--lang chi_sim
--linedata_only
--noextract_font_properties
--exposures "0"
--langdata_dir ./langdata
--tessdata_dir ./tessdata
--output_dir ./tesstutorial/chi_sim `
I have already download the https://github.com/tesseract-ocr/langdata
http://url to my folder, and I have all the folders:
./langdata
./langdata/chi_sim
./tessdata
but I still got this:
`=== Starting training for language 'chi_sim'
mktemp: illegal option -- -
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
mktemp [-d] [-q] [-u] -t prefix
training/tesstrain_utils.sh: line 189: /sample_text.txt: Permission denied
[2017年 8月 3日 星期四 15时23分24秒 CST] /usr/local/bin/text2image
--fonts_dir=/Library/Fonts --font=AR PL UKai CN Light
--outputbase=/sample_text.txt --text=/sample_text.txt --fontconfig_tmpdir=
=== Phase I: Generating training images ===
Rendering using Arial Unicode MS
Rendering using AR PL UKai CN Light
[2017年 8月 3日 星期四 15时23分25秒 CST] /usr/local/bin/text2image
--fontconfig_tmpdir= --fonts_dir=/Library/Fonts --strip_unrenderable_words
--leading=32 --char_spacing=0.0 --exposure=0 --outputbase=/var/folders/br/
xq_2dfh91y1gdd90lny2501w0000gn/T/tmp.NTXG7RhZ/chi_sim/chi_sim.AR_PL_UKai_CN_Light.exp0
--font=AR PL UKai CN Light --text=./langdata/chi_sim/chi_sim.training_text
[2017年 8月 3日 星期四 15时23分25秒 CST] /usr/local/bin/text2image
--fontconfig_tmpdir= --fonts_dir=/Library/Fonts --strip_unrenderable_words
--leading=32 --char_spacing=0.0 --exposure=0 --outputbase=/var/folders/br/
xq_2dfh91y1gdd90lny2501w0000gn/T/tmp.NTXG7RhZ/chi_sim/chi_sim.Arial_Unicode_MS.exp0
--font=Arial Unicode MS --text=./langdata/chi_sim/chi_sim.training_text
ERROR: /var/folders/br/xq_2dfh91y1gdd90lny2501w0000gn/T/
tmp.NTXG7RhZ/chi_sim/chi_sim.Arial_Unicode_MS.exp0.box does not exist or
is not readable
ERROR: /var/folders/br/xq_2dfh91y1gdd90lny2501w0000gn/T/
tmp.NTXG7RhZ/chi_sim/chi_sim.AR_PL_UKai_CN_Light.exp0.box does not exist
or is not readable
ERROR: /var/folders/br/xq_2dfh91y1gdd90lny2501w0000gn/T/
tmp.NTXG7RhZ/chi_sim/chi_sim.AR_PL_UKai_CN_Light.exp0.box does not exist
or is not readable`
would anyone like to help me?
Thank you very much!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tesseract-ocr/tesseract/issues/819#issuecomment-319892740,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE2_o-thKFCnUFwGRYnGQlFCzaFHXksYks5sUXiggaJpZM4M6-yu
.

Thank you @Shreeshrii I solved my problem by your help! Thank you very much!

@zdenop, please close this issue.

@Shreeshrii at Phase I,  I got this error:
Rendered page 2 to file /tmp/tmp.GZVqm2mm3D/fas/fas.Times_New_Roman.exp0.tif
Null box at index 0
Error: Call PrepareToWrite before WriteTesseractBoxFile!!
but the process did not stop. and it got this error:
ERROR: /tmp/tmp.GZVqm2mm3D/fas/fas.Times_New_Roman.exp0.lstmf does not exist or is not readable
probably, the problem is in the very first phase. would you please help me solve it?

Look at the tif file in question in tmp folder. Looks like first line has
nulls in it.
On 17-Oct-2017 5:59 PM, "hanikh"<EMAIL_ADDRESS>wrote:

@Shreeshrii https://github.com/shreeshrii at Phase I, I got this error:
Rendered page 2 to file /tmp/tmp.GZVqm2mm3D/fas/fas.
Times_New_Roman.exp0.tif
Null box at index 0
Error: Call PrepareToWrite before WriteTesseractBoxFile!!
but the process did not stop. and it got this error:
ERROR: /tmp/tmp.GZVqm2mm3D/fas/fas.Times_New_Roman.exp0.lstmf does not
exist or is not readable
probably, the problem is in the very first phase. would you please help me
solve it?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tesseract-ocr/tesseract/issues/819#issuecomment-337215631,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE2_o0n0rSz-Z9XhUN2g4ge8DsQXIZi9ks5stJ25gaJpZM4M6-yu
.

yes. you're right. in fact, there are just numbers and some marks in tif
file. why is that happening?
On Tue, Oct 17, 2017 at 5:29 PM, Shreeshrii<EMAIL_ADDRESS>wrote:

Look at the tif file in question in tmp folder. Looks like first line has
nulls in it.
On 17-Oct-2017 5:59 PM, "hanikh"<EMAIL_ADDRESS>wrote:

@Shreeshrii https://github.com/shreeshrii at Phase I, I got this
error:
Rendered page 2 to file /tmp/tmp.GZVqm2mm3D/fas/fas.
Times_New_Roman.exp0.tif
Null box at index 0
Error: Call PrepareToWrite before WriteTesseractBoxFile!!
but the process did not stop. and it got this error:
ERROR: /tmp/tmp.GZVqm2mm3D/fas/fas.Times_New_Roman.exp0.lstmf does not
exist or is not readable
probably, the problem is in the very first phase. would you please help
me
solve it?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/tesseract-ocr/tesseract/issues/819#
issuecomment-337215631>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AE2_o0n0rSz-
Z9XhUN2g4ge8DsQXIZi9ks5stJ25gaJpZM4M6-yu>
.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tesseract-ocr/tesseract/issues/819#issuecomment-337240137,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AZFiASKi4zAcvT6hL8SkvIybrnjQVMWrks5stLLOgaJpZM4M6-yu
.

Does Times New Roman font support Farsi? You should use fonts that have
support for your training_text language.
ShreeDevi

भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com
On Tue, Oct 17, 2017 at 9:13 PM, hanikh<EMAIL_ADDRESS>wrote:

yes. you're right. in fact, there are just numbers and some marks in tif
file. why is that happening?
On Tue, Oct 17, 2017 at 5:29 PM, Shreeshrii<EMAIL_ADDRESS>wrote:

Look at the tif file in question in tmp folder. Looks like first line has
nulls in it.
On 17-Oct-2017 5:59 PM, "hanikh"<EMAIL_ADDRESS>wrote:

@Shreeshrii https://github.com/shreeshrii at Phase I, I got this
error:
Rendered page 2 to file /tmp/tmp.GZVqm2mm3D/fas/fas.
Times_New_Roman.exp0.tif
Null box at index 0
Error: Call PrepareToWrite before WriteTesseractBoxFile!!
but the process did not stop. and it got this error:
ERROR: /tmp/tmp.GZVqm2mm3D/fas/fas.Times_New_Roman.exp0.lstmf does not
exist or is not readable
probably, the problem is in the very first phase. would you please help
me
solve it?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/tesseract-ocr/tesseract/issues/819#
issuecomment-337215631>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AE2_o0n0rSz-
Z9XhUN2g4ge8DsQXIZi9ks5stJ25gaJpZM4M6-yu>
.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<https://github.com/tesseract-ocr/tesseract/issues/819#
issuecomment-337240137>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/
AZFiASKi4zAcvT6hL8SkvIybrnjQVMWrks5stLLOgaJpZM4M6-yu>
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tesseract-ocr/tesseract/issues/819#issuecomment-337269936,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE2_o5yHcKCu2plw2oV9eKheXyc1opAYks5stMsagaJpZM4M6-yu
.

Times New Roman is in the list of Persian fonts in
training/language_specific.sh. Doesn’t it mean that the Farsi trained model
is trained for Times New Roman?Or the listed fonts are just recommended?
On Wed, Oct 18, 2017 at 5:56 AM, Shreeshrii<EMAIL_ADDRESS>wrote:

Does Times New Roman font support Farsi? You should use fonts that have
support for your training_text language.
ShreeDevi

भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com
On Tue, Oct 17, 2017 at 9:13 PM, hanikh<EMAIL_ADDRESS>wrote:

yes. you're right. in fact, there are just numbers and some marks in tif
file. why is that happening?
On Tue, Oct 17, 2017 at 5:29 PM, Shreeshrii<EMAIL_ADDRESS>wrote:

Look at the tif file in question in tmp folder. Looks like first line
has
nulls in it.
On 17-Oct-2017 5:59 PM, "hanikh"<EMAIL_ADDRESS>wrote:

@Shreeshrii https://github.com/shreeshrii at Phase I, I got this
error:
Rendered page 2 to file /tmp/tmp.GZVqm2mm3D/fas/fas.
Times_New_Roman.exp0.tif
Null box at index 0
Error: Call PrepareToWrite before WriteTesseractBoxFile!!
but the process did not stop. and it got this error:
ERROR: /tmp/tmp.GZVqm2mm3D/fas/fas.Times_New_Roman.exp0.lstmf does
not
exist or is not readable
probably, the problem is in the very first phase. would you please
help
me
solve it?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/tesseract-ocr/tesseract/issues/819#
issuecomment-337215631>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AE2_o0n0rSz-
Z9XhUN2g4ge8DsQXIZi9ks5stJ25gaJpZM4M6-yu>
.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<https://github.com/tesseract-ocr/tesseract/issues/819#
issuecomment-337240137>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/
AZFiASKi4zAcvT6hL8SkvIybrnjQVMWrks5stLLOgaJpZM4M6-yu>
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/tesseract-ocr/tesseract/issues/819#
issuecomment-337269936>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AE2_
o5yHcKCu2plw2oV9eKheXyc1opAYks5stMsagaJpZM4M6-yu>

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tesseract-ocr/tesseract/issues/819#issuecomment-337441751,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AZFiAXjsDJTN8DS85eiZHWvQF07HIP6Tks5stWHigaJpZM4M6-yu
.

still getting getting this error
-e
=== Starting training for language 'hin'
-e [Fri Apr 27 13:21:33 IST 2018] /usr/local/bin/text2image --fonts_dir=/Users/ReveriePC/Library/Fonts --font=Aparajita Bold Italic --outputbase=/var/folders/b_/kz0byg9j4mv9snmglhny3d_80000gp/T/font_tmp.XXXXXXXXXX.btO1NzLv/sample_text.txt --text=/var/folders/b_/kz0byg9j4mv9snmglhny3d_80000gp/T/font_tmp.XXXXXXXXXX.btO1NzLv/sample_text.txt --fontconfig_tmpdir=/var/folders/b_/kz0byg9j4mv9snmglhny3d_80000gp/T/font_tmp.XXXXXXXXXX.btO1NzLv
-e
=== Phase I: Generating training images ===
-e Rendering using Aparajita Bold Italic
-e [Fri Apr 27 13:21:35 IST 2018] /usr/local/bin/text2image --fontconfig_tmpdir=/var/folders/b_/kz0byg9j4mv9snmglhny3d_80000gp/T/font_tmp.XXXXXXXXXX.btO1NzLv --fonts_dir=/Users/ReveriePC/Library/Fonts --strip_unrenderable_words --leading=32 --char_spacing=0.0 --exposure=0 --outputbase=/var/folders/b_/kz0byg9j4mv9snmglhny3d_80000gp/T/tmp.JHxdRWug/hin/hin.Aparajita_Bold_Italic.exp0 --max_pages=0 --font=Aparajita Bold Italic --text=../langdata/hin/hin.training_text
-e ERROR: /var/folders/b_/kz0byg9j4mv9snmglhny3d_80000gp/T/tmp.JHxdRWug/hin/hin.Aparajita_Bold_Italic.exp0.box does not exist or is not readable
-e ERROR: /var/folders/b_/kz0byg9j4mv9snmglhny3d_80000gp/T/tmp.JHxdRWug/hin/hin.Aparajita_Bold_Italic.exp0.box does not exist or is not readable

Tried with just Aparajita, same error!

Do you have the font in --fonts_dir=/Users/ReveriePC/Library/Fonts --font=Aparajita
Try to rebuild font cache and try.
fc-cache -vf
Do other devanagari fonts work with same training text?

Yes, fonts are in that directory .
Tried rebuilding font cache , it said : "fish: The file '/Users/ReveriePC/Library/Fonts' is not executable by this user"
It isn't working for any other devanagari fonts.
I am building it in a mac-os ? do I need to change some config ?

Tried on ubuntu machine , working  !

shree please help me out. I got an error that says  " Unrecognized argument --linedata_only--noextract_font_properties" when i try create the training data
the command that i used is
/usr/share/tesseract-ocr/./tesstrain.sh
--fonts_dir  /usr/share/fonts
--lang	ben
--linedata_only
--noextract_font_properties
--langdata_dir	/home/jennil/Desktop/pro/langdata-master/ben
--tessdata_dir /usr/share/tesseract-ocr/4.00/tessdata –output_dir
/home/jennil/Desktop/pro/output/ben_output
--fontlist  “Lohit Bengali”
please someone hep me out

Hi @Shreeshrii ,
I am unable to extract the lstm component from eng.traineddata in tessdata_best. I have tried a lot but unable to gain any success in this. Could you help me please.

What command did you try? What error do you get?

I have downloaded eng.traineddata from wget https://github.com/tesseract-ocr/tessdata_best/raw/master/eng.traineddata as mentioned in https://github.com/tesseract-ocr/tesseract/wiki/TrainingTesseract-4.00#creating-training-data.
I tried the following command ./src/training/combine_tessdata -e tessdata/eng.traineddata
~/tesstutorial/finetuning/eng.lstm and got the message that their is no lstm component.
and got these:
17:lstm:size=11689099, offset=192
18:lstm-punc-dawg:size=4322, offset=11689291
19:lstm-word-dawg:size=3694794, offset=11693613
20:lstm-number-dawg:size=4738, offset=15388407
21:lstm-unicharset:size=6360, offset=15393145
22:lstm-recoder:size=1012, offset=15399505
23:version:size=80, offset=15400517
I don't know what is going wrong over here.

cd tesseract/tessdata
mkdir best
cd best
wget https://github.com/tesseract-ocr/tessdata_best/raw/master/eng.traineddata

This will put eng.traineddata in   tesseract/tessdata/best
You are extracting from  tessdata/eng.traineddata
./src/training/combine_tessdata -e tessdata/eng.traineddata ~/tesstutorial/finetuning/eng.lstm


Caprese Alternative

Caprese is a major risk for Kyo's progress. Its development has been happening with little involvement from effect system developers, the proposed abstractions don't seem principled in the FP sense of programming with values, nor are they reusable in Kyo, and it might end up baking in a concrete effect system into the language itself.
In order to provide a new avenue for Kyo's evolution that mitigates this risk, the goal of this ticket is to provide a research compiler plugin to enable seamless direct syntax. The plugin should be generic and reusable with other effect systems as an alternative to Caprese. Initial design idea:
First-class Purity
A key challenge when performing the transformation from direct syntax to monadic composition is the risk of impure code changing the execution control flow and making the transformation invalid. Kyo and other effect systems rely on the user's discipline to suspend any impure code in an IO effect, but that's an error-prone approach that requires a level of discipline newcomers are typically not accustomed to. The convenience of the direct syntax and similarities to solutions in languages like Python make misuse even more likely to happen.
I've been thinking about the different ways we could introduce first-class support for expressing pure code in Scala to overcome this challenge. One alternative is introducing something like a fun keyword to be used in place of def or defining a new kind of pure function like T ~> U instead of T => U (I think I saw something like that in Caprese some time ago). The main challenge of such approaches is being able to also express that a class constructor is pure.
I'm leaning more towards introducing a new pure keyword to be used as a modifier for classes, constructors, and methods:
// an example of a pure method
pure def myPureFunction(i: Int) = i + 1

// all members, including the constructor, are automatically marked as pure as well
pure class MyPureClass(v: Int):
         def alsoPure = 42

// an example of a class with pure and impure members
class MyMixedClass pure (i: Int): // note the pure constructor
	pure def aPureMethod(j: Int) = i + j
	def anImpureMethod(j: Int) = println(i + j)

// we'd also need a way to declare pure anon functions
pure trait ~>[T, U]:
	def apply(v: T): U

It seems the rules for validating purity could be relatively straightforward:

A pure member can only invoke other members also marked as pure, including constructors.
For compatibility with pure code not using the pure modifier, which includes standard library APIs, we could compile a static set of pure symbols and add it to the plugin.
We should also provide a way for users to extend the static set of pure symbols.
Impure language constructs like var, try, throw, etc., would be disallowed in pure members.

Effectful Types
Effectful types like Kyo’s pending type, ZIO, and IO would need to provide APIs the same way they do for for-comprehensions, with no need for an implicit Monad or specific interface. It's not clear to me what would be a good syntax to express them, but we'd need a way to mark the type parameter that specifies the "result" type of the computation:
// maybe as an annotation?
type <[+T @effectful, -S]

type ZIO[+R, -E, T @effectful]

type IO[T @effectful]

Effectful types would have special type-checking behaviors:

Impure code would see the effectful type with its regular methods and type signature.
Pure code would be able to reference an effectful type only by its "result" type.
Methods that return an effectful type can use impure by-name parameters, allowing IOs(println(1)) for example.
Effectful types would be considered pure by default, without the need for the modifier, although they may use impure code internally. The developers of effectful types would be responsible for ensuring their purity.

With these building blocks, Kyo could be used in the following way:
// regular monadic composition (`Fibers.init` returns a `Fiber`)
def a: Int < Fibers = Fibers.init(1).flatMap(_.get).map(_ + 1)

// it’d continue invalid to reference an effectful type by its result type in impure code
def b = Fibers.init(1).get + 1 // compilation failure

// but a pure member can only reference the effectful type by its result type (`Fiber`)
pure def c: Int < Fibers = Fibers.init(1).get + 1

// even the regular members of the effectful type wouldn’t be accessible in a pure member
pure def c = Fibers.init(1).map(_.get) // compilation failure

// by-name parameters of methods that return effectful types would allow impure code
pure def d: Int < IOs = IOs(scala.Console.in.readLine()).toInt

Essentially, types like Int < Fibers, ZIO[Any, Throwable, Int], and IO[Int] would behave as Int in pure members. Any and every use of them implies a monadic bind by the syntax sugar transformation. An important imitation is that a pure member must use at most one effectful type, as it’s not possible to mix different monads like Kyo’s pending type and ZIO.
Notes

Please don't take this design as set in stone. The goal here is to explore something in this direction, not prescribe the final design.
I think dotty-cps-async might have something similar. I saw a compiler plugin but I couldn't find documentation for it.
I wonder if we could build something similar using annotation macros instead, since it might be easier to maintain in the long term.
I'd prefer to have an initial implementation of the solution before considering submitting it as a SIP proposal. I wouldn't rule out Kyo using the research plugin in the long term or even forking the compiler if needed to guarantee the success of the project.
We'd need to check if the transformation from direct to monadic can be done generically for effectful types with multiple type parameters with different variance annotations like ZIO.
I think an important challenge is ensuring the new type checking behavior would work well in IDEs, but I don't think that should block the development and could be addressed as a follow-up.

One-page documentation about the dotty-cps-async compiler plugin here: https://rssh.github.io/dotty-cps-async/BasicUsage.html#direct-context-encoding-experimental.   In short,  when we see a method with implicit CpsDirect[F] parameters (can be with carrying, i.e., a method which returns context function),  let's it m(a:A)(using CpsDirect[F]): B then we transform this method into a monadic form: m(a:A)(using CpsMonadContext[F]):F[B].
Note that with an effect system, we can do better than dotty-cps-async:  we can catch not application of F[_] (in effect-system case,  F[_] is something like the composition of effects, In kyo terms we can say something like:  F[X] = (A>B> .. )[X],  so we can catch in parameters something like CpsDirect[A], CpsDirect[B], ...C,  and assemble an F[(set-of-used-effects)] for each transformed method.  This can be one of the possible student projects in GSoC-2024.
Ok,  not let's try to understand the approach described here as @effectfull and how this differ from Capreso.

What means  T@effecfull: Does this means that the annotated effect type is something like type of effect (i.e.  IO,  FileSystem, ...etc).    If yes,  how does this differ from the capreso definition:   T {*}   ?    At first glance, this is the same.
Pure functions.  As I understand:
pure def c: Int < IOs = IOs(i + 1) + 1
is analog to
def c:  (x:IOs') ?=>  IOs(i+1)+1`` in Capresso encoding.    (where IOs'is the type that tracks the IO effect). Or not -- maybe from description it's more likedef c: (x:IOs' =>. IOs(i+1) ) + 1`
I.e., it looks like we have IO(x+1) in the proposed syntax. [without call methods on IO] in Carpeso is represented by context function from effect context.

Alternative encoding makes sense because using context functions for pure computations can be annoying, especially when we have many effects.
I can imagine this as syntax sugar on top of Capreso's capabilities. Could this be semantically different from capabilities (?)—need to think. (Find an illustrative example.)

Using annotation macroses:  currently, they are useless because changes from annotated types are local to type and can not be propagated to the other part of the program.   (I.e., it is impossible to change how types are visible from the outside (i.e., other parts of the program) via annotation macros)

Interesting. I will think about this. A helpful step might be to write a few simple examples in such a hypothetical style and compare them with other approaches.

also, after some thinking, I can't understand the example.
I.e.
// even the regular members of the effectful type wouldn’t be accessible in a pure member
pure def c = Fibers.init(1).map(_ + 1) // compilation failure

If the result type of Fibers.init is.  <[Int, Fiber] then a map is totally fine. (pure function over monad]).
The compilation failure should be:
pure def c = Fibers.init(1) + 1   //. real compilation failure.

Upd.5. (will switch to many-comment mode)
If we want implement this in macros (i.e.  after typing),  we have two visible problems (except need for extra Purifier) clause:

assigning to variables:

//inside effect-gathering fun:
val. x = F[doSomething]
val y = x+1
val z = x+2

Implicit conversions for satisfying typer will be inserted in y and z ,  so we will need a technique for reverting this (i.e. memorization of x)]
2. discarding values.   i.e.
   def fprints(s:String):  F[Unit]
// inside efffect-gatherign fun:

 fprings('Hello')

Macros can insert a call to effect here,   but a compiler with enabled '-Wnounit-statement -Wno-value-discard'  will produce warnings

hey @rssh! Nice to see you here :) Thank you for your work on dotty-cps-async, it has been an instrumental solution in Kyo's development!
Yes, Caprese and Kyo are solutions in the same domain and thus share major similarities. For example, Caprese's capture checking is equivalent to Kyo's pending effect set and both require a suspension mechanism. It's a bit difficult to form a complete picture of Caprese so I've based this opinion on this recent talk. The fundamental issues I see with Caprese's design are how prescriptive and restrictive it is while still not addressing the two main challenges it sets out to solve:
1. Effect Composability
Building an effect system where multiple effects can be composed in a safe and principled way is quite challenging. It's necessary to ensure handlers are pure and impure code doesn't introduce invalid control flow when effects are handled. Caprese's current design does not address that. Handlers are based on impure short circuiting via exceptions and mutable internal state. Examples from the talk:
// Impure short circuiting via exceptions
object boundary:
  final class Label[-T]
  def break[T](value: T)(using label: Label[T]): Nothing =
    throw Break(label, value)
  inline def apply[T](inline body: Label[T] ?=> T): T = ...
end boundary

// Handlers rely on mutable internal state
trait Produce[-T]:
  def produce(x: T): Unit

def generate[T](body: () => Unit) = new Generator[T]:
  def nextOption: Option[T] = step()
  var step: () => Option[T] = () =>
    boundary:
      given Produce[T] with  // handler
        def produce(x: T): Unit =
          suspend[Unit, Option[T]]: k =>
            step = () => k.resume(())
            Some(x)
      body
    None

This approach to effects does not compose. If both boundaries and generators are used in the same computation, the behavior seems undefined since Caprese doesn't even offer a way to handle the fact that the continuation itself might have other effects. The solution doesn't seem useful other than in non-nested and locally-scoped handlers. It also doesn't enable programming with values in the FP sense and relies on fragile control flow primitives instead. For comparison, here's the equivalent in Kyo:
// a pure and composable boundary/break effect in Kyo
class Boundary[V]() extends Effect[Boundary[V]]:
    opaque type Command[T] = Left[V, Nothing]

    def break(value: V): Nothing < Boundary[V] =
        suspend(this)(Left(value))

    def run[T: Flat, S](v: T < (Boundary[V] & S)): Either[V, T] < S =
        handle(handler, v)

    private val handler =
        new ResultHandler[Const[Left[V, Nothing]], [T] =>> Either[V, T], Boundary[V], Any]:
            def pure[T](v: T) = Right(v)

            def resume[T, U: Flat, S](
                command: Left[V, Nothing],
                k: T => U < (Boundary[V] & S)
            ) = command

// pure generators (borrowed from @kitlangton)
class Generators[Input: Tag, Output: Tag] extends Effect[Generators[Input, Output]]:
    override type Command[A] = Generators.Yield[Input, A]

object Generators:
    case class Yield[Input, Output](value: Input)

    def yield_[Input: Tag, Output: Tag](value: Input): Output < Generators[Input, Output] =
        suspend(Generators[Input, Output]())(Yield(value))

    def run[Input: Tag, Output: Tag, T: Flat, S](
        value: T < (Generators[Input, Output] & S)
    )(
        handler: Input => Output < S
    ): T < S =
        val h = new Handler[[T] =>> Yield[Input, T], Generators[Input, Output], Any]:
            def resume[T, U: Flat, S](
                command: Yield[Input, T],
                k: T => U < (Generators[Input, Output] & S)
            ) =
                handle(
                    handler(command.value)
                        .map(output => k(output.asInstanceOf[T]))
                        .asInstanceOf[U < (Generators[Input, Output] & S)]
                )
        handle(h, value)

Another important difference between Caprese and Kyo at the current state of the projects is that both these implementations are already fully functional in Kyo without the need for any language changes and in a purely functional manner.
2. Effect Suspension
Caprese's answer to how to introduce suspension is still unaddressed. Its design locks in to a significantly more limited form of suspension than Kyo's that can only be provided by an underlying runtime:
// Caprese's suspension API
class Suspension[-T, +R]:
  def resume(arg: T): R = ???

def suspend[T, R](body: Suspension[T, R] => R)(using label: Label[R]): T

Note how resume returns an R directly, which means the suspension itself can only happen by pausing the execution in the runtime itself. In Kyo, suspensions use the typical building block to perform suspensions: monads. I believe other languages like Unison and Koka adopt a similar approach and model the execution of algebraic effects on top of a monadic runtime.
Caprese's design seems to come from a misconception regarding the performance characteristics of monadic runtimes. Fine-grained effect suspensions like what Kyo and Caprese want to offer would have significantly worse performance using Loom's suspension for example. Although I'm a big fan of Loom and think it's one of the major improvements of the JVM in recent years, it seems only appropriate for coarse-grained effects like async, where its overhead is generally acceptable.
In contrast, Kyo's suspensions are purely functional:
def suspend[E <: Effect[E], T](e: E)(v: e.Command[T])(
    using _tag: Tag[E]
): T < E =
    new Root[e.Command, T, E]:
        def command = v
        def tag     = _tag

This characteristic enables a more powerful form of suspension. Handlers receive a pure function as the continuation, which provides a high degree of freedom. Essentially, it enables pure multi-shot continuations.
In addition to the more flexible algebraic effect mechanism, Kyo's encoding of the pending type (<) has significantly better performance than traditional monadic encodings.
Conclusion
Caprese's current work seems to prioritize defining APIs and abstractions over addressing the main challenges of the problem. While this approach may help iterate quickly and produce papers, it seems to be leading to major limitations. The work on this ticker is intended to provide an alternative that works seamless with Kyo and other effect systems rather than baking in an impure effect system in the language, which can't be reused by them. Addressing first-class support for purity is also a necessary step to make the transformation from direct to monadic in a principled way.

If we want implement this in macros (i.e. after typing), we have two visible problems (except need for extra Purifier) clause:

I was thinking about your suggestions and I’m afraid it’d be too fragile to do that via macros and implicit conversions. The usability wouldn’t be great since it wouldn’t be able to prevent mistakes and macros can’t express the effectful type mechanism properly. Maybe we should aim for a plugin directly.
The purity check would also be necessary. I’ve taught newcomers to use Kyo and one of the first things they normally do is mix impure code. The direct syntax seems to encourage that given the similarities with other languages.

A purity check can be implemented in macros (we have all trees there,  so it's relatively easy).
Also, I forgot about the third problem, which is inferring function type via the last expression in the block.
The last expression can be monadic or non-monadic, therefore if the return type of the `pure/effect-gathering' method is not set, it will be inferred differently for these two cases,  and we can do nothing with this,  because all types of macroses started
to work after typing.
So yes,  implementing on top of implicit conversion is more fighting with existing typing than developing.
Thinking about finding another way of typing F[X] as X  other than implicit conversions without language change
(something like.   Ext[F[_]]  ?=>  X ). -- Now I don't think this is possible,  but ... need to think more.

If we want to escape implicit conversion,  we should change the primary representation of effect total values to be compatible by assignment with its result. (Also, the idea to have a type defined differently in different compile-time environments,  but it look like this required a huge amount of boilerplate)
Two possible variants of typing should be used instead of IOs[A]. I see now:
A)  'IOs.Context ?=> A'
B). A^ { IOs.Capability }.   [In this case ,^ is capreso effect tracking annotation. All 'normal' values in 'pure' functions actually will have such type].

If we want to escape implicit conversion

I think we'd need to change the type inference of effectful types via a research plugin for the proposed solution to have a reasonable usability. I wonder if it's really very complicated since pure is similar syntactic sugar transformation. Maybe having a phase that handles the pure modifier and performs the direct to monadic transformation as syntactic sugar would be enough.

Two possible variants of typing should be used instead of IOs[A]. I see now:
A) 'IOs.Context ?=> A'
B). A^ { IOs.Capability }. [In this case ,^ is capreso effect tracking annotation. All 'normal' values in 'pure' functions actually will have such type].

Both options do not seem viable. The fundamental impedance mismatch between Kyo and Caprese/Capabilities is that Kyo uses suspended computations as pure values. A type like Int < Fibers can internally be either a pure value Int or a suspended computation with the Fibers effect pending. The equivalent in Caprese would be Int ^ { Fibers.capability }, which can not be anything other than Int afaics. The effect tracking in Caprese seems meant to be compatible only with control-flow based handlers, not suspended computations via monads.

A type like Int < Fibers can internally be either a pure value Int or a suspended computation with the Fibers effect pending. > The equivalent in Caprese would be Int ^ { Fibers.capability }, which can not be anything other than Int afaics.
For the programmer.
It is possible to write a standalone plugin that internally transforms such values into monadic form.

I.e. if we have no impilicts,  exists 3 options:

phase before types.  (problem -- the transformation of high-order functions uses typing.  Maybe it is possible to streamline this or isolate and move to a later stage. )
Change types. It's unclear if this is possible without substituting unmodified typers in phase. And is it possible to find minimal changes to the type system that can be clearly described?
Find a way in type system to represent monadic values in a form, compatible with direct values and then transform such values after  typing and capture checking.

@rssh thank you for following up on this! I was taking a look at capabilities and, besides the issue with it not being designed to represent monadic effects, there are some important usage patterns of the pending effect set in Kyo that might not be possible to express with capabilities:
1. Restricting Effects
Some methods in Kyo require restricting the effects that are allowed to be present in a computation. This is an essential feature to have proper support for fibers for example since forking with effects requires special handling. The Fibers.init method uses an implicit evidence to enforce that:
object Fibers {
    def init[T, S](v: => T < (Fibers & S))(
        using
        @implicitNotFound(
            "Fibers.init only accepts Fibers and IOs-based effects. Found: ${S}"
        ) ev: S => IOs,
        f: Flat[T < (Fibers & S)]
    ): Fiber[T] < (IOs & S) = ...
}

When this method is called with effects that aren't Fibers or IOs (or based on them), the compilation fails. For instance, without this mechanism a Vars[Int] effect could be "propagated" to a fiber, which wouldn't make sense since it's not possible to safely update mutable state across different threads.
Something remarkable about this approach is how it offers a mechanism similar to Rust's borrow checking to ensure safe use of mutable state but without the significant usability issues that comes with it.
2. Aliased Effect Sets
Since Kyo's pending type leverages the type system itself, it's easy to refactor sets of effects using regular type aliases:
type Databases = Envs[Database] & Aborts[DBException] & IOs

def countPeople: Int < Databases = Envs[Database].use(_.countPeople)

This approach becomes even more powerful when paired with opaque types. For example, the Resources effect is internally 100% based on IOs but Kyo ensures users have to handle the effect by using an opaque type:
opaque type Resources <: IOs = IOs

val a: Int < (Databases & Resources) = Resources.acquire(new Database).map(_.countPeople)

val b: Int < (Databases & IOs) = Resources.run(a)

Note how Resources also indicates that it's internally IOs via type bound, which enables calling Fibers.init with a Resources effect pending as well.
3. Parametrized Effects
Kyo offers effects with a type parameter like Envs[Database & Cache], which is equivalent to Envs[Database] & Envs[Cache]. This is crucial for usability since it aids in reducing the number of elements in the pending set and still allows Kyo to define methods that operate on a single environment value:
def t1(v: Int < Envs[Int & String]): Int < Envs[String] =
    Envs[Int].run(1)(v)
def t2(v: Int < (Envs[Int] & Envs[String])): Int < Envs[Int] =
    Envs[String].run("s")(v)

Conclusion
I'm not convinced that Capabilities are reusable in Kyo and I think it's likely they'll require major redesigning to achieve reasonable usability. If we build the plugin on top of it, we risk degrading user experience and maintaining the plugin can become quite challenging as Capabilities will likely still have major changes.
That said, capture checking would still be valuable in Kyo since it'd enable a safer Resources effect for example. I think it'd make sense to explore if we could apply it to specific scenarios but I don't envision eventually migrating to Capabilities since Kyo's approach using type-level set serves library's needs quite well.

phase before types. (problem -- the transformation of high-order functions uses typing. Maybe it is possible to streamline this or isolate and move to a later stage. )
Change types. It's unclear if this is possible without substituting unmodified typers in phase. And is it possible to find minimal changes to the type system that can be clearly described?
Find a way in type system to represent monadic values in a form, compatible with direct values and then transform such values after typing and capture checking.
(can we find more ways to choose from ?)

I don't know the compiler's internals but wouldn't it be possible to treat pure as syntactic sugar in the Desugar phase similarly to for-comprehensions? If we output the tree with the equivalent monadic composition, it'd fail in case the effectful types are not used as their "result" type in a pure member.

I don't know the compiler's internals but wouldn't it be possible to treat pure as syntactic sugar in the Desugar phase similarly to for-comprehensions?

The question - how we would know without typer,  that we extract something from monad or we use plain values?
This will mean  that we should have a syntax-only way to describe monadic computations.  I.e. in next block of code:
[*] def  askValue1():  Int < IOs

[*] def askValue2(): Int.  // fake

[*] def increment1(x: Int ): Int = {
     askValue1() + 1
}

[*] def increment2(x: Int ): Int = {
     askValue2() + 1
}

Is impossible,  because increment1 and ìncement2` should be syntaxically different.
Of course,  we can reuse <-  and. write  a <- b when we want extract value.  It can be a cool addition in 'scala' style for for-comprehartion (related topic on scala-contributirs:   https://contributors.scala-lang.org/t/pre-sip-for-with-control-flow-an-alternative-to-scala-async-scala-continuations-and-scala-virtualized/4423 ,  https://contributors.scala-lang.org/t/from-freeing-leftarrow-to-better-for-and-unification-of-direct-and-monadic-style/5922 ),  but is this solve the goal of make monadic computation like non-monadic ?  They should be syntaxically differ.   I.e. something like:
[*] def  askValue1():  Int < IOs

[*] def askValue2(): Int.  // fake

[*] def increment1(x: Int ): Int = {
     val x <- askValue1()
     x + 1
}

[*] def increment2(x: Int ): Int = {
     askValue2() + 1
}

And if we have syntax for extracting value from some monads,  we don't need a { 'pure' , 'effect-gathering' } markers for functions:   if we use <- inside function,  then one is effect-gathering.  In principle we can generate two versions of. increment1 -- monadic and non-monadic.
Syntax-only way is cool ;).   But how is it suitable for kyo goals is unclear.

Yet one way - think that in effect-gathering function any value is wrapped in a monad (and use = instead '<-' ).  It's  a way of 'scala-virtualized' .   Potential problem is performance of extracting all possible values from monad.

good point! I've been discussing this proposal with other people and a common concern is the lack of an await call to make it clear that an effectful type is in use. I wanted to avoid the syntax noise of the await calls but I don't think having it would be too problematic since other languages follow the same approach. Maybe we cloud explore making both pure and await keywords that trigger the syntactic sugar? I'd avoid <- since it seems less straightforward to understand than await.
It seems with this approach we wouldn't need @effectful as well but we'd still need a way to ensure all effectful types in a computation are wrapped into an await call.

Potential problem is performance of extracting all possible non-constant values from monad.

I'm not sure what you mean by this but I don't foresee any performance issues with the translation. It'd be Kyo's regular monadic composition under the hood, which is efficiently executed.

I'm not sure what you mean by this but I don't foresee any performance issues with the translation. It'd be Kyo's regular monadic composition under the hood, which is efficiently executed.

[*]. def  calc3(x:Int, y:Int, z:Int): Int = (x+y+z)*otherFun(x,y)

translation:
  def cac3[F[_]](x:F[Int], y:F[Int], z:F[Int]):F[Int] = {
    val x1 = await(x)
    val y1 = await(y)
    val z1 = await(z)
    z1 + y1 + z1 + await(otherFun(x1,y1))
 }

Ohh,  in first comment (with askValue primer) I say, that some syntax form of monadic wrapping is needed.
In the second (to which you replied), I realized we could assume that any value is wrapped in a monad.
Therefore, we can have some function annotation with semantics which you name 'pure'   (pure/effect-gathering... actually need to find the best word).  Sorry for the mess.   It was two different encodings.   In the first comment -- with syntax sugar for await/reflect/uplift, in the second --  assuming that all values are wrapped in await/reflect/uplift. (and await, of course, mapped to map).
Now, I am thinking about the third encoding: we can assume that values are pure.  (simular to your translation),  so we have:
pure def otherFun(x: Int, y: Int): Int = ...

pure def calc3(x:Int, y:Int, z:Int): Int =
    (x+y+z) * otherFun(x,y)

// translation. (if we  have no used Fiber inside)
def calc3_withEffects[F[_]](x: Int, y: Int, z: Int): F[Int] =
    otherFun_withEffects[F](x, y).map(v => (x+y+z) + v)

If we use Fibers:
pure def c(x:Int): Int < Fibers = Fibers.init(x).get + 1

// translation
def c_wihtEffect[F[_]:HasFibers](x:Int):F[Int] =
    // await(Fibers.init(x)).get+1
    Fibers.init(x).map(_.get + 1)

Type parameter is needed for case,  when we want to call f form Int < Fibers < Either.
I will think,  is it possible without.....  maybe if all 'widening'. are put on the caller side, I'm not sure that this is possible without caveats of implicit conversions or syntax noise.

I think I understand now. Kyo is different from other effect systems because you don't need to lift pure values into the monad since any type T is also a T < Any (no pending effects). Other effect systems would indeed need to provide a way to lift pure values to perform the transformation. Maybe we could assume that their companion object has to provide an apply or pure method? I'd like to avoid having to provide implicits and have a mechanism similar to for-comprehensions that only requires source compatibility.

Hmm,  if we generate a copy of function,  then we can do this after typing

About  variation of (1). [before typing].
Example with sum will be more verbose than I have wrote:
pure def calc3(x:Int, y:Int, z:Int): Int < Fibers =
    (x+y+z) * otherFun(x,y)

def calc3_ef[F[_]](x: Int, y: Int, z: Int): F[Int] =
       ((plus_ef(x,y).flatMap( xy => plus_ef(xy,z)).flatMap(xyz => multiple_ef[F](xyz, otherFun(x,y))
   )

because before typing we don't know -- is '+' return us monadic value or not.

So,  among the options to most simple solutions that are very close to the original idea:
Variant A:
4:
- use standard plugin and represent F[_] as it now.
- use underlying implicit conversions for plumbing such functions be compilable (maybe insert imorts in the body before typing).
- Copy function and do the usual recursive top-down transformation for cps and erasing implicit conversions.
(i.e. in
  val. x = F[doSomething]
  val y = x+1
  val z = x+2

compiler after typing will generate
  val x: F[Int] = F[doSomething]
  val y:Int = convert[F[Int],Int](x)+1
  val z:Int = convert[F[Int],Int](x)+2

but we will fully remap it to normal 'virtual' representation in the near-generated method:
  val x:Int = await(F[doSomething])
  val y = (x)+1
  val z = (x)+2

which eventually become
  F[doSomething].map{ x1 =>
    val x = x1
    val y = (x)+1
    val z = (x)+2

  }

And old method will make compiletime-only.

/bounty $2000

Bounty requirements:

A new compiler plugin implementation from scratch.
The pure modifier as described with the validations to deny the use of impure language constructs.
Effectful types with custom inference behavior within pure members.
No major performance regression with the transformations.

I kinda wanna try this 😎
/attempt #211

@kitlangton I'm redistributing Kyo's bounties. Have you been able to work on this? If not, can you cancel the attempt? Thank you!


Does not work on Debian Jessie

I installed the most recent version of html2pug from npm (1.0.0) and used NodeJS v6.11.1.
When running html2pug -f <file>.html the following error is being thrown
module.exports = async (sourceHtml, opts = {}) => {                        ^ SyntaxError: Unexpected token (     at createScript (vm.js:56:10)     at Object.runInThisContext (vm.js:97:10)     at Module._compile (module.js:542:28)     at Object.Module._extensions..js (module.js:579:10)     at Module.load (module.js:487:32)     at tryModuleLoad (module.js:446:12)     at Function.Module._load (module.js:438:3)     at Module.require (module.js:497:17)     at require (internal/module.js:20:19)     at Object.<anonymous> (/usr/lib/node_modules/html2pug/src/cli.js:6:18)
This error can also be reproduced on Windows 10 using NodeJs v6.10.3 and npm 3.10.10

@phenomax This is not platform specific. Node 6.x doesn't support async functions. Please update to at least Node 7.6 to use this library.
Highly recommend nvm for switching between node versions.


Introduce "InvokedInCurrentScopeAttribute" for static analysis

Summary of the new feature/enhancement
When a command has a ScriptBlock parameter, it can be difficult for static analysis tools like PSScriptAnalyzer to determine how the ScriptBlock will be invoked.  This makes tracking some things significantly more difficult (like if an assigned variable is utilized). For example, although ForEach-Object is invoked in the current scope, it's not possible for PSSA to determine variable usage in this example:
$a = @()
0..10 | ForEach-Object { $a += $PSItem }
return $a

Proposed technical implementation details (optional)
Introduce an attribute InvokedInCurrentScopeAttribute that can be used to tell static analysis tools what scope the command plans to invoke the parameter.
using System;

namespace System.Management.Automation
{
    [AttributeUsage(AttributeTargets.Property | AttributeTargets.Parameter)]
    public sealed class InvokedInCurrentScopeAttribute : Attribute
    {
        public string WhenPresent { get; set; }
    }

    public partial sealed class ForEachObjectCommand
    {
        [InvokedInCurrentScope]
        public ScriptBlock Begin { get; set; }

        [InvokedInCurrentScope]
        public ScriptBlock[] Process { get; set; }

        [InvokedInCurrentScope]
        public ScriptBlock End { get; set; }

        [InvokedInCurrentScope]
        public ScriptBlock[] RemainingScripts { get; set; }
    }

    public partial sealed class WhereObjectCommand
    {
        [InvokedInCurrentScope]
        public ScriptBlock FilterScript { get; set; }
    }

    public partial class InvokeCommandCommand
    {
        [InvokedInCurrentScope(WhenPresent = nameof(NoNewScope))]
        public override ScriptBlock ScriptBlock { get; set; }
    }
}

Tooling support for this should be implemented similarly to how Roslyn handles nullability decorations.  Mainly, looking for this decoration should be done by type name instead of identity.  Binary modules targeting 5.1 should be able to define this attribute within their own assembly and still have it detected by analysis tools.

/cc @bergmeister Thoughts?

Perhaps we could add more for PSScriptAnalyzer in Engine/API.

Perhaps we could create a supplementary module of static analysis attributes, so that we can ship that module in-box in future versions of us, but something like PSScriptAnalyzer could just take a dependency on that module and have them available downlevel to ps5.1 as well.

Such attribute would definitely help but since I assume this issue was created due to the known issue of the PSUseDeclaredVarsMoreThanAssignments PSSA rule that has false positives when variable usage crosses multiple scriptblocks? Therefore an alternative would also be to relax the rule and assume invocation in current scope. Personally, I was never too bother about its false positives and find the rule itself still useful. The problem of PSSA not being aware of Pester syntax like e.g. a variable declared in BeforeAll would still remain with such attributes I suppose.

Would it? Both BeforeAll {} and It {} invoke their scripts in the same scope -- that of the parent Context {} or Describe {} block. As such, you could apply such an attribute to both It and the Before/After-type blocks and remove those false positives. There would still be some if you're declaring values outside a context/describe, but iirc that's generally considered poor form anyway?

Such attribute would definitely help but since I assume this issue was created due to the known issue of the PSUseDeclaredVarsMoreThanAssignments PSSA rule that has false positives when variable usage crosses multiple scriptblocks?

It's an easy example of something that already exists that could benefit, but not my motivation for creating the issue.
I was working on a sort of type inference overhaul.  Part of that was making the variable inference more accurate which involves some flow analysis.  I realized the same problem that plagues that rule would make any sort of flow analysis inaccurate whenever command with a scriptblock was used.

The problem of PSSA not being aware of Pester syntax like e.g. a variable declared in BeforeAll would still remain with such attributes I suppose.

I think you'd just put the attribute on BeforeAll no?  I guess it depends how the rule is set up, but if you're actually trying to check execution flow, marking BeforeAll with InvokedInCurrentScope and leaving the It blocks undecorated should give enough information.

Hmm, you're right about Pester's BeforeAll. I initially thought PSSA would need to know the order of execution but it actually seems that PSUseDeclaredVarsMoreThanAssignments doesn't care about the order, i.e. $foo $a; $a =42 makes it not trigger a warning, one could also argue this is actually a bug...
My main point is that if the problem we try to solve is reducing the false positives of this rule, we could relax it an just assume current scope, WDYT?

My main point is that if the problem we try to solve is reducing the false positives of this rule, we could relax it an just assume current scope, WDYT?

Like I said that rule was just a convenient example of something existing that could benefit, but not why I opened the issue.  I think a lot of potential tools could benefit from this sort of data.
That said, relaxing it to assume current scope would probably be sufficient for that rule specifically.  Personally I'd like to see it catch more like the order problem you refer to, but that would probably be a pretty significant undertaking (e.g. if you're in a loop, that order is still valid for accessing the same var).

If we look C# nullable annotations there are  many new attributes. For the issue I want to know - do we expect a set of new attributes for such annotations for static analysis? My concern is that it is public API and we need a good design before start.

Given that there is more than half a year of .Net 5 and therefore also PowerShell Previews, there should be enough time to test and iterate over the design. Now is actually the best time to experiment :)

Yeah I think there's a lot of room for more annotations to assist with static analysis.  Like @bergmeister  mentioned, there's no need to wait until we figure out what all of them will be.
At a high level the most important things to me are:

They are used purely for static analysis.  They have zero run time effect.
Tooling matches based on type name not identity.

To expand on #2, I expect to be able to target PowerShellStandard.Library 5.1 and still support a new static analysis attribute like this:
namespace System.Management.Automation
{
    [AttributeUsage(AttributeTargets.Property | AttributeTargets.Parameter)]
    internal sealed class InvokedInCurrentScopeAttribute : Attribute
    {
        public string WhenPresent { get; set; }
    }
}

namespace MyProject.Commands
{
    [Cmdlet(VerbsDiagnostic.Test, "MyCommand")]
    public InvokeMyCommandCommand : PSCmdlet
    {
        [Parameter]
        [InvokedInCurrentScope]
        public ScriptBlock Action { get; set; }
    }
}

Maybe PowerShell can be a little less strict than roslyn and ignore namespace as well.

there's no need to wait until we figure out what all of them will be.

The API is public and will be used in external repo - so we need to get approval from PowerShell Committee and security team. So it is better to start with good design so as not to waste their time.
I guess we need a base abstract class like "StaticAnalyzerAttribute".


Missing setup scripts?

I'm trying to create an sftp server. Trying to RUN /sbin/setup-sshd -c openssh throws
Sending build context to Docker daemon  12.8 kB
Step 1 : FROM alpine:3.4
 ---> 4e38e38c8ce0
....
Step 10 : RUN /sbin/setup-sshd -c openssh
 ---> Running in 5f09a706f481
/bin/sh: /sbin/setup-sshd: not found
The command '/bin/sh -c /sbin/setup-sshd -c openssh' returned a non-zero code: 127

There are no setup-* scripts in /sbin

Yea, we exclude a lot of unnecessary stuff from the base image. You can get these by installing the alpine-conf package if you really want them.


Indicate output dimensionality in AbstractInterpolation type

As a developer writing a package extension for DataInterpolations.jl (https://github.com/adrhill/SparseConnectivityTracer.jl/pull/178), it would be useful if all AbstractInterpolations indicated their output dimensionality, e.g.
abstract type AbstractInterpolation{T,N} end

or {N,T} instead of just {T}:
https://github.com/SciML/DataInterpolations.jl/blob/536cb0243038af495656ab11a219724acd56a293/src/DataInterpolations.jl#L5

That's probably fine.

Yeah, that can be done. Will work on it soon.


Request a video intro to key concepts in the Flutter framework

In our recent UX studies, we've observed a lot of initial difficulty for developers who came from other frameworks to realize UI programing in Flutter was reactive and UI changes should be done by changing the state of the UI instead of manipulating UI elements directly. Those developers could have benefited from a quick but upfront introduction to the key aspects of the framework, so they don't attempt to apply their existing UI programming knowledge inapplicable to Flutter.
Here is my request: can we turn A Tour of the Flutter Framework into a short video and feature it prominently on our website? It could pair nicely with the more marketing-oriented Introducing Flutter video.
Cc: @RedBrogdon, @galeyang, @jayoung-lee, @Sfshaza

Hmm. I'm not sure how short a video could be and still cover all of these topics:

Introduction
Hello World
Basic widgets
Using Material Components
Handling gestures
Changing widgets in response to input
Bringing it all together
Responding to widget lifecycle events
Keys
Global Keys

We do have videos in the pipeline for Stateful/Stateless/Inherited and Keys, which will go over some of this content.
If this is your primary concern:

programing in Flutter [is] reactive and UI changes should be done by changing the state of the UI instead of manipulating UI elements directly

We could totally create a short video on that. I've done two recent conference talks about how the widget/element/render trees relate to one another and the motivation behind the way things work. They could probably be distilled into something short and straightforward. Would that help here?

The research question is what is the minimum amount of upfront learning required to get an experienced developer started on Flutter and continue with on-demand learning?

We could totally create a short video on that. I've done two recent conference talks about how the widget/element/render trees relate to one another and the motivation behind the way things work. They could probably be distilled into something short and straightforward. Would that help here?

Yes, totally. It doesn't even need to touch the three trees, IMO. The primary goal of this video is to quickly point out paradigm shifts to developers not familiar with reactive UI frameworks but otherwise proficient in programming, so they can make an informed decision on when they can learn by trial and error and when they do need to pause and read stuff.
The research question, if I may frame it that way, is this: what is the minimum amount of upfront learning required to get an experienced developer started on Flutter and continue with on-demand learning?
The goal is to answer this question:

Issue moved to flutter/website #2009 via ZenHub


Crushed Algolia's free plan in 4 days

96% of the searches are empty. Need to find a new solution

https://github.com/wildpow/new-esc-gatsby/pull/126


Reccomended parameter settings for whole chromosome of genotype-density data

Hi,
I'd like to run iSAFE on a single chromosome (6), containing ~70K individuals, 35,863 SNPs covering 170Mb. I've been trying lots of different parameter settings of different window sizes, minimum region sizes etc, but can't seem to find an optimum set that will work.
What would you recommend to get the optimum results in this scenario?
Thanks,
Sam

Dear Sam,
Could you please share a few parameter set you have tested and the outputs?
Best,
Ali


[ENH] schema-less indexing collects files prefix that can be returned / filtered by query

Same but on the right base
Fixes #140
When running with use_schema = false, the file listed for a given subject must follow the pattern:
'^([a-zA-Z0-9]*)sub-'  subject_label  '.*_([a-zA-Z0-9]+){1}\..*$'

parse_filename will return an extra prefix field
  filename = '../sub-16/anat/swuasub-16_run-1_task-rest_bold.nii';
  output = bids.internal.parse_filename(filename);

  expected = struct( ...
                    'filename', 'swuasub-16_run-1_task-rest_bold.nii', ...
                    'suffix', 'bold', ...
                    'prefix', 'swua', ...
                    'ext', '.nii', ...
                    'entities', struct(...
                                       'swuasub', '16', ...     % <--------- this makes a weird entity, but querying for subject does not rely on it
                                       'run', '1', ...
                                       'task', 'rest'));

I am a bit worried about the weird shape that can now be taken by the "sub" entity. Not sure if that can cause issues.
It is possible to query for prefix
bids.query(BIDS, 'prefixes');

  data = bids.query(BIDS, 'data', ...
                    'sub', '01', ...
                    'run', '01', ...
                    'prefix', 'swua');

Merging this into the dev branch.
So schema less indexing of a bids dataset should now be able of picking up any prefix to the filename and to filter those files with query.


add L0Smooth Algorithm

I have added L0Smooth algorithm to src and added test for it. But I meet some problems when testing my work. I met a UndefVarError, especially, the function 'L0Smooth' was not defined. I don't know what's wrong with it.

But I meet some problems when testing my work. I met a UndefVarError, especially, the function 'L0Smooth' was not defined.

It works well here if you activate the package folder as the project:
$ pwd
/Users/jc/Documents/Julia/ImageSmooth.jl

$ julia --project=.
julia> using ImageSmooth

julia> L0Smooth
L0Smooth

The document used to have 2 @docs block for L0Smooth, but I met a warning which said duplicate docs found for 'L0Smooth' in @docs block. I'd like to know whether I could maintain 2 @docs block for the same function?

In this PR, we have done works as follow:

Create the SmoothAPI smooth and smooth!.
Add the image-smoothing algorithm L0Smooth， which is available for both Gray image and RGB image.
Add test files for utils, SmoothAPI and L0Smooth.
Complete the documents for ImageSmooth.jl.

The current performance of L0Smooth algorithm is  as follow:
julia> img_gray = testimage("cameraman");
julia> img_rgb = testimage("lena_color_512");

julia> fₛ = L0Smooth()
L0Smooth(0.02, 2.0, 100000.0)

julia> @btime smooth(img_gray, fₛ)
  418.836 ms (1712 allocations: 316.75 MiB)
julia> @btime smooth(img_rgb, fₛ)
  1.155 s (1934 allocations: 1.03 GiB)

Some problems are still remained:

The performance of L0Smooth can be optimized via excluding unnecessary memory allocations.
Use fdiff in JuliaImages/ImageBase.jl#11 to replace the initial version L0Smooth  implemented by using forwarddiff! and backdiff!. This can also cause a promotion of the performance.


 Currently it will show the last capture image from when live stream was running (including timestamps on top right corner) after a live stream has occurred. It will only update to 'last event' when a new event occurs after the live stream.
My preference would be that this reverts back to the 'last event detected' image i.e.. motion detection or doorbell button press.

hey. I have been checking this today and I have seen that whenever RTSP or P2P livestreaming finished, I start getting picture from previous motion triggered event, so livestreaming images will be gone.

fixed over here: https://github.com/fuatakgun/eufy_security/commit/19fe9d924358c2b2f0569085167521f202725afb


Next button arrow icon not visible until reached last slide

Hey,
I'm using the compile 'com.heinrichreimersoftware:material-intro:-SNAPSHOT' with skip button disabled. And when the material-intro starts, the arrow  icon of the next button is invisible. But if I go to the last slide and navigate backwards then the next button arrow  icon is visible.

[ ]

``


ObjCBool: Add boolValue property

Update uses of Bool to ObjCBool where appropriate

Found after testing Darwin compatibility where the ObjCBool type has a boolValue property

@phausler Does this look correct?

@swift-ci please test

What advantages/disadvantages does this have? The code looks less Swifty than before with the introduction of ObjCBool in the places that it has been used.

ObjCBool is defined in https://github.com/apple/swift/blob/master/stdlib/public/SDK/ObjectiveC/ObjectiveC.swift#L27
we should probably lift the implementation instead of aliasing Bool

I converted ObjCBool to use the struct version from the stdlib however I don't think its that useful.
Apart from having to update all of the code to use .boolValue I don't think it provides any benefit and is now burdened by using  @_fixed_layout and @_transparent which may not be allowed.
The origin commit just fixed up some tests so that they would work under Darwin compatibility, so the only possible value I can see with this change is that a library would possibly be binary compatible between sclf and native Foundation on macOS. I don't see any benefit for non-Darwin platforms.

I had another think about this. The boolValue is only required for the tests (to be compatible with native Darwin Foundation) not in the actual runtime or implementation files.
I think the original extension and property should be added just to a test file and not exported in the runtime. Its a rather meaningless type for  non-ObjC platforms, and ObjC platforms will use the real ObCBool provided by the core runtime.

it doesn't have much meaning once compiled but the one thing it does do is keep us honest for compatibility - so in the spirit of the project (to offer cross platform compatibility with the fundamentals offered by Foundation and cross platform interoperation) it seems to be the best approach even though it is named perhaps inappropriately (more a naming choice by the standard library) it statically enforces our goals to offer the same API surface area for things that can be represented as a cross platform interface.

@phausler I removed the_convertBoolToObjCBool and _convertObjCBoolToBool and rebased

@swift-ci please test and merge


Facebook messenger PPA?

Can we get a PPA for Linux? this will be a lot easier and a better solution for app updates on Linux.

Messenger for Desktop 2 will have one for sure.
--
Alexandru Rosianu Full-Stack Developer aluxian.com | startersquad.com On Sun, May 22, 2016 at 4:40 PM, gameslayer <<EMAIL_ADDRESS><EMAIL_ADDRESS>> wrote:
Can we get a PPA for Linux? this will be a lot easier and a better solution for app updates on Linux.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub [https://github.com/Aluxian/Facebook-Messenger-Desktop/issues/424]

Awesome! let me know when you make one I do want to add this to the store I am making for Linux
http://cdfrosty.wix.com/store

Messenger for Desktop 2

Woot Any more details about that?

It'll run on Electron, it'll have more features, it'll consume less battery. :)
--
Alexandru Rosianu Full-Stack Developer aluxian.com | startersquad.com On Mon, May 23, 2016 at 10:54 PM, Majki <<EMAIL_ADDRESS><EMAIL_ADDRESS>> wrote:
Messenger for Desktop 2
Woot Any more details about that?
—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub [https://github.com/Aluxian/Facebook-Messenger-Desktop/issues/424#issuecomment-221077201]

I know that it is "rude" to ask. But any e.t.a. for Messenger for Desktop 2?

When does facebook messenger 2 come out? I also have a question what exe process on messenger?

I think I might add a PPA if I have some time later. I need to work on the update system for V1.5.0


File to import not found or unreadable

When I run npm start, an error occur:
ERROR in ./~/css-loader?{"sourceMap":false,"importLoaders":1}!./~/postcss-loader                                                                                ?{"ident":"postcss"}!./~/sass-loader/lib/loader.js?{"sourceMap":false,"precision                                                                                ":8,"includePaths":[]}!./src/main.scss
Module build failed:
undefined
^
      File to import not found or unreadable: C:\Users\Admin\desktop\Jarethien-E                                                                                nglish Dictionary 2.0\node_modules\bootstrap\scss\mixins\_reset-text.scss.
Parent style sheet: C:/Users/Admin/desktop/Jarethien-English Dictionary 2.0/node                                                                                _modules/bootstrap/scss/_mixins.scss
      in C:\Users\Admin\desktop\Jarethien-English Dictionary 2.0\node_modules\bo                                                                                otstrap\scss\_mixins.scss (line 29, column 1)
 @ ./src/main.scss 4:14-187
 @ multi ./src/main.scss
webpack: Failed to compile.

I thought that this happens because my computer cannot read scss files, but even after I installed sass, the error is still not solved.

Could you post your OS, Node and NPM versions?
Also it seems that there is a space in your path, and some npm modules do not like that, please try running it from a folder without any space in its name and see if it solves the issue.

I run window 7 ultimate, npm 5.3.0, node 8.1.4. But possibly the error occurred because of the space. I will try it.

Well, it was the space in file that made the error. Thanks for pointing that out.


fix(wallet): assetsTracker never emits if no assets in total balance

Context
Write a test and fix it if there’s a bug
https://input-output-rnd.slack.com/archives/C01S2DD5B0D/p1669818151273419 - Connect to preview
Important Changes Introduced

assets tracker now emits when there aren't assets in the total balance
optimize emitting if the assets haven't changed

Commits are squashed.


hackage

Hi,
Any chance this library could go on hackage? It's a nice library, I use it to good effect and it fills a niche I can't find elsewhere.

Tony,
Glad you like the package. I plan to push this to hackage before the semester starts (within a month or so). Will this work? I've got a number of planned tools that will depend on it.
Andy

Works well for me, thanks.  Look forward to seeing the resultant toolset.


Changing distillation weights changes runtime

Describe the bug
It appears that imbalances in the distillation weights has a significant impact on performance. When I set them all equal to 1, it runs twice as fast as when I set lm_loss to 1 and the other two to 0.1.
To Reproduce

Run a med-to-small distillation with weights 1, 1, 1
Run med-to-small distillation with weights 1, 0.5, 0.5
Run a med-to-small distillation with weights 1, 0.1, 0.1

Expected behavior
I did not expect such great variation.
Proposed solution
No clue
Screenshots

Environment (please complete the following information):

GPUs: EleutherAI V100 cluster
Configs: The defaults in the distill-gpt-neox branch

Additional context
Add any other context about the problem here.

Abandoning distilling efforts


R0 'estate' bug

In R0, Christabell mentions 'estate'.
Asking about 'estate' leads to bold 'Carol'.
Asking about 'Carol' leads to Naomi talking about how she already met Carol -- but this is R0.

replaced estate with property in R0 quips
property is not a  key word

I think this may be the wrong solution... You cut off the path that leads to the player typing "ask about Carol", but I think there are multiple paths to it in R0. Instead,  the quip about Carol should be changed for R0. Naomi should never have heard of Carol yet.

I think it's a separate issue. I've walled off the mansion topic from R0 --
already enough intro material to cover there.
Carol is mentioned one time in the transcript in R0, under "lineage". There
is a separate quip for R0 to deal with Carol. I justed tested lineage and
the quip -- they seem to be working okay.
It's up the player whether to drill down on conversation in R0 or not -- it
is somewhat optional -- anything mentioned in R0 is gone over in greater
detail later.
I think we're okay

Jack

On Thu, Aug 2, 2018 at 7:41 AM, Ben Collins-Sussman <
<EMAIL_ADDRESS>wrote:

I think this may be the wrong solution... You cut off the path that leads
to the player typing "ask about Carol", but I think there are multiple
paths to it in R0. Instead, the quip about Carol should be changed for R0.
Naomi should never have heard of Carol yet.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/dhakajack/trfv/issues/31#issuecomment-409897965, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFD_SrNE158vpn3LPdL68d_BnnC45jvCks5uMuV6gaJpZM4Vriaw
.

I'm pretty sure Frances accidentally also ran into bolded word 'potence',
which led to 'carol', which led to Naomi saying she had already visited
Carol.  Is this not possible anymore?
On Thu, Aug 2, 2018 at 9:52 AM Jack Welch<EMAIL_ADDRESS>wrote:

I think it's a separate issue. I've walled off the mansion topic from R0 --
already enough intro material to cover there.
Carol is mentioned one time in the transcript in R0, under "lineage". There
is a separate quip for R0 to deal with Carol. I justed tested lineage and
the quip -- they seem to be working okay.
It's up the player whether to drill down on conversation in R0 or not -- it
is somewhat optional -- anything mentioned in R0 is gone over in greater
detail later.
I think we're okay

Jack

On Thu, Aug 2, 2018 at 7:41 AM, Ben Collins-Sussman <
<EMAIL_ADDRESS>wrote:

I think this may be the wrong solution... You cut off the path that leads
to the player typing "ask about Carol", but I think there are multiple
paths to it in R0. Instead, the quip about Carol should be changed for
R0.
Naomi should never have heard of Carol yet.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/dhakajack/trfv/issues/31#issuecomment-409897965,
or mute
the thread
<
https://github.com/notifications/unsubscribe-auth/AFD_SrNE158vpn3LPdL68d_BnnC45jvCks5uMuV6gaJpZM4Vriaw
.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dhakajack/trfv/issues/31#issuecomment-409954007, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ALo-ea3xkexQ0epo_Qr-Am7t9i4go21jks5uMxIzgaJpZM4Vriaw
.


Tabs and window controls out of window (inaccessible) when bookmark toolbar is enable in OneLine mode

I recently reset my Notebook and modified my Firefox with WaveFox since I used it on my PC already. Everything worked fine.
When I was back on my PC I updated WaveFox since my Browser used an older Version with the older userchrome strings and added the new ones (same settings as on my Notebook)
Now I have the problem that the adress bar gets so big that I cannot use my tabs anymore and the window controls are also out of window. However this occurs only when I have the bookmark toolbar enabled. I usually have it show up on new tabs only. I didn't had this issue on my Notebook but there were almost no bookmarks in the toolbar in.
Screenshots

OneLine mode without bookmark toolbar active

OneLine mode with bookmark toolbar active (new tab page)

Details
Browser / OS
Firefox Version: 109.0.1 (64-Bit)
OS: Windows 11 Pro 22H2
Relevant Settings

WaveFox Preference
Value

userChrome.OneLine.NavBarFirst.Enabled
true

userChrome.OneLine.TabBarFirst.Enabled
false

userChrome.ProtonTabs-Enabled
true

userChrome.RegularMenuIcons-Enabled
true

userChrome.Toolbar.Transparency.High.Enabled
false

userChrome.Toolbar.Transparency.Medium.Enabled
false

userChrome.Toolbar.Transparency.VeryHigh.Enabled
true

userChrome.Windows.SystemEffects.Enabled
true

Firefox Preference
Value

toolkit.legacyUserProfileCustomizations.stylesheets
true

gfx.webrender.all
true

browser.tabs.inTitlebar
2 (unmodified)

browser.toolbars.bookmarks.visibility
newtab

Indeed, when the bookmarks bar overflows, such a problem occurs. I fixed it, but it's not the final version. When I have free time, I will find a better way.

All is ready.

I'm still having this issue on WaveFox v1.6.126 on macOS

I made the necessary corrections in v1.6.127. It should be fine now.

Thanks, that's working great now!


Problem on Debian

Hi.
The scripts are using #!/bin/sh to run.
On Debian 10 the /bin/sh is linked to /bin/dash. This is a problem, because lines like this:
if [ "$1" == "-h" ] || [ "$1" == "--h" ] || [ "$1" == "/h" ] || [ $# -gt 0 ]; then
would not work on Dash.
Here is the error:

./list-active-accounts.sh: 24: [: unexpected operator

Could you please change #!/bin/sh to #!/bin/bash, so the scripts will work as expected?
Thanks.

Hello, I fixed this with commit 5f05452. I replaced the sh shebang with bash.


Prisma2 packages causes Yarn install devDependencies on --prod

I don't know which repo should this issue belong to, but apparently, prisma packages causes yarn --prod to install all devDependencies (not npm install --prod=true). Adding either @prisma/photon or nexus-prisma to dependencies cause the wrong yarn behavior, and it is fixed if I moved all prisma packages to devDependencies.
Please have a look into this Yarn issue I created as well.
https://github.com/yarnpkg/yarn/issues/7801
Could anyone please explain this problem?

Can you please share your package.json file here?

I am unable to reproduce this. I used yarn --prod and yarn didn't fetch any devdependencies for me. Maybe try clearing your yarn cache.

@pantharshit00 yep, it doesn't fetch all the devDependencies, it does fetch prisma2 though.

Yes, I can reproduce this. I think this is more related to yarn as npm doesn't fetch it and we have correctly listed it as a peer dependency in nexus-prisma(which lists @prisma/photon as peerDep) and @prisma/photon(which lists prisma2 as peerDep). I think it has to with how yarn resolves peerDependencies but I am not an expert on dependency resolution. I also tested pnpm and it also doesn't grab these packages so clear this is very yarn specific:

Right now as a workaround you can just remove it and use a global install or use it via npx. The size of prisma2 is currently big because we need to ship typescript.
To understand why we ship typescript here is our current generation process:

Call prisma2 generate (either from @prisma/photons postinstall hook or from the user directly)
prisma2 generate now calls the @prisma/photon generator located in @prisma/photon/generator-build/index.js. This generator in memory creates an index.ts file, including the individual Photon build of the project. Now typescript comes in to play: It turns the index.ts file into an index.js and index.d.ts file. We have it on our todo list to directly generate an index.js and index.d.ts file, without the use of TypeScript. That will happen fairly soon / hopefully in Q1. That will make the package size more managable.

We require prisma2 as a peer dep in the first place as @prisma/photon's hook needs prisma2 to run generate. But as soon as the generation is done, prisma2 is not needed anymore. That means when people have the build done, prisma2 is not needed anymore. So that is why it is not a direct dependency but a peerDependency.

I will wait on yarn maintainers to respond here for now

@pantharshit00 thank you for the response. Using prisma2 through npx or global installation is not solving the problem, a nexus-prisma must be in dependencies where it allows nexus to use model and crud. Leaving nexus-prisma makes prisma2 installed as well. We have no problem with moving photon to devDependencies as we generate photon before deployment anyway, so photon is actually not causing the problem.
According to types of dependencies from *yarn * docs, "having a peer dependency means that your package needs a dependency that is the same exact dependency as the person installing your package". This explains why sometimes I got complained sometimes when installing photon and prisma2 in different versions. E.g.
    "@prisma/photon": "^2.0.0-preview018",
    "prisma2": "^2.0.0-preview019",

Though, it missed the explaination for how yarn resolves peerDependencies unless that is all it does.

This explains why sometimes I got complained when installing photon and prisma2 in different versions. E.g.

Yarn doesn't generate that warning, we doesn. We mark as prisma2 dep as * so yarn won't complain.

If you remove prisma2 from devDependecies, it doesn't install it at all in prod mode:

So my workaround here is to remove prisma2 from package.json and use global cli.

So my workaround here is to remove prisma2 from package.json and use global cli.

@pantharshit00 yep, that doesn't install prisma2 in node_modules anymore, but still add @types and @prisma/photon, even though we have custom output path for photon. The final size of node_modules excluding prisma2 is still 90M in this case, which is still way bigger than npm --prod=true 40M.
I hope yarn would have more explanations on how peerDependencies is resolved on their docs.

Yes, same, I already explained why photon is huge right now and we will take actions to fix it but for now maybe stick with npm.

Yarn closed the issue:

Closing as fixed in v2 where yarn workspaces focus --production only keeps dependencies

Next step: verify that this works now

Fellas. I gave up.
I can confirm this is still a problem as of<EMAIL_ADDRESS>but I gave up trying to try it on Yarn 2. Since Yarn 1 is in maintenance mode, I don't see this getting fixed.
Workarounds:

Use NPM
Try yarn workspaces focus --production and report back if it works.

Closing for now. Time for a drink.


'tensorflow.python.framework.ops.EagerTensor' object has no attribute 'mean'

Hi, guys hope everyone is safe and healthy! I'm trying to get SHAP values from a Keras model where after loading the model and weights, I import the CSV data using pandas then got a slice as follows;
background = df_full.iloc[np.array(df_full.index.array)[np.random.choice(df_full.shape[0], 10, replace=False)]]
background = background.reset_index(drop=True)
explainer = shap.DeepExplainer(model,tf.constant(background.values, dtype=np.float32))
X  = df_full.iloc[np.array(df_full.index.array)[np.random.choice(df_full.shape[0],                                                                   10, replace=False)]]
shap_values = explainer.shap_values(tf.constant(X.values, dtype=np.float32))
But I get an error from shap/explainers/deep/deep_tf.py L308 saying as follows.
'tensorflow.python.framework.ops.EagerTensor' object has no attribute 'mean'
In deep_tf.py, I found the corresponding line which was;
phis[l][j] = (sample_phis[l][bg_data[l].shape[0]:] * (X[l][j] - bg_data[l])).mean(0)
where this .mean(0) was problematic so I changed it with the following;
phis[l][j] = np.mean(sample_phis[l][bg_data[l].shape[0]:] * (X[l][j] - bg_data[l]), axis=0)
Now it's working for me but I wanted to ask if I'm doing any mistake here, looks like fine and results are as expected. Would be appreciated if you can shine some light on the matter.
Cheers

I changed shap/explainers/deep/deep_tf.py L308 as @jackaraz suggested but the same error occurs. Any help is appreciated！

hello, will a fix be made to this problem in the latest updates of shap? I am using shap==0.37.0 with tensorflow==2.3.1. The same error mentioned at the beginning of the thread is still occurring.

Do any of you still have this issue? If you can please try it with the newest version of shape and tensorflow.

I also encountered this problem, have you solved it? @jackaraz @detrin

I had the same issue. Turns out that shap expects the input data to be an np.array, not a tensor.
So, in case anyone runs into the same problem, just make sure your data structure is correct (same as your model expects), but instead of using tensors, convert your data to simple numpy arrays.
I'm using:

shap: '0.46.0'
tensorflow: '2.18.0'


plugin-catalog: entity validation endpoint

Feature Suggestion
provide an endpoint from backstage that receives entity defintions and returns at the least validation errors, at most a diff of the catalog were those entities to be ingested (a la https://<backstage-host>/api/catalog/locations?dryRun=true
Possible Implementation

Context
one of the biggest inhibitors to backstage adoption / experimentation in our organization has been a lack of quick feedback validation of entities (e.g. catalog-info.yml files.) the initial recommendation for validating an entity was to register changes from a dev branch to a staging instance of backstage, though this has become less ideal due to related entities not existing on our staging instance and $text: references (i.e. for openapi specs) not being accessible from our staging environment.
we see that the "analyze" step of the "Register an existing component" flow hits https://<backstage-host>/api/catalog/locations?dryRun=true, which could potentially be hit from CI runs to inspect catalog-info.yml changes from CI runs (e.g. are there validation errors, what entities will be created/changed, etc.). however, this step requires a location that can be accessed from our backstage instance, e.g. a url to a branch on our VCS. ideally, developers could get feedback on an entity configuration from their local environments, without pushing code to a remote, allowing for quick iteration on creating new entities and exploring new backstage features for existing ones.
it could be nice to have an endpoint where we can send an entity configuration blob in the payload and receive back information similar to what /location?dryRun=true returns. (of course this would be a bit different as there wouldn't actually be a location being generated.) we would potentially be interested in wrapping a call to that endpoint with some custom yaml validation to match our own conventions. though if this endpoint could take into account any custom validators that could be great.
fwiw we haven't tried out https://github.com/RoadieHQ/backstage-entity-validator yet.

This actually exists already and was added in v1.1.0: https://backstage.service.us-west-2.aws.vendhq.com/api/catalog/validate-entity
You can POST a JSON object with the following shape:
{
  "entity": "<json representation of catalog entry",
  "location": "<location of the entity, can be stubbed out>"
}

It returns a 200 if the entity is valid or a 400 if there are issues

Having said that, it can do with some enhancement as it only shows you the first detected issue. If you fix that and another one shows up, you basically have to rinse and repeat than fixing everything in one go.
It also only supports one entity whereas a repo may commonly have multiple entities (system + components) so each entry has to be posted separately rather than checking all entities at ones.
Personally, I opted to use the RoadieHQ entity validator but also has the same issues of only showing the first detected issue, even if there are multiple. Your CI pipeline may not necessarily have network reachability of your Backstage instance, as ours doesn't in which case, a locally run validator is a better choice.
If you do opt to use the Roadie validator, I can recommend using it in conjunction with xargs to "autodiscover" catalog files or else it may discover other yaml files that will choke the tool: https://github.com/RoadieHQ/backstage-entity-validator/issues/28#issuecomment-1112096677

Ahhh interesting. I’ll take a look at both. Also going to take a look at the code behind the location register dry run endpoint. Very curious if that could be exposed.

@zhammer can we close this now?

sure - i may make another issue once we give the current validation endpoint a look!

Perfect! thanks!