Blu-ray.com estimator crashes when info not found

Expected behaviour:
Flexget shouldn't crash.
Actual behaviour:
Flexget crashes when certain info can't be found
Steps to reproduce:

Step 1: Make sure a movie is in a movie_list (or any input) that does not exist on blu-ray.com. (for example The Seamstress(2009)
Step 2: Use a task that tries get the release dat for the movie.

Config:
DownloadMoviesTest:
    discover:
      interval: 3 hour
      what:
        - movie_list:
            list_name: "Testing"
      from:
        - piratebay: yes

Log:
2016-08-18 11:51 DEBUG    est_released  DownloadMoviesTest The Seamstress (2009)
2016-08-18 11:51 DEBUG    est_movies_bluray DownloadMoviesTest Searching Blu-ray.com for release date of The Seamstress (2009)
2016-08-18 11:51 VERBOSE  api_bluray    DownloadMoviesTest Searching from blu-ray.com title=The Seamstress
2016-08-18 11:51 DEBUG    utils.requests DownloadMoviesTest Fetching http://m.blu-ray.com/quicksearch/search.php
2016-08-18 11:51 DEBUG    utils.requests DownloadMoviesTest Fetching http://m.blu-ray.com/countries.json.php
2016-08-18 11:51 DEBUG    utils.requests DownloadMoviesTest Fetching http://m.blu-ray.com/movies/The-Seamstress-Blu-ray/6012/
2016-08-18 11:51 CRITICAL task          DownloadMoviesTest BUG: Unhandled error in plugin discover: 'NoneType' object has no attribute 'text'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flexget/task.py", line 472, in __run_plugin
    return method(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/flexget/event.py", line 23, in __call__
    return self.func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/input/discover.py", line 289, in on_task_input
    entries = self.estimated(entries, estimation_mode)
  File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/input/discover.py", line 201, in estimated
    est_date = estimator.estimate(entry)
  File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/estimators/est_released.py", line 28, in estimate
    estimate = estimator(entry)
  File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/estimators/est_released_movies_bluray.py", line 31, in estimate
    movie = lookup(title=movie_name, year=movie_year, session=session)
  File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/internal/api_bluray.py", line 209, in lookup
    movie = BlurayMovie(title=title, year=year)
  File "<string>", line 4, in __init__
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/state.py", line 306, in _initialize_instance
    manager.dispatch.init_failure(self, args, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/state.py", line 303, in _initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/internal/api_bluray.py", line 121, in __init__
    self.rating = float(movie_info.find('div', id='ratingscore').text.strip())
AttributeError: 'NoneType' object has no attribute 'text'

Additional information:

Flexget Version: 2.3.0
Python Version: 2.7.12
Installation method: pip
OS and version: Ubuntu server 14.04
Link to crash log: http://pastebin.com/1bEAkpj6

fixed in fb6c2c3

I just encountered this crash
2016-08-24 23:24 DEBUG    est_released  get-movies      A Troll in Central Park (1994)                                                                                2016-08-24 23:24 DEBUG    est_movies_bluray get-movies      Searching Blu-ray.com for release date of A Troll in Central Park (1994)                                                             2016-08-24 23:24 VERBOSE  api_bluray    get-movies      Searching from blu-ray.com title=A Troll in Central Park                                                                                2016-08-24 23:24 DEBUG    utils.requests get-movies      Fetching http://m.blu-ray.com/quicksearch/search.php                                                                                2016-08-24 23:24 DEBUG    utils.requests get-movies      Fetching http://m.blu-ray.com/countries.json.php                                                                                2016-08-24 23:24 DEBUG    utils.requests get-movies      Fetching http://m.blu-ray.com/movies/A-Troll-in-Central-Park-and-Thumbelina-Blu-ray/58752/                                              2016-08-24 23:24 CRITICAL task          get-movies      BUG: Unhandled error in plugin discover: 'NoneType' object has no attribute 'text'                                                       2016-08-24 23:24 CRITICAL manager       get-movies      An unexpected crash has occurred. Writing crash report to C:\Users\Jonathan\flexget\crash_report.2016.08                                 .24.232403370000.log. Please verify you are running the latest version of flexget by using "flexget -V" from CLI or by using version_checker plugin at http://fl                                 exget.com/wiki/Plugins/version_checker. You are currently using version 2.3.5                                                                                2016-08-24 23:24 DEBUG    manager       get-movies      Traceback:                                                                                Traceback (most recent call last):                                                                                File "c:\python27\lib\site-packages\flexget\task.py", line 472, in __run_plugin                                                                                return method(*args, **kwargs)                                                                                File "c:\python27\lib\site-packages\flexget\event.py", line 23, in __call__                                                                                return self.func(*args, **kwargs)                                                                                File "c:\python27\lib\site-packages\flexget\plugins\input\discover.py", line 289, in on_task_input                                                                                entries = self.estimated(entries, estimation_mode)                                                                                File "c:\python27\lib\site-packages\flexget\plugins\input\discover.py", line 201, in estimated                                                                                est_date = estimator.estimate(entry)                                                                                File "c:\python27\lib\site-packages\flexget\plugins\estimators\est_released.py", line 28, in estimate                                                                                estimate = estimator(entry)                                                                                File "c:\python27\lib\site-packages\flexget\plugins\estimators\est_released_movies_bluray.py", line 31, in estimate                                                                                movie = lookup(title=movie_name, year=movie_year, session=session)                                                                                File "c:\python27\lib\site-packages\flexget\plugins\internal\api_bluray.py", line 217, in lookup                                                                                movie = BlurayMovie(title=title, year=year)                                                                                File "<string>", line 4, in __init__                                                                                File "c:\python27\lib\site-packages\sqlalchemy\orm\state.py", line 306, in _initialize_instance                                                                                manager.dispatch.init_failure(self, args, kwargs)                                                                                File "c:\python27\lib\site-packages\sqlalchemy\util\langhelpers.py", line 60, in __exit__                                                                                compat.reraise(exc_type, exc_value, exc_tb)                                                                                File "c:\python27\lib\site-packages\sqlalchemy\orm\state.py", line 303, in _initialize_instance                                                                                return manager.original_init(*mixed[1:], **kwargs)                                                                                File "c:\python27\lib\site-packages\flexget\plugins\internal\api_bluray.py", line 135, in __init__                                                                                self.overview = contents.find('p').text.strip()                                                                                AttributeError: 'NoneType' object has no attribute 'text'                                                                                2016-08-24 23:24 WARNING  task          get-movies      Aborting task (plugin: discover)


Inconsistent results ？

Hello, thank you very much for contributing such a good job. I have some doubts when executing the code you provided, and I hope you can answer them. Thank you very much.

First, I changed the model_name in ce.json to ours, dataset to cifar10, and the result is as follows:

It seems to be somewhat different from the results mentioned in your paper. I don't quite understand it. I hope I can get your explanation
Secondly, there is CENet in your code, which doesn't seem to be mentioned in the paper. Is this a new attempt you made?
Finally, you mentioned the results of FedAvg in your paper and compared them with federated incremental learning methods, including your proposed method. In this case, how does it work? I don't seem to see this part in the code.
Thanks again!

I've responded your email. Thanks for your questions


res.on("searchEntry") in next.js after build does not return data.

I have an issue getting user data from AD using res.on("searchEntry"). I have described it more here:
https://stackoverflow.com/questions/78448023/res-onsearchentry-from-ldap-js-does-not-returns-result-in-next-js-production
It work in dev mode, but not after the build. After some research, I have placed a delay 20sec before resolving data from res.on("searchEntry") - in case it delays... but I have noticed That it not receiving anything. Replies only get on :
res.on("searchRequest", (searchRequest) => {
console.log("65", searchRequest.messageId);
});
and after on :
res.on("end", async (result) => {
//   console.log(result);
resolve({ userData }); - this I am setting exactly when I get res.on("searchEntry") result.
client.destroy();
});
res.on("searchEntry").- never get reply. BUT it gets when next.js is in dev mode.... How res.on("searchEntry") - work? what can cause, that we don't get a reply on this request from LDAP?  Kind of URL setup maybe?  I am testing on localhost.
Does res.on("end", async (result))  REUSLT contain result from searchEntry ?
Appreciate for help on this!

We do not support bundling of any sort.


compile Dr.Elephant for Hadoop 2.6.0 and Spark 1.6.0

Hi,
I'm trying to compile Dr.Elephant for Hadoop 2.6.0 and Spark 1.6.0, but getting compilation errors. Can you help me to fix the error?
[info] Run completed in 17 seconds, 160 milliseconds.
[info] Total number of tests run: 147
[info] Suites: completed 15, aborted 0
[info] Tests: succeeded 141, failed 6, canceled 0, ignored 0, pending 0
[info] *** 6 TESTS FAILED ***
[error] Failed: Total 463, Failed 6, Errors 0, Passed 452, Skipped 5
[error] Failed tests:
[error] 	com.linkedin.drelephant.util.SparkUtilsTest
[error] (test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 325 s, completed Nov 27, 2018 7:22:50 PM

cd target/universal
./compile.sh: line 152: cd: target/universal: No such file or directory
++ /bin/ls '*.zip'
/bin/ls: cannot access *.zip: No such file or directory
ZIP_NAME=
unzip
UnZip 6.00 of 20 April 2009, by Info-ZIP.  Maintained by C. Spieler.  Send
bug reports using http://www.info-zip.org/zip-bug.html; see README for details.

Usage: unzip [-Z] [-opts[modifiers]] file[.zip] [list] [-x xlist] [-d exdir]
Default action is to extract files in list, except those in xlist, to exdir;
file[.zip] may be a wildcard.  -Z => ZipInfo mode ("unzip -Z" for usage).
-p  extract files to pipe, no messages     -l  list files (short format)
-f  freshen existing files, create none    -t  test compressed archive data
-u  update files, create if necessary      -z  display archive comment only
-v  list verbosely/show version info       -T  timestamp archive to latest
-x  exclude files that follow (in xlist)   -d  extract files into exdir
modifiers:
-n  never overwrite existing files         -q  quiet mode (-qq => quieter)
-o  overwrite files WITHOUT prompting      -a  auto-convert any text files
-j  junk paths (do not make directories)   -aa treat ALL files as text
-U  use escapes for all non-ASCII Unicode  -UU ignore any Unicode fields
-C  match filenames case-insensitively     -L  make (some) names lowercase
-X  restore UID/GID info                   -V  retain VMS version numbers
-K  keep setuid/setgid/tacky permissions   -M  pipe through "more" pager
-O CHARSET  specify a character encoding for DOS, Windows and OS/2 archives
-I CHARSET  specify a character encoding for UNIX and other archives
See "unzip -hh" or unzip.txt for more help.  Examples:
unzip data1 -x joe   => extract all files except joe from zipfile data1.zip
unzip -p foo | more  => send contents of foo.zip via pipe into program more
unzip -fo foo ReadMe => quietly replace existing ReadMe if archive file newer

rm
rm: missing operand
Try 'rm --help' for more information.
DIST_NAME=
chmod +x /bin/dr-elephant
chmod: cannot access ‘/bin/dr-elephant’: No such file or directory
sed -i.bak '/declare -r app_classpath/s/.$/:hadoop classpath:${ELEPHANT_CONF_DIR}"/' /bin/dr-elephant
sed: can't read /bin/dr-elephant: No such file or directory
cp /root/dr/dr-elephant/scripts/start.sh /bin/
cp /root/dr/dr-elephant/scripts/stop.sh /bin/
cp -r /root/dr/dr-elephant/app-conf
cp: missing destination file operand after ‘/root/dr/dr-elephant/app-conf’
Try 'cp --help' for more information.
mkdir /scripts/
mkdir: cannot create directory ‘/scripts/’: File exists
cp -r /root/dr/dr-elephant/scripts/pso /scripts/
zip -r .zip

zip error: Nothing to do! (.zip)

mv .zip /root/dr/dr-elephant/dist/
mv: cannot stat ‘.zip’: No such file or directory

Hello,
You have to disable tests as described in https://github.com/linkedin/dr-elephant/issues/255
'try delete "test" command in line "play_command $OPTS clean test compile dist" of compile.sh, then you will pass compile without test.'
It worked for me.

thanks

Hi,
Although we were able to compile and deploy and run DrElephant 2.1.7 in our
environment we are still unable to view Spark 2.3 jobs in the UI. We also
see an error if we try to use SparkRestClient in FetcherConfig -Method not
found. Please advise what setup you use and if you are bale to see Spark
2.3 jobs.
PS: We currently use SparkFecther in FetcherConfig and are only able to
analyze Spark1.6.3 jobs
BR,
Lyubomir
On Fri, 30 Nov 2018 at 10:30, wzxwf<EMAIL_ADDRESS>wrote:

Closed #486 https://github.com/linkedin/dr-elephant/issues/486.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/linkedin/dr-elephant/issues/486#event-1997331508, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AGaxROs7ckDmSNngLNFARPAhkvLc2CSaks5u0OyhgaJpZM4Y1Y-E
.

I use spark2.2 to work normally.  We currently use SparkFecther in FetcherConfig and are able to
analyze Spark2.2.0 jobs


Mute and scroll disable

Added a fixed red square with a mute logo at its center. scrolling will be enabled when the mute button is clicked.

@Zahraa222 it doesn't seem to be working.

The mute button looks really good. We need the mute button to be separate of the scrolling disable. Just an empty button is good. Also, please extract the button to it's own component and import it in instead of having everything right there.
The scroll disable doesn't seem to be working. I'm still able to scroll despite having clicked on the button.


Disable "prompts" automatically for Continuous Integration systems

Check for Travis/Circle by CI=true - http://docs.travis-ci.com/user/ci-environment/#CI-environment-OS // https://circleci.com/docs/environment#env-vars

Fixed in https://github.com/ivankravets/platformio/commit/e0f2bcdeaffc2cd13028c181e251be145d7d90ce


releasefunds and orderconfirmation have insufficient gas for ETH orders

Attempting to accept an offline order with orderconfirmation or accept a dispute resolution with releasefunds returns an error of insufficient funds for gas * price + value when using ETH.
This appears to be true for orders between $0.01 and $1.00 of ETH. Larger amounts couldn't be tested due to the number as string bug (#1417).

Yeah people will need to have a positive eth balance to accept eth orders due to how eth works.

Maybe we need a custom error code for insuffient gas


Feature/remove rename

Adds remove, rmdir and rename to SFTPClient.
Adds rename to SFTPServer
Corrects the enum for readlink
Adds missing request id to several messages in the serialiser

Thanks for the PR, sorry for missing it. Feel free to tag me in next PRs.


Assertion error in iam_dataset.py while running 0_handwriting_ocr.py

Are you using the original iam_dataset.py?
This condition should only be checked if output_data is set to "bb" (bounding box). It seems like you moved the code outside of the check (if self._output_data == "bb":).


Population not shown for "rest of [state]" when not enough county-level data available

Actual Behavior:
When we show data for "rest of [state]" the population of that area is listed as unknown
When I...
Click on any state in the default Doctor Visits view, outside of an individual county

go to https://covidcast.cmu.edu/?sensor=doctor-visits-smoothed_adj_cli&level=county&date=20201108&signalType=value&encoding=color&mode=overview&region=38000
Click on North Dakota, anywhere but the four county borders

Expected behavior
I propose that we do the math to determine the population of the "rest of [state]" area by subtracting the population of the delineated counties from the total population for the state.
Screenshots

Rating Scale (1 is minor, 7 is severe):
1

Context

Browser: Chrome

Additional context

This one seems to be back:

which version are you referring to? latest dev? cause it is not part of v1.11.1

that's the live site

that's the live site

the live version (v1.11.1) was just a patch release without any new features. This feature is going to be released soon.


Add support for cleaning up router interfaces on HA neutron routers

For Openstack deployments with DVR/HA, there are additional router
interfaces which must be deleted befor the router can be deleted.
A similar fix was submitted to Openstack shade:
https://github.com/openstack-infra/shade/commit/e4fb68662753546f29ab054754d13bc94b661525

Thank you for contributing to openstack/ospurge!
openstack/ospurge uses Gerrit for code review.
If you have never contributed to OpenStack before make sure you have read the
getting started documentation:
http://docs.openstack.org/infra/manual/developers.html#getting-started
Otherwise please visit
http://docs.openstack.org/infra/manual/developers.html#development-workflow
and follow the instructions there to upload your change to Gerrit.


not working on localhost

Hi
I can access the google drive, but when I click on the file to open in the app, it does not load and in the terminal it says: { statusCode: 403, statusMessage: 'Unauthorized' }
what could it be?

hum... interesting... I guess there is a bug at https://github.com/Autodesk-Forge/model.derivative-nodejs-google.drive.viewer/blob/master/server/model.derivative.google.drive.integration.js#L65, it should name the bucket with a prefix (e.g. your client ID) to avoid duplicated names across apps...
cannot test right now, but this should do the trick:
var ossBucketKey =  config.credentials.client_id + (user.displayName.replace(/\W+/g, '') + user.id).toLowerCase();

@augustogoncalves Sorry but this issue is happening to the app I've deployed at Heroku as well. It might be related to google apis. I've created the credentials, I activated the google people api, google+ and drive api.

Idk what I am doing wrong. Do you think am I setting the google api wrong? Do you have a tuto how to set the api for this app?

just committed a change that fixes this issue, please take a look, @douglaspetrin

@augustogoncalves thanks! Now it's working.


updates peering type to AzurePublicPeering in public peering section

The update public peering section mentions MicrosoftPeering as Name and PeeringType which is not correct.
This should be AzurePublicPeering.

Hi @Stijnc, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!

This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. Real humans will now evaluate your PR.

TTYL, MSBOT;

@Stijnc : Thanks for your contribution to the Azure documentation! The author, @ganesr, has been notified to review your proposed change.

#sign-off


fix: Correct subplot positioning in plot_forecasts

Fixes #179
What does this implement/fix? Explain your changes.
Plotting loop was referring to all unique entities instead of the sampled entities to be plotted. Also, adjusted the subplot position calculation to correctly handle an arbitrary number of series and columns. This change ensures that the grid layout of subplots is dynamically adjusted to accommodate the specified number of series, thus avoiding IndexError and incorrect subplot arrangements.

This fixes the plotting module only partly, in the #181 it implements the fix for all three plotting functions having the same problem.


Invalid login

After docker-compose up with the mysql example, I run the following command as the readme says.
docker run --detach --name superset [options] amancevice/superset
docker exec -it superset superset-init

But I can't login with the admin user I created. I also tried postgres, the result is the same.
The I decide to attach to the console after docker-compose up: docker exec -it mysql_superset_1 bash, then  superset-init. And it worked! I can login with the user I just created.
I though it is related with user permission as the post says but I didn't Investigate it. I post it here for record and for whom with the same issue.

I think you have misread the READMEs. The main README is a general info for the project.
If you are attempting to bring up an example you should re-read the README in the examples page.
As a general rule, I wouldn't mix docker-compose and docker commands. What you are describing would act on completely different containers, which is why your login did not work.


Chore: Remove most * imports

Description
And pub use ...
It make it very difficult to know what's in scope and what's not. Particularly hellish when refactoring.

Should go in code style


With the new CUDA folder structure, CuBlas is not found

Expected Behavior
Octo-Tiger should compile as before
Actual Behavior
it seems that cublas_v2.h is not found, even if the header is in the path.
/pscratch/sd/d/diehlpk/OctoTigerBuildChain/src/octotiger-kokkos/src/unitiger/hydro_impl/hydro_cuda_interface.cpp:1:9: warning: #pragma once in main file
    1 | #pragma once
      |         ^~~~
In file included from /pscratch/sd/d/diehlpk/OctoTigerBuildChain/build/hpx/include/hpx/async_cuda/cublas_executor.hpp:24,
                 from /pscratch/sd/d/diehlpk/OctoTigerBuildChain/build/hpx/include/hpx/modules/async_cuda.hpp:19,
                 from /pscratch/sd/d/diehlpk/OctoTigerBuildChain/src/octotiger-kokkos/src/unitiger/hydro_impl/flux_cuda_kernel.cpp:13:
/pscratch/sd/d/diehlpk/OctoTigerBuildChain/build/hpx/include/hpx/async_cuda/custom_blas_api.hpp:39:14: fatal error: cublas_v2.h: No such file or directory
   39 |     #include <cublas_v2.h>
      |              ^~~~~~~~~~~~~
compilation terminated.
gmake[2]: *** [CMakeFiles/hydrolib.dir/build.make:132: CMakeFiles/hydrolib.dir/src/unitiger/hydro_impl/flux_cuda_kernel.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
In file included from /pscratch/sd/d/diehlpk/OctoTigerBuildChain/build/hpx/include/hpx/async_cuda/cublas_executor.hpp:24,
                 from /pscratch/sd/d/diehlpk/OctoTigerBuildChain/build/hpx/include/hpx/modules/async_cuda.hpp:19,
                 from /pscratch/sd/d/diehlpk/OctoTigerBuildChain/src/octotiger-kokkos/src/unitiger/hydro_impl/reconstruct_cuda_kernel.cpp:2:
/pscratch/sd/d/diehlpk/OctoTigerBuildChain/build/hpx/include/hpx/async_cuda/custom_blas_api.hpp:39:14: fatal error: cublas_v2.h: No such file or directory
   39 |     #include <cublas_v2.h>
      |              ^~~~~~~~~~~~~
compilation terminated.
gmake[2]: *** [CMakeFiles/hydrolib.dir/build.make:146: CMakeFiles/hydrolib.dir/src/unitiger/hydro_impl/reconstruct_cuda_kernel.cpp.o] Error 1
ptxas info    : 45 bytes gmem
gmake[2]: Leaving directory '/pscratch/sd/d/diehlpk/OctoTigerBuildChain/build/octotiger/build-kokkos'
gmake[1]: *** [CMakeFiles/Makefile2:176: CMakeFiles/hydrolib.dir/all] Error 2
gmake[1]: Leaving directory '/pscratch/sd/d/diehlpk/OctoTigerBuildChain/build/octotiger/build-kokkos'
gmake: *** [Makefile:136: all] Error 2

Steps to Reproduce the Problem
... Please be as specific as possible while describing how to reproduce your problem.

Build HPX with CUDA
Compile Octo-Tiger

Specifications
... Please describe your environment

HPX Version: Master
Platform (compiler, OS): gcc 10.3/11.2 and CUDA 11.4 and cmake 3.19 and 3.21

@diehlpk could you please check whether the HPX preprocessor constant HPX_HAVE_GPUBLAS is defined (should be in hpx/config/defines.hpp)?

@diehlpk could it be that you have to add the CUDA::cublas target to the octotiger executable target? HPX does not depend on cuBLAS, so it doesn't propagate any dependencies for it.

@hkaiser I think it does, because we include the cublas_v2 here
https://github.com/STEllAR-GROUP/hpx/blob/ae45453ace32e6f5da0af9b63e09cae0876e3ada/libs/core/async_cuda/include/hpx/async_cuda/custom_blas_api.hpp#L39

@hkaiser In addition, we include hipblas as well
https://github.com/STEllAR-GROUP/hpx/blob/ae45453ace32e6f5da0af9b63e09cae0876e3ada/libs/core/async_cuda/include/hpx/async_cuda/custom_blas_api.hpp#L14

HPX does not depend on cuBLAS, so it doesn't propagate any dependencies for it.

@hkaiser I would assume that HPX propagates the dependency if HPX includes the header.

HPX does not depend on cuBLAS, so it doesn't propagate any dependencies for it.

@hkaiser I would assume that HPX propagates the dependency if HPX includes the header.

You're right, my fault. I'll have another look.

cublas/hipblas is optional because hip installations do not necessarily include hipblas (https://github.com/STEllAR-GROUP/hpx/blob/581d3fb174e55eb1a8ba68d54dc9108e3cbac82b/cmake/HPX_SetupCUDA.cmake#L47-L53). @diehlpk can you try a newer version of CMake? CMake has been notoriously bad at finding various CUDA libraries, especially in the newer versions.

cublas/hipblas is optional because hip installations do not necessarily include hipblas (https://github.com/STEllAR-GROUP/hpx/blob/581d3fb174e55eb1a8ba68d54dc9108e3cbac82b/cmake/HPX_SetupCUDA.cmake#L47-L53). @diehlpk can you try a newer version of CMake? CMake has been notoriously bad at finding various CUDA libraries, especially in the newer versions.

@msimberg even with 3.21 it does not work.

@msimberg I think the issue is that the HPX CUDA scripts do find cuBLAS, but this is not propagated to dependent projects.


„Share Story“ Button doesn’t work, error: „ZeroClipboard is deprecated“

Hi,
we discovered odyssey.js last week and built a card with it.
Unfortunately, the „share story“ button doesn’t work for us, neither in Chrome nor in Firefox or Safari.
I checked the console and got following error messages when I clicked on the button:
`new ZeroClipboard(elements, options)` is deprecated. See docs for more info:
    https://github.com/zeroclipboard/zeroclipboard/blob/master/docs/instructions.md#deprecations
C @ ZeroClipboard.min.js:9
K @ ZeroClipboard.min.js:9
(anonymous) @ sandbox.js:202
(anonymous) @ d3.js:1047
d3.js:1921 POST https://api.github.com/gists 401 (Unauthorized)
xhr.send @ d3.js:1921
xhr.(anonymous function) @ d3.js:1907
(anonymous) @ sandbox.js:569
(anonymous) @ d3.js:1918
event @ d3.js:430
respond @ d3.js:1871
load (async)
d3_xhr @ d3.js:1859
(anonymous) @ d3.js:1853
Gist @ sandbox.js:555
(anonymous) @ sandbox.js:194
(anonymous) @ d3.js:1047
sandbox.js:570 Uncaught TypeError: Cannot read property 'responseText' of undefined
    at Object.<anonymous> (sandbox.js:570)
    at Object.event (d3.js:430)
    at XMLHttpRequest.respond (d3.js:1873)

+1 please help!

I'm afraid this is because of the POST https://api.github.com/gists 401 (Unauthorized), which is trying to post a gist to Carto's account from http://cartodb.github.io/odyssey.js/sandbox/sandbox.html. @javisantana maybe this is a simple fix?

Related: github removed anonymous gist creation in February 2018, which probably broke this.
Not sure how this can be fixed, since Carto will probably not authenticate to create sandbox users' gists. Maybe the best solution for now is to teach users on how to create their own gists and view them on bl.ocks.

@bplmp unfortunately, the problem for us is that we need to utilize iframe, which was apart of the sharing feature, which bl.ocks hasn't supported since 2016.
I haven't been able to get a similar solution from using rawgit.com either.

@erlenbusch then perhaps you could use github pages? I've used iframes from github pages in other sites previously and it worked ok.


adding a chatbot

Is your feature request related to a problem? Please describe.
no
Describe the solution you'd like
i want to add a chatbot to the website.
Describe alternatives you've considered
there are multiple ways to do this.
Additional context

that chatbot will be first with the files from this project and after which i will integrate the bot into the website.
@Suraj-kumar00

while creating a flashcard , they can venture deeper intor concepts and create more specific flash cards to that topic.

Okay let me see you how it's going to look like in the preview when you'll complete this, and then i'll see if this should be added in the project or not.
@T-Rahul-prabhu-38 it's yours!

Hi @T-Rahul-prabhu-38, what is the update on the issue?

sorry for the delay.


Integrate the new Digital Product Pass Model 4.0 + Battery Pass Model 5.0

Description
Based on the new models display the Digital Product Pass v4.0 +  the Battery Pass v5.0.
Acceptance Criteria

[ ] Remove all the retrocompatibility since models are being deprecated
[ ] Create test data for the models
[ ] Create a model compatibility matrix in the fronted documentation (create a frontend readme)

Lastest Status:

Battery Passport Model 5.0.0 Integrated

Flagged:

Waiting

INT environment is now working and changed can be integrated.
Transmission pass 2.0.0 is also included

Correction: the new versions are:

BatteryPass v6.0
DigitalproductPass v5.0
TransmissionPass v3.0

Integrated and closed


[Feature] Hadoop Ingestion with Orc Support

Hadoop ingestion from parquet file is already supported in Druid #2333.
I think ingestion directly from orc file is also beneficial
because many people use Hive in their data center and Hive has better performance with Orc than Parquet.
To do so, I'm plan to make InputRowParser work on OrcStruct, which is from OrcRecordReader of OrcNewInputFormat.
One issue in implementing InputRowParser is that
even though Orc file has schema information in itself, I cannot retrieve that schema information from OrcStruct which is the only given to me in the above approach.
As an easy first step, I think a way that InputRowParser explicitly receives StructTypeInfo as a parameter and get schema information from that.

@sirpkt are you planning on doing this as an extension?

@fjy Yes. contrib extension like Parquet support #2333.

closing, fixed by #3019

Is this done as an extension? Can you pls. more info on this? How to use this etc?
Thanks,
Uday.

@udayakumarp please read the documentation

Thanks fjy - Can you pls. point me to the documentation (in GIT you mean?)

Never mind! Got it - http://druid.io/docs/0.9.2-rc2/development/extensions-contrib/orc.html
Thanks,
Uday.


Add a read-only Gitea user to repos

Description
If Zarf is initialized with the gitops-service component we will now be creating a non-admin account that will be added as a read-only user to future repos pushed via Zarf.

The credentials are displayed when zarf init completes and are stored as a secret in the zarf namespace with the name git-pull.

Related Issue
Closes #183
Type of change

[ ] New feature (non-breaking change which adds functionality)

Checklist before merging

[ ] Tests have been added/updated as necessary (add the needs-tests label)
[ ] Documentation has been updated as necessary (add the needs-docs label)
[ ] An ADR has been written as necessary (add the needs-adr label) [ 1 2 3 ]
[ ] (Optional) Changes have been linted locally with golangci-lint. (NOTE: We haven't turned on lint checks in the pipeline yet so linting may be hard if it shows a lot of lint errors in places that weren't touched by changes. Thus, linting is optional right now.)

Looks like one of the injector binaries is showing up


Modernize buildstep for cedar-14 stack and Heroku buildpack requirements

Goal
Modernization of the builder architecture so that the build process more closely resembles the Heroku
workflow. This involves the following high level changes:

ubuntu-deboostrap:14.04 base image for a smaller surface area to begin with.
unmodified cedar-14 stack install script from Heroku. Ensures the installed packages match closely.
integrate the flynn buildback installer. Removes code duplication and allows versioning installed buildpacks
correctly identify the stack as cedar-14 so that the buildpacks know which binaries to install
switch to Heroku buildpacks for Go and PHP
Remove all preinstalled custom buildpacks. Lesser maintenance burden, fair for all custom buildpack authors

Actions
Removed packages
The following packages are no longer be installed on build:

libjpeg62, libpng12-0, libpng12-dev, libreadline5
By choosing the right binaries for the stack these packages are no longer required
libmagickcore-dev, postgresql-server-dev-9.3
These packages are implicitly installed due to dependencies of other packages
nodejs
The official heroku buildpacks install a node version if it is required
libssl0.9.8
Superseded by libssl1.0.0
mercurial
The official heroku buildpacks install mercurial as required
openjdk-7-jdk, openjdk-7-jre-headless
Will be automatically installed by the java buildpack, otherwise the multi buildpack should be used
postgresql-client, mysql-client
Open for discussion. In favor of an unmodified cedar-14 stack I removed them since the benefit of having them in the container is small with a database server that should be reachable from the admin workstation.
libsqlite3-dev, sqlite3
SQLite is not available on cedar-14 and not really useful on ephemeral filesystems
libmcrypt-dev
Removed in favor of an unmodified cedar-14 stack image. Could be installed by a custom buildpack if required.

Removed custom buildpacks
The following custom buildpacks are no longer preinstalled in the buildstep image:

Metorite - https://github.com/oortcloud/heroku-buildpack-meteorite.git
Perl - https://github.com/miyagawa/heroku-buildpack-perl.git
Dart - https://github.com/igrigorik/heroku-buildpack-dart.git
NGINX - https://github.com/rhy-jot/buildpack-nginx.git
Apache - https://github.com/Kloadut/heroku-buildpack-static-apache.git
Jekyll - https://github.com/bacongobbler/heroku-buildpack-jekyll.git

Validation
Validation of buildstep has been performed with all sample apps provided on the Getting Started on Heroku website.
The following deploys worked without modification:

Ruby - OK (Ruby 2.0.0, PostgreSQL)
PHP - OK (PHP 5.5.11, Apache 2.4.10)
node.js - OK (Node.JS 0.10.32)
Python - OK (Python 2.7.8)
Java - OK (OpenJDK 1.7, Jetty 7.6.0.v20120127)
Clojure - OK (OpenJDK 1.6, Leinigen 2.4.2, Jetty 7.x.y-SNAPSHOT)
Scala - OK (OpenJDK 1.7, SBT 0.13.5, Scala 2.10.4)

Known Quirks
Since buildstep runs as root within the container newer PHP Versions (after 5.5.11) don't work without a modified php-fpm.conf. Due to CVE-2014-0185 PHP changed the default permissions for the listening socket from 0666 to 0660. Since PHP-FPM and Apache both get run as root this results in the PHP-FPM process creating the listening socket belonging to root:root with mode 0660 and Apache running as daemon:daemon which can't read the PHP socket.
The workaround for dokku involves telling PHP-FPM to set the socket to 0666 mode:
Tell the startup script to include a config fragment for PHP-FPM:
Procfile
web: vendor/bin/heroku-php-apache2 -F conf/php-fpm.inc.conf web/

Add a conf folder with the following php-fpm.inc.conf to your application
php-fpm.inc.conf
listen.owner = nobody
listen.mode = 0666

With this workaround enable the example app has been validated using PHP 5.5.16 and PHP 5.6.0 as well.
Heroku doesn't have this problems as the daemons don't get started as root and run as the same unprivileged user.
Results
Closed issues
These changes should close the following issues:

#36 Allow to switch the buildpack branch (due to flynn buildpack installer)
#61 Ruby 1.8 app won't start on image generated with buildstep (bundler is no longer installed globally)
#66 Use $app_root and $build_root instead of hard coding /app and /build
#80 scala buildpack is broken due to config_vars and quoting
#82 HHVM support (switch to official PHP buildpack with HHVM)
#84 Cannot pull down buildpack for Ruby 2.0
#91 make build fails because rubygems are not installed
#93 PHP after switch to Trusty (PHP buildback defaulted to cedar stack binaries due to missing STACK env var)
#94 update-alternatives fails in prepare script (no longer gets called)

Closed pull requests
These changes should close the following pull requests:

#38 - Generate en_US.UTF-8 locale
#41 - Install language-pack-en-base
#71 - Removed old perl buildpack, using new one
#95 - locale utf8 issue solved
#97 - Use official php buildpack
#100 - Don't add config_vars from release to .profile.d
#103 - Copy and run prepare before builder to better leverage docker caching
#104 - Fix bundler cache that was not working.
#106 - Make sure libreadline5 is installed.
#107 - Hardcode stack name
#108 - Replaced CHH's buildpack with official one.

@yabawock sorry I know this has been merged for a while... but just running into an issue with /exec symlinked to /start. Was that intentional and why? Need a way to source all files and run a command and /exec is now gone (while /start is doing way too much).

I'm having a hard time understanding how it being symlinked is causing a
problem. Can you elaborate more on what you're doing?
On Tue, Jan 6, 2015 at 11:50 AM, Ken Ip<EMAIL_ADDRESS>wrote:

@yabawock https://github.com/yabawock sorry I know this has been merged
for a while... but just running into an issue with /exec symlinked to
/start. Was that intentional and why? Need a way to source all files and
run a command and /exec is now gone (while /start is doing way too much).
—
Reply to this email directly or view it on GitHub
https://github.com/progrium/buildstep/pull/109#issuecomment-68922446.

--
Jeff Lindsay
http://progrium.com

I'm using dokku-supervisord and it overrides /start to do something else. In there it calls the old /exec to source profile.d files and run Procfile commands. I guess this isn't buildstep's fault in changing how /exec works, but just wondering if there's any migrate path for /exec or plugins should write their own to source profile.d file. See https://github.com/statianzo/dokku-supervisord/issues/21#issuecomment-68925417.

I'll have to double check but I'm pretty sure the reason /exec symlinks to /start is because you can use /start like /exec, so ... ohhhhh, no I see. If they override /start then we can't expect it to behave like /exec. Gotcha.
Perhaps if dokku-supervisord is overriding /start, it should first copy it to /exec overriding the symlink so that it can call it as expected.
In the long term, and I'll try to make a post about this for plugin maintainers to read if it applies, there is currently a branch called herokuish (https://github.com/progrium/buildstep/tree/herokuish) that hands off everything to the herokuish tool, and buildstep just becomes a backwards compatible wrapper around it. In that branch, I believe both are symlinked.

Yeah, that's basically how plushu-cedarish works right now, except I was rolling my own script to do "herokuish build". I'll submit a few issues to incorporate the necessary features of plushu-cedarish's build script into herokuish (specifically, I have an app in the Plushu sandbox that kind of needs to run as root) and then I'll just replace the build script with herokuish, and rename the plugin (which is still gonna use cedarish to run herokuish, it's just that the main component will have a more descriptive name).

I guess what's not clear is that the old /exec behavior is in there:
https://github.com/progrium/buildstep/blob/master/builder/builder#L52
If it's not called via something called start it will work like /exec ... hence, copying it to /exec will mean it will behave like old /exec. Does that make sense?

 Missed the start part in there. Thanks!

And the herokuish branch works similarly, except both are symlinks. But I should point out it will have support for supervisord-like functionality, by default using forego (Go port of foreman), which as far as I know is more ideal because it's simpler (no configuration other than Procfile) and all output goes to STDOUT by default.

@progrium sorry couldn't look at this for the past few days - I'm just looking at this now. So, if I follow you and do copy /start to /exec before overriding /start does that mean that /start has to run as root? The problem I have is that now /exec runs setuidgid (which it didn't before) so it's breaking since my /start script is not run as root.
Thanks for your help!

@kenips Both /start and /exec (being the same executable) currently expect to be run as root and both drop privileges before executing the command using setuidgid.
If you start supervisord using the unprivileged account your new /exec shouldn't try to drop the privileges again as it won't work when called from supervisord. But this would lead to permission problems when calling /exec from something like dokku run <app> <command> since that would end up being run as root.
IMHO the correct way would be to start supervisord from /start as root and have /exec handle dropping the privileges to the correct user. I only had a cursory look at dokku-supervisord but if you would remove /exec und move /start to /exec you would have a /exec that already handles the privileges correctly. Then you would just need to write a /start that generates the config and starts the supervisor daemon.

hey @yabawock thank you so much for taking a look! While you suggestion works in theory, the reason why we moved supervisord to the unprivileged account was that supervisord would print out a critical warning when running as root (and someone file a bug for it). So the current behaviour of running supervisord as the unprivileged user is desired, UX wise.
For now I've restored /exec to the old behaviour within dokku-supervisord. I'll probably recommend people going with herokuish in the future as well. But let's just use this as a reference point for the need of the unprivileged user running /exec. If you hear more use cases in the future then you can decide on whether or not to change its behaviour.
Many thanks again for looking into this regardless!

Does supervisord pass down the environment to the processes it spawns? If it does you might be able to read in the environment once before starting supervisord and make the supervisor config just contain the unmodified command from the procfile.
Then /exec would drop permissions as it should, supervisord and all children would keep running unprivileged as well.

Yes it does. And I think that might work! :+1:


Update to AdoAdapterTransaction

I was unable to perform multiple database transactions on the same shared connection because when the transaction committed it was automatically disposing the connection.
This alters the AdoAdapterTransaction so that it only disposes the connection if it is not shared.

Good catch, thank you.

Hi Mark,
No problem, I did also notice that when using the Simple.Data.Oracle you
can't use the length function for character data because simple.data.ado
always replaces Length with Len in the SQL. I was able to use Like("_%") to
check that there was at least 1 character in the data but Length would be
easier.
Is there a way to override the IFunctionNameConverter functionality within
a specific adapter? As that would allow an adapter to supply the correct
names for known functions that don't match the SQL Server implementation.
Or was I missing something when trying to call the Length function?
Thanks,
Simon
Slimtech IT Consultants Limited
38, Driftway Road, Hook, Hampshire RG27 9SB
Registered in England and Wales No. 07339148

From: "Mark Rendle"
<EMAIL_ADDRESS>ub.com>
Sent: 19 January 2012 12:56
To: "SimonH"<EMAIL_ADDRESS>Subject: Re: [Simple.Data] Update to AdoAdapterTransaction (#138)
Good catch, thank you.

Reply to this email directly or view it on GitHub:
https://github.com/markrendle/Simple.Data/pull/138#issuecomment-3562005


Auth enables itself out of nowhere and randomly goes away after reboot.

Everything else rocks, broken auth makes it all useless.

Adding the below to the config provides a one-time opportunity to register a username and password at the onboarding url.
webserver:
port: 8888
auth:
If someone can provide some background on the expected behavior of this, I'd be glad to create a PR.
A. Should having no webserver section in the config work? It seems to eventually apply it.
B. Can more than one user be added? The /#/onboarding url redirects to /#/login after a single user is added.
C. Are there other valid auth params?
D. Is my "fix" above correct? Should it remain working?
I could spend days debugging to answer those myself, but I'm not sure if I'd understand the intent, just the current state.  I'd like a little background on the expected behavior and functionality of the auth flow, so I'm not documenting or fixing them in incompatible ways.
I'll have a fair amount of time to hack on this software, we're adopting it where I work. I'd much prefer working on the code to move it the direction you want vs. creating a fork with different capabilities.

Auth enabling on its own should not be possible, could you provide more information on how to reproduce?
The only way i can image it happen is if you first enable auth and restart Viseron so it saves the config as a "last known valid config".
If you then disable auth while also breaking the config YAML format, Viseron will enable safe mode and revert back to the last known valid config.
The safe mode should also be documented (saw your other issue regarding the auth docs being non-existent)

A. Should having no webserver section in the config work? It seems to eventually apply it.

The webserver is part of the default components and as such are always loaded: https://github.com/roflcoopter/viseron/blob/cff837d248b1413ba3cec728fafd3087d1ec4874/viseron/components/init.py#L82

B. Can more than one user be added? The /#/onboarding url redirects to /#/login after a single user is added.

Atm no, only the first user can be added. There is no reason for that other than me having limited time.

C. Are there other valid auth params?

Could you elaborate a bit more on what you mean here?

I'll have a fair amount of time to hack on this software, we're adopting it where I work. I'd much prefer working on the code to move it the direction you want vs. creating a fork with different capabilities.

Awesome! Would love some contributions, I am pretty much the solo developer on this project.
User management is done here: https://github.com/roflcoopter/viseron/blob/dev/viseron/components/webserver/auth.py
As you can see the groundwork for multiple users is already there, with support for groups (admin, read, write)
The API supports creating users already: https://github.com/roflcoopter/viseron/blob/cff837d248b1413ba3cec728fafd3087d1ec4874/viseron/components/webserver/api/v1/auth.py#L32
So basically all that has to be done to support multiple users is the UI portion.
And in the long run stuff like deleting users/changing passwords etc.

Hi, I've just found this project and I love it. Is simple but complete and in couple of hours I've managed to run on my Synology.
The only thing Is missing is the user part to put on a public domain so I join this discussion!

Great to hear you are enjoying Viseron!
Just to be clear, you can enable auth already but it's only possible to have a single user atm.
I intend to circle back to this at some point but can't promise when, have been busy with #716

Great to hear you are enjoying Viseron!
Just to be clear, you can enable auth already but it's only possible to have a single user atm.

Good for me, how I enable that? I don't remember to ser that in the documentation.

You enable it by adding auth: for the webserver component:
webserver:
  auth:

And then the next time you restart Viseron and open the interface you will be redirected to an onboarding page where you can create your first user.

You enable it by adding auth: for the webserver component:
webserver:
  auth:

And then the next time you restart Viseron and open the interface you will be redirected to an onboarding page where you can create your first user.

Done...awesome!!!


Error with meterpreter

Hello,
Before all, awesome work. Im doing a entry for mi blog and i get this error in msf:
[-] Handler failed to bind to <IP_ADDRESS>:7777:-  -
[-] Handler failed to bind to <IP_ADDRESS>:7777:-  -
[-] Exploit failed [bad-config]: Rex::BindFailed The address is already in use or unavailable: (<IP_ADDRESS>:7777).
I execute this in msf:
use exploit/multi/handler
set PAYLOAD android/meterpreter/reverse_tcp
set LHOST <IP_ADDRESS>
set LPORT 7777
set ExitOnSession false
exploit -j -z
Im getting nice connection btw from the android terminal
[] Sending stage (63194 bytes) to <IP_ADDRESS>
[] Meterpreter session 1 opened (<IP_ADDRESS>:7777 -> <IP_ADDRESS>:2427) at 2017-03-30 20:28:19 +0200
But i dont know who im doing bad.
Thank your for the help.
Regards.

@durkh3im When you see something like this:
[-] Exploit failed [bad-config]: Rex::BindFailed The address is already in use or unavailable: (<IP_ADDRESS>:7777).

Generally, that means you have something leftover that is already binding to that port.
Make sure you are closing your older Meterpreter sessions.


intro notebooks

Introduction notebooks to Argopy, and pyXPCM, and a start of the project (step1.ipynb). Updated REAME

The README looks good, the notebooks are fine


data_mesgs['session'] is null

Hi, im testing your parser (great job) but i have a problem.
I have an issue with the parser.

Hi, my understanding of the FIT protocol is that there are not many mandatory fields. I don't think a valid FIT file is required to have Session messages within it.
Is it just for one file (for your Garmin 935) that you get this or all? As a workaround, you can probably derive the information in Session from the Record data.
If you're able to share a FIT file here then I can take a look. I will run it through the official FIT SDK Java program "FitToCSV" to generate a CSV file

hi , here is the file
seems ok, i can see all the data in training peaks, garmin connect and fitfileviewver.com.
i don't know where is the problem....
77212839306.txt

i have a new one (from other device) with a similar problem.
says Uncaught Exception: phpFITFileAnalysis->readHeader(): not a valid header size!
but the file is ok
78091779901.txt

Just ran the x06.fit file through the SDK parser and there is session messages in there so I am investigating...

First look and I'm a bit stumped. Looking at the protocol to see if anything has changed and I note:

An activity file shall contain file_id, activity, session, and lap messages. The file may also contain record, event, length and/or hrv messages.

So there definitely should be session information in the file (confirmed by the FitCSVTool.jar). I just don't see any messages with a global message number of 18, which relate to session messages.

Hi, how are you?
do you have any update?
thanks. greetings,


feat: add into_inner for writer

In my use case, I need to call into_inner on the WavWriter struct, which didn't exist.
Related to https://github.com/ruuda/hound/pull/4 but for the writer

@ruuda I'm also interested in this. It's particularly useful when writing to an in memory buffer e.g. Cursor<Vec<u8>>


Need an "all" option under transaction field to add all transactions

Summary
Need an additional option "all" in the transaction list to be able to select all of them.
Description
A lot of times we want to add all transactions. In most cases, we have couple hundreds of transactions, it takes a long time to add them all and it is tedious if we want to do this for all of our SLO definitions. With "all" option, we can add all of them at once.

@gem-wang @ricegi  @andrewseling - I have a prototype of this working - basically instead of adding individual transactions to a query with separate where clauses, I just remove those filters all together with an "all" checkbox above the dropdown:

When checked, the dropdown is disabled and no filters are applied to the query. Here are examples for both availability (errors) and latency (calculated):
Errors:
SELECT (1 - ((FILTER(count(*), WHERE ( duration > 3))) / (count(*)))) * 100 AS 'SLO' FROM Transaction WHERE appName = 'myApp' AND httpResponseCode IS NOT NULL SINCE<PHONE_NUMBER>292 UNTIL<PHONE_NUMBER>292
Latency:
SELECT (1 - ((FILTER(count(*), WHERE (httpResponseCode LIKE '5%' OR httpResponseCode LIKE '400' OR httpResponseCode LIKE '403' OR httpResponseCode LIKE '404'))) / (count(*)))) * 100 AS 'SLO' FROM Transaction WHERE appName = 'myApp' AND httpResponseCode IS NOT NULL SINCE<PHONE_NUMBER>713 UNTIL<PHONE_NUMBER>713

can be closed. Added in https://github.com/newrelic/nr1-slo-r/pull/133


MistServer doesn't compile on Mac 10.10.3

When doing:
mkdir build
cd build
cmake ..
cmake --build .

I get these errors:
https://gist.github.com/roxlu/b2e668151c81a856b806
D

Fixed with 2.7 release. Use -DNOSHM=1 cmake option and up-to-date xcode. :grinning:


Destroying MEJS element in IE does not work when switching renderers.

Hello,
On Internet Explorer, I noticed an issue when trying to remove a player instance. You can see a demo of it here:
http://seventhcycle.net/mejs/ie/
Please try this on IE11.
On the initial load, I am using the native_hls renderer. The movie plays fine.
When I click "Load Video", I have the system attempt to load an MP4 file, and therefore don't set a renderer.
When I do this on IE, I notice that the video is not completely removed. You'll hear a second audio stream play that is independent of the video that is available on the screen. It is not possible to stop or pause this video.
I am either looking for:

A surefire way of removing the previous mediaelements instance so that no trace of the prior instance is available on IE.

OR

A fix within the software so that remove() properly works on IE, regardless of which renderer is chosen.

Please let me know if you have any trouble reproducing this.
Thanks!

@SeventhCycle Thanks. I'll check this soon

@SeventhCycle Can you download the master branch and test to check that this is solved please? I don't have a Win machine and Browserstack was a bit slow so I couldn't verify 100% if this got fixed. Let me know

@rafa8626 My apologies. I didn't see this had a response.
I went ahead and downloaded the master branch and deployed it. I also made a few code changes to the page to demonstrate the problem a little better.
http://seventhcycle.net/mejs/ie/
On IE11, the movie will load. Try clicking on "Load Video" once the movie has loaded.
On all other browsers (edge, firefox, safari, etc), the sound stops. This makes me think that the video object isn't being entirely removed on IE11.
Either:

This is a bug in the player.
If there's some other code to completely destroy the video player other than:

p.remove();

$('video').each(function() {
   $(this).remove();
});

Let me know and I'll incorporate / test.

@Qoraiche Not sure if you're maintainer now, but I figure I'd ask since I saw you providing a few implementation details on one of the other issues.
Not sure what the proper protocol is on this one. Sorry. :/

Hi, @SeventhCycle,
Sorry, I'm not the maintainer, I'm just checking issues page time to time and answering questions that I have an answer for. (since the maintainers for this project are busy).


Alexa Skill - Sheldon game

Create an Alexa Skill for the famous "Rock, Paper, Scissor, Lizard and Spock" game.

Hi !. I've created a pull request. Could you review and provide me feedback ?

Hi !. I've created a pull request. Could you review and provide me feedback ?

U can move on to next issue. I'll review it in few days.

U can move on to next issue. I'll review it in few days.


PowerShell command should perhaps be rather Get-AzsSubscriberUsage

It seems this page correctly points to provider usage API, but uses PowerShell command for tenant API that does not return correct data when used in AzureStackAdmin context. I guess actual command is Get-AzsSubscriberUsage.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 35cd7f7c-54fa-f242-bee3-860e5378b135
Version Independent ID: 1f878748-b633-f194-e27b-43c9f7f43924
Content: Provider resource usage API
Content Source: azure-stack/operator/azure-stack-provider-resource-api.md
Service: azure-stack
GitHub Login: @sethmanheim
Microsoft Alias: sethm

@tkubica12 Thanks for the feedback! Get-AzureUsageAggregates does correlate to the Tenant Resource Provider API call. I have assigned the issue to the content author to evaluate and update as appropriate.

@tkubica12 @TravisCragg-MSFT I'm really not sure what " Get-AzureUsageAggregates does correlate to the Tenant Resource Provider API call" means, but it looks like you're saying this is not the correct cmdlet. I just submitted a PR to change the doc to use Get-AzsSubscriberUsage instead. Thank you for catching that!

This is live now, please let us know if there are any other issues. #please-close


Port number for Azure SQL VM

We are using Azure SQL VM for importing large TFS collection to Azure DevOps. My question is can we use port 13164 rather than 1433 for doing the import. In that case all inbound access to be provided for 13164, Is it a workable approach?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 5a35b397-3fe8-122e-042e-280326f88a38
Version Independent ID: 3e0f0afc-abeb-01be-d767-813cd6ad78db
Content: Migration import from TFS to Azure DevOps Services - Azure DevOps Services & TFS
Content Source: docs/articles/migration-import.md
Product: devops
GitHub Login: @elbatk
Microsoft Alias: elbatk

@NilavaDgupta -- you may find answers here:

Azure DevOps Support Bot
Azure DevOps on Stack Overflow

I am closing this issue now.  You are welcome to @ mention me for any followup.
We hope to hear from you again.
My apologies for the delay in responding.


retargeted for .NET Standard 1.2

I retargeted it to .NET Standard 1.2
This allowas FamilyTreeProject.GEDCOM to also retarget to .NET Standard 1.2
The library can then be consumed from Windows (Phone) 8.1, Mono/Xamarin, Windows Universal and .NET Framework 4.5.1 and above.

Thanks Michael.  I noticed you were using the project.json project file. I am going to consider taking this but then updating to VS 2017 support which shouldn't tneed to project.json anymore

Looks like project.lock.json also got committed.. might want to gitignore that first.


Text underneath an arrow produced [Math Processing Error]

Issue Type: Bug
In a Jupyter Notebook file, this code didn't work:
$$
\require{AMScd}
\begin{CD}
A @>>i> B
\end{CD}
$$
And, wrapped in an array, \xrightarrow[i]{} did not work.
$$
\require{extpfeil}
\begin{array}{ccc}
a & \xrightarrow[i]{} & b
\end{array}
$$
VS Code version: Code 1.46.1 (cd9ea6488829f560dc949a8b2fb789f3cdc05f5d, 2020-06-17T21:13:20.174Z)
OS version: Windows_NT x64 10.0.17134
Extensions (1)

Extension
Author (truncated)
Version

python
ms-
2020.6.91350

/extPython


Passwords are encrypted rather than hashed

IPT user passwords (including for the admin) can be recovered by anyone who gains access to the cyphertext (i.e. users.xml).
The same salt is used for all passwords.
This should be changed to hashed passwords using a per-user salt.
Note: needs to have a upgrade for existing installations

See https://github.com/gbif/ipt/wiki/FAQ.wiki#8-how-do-i-reset-the-admin-password for using this to reset a lost password.

This is really bad practice. An attacker getting access to the data directory can trivially recover all the users' passwords and use them in other places. This is bad as people often re-use their passwords in many services (yeah, it's not a good idea either). On top of this, the crypto implementation is pretty lowlevel so also prone to errors.
https://github.com/patrickfav/bcrypt or something similar should probably be used. This enhances security dramatically while also reducing code complexity. I might look into creating a PR implementing these changes if gbif is interested.

This presumably means the IPT needs to re-store all user passwords. It would be useful to hear from IPT operators how they'd like this rolled out. Two scenarios could be:

All passwords are invalidated on upgrade and the administrator needs to distribute new passwords to users
As users log in they are required to change their password which is then encrypted using the chosen approach

While the first may be a nuisance it is the safest as some users may rarely log in, if ever.

The issue with option 1 is that administers might give everyone the same password and users don’t update the one assigned, which isn’t great either.
What about an option 3 where no one can login using the old passwords, but on login attempt the user can reset their password with:

send code to registered email
enter code received by email
create new password
login using new password

There's a 3rd option: when upgrading, decrypt all the passwords, hash them, and store the hashes.
It leaves it to the administrator to decide whether to reset all the passwords or not, but protects them from the moment of upgrade.

My apologies - I misunderstood the issue, even though it is in the title and that they are currently encrypted and not hashed.
@MattBlissett is correct, that during the update process the passwords can be decrypted and then securely hashed and restored. Optionally, an admin may wish to invalidate them all.

https://github.com/patrickfav/bcrypt or something similar should probably be used.

Bcrypt was also my first thought, but I decided to read up on it.  There are several people recommending Argon2, including the Open Web Application Security Project, but this seems to be a misunderstanding -- Argon2 is better for key derivation (generating a key for use in an encryption process), but Bcrypt is better for web application password hashing.
Check the number of rounds is appropriate; aiming for about 250ms of CPU time to calculate the hash seems common.


Fontmake is slow

Building Mada using fontmake takes about ~15 minutes on my (not terribly slow) machine. I can live with some slowness, but 15 minutes is a bit too much. It was actually ~5 minutes before https://github.com/alif-type/mada/commit/a367974b61ac6d6d8ef8093fbdd0adab5a7dfd55 (5 minutes is still a bit too much as well).

Yes, we know it's very slow.
For my part, I'm mostly focusing on correctness and feature-completeness more than performance at this stage. But we definitely plan on working towards making it faster.
It would be great if you could help us identify where the bottlenecks really are.

It seems to that interpolation is the slowest part, based on the fact that generating variable font is several times faster than generating static instances.

cProfile run on UFO to ttf build in Python 3.6.3 with the FontProject.run_from_ufos() method, current fontmake master branch.

Ranked by cumulative time over top 50 function calls:
         26123163 function calls (26024827 primitive calls) in 16.939 seconds

   Ordered by: cumulative time
   List reduced from 2160 to 50 due to restriction <50>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000   16.832   16.832 font_project.py:431(run_from_ufos)
        1    0.000    0.000   16.800   16.800 font_project.py:162(build_otfs)
        3    0.000    0.000   16.800    5.600 loggingTools.py:369(wrapper)
        1    0.019    0.019   11.650   11.650 font_project.py:96(remove_overlaps)
     1567    0.054    0.000   10.511    0.007 booleanOperationManager.py:102(union)
     1567    0.054    0.000   10.457    0.007 booleanOperationManager.py:62(_performOperation)
     1567    0.007    0.000    7.650    0.005 booleanOperationManager.py:64(<listcomp>)
     2878    0.026    0.000    7.639    0.003 flatten.py:65(__init__)
     5756    0.077    0.000    7.241    0.001 flatten.py:421(_convertPointsToSegments)
    51828    0.150    0.000    7.155    0.000 flatten.py:142(__init__)
     9606    0.505    0.000    5.348    0.001 flatten.py:1141(_flattenSegment)
  1162553    0.878    0.000    3.686    0.000 flatten.py:1176(_pointOnLine)
        1    0.000    0.000    3.229    3.229 font_project.py:225(save_otfs)
  3583719    2.492    0.000    2.778    0.000 flatten.py:1172(_distance)
        1    0.000    0.000    2.337    2.337 __init__.py:18(compileOTF)
        2    0.000    0.000    2.175    1.088 __init__.py:192(save)
    26/20    0.000    0.000    2.168    0.108 __init__.py:647(_writeTable)
       20    0.000    0.000    2.163    0.108 __init__.py:665(getTableData)
     9408    0.011    0.000    1.929    0.000 font.py:214(__iter__)
        1    0.002    0.002    1.920    1.920 font_project.py:116(decompose_glyphs)
     9415    0.006    0.000    1.912    0.000 layer.py:259(__getitem__)
     1567    0.021    0.000    1.906    0.001 layer.py:172(loadGlyph)
        2    0.000    0.000    1.586    0.793 C_F_F_.py:18(compile)
        2    0.000    0.000    1.586    0.793 __init__.py:92(compile)
     4701    0.007    0.000    1.519    0.000 psCharStrings.py:981(calcBounds)
     4701    0.011    0.000    1.497    0.000 psCharStrings.py:974(draw)
     4701    0.207    0.000    1.441    0.000 psCharStrings.py:282(execute)
        1    0.000    0.000    1.305    1.305 postProcessor.py:12(__init__)
        2    0.005    0.003    0.990    0.495 __init__.py:2322(recalcFontBBox)
     2878    0.110    0.000    0.956    0.000 flatten.py:969(drawPoints)
     8634    0.308    0.000    0.938    0.000 contour.py:592(drawPoints)
   682408    0.917    0.000    0.934    0.000 flatten.py:1207(_getCubicPoint)
     4701    0.038    0.000    0.908    0.000 glifLib.py:218(getGLIF)
     1567    0.005    0.000    0.900    0.001 layer.py:585(_stampGlyphDataState)
     1567    0.004    0.000    0.884    0.001 glifLib.py:266(readGlyph)
    25914    0.827    0.000    0.873    0.000 flatten.py:1118(_checkFlatPoints)
     1567    0.004    0.000    0.872    0.001 glifLib.py:252(getGLIFModificationTime)
    90249    0.053    0.000    0.870    0.000 glyphObjectPointPen.py:25(addPoint)
262867/236745    0.188    0.000    0.867    0.000 base.py:206(postNotification)
    90249    0.135    0.000    0.817    0.000 contour.py:575(addPoint)
     1594    0.816    0.001    0.816    0.001 {built-in method io.open}
     3115    0.007    0.000    0.741    0.000 glyph.py:448(draw)
     3115    0.006    0.000    0.726    0.000 glyph.py:456(drawPoints)
271501/239623    0.514    0.000    0.716    0.000 notifications.py:117(postNotification)
     1567    0.004    0.000    0.704    0.000 glifLib.py:823(_readGlyphFromTree)
     1567    0.013    0.000    0.699    0.000 glifLib.py:843(_readGlyphFromTreeFormat1)
    25914    0.015    0.000    0.690    0.000 flatten.py:1089(_scalePoints)
    25914    0.376    0.000    0.676    0.000 flatten.py:1095(<listcomp>)
     1567    0.003    0.000    0.649    0.000 booleanOperationManager.py:72(<listcomp>)
     2878    0.006    0.000    0.646    0.000 flatten.py:458(__init__)

Ranked by internal time over top 50 function calls:
         26123179 function calls (26024843 primitive calls) in 16.069 seconds

   Ordered by: internal time
   List reduced from 2162 to 50 due to restriction <50>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  3583719    2.435    0.000    2.716    0.000 flatten.py:1172(_distance)
   682408    0.906    0.000    0.922    0.000 flatten.py:1207(_getCubicPoint)
  1162553    0.858    0.000    3.605    0.000 flatten.py:1176(_pointOnLine)
    25914    0.807    0.000    0.852    0.000 flatten.py:1118(_checkFlatPoints)
     1594    0.715    0.000    0.715    0.000 {built-in method io.open}
     2878    0.521    0.000    0.647    0.000 flatten.py:463(<listcomp>)
     9606    0.494    0.000    5.238    0.001 flatten.py:1141(_flattenSegment)
271501/239623    0.456    0.000    0.648    0.000 notifications.py:117(postNotification)
    25914    0.358    0.000    0.627    0.000 flatten.py:1095(<listcomp>)
     1567    0.343    0.000    0.343    0.000 {method 'Execute' of 'pyclipper.Pyclipper' objects}
  1414542    0.317    0.000    0.317    0.000 {built-in method builtins.round}
     8634    0.290    0.000    0.887    0.000 contour.py:592(drawPoints)
  3583803    0.281    0.000    0.281    0.000 {built-in method math.sqrt}
   277371    0.255    0.000    0.477    0.000 psCharStrings.py:1046(getToken)
  2547697    0.223    0.000    0.223    0.000 {method 'append' of 'list' objects}
     2878    0.220    0.000    0.246    0.000 flatten.py:654(reCurveSubSegments)
     4701    0.180    0.000    1.311    0.000 psCharStrings.py:282(execute)
262867/236745    0.179    0.000    0.792    0.000 base.py:206(postNotification)
     2904    0.175    0.000    0.267    0.000 flatten.py:498(reCurveFromEntireInputContour)
     2878    0.167    0.000    0.242    0.000 glifLib.py:1178(_validateAndMassagePointStructures)
    51828    0.148    0.000    6.975    0.000 flatten.py:142(__init__)
     2878    0.142    0.000    0.142    0.000 booleanOperationManager.py:59(<listcomp>)
     3134    0.141    0.000    0.483    0.000 psCharStrings.py:994(compile)
    90249    0.140    0.000    0.578    0.000 contour.py:172(insertPoint)
   517113    0.139    0.000    0.166    0.000 base.py:69(_get_dispatcher)
   656711    0.139    0.000    0.139    0.000 flatten.py:1016(__init__)
    90249    0.130    0.000    0.781    0.000 contour.py:575(addPoint)
     1567    0.118    0.000    0.118    0.000 {method 'feed' of 'xml.etree.ElementTree.XMLParser' objects}
     2878    0.106    0.000    0.891    0.000 flatten.py:969(drawPoints)
   257449    0.104    0.000    0.104    0.000 {built-in method builtins.min}
757713/757712    0.100    0.000    0.104    0.000 {built-in method builtins.isinstance}
134423/118484    0.095    0.000    0.593    0.000 base.py:379(_set_dirty)
     9606    0.093    0.000    0.304    0.000 flatten.py:1180(_estimateCubicCurveLength)
     1567    0.093    0.000    0.106    0.000 specializer.py:268(specializeCommands)
  1162670    0.091    0.000    0.091    0.000 {built-in method builtins.abs}
806340/792551    0.082    0.000    0.088    0.000 {built-in method builtins.len}
   146266    0.079    0.000    0.139    0.000 psCharStrings.py:179(encodeInt)
     5756    0.075    0.000    7.058    0.001 flatten.py:421(_convertPointsToSegments)
   121388    0.072    0.000    0.223    0.000 arrayTools.py:29(updateBounds)
    45126    0.070    0.000    0.074    0.000 glyphObjectPointPen.py:64(addPoint)
   242574    0.068    0.000    0.068    0.000 py23.py:102(bytechr)
    90249    0.063    0.000    0.063    0.000 point.py:12(__init__)
     5756    0.063    0.000    0.325    0.000 pointPen.py:151(_flushContour)
   242137    0.056    0.000    0.086    0.000 py23.py:104(byteord)
   254085    0.053    0.000    0.054    0.000 {built-in method builtins.max}
    90249    0.052    0.000    0.833    0.000 glyphObjectPointPen.py:25(addPoint)
     1567    0.052    0.000   10.156    0.006 booleanOperationManager.py:62(_performOperation)
     1567    0.051    0.000   10.207    0.007 booleanOperationManager.py:102(union)
    45126    0.050    0.000    0.068    0.000 flatten.py:345(addPoint)
153554/151812    0.049    0.000    0.057    0.000 {built-in method builtins.getattr}

@khaledhosny do you have a smaller (than Mada) body of source that you build from a designspace (or is it possible to push one)?  Happy to add data on run_from_glyphs() and run_from_designspace() as well so that there is a starting point for this work.

There is https://github.com/alif-type/reem-kufi, but it does not do any interpolation which I think is the interesting difference. But I can run with cProfile myself. I’ll add my results here.

@davelab6 just asked me if we could run fontmake in parallel on the Google Cloud Platform, since I'm doing something similar for Font Bakery right now.
Sure, we could, it would need some modifications though, but we could reuse or extend what we have. It would probably build one instance per pod. Similarly, one quick improvement on a local machine would be to just spawn one fontmake process per instance to create. The kernel would take care of distributing this to different processor cores.
It would be nice though, to run the work within an instance build in parallel, i.e. map per glyph, interpolate and then reduce to the new font, on a multiprocessor machine that would be awesome.
Still, my guess is that something fishy is going on in that code, I believe it should be much faster judging from what it does. So, a thorough performance review may help as well.

It would be great if you could help us identify where the bottlenecks really are.

Yeah, that's the hard part :-) Usually it's either moving calls to functions out of a loop, to execute just before or to do some memoization (=buy speed with memory) etc. Caching is another form of hell though.
The problem is, a performance review is time-consuming since you need to get to know the code very well. About profiling to identify bottlenecks, I remember reading a blog post (which I can't find anymore of course). The author used to interrupt the running program at random points in time and claimed that the performance critical code would be very often where the program was at that moment, because that's where most time is spent. So he would take a deep look at these common locations and start optimizing there. This is a bit brutal and dirty, but not a really bad idea. For python, this would mean to start fontmake and then hit Ctrl+c while it's running. This will produce a KeyboardInterrupt with a full stack trace. I never tried this, but I would give it a shot in this case :grin:

for parallelizing python programs that do CPU intensive stuff (non I/O intensive) one cannot use multi-threading because of the infamous CPython's global interpreter lock (GIL). So one has to use multiprocessing. But unlike threads, sub-processes by default don't share memory with the parent process and communication between them may require serialization and deserialization (read pickle); plus, on platforms that don't have fork like windows spawning new process is relatively expensive.
But yes, we definitely need to look into that at some point.

because it is truly a waste that 7 of my 8 cores sit idly in the background while fontmake is crunching my fonts.

Yeah, for now, would it be possible to just start say 8 processes of fontmake by hand, to build different instances of the same project, with a shell-script or so? Or would they race for the same intermediate folders on the harddrive?

So one has to use multiprocessing. But unlike threads, sub-processes by default don't share memory with the parent process and communication between them may require serialization and deserialization (read pickle)

Sure, maybe not so bad though, since shared memory can cause problems as well.

Yeah, for now, would it be possible to just start say 8 processes of fontmake by hand, to build different instances of the same project

yes, if you have one UFO for each output font
$ fontmake -u MyFont-Regular.ufo -o ttf
$ fontmake -u MyFont-Bold.ufo -o ttf

If you have interpolated instances to make, you could run fontmake -m MyFont.designspace -o ufo first, and then run fontmake on each of the generated instance_ufo/.

Maybe something along these lines (Py3)?
concurrent_build.py
from multiprocessing import Pool
from font_project import FontProject

def build_fonts(ufo_path):
    fp = FontProject()
    fp.run_from_ufos(ufo_path, output=('ttf', 'otf'))

if __name__ == '__main__':
    with Pool(4) as p:
        print(p.map(build_fonts, ["UFOPATH1", "UFOPATH2", "UFOPATH3", "UFOPATH4"]))

The standard output/logging will get ugly I suspect but I believe that would do the trick to build in four parallel processes.
Perhaps parallel builds of ttf and otf would be of benefit too?  I don't know how much shared compilation happens there.  If that works, the build script can be modified to accept arguments for the source paths to replace the ufo path list.

Removal of the logger and timer may be low hanging fruit in production code too.  I can look at this tonight if you haven't previously assessed.  Isn't immediately clear to me what the timer is doing.

the timer does nothing else than timing. It doesn't uses resources. You see it in the profiler because it wraps the other method calls as a decorator.

I suggest you simply redirect stderr to /dev/null for now. We'll have to rethink they whole logging interface in a multiprocessing fontmake.

@anthrotype wondering if the decorated method calls are leading to this issue:
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
4    0.000    0.000   16.475    4.119 loggingTools.py:369(wrapper)

which issue?

as you can see from the legend, the total time (tottime) spent in that loggingTools decorator excluding the time made in calls to sub-functions is 0.000. The cumtime is the cumulative time spent in this and all subfunctions.

Misread the percall columns.  Sorry!

What project is flatten.py located in?

actually to mute the fontmake logger while multiprocessing, instead of redirecting stderr to /dev/null, you can just configure the global logging verbosity with a level of "ERROR" or "CRITICAL", e.g. logging.basicConfig(level="ERROR").
BTW, I noted this elsewhere, the fact fontmake's FontProject class takes a "verbose" argument to configure logging is a lie: the basicConfig function only works the first time is run, as the whole logging settings are global, not class or instance specific.

@graphicore This might get you the line by line profiling that you need once a bit more digging has been performed at a higher level?
https://github.com/rkern/line_profiler

What project is flatten.py located in?

boolenaOperations
it's not a chance that that is one of the most computationally heavy tasks

Cython possible there?

Cython is already used for the clipping. Pyclipper is a Cython wrapper for the C++ Clipper library. BooleanOperations is a python wrapper that "flatten" (converts curves into polygons) the contours, passes then on to pyclipper, and re-curves the output polygons back into beziers.
I'm afraid there's no easy shortcut here, Chris.

I can use a Makefile for this kind of parallelism, but it still wouldn’t help me that much since I build from design space not from individual UFOs. Let me try to interpolate then build the OTFs and see if it saves me much.

 :)

A comparison of sequential builds of the Hack UFO source x all four variant sources x .otf and .ttf builds vs. concurrent builds using the script in https://github.com/googlei18n/fontmake/issues/367#issuecomment-341171452 (on filepath concurrent.py for these tests).
Machine:

MacBook Pro (Retina, 15-inch, Mid 2015)
2.8 GHz Intel Core i7
16 GB 1600 MHz DDR3
macOS Sierra (10.12.6)

Timed with:
$ /usr/bin/time -p python3 concurrent.py

Sequential x 3 trials:
# Trial 1
real       101.76
user        93.10
sys          0.87

# Trial 2
real       104.27
user        96.68
sys          1.04

# Trial 3
real       107.07
user        95.90
sys          1.18

Concurrent x 3 trials
# Trial 1
real        33.45
user       121.67
sys          1.46

real        31.74
user       116.43
sys          1.17

real        31.71
user       116.26
sys          1.17

Definitely makes a significant difference.  All file writes worked without issues.  I didn't modify logging for the tests.

Worth adding optional concurrency support in the application?  Unclear to me what kind of breakage this might cause and how best to test for it.  If not, maybe additional documentation for users who have extended compile times?

Parallel builds using cPython 3.6.3 vs PyPy 5.9.0 on the same system as tests in https://github.com/googlei18n/fontmake/issues/367#issuecomment-341294388.  Same timing approach.  Modified script to support PyPy (multiprocessing.Pool is not a context manager in Py 2) tests at bottom.
cPython 3.6.3
Trial # 1
real        32.24
user       110.60
sys          1.19

Trial # 2
real        30.82
user       112.36
sys          1.16

Trial # 3
real        31.88
user       116.27
sys          1.27

PyPy 5.9.0
Trial # 1
real        24.38
user        92.77
sys          2.01

Trial # 2
real        24.54
user        92.90
sys          2.01

Trial # 3
real        25.16
user        94.69
sys          2.19

Test script modified to the following to support PyPy tests in Py2.  Works in both Py 2 and Py 3.
#!/usr/bin/env python
# -*- coding: utf-8 -*-

from multiprocessing import Pool
from fontmake.font_project import FontProject

def build_fonts(ufo_path):
    fp = FontProject()
    fp.run_from_ufos(ufo_path, output=('ttf', 'otf'))

if __name__ == '__main__':
    # with Pool(4) as p:
    #     p.map(build_fonts, ["Hack-Regular.ufo", "Hack-Bold.ufo", "Hack-Italic.ufo", "Hack-BoldItalic.ufo"])

    p = Pool(4)
    p.map(build_fonts, ["Hack-Regular.ufo", "Hack-Bold.ufo", "Hack-Italic.ufo", "Hack-BoldItalic.ufo"])

And for comparison here is PyPy on a single core:
Trial #1
real        67.66
user        66.09
sys          1.29

Trial #2
real        67.83
user        66.29
sys          1.35

Trial #3
real        66.17
user        64.73
sys          1.33

Can compare to the cPython 3.6.3 data in https://github.com/googlei18n/fontmake/issues/367#issuecomment-341294388 (I re-ran the tests and results ~same as shown there).
For those who can't use parallel builds, there may still be a significant performance improvement with a switch to PyPy.

With PyPy it takes ~6 minutes to build everything, but without overlap removal and probably CFF optimization:
real	5m55.376s
user	5m50.792s
sys	0m5.334s

Python 3:
real	9m44.509s
user	9m39.684s
sys	0m4.949s

but without overlap removal and probably CFF optimization

@khaledhosny is it necessary to modify the logging level to see that is the case during the build?  The default standard output looks the same for me (i.e. no notification of error with any of the steps in the build process) as it does for our cPython builds (PyPy stdout shown):
INFO:fontmake.font_project:Building OTFs
INFO:fontmake.font_project:Decomposing glyphs for Hack-Regular
INFO:fontmake.font_project:Removing overlaps for Hack-Regular
INFO:fontmake.font_project:Saving OTF for Hack-Regular
INFO:fontmake.font_project:Building TTFs
INFO:fontmake.font_project:Decomposing glyphs for Hack-Regular
INFO:fontmake.font_project:Removing overlaps for Hack-Regular
INFO:fontmake.font_project:Converting curves for Hack-Regular
INFO:cu2qu.ufo:New spline lengths: 1: 4908, 2: 4107, 3: 504, 4: 71, 5: 16
INFO:fontmake.font_project:Saving TTF for Hack-Regular
INFO:fontmake.font_project:Building OTFs
INFO:fontmake.font_project:Decomposing glyphs for Hack-Bold
INFO:fontmake.font_project:Removing overlaps for Hack-Bold
INFO:fontmake.font_project:Saving OTF for Hack-Bold
INFO:fontmake.font_project:Building TTFs
INFO:fontmake.font_project:Decomposing glyphs for Hack-Bold
INFO:fontmake.font_project:Removing overlaps for Hack-Bold
INFO:fontmake.font_project:Converting curves for Hack-Bold
INFO:cu2qu.ufo:New spline lengths: 1: 4723, 2: 4510, 3: 448, 4: 69, 5: 17, 6: 4, 7: 3
INFO:fontmake.font_project:Saving TTF for Hack-Bold
INFO:fontmake.font_project:Building OTFs
INFO:fontmake.font_project:Decomposing glyphs for Hack-Italic
INFO:fontmake.font_project:Removing overlaps for Hack-Italic
INFO:fontmake.font_project:Saving OTF for Hack-Italic
INFO:fontmake.font_project:Building TTFs
INFO:fontmake.font_project:Decomposing glyphs for Hack-Italic
INFO:fontmake.font_project:Removing overlaps for Hack-Italic
INFO:fontmake.font_project:Converting curves for Hack-Italic
INFO:cu2qu.ufo:New spline lengths: 1: 5144, 2: 5262, 3: 524, 4: 80, 5: 12, 6: 1
INFO:fontmake.font_project:Saving TTF for Hack-Italic
INFO:fontmake.font_project:Building OTFs
INFO:fontmake.font_project:Decomposing glyphs for Hack-BoldItalic
INFO:fontmake.font_project:Removing overlaps for Hack-BoldItalic
INFO:fontmake.font_project:Saving OTF for Hack-BoldItalic
INFO:fontmake.font_project:Building TTFs
INFO:fontmake.font_project:Decomposing glyphs for Hack-BoldItalic
INFO:fontmake.font_project:Removing overlaps for Hack-BoldItalic
INFO:fontmake.font_project:Converting curves for Hack-BoldItalic
INFO:cu2qu.ufo:New spline lengths: 1: 4971, 2: 5395, 3: 455, 4: 85, 5: 7, 6: 1
INFO:fontmake.font_project:Saving TTF for Hack-BoldItalic
real        67.83
user        66.29
sys          1.35

Sorry my wording was misleading, I intentionally disabled overlap removal and CFF optimization to speed non-release builds (I was saying something else above originally, but edited the text before posting).

There are specific options for keeping overlaps and disable subroutinization. Those have nothing to do with logging

Ah, I see.  Misunderstood.  Thought that you meant that the decrease in time was due to skipped routines.

Thanks both for the profiler data you have collected, it's very helpful.
I prefer to defer this to when I'll have time to properly tackle this, as it requires some rethinking of the fontmake interface and also of some of its dependency.
For now I'd recommend you parallelize with GNU make or using your ad-hoc python scripts, or use PyPy which gets you 30% faster for free.
I don't think it's fontmake's responsibility to document how to do the above.

It would be helpful, though, if there was a way to generate a single UFO instance from a designspace file, right now all instances are generated once and this makes it hard to do things in parallel (more so with make, as one rule making multiple targets is not compatible with parallelism).

I know... But I believe mutatorMath API doesn't allow that. At least mutatorMath.ufo.build (which is what fontmake is using) always instantiates all the instances in the designspace.

Nice discussion, I have had missed this.
So, this is the first time I'm seeing PyPy perform better than cPython.  Good to know.
Looks like majority of time is spent in booleanOperations.flatten; so we're back to replacing that. Let's figure out who's going to work on that. cc @adrientetar

the majority of time for compiling a single UFO is spent on booleanOperations, yes. But when compiling a designspace, the majority of time is spent in mutatorMath/fontMath/defcon/...

For starters, how about changing epsilon in booleanOperations flatten.py from 1e-8 to 1e-4? @typemytype

And for mutatorMath/fontMath/defcon, we should add a lean ufoLib to fontTools, with minimal Glyph and Font and Contour etc objects that already do math, and use varLib on them.

this is the first time I'm seeing PyPy perform better than cPython

I hear pypy works best with long running processes as JIT takes some time to heat up

best with long running processes as its JIT compiler takes some time to heat up

@anthrotype below is from PyPy web site:
There are two cases that you should be aware where PyPy will not be able to speed up your code:

Short-running processes: if it doesn't run for at least a few seconds, then the JIT compiler won't have enough time to warm up.
If all the time is spent in run-time libraries (i.e. in C functions), and not actually running Python code, the JIT compiler will not help.

So the case where PyPy works best is when executing long-running programs where a significant fraction of the time is spent executing Python code. This is the case covered by the majority of our benchmarks, but not all of them – the goal of PyPy is to get speed but still support (ideally) any Python program.

For starters, how about changing epsilon in booleanOperations flatten.py from 1e-8 to 1e-4? @typemytype

With this change, time fontmake -g NotoSansArabic-MM.glyphs -o ttf goes down from 2min to... 2min!

1    0.000    0.000   16.513   16.513 font_project.py:431(run_from_ufos)
1    0.000    0.000   16.476   16.476 font_project.py:172(build_ttfs)
4    0.000    0.000   16.475    4.119 loggingTools.py:369(wrapper)
1    0.017    0.017   11.306   11.306 font_project.py:96(remove_overlaps)
1567    0.051    0.000   10.227    0.007 booleanOperationManager.py:102(union)
1567    0.052    0.000   10.176    0.006 booleanOperationManager.py:62(_performOperation)
1567    0.006    0.000    7.465    0.005 booleanOperationManager.py:64()
2878    0.024    0.000    7.454    0.003 flatten.py:65(init)
5756    0.075    0.000    7.075    0.001 flatten.py:421(_convertPointsToSegments)
51828    0.148    0.000    6.991    0.000 flatten.py:142(init)
9606    0.493    0.000    5.256    0.001 flatten.py:1141(_flattenSegment)

I cannot reproduce this.  flatten.py takes, like, 10 seconds out of 180s runtime on NotoSansArabic-MM.glyphs for me.

@behdad If you'd like to replicate with the source that I used:
Data that I showed indicating the issue with flatten.py in:

https://github.com/googlei18n/fontmake/issues/367#issuecomment-340969823
https://github.com/googlei18n/fontmake/issues/367#issuecomment-340970379

were direct builds from UFO source files to ttf / otf binaries using FontProject.FontProject.run_from_ufos() with the regular variant of the UFO source directories located here:
https://github.com/source-foundry/Hack/tree/master/source
tar.gz download: https://github.com/source-foundry/Hack/releases/download/v3.000/Hack-v3.000-ttf.tar.gz
The parallel build data that I posted above included all four variant UFO source directories in the repository link above.
I don't believe anyone has presented profiling directly from glyphs source files yet.  Not sure how that might influence the booleanOperations issue that occurred with my testing.

For starters, how about changing epsilon in booleanOperations flatten.py from 1e-8 to 1e-4? @typemytype

dont know if that is going to change that much, BooleanOperations is and always will be a heavy and time consuming call...
I guess the best option is to port to a more completely non py package... as proposed here https://github.com/typemytype/booleanOperations/issues/40

Pushed this for anyone who is building directly from UFO source and might find it of use.  It automates the number of spawned processes for requested compiles up to a max of the number of cores detected on the system (can be manually set if desired):
fmp
Builds ttf + otf by default. Modify directly in the script.  Details on README.

Thanks!  We should add a mode to fontmake to do this automatically.

Thanks! We should add a mode to fontmake to do this automatically.

Agree!  It sounds like @anthrotype plans to tackle this at some stage.  https://github.com/googlei18n/fontmake/issues/367#issuecomment-341896459

Khaled, I'm curious, what's the current build time for mada?

I don’t know, but it does not really matter. Python is slow and I don’t think this is going to change.

FWIW, I manged to cut the build time by half (down to 15 minutes from 30 minutes) by using --optimize-cff=0 and --keep-overlaps, this produces bigger files and overlapping glyphs, but meh.
I managed to cut down to 4 minutes by building only the masters with fontmake, using varLib to build a variable font out of them, and varLib.mutator to create the instances from it.

@khaledhosny If you're bored, try building Mada instances with https://gitlab.gnome.org/GNOME/cantarell-fonts/blob/master/scripts/make-static-fonts.py and https://gitlab.gnome.org/GNOME/cantarell-fonts/blob/master/scripts/instantiator.py. It launches a subprocess per instance and psautohints the resulting OTF. I can build the 14 Noto Sans uprights OTFs in 2:50 mins (without autohinting) on my 16 logical cores, only takes around 7 GB of RAM for the duration.

Not sure if this helps but the command checkoutlinesufo -e from FDK adds another layer named glyphs.com.adobe.type.processedglyphs to UFO  which contains glyphs that have their overlaps removed. It uses a hash map to make sure if the glyph has changed or not. This fastens the build process and I think this could be useful to make ufo2ft faster.


Find a way to avoid resetting camera when changing surface selections

Clicking to change the surface or vector/scalar field selection in a surface view (e.g. https://www.figurl.org/f?v=gs://figurl/volumeview-2&d=e9d66edac2246a9db4aca6c515d4b79aaa450796&channel=flatiron1&label=Test volumeview workspace or https://www.figurl.org/f?v=gs://figurl/volumeview-2&d=a997d5dd425a92ee67aa750aa48a9d9c3193f474&channel=flatiron1&label=red blood cell ) results in resetting the camera to its initial position. This is less than ideal, because it interferes with the user's ability to toggle back and forth between different field displays if the camera has been repositioned.

I agree, this should be high priority.