Is the config option "end" available?

Hello！
I'm running the docker version of elastalert, when I set endtime in the config.json like this:
"start": "2015-05-18T00:00:00",
"end": "2015-05-23T00:00:00"
then the ElastAlert exited with code 0 in a second without any warning.
I found these lines in the start(self) function in elastalert.py:
        while self.running:
            next_run = datetime.datetime.utcnow() + self.run_every

            # Quit after end_time has been reached
            if self.args.end:
                endtime = ts_to_dt(self.args.end)

                if next_run.replace(tzinfo=dateutil.tz.tzutc()) > endtime:
                    exit(0)

It seems the next_run is always now+run_every,so when I set endtime(2015-05-23),the elastAlert will stop running and exit,which means I cant set end in the config.
Is there other ways to set endtime in the past?Thanks!

elastalert --verbose --start 2015-05-18 --end 2015-0523

Thanks for your reply!
However I tried the config --end 2015-0523,and 0523 seems can't be parse to the right month format.
Here's the log:

08:35:54.142Z ERROR elastalert-server:
ProcessController:  Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
08:35:54.164Z ERROR elastalert-server:
ProcessController:      "main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/opt/elastalert/elastalert/elastalert.py", line 2019, in
sys.exit(main(sys.argv[1:]))
File "/opt/elastalert/elastalert/elastalert.py", line 2015, in main
client.start()
File "/opt/elastalert/elastalert/elastalert.py", line 1148, in start
endtime = ts_to_dt(self.args.end)
File "elastalert/util.py", line 127, in ts_to_dt
dt = dateutil.parser.parse(timestamp)
File "/usr/lib/python2.7/site-packages/python_dateutil-2.6.1-py2.7.egg/dateutil/parser.py", line 1182, in parse
return DEFAULTPARSER.parse(timestr, **kwargs)
File "/usr/lib/python2.7/site-packages/python_dateutil-2.6.1-py2.7.egg/dateutil/parser.py", line 578, in parse
if cday > monthrange(cyear, cmonth)[1]:
File "/usr/lib/python2.7/calendar.py", line 120, in monthrange
08:35:54.185Z ERROR elastalert-server:
ProcessController:      raise IllegalMonthError(month)
calendar.IllegalMonthError: bad month number 523; must be 1-12

I have made a comment on this in #2666


Reskin - update trash icon to match ghost icon spec

Component instance usage design: https://www.figma.com/file/9hUYtvCKjOyB92vt8ukYtD/Product-Handoff?node-id=258-32926&mode=dev
Implementation:

Closes #1682

@mudrila updated the create proposal designs in the handoff file
I changed small ghost button-icon padding from 4px to 6px. This makes the small ghost button-icon match the height of the other small buttons in the page header. In addition I changed the hover value from White Alpha/04 to 08
Button-icons in design system

Button size is off BUT I also noticed an inconsistency on the design side with the icon buttons. Will provide updated figma link.
UPDATE @mudrila updated the create proposal designs in the handoff file
I changed small ghost button-icon padding from 4px to 6px. This makes the small ghost button-icon match the height of the other small buttons in the page header. In addition I changed the hover value from White Alpha/04 to 08
Button-icons in design system

@xraystyle1980 I checked the figma and it still says 4px padding both on https://www.figma.com/file/ur5hElxplkmlY9gnGKlwrm/Components-%26-Styles---Decent-Design-System-1.0.0?node-id=1362-18722&mode=dev and https://www.figma.com/file/9hUYtvCKjOyB92vt8ukYtD/Product-Handoff?node-id=2670-23059&mode=dev But I see you updated the size on Proposal Builder / DAO Creation flows. So, just to confirm - icon size is 20x20 and padding is 4px. Is that right? I'll update just colors then

Yes you are correct @mudrila. Sorry for the confusion. I made another adjustment to the icons to increase their size to 20px and updated padding to 4px for the same 28px button height.

Are these the right sizes now?
one of these I took from figma and the other from the app?

ow?

Note on this PR @mudrila - the trash icon seems to have unintended behaviour of scrolling into the nav bar
see screenshots pre

and post scroll

shall we  create a new issue instead of tackling it here @adamgall ?

makes sense and cheers for heads up


fix url_valid? for checking installed npm packages

Description
Fixes not_if guard on npm packages by correctly checking installed npm packages.
Issues Resolved
Without this fix, npm packages will attempt to be installed on every Chef run, because the npm list check doesn't match against the resolved urls correctly (it is looking for the 'url' string, while the resolved string can actually contain any valid URLs without the exact string 'url' in them.)
I was able to manually reproduce the issue with multiple converge runs against the existing test-kitchen integration cookbook, on the package-ubuntu-1404 platform.
Check List

[ ] All tests pass.
[ ] New functionality includes testing. (could use help on this)
[ ] New functionality has been documented in the README if applicable

seems this issue has already been identified fixed several times in other PRs; closing..


Set ImageNet data augmentation by default

https://github.com/apache/incubator-mxnet/blob/a38278ddebfcc9459d64237086cd7977ec20c70e/example/image-classification/train_imagenet.py#L42
When I try to train imagenet with this line commented, the train-accuracy reaches 99% while the validation-accuracy is only less than 50% (single machine, 8 GPUs, global batchsize=2048, Resnet50, fp32). Absolutely this is overfitting.
Then I uncomment this line and try again with the same experiment settings. This time both train and validation accuracy converge to about 66%, which looks like normal result.
Thus, it seems that this data augmentation is pretty important for ImageNet training. Perhaps it will be better to uncomment this as default, so that future developers won't get confused by the overfitting issue.

@sandeep-krishnamurthy @eric-haibin-lin Can you take a look?
@mxnet-label-bot Add [pr-awaiting-review]

@mxnet-label-bot update [pr-awaiting-response]

Hi @sandeep-krishnamurthy , I agree with @vishaalkapoor that the parameter argument should be set as default. But I see the parameter is already provided in  set_imagenet_aug. Perhaps one neat way would be to directly set it as uncommented? No other change will be involved.

@vishaalkapoor Could you suggest a way forward on this PR?

@ymjiang can you please set a command line argument to either override or keep set_imagenet_aug line. I think that is what @vishaalkapoor was suggesting.

@ymjiang Could you please address the review comments made by @anirudhacharya. It seems no updates since last 2 weeks. Thanks!

@karan6181 @anirudhacharya Sorry for my delay. I committed two new changes to enable data-augmentation with command-line argument. Please review if they are appropriate.

@anirudhacharya Ping for review.
@ymjiang Can you look into the CI failures ?

@vishaalkapoor Can you take a look at this PR again?

@ymjiang Hi, could you rebase to latest master? it should resolve the failing CI test

@ymjiang Hi, could you rebase to latest master? it should resolve the failing CI test

@ymjiang Gentle ping...

Rebased to master now: https://github.com/apache/incubator-mxnet/pull/15189. Will close this issue.


also use the onHasStreamData callback for the crypto stream

Depends on #1069.

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@ad3fbd3). Click here to learn what that means.
The diff coverage is 92.86%.

@@            Coverage Diff            @@
##             master    #1076   +/-   ##
=========================================
  Coverage          ?   90.89%
=========================================
  Files             ?      107
  Lines             ?     7279
  Branches          ?        0
=========================================
  Hits              ?     6616
  Misses            ?      432
  Partials          ?      231

Impacted Files
Coverage Δ

streams_map.go
89.76% <ø> (ø)

crypto_stream.go
100% <ø> (ø)

stream.go
88.24% <ø> (ø)

packet_packer.go
90.35% <100%> (ø)

session.go
84.71% <100%> (ø)

stream_framer.go
100% <100%> (ø)

internal/handshake/tls_extension_handler_client.go
90.16% <33.33%> (ø)

internal/handshake/tls_extension_handler_server.go
88.89% <33.33%> (ø)

send_stream.go
95.71% <85.71%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ad3fbd3...357c594. Read the comment docs.


Add a new enum binding

In line with the changes at https://github.com/kinetiknz/cubeb/pull/501.
How can we get this into gecko? Is it necessary to make a release and to bump the manifest and to re-vendor or something? Do we have instructions to do this?

This change is

@kinetiknz, @djg, if you can have a look?

Is it necessary to make a release and to bump the manifest and to re-vendor or something?

Yeah, sadly.  And that requires push access to crates.io.  Need to get around to merging the various cubeb/audioipc repos and dropping Gecko dependency on the crates.io stuff.
For this patch, I think we can get away with not updating cubeb-rs in Gecko for now - we're not using CUBEB_STREAM_PREF_VOICE in the Rust code yet.

Sorry that's my fault. Dealing with vendoring rust into gecko used to be a
huge pain-in-the-arse and publishing to crates.io was easier.
On Thu, 11 Apr 2019 at 07:43, Matthew Gregan<EMAIL_ADDRESS>wrote:

Is it necessary to make a release and to bump the manifest and to
re-vendor or something?
Yeah, sadly. And that requires push access to crates.io. Need to get
around to merging the various cubeb/audioipc repos and dropping Gecko
dependency on the crates.io stuff.
For this patch, I think we can get away with not updating cubeb-rs in
Gecko for now - we're not using CUBEB_STREAM_PREF_VOICE in the Rust code
yet.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/djg/cubeb-rs/pull/41#issuecomment-481878707, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAJhrWva4XQoQsfQivizTTZzmdFudoQjks5vflsRgaJpZM4cnumQ
.

--
Dan Glastonbury, Dan dot Glastonbury at gmail dot com
`Pour encourjay lays ortras'


Fastparquet reader with degraded performance in dask==2.2

Versions
dask==2.2
fastparquet==0.3.1
pyarrow==0.14.1
Code to reproduce
Create and save parquet files
import numpy as np
import pandas as pd
import dask.dataframe as dd
import random
import shutil
import fastparquet
import os

#generate dummy df
def generate_df(size, ncols):

    dates = pd.date_range('1980-01-01', periods=size, freq='1T')
    data = np.random.rand(size, ncols)

    df = pd.DataFrame(data=data, columns=[str(col) for col in range(ncols)], index=dates)
    return df

Generate dask dataframe
small_df = generate_df(1000000, ncols=20)

df1 = dd.from_pandas(small_df, npartitions=40)
if os.path.exists('test_fastparquet'):
    shutil.rmtree('test_fastparquet')

df1.to_parquet('test_fastparquet', compression='snappy', engine='fastparquet')

small_df = generate_df(1000000, ncols=20)

df2 = dd.from_pandas(small_df, npartitions=40)
if os.path.exists('test_pyarrow'):
    shutil.rmtree('test_pyarrow')

df2.to_parquet('test_pyarrow', compression='snappy', engine='pyarrow')

reading metadata and compute for 20 columns
%%timeit
dd.read_parquet('test_fastparquet/', engine='fastparquet', gather_statistics=True)
19.2 s ± 1.91 s per loop (mean ± std. dev. of 7 runs, 1 loop each)

%%timeit
dd.read_parquet('test_fastparquet/', engine='fastparquet', gather_statistics=False)
104 ms ± 4.59 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

%%timeit
dd.read_parquet('test_pyarrow/', engine='pyarrow', gather_statistics=True)
37.8 ms ± 1.02 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

%%timeit
dd.read_parquet('test_pyarrow/*.parquet', engine='pyarrow', gather_statistics=False)
9.74 ms ± 121 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

ddf1 = dd.read_parquet('test_fastparquet/', engine='fastparquet', gather_statistics=True)
ddf2 = dd.read_parquet('test_fastparquet/', engine='fastparquet', gather_statistics=False)
ddf3 = dd.read_parquet('test_pyarrow/', engine='pyarrow', gather_statistics=True)
ddf4 = dd.read_parquet('test_pyarrow/*.parquet', engine='pyarrow', gather_statistics=False)

%%timeit
ddf1.compute()
9.19 s ± 264 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%%timeit
ddf2.compute()
9.06 s ± 254 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%%timeit
ddf3.compute()
1.05 s ± 37.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%%timeit
ddf4.compute()
909 ms ± 52.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Generate dask dataframe
small_df = generate_df(1000000, ncols=2)

df1 = dd.from_pandas(small_df, npartitions=40)
if os.path.exists('test_fastparquet'):
    shutil.rmtree('test_fastparquet')

df1.to_parquet('test_fastparquet', compression='snappy', engine='fastparquet')

small_df = generate_df(1000000, ncols=2)

df2 = dd.from_pandas(small_df, npartitions=40)
if os.path.exists('test_pyarrow'):
    shutil.rmtree('test_pyarrow')

df2.to_parquet('test_pyarrow', compression='snappy', engine='pyarrow')

reading metadata and compute for 2 columns
%%timeit
dd.read_parquet('test_fastparquet/', engine='fastparquet', gather_statistics=True)
476 ms ± 13 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%%timeit
dd.read_parquet('test_fastparquet/', engine='fastparquet', gather_statistics=False)
23.3 ms ± 1.18 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

%%timeit
dd.read_parquet('test_pyarrow/', engine='pyarrow', gather_statistics=True)
22.7 ms ± 3.78 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

%%timeit
dd.read_parquet('test_pyarrow/*.parquet', engine='pyarrow', gather_statistics=False)
9.16 ms ± 2.18 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)

ddf1 = dd.read_parquet('test_fastparquet/', engine='fastparquet', gather_statistics=True)
ddf2 = dd.read_parquet('test_fastparquet/', engine='fastparquet', gather_statistics=False)
ddf3 = dd.read_parquet('test_pyarrow/', engine='pyarrow', gather_statistics=True)
ddf4 = dd.read_parquet('test_pyarrow/*.parquet', engine='pyarrow', gather_statistics=False)

%%timeit
ddf1.compute()
2.18 s ± 151 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%%timeit
ddf2.compute()
2.02 s ± 59.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%%timeit
ddf3.compute()
326 ms ± 54.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%%timeit
ddf4.compute()
297 ms ± 15.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Now with dask=2.1.0
Versions
dask==2.1
fastparquet==0.3.1
pyarrow==0.14.1
Generate dask dataframe
small_df = generate_df(1000000, ncols=20)

df1 = dd.from_pandas(small_df, npartitions=40)
if os.path.exists('test_fastparquet'):
    shutil.rmtree('test_fastparquet')

df1.to_parquet('test_fastparquet', compression='snappy', engine='fastparquet')

small_df = generate_df(1000000, ncols=20)

df2 = dd.from_pandas(small_df, npartitions=40)
if os.path.exists('test_pyarrow'):
    shutil.rmtree('test_pyarrow')

df2.to_parquet('test_pyarrow', compression='snappy', engine='pyarrow')

reading metadata and compute for 20 columns
%%timeit
dd.read_parquet('test_fastparquet/', engine='fastparquet', infer_divisions=True)
105 ms ± 455 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

%%timeit
dd.read_parquet('test_fastparquet/', engine='fastparquet', infer_divisions=False)
100 ms ± 1.68 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

%%timeit
dd.read_parquet('test_pyarrow/', engine='pyarrow', infer_divisions=True)
42 ms ± 8.58 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
dd.read_parquet('test_pyarrow/*.parquet', engine='pyarrow', infer_divisions=False)

%%timeit
dd.read_parquet('test_pyarrow/*.parquet', engine='pyarrow', infer_divisions=False)
26.2 ms ± 353 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

ddf1 = dd.read_parquet('test_fastparquet/', engine='fastparquet', infer_divisions=True)
ddf2 = dd.read_parquet('test_fastparquet/', engine='fastparquet', infer_divisions=False)
ddf3 = dd.read_parquet('test_pyarrow/', engine='pyarrow', infer_divisions=True)
ddf4 = dd.read_parquet('test_pyarrow/*.parquet', engine='pyarrow', infer_divisions=False)

%%timeit
ddf1.compute()
558 ms ± 4.85 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%%timeit
ddf2.compute()
563 ms ± 12.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%%timeit
ddf3.compute()
481 ms ± 3.88 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%%timeit
ddf4.compute()
503 ms ± 23.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Generate dask dataframe
small_df = generate_df(1000000, ncols=20)

df1 = dd.from_pandas(small_df, npartitions=40)
if os.path.exists('test_fastparquet'):
    shutil.rmtree('test_fastparquet')

df1.to_parquet('test_fastparquet', compression='snappy', engine='fastparquet')

small_df = generate_df(1000000, ncols=20)

df2 = dd.from_pandas(small_df, npartitions=40)
if os.path.exists('test_pyarrow'):
    shutil.rmtree('test_pyarrow')

df2.to_parquet('test_pyarrow', compression='snappy', engine='pyarrow')

reading metadata and compute for 2 columns
%%timeit
dd.read_parquet('test_fastparquet/', engine='fastparquet', infer_divisions=True)
20.6 ms ± 784 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

%%timeit
dd.read_parquet('test_fastparquet/', engine='fastparquet', infer_divisions=False)
19.5 ms ± 933 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

%%timeit
dd.read_parquet('test_pyarrow/', engine='pyarrow', infer_divisions=True)
24.3 ms ± 2 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

%%timeit
dd.read_parquet('test_pyarrow/*.parquet', engine='pyarrow', infer_divisions=False)
17.8 ms ± 577 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

ddf1 = dd.read_parquet('test_fastparquet/', engine='fastparquet', infer_divisions=True)
ddf2 = dd.read_parquet('test_fastparquet/', engine='fastparquet', infer_divisions=False)
ddf3 = dd.read_parquet('test_pyarrow/', engine='pyarrow', infer_divisions=True)
ddf4 = dd.read_parquet('test_pyarrow/*.parquet', engine='pyarrow', infer_divisions=False)

%%timeit
ddf1.compute()
124 ms ± 11.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%%timeit
ddf2.compute()
140 ms ± 19.3 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

%%timeit
ddf3.compute()
267 ms ± 24.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%%timeit
ddf4.compute()
331 ms ± 42.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

This behaviour is obviously not desired and is most likely related with the last refactor done to parquet engine. I did not go further to isolate the problem since you guys can have a better clue of what is going on. I'm more than happy to help on this btw!
Note: Everything is running locally on my desktop.

Locally, I am seeing that .compute() takes ~380ms for fastparquet, all columns (but I am running without snappy).
I can confirm that
dd.read_parquet('test_fastparquet/', engine='fastparquet', gather_statistics=True) is taking a very long time. The profile looks like
         60992183 function calls (58035069 primitive calls) in 25.900 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
2960160/3360 7.722    0.000   25.498    0.008 api.py:642(statistics)
  5644800    4.290    0.000    4.290    0.000 {built-in method numpy.frombuffer}
  5644800    2.712    0.000    8.499    0.000 encoding.py:43(read_plain)
   282240    2.706    0.000    4.753    0.000 api.py:699(<listcomp>)
 22579414    2.047    0.000    2.047    0.000 {method 'get' of 'dict' objects}
   134400    1.830    0.000   19.880    0.000 api.py:694(<dictcomp>)
  5644800    1.498    0.000    1.498    0.000 util.py:93(byte_buffer)
  5644800    1.094    0.000    1.713    0.000 util.py:76(ensure_bytes)
  8821295    0.921    0.000    0.922    0.000 {built-in method builtins.isinstance}
        1    0.300    0.300   25.998   25.998 fastparquet.py:158(read_metadata)
  2893155    0.231    0.000    0.231    0.000 {method 'join' of 'str' objects}
     3360    0.180    0.000   25.678    0.008 api.py:171(statistics)
    13480    0.104    0.000    0.160    0.000 api.py:166(<listcomp>)
     6720    0.052    0.000    0.083    0.000 converted_types.py:86(convert)
6812/6810    0.038    0.000    0.038    0.000 {built-in method numpy.array}
   285840    0.038    0.000    0.038    0.000 {built-in method builtins.getattr}
    70560    0.030    0.000    0.038    0.000 schema.py:97(schema_element)
    13480    0.020    0.000    0.182    0.000 api.py:163(columns)
284148/283922    0.019    0.000    0.019    0.000 {built-in method builtins.len}

@rjzamora

This is how quickly is should go:
In [12]: import fastparquet
In [13]: pf = fastparquet.ParquetFile('test_fastparquet/')
In [14]: %timeit pf.statistics
4.66 ms ± 88.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Thanks for raising this @RodrigoNeves95! -  Yep, definitely a bug.  Thanks for finding the fix so quickly @martindurant!

I can also fix this pretty easily on the fastparquet side, and release that, rather than needing a new Dask release. @mrocklin , a preference between where to fix?

No strong preference.  I think it's reasonable for us to push out a bugfix
Dask release soon if that would be helpful.  It shouldn't take someone more
than an hour.  I would probably wait a week though to let things accrue.
On Thu, Aug 1, 2019 at 8:37 AM Martin Durant<EMAIL_ADDRESS>wrote:

I can also fix this pretty easily on the fastparquet side, and release
that, rather than needing a new Dask release. @mrocklin
https://github.com/mrocklin , a preference between where to fix?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/5200?email_source=notifications&email_token=AACKZTEPED4PM5RY2C4N4EDQCL7NHA5CNFSM4IIPTWQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3K767A#issuecomment-517341052,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AACKZTAQMYY7M5DSI5LOQHTQCL7NHANCNFSM4IIPTWQA
.

As far as the other measurements go, there are more worries there, but they don't seem to match up with what I am seeing running timeit here.
Note that I (or, preferably, someone else) could maybe also do work on the fastparquet.api.statistics() function  to shortcut whenever a row_group has no stats or isn't ordered, or to ignore columns that are not of interest, etc.

Releasing fastparquet with that fix. That will be the last of fastparquet for py2!

Thank you @martindurant for the quick fix!

Locally, I am seeing that .compute() takes ~380ms for fastparquet, all columns (but I am running without snappy).
I can confirm that
dd.read_parquet('test_fastparquet/', engine='fastparquet', gather_statistics=True) is taking a very long time. The profile looks like
         60992183 function calls (58035069 primitive calls) in 25.900 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
2960160/3360 7.722    0.000   25.498    0.008 api.py:642(statistics)
  5644800    4.290    0.000    4.290    0.000 {built-in method numpy.frombuffer}
  5644800    2.712    0.000    8.499    0.000 encoding.py:43(read_plain)
   282240    2.706    0.000    4.753    0.000 api.py:699(<listcomp>)
 22579414    2.047    0.000    2.047    0.000 {method 'get' of 'dict' objects}
   134400    1.830    0.000   19.880    0.000 api.py:694(<dictcomp>)
  5644800    1.498    0.000    1.498    0.000 util.py:93(byte_buffer)
  5644800    1.094    0.000    1.713    0.000 util.py:76(ensure_bytes)
  8821295    0.921    0.000    0.922    0.000 {built-in method builtins.isinstance}
        1    0.300    0.300   25.998   25.998 fastparquet.py:158(read_metadata)
  2893155    0.231    0.000    0.231    0.000 {method 'join' of 'str' objects}
     3360    0.180    0.000   25.678    0.008 api.py:171(statistics)
    13480    0.104    0.000    0.160    0.000 api.py:166(<listcomp>)
     6720    0.052    0.000    0.083    0.000 converted_types.py:86(convert)
6812/6810    0.038    0.000    0.038    0.000 {built-in method numpy.array}
   285840    0.038    0.000    0.038    0.000 {built-in method builtins.getattr}
    70560    0.030    0.000    0.038    0.000 schema.py:97(schema_element)
    13480    0.020    0.000    0.182    0.000 api.py:163(columns)
284148/283922 0.019    0.000    0.019    0.000 {built-in method builtins.len}

@rjzamora

regarding .compute() time from what I saw on the dask dashboard, it seemed that the most time lost on doing it was about to serialize and deserialize the data.

Ah, that would be another matter, I didn't realise that you were running on distributed.
For de/serialisation, we could do some work to improve the speed, and such code was in the earlier version of the refactor PR. Specifically, the column_chunk.metadata.encoding_stats are never used anywhere, and all the column_chunk.meta_data.statistics have already been used, and so we won't need them again. I would have to check, but they key-values can probably be removed (all the pandas stuff was already parsed).

With the following rather shonky code, I go from
In [4]: df = dd.read_parquet('test_fastparquet/', engine='fastparquet', gather_statistics=False)
In [5]: %timeit len(df)
489 ms ± 152 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

to
In [6]: %timeit len(df)
328 ms ± 97.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

on (local) distributed
--- a/dask/dataframe/io/parquet/fastparquet.py
+++ b/dask/dataframe/io/parquet/fastparquet.py
@@ -292,16 +292,22 @@ class FastParquetEngine(Engine):
         # This is a list of row-group-descriptor dicts, or file-paths
         # if we have a list of files and gather_statistics=False
         if not parts:
-            parts = pf.row_groups
+            partsin = pf.row_groups
+            pf.fmd.key_value_metadata = None
         else:
             pf = None
-        parts = [
-            {
+            partsin = parts
+        parts = []
+        for piece in partsin:
+            if pf is not None:
+                for col in piece.columns:
+                    col.meta_data.statistics = None
+                    col.meta_data.encoding_stats = None
+            part = {
                 "piece": piece,
                 "kwargs": {"pf": pf, "categories": categories_dict or categories},
             }
-            for piece in parts
-        ]
+            parts.append(part)

         return (meta, stats, parts)

@rjzamora , maybe you'd like to tidy that up a bit?

This is already an awesome improvement, but can you test on your side with dask==2.2.0 and distributed==2.2.0 if you have the same benchmarks as I do?. Because the %%timeit benchmarks differences between dask/distributed versions are really weird to me. I suspect that the problem is on the distributed side right now. I'm delving on it right now.

 On a quick check, I find that submitting the work is fast (~<30ms), although surprisingly cloudpickle.dumps is being called both via tokenize and in _graph_to_futures; I would expect only the latter.
Note that when using fastparquet, the first run in a session will involve numba compilation, which may be significant for data of this size.
@mrocklin , would it be appropriate to try to define __dask_tokenize__ for whatever is taking the time (I expect the row_group objects), or to provide tokens explicitly somewhere?

Something like dask_tokenize makes sense to me, but I recommend
asking @jcrist who is really the expert on dask protocols.
On Fri, Aug 2, 2019 at 6:40 AM Martin Durant<EMAIL_ADDRESS>wrote:

Are you able to determine what the time is being spent on? On a quick
check, I find that submitting the work is fast (~<30ms), although
surprisingly cloudpickle.dumps is being called both via tokenize and in
_graph_to_futures; I would expect only the latter.
Note that when using fastparquet, the first run in a session will involve
numba compilation, which may be significant for data of this size.
@mrocklin https://github.com/mrocklin , would it be appropriate to try
to define dask_tokenize for whatever is taking the time (I expect the
row_group objects), or to provide tokens explicitly somewhere?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/5200?email_source=notifications&email_token=AACKZTHQQS4I76H2HA6ILCLQCQ2LVA5CNFSM4IIPTWQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3NYXYQ#issuecomment-517704674,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AACKZTAIUWVTKMIMUL7YXL3QCQ2LVANCNFSM4IIPTWQA
.

You could define __dask_tokenize__, or register something with dask.base.normalize_token.register:
https://docs.dask.org/en/latest/custom-collections.html#deterministic-hashing
I'm not sure where the location of this definition should live, up to you.

or to provide tokens explicitly somewhere

This would be another option. It's nice because you don't have to define for custom objects, but potentially fragile if the important state to be hashed changes.

@martindurant I can do it whenever I have time. Something I did not understand is that if you hot the same benchmarks as I.

 On a quick check, I find that submitting the work is fast (~<30ms), although surprisingly cloudpickle.dumps is being called both via tokenize and in _graph_to_futures; I would expect only the latter.
Note that when using fastparquet, the first run in a session will involve numba compilation, which may be significant for data of this size.
@mrocklin , would it be appropriate to try to define __dask_tokenize__ for whatever is taking the time (I expect the row_group objects), or to provide tokens explicitly somewhere?

if you got the same benchmarks as I did

No, but I skipped snappy, so that will account for at least some differences

@RodrigoNeves95 It looks like this was addressed by #5218. If there's still more to do, please feel free to reopen! Thanks 😄


Alert info: i ball doesn't pass colour contrast test

In the alert info, the i ball doesn't pass colour contrast test. The current contrast ratio is 2.956:1. And the minimum contrast ratio should be 3:1 based on WCAG 2.1 1.4.11 Non-text Contrast. This is an accessibility issu.e
Foreground colour: #D7FAFF
Background colour: #269ABC
Example pages that have the alert info:
https://cenw-wscoe.github.io/sgdc-cdts/docs/index-en.html
https://cenw-wscoe.github.io/sgdc-cdts/docs/contact-en.html

Please refer to https://wet-boew.github.io/GCWeb/components/components-en.html which has the most recent info alert.

Hello, the alert info has been updated. We are just waiting for deployment.

The alert has now been updated and is live.

Issue is fixed and closing the issue.


[deprecation] CLI examples will be removed from the npm-package in the next major version

Please have a look at the successor xst
When that is released the CLI examples will only be available in the repository for documentation purposes - actual examples.

CLI examples are no longer part of the released package since v5.0.0


Confirm flag

Would love to have something like vim's /gc mode in which replace would ask whether to replace each segment and get a y/n answer. I'm game to implement this if it's desirable.

Sorry, I probably won't get the time to implement this.


Add quotes around variable to handle empty value

When $1 is empty, the script prints an error since it expands to if [  = '--tweaks' ].
Adding quotes around the variable prevents this and potential other errors.

Thanks for the pr!


fix: downgrade meta dependency

Fixes #120

Thanks for this, sorry for the inconvenience. 4.4.2 should fix this.

No problem, man! Thanks for this amazing package! Glad I could be of help 😇


nacos registry question

Prerequisites
Please check the FAQ, and search existing issues for similar questions before creating a new issue.YOU MAY DELETE THIS PREREQUISITES SECTION.

[ ] I have checked the FAQ, and issues and found no answer.

when we use the Nacos registry , the gateway data is not displayed in the pinpoint

hi @xiaofeng886521
Pinpoint does not support Nacos plugin.
If you could make a Nacos plugin and contribute to us, it would be very helpful for us.
thanks :)


FR: open 'new related model' popup in Bootstrap modal

Hi!  love DAB, thanks!
(I'm new to Django - hope this makes sense)
When one model requires another to exist in order to create a new record, a green plus icon is shown beside the select box, which opens up the create view for the related model on click.
I'd like to prevent the new window being spawned, and load that form in a bootstrap modal - does this sound doable? any tips on where I might start looking to make this change?

So django upstream changes are not enough, opened a new ticket with a WIP upstream modal implementation https://code.djangoproject.com/ticket/24405

fantastic, thanks :)
fwiw, I'd love to see this as flexible as possible.. I've been avoiding modals in other projects due to the difficulties they've presented on mobile devices, and am starting to prefer accordion style in some contexts, but considering your comment upstream re: 'modern', I'm thinking about how knockout or react or ui-bootstrap, etc might complement admin... ach.. this comment probably belongs upstream, what with this project being bootstrapped, after all.. tx!

https://code.djangoproject.com/ticket/24405 has been fixed upstream
https://github.com/django/django/commit/0decef337f7f00d73b380531ac22cbc843d6e806

@xrmx - many thanks for championing this! The more configurable django admin is, the more I'll be inclined to use it!

is this issue still relevant?


Getting an exception while capturing an existing

How do you use Sentry?
Sentry Saas (sentry.io)
Version
2.13.0
Steps to Reproduce
We are using sentry-sdk 2.13.0 for python. We also alongside use   "structlog-sentry ~= 2.1.0". We are seeing a large number of Sentry errors while capturing the exception stack-trace. It seems like there's an issue with capturing breadcrumbs, in what is being captured in timestamp:
...
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/integrations/wsgi.py", line 106, in __call__
    response = self.app(
               ^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/integrations/flask.py", line 86, in <lambda>
    lambda *a, **kw: old_app(self, *a, **kw),
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/flask/app.py", line 1498, in __call__
    return self.wsgi_app(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/opentelemetry/instrumentation/flask/__init__.py", line 390, in _wrapped_app
    result = wsgi_app(wrapped_app_environ, _start_response)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/werkzeug/middleware/proxy_fix.py", line 183, in __call__
    return self.app(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/flask/app.py", line 1476, in wsgi_app
    response = self.handle_exception(e)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/flask/app.py", line 803, in handle_exception
    got_request_exception.send(self, _async_wrapper=self.ensure_sync, exception=e)
  File "/usr/local/lib/python3.12/site-packages/blinker/base.py", line 279, in send
    result = receiver(sender, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/utils.py", line 1730, in runner
    return sentry_patched_function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/integrations/flask.py", line 207, in _capture_exception
    sentry_sdk.capture_event(event, hint=hint)
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/api.py", line 162, in capture_event
    return get_current_scope().capture_event(event, hint, scope=scope, **scope_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/scope.py", line 1139, in capture_event
    event_id = self.get_client().capture_event(event=event, hint=hint, scope=scope)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/client.py", line 745, in capture_event
    event_opt = self._prepare_event(event, hint, scope)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/client.py", line 453, in _prepare_event
    event_ = scope.apply_to_event(event, hint, self.options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/scope.py", line 149, in wrapper
    return fn(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/scope.py", line 1452, in apply_to_event
    self._apply_breadcrumbs_to_event(event, hint, options)
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/scope.py", line 1309, in _apply_breadcrumbs_to_event
    event["breadcrumbs"]["values"].sort(key=lambda crumb: crumb["timestamp"])
TypeError: '<' not supported between instances of 'str' and 'datetime.datetime'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/gunicorn/workers/sync.py", line 134, in handle
    self.handle_request(listener, req, client, addr)
  File "/usr/local/lib/python3.12/site-packages/gunicorn/workers/sync.py", line 177, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/integrations/flask.py", line 89, in sentry_patched_wsgi_app
    return middleware(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/integrations/wsgi.py", line 113, in __call__
    reraise(*_capture_exception())
             ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/integrations/wsgi.py", line 195, in _capture_exception
    sentry_sdk.capture_event(event, hint=hint)
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/api.py", line 162, in capture_event
    return get_current_scope().capture_event(event, hint, scope=scope, **scope_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/scope.py", line 1139, in capture_event
    event_id = self.get_client().capture_event(event=event, hint=hint, scope=scope)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/client.py", line 745, in capture_event
    event_opt = self._prepare_event(event, hint, scope)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/client.py", line 453, in _prepare_event
    event_ = scope.apply_to_event(event, hint, self.options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/scope.py", line 149, in wrapper
    return fn(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/scope.py", line 1452, in apply_to_event
    self._apply_breadcrumbs_to_event(event, hint, options)
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/scope.py", line 1309, in _apply_breadcrumbs_to_event
    event["breadcrumbs"]["values"].sort(key=lambda crumb: crumb["timestamp"])
TypeError: '<' not supported between instances of 'str' and 'datetime.datetime'

Expected Result
No error is thrown while capturing the sentry exception
Actual Result
When this issue happens, the error is not captured in Sentry dashboard, so we are dropping events.

Also seeing this, same versions of sentry and sentry-struct-logs, and error 🙈

@jainankit or @jonas-debeuk, could one of you please share a minimal reproduction? I am not very familiar with structlog or the structlog-sentry integration, since it is not an official Sentry integration.

Hey folks, the breadcrumb sorting code that the error originates from was added recently. If you revert to SDK version 2.10.0, you shouldn't see this anymore.
FWIW the structlog-sentry issue is here: https://github.com/kiwicom/structlog-sentry/issues/159 But we should patch this in the SDK as well. I'll look into it.


Forward and backward button indicates "10 seconds" but jumps 30 seconds

The forward and backward button has a "10" [seconds] text on the visual button. But by clicking it jumps 30 seconds. We should adjust the jumping time to 10 seconds. Tooltip indicates also 30 seconds and must be adjusted to 10:

Just as reference: YouTube has also a 10 second leap for the shortcut "l" / "j" and aditional 5 seconds for

l (10s) / cursor right (5s)
j (10s) /  cursor left (5s)

Remark @naweidner: We should align the leap to the same time in OC Studio if there it is not 10 seconds. OC Editor has currently a jumping time of 1 second. But due to a cutting tool this makes sense and should not be changed to a lager time.

Fixed in commit f152cb56fe4206c6c3ec7e4c72cc0f0f1ca11d23
Test it in: https://polimediaupv.github.io/paella-ethz/?id=belmar-multiresolution-remote

Looks good to me.

Thx for fixing!


add tracee from AquaSecurity

Pre-submission checklist:
Please check each of these after submitting your pull request:

[x] Are you only including a repo_url if your project is 100% open source? If so, you need to pick the single best GitHub repository for your project, not a GitHub organization.
[x] Is your project closed source or, if it is open source, does your project have at least 250 GitHub stars?
[x] Have you picked the single best (existing) category for your project?
[x] Does it follow the other guidelines from the new entries section?
[x] Have you included a URL for your SVG or added it to hosted_logos and referenced it there?
[x] Does your logo clearly state the name of the project/product and follow the other logo guidelines?
[x] Does your project/product name match the text on the logo?
[x] Have you verified that the Crunchbase data for your organization is correct (including headquarters and LinkedIn)?
[x] ~10 minutes after opening the pull request, the CNCF-Bot will post the URL for your staging server. Have you confirmed that it looks good to you and then added a comment to the PR saying "LGTM"?

PTAL @tracymiranda @itaysk

Thanks for submitting this. Looks like the signed with category is full! I'll have to extend the landscape then we should be good!

I applied changes accordingly PTAL @itaysk

yeah I saw. looks great Thanks!!

Apologies for the delay but I've now expanded the signed with box. That means if you sync your changes with the latest then it should hopefully build and we can get this merged!

cool!

everything seems ready for merge.

LGTM


Added support for Attribute Bag

Can pass in instance of AttributeBag to SessionProvider to add support
for session attribute storage

What functionality does this provide? Please include unit tests for new features.
I haven't tested this yet, but are you sure you want a single AttributeBag shared between multiple different sessions?

It allows applications that make use of storing against the $_SESSION super-global to transition to using SessionProvider without the risk of breaking other applications making use of the $_SESSION global. I will take your advice and write a unit test for this.
I see what you mean, I will change the code to initialise the Attribute Bag within the SessionProvider; clearing the attribute bag when the SessionProvider is closed, which is the expected behaviour.

I am closing this because it is 3+ years old.


Extract validation logic from repositories

Depends on #82
This PR aims to remove the validation internals from repositories. To achieve that, I've done several things:

I created a generic IValidator<T> interface to define a common contract to all validators.
I moved all the validation logic to Validator classes, which will be responsible of registering Schemas and managing ValidationFunctions.
Those Validators will also throw validation errors.
I adapted testing classes to this new flow.

Note: ExchangeValidator is the most interesting because it handles several schemas, so we're hiding from the caller (repository) all the logic regarding linking types and schemas. Thats the reason I refactored a bit the interfaces into the Exchange domain and I created the ExchangeResult hierarchy and the type guards, to get advantage of Typescript capabilities. I think this design is pretty extensible to multiple schemas and type, so just one Validator pero domain would suffice.

Coverage increased (+0.9%) to 89.641% when pulling 6ad5d7498277418dfac8872214409a5cf9fc1997 on 72-centralise-datasource-data-validations-logic into c874b4c739342a893d61f5ea13c8153b92e46afc on main.


Use {{next_version}} in default for pro_release_commit_message

The default value for pro_release_commit_message was using {{version}} instead of {{next_version}}.

Looking into this to confirm what the behavior should be, it looks like this commit introduced the split between {{version}} and {{next_version}} but the default never got updated.
I was surprised we didn't catch this with our own history, so I assumed something must be off but instead our history does read in a weird way

This seems to be a regression issue introduced in our refactoring.


Flattened Collect

annotatedCollect has been changed to flattenedCollect which implies that SuccessfulQueryAnnotatedResult server answer disappears.
annotatedCollect result now always contains a fully flattened schema where
each column name is also fully qualified.

+1


Restored Killed console message

Added Started console message
Captured possible exception from node process

Hi @rjperes, I'm your friendly neighborhood .NET Foundation Pull Request Bot (You can call me DNFBOT). Thanks for your contribution!

This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. Real humans will now evaluate your PR.

TTYL, DNFBOT;

Thanks! I've merged some of the changes here (the better logging of Node startup failures, and the whitespace fixes). I didn't merge the console messages that were being issued on successful Node process startup or shutdown, since we don't want to produce extra log messages if everything is OK.


install PixelKit error

I'd love PixelKit,but,I find that ,Unable to find a pod with name, author, summary, or description matching PixelKit

Hi, the Pod can be found here:
https://cocoapods.org/pods/PixelKit

mabye some error for pods

will be success,As follows:
target 'Hello Pixels iOS' do
  use_frameworks!
  platform :ios, '11.0'

  pod 'PixelKit',:path =>'./PixelKit.podspec'

end

target 'Hello Pixels macOS' do
  use_frameworks!
  platform :osx, '10.13'

  pod 'PixelKit',:path =>'./PixelKit.podspec'

end


Problems installing exgboost in livebook

Hi,
I've been using livebook for a while and I'm trying to run the Iris classification example  at:
https://hexdocs.pm/exgboost/0.5.0/iris_classification.html
I write this in the setup cell (just like in the example):
Mix.install([
{:exgboost, "~> 0.4"},
{:nx, "~> 0.5"},
{:scidata, "~> 0.1"},
{:scholar, "~> 0.1"}
])
Unfortunately, I get this warning:
23:25:17.977 [warning] The on_load function for module Elixir.EXGBoost.NIF returned:
{:error,
{:load_failed,
~c"Failed to load NIF library: 'dlopen(/Users/hiram/Library/Caches/mix/installs/elixir-1.15.7-erts-14.1.1/102c8b1d69b78ce545ebc04ce45ad196/_build/dev/lib/exgboost/priv/libexgboost.so, 0x0002): Library not loaded: /opt/homebrew/opt/libomp/lib/libomp.dylib\n  Referenced from: <7834DF63-8443-3E17-8C61-5AD8FC9345BC> /Users/hiram/Library/Caches/mix/installs/elixir-1.15.7-erts-14.1.1/102c8b1d69b78ce545ebc04ce45ad196/_build/dev/lib/exgboost/priv/lib/libxgboost.dylib\n  Reason: tried: '/opt/homebrew/opt/libomp/lib/libomp.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/opt/libomp/lib/libomp.dylib' (no such file), '/opt/homebrew/opt/libomp/lib/libomp.dylib' (no such file)'"}}
23:25:17.977 [notice] Application exgboost exited: exited in: EXGBoost.Application.start(:normal, [])
** (EXIT) an exception was raised:
** (UndefinedFunctionError) function EXGBoost.NIF.set_global_config/1 is undefined (module EXGBoost.NIF is not available)
(exgboost 0.5.0) EXGBoost.NIF.set_global_config("{"verbosity":0,"use_rmm":false}")
(exgboost 0.5.0) lib/exgboost.ex:184: EXGBoost.set_config/1
(exgboost 0.5.0) lib/exgboost/application.ex:6: EXGBoost.Application.start/2
(kernel 9.1) application_master.erl:293: :application_master.start_it_old/4
I've checked and the "libexgboost.so" file actually is found in "exgboost/priv".
When i run the cell for training: "... EXGBoost.train( ..." I get the error:
** (UndefinedFunctionError) function EXGBoost.NIF.get_binary_address/1 is undefined (module EXGBoost.NIF is not available)
(exgboost 0.5.0) EXGBoost.NIF.get_binary_address(<<102, 102, 246, 64, 0, 0, 64, 64, 51, 51, 195, 64, 51, 51, 19, 64, 102, 102, 150, 64, 205, 204, 76, 64, 205, 204, 204, 63, 205, 204, 76, 62, 205, 204, 204, 64, 205, 204, 44, 64, 154, 153, 169, 64, 51, 51, 243, 63, 205, 204, ...>>)
(exgboost 0.5.0) lib/exgboost/array_interface.ex:129: EXGBoost.ArrayInterface.from_tensor/1
(exgboost 0.5.0) lib/exgboost/dmatrix.ex:309: EXGBoost.DMatrix.from_tensor/2
(exgboost 0.5.0) lib/exgboost.ex:249: EXGBoost.train/3
#cell:fhe4zu7q6payxerwyhtao572majr44yt:2: (file)
BTW, version "~>0.4" is used in the example, but the documentation is for version "0.5.0", is it supposed to be like that?
I've also tried using {:exgboost, "~> 0.5"} but got the same error.
Do I need to install something outside livebook, in the host computer?
I'm running livebook in a macbook pro with Sonoma, maybe that is the problem.
Thanks for your help in advance!
Best regards,
/Hiram Galicia

For MacOS you need to intall libomp which you can do with brew install libomp. Depending on how your livebook is set up it could be easier to compile and start an iex session outside of live book and connect it it separately rather than using a standalone livebook instance.
Try installing libomp and let me know if that resolves it for you

Also, regarding the livebook specifying the old version, that's an error. It just got missed in the update.

Hi Andrés,
Yes, it worked like a charm after running "brew install libomp". I just reopened the notebook in livebook and worked.
I saw this in the requirements but misunderstood it. Thought it was only if I'd be contributing to the library.
By the way, do you know if Apple Metal will be supported in the future? I see CUDA is in the roadmap.
Many thanks again for the help and great job with the library.
Best regards,
/Hiram Galicia

Glad it worked! I'll go ahead and close the issue.
I do think there are plans to add a metal backend. I believe it's actively being developed.
In the meantime, you can use https://github.com/acalejos/mockingjay to compile your trained EXGBoost.Booster's to Nx code and then leverage all of the benefits from that ecosystem.
You can see an example of how to do that at https://www.thestackcanary.com/serving-spam-detection-with-xgboost-and-elixir/

Thanks Andrés!
That sounds like good news with metal backend!
Absolutely, I'll take a look at the post.
Thanks for your great job!
Best regards,
/Hiram Galicia


Sudden inexplicable under estimation of coverage for multimethod.interface [should be 100%, measured 68%]

Codecov reports a quite low coverage of the interface class, specifically complaining about all error handling statements not being reached:

But, from the CI log we have:

Which pretty much shows that those lines are covered, specifically in: https://github.com/bellomia/matlab-multiple-dispatch/blob/b8ba9dd7714fbcc98f45e17d293d1ed05873e38c/.test/test_error_handling.m#L8-L14
So what's the matter with those lines? Why does Codecov miss them in the report?
Also note that in just one of the previous commits (bc586fc723981e492ea0cb965695d5454281767a) the report measured a 100% coverage in interface.m, which should be the correct assessment. So why it stopped working?

The line coverage report for commit https://github.com/bellomia/matlab-multiple-dispatch/commit/bc586fc723981e492ea0cb965695d5454281767a, where all relevant lines where correctly detected.

Apparently has solved itself. I guess it was a bug in Codecov API. Let's close the issue.


correct the behavior of Vector#reorder! method

Vector#reorder! method overrode @data with Array.
@data must be instance of Accessors::ArrayWrapper.
Same for #276 , only branch name differs.

Also make sure that the Travis CI build passes.

@v0dro: On Travis CI, some rubocop checks have failed.
However, in my local environment, no rubocop check fails.
In fact, these checks didn't failed in the previous Travis CI check, and I haven't modify that file since then.
Do you have any ideas why these rubocop checks fail?

Try bundle update.  It could be that your gems are not up to date.
On Nov 3, 2016 6:02 PM, "Yusuke Sangenya"<EMAIL_ADDRESS>wrote:

@v0dro https://github.com/v0dro: On Travis CI, some rubocop checks have
failed.
However, in my local environment, no rubocop check fails.
In fact, these checks didn't failed in the previous Travis CI check, and I
haven't modify that file since then.
Do you have any ideas why these rubocop checks fail?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/v0dro/daru/pull/277#issuecomment-258129938, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AHUmVHIOvF_hurrD2lpLg35O5yN53n8cks5q6dRvgaJpZM4KiGPv
.

@lokeshh Thanks! bundle update updated my rubocop, and same rubocop checks failed.
@v0dro Should I include commits that fixes rubocop checks, or make another Pull Request that fixes rubocop check?


Disable Dispatch logs using debug packages

The current way of disabling logs is not standard.
npm libraries usually use debug to log things so when everybody wants to disable them he can do that without altering the package itself.
Can you use debug for logging so we can disable logs?

Would the config object we send to nuclearJS do the job?
From the API:
Nuclear.Reactor
var reactor = new Nuclear.Reactor(config)
// or
var reactor = Nuclear.Reactor(config)

Configuration Options
config.debug Boolean - if true it will log the entire app state for every dispatch.

As I said it's not a standard. npm packages mostly use debug for this purpose.

@HosseinAgha
A few thoughts:
Not everyone will use nuclearJS in a build (eg npm/node), we show some folks shipping a pre-built bundle and just including a minified version in their website. In that case, debugging via the console is (almost) the only way. So whichever approach we come to, it should be universal.
You are right to bring this up and it looks like there are a few approaches to solving this: https://www.codementor.io/mattgoldspink/nodejs-best-practices-du1086jja#11-use-a-logging-library-to-increase-errors-visibility
Would you be willing to submit a revision for code review? Right now, this is a low priority at Optimizely, but we'd be willing to consider a revision that supports this.


i just like this

i just like this

This is my true love!!!!

Me too. Let get a group marriage! :kissing_cat:


Update RFP parts to RAC 3.0

I've noticed that application is built upon the ReactiveCocoa 2.0 core concepts, I mean RACSignal and RACComands.
Does it make sense to update them to the "more Swifty" RAC 3.0 Signal/SignalProducer/Action abstractions?

Hey! That's correct – we began building this application before RAC 3 had even reached an alpha!
ReactiveCocoa 3 supports Swift 1.2 only, and the 4.0 release for Swift 2 is still in alpha. We're planning on making the switch within the next few months. Maybe to ReactiveCocoa 4, maybe to RxSwift. When we do, I'll reference this issue on the pull request – if you have any feedback, we'd love to hear it!

Thx for quick response, @ashfurrow. I see your point. Could I help with anything?
Also I'm very appreciate this open-sourced application that demonstrates the concepts of RFP in iOS. There are not to much real-world practices in Swift related to RFP.

Hey, thanks @skyylex – it's ok, it' gonna be a big project :)
Happy to open our source code :cake:

@ashfurrow Could you please share you concerns about upgrading to RAC 4.0? Why moving to RxSwift?
After almost 2 years using RAC 3, I found myself learning version 4 concepts from scratch. They don't look simpler and routine tasks have to be solved in rather different way. Maybe it's Swift style + new names for old things.
At the moment I'm hesitating to switch to RXSwift  or continue with RAC 4.0.

Most of our discussion around RAC was summed up in this blog post

Yeah, we're not sure what we're going to do. I can't really comment on the motivations for the changes to RAC 4. Right now, we're leaning towards RxSwift at the moment, just to try something new. Additionally, RxSwift adheres to the common APIs at ReactiveX.io, so the domain knowledge is transferrable to other platforms, and also very stable and predictable.

Hey there! Just following up on this. This week, we completed our transition to RxSwift. I wrote up a summary of the process on Artsy's blog. Let me know if I can clarify anything!


Do not use this

Author has lost interest (see issue list).
Probably does not support the SEPA version your bank wants from you.

I have not entirely lost interest, I still intend to support newer SEPA versions, I just have a lack of time on my hands.
The annoying thing is that codebase is a little bit of mess and hard to extend. I did not write this code, I mostly took existing free code and packaged and tested it properly. :\

Sorry for venting my frustration after my bank stopped supporting pain.008.002.02 without prior announcement. It turned out they still support pain.008.003.02 - which is supported by sepaxml. Unfortunately, my bank will phase out pain.008.003.02 next November - so you have one year :-)
Whoever is behind those standards is not of this world but on the spot with naming.

I had the same issue with my bank yesterday and went the quick route of just changing the version number in the file by hand, yes… Glad you found a workaround for now ;)


Create proper type definitions and object (JSON?) schema for JS code

This should be in line with type definitions in the k6 - DefinitelyTyped project, so the editors and IDEs could correctly display types, docstrings and related information.

Related:

#149


Replace text confirmation

When running the command,
:%s/replace/text/gc
the usual y/n/q etc works -- but there is no visual highlight of the current word, or a list of options available.  If you hit 'y' on a word it doesn't change immediately, but is updated after pressing 'q' or after you've answered all the options.

Try enabling highlights in the actualvim user settings. Thanks for the repro, I'll look into this.

"doesn't change immediately" should be fixed in master + neovim 0.3

nvm, vim is blocking the UI.
it throws us into the r? mode, and blocking = True... so I can't ask for the cursor position

@justinmk to fix this and https://github.com/lunixbochs/ActualVim/issues/119 I need a cursor position event that fires even when the nvim api is fully blocking

There should be a redraw event already. But we need to externalize dialogs, too.

redraw doesn't tell me the viewport and selection region, so there's no way for me to follow modal searches properly - each time you say y/n on a replacement search, I need to be get the new cursor range(s) and viewport so I can update the host editor


Remove the "open a rust-lang/rust issue" link

The "Open an issue on the Rust GitHub repository" link on the Share window is frequently used to open spam issues that we then have to close manually. Since opening an issue manually just involves copying code and output by hand, I am not sure it is pulling its weight. Perhaps it would be better to remove it?

Can you link me to some of the spam issues?

Can you link me to some of the spam issues?

The recent ones were:
https://github.com/rust-lang/rust/issues/81009
https://github.com/rust-lang/rust/issues/81010
https://github.com/rust-lang/rust/issues/81014
https://github.com/rust-lang/rust/issues/81016
https://github.com/rust-lang/rust/issues/81019
https://github.com/rust-lang/rust/issues/81020
Some older ones:
https://github.com/rust-lang/rust/issues/80773
https://github.com/rust-lang/rust/issues/76125
https://github.com/rust-lang/rust/issues/72418
https://github.com/rust-lang/rust/issues/67275
https://github.com/rust-lang/rust/issues/66992
https://github.com/rust-lang/rust/issues/65814
https://github.com/rust-lang/rust/issues/58076

The recent ones were:
https://github.com/rust-lang/rust/issues/81009
https://github.com/rust-lang/rust/issues/81010
https://github.com/rust-lang/rust/issues/81014
https://github.com/rust-lang/rust/issues/81016
https://github.com/rust-lang/rust/issues/81019
https://github.com/rust-lang/rust/issues/81020
Some older ones:
https://github.com/rust-lang/rust/issues/80773
https://github.com/rust-lang/rust/issues/76125
https://github.com/rust-lang/rust/issues/72418
https://github.com/rust-lang/rust/issues/67275
https://github.com/rust-lang/rust/issues/66992
https://github.com/rust-lang/rust/issues/65814
https://github.com/rust-lang/rust/issues/58076

Some more spam issues caused by this:
https://github.com/rust-lang/rust/issues/83617
https://github.com/rust-lang/rust/issues/83620

I understand that this was used to create spam issues, but I have found this feature very useful and use it a lot. The primary way I use it is to quickly be able to copy

code,
playground link, and
output

all at once. I find it tedious to do it manually. Is there some kind of replacement feature that could be added where it would let me copy what used to go into the GitHub issue body? E.g., it would output this to a textbox that could be copied:
Code ([playground](...)):

```rust
...
```

Output:

```
...
```

I would find that very useful.

There's a few possibilities, sure. Off the top of my head:

We could remove the <a> tag and only have the copy icon. That way you'd have to deliberately paste it into your address bar, but otherwise it'd be the same as what previously existed.
Use a collapsed/expandable textbox like you mentioned.

@jonas-schievink do you have any intuition about why GitHub issues were comparatively spammed? I've not heard anyone complain about the same misuse on the user's forum, for example.

I've seen one URLO spam post that clearly came from the playground: https://users.rust-lang.org/t/marvinrivera1234/58204
I'm not aware of others, but I'm not too active on URLO. I suspect GitHub gets more spam because more people have an account there (and it keeps you logged in permanently).
I'd be fine with outputting copy-pastable markdown for the forum and GitHub, that shouldn't cause any spam issues as long as you omit the link. Having a non-clickable, but copyable link might still cause spam, so I'd rather avoid that.

Yeah, the copy-pastable textbox would actually work even better than the GitHub issue link for me because I often copy-paste from the issue body into Zulip or an issue or PR comment instead of opening the issue.

I opened an issue: #708


Ffmpeg exited with error code: null

Hi guys, I'm having a problem with getPngStream, which causes the process to exit with: Ffmpeg exited with error code: null.
Does anyone have a clue as to why this is happening?
I'm using ffmpeg-2.3.2.

Hi @marcooliveira, I'm having the same problem, did you manage to correct it?
Appreciate your help


how do you support "jagged" shapes with td3?

our custom environment has a differently shaped observation/action space each episode. We want to use td3 algorithm and wonder how do you change the replay buffer and training to support jagged shapes?
this might be handy for nlp or molecules/graphs where the number of variables is not fixed

Hello @bionicles! Sorry for the late response.
It's not obvious to me how to best deal with obs / action spaces that change shape during each episode. I think there are some exotic kinds of neural networks that can handle variable input shapes, but I'm not familiar with that literature or how to implement them. Graph Neural Networks might be what you are looking for.
Another thing you could try is using RNNs. Eg an RNN processes the input stream and returns a fixed-size output, or an RNN takes a fixed-size input and then autoregressively generates a sequence of output components.


Functionality to edit the OAS specification document through forms

Problem
Currently, the OAS files added to create the Rest APIs or the OAS files generated from the API definition generally lack descriptions about the fields. Since we have a generating documentation from the OAS specification feature which extracts the information from the OAS specification files to display in the documentation section, we need a user-friendly method to add descriptive fields.
Solution
Instead of letting the api creators and publishers add or edit the descriptive fields of the OAS documentation, have an interactive form through which they can edit the swagger files which does not change the functionality of the api.
Affected Component
APIM
Version
No response
Implementation
No response
Related Issues
No response
Suggested Labels
No response

Progress update

Initial design wireframes were produced and discussed over the initial discussion

Rest apis created in apim pack
1. Created from scratch 		        : oas 3.0
2. Created from url from oas 2.0 	: oas 2.0
3. Created from url from oas 3.0 	: oas 3.0
4. Created from file from oas 2.0	: oas 2.0
5. Created from file from oas 3.0	: oas 3.0

Options to deal with oas 2.0 apis

Convert to oas 3.0 using open api converter
Swagger2openapi: https://www.npmjs.com/package/swagger2openapi
Swaggerconverter: https://converter.swagger.io/#/
Let the user to edit the 2.0 swagger : swaggerv2

Design review was scheduled on 10.04.2023 to finalise the design and to discuss the how to handle rest apis created with oas 2.0

Design review held on 10.04.2023
There exists several concerns about this feature

There are already places where you can edit the swagger / edit the api artifacts details.
Api artifacts and swagger are not synced

Details that were finalized

UI Flow 2 for edit forms
Have descriptions for every fields
Don't convert oas 2.0 to oas 3.0. Implement a different form.

It was decided to schedule another design review after researching more about the following details.

Vendor specific extensions allowed by stoplight
Whether oas 2.0 and oas 3.0 specifications can be edited through a single form

Design review held on 12.04.2023
The following were discussed.

Vendor specific extensions are not supported by stoplight elements
Oas 2.0 and oas 3.0 will be edited through separate forms, since having the same form for both would require converting oas 2.0 to oas 3.0, editing through the form and saving it back in oas 2.0. Since several fields available in oas 3.0 are not available in oas 2.0 this is not possible.

A suggestion was given by Chanaka Jayasena to edit the generated documentation as it is by manipulating the CSS of the generated documentation. A PoC will be made and presented in the next meeting.

The PoC where the fields to be edited are edited on the generated document itself is infeasible technically, since individual elements do not have ids to identify themselves. Therefore a different approach to give the same look and feel was proposed.
In this approach, the look and feel of the generated document will be matched by manually generating the generated documentation. The left menu will be as the same as the generated document and the content section will have the same view of the generated document but will be editable. The PoC for this approach is scheduled on 24.04.2023.

PoC review held on 24.04.2023
The PoC for the current approach was approved in this meeting.
It was decided to park the feature until the next allocation.


Remove: vim.vim version 8.2.4501

## .\Tools\YamlCreate.ps1 -PackageIdentifier vim.vim -Mode 5 -SkipPRCheck -PackageVersion 8.2.4501

[X] Have you signed the Contributor License Agreement?
[X] Have you checked that there aren't other open pull requests for the same manifest update/change?

[X] Does your manifest conform to the 1.0 schema?

Microsoft Reviewers: Open in CodeFlow

/azp run


Delete RenderQScriptDSL

I was going to fix (or delete) RenderQScriptDSLSpec but then figured out that RenderQScriptDSL is no longer in use (and hasn't been since we deleted mongo and the old file system impl).

And fwiw, RenderQScriptDSLSpec has been silently not running for at least a few days, if not longer.


only support for ubuntu, not for centos

only support for ubuntu, not for centos

apt.dockerproject.org is too slow
E: Failed to fetch https://apt.dockerproject.org/repo/pool/main/d/docker-engine/docker-engine_17.05.0~ce-0~ubuntu-xenial_amd64.deb  Operation too slow. Less than 10 bytes/sec transferred the last 120 seconds


EPICS motor records have a .DISA field. Check it before trying a move?

The EPICS motor record, like all EPICS records, has a .DISA field that says if the record has been disabled (which means that it will not be processed).  When disabled, the motor will not accept attempts to change the target (.VAL) value.  It would be nice if that could be checked before trying to move the motor.  Then, a clear, concise message could be delivered about why the move was not allowed.
Here's an example of what happens now for a motor that has been disabled:
In [24]: movr detu.x 2
CA.Client.Exception...............................................
    Warning: "Channel write request failed"
    Context: "op=1, channel=8idi:m90.VAL, type=DBR_DOUBLE, count=1, ctx="8idi:m90""
    Source File: ../oldChannelNotify.cpp line 158
    Current Time: Mon Aug 17 2020 16:03:25.038080194
..................................................................
  A 'deferred pause' has been requested. The RunEngine will pause at the next checkpoint. To pause immediately, hit Ctrl+C again in the next 10 seconds.
Deferred pause acknowledged. Continuing to checkpoint.
^Ctrying a second time
Pausing...
The RunEngine invoked by magics cannot be resumed.
Aborting...
CA.Client.Exception...............................................
    Warning: "Channel write request failed"
    Context: "op=1, channel=8idi:m90.STOP, type=DBR_SHORT, count=1, ctx="8idi:m90""
    Source File: ../oldChannelNotify.cpp line 158
    Current Time: Mon Aug 17 2020 16:03:42.591596768
..................................................................
Aborting: running cleanup and marking exit_status as 'abort'...
Run aborted
Traceback (most recent call last):
  File "/home/beams/8IDIUSER/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py", line 1341, in _run
    msg = self._plan_stack[-1].throw(
  File "/home/beams/8IDIUSER/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py", line 299, in mvr
    return (yield from inner_mvr())
  File "/home/beams/8IDIUSER/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py", line 1066, in dec_inner
    return (yield from plan)
  File "/home/beams/8IDIUSER/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py", line 1075, in relative_set_wrapper
    return (yield from plan)
  File "/home/beams/8IDIUSER/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py", line 245, in msg_mutator
    ret = yield msg
bluesky.utils.RequestAbort
CA.Client.Exception...............................................
    Warning: "Channel write request failed"
    Context: "op=1, channel=8idi:m90.STOP, type=DBR_SHORT, count=1, ctx="8idi:m90""
    Source File: ../oldChannelNotify.cpp line 158
    Current Time: Mon Aug 17 2020 16:03:42.626892969
..................................................................

This is easier to implement for EpicsMotor since it is a Device.  Could this be implemented for EpicsSignal as well?

It will also need .DISV to compare with .DISA and obtain the result if the record is disabled or not. 👍


New Linearization and Mode-Shape Capabilities (and more)

Complete this sentence
THIS PULL REQUEST IS READY TO MERGE
Summary
This code is the result of the Envision-NREL 2019 collaboration on finding a trim solution for linearization and visualization of mode shapes.
As warned, this is a BIG pull request. In an ideal world, this would be broken up into several different pull requests. However, that was not possible given the current states of our respective code bases and time considerations.
I have not provided much in terms of updated documentation in the pull request because the original documentation is not in a place I can easily modify it. I can help update the documentation after the original documentation has been merged in.
Feature or improvement description

Trim Solution

I added new capability to find a periodic steady-state trim solution where linearization will be performed, based on this plan from NREL: https://github.com/ebranlard/temp-lin. OpenFAST will calculate the times to perform linearization instead of requiring the user to specify them.

Mode-Shape Visualization

I added capability to visualize mode shapes using VTK output files, based on an algorithm from @jjonkman
Generating these figures is a multi-step process that requires restarting OpenFAST after performing MBC3 analysis. I added the -VTKLIN command-line switch to indicate that OpenFAST is being restarted with the visualization option.
I created a Matlab script to describe the process here: https://github.com/bjonkman/r-test/blob/pullrequest/bjonkman-linear/glue-codes/openfast/5MW_Land_ModeShapes/AnalyzeModeShapes.m
You can also view some documentation about viewing mode shapes here: https://github.com/OpenFAST/r-test/blob/pullrequest/bjonkman-linear/glue-codes/openfast/5MW_Land_ModeShapes/vtk-visualization.md

Nodal Outputs on Blades

We added (optional) additional sections at the end of ElastoDyn, BeamDyn, and AeroDyn input files to request channel outputs at every node on a blade.
The structural codes use naming conventions B#N###channel, and AeroDyn's nodal output naming convention is AB#N###channel. This allows the user to distinguish between aero and structural channels that may have the same name.
The available channel names are found in https://github.com/bjonkman/openfast/blob/f/Linear/docs/OtherSupporting/OutListParameters.xlsx

Longer channel names

I set the max channel length to allow up to 20 characters in a channel name in OpenFAST. This number can be easily changed by modifying the value of  ChanLen in NWTC_Base.f90 and rebuilding.
The compressed binary file format will now contain the length of the channel names.
FAST.Farm should be checked for compatibility because I removed the now-unnecessary ChanLenFF variable and corresponding type

Registry Improvements

The Registry now allows for interpolation/extrapolation of periodic signals. Angles that have 2pi periodicity are treated as a special case to avoid issues where interpolating between 0 and 360 degrees would give an answer 180 degrees off.
The pack and unpack routines have been updated to use less stack space. This was necessary since the mode-shape visualization uses these routines.

Performance improvements

WriteOutput variables in AeroDyn and BeamDyn are not computed if they will not be used at that step.

Bug fixes

The AeroDyn driver OutFileRoot is now specified relative to the location of the driver input file instead of the current working directory.
Fixed a problem where AeroDyn could potentially think it was using a different DT than what it reported to the glue code.

Related issue, if one exists

https://github.com/OpenFAST/openfast/issues/319
https://github.com/OpenFAST/openfast/issues/304
https://github.com/OpenFAST/openfast/issues/99

Impacted areas of the software

ElastoDyn

New nodal outputs on blades
Linearization

AeroDyn

New nodal outputs on blades
Linearization
New method of calculating the D matrix in linearization (will not produce error about being non-differentiable between AeroDyn solution regions). Warning: There may be an issue with FrozenWake with this modification. It is not apparent in the cases I ran, but the logic seems a little off to me. This has been fixed in a future pull request for linearized aerodynamics from Envision.
Performance improvements: do not calculate WriteOutputs unless they are required

BeamDyn

New nodal outputs
Linearization
Performance improvements: do not calculate WriteOutputs unless they are required

ServoDyn

Linearization
Bladed DLL interface code: partial implementation of API from Bladed 4.9
Can be built with preprocessor definition LOAD_DLL_TWICE_FOR_LOGGING_CHANNELS to output logging channels from the DLL (including legacy version) in the OpenFAST output file. This, however, requires the DLL to be unloaded and loaded again.
Removed ElecPwr_prev and GenTrq_prev from ServoDyn's inputs. These are now stored in misc vars to send back to the Bladed DLL (and simplifies the glue code)
The DLL controller is now called in UpdateStates (and when it is first called in CalcOutput, if it hasn't already been initialized)
Updates to how the HSS brake is applied from the Bladed-stlye DLL (uses linear ramp)

Glue-code

Linearization
Changes to input file
Changes to linearization output file format

Registry

Pack/Unpack routines
Extrap/Interp routines

FAST Library (Simulink interface)

I increased the number of outputs that are allowed through the interface from 1000 to 4000 (the new nodal outputs gives the ability to generate a lot of data!)

NWTC Library

Modifications to the binary file format
Increased tolerance on "close" node in mesh mapping to 7.5mm
Routines to read real variables now check that they are finite values that are not NaN.
Removed OutParmFFType type (please verify that this is not a problem with FAST.Farm)

Documentation

Updated example input files
Fixed some text about where the static/dynamic switch is located in BeamDyn

Additional supporting information

I added an example of the trim solution and mode-shape visualization in https://github.com/bjonkman/r-test/blob/pullrequest/bjonkman-linear/glue-codes/openfast/5MW_Land_ModeShapes/. This is not very robust, but is a good starting point for NREL (or someone else) to automate the mode-shape visualization with different options.
The matlab-toolbox has been updated in https://github.com/OpenFAST/matlab-toolbox/pull/11 to work with this code base

Test results, if applicable

Input files for the regression tests have been updated here: https://github.com/bjonkman/r-test/tree/pullrequest/bjonkman-linear
I would not expect results to change except for

minor differences in linearization output in models that have enabled AeroDyn
cases that may have different Bladed DLL controller options enabled (e.g. HSS brake), though I don't think any of the NREL regression tests test that feature.

For fun, here are the first two mode-shape plots that I generated with the scripts in the 5MW_Land_ModeShapes regression test directory:
Mode 1 (tower side-side)

Mode 2 (tower fore-aft)

Hi,
I am trying to compile OpenFAST from branch f/Linear in order to try the linearization tool. Nevertheless, y found some problems when compiling it:
(I am using Cygwin 64 bit + CMake since even if I do have Visual Studio, I do not have license for Intel Fortran Compiler and therefore I think that I am not able to compile that way).

I got an error saying that the file src/BeamDyn_BldNdOuts_IO was not found in modules/beamdyn/src/CMakeLists.txt

I was able to solve it by changing the name of the call to: src/BeamDyn_BldNdOuts_IO.f90 (I think this is a bug)

set(BD_SOURCES
src/BeamDyn.f90
src/BeamDyn_IO.f90
src/BeamDyn_BldNdOuts_IO.f90
src/BeamDyn_Subs.f90
src/BeamDyn_Types.f90
)

After doing that, I have tried to compile again, and now I get the following message:

[ 60%] Building Fortran object modules/hydrodyn/CMakeFiles/hydrodyn_driver.dir/src/HydroDyn_DriverCode.f90.o
/cygdrive/d/Users/jmrico/Documents/9131_SEAPOWER/openfastLinearBJ/modules/hydrodyn/src/HydroDyn_DriverCode.f90:543:105:
call RunTimes( StrtTime, REAL(UsrTime1,ReKi), SimStrtTime, REAL(UsrTime2,ReKi), time, UsrTimeDiff )
1
Error: Type mismatch in argument ‘unsum’ at (1); passed REAL(8) to INTEGER(4)
make[2]: *** [modules/hydrodyn/CMakeFiles/hydrodyn_driver.dir/build.make:63: modules/hydrodyn/CMakeFiles/hydrodyn_driver.dir/src/HydroDyn_DriverCode.f90.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:862: modules/hydrodyn/CMakeFiles/hydrodyn_driver.dir/all] Error 2
make: *** [Makefile:130: all] Error 2

I am not sure what this ‘unsum’ makes reference to, but I was not able to find it.
I would really appreciate if someone could help me with this problem.
Thanks in advance,
Jon

@jonmtz, Thanks for reporting those issues. I've pushed some changes that should hopefully fix both problems. Let me know if you find anything else.

Dear @bjonkman , thank you for the quick answer. With the new version I compiled without problems, so seems to be OK.
If I find any problems when performing simulations I will let you know.
Best regards,
Jon

Items to check on this PR:

[ ] remove DBGOUTS from AD15 -- these are superseded by the new outputs
[ ] add documentation on the node based outputs (in user guides for each module)
[ ] check BD changes against the bugfix branch (in progress)
[ ] add documentation on how to use the linearization and mode shape plotting

Notes on regression tests for PR 373 (after updates are merged from my branch to @bjonkman branch):

5MW_OC3Mnpl_DLL_WTurb_WavesIrr: usual numerical noise

5MW_OC3Trpd_DLL_WSt_WavesReg: usual numerical noise

5MW_OC4Jckt_DLL_WTurb_WavesIrr_MGrowth: usual numerical noise

5MW_Land_BD_DLL_WTurb: the timestep changed so comparisons to older baselines don't make sense.  I verified it is still correct.

WP_Stationary_Linear: new test case. No prior baselines

Ideal_Beam_Fixed_Free_Linear: not sure

Ideal_Beam_Free_Free_Linear: not sure

5MW_OC4Semi_Linear: THIS DOES NOT RUN!  FAST_InitializeAll:HydroDyn_Init:WAMIT_Init:SS_Exc_Init:HydroDyn Wave direction does not match the wave excitation wave direction

Just a small note. I think if we want to do linearization at WS=0, InflowWind report an error, so the user has to set CompInflow=0. For VTK surface outputs, we still need AeroDyn, so in that case we need to trust AeroDyn without InflowWind. It seems to work fine.

@andrew-platt, a few comments on your regression test notes:

WP_Stationary_Linear isn't exactly new. :) It's the old "Test14" that was there already in FAST 6. Originally we ran the MBC3 analysis on it and compared the frequencies it reported.

I found that one of the many WaveDir variables wasn't getting set properly in HydroDyn for the 5MW_OC4Semi_Linear case. I'm not sure my fix is the best way to do that, so someone more familiar with all the HydroDyn data flow should maybe review that. Hopefully that fixes the problem you were seeing (the earlier version ran fine on my VS build).

I think it would be helpful to add some Matlab scripts for the linearization cases in r-test so that when the output files don't match exactly, we can still manually check what frequencies and damping are actually predicted from those matrices.

I think I've encountered two small corner cases, more or less related.  If somehow I run openfast from another directory: something like:
openfast .exe ../path/to/Main.fst

Then I can run again for VTKLin
 openfast .exe -VTKLin ../path/to/Main.viz

As long as the viz file inputs have CheckpointRoot and MatlabFileName as follows:
      ../path/to/Main.ModeShapeVTK

I'm wondering if these paths should be relative to the VIZ file instead?
Because if I go to the folder, then
cd ../path/to/
openfast.exe -VTKLin Main.viz

It would fail with checkpoint file not found. (FAST_RestoreForVTKModeShape_Tary:FAST_RestoreFromCheckpoint_Tary:FAST_RestoreFromCheckpoint_T:Open BInpFile)
So I'd have to replace the variables CheckpointRoot and MatlabFileName.
But, if I do replace them, the checkpoint file is opened, but somehow, OpenFAST cannot find the .outb file.
FATAL ERROR when writing binary output file: WrBinFAST:OpenBOutFile:Cannot open file
"../path/to/Main.outb"

Likely because the checkpoint file "remembers" that it was created while called from another directory.
It might make sense to make the Viz inputs relative to the viz file like other input files? As for the checkpoint, that might be related to the Rootname of the simulation needing reinitialization.
There are some design choices there I guess, I just wanted to point that out.

@ebranlard , Yes, this is because we used the checkpoint capability. You are basically restarting the simulation with all the data it had previously stored--including path names, location of DLLs, etc. (and EVERY module has the potential to store some of these variables, so I don't think it's feasible to overwrite the RootName in just one place). I thought I had written some of these limitations of the checkpoint-restart capability in the FAST8 ReadMe, but since the NWTC website is semi-permanently down, I don't have access to that right now. But, basically the main limitation is that you should be restarting it from the same directory as the code was originally run. I think MAP++ might even need to re-read its input file (or maybe I stored that info another way, I don't remember at the moment).
Anyway, I just modified the code so that those two files can have paths specified to the location of the input file where they are specified. Just keep in mind those other checkpoint-restart limitations if you do run it somewhere else.

Great thanks!

Another detail, I think the vtk folder is not created when running with -VTKLin, which would make openfast crash.

@ebranlard , yes, the vtk folder is created on the initialization step. If someone deletes it between initially running the code and restarting with -vtklin, then OpenFAST would likely just not write the vtk files. It does check for an error when writing files (though the errors are not reported back to OpenFAST), so I'd be surprised if it actually "crashed". Did you try this?
I think this is something that should be noted as a limitation of checkpoint-restart capability (not just vtklin). There are MANY things that could go wrong between writing a checkpoint file and restarting if the user changes/deletes files/folders or even decides to use a different version of OpenFAST.

Thank you, I was running from the matlab command prompt, and don't think I saw an error message. But I'll double check.
A final question. I have the feeling the code uses NLinTimes to generate the vtk files. Is it possible to perform visualization, for a case at 0 rpm? I typically use NLinTimes=1 in this case and only one VTK is generated then with -VTKLin. Thank you!!

I'm not sure there will be an error message because the way the existing WriteVTK routine currently works, it ignores the errors and just continues the simulation.
At 0 RPM, the trim-solution plan said to overwrite NLinTimes (or NAzimStep) to be 1, and "the number of rotations is then understood as the number of time steps." You should be able to perform visualization with this setting, but you will have to set the VTKLinTim input setting to be 2 (separate animations for each LinTimes).
Does that answer your question?

Thanks, yes! Having VTKLinTim=2 works with 0 rpm. I'll try to add some checks for that in the matlab scripts.
And I confirm that openfast exits "normally" even when the vtk folder is not here. Maybe we could add a small check when the  VTKLin flag is there? But I understand it's just one of the cornercases related to the checkpoint.

@ebranlard I created an issue  to have someone fix that issue in the future. It feels like a lot of code would need to be changed, and I don't have the time to do that right now. It's mostly mindless work, but there are a lot of lines with calls to MeshWrVTK that need to be checked. I'd prefer to address that issue outside of this particular PR.

that's totally fine by me!

Hi Bonnie, I'm trying to view the documentation for the linearization mode shape visualization but when I click https://github.com/bjonkman/r-test/blob/pullrequest/bjonkman-linear/glue-codes/openfast/5MW_Land_ModeShapes/vtk-visualization.md I get a 404 page not found. Also, can you let me know which version of the matlab toolbox works with AnalyzeModeShapes.m?
Thank you!!!

@nickjohnson13 , that r-test branch got merged in. Here's an updated link: https://github.com/OpenFAST/r-test/blob/pullrequest/bjonkman-linear/glue-codes/openfast/5MW_Land_ModeShapes/vtk-visualization.md (and I fixed it in the comments above).
The example should work with the current matlab-toolbox master branch, but it's been quite a while since I've run these. :)

Thanks, guys!


KITTI Evaluation

Hi @zhyever, thank you so much for open-sourcing this excellent repository! I am new to depth estimation. I have a question regarding the evaluation of the KITTI dataset.
For the self-supervised depth estimation methods like MonoDepth2, the models are evaluated with a ground-truth file named gt_depths.npz. For evaluations in your repository, however, it seems that we need to download the annotated depth ground-truth.
I would like to ask what is the difference between these two. Can I evaluate your models in a way just like the self-supervised depth estimation methods? Besides, will the size of evaluation images affect the evaluation performance? For example, can I evaluate your DepthFormer model trained with the 1241 × 376 resolution with evaluation images of size 640 x 192?
Thanks again for your contribution in this area. Looking forward to your reply!

Thanks for your interest.
I read the codes of MonoDepth2 a long time ago, so the views below can be not correct but only from my own understanding. For unsupervised depth estimation, since there is no GT, the model output can only be in a range of 0-1, as an output of a sigmoid function. That is a sort of relative depth. When evaluating, we need to apply the mean depth to scale the predicted depth range to the real depth range for calculating metrics between GT.
For the GT gt_depths.npz. I remember the evaluation of self-supervised depth and supervised depth is not aligned till now. You can refer to the Toyota repos/papers of PackNet3D for some details, where I remember the authors highlight this issue and also modify something about it in their codes. In short words, the GT for supervised depth is more elaborate and accurate, but the GT for self-supervised depth is closer to the raw point cloud projections (maybe it is).
So, if you'd like to evaluate our models in a self-supervised manner, that can be implemented directly to calculate metrics with the gt_depth.npz but skip the scaling stage since the supervised model gives a real depth. Also I need to highlight, when you evaluate your DepthFormer model trained with the 1241 × 376 resolution with evaluation images of size 640 x 192, you will get totally wrong depth results. Not only for DepthFormer but all supervised methods cannot work for evaluation with various resolutions. I recommend our paper STMono3D (Unsupervised Domain Adaptation for Monocular 3D Object Detection via Self-Training), where we analyze this problem called depth shift in a mono3d setting, but I observed this problem also exists in monodepth.

Hi @zhyever, thank you for your prompt and detailed reply! Your explanation resolves my questions and helps me understand the difference between supervised and self-supervised depth estimation better.

There is a long time since I have not studied self-supervised methods. So, if there is something wrong, hope that you can point it out here, avoiding misleading other people seeing this issue. Thanks, hope that can be helpful for you. 😄

There is a long time since I have not studied self-supervised methods. So, if there is something wrong, hope that you can point it out here, avoiding misleading other people seeing this issue. Thanks, hope that can be helpful for you. 😄

Sure, I will provide more details of this after testing the models. Thanks again!


iOS app missing dependencies //build/toolchain/ios:ios_x64

Problem
I've cloned the repo, setup ninja, gn, openssl and pkg-info but still stuck with Xcode not finishing the chip_xcode_build_connector.sh script
I'm getting an Error Unresolved dependencies looking for //build/toolchain/ios:ios_x64 and can't figure out what that is.  I've seen various issues reported with the iOS project but all other workarounds/solutions doesn't work so maybe this is a recent bug or I completely missed a step in the CHIP setup guide?
+ mkdir -p /Users/someuser/Library/Developer/Xcode/DerivedData/Darwin-fjaqranellsxxnfsmhjpezwgeroc/Build/Intermediates.noindex/CHIP.build/Debug-iphonesimulator/CHIP.build
+ export
+ defines=()
+ declare -a defines
+ read -r -a defines
+ declare target_defines=
+ for define in '"${defines[@]}"'
+ case "$define" in
+ target_defines+=',"CHIP_PROJECT_CONFIG_INCLUDE=<CHIPProjectConfig.h>"'
+ for define in '"${defines[@]}"'
+ case "$define" in
+ target_defines+=',"CHIP_PLATFORM_CONFIG_INCLUDE=<platform/Darwin/CHIPPlatformConfig.h>"'
+ for define in '"${defines[@]}"'
+ case "$define" in
+ target_defines+=',"CHIP_DEVICE_PLATFORM_CONFIG_INCLUDE=<platform/Darwin/CHIPDevicePlatformConfig.h>"'
+ for define in '"${defines[@]}"'
+ case "$define" in
+ continue
+ for define in '"${defines[@]}"'
+ case "$define" in
+ continue
+ for define in '"${defines[@]}"'
+ case "$define" in
+ continue
+ for define in '"${defines[@]}"'
+ case "$define" in
+ continue
+ for define in '"${defines[@]}"'
+ case "$define" in
+ continue
+ for define in '"${defines[@]}"'
+ case "$define" in
+ continue
+ for define in '"${defines[@]}"'
+ case "$define" in
+ continue
+ for define in '"${defines[@]}"'
+ case "$define" in
+ continue
+ for define in '"${defines[@]}"'
+ case "$define" in
+ continue
+ for define in '"${defines[@]}"'
+ case "$define" in
+ continue
+ for define in '"${defines[@]}"'
+ case "$define" in
+ target_defines+=',"DEBUG=1"'
+ target_defines='["CHIP_PROJECT_CONFIG_INCLUDE=<CHIPProjectConfig.h>","CHIP_PLATFORM_CONFIG_INCLUDE=<platform/Darwin/CHIPPlatformConfig.h>","CHIP_DEVICE_PLATFORM_CONFIG_INCLUDE=<platform/Darwin/CHIPDevicePlatformConfig.h>","DEBUG=1"]'
+ declare target_cpu=
+ case $PLATFORM_PREFERRED_ARCH in
+ target_cpu=x64
+ declare 'target_cflags="-target","x86_64-apple-ios13.4"'
+ read -r -a archs
+ for arch in '"${archs[@]}"'
+ target_cflags+=',"-arch","x86_64"'
+ [[ NO == YES ]]
+ args=('default_configs_cosmetic=[]' 'chip_crypto="mbedtls"' 'chip_build_tools=false' 'chip_build_tests=false' 'chip_ble_project_config_include=""' 'chip_device_project_config_include=""' 'chip_inet_project_config_include=""' 'chip_system_project_config_include=""' 'target_cpu="'"$target_cpu"'"' 'target_defines='"$target_defines" 'target_cflags=['"$target_cflags"']')
+ declare -a args
+ [[ Debug != Debug* ]]
+ [[ iOS != macOS ]]
+ args+=('target_os="ios"' 'import("//config/ios/args.gni")')
+ [[ iOS == macOS ]]
+ cd /Users/someuser/proj_source/connectedhomeip
++ find_in_ancestors chip_xcode_build_connector_env.sh
+ ENV=
+ [[ -n '' ]]
+ git submodule update --init
+ [[ -z '' ]]
+ env -i PW_ENVSETUP_NO_BANNER=1 PW_ENVSETUP_QUIET=1 bash -c '. scripts/activate.sh'
20210909 12:26:35 INF Environment passes all checks!
+ set +ex
20210909 12:26:36 INF Environment passes all checks!
+ cd /Users/someuser/Library/Developer/Xcode/DerivedData/Darwin-fjaqranellsxxnfsmhjpezwgeroc/Build/Intermediates.noindex/CHIP.build/Debug-iphonesimulator/CHIP.build
+ gn --root=/Users/someuser/proj_source/connectedhomeip gen --check out '--args=default_configs_cosmetic=[] chip_crypto="mbedtls" chip_build_tools=false chip_build_tests=false chip_ble_project_config_include="" chip_device_project_config_include="" chip_inet_project_config_include="" chip_system_project_config_include="" target_cpu="x64" target_defines=["CHIP_PROJECT_CONFIG_INCLUDE=<CHIPProjectConfig.h>","CHIP_PLATFORM_CONFIG_INCLUDE=<platform/Darwin/CHIPPlatformConfig.h>","CHIP_DEVICE_PLATFORM_CONFIG_INCLUDE=<platform/Darwin/CHIPDevicePlatformConfig.h>","DEBUG=1"] target_cflags=["-target","x86_64-apple-ios13.4","-arch","x86_64"] target_os="ios" import("//config/ios/args.gni")'
ERROR Unresolved dependencies.
//:check(//build/toolchain/ios:ios_x64)
  needs //build/toolchain/ios:ios_x64()
//:default(//build/toolchain/ios:ios_x64)
  needs //build/toolchain/ios:ios_x64()

Proposed Solution
Add missing setup guide if any?

I figured this one out as actually running the app instead of just building it against a simulator.  I should have figured as this was more of device integration I should have thought of it this way.  It may help if there's a mention in any of the documents to run the iOS app against an actual device or build it to an arm64 target instead of the simulator so it's quick for anyone starting on this project.


Saving on staging crashes server

If i try to save any article on staging, the requests timeout and other pages show a 504 / cloudflare notice that the server is down, this started to happen about after https://github.com/serlo/serlo.org/pull/570

It worked for me this time (both with and without review).

It worked for me this time (both with and without review).

Trying to save trivial change here: https://de.serlo-staging.dev/entity/repository/add-revision/1875
Giving up after 3 unsuccessful tries. It's a non-converted article. Maybe this is causing the bug?
It worked with https://de.serlo-staging.dev/entity/repository/add-revision/25812 after 2 tries.

Trying to save trivial change here: https://de.serlo-staging.dev/entity/repository/add-revision/1875
Giving up after 3 unsuccessful tries. It's a non-converted article. Maybe this is causing the bug?
It worked with https://de.serlo-staging.dev/entity/repository/add-revision/25812 after 2 tries.

Saving now uses new infrastructure.


Update README.md

Related Github tickets

a list of github issues that are relevant for this PR

Background
A short paragraph about what exactly this PR solves and what is the problem.
Testing completed

[ ] a list of actions that you've performed to ensure that this PR works as intended
[ ] 99% of the times you should have an item: "wrote tests"

Change pigeon version from 0.5.0 to 0.5.1