Doctests marked should_panic should not panic

It seems that panicking doctests marked as should_panic causes the tests run with tarpaulin to panic instead marking it a success.
This happens in my github action (output, with Rust nightly and run with --run-targets Tests,Doctests),
[INFO tarpaulin] running /home/runner/work/pineappl/pineappl/target/doctests/src_lumi_rs_36_0/rust_out
thread 'main' panicked at 'assertion failed: !entry.is_empty()', pineappl/src/lumi.rs:43:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

for a doctest which expects a panic in this situation and is marked as should_panic. Running with RUST_BACKTRACE=1 shows that the test panics at line 43, as expected. When run with cargo test the test succeeds.

I was a bit thrown initially since I added should_panic handling for doc tests previously and it worked with my tests. But I think after looking at the test I see what it is! Your test is marked as rust,should_panic but I didn't take into account the markdown formatting so it might only work with should_panic
This should be a simple enough fix in src/cargo.rs I'll look into it when I'm finished at work :+1:

Thanks for looking into it! Removing rust, did not fix it, unfortunately; there must be something else. I also checked the following attributes, maybe that helps tracking down the problem:

ignore and rust,ignore correctly ignore the test,
but compile_fail and rust,compile_fail seem to work a bit differently; if the test doesn't compile, it doesn't appear with an [INFO tarpaulin] line (probably correct), but if the test does compile, it will be executed and the outcome will decide whether it's a fail or not (wrong). In this case compilation should mean fail.

hmm if you use --debug it should give some more useful information (but also a lot of cargo build output). I also wasn't aware of compile_fail tests in doctests I'll aim to add something for that

I didn't learn anything useful from adding --debug. But if you point me to where should_panic is handled in tarpaulin I could try to see where the problem is.

src/cargo.rs finds which doc tests should panic and in src/lib.rs the return codes are checked to make sure tests passed or panicked as expected

@cschwan ah I've figured out where the issue is. Doctest filenames are generated from the file path and line number. My doctest handling in get_panic_handling checks this relative to workspace root, but it's actually done relative to package root. Once I fix that it should analyse the files and find the should_panics

You can try out a patch I've done on branch issue_493

Great, that works and fixes the reported problem. Thanks a lot!

Great I'll endeavour to finish any polishing and get it into the develop branch tomorrow then.
I'll probably have another release at the weekend as well just gonna see if I can fix a few more things easily before then and try it on a few more projects with doc tests just to be safe


Text in Terminal in Windows is unreadable

When I create a new terminal session in Lens for Windows and start to type, the text becomes unreadable:

I am using the following Lens version for Windows:

Microsoft Windows [Version 10.0.17763.1457]

Is this still a problem with 4.1.4?

No it is solved now. Thank you.


How to get the point cloud in camera coordinate?

Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

Consider checking out SDK examples.
Have you looked in our documentations?
Is you question a frequently asked one?
Try searching our GitHub Issues (open and closed) for a similar issue.

All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

Required Info

Camera Model
D435

Firmware Version
<IP_ADDRESS>

Operating System & Version
Win10

Platform
PC

SDK Version
SDK2.0

Language
C++

Issue Description
<Describe your issue / question / feature request / etc..>
From the rs_pointcloud.cpp, we can know that the output of point cloud objects is a collection of {x,y,z}coordinates in the real world.
From the RS viewer 3D model, we can get the point cloud in .ply format, which is in the pixel coordinates, right? The depth format I used is Z16.
But how can I get the point cloud in camera coordinates?
https://forums.intel.com/s/question/0D50P0000490XLwSAM/getting-complete-point-cloud-from-d435?language=en_US
I ran the code and saved the results in .ply format. And I think the point cloud is in the camera coordinate system,because the function deproject_pixel_to_point was called. I also saved point cloud from RS viewer.
But when I open those two .ply files in the meshlab,the two point clouds are almost same. Shouldn't these two point clouds belong to different coordinate systems?
My main steps: get frames from camera--align depth and color frame--deroject pixel to point--save to disc. Is there anything wrong with that step?
How can I get the aligned and correct point cloud in camera coordinates?

[Realsense Customer Engineering Team Comment]
Hi @anranx,
The .ply saving function(in realsense-viewer example) is points::export_to_ply() in /src/archive.cpp.
https://github.com/IntelRealSense/librealsense/blob/master/src/archive.cpp#L44-L135
The points data saved into .ply file is the same as the points in rs-pointcloud.cpp.
BTW, which version of librealsense do you use?


Add --unset to rbenv-global

Support unsetting global ruby version. Makes rbenv global consistent with
rbenv local and rbenv shell

I'm not sure if I want people to be able to unset the global version. One should change it to "system" instead. Maybe --unset should be an equivalent of that? But then it's a misnomer, because you can't truly unset a global rbenv version— it just falls back to system if the file is missing.
Related: #631 #362

I know there isn't a technical difference between removing the global version file and setting rbenv global system. Do you think there is a semantic difference?
When my goal is to revert to the system ruby, for some reason my brain doesn't think rbenv {global,local,shell} system, I think --unset. When I unset local/shell versions, it's because I want to remove the more-specific setting and just fall back to global. Similarly, I mentally think of the system ruby as another level of fallback behind global. (shell -> local -> global -> system) So when I want the global version to be system, I don't think in terms of 'setting' the global version to system, but rather clearing the global version and letting it fall back to system. As if I'm (sorta) removing rbenv from the equation. In this view, there is a (hair-splitting) semantic difference between removing the global file and setting it to system. I just think in terms of clearing the global version, not setting it to system.
I think this is why I find the lack of global-unset a little incongruent.

FWIW, this change was originally prompted by nodenv. Since node doesn't ship with OS X, I find that many people have the 'system' node installed via homebrew, and use nodenv to manage additional nodes. Since homebrew deps are generally upgraded regularly, one's system node is (generally) fairly recent. Thus, I find myself frequently pegging the global node to a nodenv version for a short time (for a number of shells across a number of directories, so local/shell are not suitable). But then when I'm finished with that I want to clear the global setting, and let it fall back to the homebrew node. nodenv global system would work fine, but as I described above, my brain doesn't reach for that for whatever reason.
I would be happy with making --unset just do a global system as you describe. I still think that having --unset available would be nice.

Thanks for explaining. I'm still not convinced we need --unset. In the case of shell and local, the --unset flag literally deletes a setting. In the case of global, it would just change it.

one's system node is (generally) fairly recent. Thus, I find myself frequently pegging the global node to a nodenv version for a short time

Sounds like you want to temporarily set your global nodenv version to "system" node. I don't see why nodenv global system would be counterintuitive here.
Not many people ask for this, therefore I'm not terribly motivated to approve this change yet.

In the case of shell and local, the --unset flag literally deletes a setting. In the case of global, it would just change it.

well.... yeah, I guess. As this PR is currently authored, global --unset deletes a setting, too. It's a perfect mirror of the implementation for shell/local --unset. As long as the global version is set via a file, it's possible to delete the version file just as one could delete a local version file. Having a way to delete the file via the cli would be convenient and makes the shell/local/global subcommands all consistent.

I don't see why nodenv global system would be counterintuitive here.

I'm not necessarily saying it's counter-intuitive for others, just that I personally didn't even think to try it. TBH, until you mentioned it, I never even realized that system was a valid value that could be set. I just assumed without looking into it deeply, that 'system' was simply a helpful display-only message for when rbenv didn't set a ruby version. I had no idea that rbenv itself could actually be set to system.

Hey! I think I am facing this issue as well. Maybe it's the same use case.
I had Ruby 2.5.1 installed on an Ubuntu installation that had no Rubies itself and set it to be the global version. When I uninstalled 2.5.1 I get this output from rbenv versions
$ rbenv versions
rbenv: version `2.5.1' is not installed (set by /usr/local/rbenv/version)
Warning: no Ruby detected on the system

While I expected the warning, I didn't expect the first line which made me google and find out if there's a better way than deleting the version file (or emptying it?) which led me to this issue.

Not many people ask for this, therefore I'm not terribly motivated to approve this change yet.

Well, once every 3 years it seems ;)
Do we still not want this feature? If so, is deleting or emptying the version file preferred?

If so, is deleting or emptying the version file preferred?

Yes, that is preferred. Closing since there was not much demand for this, and the feature set of rbenv is basically frozen for the time being. Thank you @jasonkarns for the idea!


SkiaRenderer broken

see "Render into Multiple Viewports" help patch

https://discourse.vvvv.org/t/howto-play-an-audio-file-vl-audio/19741

See 7fd7e58e64c67272b5b349d1445a2e079a0c0e64


Pass generateLabel flag as Y while retrying generating shipping label

Current behavior
Label is not getting generated if the facility is not part of auto generate label facility group even if retrying.
Expected behavior
Label should be generated successfully.
Steps to reproduce the issue

Can you handle fixing this bug by yourself?

[ ] YES
[ ] NO

Environment details

Browser:
OS:
Code Version:

Additional information

Auto label generation enhancement changes are dependent on OMS release v5.16.0. Hence, marking this ticket as on-hold.

Verified and working fine in release v2.9.1. Hence, closing this issue.


Use better app import format

Currently we use the following format, where import only the module:
python -m aiotaskq worker <import-path-to-module>
# Example: python -m aiotaskq worker some.import.path.to.module

But it's better to follow a more popular format, like the one uvicorn uses:
python -m aiotaskq worker <import-path-to-module>:<path-to-attribute>
# Example: python -m aiotaskq worker some.import.path.to.module:app.some.attribute

See uvicorn import source code
This way we can do wrap all the tasks in an object, most likely something we can call an app (borrowing celery convention):
# file: ./src/project/app/app.py

from aiotaskq import AIOTaskQ

app = AIOTaskQ(
    # ...some configs
)

# file: ./src/project/tasks.py
from project.app import app

@app.task
def some_task_1():
    pass

@app.task
def some_task_2():
    pass

And start the worker like the following:
python -m aiotaskq worker project.app.app:app

So now tasks can be defined anywhere in the project since it's all captured by the object project.app.app:app, or in general <import-path-to-module>:<path-to-attribute>

As part of #42 we want to support the common app being used in a typical starlette application: path.to.module:some_app_object where some_app_object encapsulates the whole starlette application. So we want to do something similar, whereby all tasks should also be encapsulated within a single application instance. Currently, we're not able to do that, hence #42 is blocked.


[CLI] When logging df.plot(), the datetime index is ignored

Description
When I log df.plot(), the datetime index is ignored and arbitrary  xlabels are displayed.
Wandb features
wandb.log()
How to reproduce
# df is a pandas dataframe with datetime index
plt = df.plot()
wandb.log({'return':plt}
However:
import matplotlib.pyplot as plt
plt.plot(df)
wandb.log({'return': plt})
works well
Environment

OS: Ubuntu 18
Environment: AWS
Python Version: 3.8.10

Hi @yanghoonkim , thanks for writing in. I tried to reproduce but both the plots display same labels for me as seen in the UI here. Here's the script I used:
date_today = datetime.now()
days = pd.date_range(date_today, date_today + timedelta(7), freq='D')

np.random.seed(seed=1111)
data = np.random.randint(1, high=100, size=len(days))
df = pd.DataFrame({'test': days, 'col2': data})
df = df.set_index('test')

run = wandb.init(project='debug_df_timestamp')
plt = df.plot()
wandb.log({'return_1': plt})

plt.plot(df)
wandb.log({'return_2': plt})

run.finish()

Thanks for your answer @anmolmann
Would you please try this code?
import pandas as pd
import numpy as np
import wandb

date_today = datetime.now()
days = pd.date_range(date_today, date_today + timedelta(300), freq='M').floor('D')

np.random.seed(seed=1111)
data = np.random.randint(1, high=100, size=len(days))
data1 = np.random.randint(1, high=100, size=len(days))

df = pd.DataFrame({'test': days, 'col2': data, 'col1':data1})
df = df.set_index('test')

run = wandb.init(project='debug_df_timestamp')
plt = df.plot()
wandb.log({'return_1': plt})

plt.plot(df)
wandb.log({'return_2': plt})

run.finish()```

I found the error occurs when there are multiple columns

@yanghoonkim , this is happening because the obj returned by df.plot() and plt.plot(df) are different and therefore, the way matplotlib converts them into a Plotly obj differs as well. If you try to log the plots like this:
date_today = datetime.now()
days = pd.date_range(date_today, date_today + timedelta(7), freq='D')

np.random.seed(seed=1111)
data = np.random.randint(1, high=100, size=len(days))
data1 = np.random.randint(1, high=100, size=len(days))
df = pd.DataFrame({'test': days, 'col1': data1, 'col2': data})
df = df.set_index('test')

plt = df.plot()
run = wandb.init(project='debug_pd_timestamp')
wandb.log({'return_1': plt})
plt.plot(df)
wandb.log({'return_2': plt})
run.finish()

This would result in identical plots like this:

@anmolmann By comparing the following code block, you can find that the source of the problem is not the obj returned by df.plot and the plt.plot(df), but the precision of the datetime index
import pandas as pd
import numpy as np
import wandb
import matplotlib.pyplot as plt

date_today = datetime.now()
days = pd.date_range(date_today, date_today + timedelta(300), freq='M').floor('D')

np.random.seed(seed=1111)
data = np.random.randint(1, high=100, size=len(days))
data1 = np.random.randint(1, high=100, size=len(days))

df = pd.DataFrame({'test': days, 'col2': data, 'col1':data1})
df = df.set_index('test')
run = wandb.init(project='debug_df_timestamp')
fig = df.plot()
wandb.log({'return_1': fig})

plt.plot(df)
wandb.log({'return_2': plt})

run.finish()

If I remove .floor('D'), there is no problem
import pandas as pd
import numpy as np
import wandb
import matplotlib.pyplot as plt

date_today = datetime.now()
days = pd.date_range(date_today, date_today + timedelta(300), freq='M')

np.random.seed(seed=1111)
data = np.random.randint(1, high=100, size=len(days))
data1 = np.random.randint(1, high=100, size=len(days))

df = pd.DataFrame({'test': days, 'col2': data, 'col1':data1})
df = df.set_index('test')
run = wandb.init(project='debug_df_timestamp')
fig = df.plot()
wandb.log({'return_1': fig})

plt.plot(df)
wandb.log({'return_2': plt})

run.finish()

@anmolmann Would you please reopen the issue?

I found that if the datetimeindex doesn't have microseond value, then the situation happens

Hello @yanghoonkim,
Thank you for the reproducible issue code and the context into the issue itself. The engineers are looking into this and we will update you as soon as progress is made!

@ash0ts Thanks a lot! I'm looking forward to it!

Is this bug fixed?

This problem is still happening in the latest version of wandb


[test-triage] chip_sw_sleep_pwm_pulses

Hierarchy of regression failure
Chip Level
Failure Description
UVM_ERROR @ * us: (i2c_monitor.sv:66) [monitor] Check failed r_bit === *'b* (* [*] vs * [*]) has 14 failures:

Test chip_sw_sleep_pwm_pulses has 1 failures.

0.chip_sw_sleep_pwm_pulses.605364621
Line 628, in log /container/opentitan-public/scratch/os_regression/chip_earlgrey_asic-sim-vcs/0.chip_sw_sleep_pwm_pulses/latest/run.log

  UVM_ERROR @ 9425.130260 us: (i2c_monitor.sv:66) [uvm_test_top.env.m_i2c_agent0.monitor] Check failed r_bit === 1'b0 (0x1 [1] vs 0x0 [0])
  UVM_INFO @ 9425.130260 us: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

Steps to Reproduce

GitHub Revision: 47d4bfa34
dvsim invocation command to reproduce the failure, inclusive of build and run seeds:
./util/dvsim/dvsim.py hw/top_earlgrey/dv/chip_sim_cfg.hjson -i TEST_NAME --build-seed<PHONE_NUMBER> --fixed-seed 605364621 --waves fsdb

Tests with similar or related failures

[ ] chip_sw_sleep_pwm_pulses

Recently the nightlies have started to not fully succeed. Last 3 nights have shown partial success.

Tests
2022-11-14 08:10:42
2022-11-13 08:12:54
2022-11-12 08:04:45
2022-11-11 08:07:48
2022-11-10 08:10:06
2022-11-09 08:04:06
2022-11-08 08:03:42
2022-11-07 08:08:10
2022-11-06 07:11:02
2022-11-05 07:11:23
2022-11-04 07:12:05
2022-11-03 07:11:23
2022-11-02 07:03:26
2022-11-01 07:10:18
Suite

178
chip_sw_sleep_pwm_pulses
66.67
66.67
66.67
100
100
100
100
100
100
100
100
100
66.67
100
chip_sw_sleep_pwm_pulses

@jdonjdon
@weicaiyang
i think this is probably another one that should be fixed with monitor_en.
Should we just mark it as root caused?

@tjaychen @weicaiyang Let me scan all share io agent to make sure it is properly disabled in chip_env.

This is the same failure as #16296
Fixed by #16333

Passing after the fix:

Tests
2022-11-21 08:11:59
2022-11-20 08:10:00
2022-11-19 08:10:46
2022-11-18 08:16:52
2022-11-17 08:11:05
2022-11-16 08:14:00
2022-11-15 08:11:58
2022-11-14 08:10:42
2022-11-13 08:12:54
2022-11-12 08:04:45
2022-11-11 08:07:48
2022-11-10 08:10:06
2022-11-09 08:04:06
2022-11-08 08:03:42
Suite

187
chip_sw_sleep_pwm_pulses
100
100
100
100
100
100
100
66.67
66.67
66.67
100
100
100
100
chip_sw_sleep_pwm_pulses


Advanced Brute Medication Guidebook entry

About the PR
Requires #57

haha imagine if i pushed the entire branch again instead of just the guidebook that would be crazy


Basic Unary Iteration / QROM

unary iteration / qrom bloqs support a lot of features (multi-dimensional select; break) that obscure the key points of the design. They're also written cirq-style where each target op op[l] is a bundle of gates instead of one bloq per index. Writing an idiomatic "basic" 1d unary iteration and/or qrom bloq would be very instructive.

A sketch of how this might look
class Set(Bloq):
  datum: int
  bitsize: int

  return Signature.build(x=bitsize)

  def decompose(bb, x):
    for bit in iter_bits(self.datum):
       if bit:
         bb.Add(XGate()

class ApplyL(Bloq):
  ops: List[Bloq]
  selection_bitsize: int = field(validator=len(ops) fits)
  target_bitsize: int

  return Signature.build(ctrl=1, selection=self.selection_bitsize, target=self.target_bitsize)

  def decompose(bb, ctrl, selection, target):
    selection = selection.split()
    l = 0
    for ...:
        # construct and ladder
        _, add_ctrl_op = self.ops[l].get_ctrl_system()
        s, target = add_ctrl_op(ctrl = [current selection bit], {'x': target})
        l += 1
        # unconstruct and ladder

qrom = ApplyL([Set(d, target_bitsize) for d in data], selection_bitsize, target_bitsize)

This should go in a tutorial notebook. We shouldn't have two QROM implementations.

After ApplyLthBloq and UnaryIterationGate, I think this can be closed. The "basic" unary iteration can be accessed using the ApplyLthBloq interface and power users can derive from UnaryIterationGate for more control.


 Every 10ms I want to yield to my UI task (running on requestAnimationFrame). My code is something like this:
async fn decode_in_chunks(
    mut decoder: Decoder,
    process: Box<dyn Fn(Message)>,
) {
    let mut last_yield = instant::Instant::now();

    for msg in decoder {
        process(msg);

        if last_yield.elapsed() > instant::Duration::from_millis(10) {
            // yield to the ui task
            yield_().await;
            last_yield = instant::Instant::now();
        }
    }
}

I spawn this with wasm_bindgen_futures::spawn_local
The problem is the yield_ function. I've currently defined it as so:
/// Yield to other tasks
async fn yield_() {
    sleep_ms(1).await;
}

// Hack to get async sleep on wasm
async fn sleep_ms(millis: i32) {
    let mut cb = |resolve: js_sys::Function, _reject: js_sys::Function| {
        web_sys::window()
            .unwrap()
            .set_timeout_with_callback_and_timeout_and_arguments_0(&resolve, millis)
            .expect("Failed to call set_timeout");
    };
    let p = js_sys::Promise::new(&mut cb);
    wasm_bindgen_futures::JsFuture::from(p).await.unwrap();
}

this feels very hacky, for obvious reasons.
Any suggestions for how to improve this?

Doesn't the usual Rust async yield work?
struct Yield {
    yielded: Option<bool>,
}

impl Yield {
    fn new() -> Self {
        Self {
            yielded: Some(false),
        }
    }
}

impl Future for Yield {
    type Output = ();

    fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
        let yielded = self.yielded.unwrap();

        if yielded {
            self.yielded = None;
            Poll::Ready(())
        } else {
            self.yielded = Some(true);
            Poll::Pending
        }
    }
}

Also it seems that the future you are spawning here is really blocking. Instead of putting a timer here I believe you should just yield in-between every call to process(), presumably the function that you can't make non-blocking, making this basically an async iterator/stream.

@daxpedda unfortunately that yield doesn't work at all for me. It compiles, but my async task seems to get stuck.

Ah, apologies, my yield example has a bug, it doesn't wake the Waker.
Now that I think about it, it kinda depends on how the executor works.
I fixed it in my original post.
Will play around with it myself and see if it works.

Alright, this doesn't seem to work at all. I'm guessing the wasm-bindgen-futures executor simply rechecks if something is ready again right away.

So after some testing basically my findings are that you can yield, but not how you want it. Basically you yield to the executor. So you can multiplex multiple futures by yielding, but you can't yield back to the "native" runtime because it's not part of the executor.
Obviously web workers would be the perfect solution here, unfortunately the ecosystem around that is not exactly in a good state. I can't come up with a better solution then yours otherwise.

Thanks @daxpedda, that at least clears things up a little bit for me!

I actually found two viable solutions for this in the meantime:

window.requestIdleCallback() which can be used to do exactly what you want here. Just let the future sleep and wake it up in the callback. Though this is unsupported in Safari (Bugzilla).

While looking for Safari polyfills, I found that using setTimeout() with a delay of 0 can also do the job:

If this parameter is omitted, a value of 0 is used, meaning execute "immediately", or more accurately, the next event cycle.

Unfortunately nested timeouts will prevent this behavior and instead delay it by 4ms.

I discovered even more in the meantine:

Scheduler.postTask(), only available in Chromium.
Scheduler.yield(), available nowhere.

Both could do what you want and much more, really interesting developments.


Treats edges pointed the wrong direction

For relationships coming from uberongraph with MONDO:disease_responds_to, the edges are pointing from disease to chemical_substance:
match (d:disease)-[:treats]->(c:chemical_substance) return * limit 10

the robokopkg graph is the graph that contains this incorrect data. the data in this graph was populated using the legacy robokop services (no data services data).
when i get to refactoring the ubergraph data in data services for the load manager i will make a point of insuring this is taken care of.

Fixed


Compatibility : experimental/filesystem unavailable in some versions of LLVM compiler/library

Dear developers,
I noticed that some versions of LLVM (including the MacOS derivatives) do not have support for #include <experimental/filesystem> in their version of libc++ library.  Which causes the compilation of armnnTfLiteParser/test/Activations.cpp.o to fail.
So on a Mac I either had to compile without TFlite support or turn off the unit tests.
However it seems that the functionality from #include <experimental/filesystem> is not actually used, since when I commented out this line, the compilation just continued. Could this #include line be removed?

Yep, happy for unused stuff to be removed. experimental/filesystem should not be included, as we're using boost/filesystem anyway and only in test/LoadModel.cpp. I'll happily +2 a change on this, otherwise I'll get round to it in the next couple of days.

FIxed by aab709b3598b07223ba2da1ab8a18f95e30ff70d


Makefile per module using external library

We will need a way to have the Makefile working with other Makefile in subdirs, as in modules/*/Makefile.
Some module surely will need to use external libraries, so a specific Makefile needs be done for such.

Closed as we have recently moved to CMake.


Not able to copy images from docker storage to containers storage using skopeo when the graph driver is overlay2, kernel version is 3.x

It fails with the following message:
kernel too old to provide multiple lowers feature for overlay: driver not supported.
version used:
skopeo --version
skopeo version 0.1.24-dev commit: 6ac96b7874815c8fb0ae7eff563942334db5d45d
uname -r
3.10.0-693.el7.x86_64

It comes from containers/storage which is vendored into skopeo: https://github.com/containers/storage/blob/master/drivers/overlay/overlay.go#L136

@nalind ?

I believe you need to add `override_kernel_check=true to /etc/containers/storage.conf.

Under storage options.

We should ship that by default on.  @lsm5

@rhatdan Thanks, I am able to copy the images after adding the check.

Skopeo now ships with this on by default.


feat: cla support

cla support & bug fixes

:tada: This PR is included in version 1.5.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


How can we help?
None
Please read and ensure the following

[X] The installer meets the above requirements

Please provide the following information
https://aka.ms/vs/17/release/vc_redist.arm64.exe
Microsoft.VCRedist.2015+.x64 and Microsoft.VCRedist.2015+.x86 variants are available; not sure why arm64 variant isn't packaged yet.

Hi @rollingmoai
Your package is in PR

It exists, but it's just named differently - likely due to how the Arm64 installers work. In this case it's Microsoft.VCRedist.2022.arm64 or Microsoft.VCRedist.2019.arm64

It exists, but it's just named differently - likely due to how the Arm64 installers work. In this case it's Microsoft.VCRedist.2022.arm64 or Microsoft.VCRedist.2019.arm64

If so, why Microsoft names one package with three names?


Moved the logic for counting domain limits to a new class

Goal

Testing

Release Notes

WHAT:
WHY:
WHO:

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#749  👈
#748
#747
#746
#745
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @priettt and the rest of your teammates on  Graphite

Merge activity

Apr 18, 7:50 PM EDT: @bidetofevil started a stack merge that includes this pull request via Graphite.


Unable to create new credential

I am unable to create a new credential. The NTFY API doesn't show up in the dropdown.
I already tried restarting the instance.
This happens on the latest version of n8n.

@Quadrubo On a fresh local install of n8n and this NTFY node, I'm unable to replicate the issue. Could you provide a screenshot of the installed node within your community nodes? Also, does the node itself within the workflow allow you to create the credential?


Add a version command

Add a cobra command so that a user can run ghost version and have the version print out

@all-contributors please add @Galzzly for code


Turning on at low brightness turns on at higher brightness and dims down

Issue
Hi there,
i have a hue whit filament bulb which i want to turn on at 1% brightness. The bulb then turns on with a higher brightness and dims down to 1%
Log Messages
no messages
Debug Files

homebridge-hue.json.gz

You're using the Hue bridge to expose Hue lights natively.  You'll have to take this up with Signify.

I need to see a debug log file to see what's happening, but it sounds like the HomeKit app sends On and Brightness separately.  When they're sent simultaneously, Homebridge Hue combines them into a single API call, which deCONZ should translate to turning the light on at minimum brightness and then dimming up to the requested brightness.

There you go:
`[20/08/2021, 19:23:42] [Deconz] Affe: homekit on changed from false to true
[20/08/2021, 19:23:42] [Deconz] Zigbee-GW: request 19: PUT /lights/16/state {"on":true}
[20/08/2021, 19:23:42] [Deconz] Zigbee-GW: request 19: 200 OK
[20/08/2021, 19:23:42] [Deconz] Affe: state changed event: {"alert":null,"bri":3,"on":true,"reachable":true}
[20/08/2021, 19:23:43] [Deconz] Diele 1: attr changed event: {"id":"10","lastannounced":"2021-08-13T19:48:12Z","lastseen":"2021-08-20T17:23Z","manufacturername":"IKEA of Sweden","modelid":"TRADFRI bulb E27 WW clear 250lm","name":"Diele 1","swversion":"2.1.022","type":"Dimmable light","uniqueid":"ec:1b:bd:ff:fe:9c:e8:45-01"}
[20/08/2021, 19:23:43] [Deconz] Zigbee-GW: request 20: GET /sensors
[20/08/2021, 19:23:43] [Deconz] Zigbee-GW: request 20: 200 OK
[20/08/2021, 19:23:43] [Deconz] Zigbee-GW: request 21: GET /lights
[20/08/2021, 19:23:43] [Deconz] Zigbee-GW: request 21: 200 OK
[20/08/2021, 19:23:47] [Deconz] Affe: homekit on changed from true to false
[20/08/2021, 19:23:47] [Deconz] Zigbee-GW: request 22: PUT /lights/16/state {"on":false}
[20/08/2021, 19:23:47] [Deconz] Zigbee-GW: request 22: 200 OK
[20/08/2021, 19:23:47] [Deconz] Affe: state changed event: {"alert":null,"bri":3,"on":false,"reachable":true}
[20/08/2021, 19:23:48] [Deconz] Zigbee-GW: request 23: GET /sensors
[20/08/2021, 19:23:48] [Deconz] Zigbee-GW: request 23: 200 OK
[20/08/2021, 19:23:48] [Deconz] Zigbee-GW: request 24: GET /lights
[20/08/2021, 19:23:48] [Deconz] Zigbee-GW: request 24: 200 OK
[20/08/2021, 19:23:53] [Deconz] Zigbee-GW: request 25: GET /sensors
[20/08/2021, 19:23:53] [Deconz] Zigbee-GW: request 25: 200 OK
[20/08/2021, 19:23:53] [Deconz] Zigbee-GW: request 26: GET /lights
[20/08/2021, 19:23:53] [Deconz] Zigbee-GW: request 26: 200 OK
[20/08/2021, 19:23:54] [Deconz] Farbiges Licht: attr changed event: {"colorcapabilities":8,"id":"1","lastannounced":"2021-08-01T15:11:17Z","lastseen":"2021-08-20T17:23Z","manufacturername":"IKEA of Sweden","modelid":"TRADFRI bulb E27 CWS opal 600lm","name":"Farbiges Licht","swversion":"1.3.002","type":"Color light","uniqueid":"bc:33:ac:ff:fe:98:20:9b-01"}
[20/08/2021, 19:23:55] [Deconz] Power 40: state changed event: {"current":0,"lastupdated":"2021-08-20T17:23:55.409","power":0,"voltage":229}
[20/08/2021, 19:23:58] [Deconz] Affe: attr changed event: {"id":"16","lastannounced":"2021-06-23T15:43:52Z","lastseen":"2021-08-20T17:23Z","manufacturername":"Philips","modelid":"LWO001","name":"Affe","swversion":"1.55.8_r28815","type":"Dimmable light","uniqueid":"00:17:88:01:08:9f:f3:67-0b"}
[20/08/2021, 19:23:58] [Deconz] Zigbee-GW: request 27: GET /sensors
[20/08/2021, 19:23:58] [Deconz] Zigbee-GW: request 27: 200 OK
[20/08/2021, 19:23:58] [Deconz] Zigbee-GW: request 28: GET /lights
[20/08/2021, 19:23:58] [Deconz] Zigbee-GW: request 28: 200 OK
[20/08/2021, 19:23:59] [Deconz] Waschmaschine: attr changed event: {"id":"11","lastannounced":"2021-05-04T21:12:42Z","lastseen":"2021-08-20T17:23Z","manufacturername":"_TZ3000_g5xawfcq","modelid":"TS0121","name":"Waschmaschine","swversion":null,"type":"Smart plug","uniqueid":"58:8e:81:ff:fe:d3:91:19-01"}
‘

[20/08/2021, 19:23:42] [Deconz] Affe: homekit on changed from false to true
[20/08/2021, 19:23:42] [Deconz] Zigbee-GW: request 19: PUT /lights/16/state {"on":true}
[20/08/2021, 19:23:42] [Deconz] Zigbee-GW: request 19: 200 OK

Assuming Affe is indeed /lights/16, HomeKit only sends the On characteristic.  Homebridge Hue makes the API call to the turn on the light.  Depending on the light firmware and/or settings, this will be at a fixed level, or at the previous level.  The Hue lights I've seen are hard-coded (in firmware) to use the previous level.
I don't see any change in brightness, neither requested nor executed.

So there is no way to change this. Thank you

You might try a HomeKit scene, which sets both On and Brightness.

This does not help. So its bulb related i think?

I have no experience with the Hue filament bulbs, but I find it hard to believe that they would work differently from other Hue bulbs.  Do you see a single PUT in the log, setting both on and bri?


[LIVY-773] Fix status code when session limit reached

What changes were proposed in this pull request?
I propose to change http status code from 400 (Bad Request) to 503 (Service Unavailable) when Livy server cannot start new sessions because livy.server.session.max-creation is reached.
How was this patch tested?
Modified existing unit tests.

Codecov Report

Merging #297 into master will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##             master     #297   +/-   ##
=========================================
  Coverage     68.24%   68.24%
  Complexity      965      965
=========================================
  Files           104      104
  Lines          5952     5952
  Branches        900      900
=========================================
  Hits           4062     4062
  Misses         1311     1311
  Partials        579      579

Impacted Files
Coverage Δ
Complexity Δ

.../scala/org/apache/livy/server/SessionServlet.scala
71.57% <100.00%> (ø)
20.00 <0.00> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3b9bbef...66b7a84. Read the comment docs.


Event Emitter Exception Logging

Is there an existing issue that is already proposing this?

[X] I have searched the existing issues

Is your feature request related to a problem? Please describe it
Currently any event listeners (ie @OnEvent("MY_EVENT")) default to suppressing exceptions.  The default logger message when suppressing exceptions only prints the error message and ignores the stack if it exists.  This causes it to be very tricky to trace where errors actually occurred.
Describe the solution you'd like
Print the stack trace if available when an exception occurs in an event listener.  I believe the stack trace just needs added to the following line.  https://github.com/nestjs/event-emitter/blob/master/lib/event-subscribers.loader.ts#L187
Teachability, documentation, adoption, migration strategy
No response
What is the motivation / use case for changing the behavior?
It is very difficult to track where an error originated without the stack trace.

Would you like to create a PR for this issue?

Let's track this here https://github.com/nestjs/event-emitter/pull/1283


(nit) Minor fix in request domain handling for origin verification for IDN

This is very nit picking and not sure if the additional computation cost makes sense, but the current implementation seems to be missing a corner case for handling IDN subdomains where user agents convert entire domain name to puny code.

Sorry i missed the change you've made. Closing this PR in favor of your change.


FileTarget - Only enable FileSystemWatcher when ConcurrentWrites = true

Only enable Win32-API-call when overriding FileAttributes. Refactor MultiFileWatcher to better handle medium trust.
Better chance of NLog FileTarget working on low-permission-platforms (UWP / Xamarin Android / Xamarin iOS). But it will take longer time (up to 1 sec) for NLog to recover from the user deleting the active file (before 100 ms)

Updated Wiki:  https://github.com/NLog/NLog/wiki/File-target (For enableFileDelete about FileSystem watchers being used when ConcurrentWrites=true)


Zephyr stdio has broken newline handling

Zephyr stdin hook eats newline characters. Zephyr stdout hook adds \r after each \n.
This seems to be applied for all libc implementations, but I have tested mostly with newlib.
Unfortunately that is not what libc implementations expect - they expect to remove newlines as they see fit and expect to write the bytes they actually want to write.
This makes scanf, printf, read, write etc. broken in some cases when one cannot actually receive or send newline characters correctly.

@rszkodzinski thanks for the issue. Would you consider opening a Pull Request with the fix?

Unfortunately I'm somewhat busy writing completely unrelated code and there are bound to be unintended consequences in console or shell code for example.
Fortunately for me the hooks are not broken for getchar case, only for all the reads longer than 1.

@rszkodzinski the hooks are not installed by default AFAIK, see here:
https://github.com/zephyrproject-rtos/zephyr/issues/60741#issuecomment-1649886273
Did you modify the code to install hooks?

I did install the input hooks, yes. In particular, one that does RTT waitkey. (I was using RTT for testing.)
The output has the hooks installed by default, and those have interesting newline handling in them too.

I did install the input hook, yes, in my main. In particular, one that does RTT waitkey. (I was using RTT for testing.)

Thanks for the clarification and code @rszkodzinski. I will follow up.


avoid loading invisible files for partial panos of any image type

This is the first of a series of contributions that I plan to do over the next days or weeks,
providing various extensions/improvements of the code that I've done for myself so far,
supporting partial cylindrical panoramas like this one.
This PR extends the support for partial panoramas of any image type.
It avoids loading image/tile files that would represent just background (i.e., invisible areas),
which helps saving storage and transfer bandwidth for files that anyway would not be shown.

Thanks! This looks good. I removed the ES6 syntax to maintain backward compatibility and merged it.

Thank you too!
This was swift :)
This message was sent using my Vernee Apollo Lite with K-9 Mail. Please excuse my brevity.
On March 4, 2018 3:06:19 AM GMT+01:00, Matthew Petroff<EMAIL_ADDRESS>wrote:

Thanks! This looks good. I removed the ES6 syntax to maintain backward
compatibility and merged it.
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/mpetroff/pannellum/pull/562#issuecomment-370195667


Backend: Cluster articles using Elasticsearch MoreLikeThis

I love NewsBlur, been a paying customer for a while now. the one feature that I would love to have is removing duplicate articles at the folder level. preferably some sort of checkbox when I open "folder settings" maybe
I keep my feeds in folders, and I always find duplicated articles inside the folder. Not surprising really, if there's a good subject out there multiple feeds are going to have articles about it.
Looks like inoreader has this feature.

NewsBlur already tries very hard to remove duplicate articles automatically. Unfortunately they still come through, so let's use this ticket as a reminder to double check the dupe checker and to write tests for it to ensure it works as we expect it to. I see no reason to have this as a preference. Duplicates should be (and are) filtered out automatically.
Here's the code that does the work:
https://github.com/samuelclay/NewsBlur/blob/master/apps/rss_feeds/models.py#L2014-L2117

Oh, you mean from unrelated feeds in the folder? That's a different kind of check, one that I'm working on now using ElasticSearch's MoreLikeThis query.

Yes! From unrelated feeds in a folder

I like the idea very much. It is similar to what Google News is doing: Cluster the news for a specific event / topic across different sources. See screensho

t.

One idea I had about this: I guess it would already help, if Newsblur could cluster by the trained tags. E.g. I train the word "Amazon" and it clusters everything that has Amazon in the title - independent of the source


Reduce likelihood of macro collision

. . . by removing one macro and prefacing the other with the logical module it's associated with.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 93.871%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
2328

Relevant Lines:
2480

💛  - Coveralls

I feel like it might be worth doing this for the other constants, too?
Alternatively, would it make more sense to make this a namedspaced c++ thing?

I feel like it might be worth doing this for the other macros, too?

Done.

Alternatively, would it make more sense to make this a namedspaced c++ thing?

ROTL64 and CSA (new in this commit) could be replaced by inline functions, which could be namespaced. The others cannot.

I was thinking that we could alternatively restructure as a proper C++ class, but this certainly works for now and does improve naming safety. Thank you!


Fix an indentation in cdl2spi.py

I already fixed this on my end yesterday, but I have not pushed that yet.  I will do so in a little bit, so I'm going to go ahead and close this pull request.  Thanks for going ahead and fixing it, though, since I left it in a broken state. . .


Make fullscreen video screen height

Make preview as high as the screen to be able to see full video without scrolling. Width will adjust to keep proportions.

Thanks. Now merged in


chore(dockerfile): add docker for deployment

:tada: This PR is included in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Fix Instance Cert Bug (Hotfix branch)

Background
https://github.com/AthenZ/k8s-athenz-sia/issues/55
Related PRs

Hotfix: https://github.com/AthenZ/k8s-athenz-sia/pull/52/files
Latest: #53

TODOs

[ ] Fix the variables as the latest (to avoid confusion)
[ ] Fix the changes as the latest with the bug fix in #53

Not Dones

Since athenz-sia.env was not managed that much, it wont be added here.

Operation Check Done Internally.


Added code in Readme to remove error

Summary
Changes proposed in this pull request
Checklist (if any)

@imdhanish It would be better if you can actually remove all the errors instead of ignoring it.

@imdhanish all the courses already has PRs to fix pep8 so there's nothing much I can help.

you can mergemy PR bro

you can merge my PR bro


Ruby build on Kokoro MacOS broken "NoMethodError: undefined method `last_comment' "

Build have been consistently broken for a while with message NoMethodError: undefined method last_comment' for #Rake::Application:0x007fc0739c6a90`
it seems to be connected to a recent change in rake
https://stackoverflow.com/questions/35893584/nomethoderror-undefined-method-last-comment-after-upgrading-to-rake-11
I gave it a few attempts to fix but ruby build system is pretty cryptic so I wasn't successful yet.
2017-09-01 07:06:49,970 + export GRPC_CONFIG=dbg
+ GRPC_CONFIG=dbg
++ dirname tools/run_tests/helper_scripts/build_ruby.sh
+ cd tools/run_tests/helper_scripts/../../..
+ rm -rf ./tmp
+ rake compile
rake aborted!
NoMethodError: undefined method `last_comment' for #<Rake::Application:0x007fc0739c6a90>
/Users/kbuilder/.rvm/gems/ruby-2.4.0/gems/rubocop-0.30.1/lib/rubocop/rake_task.rb:23:in `initialize'
/Volumes/BuildData/tmpfs/src/github/grpc/workspace_ruby_macos_dbg_native/Rakefile:13:in `new'
/Volumes/BuildData/tmpfs/src/github/grpc/workspace_ruby_macos_dbg_native/Rakefile:13:in `<top (required)>'
(See full trace by running task with --trace)
real         0.77
user         0.73
sys          0.03

2017-09-01 07:06:49,970 FAILED: tools/run_tests/helper_scripts/build_ruby.sh [ret=1, pid=30129]

I has some partial success with gemspec changes from  https://github.com/gds-operations/vcloud-core/pull/189/files#diff-3019d21ddaeb5517843392a0f60eb64f, but they will also require some ruby style fixes it seems.
https://sponge.corp.google.com/target?id=2491af25-5d38-4e58-969a-1a973be276ea&target=github/grpc/ruby_macos_dbg_native&searchFor=&show=ALL&sortBy=STATUS


Just can't get PUAE working

Just can't get PUAE working. I tried several versions but they all crash. I think they rely on some stuff that isn't implemented yet.
[2021.10.23-22.08.06:639][423]Libretro: Warning: Unhandled env #65572
[2021.10.23-22.08.06:776][437]Libretro: 6888x no FPU: f201 00009000 PC=00f80ca0
[2021.10.23-22.08.06:776][437]Libretro: B-Trap f201 at f80ca0 (000001d782714450)

I tried messing with this and found it segfaulted trying to call an led callback function I didn't implement when trying to run Monkey Island 2. So I pushed a commit to fix that. Not sure if you ever ran into the same issue.
Also I saw you were working on config support. If you get that working don't hesitate to make a pull request I can probably fix the conflicts too depending on how familiar you are with git.

Ah haven't tried MI2 yet. Thanks a lot for fixing the LED callback.
I struggled a lot with C++ - not my language i must admit. My idea was to just dump the default core config into a text file instead of a table and read it out if it already exists. But yeah I wasn't even able to create a simple text file in C++... The types in C++ drive me crazy.


EC2 User Data Limit Exceeded

When following the demo install shown here:  https://github.com/FINRAOS/herd/wiki/demo-install
It appears that the user data exceeds an AWS limit.  This is the error I got:
The following resource(s) failed to create: [herdApplicationServer]. . Rollback requested by user.
CREATE_FAILED | AWS::EC2::Instance | herdApplicationServer | User data is limited to 16384 bytes (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterValue; Request ID: 3a...)
A bit of Googling seems to indicate that AWS has an upper limit of 16K
By way of troubleshooting I hacked out a big chunk of the UserData, just to see if I could get past this error.  That test was successful.  However, I do not fully understand the impacts of doing so.
 Received 0 conditions when expecting 1
  | 14:52:36 UTC-0700 | CREATE_IN_PROGRESS | AWS::CloudFormation::WaitCondition | herdServerWaitCondition
How can I help restructure this go get around this error.  Happy to contribute to the project, just need some idea how I could do so.

Hi Steve -
Thanks for taking an interest in Herd! You are correct, it looks like we
exceeded the limit for UserData. But by removing those database scripts, it
killed the product install. We will get a fix for this in the near future.
But in the meantime I suggest you take a look at Herd-MDL - it's the next
evolution for Herd. It is a new product that includes Herd and the Herd UI
but it also adds a Hive Metastore so other tools in the ecosystem can point
to data managed by Herd -- and it includes an example of such a tool, a
Presto cluster for interactive queries.
To install Herd-MDL you can go here
https://finraos.github.io/herd-mdl/docs.html and follow the instructions in
the Basic Install section. This will install Herd, Herd-UI, Hive Metastore,
and a Presto cluster, all integrated and ready to go.
One more thing to note - the cloudformation you started working with in
Herd was very minimal - all stuffed in one instance just for very light
experimentation. The Herd-MDL cloudformation installs a much larger set of
resources. It can get kind of expensive if you are just experimenting on
your personal account. If you want to start with something a bit smaller,
set the 'DeployComponents' setting to 'Herd' and it will skip the Hive
Metastore and Presto cluster.
Also feel free to let us know why you are interested in Herd. We always
like to hear from people who are building data lakes - it's interesting to
know what you are tying to accomplish, where you are in the process, etc.
Hope you continue experimenting and let us know how it goes.

Nate Weisz
Herd Product Manager

On Mon, Oct 29, 2018 at 6:07 PM stevejhall<EMAIL_ADDRESS>wrote:

When following the demo install shown here:
https://github.com/FINRAOS/herd/wiki/demo-install
It appears that the user data exceeds an AWS limit. This is the error I
got:
The following resource(s) failed to create: [herdApplicationServer]. .
Rollback requested by user.
CREATE_FAILED | AWS::EC2::Instance | herdApplicationServer | User data is
limited to 16384 bytes (Service: AmazonEC2; Status Code: 400; Error Code:
InvalidParameterValue; Request ID: 3a...)
A bit of Googling seems to indicate that AWS has an upper limit of 16K
By way of troubleshooting I hacked out a big chunk of the UserData, just
to see if I could get past this error. That test was successful. However, I
do not fully understand the impacts of doing so.
 Received 0 conditions
when expecting 1
| 14:52:36 UTC-0700 | CREATE_IN_PROGRESS |
AWS::CloudFormation::WaitCondition | herdServerWaitCondition
How can I help restructure this go get around this error. Happy to
contribute to the project, just need some idea how I could do so.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/FINRAOS/herd/issues/393, or mute the thread
https://github.com/notifications/unsubscribe-auth/ADFd9ar0j4BMLDmTTNg9oVp4pzgydRVTks5up3wJgaJpZM4YApC5
.

Thanks Nate.
I appreciate the detailed response.
I have some feedback to your inquiry and a new issue based on attempting to
Per your inquiry.  I work on a team at a Fortune 100 company.  We are looking at building out a next generation data lake largely around raw storage in S3 with many of the attributes that FINRA has highlighted.  Separation of storage and compute, catalog capabilities, governance, self service, and on demand compute.  A few of us have looked at the capabilities in this stack, and think it is very attractive as a starting point.  The key goal is to empower our business users by making data more available so that business decision makers can make data driven decisions.  The specific use case that I began to investigate is empowerment Data Engineers.
I am not sure what the team may ultimately adopt.  If this stack proves to be something we want to adopt we can likely bring many resources to the open source road map and likely become a significant contributor.
So, the issue
I have tried the setup at https://finraos.github.io/herd-mdl/docs.html as suggested.
That stack has a different problem.
CREATE_FAILED | AWS::CloudFormation::Stack | MdlStack | Parameters: [CloudWatchRetentionDays, PrestoMaxMemoryPerNode] must have values
I have tried twice.  First with the default choice of all, Second with the choice of Prereqs Only.  Both tries had the same outcome.
Prereqs Only we got a few artifacts to be created, we can see the failure as the last element of the list.
ArtifactCopyCustomResource | CopyArtifactsLambdaPhysicalID | Custom::ArtifactCopierResource | CREATE_COMPLETE |
ArtifactCopyLambdaFunction | eap-core-finra-herd-shall-ArtifactCopyLambdaFuncti-1RMEH60PB6MN2 | AWS::Lambda::Function | CREATE_COMPLETE |
DeployBucket |<PHONE_NUMBER>92-mdl-staging-prod | AWS::S3::Bucket | CREATE_COMPLETE |
LambdaExecutionRole | eap-core-finra-herd-shall7-LambdaExecutionRole-9KQWJF2M14TY | AWS::IAM::Role | CREATE_COMPLETE |
MdlStack | null | AWS::CloudFormation::Stack | CREATE_FAILED | Parameters: [CloudWatchRetentionDays, PrestoMaxMemoryPerNode] must have values
The parameters specified for the stack are:
BdsqlCoreInstanceType | m4.4xlarge |
BdsqlMasterInstanceType | m4.4xlarge |
CertificateArn |   |
CertificateInfo | CN=mdl.org,OU=MDL,O=FINRA,L=WASHINGTON,ST=DC,C=US |
CreateCloudFrontDistribution | true |
CreateDemoObjects | true |
CreateIAMRoles | true |
CreateKeypair | true |
CreateOpenLDAP | false |
CreateRDSInstances | true |
CreateS3Buckets | true |
CreateSecurityGroups | true |
CreateSQS | true |
CreateVPC | true |
CustomTagName | application |
CustomTagValue | herd-mdl |
DeployComponents | Prereqs Only |
DomainNameSuffix |   |
EnableSSLAndAuth | false |
Environment | prod |
EsInstanceType | t2.medium |
HerdDBClass | db.m4.large |
HerdDBSize | 10 |
HerdInstanceType | m4.2xlarge |
HostedZoneName |   |
ImageId | ami-1853ac65 |
LdapDN | dc=mdl,dc=org |
LdapInstanceType | t2.medium |
MDLInstanceName | mdl |
MetastorDBClass | db.m4.large |
MetastorDBSize | 10 |
MetastorInstanceType | m4.2xlarge |
NumberOfBdsqlCoreInstances | 1 |
RefreshDatabase | true |
ReleaseVersion | 1.2.0
I searched the installMDL.yml file to see if I could figure out how to set these parameters.  It was not evident.   Does the team have any suggestions?
Thanks,
Steve

Nate,
Following your suggestion, I am trying to install herd-mdl through the provided installMDL.yml file. The installation is failing with the following error:

2019-07-01 12:11:23 UTC-0400
ice-poc-herd-mdl
ROLLBACK_IN_PROGRESS
The following resource(s) failed to create: [MdlStack]. . Rollback requested by user.

2019-07-01 12:11:23 UTC-0400
MdlStack
CREATE_FAILED
Embedded stack arn:aws:cloudformation:us-east-1:681636940503:stack/ice-poc-herd-mdl-MdlStack-16AJ6J361EUU6/ca1a8850-9c1a-11e9-8652-0eb49284c068 was not successfully created: The following resource(s) failed to create: [PrerequisitesPrimaryStack].

I tried to get the PowerUserAccess policy, but the link provided to get the sample policy is giving 404. Can you please help?

Hi @ssaha01 -
Glad you are looking at Herd! The best course of action here is to use the Herd-MDL install as discussed in the comment above. We also recommend setting 'DeployComponents' to 'Herd' -- unless you are interested in the integrated Hive Metastore and Presto cluster.
@rongwang is an engineer from our team and she can dig into the current resource creation issue.

@ssaha01 For better assist you, we'd like you retry the creation and send us more detail information.
1.can you retry your stack creation with Rollback on failure set to No (Rollback on failure setting is under advanced section when you create cft from console)?
2. find the actual failed nested stack inside PrerequisitesPrimaryStack, and send us a screenshot of the Events history for the failed nested stack

I found that you can get below the EC2 user data limit by removing the parts of herd.localdb.template that loads the demo data: all the curl POSTs between lines 633-729.
That allows the CloudFormation script to run but herd-app doesn't start. Tomcat logs show a permission denied exception when trying to write "velocity.log". Caused by: java.io.FileNotFoundException: velocity.log (Permission denied)
at java.io.FileOutputStream.open0(Native Method) ~[?:1.8.0_242]
at java.io.FileOutputStream.open(FileOutputStream.java:270) ~[?:1.8.0_242]
at java.io.FileOutputStream.(FileOutputStream.java:213) ~[?:1.8.0_242]
at java.io.FileOutputStream.(FileOutputStream.java:133) ~[?:1.8.0_242]
at org.apache.log4j.FileAppender.setFile(FileAppender.java:294) ~[log4j-1.2.17.jar:?]
at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207) ~[log4j-1.2.17.jar:?]
at org.apache.log4j.FileAppender.(FileAppender.java:110) ~[log4j-1.2.17.jar:?]
at org.apache.log4j.RollingFileAppender.(RollingFileAppender.java:79) ~[log4j-1.2.17.jar:?]
at org.apache.velocity.runtime.log.Log4JLogChute.initAppender(Log4JLogChute.java:118) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.runtime.log.Log4JLogChute.init(Log4JLogChute.java:85) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.runtime.log.LogManager.createLogChute(LogManager.java:157) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.runtime.log.LogManager.updateLog(LogManager.java:269) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.runtime.RuntimeInstance.initializeLog(RuntimeInstance.java:871) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.runtime.RuntimeInstance.init(RuntimeInstance.java:262) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.runtime.RuntimeSingleton.init(RuntimeSingleton.java:112) ~[velocity-1.7.jar:1.7]
at org.apache.velocity.app.Velocity.init(Velocity.java:74) ~[velocity-1.7.jar:1.7]
at org.finra.herd.service.helper.VelocityHelper.(VelocityHelper.java:41) ~[herd-service-0.116.0.jar:?]
Not sure where Tomcat is trying to write velocity.log or how to change it.
Also as far as I can tell, there is not default user/password created for herd-ui, so even if herd-app starts, it wont be possible to login.
I am trying to switch to herd-mdl but our restricted environment doesn't allow creation of the required IAM entities, so having to rewrite the CloudFormation scripts to use existing entities. (which we did for the herd scripts but herd-mdl is more complex)

I tried to use version 0.136.0 and could pass through User data is limited to 16384 bytes error by replacing the db upgrade script with for loop.
          "psql -f herd.postgres.0.1.0.create.sql\n",
          "psql -f herd.postgres.0.1.0.refdata.sql\n",
          "psql -f herd.postgres.0.1.0.cnfgn.sql\n",
          "for i in {1..136}\n",
          "do\n",
          "  in=`expr $i + 1`\n",
          "  v_old=`printf %03d $i`\n",
          "  v_new=`printf %03d $in`\n",
          "  psql -f herd.postgres.0.${v_old}.0-to-0.${v_new}.0.upgrade.sql\n",
          "done\n",
          "psql -f activiti.postgres.create.engine.sql\n",
          "psql -f activiti.postgres.create.history.sql\n",
          "psql -f activiti.postgres.create.identity.sql\n",
          "psql -f quartz_tables_postgres.sql\n",
          "psql -f elasticsearch.configuration.values.sql\n",

But the UserData script still failed in the end. I got 404 during 'wget http://localhost:8080/herd-app/rest/buildInfo`.
Sorry for the Japanese output, but it means you got 404.
/usr/bin/wget http://localhost:8080/herd-app/rest/buildInfo
--2021-01-19 01:48:49--  http://localhost:8080/herd-app/rest/buildInfo
localhost (localhost) をDNSに問いあわせています... <IP_ADDRESS>
localhost (localhost)|<IP_ADDRESS>|:8080 に接続しています... 接続しました。
HTTP による接続要求を送信しました、応答を待っています... 404
2021-01-19 01:48:49 エラー 404: (説明なし)。

I think herd.localdb.template doesn't work anymore. I just want to know what herd is like and don't want to use herd-mdi because it requires multiple resources, so I gave up on it.

I tried to use version 0.136.0 and could pass through User data is limited to 16384 bytes error by replacing the db upgrade script with for loop.
          "psql -f herd.postgres.0.1.0.create.sql\n",
          "psql -f herd.postgres.0.1.0.refdata.sql\n",
          "psql -f herd.postgres.0.1.0.cnfgn.sql\n",
          "for i in {1..136}\n",
          "do\n",
          "  in=`expr $i + 1`\n",
          "  v_old=`printf %03d $i`\n",
          "  v_new=`printf %03d $in`\n",
          "  psql -f herd.postgres.0.${v_old}.0-to-0.${v_new}.0.upgrade.sql\n",
          "done\n",
          "psql -f activiti.postgres.create.engine.sql\n",
          "psql -f activiti.postgres.create.history.sql\n",
          "psql -f activiti.postgres.create.identity.sql\n",
          "psql -f quartz_tables_postgres.sql\n",
          "psql -f elasticsearch.configuration.values.sql\n",

But the UserData script still failed in the end. I got 404 during 'wget http://localhost:8080/herd-app/rest/buildInfo`.
Sorry for the Japanese output, but it means you got 404.
/usr/bin/wget http://localhost:8080/herd-app/rest/buildInfo
--2021-01-19 01:48:49--  http://localhost:8080/herd-app/rest/buildInfo
localhost (localhost) をDNSに問いあわせています... <IP_ADDRESS>
localhost (localhost)|<IP_ADDRESS>|:8080 に接続しています... 接続しました。
HTTP による接続要求を送信しました、応答を待っています... 404
2021-01-19 01:48:49 エラー 404: (説明なし)。

I think herd.localdb.template doesn't work anymore. I just want to know what herd is like and don't want to use herd-mdi because it requires multiple resources, so I gave up on it.


Setting up GXF platform no logging after deploying applications

Question:
I'v setting up a VM with Postgres, ActiveMQ and Tomcat. All the directory's are set correct.
I've got the issue that the applications that i am doplying into tomcat won't set any logging in "/var/log/osp/logs/".
For example i am deploying 'osgp-core-5.52.0-SNAPSHOT.war', so Flyway will generate the initial tables in my postgres database.
in /opt/tomcat/logs/catalina.out i am seeing:
"14-Mar-2024 09:52:02.822 INFO [http-nio-8080-exec-14] org.apache.catalina.core.StandardContext.reload Reloading Context with name [/osgp-core-5.52.0-SNAPSHOT] is completed
"
And no errors in this logging.
I am running tomcat under user tomcat so the dir /var/log/osp is of owner tomcat:tomcat amd also the directory /var/log/osp/logs/.
the context.xml in /opt/tomcat/conf/ is the same one as given in the project. so all the environment names are set to /etc/osp/ and i've placed there all configuration files (ive put the sample ones)
background:
I am running all the platform application on one VM. This VM is a AlmaLinux release 9.3 (Shamrock Pampas Cat).
I think that i missing something but i don't know what...

Which Java and Tomcat versions are you using?

I am using java17 and Tomcat9.
What i am seeing in the code is this line:
@PropertySource(value = "file:${osgp/Global/config}", ignoreResourceNotFound = true)
Where 'ignoreResourceNotFound = true', this indicates that there will be no errors what so ever when configfile isn't found, i don't know if this is helpfull for finding issues.

Could you give it a try using Tomcat 10? We have recently updated some libraries, after which Tomcat 10 is required.
For osgp-core the log configuration to be used is configured in the OsgpCoreInitializer class: "java:comp/env/osgp/Core/log-config". This should match with an entry in the Tomcat context.xml:
<Environment name="osgp/Core/log-config" value="/etc/osp/osgp-core-logback.xml" type="java.lang.String" override="false" />
Do you have an osgp-core-logback.xml file in /etc/osp? If so, what log levels are configured?

Regarding the @PropertySource annotation, the components already have a properties file included on their class path.
The properties in this file could be overridden by optional properties files in /etc/osp, hence the ignoreResourceNotFound = true

Yes i have a osgp-core-logback.xml placed in /etc/osp but it looks like this file won't load, because when i deliberately make a mistake in that config, still osgp-core component is deployed succesfull loaded in tomcat. And no errors are logged in catalina.out
In the context.xml in /opt/tomcat/conf is the enviroment var set it look like this:
'
<!-- Default set of monitored resources -->
<WatchedResource>WEB-INF/web.xml</WatchedResource>

<!-- Uncomment this to disable session persistence across Tomcat restarts -->
<!--
<Manager pathname="" />
-->

<!-- Uncomment this to enable Comet connection tacking (provides events
     on session expiration as well as webapp lifecycle) -->
<!--
<Valve className="org.apache.catalina.valves.CometConnectionManagerValve" />
-->

<!-- App config files -->
<Environment name="osgp/Global/config" value="/etc/osp/global.properties" type="java.lang.String" override="false" />

<Environment name="osgp/AdapterDomainAdmin/config" value="/etc/osp/osgp-adapter-domain-admin.properties" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterDomainAdmin/log-config" value="/etc/osp/osgp-adapter-domain-admin-logback.xml" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterDomainCore/config" value="/etc/osp/osgp-adapter-domain-core.properties" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterDomainCore/log-config" value="/etc/osp/osgp-adapter-domain-core-logback.xml" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterDomainMicrogrids/config" value="/etc/osp/osgp-adapter-domain-microgrids.properties" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterDomainMicrogrids/log-config" value="/etc/osp/osgp-adapter-domain-microgrids-logback.xml" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterDomainDistributionAutomation/config" value="/etc/osp/osgp-adapter-domain-distributionautomation.properties" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterDomainDistributionAutomation/log-config" value="/etc/osp/osgp-adapter-domain-distributionautomation-logback.xml" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterDomainPublicLighting/config" value="/etc/osp/osgp-adapter-domain-publiclighting.properties" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterDomainPublicLighting/log-config" value="/etc/osp/osgp-adapter-domain-publiclighting-logback.xml" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterDomainTariffSwitching/config" value="/etc/osp/osgp-adapter-domain-tariffswitching.properties" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterDomainTariffSwitching/log-config" value="/etc/osp/osgp-adapter-domain-tariffswitching-logback.xml" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterDomainSmartMetering/config" value="/etc/osp/osgp-adapter-domain-smartmetering.properties" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterDomainSmartMetering/log-config" value="/etc/osp/osgp-adapter-domain-smartmetering-logback.xml" type="java.lang.String" override="false" />

<Environment name="osgp/JasperInterface/config" value="/etc/osp/jasper-interface.properties" type="java.lang.String" override="false" />

<Environment name="osgp/AdapterWsAdmin/config" value="/etc/osp/osgp-adapter-ws-admin.properties" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterWsAdmin/log-config" value="/etc/osp/osgp-adapter-ws-admin-logback.xml" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterWsCore/config" value="/etc/osp/osgp-adapter-ws-core.properties" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterWsCore/log-config" value="/etc/osp/osgp-adapter-ws-core-logback.xml" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterWsMicrogrids/config" value="/etc/osp/osgp-adapter-ws-microgrids.properties" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterWsMicrogrids/log-config" value="/etc/osp/osgp-adapter-ws-microgrids-logback.xml" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterWsDistributionAutomation/config" value="/etc/osp/osgp-adapter-ws-distributionautomation.properties" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterWsDistributionAutomation/log-config" value="/etc/osp/osgp-adapter-ws-distributionautomation-logback.xml" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterWsPublicLighting/config" value="/etc/osp/osgp-adapter-ws-publiclighting.properties" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterWsPublicLighting/log-config" value="/etc/osp/osgp-adapter-ws-publiclighting-logback.xml" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterWsSharedDb/config" value="/etc/osp/osgp-adapter-ws-shared-db.properties" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterWsSmartMetering/config" value="/etc/osp/osgp-adapter-ws-smartmetering.properties" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterWsSmartMetering/log-config" value="/etc/osp/osgp-adapter-ws-smartmetering-logback.xml" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterWsTariffSwitching/config" value="/etc/osp/osgp-adapter-ws-tariffswitching.properties" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterWsTariffSwitching/log-config" value="/etc/osp/osgp-adapter-ws-tariffswitching-logback.xml" type="java.lang.String" override="false" />

<Environment name="osgp/Core/config" value="/etc/osp/osgp-core.properties" type="java.lang.String" override="false" />
<Environment name="osgp/Core/log-config" value="/etc/osp/osgp-core-logback.xml" type="java.lang.String" override="false" />
<Environment name="osgp/CoreDbApi/config" value="/etc/osp/osgp-core-db-api.properties" type="java.lang.String" override="false" />
<Environment name="osgp/CoreDbApiIec61850/config" value="/etc/osp/osgp-core-db-api-iec61850.properties" type="java.lang.String" override="false" />

<Environment name="osgp/Logging/config" value="/etc/osp/osgp-logging.properties" type="java.lang.String" override="false" />
<Environment name="osgp/Logging/log-config" value="/etc/osp/osgp-logging-logback.xml" type="java.lang.String" override="false" />

<Environment name="osgp/DomainLogging/config" value="/etc/osp/osgp-domain-logging.properties" type="java.lang.String" override="false" />

<Environment name="osgp/SigningServer/config" value="/etc/osp/signing-server.properties" type="java.lang.String" override="false" />
<Environment name="osgp/SigningServer/log-config" value="/etc/osp/signing-server-logback.xml" type="java.lang.String" override="false" />

<Environment name="osgp/AdapterProtocolDlms/config" value="/etc/osp/osgp-adapter-protocol-dlms.properties" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterProtocolDlms/log-config" value="/etc/osp/osgp-adapter-protocol-dlms-logback.xml" type="java.lang.String" override="false" />

<Environment name="osgp/AdapterProtocolOslp/config" value="/etc/osp/osgp-adapter-protocol-oslp.properties" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterProtocolOslp/log-config" value="/etc/osp/osgp-adapter-protocol-oslp-logback.xml" type="java.lang.String" override="false" />

<Environment name="osgp/AdapterProtocolOslpElster/config" value="/etc/osp/osgp-adapter-protocol-oslp-elster.properties" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterProtocolOslpElster/log-config" value="/etc/osp/osgp-adapter-protocol-oslp-elster-logback.xml" type="java.lang.String" override="false" />

<Environment name="osgp/AdapterProtocolIec61850/config" value="/etc/osp/osgp-adapter-protocol-iec61850.properties" type="java.lang.String" override="false" />
<Environment name="osgp/AdapterProtocolIec61850/log-config" value="/etc/osp/osgp-adapter-protocol-iec61850-logback.xml" type="java.lang.String" override="false" />

<Environment name="osgp/WebDeviceSimulator/config" value="/etc/osp/web-device-simulator.properties" type="java.lang.String" override="false" />
<Environment name="osgp/WebDeviceSimulator/log-config" value="/etc/osp/web-device-simulator-logback.xml" type="java.lang.String" override="false" />

'

De logging of catalina. out is:

20-Mar-2024 10:48:16.233 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:               Linux
20-Mar-2024 10:48:16.233 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            5.14.0-362.18.1.el9_3.x86_64
20-Mar-2024 10:48:16.233 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:          amd64
20-Mar-2024 10:48:16.233 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             /usr/lib/jvm/java-17-openjdk-<IP_ADDRESS>.7-2.el9.x86_64
20-Mar-2024 10:48:16.233 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           17.0.10+7-LTS
20-Mar-2024 10:48:16.233 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:            Red Hat, Inc.
20-Mar-2024 10:48:16.233 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         /opt/tomcat
20-Mar-2024 10:48:16.234 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         /opt/tomcat
20-Mar-2024 10:48:16.389 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties
20-Mar-2024 10:48:16.390 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
20-Mar-2024 10:48:16.390 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.awt.headless=true
20-Mar-2024 10:48:16.390 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.security.egd=file:/dev/./urandom
20-Mar-2024 10:48:16.391 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
20-Mar-2024 10:48:16.394 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
20-Mar-2024 10:48:16.394 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
20-Mar-2024 10:48:16.394 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
20-Mar-2024 10:48:16.394 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
20-Mar-2024 10:48:16.394 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util=ALL-UNNAMED
20-Mar-2024 10:48:16.398 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
20-Mar-2024 10:48:16.398 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
20-Mar-2024 10:48:16.398 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms512M
20-Mar-2024 10:48:16.398 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx1024M
20-Mar-2024 10:48:16.398 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+UseParallelGC
20-Mar-2024 10:48:16.398 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/opt/tomcat
20-Mar-2024 10:48:16.398 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/opt/tomcat
20-Mar-2024 10:48:16.398 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/opt/tomcat/temp
20-Mar-2024 10:48:16.409 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib]
20-Mar-2024 10:48:17.567 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
20-Mar-2024 10:48:17.737 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["ajp-nio-0:0:0:0:0:0:0:1-8009"]
20-Mar-2024 10:48:17.744 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [2182] milliseconds
20-Mar-2024 10:48:17.933 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
20-Mar-2024 10:48:17.933 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/10.1.19]
20-Mar-2024 10:48:17.937 SEVERE [main] org.apache.catalina.startup.HostConfig.beforeStart Unable to create directory for deployment: [/opt/tomcat/conf/Catalina/localhost]
20-Mar-2024 10:48:18.049 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/opt/tomcat/webapps/osgp-secret-management-5.52.0-SNAPSHOT.war]
20-Mar-2024 10:48:37.879 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
20-Mar-2024 10:48:38.110 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/opt/tomcat/webapps/osgp-secret-management-5.52.0-SNAPSHOT.war] has finished in [20,061] ms
20-Mar-2024 10:48:38.134 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/opt/tomcat/webapps/web-demo-app-5.52.0-SNAPSHOT.war]
20-Mar-2024 10:48:44.377 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
20-Mar-2024 10:48:44.399 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/opt/tomcat/webapps/web-demo-app-5.52.0-SNAPSHOT.war] has finished in [6,264] ms
20-Mar-2024 10:48:44.457 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/opt/tomcat/webapps/osgp-adapter-ws-admin-5.52.0-SNAPSHOT.war]
20-Mar-2024 10:48:56.825 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
20-Mar-2024 10:48:56.837 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/opt/tomcat/webapps/osgp-adapter-ws-admin-5.52.0-SNAPSHOT.war] has finished in [12,379] ms
20-Mar-2024 10:48:56.868 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/opt/tomcat/webapps/osgp-adapter-domain-core-5.52.0-SNAPSHOT.war]
20-Mar-2024 10:49:05.535 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
20-Mar-2024 10:49:05.546 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/opt/tomcat/webapps/osgp-adapter-domain-core-5.52.0-SNAPSHOT.war] has finished in [8,678] ms
20-Mar-2024 10:49:05.548 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/opt/tomcat/webapps/osgp-protocol-adapter-iec61850-5.52.0-SNAPSHOT.war]
20-Mar-2024 10:49:13.561 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
20-Mar-2024 10:49:13.571 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/opt/tomcat/webapps/osgp-protocol-adapter-iec61850-5.52.0-SNAPSHOT.war] has finished in [8,024] ms
20-Mar-2024 10:49:13.588 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/opt/tomcat/webapps/osgp-logging-5.52.0-SNAPSHOT.war]
20-Mar-2024 10:49:25.017 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
20-Mar-2024 10:49:25.030 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/opt/tomcat/webapps/osgp-logging-5.52.0-SNAPSHOT.war] has finished in [11,442] ms
20-Mar-2024 10:49:25.094 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/opt/tomcat/webapps/web-device-simulator-5.52.0-SNAPSHOT.war]
20-Mar-2024 10:49:37.490 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
20-Mar-2024 10:49:37.526 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/opt/tomcat/webapps/web-device-simulator-5.52.0-SNAPSHOT.war] has finished in [12,432] ms
20-Mar-2024 10:49:37.575 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/opt/tomcat/webapps/osgp-adapter-ws-core-5.52.0-SNAPSHOT.war]
20-Mar-2024 10:49:52.216 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
20-Mar-2024 10:49:52.224 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/opt/tomcat/webapps/osgp-adapter-ws-core-5.52.0-SNAPSHOT.war] has finished in [14,648] ms
20-Mar-2024 10:49:52.262 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/opt/tomcat/webapps/osgp-adapter-domain-admin-5.52.0-SNAPSHOT.war]
20-Mar-2024 10:50:02.827 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
20-Mar-2024 10:50:02.838 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/opt/tomcat/webapps/osgp-adapter-domain-admin-5.52.0-SNAPSHOT.war] has finished in [10,575] ms
20-Mar-2024 10:50:02.852 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/opt/tomcat/webapps/osgp-core-5.52.0-SNAPSHOT.war]
20-Mar-2024 10:50:11.250 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
20-Mar-2024 10:50:11.261 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/opt/tomcat/webapps/osgp-core-5.52.0-SNAPSHOT.war] has finished in [8,409] ms
20-Mar-2024 10:50:11.273 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/opt/tomcat/webapps/osgp-protocol-simulator-iec61850-5.52.0-SNAPSHOT.war]
20-Mar-2024 10:50:12.618 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
20-Mar-2024 10:50:12.621 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/opt/tomcat/webapps/osgp-protocol-simulator-iec61850-5.52.0-SNAPSHOT.war] has finished in [1,348] ms
20-Mar-2024 10:50:12.711 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/opt/tomcat/webapps/host-manager]
20-Mar-2024 10:50:13.300 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
20-Mar-2024 10:50:13.787 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/opt/tomcat/webapps/host-manager] has finished in [1,077] ms
20-Mar-2024 10:50:13.787 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/opt/tomcat/webapps/ROOT]
20-Mar-2024 10:50:13.886 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
20-Mar-2024 10:50:13.889 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/opt/tomcat/webapps/ROOT] has finished in [102] ms
20-Mar-2024 10:50:13.893 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/opt/tomcat/webapps/docs]
20-Mar-2024 10:50:13.997 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
20-Mar-2024 10:50:14.002 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/opt/tomcat/webapps/docs] has finished in [109] ms
20-Mar-2024 10:50:14.003 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/opt/tomcat/webapps/examples]
20-Mar-2024 10:50:14.293 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
20-Mar-2024 10:50:14.556 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/opt/tomcat/webapps/examples] has finished in [553] ms
20-Mar-2024 10:50:14.557 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/opt/tomcat/webapps/manager]
20-Mar-2024 10:50:14.676 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
20-Mar-2024 10:50:14.679 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/opt/tomcat/webapps/manager] has finished in [122] ms
20-Mar-2024 10:50:14.784 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
20-Mar-2024 10:50:14.952 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-0:0:0:0:0:0:0:1-8009"]
20-Mar-2024 10:50:15.001 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [117257] milliseconds

De content of /etc/osp/osgp-core-logback.xml is:
[root@gfxtest osp]# cat  osgp-core-logback.xml
<?xml version="1.0" encoding="UTF-8"?>

<!--
SPDX-FileCopyrightText: Contributors to the GXF project

SPDX-License-Identifier: Apache-2.0
-->

<configuration debug="true">

  <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
      <pattern>[%d{yyyy-MM-dd HH:mm:ss.SSS}] [${HOSTNAME}] [%thread] %level %logger{36}@%method:%line - %msg%n</pattern>
    </encoder>
  </appender>

  <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>/var/log/osp/logs/osgp-core.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
      <fileNamePattern>/var/log/osp/logs/osgp-core.log.%d{yyyy-MM-dd}.%i.log</fileNamePattern>
      <maxFileSize>100MB</maxFileSize>
      <maxHistory>10</maxHistory>
      <totalSizeCap>20GB</totalSizeCap>
    </rollingPolicy>
    <encoder>
      <pattern>[%d{yyyy-MM-dd HH:mm:ss.SSS}] [${HOSTNAME}] [%thread] %level %logger{36}@%method:%line - %msg%n</pattern>
    </encoder>
  </appender>

  <root level="DEBUG" additivity="false">
    <appender-ref ref="CONSOLE"/>
    <appender-ref ref="FILE"/>
  </root>

</configuration>

Could it be an issue with file permissions on  /etc/osp and/or /var/log/osp?
Or maybe SELinux is blocking access?

I am running tomcat with user:group permissions tomcat:tomcat. And i start it as a service. The service file is placed in /etc/systemd/system/tomcat.service.
[Unit]
Description=Tomcat webs servlet container
After=network.target

[Service]
Type=forking

User=tomcat
Group=tomcat

Environment="JAVA_HOME=/usr/lib/jvm/jre"
Environment="JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom"

Environment="CATALINA_BASE=/opt/tomcat"
Environment="CATALINA_HOME=/opt/tomcat"
Environment="CATALINA_PID=/opt/tomcat/temp/tomcat.pid"
Environment="CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC"
Environment="CATALINA_OUT=/var/log/tomcat/catalina.out"

ExecStart=/opt/tomcat/bin/startup.sh
ExecStop=/opt/tomcat/bin/shutdown.sh
ReadWritePaths=/var/log/

[Install]
WantedBy=multi-user.target

The file permissions on /etc/osp is drwxr-xr-x.  6 tomcat tomcat   4096 Mar 20 10:47 osp
The file permissions on /var/log/osp is drwxr-xr-x. 3 tomcat tomcat    4096 Mar  6 13:38 osp
i've checked SELinux by following this page:
https://access.redhat.com/documentation/de-de/red_hat_enterprise_linux/8/html/using_selinux/troubleshooting-problems-related-to-selinux_using-selinux
i don't see any messages regarding writing to the log directories.

I've just noticed this line in your logging:
20-Mar-2024 10:48:17.937 SEVERE [main] org.apache.catalina.startup.HostConfig.beforeStart Unable to create directory for deployment: [/opt/tomcat/conf/Catalina/localhost].
Could you try creating this folder manually (with the appropriate permissions) and see if it fixes the logging issues?
As a side note some more information regarding the logging configuration:
For the logging configuration an override/fallback mechanism is implemented in the AbstractApplicationInitializer class. The logging configuration is resolved in the following order:

a specific logback file resolved from the context (java:comp/env/osgp/Core/log-config passed from the OsgpCoreInitializer class)
a global logback file resolved from the context (java:/comp/env/osgp/Global/log-config)
a logback file from the classpath (classpath:logback-config.xml)

If the logging is working it should be visible in the logging which configuration file is used:

DEBUG org.opensmartgridplatform.core.application.config.ApplicationContext -- Using log config /etc/osp/osgp-core-logback.xml found through JNDI name java:comp/env/osgp/Core/log-config INFO o.o.c.a.config.ApplicationContext@initializeLoggingContext:124 - Initialized logging using /etc/osp/osgp-core-logback.xml
DEBUG org.opensmartgridplatform.core.application.config.ApplicationContext -- Using log config /etc/osp/global-logback.xml found through JNDI name java:/comp/env/osgp/Global/log-config INFO o.o.c.a.config.ApplicationContext@initializeLoggingContext:124 - Initialized logging using /etc/osp/global-logback.xml
INFO o.o.c.a.config.ApplicationContext@initializeLoggingContext:124 - Initialized logging using classpath:logback-config.xml

The src/webapp/META-INF/ folder inside the osgp-core project also includes a context.xml which also contains the
<Environment name="osgp/Core/log-config" value="/etc/osp/osgp-core-logback.xml" type="java.lang.String" override="true" /> entry.

The SERVERE log is something that has to with the atrribute createDirs on the  in the server.xml of tomcat. Default it is set 'true'.
https://tomcat.apache.org/tomcat-9.0-doc/config/host.html
This has no negative impact on the startup of the server. When i can set this attribute to false it prevent the error from logging.
Thanks for the extra information about the logging structure.

Ok i have some information:
I probably didn't build the application correctly. I'm missing the webhook of the war file to actually start the application.
WEB-INF/web.xml is often used for this. However, this file does not exist in my WAR.

web.xml should not be needed to start the application.

I see, i can run osgp-core now on my windows machine with Tomcat 9.0.86. I see application logging placed in the catalina.out.
Still i have no idea why no logging appear on my linux VM. So it is something in the enviroment that's for sure.

Good to read that you succeeded in running osgp-core on your windows machine.
I've been looking over the messages in this conversation in order to see if I missed something.
Would it make a difference if you modify ReadWritePaths in the service file to ReadWritePaths=/var/log/osp/ (followed by systemctl daemon-reload) since the tomcat user probably does not have write access to /var/log/?
Does the path /var/log/osp/logs/ already exists? If it does, is it also owned by the tomcat user?

Closing the issue, as it seems to be something in the environment and not in the application.


[Feature request] playback mp4 ISO multiplex files

Currently player is really nice, functionality is lovely. It does playback ISO packaged mp4 files, however the functionality of the multiplexed media is not available via player. It would be brilliant if the player was enable to select multiplex features, such as multiplexed audio (various soundtracks), video (various bitrate renditions) and presentation data (including subtitles) from mp4 packages in the ISO media. MP4box does a nice job of preparing ISO media mp4 files and it helps to keep personal library of videos tidy and to package adaptive bitrate mp4 files, unfortunately ISO package functionality is lost to player and it plays the highest rendition of adaptive packaged files. This ability to playback ISO media files and to carry multiplex functionality to player would be a wonderful addition. Thanks again for a great project ;-)

It would indeed be great, but that functionality is bound on the browser implementations. If you want support for multiple audio/video streams you have to annoy mozilla and the google chrome team into implementing it. Streama is only using features provided by the browsers, it doesn't really do anything with the media itself.


Not able to find Core Enums in IdentityServer4

HI,
I'm trying to figure out where are the Core Model Enums in IdentityServer4. For Eg, openid connect flows enum. Do I need to create my own core models or is there one dll available?
Thanks,
Navya.

There is the IdentityServerConstants class - all other constants are coming from IdentityModel. What exactly are you looking for?

Hi,
Thanks, I'm looking for below enum (and few other) in IdentityServer4.
I could find it in IdentityServer3 but not in 4
namespace IdentityServer3.Core.Models
///
/// OpenID Connect flows.
///
public enum Flows
{
///
/// authorization code flow
///
AuthorizationCode = 0,
    /// <summary>
    /// implicit flow
    /// </summary>
    Implicit = 1,

    /// <summary>
    /// hybrid flow
    /// </summary>
    Hybrid = 2,

    /// <summary>
    /// client credentials flow
    /// </summary>
    ClientCredentials = 3,

    /// <summary>
    /// resource owner password credential flow
    /// </summary>
    ResourceOwner = 4,

    /// <summary>
    /// custom grant
    /// </summary>
    Custom = 5,

    /// <summary>
    /// authorization code with proof key flow
    /// </summary>
    AuthorizationCodeWithProofKey = 6,

    /// <summary>
    /// hybrid flow with proof key
    /// </summary>
    HybridWithProofKey = 7
}

Hello, Can anyone answer this for me? If a dll is available in Identity4.

We don't have something like this. Just copy the code.


Refactor PlayerTableViewController logic to allow resizable tableView…

…, to leave room for the music player sticky cell at bottom

After you make the minor spelling change, this is good to merge in!


fastify  how to use?

Maybe this plugin can help you, https://github.com/fastify/fastify-express

You can set it up with fastify-express as @felixmosh mentioned. For instance to server at /queues you can:
    fastify.register(require('fastify-express'))
      .after(() => {fastify.use('/queues', router)})
    await fastify.listen(__PORT__)

Since v3.0.0 bull-board supports fastify natively, check the with-fastify example


Add sent/read status icons

Add a single ✔ check indicator when the message has been sent and change it to a double when the server has received it (WhatsApp-style).
This automatically shows the user a hint when the server is unreachable.

Made a PR for it #38.

🎉


Does not work with directories

I'm not saying that it should hash directories, but it should be able to handle (add the hash or ignore) directories if they happen to match the glob pattern.
Ignoring them and going strictly with using the glob pattern from the CLI (not API), adding the nodir option should do the trick.  But you probably want to perform this at a more core/common level to include the API.

Hey @TheSharpieOne thanks for the issue! Good call on it. Fancy making a PR? What you've said sounds like a good idea to implement.

Will do. I'll put something up after this holiday weekend.
Cheers.