terminated by signal SIGSEGV (Address boundary error) if import onnxruntime before onnxruntime-genai

Cannot create og.Model if user import onnxruntime before the onnxruntime-gneai.
The job is terminated by signal SIGSEGV (Address boundary error)

import onnxruntime as ort

def genai_run(prompt, model_path, max_length=200):

    import time

    import onnxruntime_genai as og

    print("Loading model...")
    app_started_timestamp = time.time()
    model = og.Model(model_path)
    model_loaded_timestamp = time.time()
    print("Model loaded in {:.2f} seconds".format(model_loaded_timestamp - app_started_timestamp))
    tokenizer = og.Tokenizer(model)
    tokenizer_stream = tokenizer.create_stream()
    input_tokens = tokenizer.encode(prompt)
    started_timestamp = time.time()

    print("Creating generator ...")
    params = og.GeneratorParams(model)
    params.set_search_options(
        {
            "do_sample": False,
            "max_length": max_length,
            "min_length": 0,
            "top_p": 0.9,
            "top_k": 40,
            "temperature": 1.0,
            "repetition_penalty": 1.0,
        }
    )
    params.input_ids = input_tokens
    generator = og.Generator(model, params)
    print("Generator created")

    first = True
    new_tokens = []

    while not generator.is_done():
        generator.compute_logits()
        generator.generate_next_token()
        if first:
            first_token_timestamp = time.time()
            first = False

        new_token = generator.get_next_tokens()[0]
        print(tokenizer_stream.decode(new_token), end="")
        new_tokens.append(new_token)

    run_time = time.time() - started_timestamp
    print(
        f"Prompt tokens: {len(input_tokens)}, New tokens: {len(new_tokens)},"
        f" Time to first: {(first_token_timestamp - started_timestamp):.2f}s,"
        f" New tokens per second: {len(new_tokens)/run_time:.2f} tps"
    )

model_path = "xxxx"
genai_run("helo world", model_path)

How to reproduce:

python -m onnxruntime_genai.models.builder -m microsoft/phi-2 -e cpu -p int4 -o ./models/phi2
run above scripts

Not only onnxruntime, it failed when import torch, transformer.

Could you share more details on this error? And if possible, a stacktrace. That would help me reproduce this.
I tried running your script and was able to successfully execute it. Here is the output:
Loading model...
Model loaded in 15.46 seconds
Creating generator ...
Generator created
.

In the end, the friends realized that their journey had not only deepened their understanding of the world but also strengthened their bond. They had learned the importance of embracing diversity, respecting different perspectives, and finding common ground.

As they bid farewell to the Enchanted Forest, they carried with them the memories of their adventure and the wisdom gained from their encounters. They knew that their shared experiences would forever shape their lives and inspire them to continue exploring the wonders of the world.

And so, the three friends returned to their small town, forever changed by their journey. They became advocates for unity, spreading the message of acceptance and understanding wherever they went. Their story became a testament to the power of friendship, curiosity, and the pursuit of knowledge.

As they looked back on their adventure, they realized that the Enchanted Forest had not only taught them about the world but also about themselves. They had discovered their own strengths, overcome their fears, andPrompt tokens: 3, New tokens: 197, Time to first: 0.25s, New tokens per second: 14.55 tps

I am using onnxruntime-genai 0.1.0 from PyPI for this test. Could you share the following information:
Platform: windows, linux...
ort version
ort-genai version
python version
torch version
transformers version
stacktrace if possible

Platform: linux ubuntu20.04

ort version: onnxruntime-gpu 1.17.1

ort-genai version:  onnxruntime-genai-cuda (pip install onnxruntime-genai-cuda --pre --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/)
I tested cpu version of ort-genai just now and it worked for me. The failure happened on genai-cuda version no matter we target to run the mode in cpu or gpu.

python version: 3.8

torch version: 2.2.2

transformers version: 4.39.3

stacktrace if possible: only terminated by signal SIGSEGV (Address boundary error) was throwed when model = og.Model(model_path)

Thanks for you trying @baijumeswani :) , please let me know if anything is missed. Thanks!!

Ok, I was able to reproduce the problem on my end. I was also able to narrow does the problem to be related to std::filesystem.
Summary of the problem:

onnxruntime-genai-cuda is built with GCC 8.5.
onnxruntime-genai (cpu variant) is built with GCC 12.
onnxruntime-gpu and probably torch/transformers use GCC > 8 for their build.

The problem is introduced because when importing onnxruntime/torch/transformers first, the symbols for std::filesystem are loaded from libstdc++.so.6. These symbols are incompatible with the symbols needed for std::filesystem for GCC 8. There is more meaningful information provided here: https://bugs.launchpad.net/ubuntu/+source/gcc-8/+bug/1824721/comments/6
I think to resolve this problem, we might need publish a patch release that is built with a higher GCC version.
cc @jchen351

Platform: linux ubuntu20.04
ort version: onnxruntime-gpu 1.17.1
ort-genai version:  onnxruntime-genai-cuda (pip install onnxruntime-genai-cuda --pre --index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-genai/pypi/simple/)

I can reproduce the problem on Mariner2 Linux. Supposedly the two packages use the same compiler and there is no discrepancy between them.  I could see the genai package uses a fs implementation from itself, instead of libstdc++.

I am rebuilding the packages with symbols.

Now I have an onnxruntime-gpu  debug package, I don't have a genai debug package yet. So I still do not have enough clues.  But I see when it crashed the callstack has both packages' native code there. It seems that the onnxruntime-gpu package somehow calls into  GenAI, which I didn't expect to happen. Is it possible that the onnxruntime-gpu code constructed a std::filesystem::path object then passed it to the GenAI package's C++ code?

Do the two packages pass C++ objects around?

No, there should be no interaction between the two packages.
My hypothesis is that this is caused because we statically link against stdc++fs and when importing any python module that requires libstdc++.so, the symbols are not distinguishable (between the statically linked code and the shared lib) and ort-genai invokes std::filesystem symbols from libstdc++.so which would result in the crash.

I am guessing that if we tried doing this same thing on an ubuntu 18.04 machine, we wouldn't see this crash.

But when it crashed in the callstack I see both packages' native code, which is abnormal.

ort-genai calls into onnxruntime.so library which is embedded inside the ort-genai python package. Are you sure you saw native code from ort python package in the callstack? Or from ort-genai calling into ort.so inside the ort-genai python package? Could you paste your callstack?

Native code from ort python package in the callstack. I am 100% sure on that.

Here is the callstack.
#0  0x00007fff8c858d57 in std::filesystem::__cxx11::path::~path() ()
   from /home/chasun/.local/lib/python3.9/site-packages/onnxruntime_genai/onnxruntime_genai.cpython-39-x86_64-linux-gnu.so
#1  0x00007fff8c858d74 in std::filesystem::__cxx11::path::~path() ()
   from /home/chasun/.local/lib/python3.9/site-packages/onnxruntime_genai/onnxruntime_genai.cpython-39-x86_64-linux-gnu.so
#2  0x00007fff8c836972 in ?? ()
   from /home/chasun/.local/lib/python3.9/site-packages/onnxruntime_genai/onnxruntime_genai.cpython-39-x86_64-linux-gnu.so
#3  0x00007fff8c84159a in ?? ()
   from /home/chasun/.local/lib/python3.9/site-packages/onnxruntime_genai/onnxruntime_genai.cpython-39-x86_64-linux-gnu.so
#4  0x00007fff8c84fb5f in ?? ()
   from /home/chasun/.local/lib/python3.9/site-packages/onnxruntime_genai/onnxruntime_genai.cpython-39-x86_64-linux-gnu.so
#5  0x00007ffff7d849d2 in cfunction_call (func=0x7fff8fd0fef0, args=<optimized out>, kwargs=<optimized out>)
    at Objects/methodobject.c:543
#6  0x00007ffff7d68390 in _PyObject_MakeTpCall (tstate=0x555555559b80, callable=0x7fff8fd0fef0, args=0x7fffffffd7e0,
    nargs=2, keywords=0x0) at Objects/call.c:191
#7  0x00007ffff7cc351c in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fffffffd7e0,
    callable=0x7fff8fd0fef0, tstate=0x555555559b80) at ./Include/cpython/abstract.h:116
#8  _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fffffffd7e0, callable=0x7fff8fd0fef0,
    tstate=0x555555559b80) at ./Include/cpython/abstract.h:103
#9  method_vectorcall (method=<optimized out>, args=0x7ffff784a838, nargsf=<optimized out>, kwnames=0x0)
    at Objects/classobject.c:83
#10 0x00007ffff7d8f6a0 in slot_tp_init (self=self@entry=0x7fff8fcffa70, args=args@entry=0x7ffff784a820,
    kwds=kwds@entry=0x0) at Objects/typeobject.c:6974
#11 0x00007ffff7d8e59f in type_call (type=<optimized out>, args=0x7ffff784a820, kwds=0x0) at Objects/typeobject.c:1028
#12 0x00007ffff528b911 in pybind11::detail::pybind11_meta_call (type=0x55555619b370, args=0x7ffff784a820, kwargs=0x0)
--Type <RET> for more, q to quit, c to continue without paging--
    at /build/Debug/_deps/pybind11_project-src/include/pybind11/detail/class.h:187
#13 0x00007ffff7d68390 in _PyObject_MakeTpCall (tstate=0x555555559b80, callable=0x55555619b370, args=0x5555555904b0,
    nargs=1, keywords=0x0) at Objects/call.c:191
#14 0x00007ffff7daf4ee in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775809, args=0x5555555904b0,
    callable=0x55555619b370, tstate=<optimized out>) at ./Include/cpython/abstract.h:116
#15 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775809, args=0x5555555904b0, callable=0x55555619b370,
    tstate=<optimized out>) at ./Include/cpython/abstract.h:103
#16 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775809, args=0x5555555904b0, callable=0x55555619b370)
    at ./Include/cpython/abstract.h:127
#17 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x555555559b80)
    at Python/ceval.c:5077
#18 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0x5555555902a0, throwflag=<optimized out>)
    at Python/ceval.c:3489
#19 0x00007ffff7da9b2c in _PyEval_EvalFrame (throwflag=0, f=0x5555555902a0, tstate=0x555555559b80)
    at ./Include/internal/pycore_ceval.h:40
#20 _PyEval_EvalCode (tstate=tstate@entry=0x555555559b80, _co=<optimized out>, globals=<optimized out>,
    locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x5555555b3f80,
    kwcount=0, kwstep=1, defs=0x7ffff784ada8, defcount=1, kwdefs=0x0, closure=0x0, name=0x7ffff781d5b0,
    qualname=0x7ffff781d5b0) at Python/ceval.c:4329
#21 0x00007ffff7d68ab5 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:396
#22 0x00007ffff7daaaaa in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x5555555b3f70,
    callable=0x7ffff78cc160, tstate=0x555555559b80) at ./Include/cpython/abstract.h:118
#23 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x5555555b3f70, callable=<optimized out>)
    at ./Include/cpython/abstract.h:127
#24 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x555555559b80)
    at Python/ceval.c:5077
#25 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0x5555555b3e00, throwflag=<optimized out>)
    at Python/ceval.c:3520
--Type <RET> for more, q to quit, c to continue without paging--
#26 0x00007ffff7da9b2c in _PyEval_EvalFrame (throwflag=0, f=0x5555555b3e00, tstate=0x555555559b80)
    at ./Include/internal/pycore_ceval.h:40
#27 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=<optimized out>,
    args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x0, kwcount=0, kwstep=2, defs=0x0,
    defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at Python/ceval.c:4329
#28 0x00007ffff7e1f675 in _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>,
    locals=0x7ffff788d440, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>, kwargs=0x0,
    kwcount=0, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at Python/ceval.c:4361
#29 0x00007ffff7e1f60d in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>,
    args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0,
    closure=0x0) at Python/ceval.c:4377
#30 0x00007ffff7e1f5bf in PyEval_EvalCode (co=co@entry=0x7ffff781f500, globals=globals@entry=0x7ffff788d440,
    locals=locals@entry=0x7ffff788d440) at Python/ceval.c:828
#31 0x00007ffff7e326d4 in run_eval_code_obj (tstate=0x555555559b80, co=0x7ffff781f500, globals=0x7ffff788d440,
    locals=0x7ffff788d440) at Python/pythonrun.c:1221
#32 0x00007ffff7e32666 in run_mod (mod=<optimized out>, filename=<optimized out>, globals=0x7ffff788d440,
    locals=0x7ffff788d440, flags=<optimized out>, arena=<optimized out>) at Python/pythonrun.c:1242
#33 0x00007ffff7d1ded0 in pyrun_file (fp=fp@entry=0x555555559520, filename=filename@entry=0x7ffff790b760,
    start=start@entry=257, globals=globals@entry=0x7ffff788d440, locals=locals@entry=0x7ffff788d440,
    closeit=closeit@entry=1, flags=0x7fffffffe098) at Python/pythonrun.c:1140
#34 0x00007ffff7d1dc27 in pyrun_simple_file (flags=0x7fffffffe098, closeit=1, filename=0x7ffff790b760,
    fp=0x555555559520) at Python/pythonrun.c:450
#35 PyRun_SimpleFileExFlags (fp=fp@entry=0x555555559520, filename=<optimized out>, closeit=closeit@entry=1,
    flags=flags@entry=0x7fffffffe098) at Python/pythonrun.c:483
#36 0x00007ffff7d1e9c3 in PyRun_AnyFileExFlags (fp=fp@entry=0x555555559520, filename=<optimized out>,
    closeit=closeit@entry=1, flags=flags@entry=0x7fffffffe098) at Python/pythonrun.c:92
#37 0x00007ffff7e3acbd in pymain_run_file (cf=0x7fffffffe098, config=0x55555555b580) at Modules/main.c:373
#38 pymain_run_python (exitcode=0x7fffffffe090) at Modules/main.c:598
#39 Py_RunMain () at Modules/main.c:677
--Type <RET> for more, q to quit, c to continue without paging--
#40 0x00007ffff7e3a85d in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:731
#41 0x00007ffff7a6e57d in __libc_start_call_main (main=main@entry=0x555555555050 <main>, argc=argc@entry=2,
    argv=argv@entry=0x7fffffffe2b8) at ../sysdeps/nptl/libc_start_call_main.h:58
#42 0x00007ffff7a6e630 in __libc_start_main_impl (main=0x555555555050 <main>, argc=2, argv=0x7fffffffe2b8,
    init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe2a8)
    at ../csu/libc-start.c:392
#43 0x0000555555555081 in _start () at ../sysdeps/x86_64/start.S:115

The function call at the line "#12 0x00007ffff528b911 in pybind11::detail::pybind11_meta_call" was from ORT's python package.
pybind11_meta_call + 77 in section .text of /home/chasun/.local/lib/python3.9/site-packages/onnxruntime/capi/onnxruntime_pybind11_state.cpython-39-x86_64-linux-gnu.so

It is unexpected for ort and ort-genai to share any objects specially since the python script does not invoke anything on onnxruntime (except for importing it and the crash does not happen during importing onnxruntime).

I got a genai debug package, but the callstack is wired.  But, I found an issue.
When I ran
nm -C -g --defined-only /home/chasun/.local/lib/python3.9/site-packages/onnxruntime/capi/onnxruntime_pybind11_state.cpython-39-x86_64-linux-gnu.so

It only prints two things:
00000000000e8595 T PyInit_onnxruntime_pybind11_state
0000000000000000 A VERS_1.0

However, with the same command the GenAI package's binary prints a lot of stdc++ symbols.  Which means you didn't hide the symbols per suggestion from https://gcc.gnu.org/wiki/Visibility

Does GenAI has this file? python/version_script.lds

The symbol exported from GenAI should be PyInit_onnxruntime_genai

Got a new callstack with GenAI's debug symbols.
#0  0x00007fff8cc5ec25 in std::__cxx1998::vector<std::filesystem::__cxx11::path::_Cmpt, std::allocator<std::filesystem::__cxx11::path::_Cmpt> >::~vector (this=0x23, __in_chrg=<optimized out>) at /usr/include/c++/8/bits/stl_vector.h:567
#1  0x00007fff8cc5bb26 in std::filesystem::__cxx11::path::~path (this=0x3, __in_chrg=<optimized out>)
    at /usr/include/c++/8/bits/fs_path.h:209
#2  0x00007fff8cc62fb8 in std::filesystem::__cxx11::path::_Cmpt::~_Cmpt (this=0x3, __in_chrg=<optimized out>)
    at /usr/include/c++/8/bits/fs_path.h:644
#3  0x00007fff8cc62fd3 in std::_Destroy<std::filesystem::__cxx11::path::_Cmpt> (__pointer=0x3)
    at /usr/include/c++/8/bits/stl_construct.h:98
#4  0x00007fff8cc61efd in std::_Destroy_aux<false>::__destroy<std::filesystem::__cxx11::path::_Cmpt*> (__first=0x3,
    __last=0x0) at /usr/include/c++/8/bits/stl_construct.h:108
#5  0x00007fff8cc60cd5 in std::_Destroy<std::filesystem::__cxx11::path::_Cmpt*> (__first=0x3, __last=0x0)
    at /usr/include/c++/8/bits/stl_construct.h:137
#6  0x00007fff8cc5fcc5 in std::_Destroy<std::filesystem::__cxx11::path::_Cmpt*, std::filesystem::__cxx11::path::_Cmpt>
    (__first=0x3, __last=0x0) at /usr/include/c++/8/bits/stl_construct.h:206
#7  0x00007fff8cc5ec3b in std::__cxx1998::vector<std::filesystem::__cxx11::path::_Cmpt, std::allocator<std::filesystem::__cxx11::path::_Cmpt> >::~vector (this=0x7fffffffcdb0, __in_chrg=<optimized out>)
    at /usr/include/c++/8/bits/stl_vector.h:567
#8  0x00007fff8cc5bb26 in std::filesystem::__cxx11::path::~path (this=0x7fffffffcd90, __in_chrg=<optimized out>)
    at /usr/include/c++/8/bits/fs_path.h:209
#9  0x00007fff8cc71291 in std::make_unique<Generators::Config, char const*&> ()
    at /usr/include/c++/8/bits/unique_ptr.h:835
#10 0x00007fff8cc6cd0c in Generators::CreateModel (ort_env=..., config_path=0x7fffffffcfc0 "xxxx")
    at /ort_genai_src/src/models/model.cpp:345
#11 0x00007fff8cbee330 in Generators::<lambda(const string&)>::operator()(const std::__cxx11::string &) const (
    __closure=0x5555561a0c58, config_path="xxxx") at /ort_genai_src/src/python/python.cpp:212
#12 0x00007fff8cbefee6 in pybind11::detail::initimpl::factory<Generators::pybind11_init_onnxruntime_genai(pybind11::module_&)::<lambda(const string&)>, pybind11::detail::void_type (*)(), std::shared_ptr<Generators::Model>(const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&), pybind11::detail::void_type()>::<lambda(pybind11::detail::value_and_holder&, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)>::op--Type <RET> for more, q to quit, c to continue without paging--
erator()(pybind11::detail::value_and_holder &, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > &) const (this=0x5555561a0c58, v_h=..., args#0="xxxx")
    at /ort_genai_src/build/cuda/_deps/pybind11_project-src/include/pybind11/detail/init.h:297
#13 0x00007fff8cbf3a97 in pybind11::detail::argument_loader<pybind11::detail::value_and_holder&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>::call_impl<void, pybind11::detail::initimpl::factory<Func, pybind11::detail::void_type (*)(), Return(Args ...)>::execute(Class&, const Extra& ...) && [with Class = pybind11::class_<Generators::Model, std::shared_ptr<Generators::Model> >; Extra = {}; Func = Generators::pybind11_init_onnxruntime_genai(pybind11::module_&)::<lambda(const string&)>; Return = std::shared_ptr<Generators::Model>; Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&}]::<lambda(pybind11::detail::value_and_holder&, const std::__cxx11::basic_string<char>&)>&, 0, 1, pybind11::detail::void_type>(pybind11::detail::initimpl::factory<Generators::pybind11_init_onnxruntime_genai(pybind11::module_&)::<lambda(const string&)>, pybind11::detail::void_type (*)(), std::shared_ptr<Generators::Model>(const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&), pybind11::detail::void_type()>::<lambda(pybind11::detail::value_and_holder&, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)> &, std::index_sequence, pybind11::detail::void_type &&) (
    this=0x7fffffffcfb0, f=...) at /ort_genai_src/build/cuda/_deps/pybind11_project-src/include/pybind11/cast.h:1439
#14 0x00007fff8cbf3985 in pybind11::detail::argument_loader<pybind11::detail::value_and_holder&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>::call<void, pybind11::detail::void_type, pybind11::detail::initimpl::factory<Func, pybind11::detail::void_type (*)(), Return(Args ...)>::execute(Class&, const Extra& ...) && [with Class = pybind11::class_<Generators::Model, std::shared_ptr<Generators::Model> >; Extra = {}; Func = Generators::pybind11_init_onnxruntime_genai(pybind11::module_&)::<lambda(const string&)>; Return = std::shared_ptr<Generators::Model>; Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&}]::<lambda(pybind11::detail::value_and_holder&, const std::__cxx11::basic_string<char>&)>&>(pybind11::detail::initimpl::factory<Generators::pybind11_init_onnxruntime_genai(pybind11::module_&)::<lambda(const string&)>, pybind11::detail::void_type (*)(), std::shared_ptr<Generators::Model>(const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&), pybind11::detail::void_type()>::<lambda(pybind11::detail::value_and_holder&, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)> &) (this=0x7fffffffcfb0, f=...)
    at /ort_genai_src/build/cuda/_deps/pybind11_project-src/include/pybind11/cast.h:1413
#15 0x00007fff8cbf3463 in pybind11::cpp_function::<lambda(pybind11::detail::function_call&)>::operator()(pybind11::detail::function_call &) const (this=0x0, call=...)
--Type <RET> for more, q to quit, c to continue without paging--
    at /ort_genai_src/build/cuda/_deps/pybind11_project-src/include/pybind11/pybind11.h:249
#16 0x00007fff8cbf34e6 in pybind11::cpp_function::<lambda(pybind11::detail::function_call&)>::_FUN(pybind11::detail::function_call &) () at /ort_genai_src/build/cuda/_deps/pybind11_project-src/include/pybind11/pybind11.h:224
#17 0x00007fff8cc049bb in pybind11::cpp_function::dispatcher (self=0x7fff90097ed0, args_in=0x7ffff77cc500,
    kwargs_in=0x0) at /ort_genai_src/build/cuda/_deps/pybind11_project-src/include/pybind11/pybind11.h:929
#18 0x00007ffff7d849d2 in cfunction_call (func=0x7fff9009e270, args=<optimized out>, kwargs=<optimized out>)
    at Objects/methodobject.c:543
#19 0x00007ffff7d68390 in _PyObject_MakeTpCall (tstate=0x555555559bc0, callable=0x7fff9009e270, args=0x7fffffffd780,
    nargs=2, keywords=0x0) at Objects/call.c:191
#20 0x00007ffff7cc351c in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fffffffd780,
    callable=0x7fff9009e270, tstate=0x555555559bc0) at ./Include/cpython/abstract.h:116
#21 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=2, args=0x7fffffffd780, callable=0x7fff9009e270,
    tstate=0x555555559bc0) at ./Include/cpython/abstract.h:103
#22 method_vectorcall (method=<optimized out>, args=0x7ffff784a808, nargsf=<optimized out>, kwnames=0x0)
    at Objects/classobject.c:83
#23 0x00007ffff7d8f6a0 in slot_tp_init (self=self@entry=0x7fff9007b270, args=args@entry=0x7ffff784a7f0,
    kwds=kwds@entry=0x0) at Objects/typeobject.c:6974
#24 0x00007ffff7d8e59f in type_call (type=<optimized out>, args=0x7ffff784a7f0, kwds=0x0) at Objects/typeobject.c:1028
#25 0x00007fff8cbfe650 in pybind11::detail::pybind11_meta_call (type=0x5555561a0780, args=0x7ffff784a7f0, kwargs=0x0)
    at /ort_genai_src/build/cuda/_deps/pybind11_project-src/include/pybind11/detail/class.h:187
#26 0x00007ffff7d68390 in _PyObject_MakeTpCall (tstate=0x555555559bc0, callable=0x5555561a0780, args=0x555555e01050,
    nargs=1, keywords=0x0) at Objects/call.c:191
#27 0x00007ffff7daf4ee in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775809, args=0x555555e01050,
    callable=0x5555561a0780, tstate=<optimized out>) at ./Include/cpython/abstract.h:116
#28 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775809, args=0x555555e01050, callable=0x5555561a0780,
    tstate=<optimized out>) at ./Include/cpython/abstract.h:103
#29 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775809, args=0x555555e01050, callable=0x5555561a0780)
    at ./Include/cpython/abstract.h:127
#30 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x555555559bc0)
--Type <RET> for more, q to quit, c to continue without paging--
    at Python/ceval.c:5077
#31 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0x555555e00e40, throwflag=<optimized out>)
    at Python/ceval.c:3489
#32 0x00007ffff7da9b2c in _PyEval_EvalFrame (throwflag=0, f=0x555555e00e40, tstate=0x555555559bc0)
    at ./Include/internal/pycore_ceval.h:40
#33 _PyEval_EvalCode (tstate=tstate@entry=0x555555559bc0, _co=<optimized out>, globals=<optimized out>,
    locals=locals@entry=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x5555555b3ec0,
    kwcount=0, kwstep=1, defs=0x7ffff784ad78, defcount=1, kwdefs=0x0, closure=0x0, name=0x7ffff781d5f0,
    qualname=0x7ffff781d5f0) at Python/ceval.c:4329
#34 0x00007ffff7d68ab5 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:396
#35 0x00007ffff7daaaaa in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x5555555b3eb0,
    callable=0x7ffff78cc0d0, tstate=0x555555559bc0) at ./Include/cpython/abstract.h:118
#36 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x5555555b3eb0, callable=<optimized out>)
    at ./Include/cpython/abstract.h:127
#37 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x555555559bc0)
    at Python/ceval.c:5077
#38 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=0x5555555b3d40, throwflag=<optimized out>)
    at Python/ceval.c:3520
#39 0x00007ffff7da9b2c in _PyEval_EvalFrame (throwflag=0, f=0x5555555b3d40, tstate=0x555555559bc0)
    at ./Include/internal/pycore_ceval.h:40
#40 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=<optimized out>,
    args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x0, kwcount=0, kwstep=2, defs=0x0,
    defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at Python/ceval.c:4329
#41 0x00007ffff7e1f675 in _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>,
    locals=0x7ffff788d480, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>, kwargs=0x0,
    kwcount=0, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at Python/ceval.c:4361
#42 0x00007ffff7e1f60d in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>,
    args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0,
--Type <RET> for more, q to quit, c to continue without paging--
    closure=0x0) at Python/ceval.c:4377
#43 0x00007ffff7e1f5bf in PyEval_EvalCode (co=co@entry=0x7ffff781f500, globals=globals@entry=0x7ffff788d480,
    locals=locals@entry=0x7ffff788d480) at Python/ceval.c:828
#44 0x00007ffff7e326d4 in run_eval_code_obj (tstate=0x555555559bc0, co=0x7ffff781f500, globals=0x7ffff788d480,
    locals=0x7ffff788d480) at Python/pythonrun.c:1221
#45 0x00007ffff7e32666 in run_mod (mod=<optimized out>, filename=<optimized out>, globals=0x7ffff788d480,
    locals=0x7ffff788d480, flags=<optimized out>, arena=<optimized out>) at Python/pythonrun.c:1242
#46 0x00007ffff7d1ded0 in pyrun_file (fp=fp@entry=0x555555559520, filename=filename@entry=0x7ffff790b760,
    start=start@entry=257, globals=globals@entry=0x7ffff788d480, locals=locals@entry=0x7ffff788d480,
    closeit=closeit@entry=1, flags=0x7fffffffe038) at Python/pythonrun.c:1140
#47 0x00007ffff7d1dc27 in pyrun_simple_file (flags=0x7fffffffe038, closeit=1, filename=0x7ffff790b760,
    fp=0x555555559520) at Python/pythonrun.c:450
#48 PyRun_SimpleFileExFlags (fp=fp@entry=0x555555559520, filename=<optimized out>, closeit=closeit@entry=1,
    flags=flags@entry=0x7fffffffe038) at Python/pythonrun.c:483
#49 0x00007ffff7d1e9c3 in PyRun_AnyFileExFlags (fp=fp@entry=0x555555559520, filename=<optimized out>,
    closeit=closeit@entry=1, flags=flags@entry=0x7fffffffe038) at Python/pythonrun.c:92
#50 0x00007ffff7e3acbd in pymain_run_file (cf=0x7fffffffe038, config=0x55555555b580) at Modules/main.c:373
#51 pymain_run_python (exitcode=0x7fffffffe030) at Modules/main.c:598
#52 Py_RunMain () at Modules/main.c:677
#53 0x00007ffff7e3a85d in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:731
#54 0x00007ffff7a6e57d in __libc_start_call_main (main=main@entry=0x555555555050 <main>, argc=argc@entry=2,
    argv=argv@entry=0x7fffffffe258) at ../sysdeps/nptl/libc_start_call_main.h:58
#55 0x00007ffff7a6e630 in __libc_start_main_impl (main=0x555555555050 <main>, argc=2, argv=0x7fffffffe258,
    init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe248)
    at ../csu/libc-start.c:392
#56 0x0000555555555081 in _start () at ../sysdeps/x86_64/start.S:115

Any updates? :)

A few solutions for us to explore:

Try making the pybind symbols private using version_script.lds (the way ort does).
If that does not help, we can try avoiding the use of std::filesystem
And try using a newer GCC compiler.

I'll work on finding the right path forward.

I think I can confirm the root cause is  https://stackoverflow.com/questions/63902528/program-crashes-when-filesystempath-is-destroyed  , because after I set a breakpoint at "std::filesystem::__cxx11::path::_M_split_cmpts()", I got the following stacktrace:
#0  0x00007ffff55e14a0 in std::filesystem::__cxx11::path::_M_split_cmpts() () from /lib/libstdc++.so.6
#1  0x00007fff8cc6efb6 in std::filesystem::__cxx11::path::path<char const*, std::filesystem::__cxx11::path> (this=0x7fffffffcdf0,
    __source=@0x7fffffffce58: 0x7fffffffd020 "xxxx") at /usr/include/c++/8/bits/fs_path.h:185
#2  0x00007fff8cc71234 in std::make_unique<Generators::Config, char const*&> () at /usr/include/c++/8/bits/unique_ptr.h:835
#3  0x00007fff8cc6cd0c in Generators::CreateModel (ort_env=..., config_path=0x7fffffffd020 "xxxx") at /ort_genai_src/src/models/model.cpp:345
#4  0x00007fff8cbee330 in Generators::<lambda(const string&)>::operator()(const std::__cxx11::string &) const (__closure=0x5555561a0ad8,
    config_path="xxxx") at /ort_genai_src/src/python/python.cpp:212
#5  0x00007fff8cbefee6 in pybind11::detail::initimpl::factory<Generators::pybind11_init_onnxruntime_genai(pybind11::module_&)::<lambda(const string&)>, pybind11::detail::void_type (*)(), std::shared_ptr<Generators::Model>(const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&), pybind11::detail::void_type()>::<lambda(pybind11::detail::value_and_holder&, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)>::operator()(pybind11::detail::value_and_holder &, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > &) const (this=0x5555561a0ad8, v_h=..., args#0="xxxx") at /ort_genai_src/build/cuda/_deps/pybind11_project-src/include/pybind11/detail/init.h:297
#6  0x00007fff8cbf3a97 in pybind11::detail::argument_loader<pybind11::detail::value_and_holder&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>::call_impl<void, pybind11::detail::initimpl::factory<Func, pybind11::detail::void_type (*)(), Return(Args ...)>::execute(Class&, const Extra& ...) && [with Class = pybind11::class_<Generators::Model, std::shared_ptr<Generators::Model> >; Extra = {}; Func = Generators::pybind11_init_onnxruntime_genai(pybind11::module_&)::<lambda(const string&)>; Return = std::shared_ptr<Generators::Model>; Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&}]::<lambda(pybind11::detail::value_and_holder&, const std::__cxx11::basic_string<char>&)>&, 0, 1, pybind11::detail::void_type>(pybind11::detail::initimpl::factory<Generators::pybind11_init_onnxruntime_genai(pybind11::module_&)::<lambda(const string&)>, pybind11::detail::void_type (*)(), std::shared_ptr<Generators::Model>(const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&), pybind11::detail::void_type()>::<lambda(pybind11::detail::value_and_holder&, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)> &, std::index_sequence, pybind11::detail::void_type &&) (this=0x7fffffffd010, f=...)
    at /ort_genai_src/build/cuda/_deps/pybind11_project-src/include/pybind11/cast.h:1439
#7  0x00007fff8cbf3985 in pybind11::detail::argument_loader<pybind11::detail::value_and_holder&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>::call<void, pybind11::detail::void_type, pybind11::detail::initimpl::factory<Func, pybind11::detail::void_type (*)(), Return(Args ...)>::execute(Class&, const Extra& ...) && [with Class = pybind11::class_<Generators::Model, std::shared_ptr<Generators::Model> >; Extra = {}; Func = Generators::pybind11_init_onnxruntime_genai(pybind11::module_&)::<lambda(const string&)>; Return = std::shared_ptr<Generators::Model>; Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&}]::<lambda(pybind11::detail::value_and_holder&, const std::__cxx11::basic_string<char>&)>&>(pybind11::detail::initimpl::factory<Generators::pybind11_init_onnxruntime_genai(pybind11::module_&)::<lambda(const string&)>, pybind11::detail::void_type (*)(), std::shared_ptr<Generators::Model>(const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&), pybind11::detail::void_type()>::<lambda(pybind11::detail::value_and_holder&, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)> &) (this=0x7fffffffd010, f=...) at /ort_genai_src/build/cuda/_deps/pybind11_project-src/include/pybind11/cast.h:1413
#8  0x00007fff8cbf3463 in pybind11::cpp_function::<lambda(pybind11::detail::function_call&)>::operator()(pybind11::detail::function_call &) const (
    this=0x0, call=...) at /ort_genai_src/build/cuda/_deps/pybind11_project-src/include/pybind11/pybind11.h:249
#9  0x00007fff8cbf34e6 in pybind11::cpp_function::<lambda(pybind11::detail::function_call&)>::_FUN(pybind11::detail::function_call &) ()
    at /ort_genai_src/build/cuda/_deps/pybind11_project-src/include/pybind11/pybind11.h:224
#10 0x00007fff8cc049bb in pybind11::cpp_function::dispatcher (self=0x7fff90095ed0, args_in=0x7ffff77cf580, kwargs_in=0x0)
    at /ort_genai_src/build/cuda/_deps/pybind11_project-src/include/pybind11/pybind11.h:929
#11 0x00007ffff7d849d2 in cfunction_call (func=0x7fff9009c220, args=<optimized out>, kwargs=<optimized out>) at Objects/methodobject.c:543
#12 0x00007ffff7d68390 in _PyObject_MakeTpCall (tstate=0x555555559b80, callable=0x7fff9009c220, args=0x7fffffffd7e0, nargs=2, keywords=0x0)
    at Objects/call.c:191

Though onnxruntime_genai.cpython-39-x86_64-linux-gnu.so contains a copy of the function:
$ nm -C  /home/chasun/.local/lib/python3.9/site-packages/onnxruntime_genai/onnxruntime_genai.cpython-39-x86_64-linux-gnu.so |grep std::filesystem::__cxx11::path::_M_split_cmpts
00000000002d2140 T std::filesystem::__cxx11::path::_M_split_cmpts()
00000000001ec838 t std::filesystem::__cxx11::path::_M_split_cmpts() [clone .cold.121]

At runtime from the callstack you can see actually an implementation from /lib/libstdc++.so.6 was used.  Given the layout of std::filesystem::path object between GCC 8 and higher GCC versions are different(and incompatible), we should not see one calls into another.

Thanks!

I started upgrading the GCC.


Respect Public Marketplace Appliance compatible ONE versions

Description
Public Marketplace Appliances have (an optional) parameter with list of compatible comma separated  OpenNebula versions for the particular appliance. E.g.:
https://github.com/OpenNebula/marketplace/blob/c4b42603f5b6beab5a36bb017a5b0dd9cbe00782/appliances/OpenNebula_Systems/kvm/4e3b2788-d174-4151-b026-94bb0b987cbb.yaml#L22
This list is just informative (for the web view) and is not respected right now in the OpenNebula.
The idea is to parse the non-empty opennebula_version string and as part of the monitoring action of the Public Marketplace, return back into the OpenNebula only appliances matching the installed version.
It's also necessary to review the opennebula_version in existing appliances metadata as part of this issue (drop or update the parameters).

Progress Status

[ ] Branch created
[ ] Code committed to development branch
[ ] Testing - QA
[ ] Documentation
[ ] Release notes - resolved issues, compatibility, known issues
[ ] Code committed to upstream release/hotfix branches
[ ] Documentation committed to upstream release/hotfix branches

Implemented on MP server.


get_activations for pytorch_estimator returns incorrect values when framework=True?

Describe the bug
The implementation of get_activations() for the pytorch estimator returns the value of x when framework=True.
https://github.com/Trusted-AI/adversarial-robustness-toolbox/blob/main/art/estimators/classification/pytorch.py#L853
Expected behavior
It should return the value of the tensor for x_preprocessed

Hi @keykholt Thank you very much for reporting this issue! We will fix it with ART 1.9.1


MekanismGenerators crash on Server

Issue description:
Crash when starting the Server.
Steps to reproduce:

Start the Server
Crash

Version (make sure you are on the latest version before reporting):
Forge: 2556
Mekanism: <IP_ADDRESS>7
If a (crash)log is relevant for this issue, link it here: (It's almost always relevant)
https://gist.github.com/D3nnis3n/8bb1d3b89222e0f9d90e78036ea23ecb

Thanks for reporting, but please use the search functionality of github as this is reported multiple times.
Closing as duplicate of #4827


Tags are displayed incorrectly or aren't displayed at all

ISSUE TYPE

Bug Report

COMPONENT NAME
UI

CLOUDSTACK VERSION
4.16.0

CONFIGURATION
XCP-NG 8.2
Advanced Zone
OS / ENVIRONMENT
Oracle Linux 8.4 - Management server, MYSQL 8.0.26
XCP-NG 8.2 - hypervisor
SUMMARY
Storage tags are displayed incorrectly:

Host tags aren't displayed

STEPS TO REPRODUCE
Add tag to storage

EXPECTED RESULTS
Storage's tags are displayed correctly. Host's tags are displayed.

ACTUAL RESULTS
Storage tags are displayed incorrectly. Host tags aren't displayed.

@edikevich Tags in the info card of the UI is different from host, storage tags. Please see #5798 fixes the behaviour

Fixed in https://github.com/apache/cloudstack/pull/5798


Add support for a special file to exclude a directory from being rendered

Is your feature request related to a problem? Please describe.
I have request that would reduce some logic needed in files if a var is not true. In this directory all files are wrapped with an if statement of if bootstrap_distribution == 'talos'. If you take a look around the ./bootstrap/templates folder you will see I am doing this quite a bit to exclude folders and files from being templated based on variables.
Describe the solution you'd like
I was thinking it would be cool to support like a .pathignore.j2 file where the contents would hold the logic if that folder, subfolders and files get templated. I could drop this file in ./bootstrap/templates/kubernetes/talos and omit the logic in the files.
{% bootstrap_distribution == 'talos' %}

I am not sure if this can be done in custom Loader but I am all ears if there is already a way to do this but needs some special sauce added on my end.

Thanks for the suggestion, I think your use-case is totally valid! I feel that the most "elegant" way of implementing such a feature would be to extend of the loader class with support for dynamic exclusions (i.e., allowing users to supply a function with type Callable[[Path], bool] that receives the currently processed input path returns True to continue processing or False to skip it. This also has the advantage that one has access to standard Python code for the checks and that all exclusions are defined centrally. I will try to come up with an early version of this as soon as possible.

That sounds like it could work. I might need to see an example of the implementation to wrap my brain around it. No rush on this, the currently method of having if statements in each files works for now but as my repo grows it's going to get a little cumbersome.

My idea is to allow something like this:
from pathlib import Path

class Loader:
    def __init__(self, data) -> None:
        self.data = data

    def filter_path(self, path: Path) -> bool:
        if (
            str(path).startswith("bootstrap/templates/kubernetes/talos")
            and self.data.bootstrap_distribution == "talos"
        ):
            return False

        return True

Or maybe the following (which would be more in line with the existing members of the loader class):
from pathlib import Path

class Loader:
    def __init__(self, data) -> None:
        self.data = data

    def path_filters(self):
        return [self._filter_talos]

    def _filter_talos(self, path: Path) -> bool:
        if (
            str(path).startswith("bootstrap/templates/kubernetes/talos")
            and self.data.bootstrap_distribution == "talos"
        ):
            return False

        return True

That's much better than the solution I came up with and makes it very extendable!

Hey @mirkolenz after thinking on this for the past day it would still be neat to support my original request of a "special file", let me explain.
The major benefit I see to having support for this is I don't need to manage the Loader on if/when paths are remove or changed. I could keep the logic coupled to the folder where the files are located. I have a lot of directories that could make use of a feature like this and having it wired into the loader file will increase the maintenance burden on my end to make sure that file is updated, also it might cause issues on if paths are changed and the loader it not updated.

I see! I will keep the decentralized use case in mind when designing the solution and provide you with a code example how to apply it to your use case.

Here is a draft for a loader that could support this with not-yet-released changes in makejinja:
from collections.abc import Callable
from pathlib import Path
from typing import Any

import makejinja
from jinja2.utils import import_string

class Loader:
    def __init__(self, data: dict[str, Any], config: makejinja.config.Config):
        self.data = data
        self.config = config

        self._excluded_dirs: set[Path] = set()

        for input_path in config.inputs:
            for filter_file in input_path.rglob(".mjfilter"):
                filter_func: Callable[[dict[str, Any]], bool] = import_string(
                    f"{filter_file}:main"
                )
                if filter_func(data) is False:
                    self._excluded_dirs.add(filter_file.parent)

    def path_filters(self):
        return [self._mjfilter_func]

    def _mjfilter_func(self, path: Path) -> bool:
        # Render the file if there is no parent directory that is excluded
        return not any(
            path.is_relative_to(excluded_dir) for excluded_dir in self._excluded_dirs
        )

The __init__ function now receives the complete config so that things like the input folders are available.
The loader then recursively searches for any file .mjfilter (just an example) in each input path.
Such a .mjfilter is a Python file that should contain a function main (may be changed) that receives the data dict and returns True if the folder contents shall be rendered and False if it shall be skipped.
The functions are then evaluated and each Path returning False is saved to a set.
The path filter then checks if the current file is contained in an excluded folder and consequently returns False in this case.

This is obviously not finalized, I just wanted to ask for your feedback before doing the actual implementation. What do you think?

For completeness, here is an example for a .mjfilter file:
def main(data):
    return data["bootstrap_distribution"] == "talos"

That would render the folder only if bootstrap_distribution is talos. To reverse the logic, one can use !=.

Using python over jinja makes a ton of more sense, I can't wait to try it out!

The mentioned changes are now live: https://github.com/mirkolenz/makejinja/releases/tag/v2.4.0
The code snippet provided by me should still hold. The only thing to keep in mind is that the .mjfilter files should probably be added to --exclude-pattern, otherwise they will be copied over.
Please let me know if the code works as expected. Also check whether the runtime is negatively impacted by adding path filters. Depending on the logic, it may have a huge impact since the functions are called for every file and directory that is matched by the --include-pattern. That is the reason why in my code example, the excluded paths are added to a set: To minimize runtime impact.
In the new version, I changed the default for --keep-trailing-newlines to true, so one needs to set --strip-trailing-newlines for the old behavior. It also means that this setting may be removed from your makejinja.toml if you would like to.
Lastly, I would like to note that I renamed "loaders" to "plugins" since that more closely resembles what they are actually used for. It also makes it easier for users to differentiate between the Jinja loaders and the makejinja plugins. This change should be backwards-compatible (i.e., I added aliases for all deprecated names like AbstractLoader and so on). If something breaks with the new version then I consider this to be a bug that should be fixed.

2.5.0 looks good! I was able to migrate over to the Plugins without an issue.
However I am having trouble with implementing the mjfilters, I tried following along with what you posted eariler.
More information here: https://github.com/onedr0p/flux-cluster-template/pull/1222

Ok so the problem is that Jinja's import_string expects a module name like path.to.file, but it receives a path like path/to/file instead. This dot-based notation means that the filter files need to follow Python file naming conventions, so you need to rename them to something like _mjfilter.py. I currently do not have the time to test it, but does the following work with these modifications?
from collections.abc import Callable
from pathlib import Path
from typing import Any

import makejinja
from jinja2.utils import import_string

class Plugin:
    def __init__(self, data: dict[str, Any], config: makejinja.config.Config):
        self.data = data
        self.config = config

        self._excluded_dirs: set[Path] = set()

        for input_path in config.inputs:
            for filter_file in input_path.rglob("_mjfilter.py"):
                filter_file = filter_file.relative_to(Path.cwd())
                filter_file = filter_file.with_suffix("")
                filter_module = str(filter_file).replace("/", ".")
                filter_func: Callable[[dict[str, Any]], bool] = import_string(
                    f"{filter_module}:main"
                )
                if filter_func(data) is False:
                    self._excluded_dirs.add(filter_file.parent)

    def path_filters(self):
        return [self._mjfilter_func]

    def _mjfilter_func(self, path: Path) -> bool:
        # Render the file if there is no parent directory that is excluded
        return not any(
            path.is_relative_to(excluded_dir) for excluded_dir in self._excluded_dirs
        )

If it does not work, you need to modify how the module name is constructed (i.e., all lines where filter_file and filter_module is assigned some value).
If you do not get it to work, could you maybe assemble a minimal working example that I can tinker with and upload it as a ZIP file? That would greatly help me in constructing the right code.

I can't even get this line to work 😄
class Plugin(makejinja.plugin.Plugin):
    def __init__(self, data: dict[str, Any], config: makejinja.config.Config):
        self.data = data
        self.config = config

Logs
Traceback (most recent call last):
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/bin/makejinja", line 8, in <module>
    sys.exit(makejinja_cli())
             ^^^^^^^^^^^^^^^
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/lib/python3.11/site-packages/rich_click/rich_command.py", line 126, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/lib/python3.11/site-packages/typed_settings/cli_click.py", line 247, in new_func
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/lib/python3.11/site-packages/makejinja/cli.py", line 37, in makejinja_cli
    makejinja(config)
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/lib/python3.11/site-packages/makejinja/app.py", line 60, in makejinja
    plugins.append(load_plugin(plugin_name, env, data, config))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/lib/python3.11/site-packages/makejinja/app.py", line 358, in load_plugin
    env.globals.update(plugin.data())
                       ^^^^^^^^^^^^^
TypeError: 'dict' object is not callable

Actually I renamed these and it seems to have gotten pass that error. Maybe some shadowing happening.
    def __init__(self, data: dict[str, Any], config: makejinja.config.Config):
        self.user_data = data
        self.user_config = config

If/when you have a moment I included a minimum example of what I am trying to achieve here, you should just be able to change config in bootstrap/vars/config.yaml and then run makejinja to test.
https://github.com/onedr0p/makejinja-mjfilter-example

Thanks for the MWE, I will investigate as soon as I have some spare time.
The error with the variable data was an error from my side: A function called data is part of the plugin protocol. If a class has such an attribute, makejinja tries to execute it, leading to the observed error. Probably the safest way is to prefix all private attributes with an underscore.

I finally had the time to tinker with it. I got it to work and given the makejinja output it seems to correctly apply the filters with my modifications. There are two changes that need to be done to the MWE:

The filter files were named _mkfilter.py, but the pattern matched _mjfilter.py. Thus, I renamed all filter files.
I used the following code for the plugin:

import importlib.util
import sys
from collections.abc import Callable
from pathlib import Path
from typing import Any

import makejinja

def import_filter(file: Path) -> Callable[[dict[str, Any]], bool]:
    # https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly

    module_path = file.relative_to(Path.cwd()).with_suffix("")
    module_name = str(module_path).replace("/", ".")
    spec = importlib.util.spec_from_file_location(module_name, file)

    assert spec is not None

    module = importlib.util.module_from_spec(spec)
    sys.modules[module_name] = module

    assert spec.loader is not None

    spec.loader.exec_module(module)

    return module.main

class Plugin(makejinja.plugin.Plugin):
    def __init__(self, data: dict[str, Any], config: makejinja.config.Config):
        self._data = data
        self._config = config

        self._excluded_dirs: set[Path] = set()

        for input_path in config.inputs:
            for filter_file in input_path.rglob("_mjfilter.py"):
                filter_func = import_filter(filter_file)
                if filter_func(data) is False:
                    self._excluded_dirs.add(filter_file.parent)

    def path_filters(self):
        return [self._mjfilter_func]

    def _mjfilter_func(self, path: Path) -> bool:
        # Render the file if there is no parent directory that is excluded
        return not any(
            path.is_relative_to(excluded_dir) for excluded_dir in self._excluded_dirs
        )

I did not get it to work using Jinja's import_string since this assumes that Python modules are imported. AFAIK, this only works if all "parent modules" include an __init__.py file which is not the case here. Thus, I resorted to an approach that is a bit more involved but seems to be the officially recommended way of dynamically importing arbitrary files.

Here is the output I got when running `makejinja` in the MWE (click to expand):
Load data './bootstrap/vars/config.yaml'
Create dir './bootstrap/templates/kubernetes' -> './kubernetes'
Create dir './bootstrap/templates/kubernetes/apps' ->
'./kubernetes/apps'
Create dir './bootstrap/templates/kubernetes/apps/test0' ->
'./kubernetes/apps/test0'
Create dir './bootstrap/templates/kubernetes/apps/test0/__pycache__' ->
'./kubernetes/apps/test0/__pycache__'
Copy file './bootstrap/templates/kubernetes/apps/test0/__pycache__/_mjfilter.cpython-311.pyc' ->
'./kubernetes/apps/test0/__pycache__/_mjfilter.cpython-311.pyc'
Skip excluded path './bootstrap/templates/kubernetes/apps/test0/_mjfilter.py'
Render file './bootstrap/templates/kubernetes/apps/test0/app.yaml.j2' ->
'./kubernetes/apps/test0/app.yaml'
Create dir './bootstrap/templates/kubernetes/apps/test0/config' ->
'./kubernetes/apps/test0/config'
Render file './bootstrap/templates/kubernetes/apps/test0/config/config.yaml.j2' ->
'./kubernetes/apps/test0/config/config.yaml'
Skip excluded path './bootstrap/templates/kubernetes/apps/test1'
Skip excluded path './bootstrap/templates/kubernetes/apps/test1/__pycache__'
Skip excluded path './bootstrap/templates/kubernetes/apps/test1/__pycache__/_mjfilter.cpython-311.pyc'
Skip excluded path './bootstrap/templates/kubernetes/apps/test1/_mjfilter.py'
Skip excluded path './bootstrap/templates/kubernetes/apps/test1/app.yaml.j2'
Skip excluded path './bootstrap/templates/kubernetes/apps/test1/config'
Skip excluded path './bootstrap/templates/kubernetes/apps/test1/config/config.yaml.j2'
Create dir './bootstrap/templates/kubernetes/k0s' ->
'./kubernetes/k0s'
Create dir './bootstrap/templates/kubernetes/k0s/__pycache__' ->
'./kubernetes/k0s/__pycache__'
Copy file './bootstrap/templates/kubernetes/k0s/__pycache__/_mjfilter.cpython-311.pyc' ->
'./kubernetes/k0s/__pycache__/_mjfilter.cpython-311.pyc'
Skip excluded path './bootstrap/templates/kubernetes/k0s/_mjfilter.py'
Create dir './bootstrap/templates/kubernetes/k0s/files' ->
'./kubernetes/k0s/files'
Render file './bootstrap/templates/kubernetes/k0s/files/config.yaml.j2' ->
'./kubernetes/k0s/files/config.yaml'
Render file './bootstrap/templates/kubernetes/k0s/k0s.yaml.j2' ->
'./kubernetes/k0s/k0s.yaml'
Skip excluded path './bootstrap/templates/kubernetes/k3s'
Skip excluded path './bootstrap/templates/kubernetes/k3s/__pycache__'
Skip excluded path './bootstrap/templates/kubernetes/k3s/__pycache__/_mjfilter.cpython-311.pyc'
Skip excluded path './bootstrap/templates/kubernetes/k3s/_mjfilter.py'
Skip excluded path './bootstrap/templates/kubernetes/k3s/files'
Skip excluded path './bootstrap/templates/kubernetes/k3s/files/config.yaml.j2'
Skip excluded path './bootstrap/templates/kubernetes/k3s/k3s.yaml.j2'
Skip excluded path './bootstrap/templates/kubernetes/talos'
Skip excluded path './bootstrap/templates/kubernetes/talos/__pycache__'
Skip excluded path './bootstrap/templates/kubernetes/talos/__pycache__/_mjfilter.cpython-311.pyc'
Skip excluded path './bootstrap/templates/kubernetes/talos/_mjfilter.py'
Skip excluded path './bootstrap/templates/kubernetes/talos/files'
Skip excluded path './bootstrap/templates/kubernetes/talos/files/config.yaml.j2'
Skip excluded path './bootstrap/templates/kubernetes/talos/talos.yaml.j2'
Render file './bootstrap/templates/main.yaml.j2' ->
'./main.yaml'

Please let me know if this code works for you.

Thank you so much for taking the time to look into this, I tried it out in my test repo and it appears to be working 🎉  I'll be giving it a try in my main repo soon and hopefully all goes well and does not speed down the execution drastically. This helps immensely with conditional based generation of folders and files, you have no idea how much of an effect this means in my project.
One note is I had to do was set an env var of PYTHONDONTWRITEBYTECODE="1" to have those pesky _pycache folders not be generated.

I am going to go ahead and close this issue since it is all working now. I honestly can't tell any speed difference as it still seems fast.
It would still be nice for makejinja to support this special file out of the box but I will take this win and again, I have so much gratitude for this project and you helping me out here.

Thanks for the feedback, great that it works for you now! I agree that such a feature may be interesting for others in the future, but am a bit hesitant to integrate it into makejinja since there are a few potential parameters (e.g., the file name, the arguments passed to the functions). Another idea that I have would be to include the plugin code into makejinja so that it can be loaded by adding a module name like makejinja.plugins.mjfilter to the list of plugins in the config (the ability to offer such bundled plugins was the main reasons why plugin is a list and not a single entry). In that case however, no customization would be possible and anyone wanting to modify it would need to duplicate the code (which is not ideal IMHO). I will let you know once/if I made a decision on this matter.


Upgrade to Angular 18

Hi,
Would be great if this package could be updated to support Angular 18.
Thanks for all the efforts!

I have created this PR: https://github.com/danielmoncada/date-time-picker-moment-adapter/pull/20
Just need to merge it.

@GKersten the latest package on npm, v18 was updated about a month ago https://www.npmjs.com/package/@danielmoncada/angular-datetime-picker

published: https://www.npmjs.com/package/@danielmoncada/angular-datetime-picker-moment-adapter


SCB-315 Config Center module can't get config by DynamicPropertyFacto…

…ry.getInstance().getStringProperty, so has to use ConcurrentCompositeConfiguration
Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[x] Each commit in the pull request should have a meaningful subject line and body.
[x] Format the pull request title like [SCB-XXX] Fixes bug in ApproximateQuantiles, where you replace SCB-XXX with the appropriate JIRA issue.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Run mvn clean install to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[x] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Coverage decreased (-0.06%) to 87.254% when pulling 208490181e867297a480e679ccf23decb577014b on jeho0815:Branch_SCB-315 into 769dc9dd52f3c003f763ddaacd5d601a37ed273e on apache:master.


Enable SQLITE_ENABLE_UPDATE_DELETE_LIMIT

The SQLITE_ENABLE_UPDATE_DELETE_LIMIT enables ORDER BY + LIMIT clauses in the DELETE/UPDATE statements. The problem is that some SQLite options cannot be enabled with the default upstream amalgamation, since they modify SQLite parser generator output.
To enable this option I have created a small shell script (runs on linux only) that clones SQLite repository and generates a custom amalgamation tarball with the option enabled. This script only needs to be executed when updating SQLite to a new version (simply change the checkout version in the script).
Then I modified the build scripts to use the generated tarball, which must have been previously checked in, and included a locally generated tarball as the last commit.
If you want to generate the tarball yourself, feel free to remove the last commit and replace with your own.

I've expanded upon these ideas and added this compile-time option to version 5.0.0.


[HUDI-5822] Fix FileId not found exception when FileId is passed to HoodieMergeHa...

...ndle
Applying the fix from #5185 will fix write issues for MOR tables, but will cause write issues for COW tables.
More information on how to reproduce the COW error in this jira issue:
https://issues.apache.org/jira/browse/HUDI-5822
This addresses the issue raised here: https://github.com/apache/hudi/issues/5782
Change Logs

Complement the change of getLatestFileSlices -> getAllFileGroups in #5185

Impact

Allow jobs to recover properly
Ensure that the correct fileSlices are being read

Risk level (write none, low medium or high below)
None
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

[ ] Read through contributor's guide
[ ] Change Logs and Impact were stated clearly
[ ] Adequate tests were added if applicable
[ ] CI passed

Do not merge this in yet, my fix rollback the fix done here:
https://github.com/apache/hudi/pull/5185

@voonhous, could you please add the test cases for the changes?

@voonhous, could you please add the test cases for the changes?

Hmmm, not easy to write a test for this. This bug can only be triggered if JM's rollback is running behind TM.

@hbgstc123 @TengHuo for visibility

@SteNicholas @danny0405 Can you please help to review this commit? Thank you

@hudi-bot run azure

Close because it should be fixed in https://github.com/apache/hudi/pull/8077

@danny0405 This is not fixed... The rollback standardises the rollback logic between markers and listing for MOR tables.
The bug is still present in COW tables... TM might run ahead of of JM. If we continue to use getAllFileGroups, TM might still be able to fetch the fileGroup that is destined to be deleted, causing the fileId/Group not found in issue.

TM might run ahead of of JM

How could this happen? The write tasks on JM would wait for the JM coordinator to finish initialization then start to hande the writiing process.

JM
2023-02-20 11:36:34,826 INFO  org.apache.hudi.client.BaseHoodieWriteClient                 [] - Begin rollback of instant<PHONE_NUMBER>2929727
2023-02-20 11:36:34,833 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading HoodieTableMetaClient from hdfs://hudi_path
2023-02-20 11:36:34,947 INFO  org.apache.hudi.common.table.HoodieTableConfig               [] - Loading table properties from hdfs://hudi_path/.hoodie/hoodie.properties
2023-02-20 11:36:34,952 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from hdfs://hudi_path
2023-02-20 11:36:34,952 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading Active commit timeline for hdfs://hudi_path
2023-02-20 11:36:35,320 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[==>20230220112929727__commit__INFLIGHT]}
2023-02-20 11:36:35,321 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating View Manager with storage type :REMOTE_FIRST
2023-02-20 11:36:35,321 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating remote first table view
2023-02-20 11:36:35,323 INFO  org.apache.hudi.client.BaseHoodieWriteClient                 [] - Scheduling Rollback at instant time :<PHONE_NUMBER>3634829 (exists in active timeline: true), with rollback plan: false
2023-02-20 11:36:35,612 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[==>20230220113634829__rollback__REQUESTED]}
2023-02-20 11:36:35,612 INFO  org.apache.hudi.table.action.rollback.BaseRollbackPlanActionExecutor [] - Requesting Rollback with instant time [==>20230220113634829__rollback__REQUESTED]
2023-02-20 11:36:35,620 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[==>20230220113634829__rollback__REQUESTED]}
2023-02-20 11:36:35,694 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Checking for file exists ?hdfs://hudi_path/.hoodie/20230220113634829.rollback.requested
2023-02-20 11:36:35,706 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Create new file for toInstant ?hdfs://hudi_path/.hoodie/20230220113634829.rollback.inflight
2023-02-20 11:36:35,709 INFO  org.apache.hudi.table.action.rollback.CopyOnWriteRollbackActionExecutor [] - Clean out all base files generated for commit: [==>20230220112929727__commit__INFLIGHT]
2023-02-20 11:36:35,720 INFO  org.apache.hudi.table.action.rollback.CopyOnWriteRollbackActionExecutor [] - Time(in ms) taken to finish rollback 11
2023-02-20 11:36:35,720 INFO  org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor [] - Rolled back inflight instant<PHONE_NUMBER>2929727
2023-02-20 11:36:35,721 INFO  org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor [] - Index rolled back for commits [==>20230220112929727__commit__INFLIGHT]
2023-02-20 11:36:35,725 INFO  org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor [] - Deleting instant=[==>20230220112929727__commit__INFLIGHT]
2023-02-20 11:36:35,725 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Deleting instant [==>20230220112929727__commit__INFLIGHT]
2023-02-20 11:36:35,728 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Removed instant [==>20230220112929727__commit__INFLIGHT]
2023-02-20 11:36:35,728 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Deleting instant [==>20230220112929727__commit__REQUESTED]
2023-02-20 11:36:35,731 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Removed instant [==>20230220112929727__commit__REQUESTED]
2023-02-20 11:36:35,732 INFO  org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor [] - Deleted pending commit [==>20230220112929727__commit__REQUESTED]
2023-02-20 11:36:35,733 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Checking for file exists ?hdfs://hudi_path/.hoodie/20230220113634829.rollback.inflight
2023-02-20 11:36:35,786 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Create new file for toInstant ?hdfs://hudi_path/.hoodie/20230220113634829.rollback
2023-02-20 11:36:35,786 INFO  org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor [] - Rollback of Commits [20230220112929727] is complete
2023-02-20 11:36:35,805 INFO  org.apache.hudi.common.fs.FSUtils                            [] - Removed directory at hdfs://hudi_path/.hoodie/.temp/20230220112929727
2023-02-20 11:36:35,806 INFO  org.apache.hudi.metrics.HoodieMetrics                        [] - Sending rollback metrics (duration=973, numFilesDeleted=2)
2023-02-20 11:36:35,812 INFO  org.apache.hudi.client.BaseHoodieWriteClient                 [] - Generate a new instant time:<PHONE_NUMBER>3635812 action: commit
2023-02-20 11:36:35,815 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[20230220113634829__rollback__COMPLETED]}

TM
 11:36:33,837 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from hdfs://hdfs_path
2023-02-20 11:36:33,837 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading Active commit timeline for hdfs://hdfs_path
2023-02-20 11:36:33,840 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[==>20230220112929727__commit__INFLIGHT]}
2023-02-20 11:36:33,841 INFO  org.apache.hudi.client.BaseHoodieClient                      [] - Embedded Timeline Server is disabled. Not starting timeline service
2023-02-20 11:36:33,843 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating View Manager with storage type :REMOTE_FIRST
2023-02-20 11:36:33,843 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating remote first table view
2023-02-20 11:36:33,849 INFO  org.apache.hudi.common.table.view.AbstractTableFileSystemView [] - Took 2 ms to read  0 instants, 0 replaced file groups
2023-02-20 11:36:33,850 INFO  org.apache.hudi.sink.common.AbstractStreamWriteFunction      [] - Send bootstrap write metadata event to coordinator, task[0].
2023-02-20 11:36:33,850 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[==>20230220112929727__commit__INFLIGHT]}
2023-02-20 11:36:33,853 INFO  org.apache.flink.runtime.taskmanager.Task                    [] - bucket_write: dim_buyer_info_test (1/2)#0 (e2e7a38d69393b8d814ad48544573435_829468138015e9cb689e833f1800885e_0_0) switched from INITIALIZING to RUNNING.
2023-02-20 11:36:33,857 INFO  org.apache.hudi.sink.CleanFunction                           [] - Executor executes action [wait for cleaning finish] success!
2023-02-20 11:36:33,860 INFO  org.apache.hudi.sink.bucket.BucketStreamWriteFunction        [] - Loading Hoodie Table dim_buyer_info_test, with path hdfs://hdfs_path/age=0
2023-02-20 11:36:33,860 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading HoodieTableMetaClient from hdfs://hdfs_path
2023-02-20 11:36:33,867 INFO  org.apache.hudi.common.table.HoodieTableConfig               [] - Loading table properties from hdfs://hdfs_path/.hoodie/hoodie.properties
2023-02-20 11:36:33,868 INFO  org.apache.hudi.common.util.ClusteringUtils                  [] - Found 0 files in pending clustering operations
2023-02-20 11:36:33,868 INFO  org.apache.hudi.common.table.view.AbstractTableFileSystemView [] - Building file system view for partition (age=1)
2023-02-20 11:36:33,872 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from hdfs://hdfs_path
2023-02-20 11:36:33,872 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading Active commit timeline for hdfs://hdfs_path
2023-02-20 11:36:33,878 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[==>20230220112929727__commit__INFLIGHT]}
2023-02-20 11:36:33,879 INFO  org.apache.hudi.common.table.view.AbstractTableFileSystemView [] - addFilesToView: NumFiles=2, NumFileGroups=1, FileGroupsCreationTime=6, StoreTimeTaken=1
2023-02-20 11:36:33,880 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating View Manager with storage type :REMOTE_FIRST
2023-02-20 11:36:33,880 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating remote first table view
2023-02-20 11:36:33,880 INFO  org.apache.hudi.sink.bucket.BucketStreamWriteFunction        [] - bootstrapIndexIfNeed with timeline: [[==>20230220112929727__commit__INFLIGHT]]
2023-02-20 11:36:33,880 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading HoodieTableMetaClient from hdfs://hdfs_path
2023-02-20 11:36:33,880 INFO  org.apache.hudi.sink.bucket.BucketStreamWriteFunction        [] - Should load this partition bucket 0 with fileID 00000000-ee86-4b41-a704-9e075dd253d8
2023-02-20 11:36:33,880 INFO  org.apache.hudi.sink.bucket.BucketStreamWriteFunction        [] - Adding fileID 00000000-ee86-4b41-a704-9e075dd253d8 to the bucket 0 of partition age=1.

@danny0405 Running the tests included with this PR without the fix fails... i.e. The fix in this PR is still required.

@danny0405 Running the tests included with this PR without the fix fails... i.e. The fix in this PR is still required.

Got ya, can you rebase with the latest master and force-push again

@danny0405 Running the tests included with this PR without the fix fails... i.e. The fix in this PR is still required.


Identical but renamed courses have different coverage of years

EL ENG 126: https://berkeleytime.com/grades/0-2418-all-all
EECS 126: https://berkeleytime.com/grades/0-14673-all-all

Might be worth giving users more control over the all semesters option or listing the semester's we're averaging for better transparency.


Humbug and Laravel Valet

I am trying Humbug out for the first time and I am having a small problem with Humbug + Laravel Valet.
Humbug is trying to load the Valet helpers file and the Illuminate/Foundation/helpers.php file at the same time and they both have a function named info(). So I am getting a 'cannot redeclare info() error'. This error isn't present when I run PHPUnit on the same project.
How do I tell Humbug not to load the global Valet files (located in ~/.composer/vendor/laravel/valet/cli/includes/helpers.php)?
I can fix this by wrapping the info function in helpers.php in an "if(!function_exists('info'){})" statement, but that doesn't sound like a long term solution.

If I remove Laravel Valet, then Humbug works fine.
I just re-installed Valet and I am using 1.1.19. Humbug is 1.0-dev.
I had originally installed humbug globally via composer. I just uninstalled it globally and re-installed it locally in the project and everything works now. I can have Valet and Humbug installed at the same time.

@greenbicycle any application/tool executing your code may have dependencies that can conflict with yours. If you want to avoid that, installing it as a dependency of your project solve that problems.

Thanks. Problem solved.

Probably worth noting somewhere prominently in the docs.


Warning : mysqli_fetch_assoc() expects parameter 1 to be mysqli_result, null given in C:\xampp\htdocs\Advance_Shopping_cart-master\index.php on line 71

how can i solve this problem please.?
Warning : mysqli_fetch_assoc() expects parameter 1 to be mysqli_result, null given in C:\xampp\htdocs\Advance_Shopping_cart-master\index.php on line 71

same problem

Error is occuring
Warning : mysqli_fetch_assoc() expects parameter 1 to be mysqli_result, null given in C:\xampp\htdocs\Advance_Shopping_cart-master\index.php on line 71

how can i solve this problem please.?
Warning : mysqli_fetch_assoc() expects parameter 1 to be mysqli_result, null given in C:\xampp\htdocs\Advance_Shopping_cart-master\index.php on line 71

Same Here..

create database and insert the given records properly

after creating database then it also not working why I don't know pls help

after creating database then it also not working why I don't know pls help

Fill db with dummy data


Cant work in my car :(

I try fix now

Pls, I would like to know if this issue will be fixed

Oh, for the love of pancakes. Some random person on the other side of the planet has probably gotten DOOM running on your model of car. All you need is that no-code spirit!!

Send the debug log

```  Here is the log

Had same issue, disabling break lights fixed it.


RemoteEvents don't function correctly when using rbxnet

Unsure whether it's my issue or not, but any request gets rejected when using Iris to send it.
Button code:
if Iris.Button({ "Button" }).clicked() then
    local remote = remotes.Client:Get("PrintMessage")
    remote:SendToServer("This should be printed")
end

Server-side code:
remotes.Server:Get("PrintMessage"):Connect(function(plr, message)
	print(message)
end)

Error itself:
net.out.definitions.ClientDefinitionBuilder:54: Expected Promise rejected with no value.
Everything works perfectly fine when using a vanilla Roblox button and the message gets printed.
Rbxnet: https://rbxnet.australis.dev/

Seems more like an rbxnet issue

I tried having a look at the documentation for RBXNet but I wasn't quite sure how it works. Within the Iris loop, there shouldn't be any asynchronous code which would prevent the entire code from finishing. I presume that would be the issue, but I'm not quite sure. I'll see if I can do some testing with this.

This appears to be intended behavior. In an Iris execution cycle, a coroutine is spawned and if its lifecycle is not concluded when the next Iris cycle is initialized, then the coroutine is ended. In most scenarios involving asynchronous code inside of an Iris cycle it is best practice to spawn a thread and keep all GUI code synchronous.


[$1000] Two line markers appear for a second

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Create a task
Send two messages
Mark as Unread the second message
delete the first message

Expected Result:
The new line marker and the message separating the line align immediately
Actual Result:
Two line markers appear for a second
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android / native
[ ] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[x] MacOS / Chrome / Safari
[ ] MacOS / Desktop

Version Number: 1.3.54-11
Reproducible in staging?: Yes
Reproducible in production?: Yes
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
https://github.com/Expensify/App/assets/38435837/f7193735-c044-4a12-b754-0ad7677650ba
https://github.com/Expensify/App/assets/38435837/e1384077-09e6-456c-97c6-f5ee1f720780
Expensify/Expensify Issue URL:
Issue reported by: @daveSeife
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1691459749715329
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~0110cb0187ff6d5cbd
Upwork Job ID:<PHONE_NUMBER>512989696
Last Price Increase: 2023-08-17

Proposal
Please re-state the problem that we are trying to solve in this issue.
Two line markers appear for a second
What is the root cause of that problem?
When removing the message, this message isn't remove from sortedReportActions until the BE return new data. It caused that 2 divider lines are displayed at the same time.
But we can't see this issue on thread chat because in thread chat the new marker and divider line have the same position, the new marker line will overwrite the divider line
in the task report and money report view, we apply margin vertical 8px to divider line
https://github.com/Expensify/App/blob/0bdf18378356d6f4d5dd0aada5182b3c1648fa4e/src/styles/styles.js#L3537
It make the position of the new marker line and the divider line different.
What changes do you think we should make in order to solve the problem?
In the task report and money report view, I suggest that  we should remove margin (as we did in thread chat) vertical or only use margin top in the divider line
This solution also make report action not flicker when mark as unread
Result
https://github.com/Expensify/App/assets/141406735/51d7e899-569a-4880-808c-43c305d2df76

Reproduced on staging and prod.

Proposal
Please re-state the problem that we are trying to solve in this issue.
Two line markers appear for a second
What is the root cause of that problem?
after deleting the first comment  here its changed to sortedReportActions[sortedReportActions.length - 2].pendingAction = 'delete' until delete api finishes at the time reportActionID === newMarkerReportActionID also doesn't match that why showing the line  at the time shouldDisplayNewMarker also true
https://github.com/Expensify/App/blob/e043331def1babf5c70021a22043faed858f38d0/src/pages/home/report/ReportActionsList.js#L128

What changes do you think we should make in order to solve the problem?
                                 const sortedReportActionsFilter = sortedReportActions.filter(item => props.network.isOffline || item.pendingAction != CONST.RED_BRICK_ROAD_PENDING_ACTION.DELETE);
            const sortedReportActionsFirst = sortedReportActionsFilter.length > 1 && sortedReportActionsFilter[sortedReportActionsFilter.length - 2];
            const shouldHideThreadDividerLine = sortedReportActionsFirst && sortedReportActionsFirst.reportActionID === newMarkerReportActionID

@parasharrajat, let me what is the mistake I am done.

@parasharrajat Sorry, I missed that before the help+wanted  label posted the proposals. so I missed that.

@dukenv0307 do you need anything else from us to get the PR up?

@dukenv0307 I just tested this and it's performing a bit differently, with the line below the first message. Mind giving this one another look?

Summarizing payments for this issue:

Urgency bonus: No
Reveiwer: @parasharrajat $1000
Contributor: @dukenv0307 $1000 paid via Upwork
Reporter: @daveSeife $250 paid via Upwork

Upwork job: https://www.upwork.com/jobs/~0110cb0187ff6d5cbd

@parasharrajat please submit payment once you've completed the checklist

$1,000 payment approved for @parasharrajat based on summary above.


port to new gongc

[x] bare application
[x] add gongfly specific gongfly-dashbord
[x] add sim stack (like adding in gonggantt)
[x] display engine control in gongfly-dashbord
[x] add leaflet stack
[x] display leaflet stack in gongfly-dashbord

compilation issue:
vendor\github.com\fullstack-lang\gongleaflet\ng\projects\gongleafletspecific\src\lib\mapoptions\mapoptions.component.ts
import * as L from 'leaflet'

Cannot find module 'leaflet' or its corresponding type declarations.

This is a regression compared to the working version in main. Is it:

module import ?

the Leaflet module is imported in the  gongflyspecific angular lib but to no avail. The leaflet module is also imported in the the app module.

path ?

      "@types/leaflet": [
        "./node_modules/@types/leaflet"
      ],
      "@asymmetrik/ngx-leaflet": [
        "./node_modules/@asymmetrik/ngx-leaflet"
      ],
      "leaflet": [
        "./node_modules/leaflet"
      ],

in tsconfig seems to be correct

leaving the subject to solve issue with gongsim.
Error: ../vendor/github.com/fullstack-lang/gongsim/ng/projects/gongsimcontrol/src/lib/engine-control/engine-control.component.ts:60:24 - error TS2554: Expected 1 arguments, but got 0.

60     this.engineService.getEngines().subscribe(

problem, objects do not appear on the map.
	log.Printf("Unknown layer %s ", layerGroupName)


Change URL for sqlfluff rules documantation

The current rules documentation URL appears to have changed and has been corrected.

Before

https://docs.sqlfluff.com/en/stable/rules.html#sqlfluff.rules.Rule_AL01

After (Looks like sphinx has been addedsphinx)

https://docs.sqlfluff.com/en/stable/rules.html#sqlfluff.rules.sphinx.Rule_AL01

Thanks for the PR!


Example doesn't work anymore

Using the following dependencies:
 "dependencies": {
    "@hono-storage/core": "^0.0.9",
    "@hono-storage/node-disk": "^0.0.10",
    "@hono/node-server": "^1.11.1",
    "hono": "^4.3.2"
  },
  "devDependencies": {
    "@types/node": "^20.11.17",
    "tsx": "^4.7.1"
  }

index.ts
import { serve } from "@hono/node-server";
import { HonoDiskStorage } from "@hono-storage/node-disk";
import { Hono } from "hono";

const app = new Hono();
const storage = new HonoDiskStorage({
  dest: "./uploads",
  filename: (_, file) => `${file.originalname}-${Date.now()}.${file.extension}`,
});

app.post("/", storage.single("file"), (c) => c.text("OK"));

serve(app);

Error message over storage.single()
No overload matches this call.
  The last overload gave the following error.
    Argument of type 'MiddlewareHandler<{ Variables: { files: { file?: FieldValue | undefined; }; }; }>' is not assignable to parameter of type 'H<Env, "/", BlankInput, HandlerResponse<any>>'.
      Type 'MiddlewareHandler<{ Variables: { files: { file?: FieldValue | undefined; }; }; }>' is not assignable to type 'MiddlewareHandler<Env, "/", BlankInput>'.
        Types of parameters 'c' and 'c' are incompatible.
          Type 'Context<Env, "/", BlankInput>' is not assignable to type 'Context<{ Variables: { files: { file?: FieldValue | undefined; }; }; }, string, {}>'.
            Property '#private' in type 'Context' refers to a different member that cannot be accessed from within type 'Context'.ts(2769)
types.d.ts(154, 5): The last overload is declared here.

You got an idea, what's wrong?

It seems that there is a type discrepancy when using Hono 4 series Hono because they have not updated the Hono version.
I’ll patch it the next version, Thank you.


Unable to filter by lead status in Student list

Because the lead status is a computed property, we cannot filter from CRUD.

Hello! Can I work on this issue?

Hello! You are most welcome to work on this issue, and you will be credited as a contributer when your PR is merged. Please feel free to reach out if you need any help or have any questions!

I have pulled a request!

Thanks for the PR. Unfortunately, the getLeadStatusAttribute() method is not simply a getter for the lead_status_type_id property. It returns a computed value that makes it impossible to build the filter based on the database colum. We'd have to run the query first and retrieve all the values, and then filter. Which isn't really compatible with Backpack's filter system...
I guess there are different ways to address this issue, one of them could be to ensure that lead_status_type_id is systematically updated whenever the student is enrolled / removed from a course.
If you have other suggestions, please feel free to share them :)

What's about listening to Enrollments events and dynamically update lead_status_type_id  ? This way we could implement filter based on database column?

Yes, I think this would work :)

Hi! I've just pulled a request!


PHOENIX-4763: Changing a base table property value should be reflected in child views (if the property wasn't changed)

[WIP]

When a view alters a property, add a tag for the corresponding cell in SYSCAT
Introduce a bitset to keep track of these properties in PTable
Add tests
TODO:

Handle the case where we "CREATE VIEW <>" with properties specified
Add unit tests for tagging changes

@twdsilva Please review. I will rebase changes once the approach looks good.

Added cell tagging change in 'CREATE VIEW' with properties path
Added tests
Added refactoring as per PHOENIX-4951
@twdsilva please review. Thanks!

Thanks for the review @twdsilva


Linter-Flake8 keeps on crashing

When ever I try to use Atom, I get this error message:
linter-flake8:: Flake8 threw an error related to:
flake8.exceptions.PluginRequestedUnknownParameters: "pycodestyle" requested unknown parameters causing 'FileProcessor' object has no attribute 'indent_size'
Please check Atom's Console for more detail

I use pycodestyle 2..7.0

I had the same problem, I installed Flake8 using apt, which installs an old version of Flake8 that conflicts with pycodestyle.
Related issue: https://github.com/PyCQA/pycodestyle/issues/983
To solve this, I removed the package and then installed it with pip, and made an alias to flake8 in bashrc
sudo apt-get remove flake8
python3 -m pip install flake8
// Alias in bashrc
alias flake8="python3 -m flake8"


how to choose the noise parameters in sampling?

Hi Campbell, I saw in the notebook demo and the sampling code, there is a hyperparameter noise. In sampling.py it is set to 0.0, and it's 1.0 in the uniform demo and 10.0 in the mask demo. Is there any principle to choose this parameter? thanks

So the noise parameter will control how much 'mixing' happens during sampling e.g. with the masking process how much it flips back and forth between mask and unmask. In the masking case, if we integrate with time step dt and have D dimensions, we have that  noise * dt * D  is the average number of dimensions that get set back to mask in each integration step. We don't want this to be too big a proporition of all dimensions otherwise the process could become degenerate with everything getting set back to mask all the time. So maybe as a rule of thumb we wouldn't want more than say 10% of the dimensions to get switched back each integration step noise * dt * D < 0.1 D  => noise < 0.1/dt so for dt = 0.001 we would have noise < 100. From this we also see that the higher you make dt, the smaller you will want to set noise (noise = 0 is likely the easiest to simulate with highest dt).
Other than this very rough upper bound, at the end of the day it should be set so that you observe best sampling performance on the task you are interested in. In theory, any value of noise will achieve the desired marginals and result in a sample from the data distribution. In practice however, our denoising distribution is only approximate and we introduce discretization error during simulation as well. In this case, we are forced to empirically choose the noise that works best in our real world approximate setting.
One final thing to note is that it is also possible to have the noise hyperparameter depend on the time variable. E.g. you can have more noise during the final parts of simulation and less noise nearer the beginning. In my previous work https://arxiv.org/pdf/2205.14987 top of page 39, we found it is most beneficial to introduce noise near the end of sampling. Though I haven't systematically investigated this for the discrete flow models case.

Thank you for the explanation. Is it okay to set noise=0 for uniform diffusion? Also, do you mind sharing the unigram_stats.pkl file?


refactor: add a utility type for validation functions

Codecov Report

Merging #8 (5da81df) into main (db48c2b) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##              main        #8   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            2         2
  Lines          356       359    +3
  Branches        63        63
=========================================
+ Hits           356       359    +3

Impacted Files
Coverage Δ

src/index.tsx
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update db48c2b...5da81df. Read the comment docs.


AudioStreamInteractive transition can play clips with wrong volumes (very silent)

Tested versions

Reproducible in v4.3.stable.mono.official [77dcf97d8]

System information
Windows 10 - Vulkan (forward+) - dedicated
Issue description
Using a transition to Same or previous position that has fade, makes the audio clip play incorrectly (very low volume)
This doesn't happen in all cases, there is something strange going on.
I found two instances where this happen that I could replicate inside the MRP.
Video of the problem (needs audio)
Using the transition from Tower Low to Battle intro (to current position, with fade 1) -> Low audio
Using the transition from Battle to Battle intro (no transition set, so instant) -> Normal audio
https://github.com/user-attachments/assets/e15f774c-0bbc-4df0-b544-e36dee63808c
Steps to reproduce
The other example I found, set the start transition it itself with fade 1 and transition to same position.

Open "audio_volume_transition.tscn"
Then set parameter Switch to Clip to the starting clip(Tower Intro) and start start the playback.

Result: Will trigger the transition when playback starts and start the clip with low audio.
Example 2

Open "duplicated audio.tscn"
When "Loop" is playing, set Switch to Clip to "Battle Intro"

Result: Battle Intro is played with low volume
I couldn't replicate this with a minimum example, putting transition of Loop->BattleIntro with fade and same position in a new InteractiveAudio without any other transitions worked as normal. So something strange is going on here.
Thinking about it now, maybe it has something to do with the automatic fade mode option.
My guess is that this only happens in one type of fade option, and because I used automatic it didn't happen when I tried to reproduce it in a clean InteractiveStream because the automatic fade might have chose a different type.
Minimal reproduction project (MRP)
fade_low_volume_interactive_music.zip

I've tested a bit more, and I think I figure out more about the problem.
This problem is present when using a fade in or cross fade transition. Even if transition mode is set to clip_start.
So I've thought the problem had something to do with transitions from a low to a high BPM song and somehow calculating the fade values wrong.
But it turns out the problem is with the Auto Advance feature.
This bug happens when the transition destination audio stream has Auto Advance enabled, and uses a fade in or cross fade.
Not sure why this would happen, but from more testing I feel pretty sure that is the problem, and I can now perfectly replicate this bug.
Steps:

Add a transition with (cross-fade, fade beats=1, to a clip that has Auto Advanced enabled)

Result: clip is played with low volume


Love this plugin, just need some additional features for it to be usable

First, would love to have basic data validation. Since it only supports CSV, we should be able to do this:
<input type="file" accept=".csv" />

So that at the HTML level there is only CSV types allowed. Right now the app just crashes if you try to upload, say, a .Numbers file.
Second, and more importantly, we really really need a callback function somehow that gets fired when the user selects the file. Otherwise we don't really have a way to handle file saving once it's been uploaded. I feel like this is so basic that I must be missing something here??

Hey Martyn these are valuable and relevant features, I can have a look at it, meanwhile feel free to send a pull request :)


Update language for v5.0.0: it

include new/changed strings for the new features

@elax46 Thank you so much!


Jerryliu sc 44386 migrate web3 related magic ext repos to magic (PART 1)

📦 Pull Request

Local tested with RN env
All extension packages will have major version bumps.

✅ Fixed Issues
https://app.shortcut.com/magic-labs/story/44386/migrate-web3-related-magic-ext-repos-to-magic-js-sdk-monorepo
🚨 Test instructions
[Describe any additional context required to test the PR/feature/bug fix.]
⚠ Don't forget to add a semver label!

patch: Bug Fix?
minor: New Feature?
major: Breaking Change?
skip-release: It's unnecessary to publish this change.

Please add a major semver label, when you get a chance!

:rocket: PR was released in<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>:rocket:


Bugfix: ArchiveBox ignoring config values via environment variable and ArchiveBox.conf

Downloading sites without enforcing SSL verification doesen't function

CHECK_SSL_VALIDITY=false

does not permit user to download from site with outdated or incorrectly setup certificates.

[!] Failed to download https://site-with-ssl-issue.com
<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:841)>

ArchiveBox.conf.default.txt
Aliasing in ~/.bashrc did not resolve the issue with ArchiveBox SSL verification.

alias wget='wget --no-check-certificate'
alias curl='curl -k'

Issue resolved, needs to be in proper directory.

CHECK_SSL_VALIDITY malfunctions regardless of file location.
It only downloads sites with SSL issues when line in archivebox/config.py

CHECK_SSL_VALIDITY =     os.getenv('CHECK_SSL_VALIDITY',     'False'             ).lower() == 'False'

is modified

Can you verify that the environment variable is actually making it to archivebox by adding a line that line to print:
print('CHECK_SSL_VALIDITY=', os.getenv('CHECK_SSL_VALIDITY', 'UNSET'))

Or by running archivebox config --get=CHECK_SSL_VALIDITY.

Outputs CHECK_SSL_VALIDITY= UNSET
This is likely identical culprit to bug https://github.com/pirate/ArchiveBox/issues/268

Going to close the other issue and rename this one, since it seems to be a generic issue with config handling.

Do you mind posting your archivebox version, and the exact command you're using to run archivebox.  Also the output of archivebox --version.

$ ./archive version
ArchiveBox version b7ea44a1e
Config files are identical
~/.ArchiveBox.conf
ArchiveBox/etc/ArchiveBox.conf.default

From git
git clone https://github.com/pirate/ArchiveBox
$ ./archive version
ArchiveBox version b7ea44a1e

Config files are identical
~/.ArchiveBox.conf
ArchiveBox/etc/ArchiveBox.conf.default

To archive
./archive http://www.site-to-archive.com

Ah I thought you were running the v0.4 branch.  The current version on master actually doesn't support config files at all, it's all done via environment variables, so you'd have to source your config file before running the command manually.
source ~/.ArchiveBox.conf
echo $CHECK_SSL_VALIDITY
./archive http://www.site-to-archive.com

Can you confirm that that outputs false before archivebox starts.

It used to archive sites with ssl issues fine if CHECK_SSL_VALIDITY was manually exported but the media option doesn't work regardless if its both exported manually and in both config files.
$ source ~/.ArchiveBox.conf
$ echo $CHECK_SSL_VALIDITY
False

There is no issue any longer, setting enviorment variables works really well including for disabling media downloads.