Questions about child processes and manually reloading

Good day!
I use Bottle + WSGIRefServer and have some questions.

In which point of code is child process created?
Are there fixed number of child processes or "one process per request"?
How do I limit number of child processes? I want to have <= 1 child process at every moment.
How do I manually reload server? So I want to reload server by a signal...

Thanks very much!

P.S. Maybe it's WSGIRef questions not Bottle. Sorry.

WSGIRef is a single-threaded development server not suitable for production. Even for development it has its quirks, for example that it can only handle a single request at a time. If you need more, I strongly recommend using a different server backend instead of fixing or monkey-patching WSGIRef.

Never. If you are using bottle auto-reload, then there is a fork just before the server is started, but this child process is only there to allow reloading, not to load-balance requests. There is always only a single process handling requests.
One process per everything. Some server backends maintain a thread-pool to handle requests in parallel. Multi-Process backends are rare, since you usually don't need it.
Use a different server backend that supports threads, and configure the thread pool according to your requirements.
If you want to manually restart the server, you don't need the bottle reload feature anymore. Just kill the server with SIGINT (the same singal sent by Ctrl+c) and have a process supervisor (supervisord, systemd, a bash script with a loop) restart it.

Thanks very much!


Adding documentation to clarify the result of DispatchTime(uptimeNanoseconds: 0)

The documentation does not make it clear that DispatchTime(uptimeNanoseconds: 0) produces a value that represents the current nanoseconds since boot, not the time of boot. The fix adds a documentation comment to that effect.
(Radar 28814085)
Resolves SR-2807.

@moiseev @mwwa @das please review

@swift-ci please smoke test

@swift-ci please smoke test


Pass passphrase to gpg using --passphrase-fd 0

Hi,
Thanks for the guide. It's very informative.
I would like to suggest a change to pass the passphrase to gpg from stdin via the --passphrase-fd 0 option instead of via the --passphrase option. The latter exposes the passphrase to observation by other processes on the system and the gpg man page includes a disclaimer for it: "Don't use this option if you can avoid it". Although I acknowledge that the README recommends a single user, ephemeral Debian Live environment, thus reducing the risk of exposing the passphrase as an argument, users may choose to ignore that recommendation so it seems best to protect them from themselves where possible. It's potentially also arguable that some users may naively take the current command and use it in other, more risky scenarios, not realizing the full context of the intended use.

This is an excellent suggestion.
Can you please confirm this GPG feature exists on the latest Debian Live image and also that the built-in version of echo is being used for effect?
Are the Transfer Subkeys operations in need of this update as well?

@forbytten thank you for the exceptional analysis. Your current change looks good.
I welcome and look forward to additional contributions and suggestions for improvements from you.


Add Appointment converter

Description

The converter shall be able to convert:

AppointmentModel to Appointment

Appointment to AppointmentModel

Appointment  to UpdateCompanion

The converter shall extends EntityModelConverter class with its generics: <Appointment, AppointmentModel>

Depends on

[x] #13
[x] #26

Part of

#25

closed by #80


Recursively skip verbatim elements

This handles cases like

        https://example.org

where a link was embedded in a verbatim block, but embedded in a non-verbatim element ().
This was correctly handled in html5ever, but not html5gum (which is our default HTML parser in lychee)
Fixes https://github.com/lycheeverse/lychee/issues/259#issuecomment-1328292006.
@untitaker, is there a better way to handle nested tags in html5gum or would you have resolved the issue in a similar way?

Oh I see. Would you have the time to create a pull request to change it to the new structure? As the author of the library it's probably way easier to do for you than for me. 😅 Maybe you want to ignore what I did in this PR and base it off of the last version of master just before this commit?

Probably not the cleanest implementation, but yeah. Thanks for the feedback. 👍


pip config: Support multiple resolvers

Problem:
The packages I want to install via pip are hostet in different artifactory repos of the same artifactory instance (generally the PYPI mirror plus some repo for internal packages) but it appears as if the JF CLI does not support more than one  resolver per project. Switching the config per installation will not help if for example an internal package has a dependency to another package that is hosted on pypi. Also the pip meachanisms like --extra-index-url have no effect for the CLI.
Proposal
Allow list or dict entries in the pip.yaml config file to store more than one resolver. This could either be implemented analogously to pip (index-url, extra-index-url) or just be a list of resolver that are iterated through.
Exemplary pip.yaml:

version: 1
type: pip
resolvers:
- {repo: my-repo-one, serverId: my-server}
- {repo: my-repo-two, serverId: my-server}

Alternative approaches:

Allow specification of a resolver as option for the "jf.exe pip install" command. (e.g. jf.exe pip install my-package --resolver=my-repo). Important: The specified resolver should not overwrite the configured base resolver but rather be available additionally. This way at least 2 resolvers/indices would be available for each installation call.
Parse the given pip configuration to use and append index-url and extra-index-url with credentials.

Kind regards
Enrico

Hi @enrlc0 -
Actually, resolving dependencies from multiple repositories is already supported, and is a fundamental feature of JFrog Artifactory.
All you need to do, is to create a virtual Pypi repository in Artifactory, and have it include all the repositories which include your dependencies. You can then have this virtual repository configured in JFrog CLI. This allows for great flexibility, because you can add or remove repositories to/from the virtual repository, without having your build scripts modified.

Hi @eyalbe4
thanks for the reply. Unfortunately I do not have permissios to create an repositories in Artifactory but I'll reach out to our admins.
For a large company with complicated permission management, I still think the ability to add more than one resolver would be helpful. In our case there would probably require more than a handful virtual repositories and double maintenance if any change of permissions was to be conducted. User targeted changes would also not be possible without creating yet another repo.
For now I think the administrators might just create a virtual repo as I'm one of a few people who actually usees the jfrog-cli but I hope you understand why I believe the feature might be helpful.
Kind regards
Enrico


Documentation for Static

Default static will need documentation and new screenshots. I'll be happy to add this - perhaps a writeup of a CTF problem like the old ezhp writeup will be nice.

+1
Also, I renamed the default on issue to make static nonblocking


fix undefined symbols problem

pm is used to show plus-minus, is yet not implemented into sympy. #5305
Example latex: \frac{-b\pm\sqrt{b^2-4ac}}{2a}
when converted into sympy using latex2sympy, I get
-b*pm*sqrt(-4*a*c + b**2)/(2*a)
So, the undefined pm simply gets multiplied. Hence getting wrong expression.
It should have probably shown an error.
In fact, any undefined thing simply gets multiplied
Ex: -b*pm*blabla*(-4*a*c + b**2)/(2*a)

Maybe \pm should be blacklisted as an invalid symbol because it produces such an unexpected result.
In regards to \blabla or something like that, I don't think it's a good idea to have a whitelist of valid symbols because it would be hard to account for the variety of symbols that people would want to use.


Laravel does not drop system versioned tables when running artisan migrate:fresh command.

Laravel Version
10.37.3
PHP Version
8.2.12
Database Driver & Version
MariaDB 10.7
Description
MariaDB supports System Versioned Tables, allowing operations on temporal data.
If a migration-managed table is system versioned, the artisan command migrate:fresh command does not drop the table, and subsequently bringing the migrations up fails, since the migration tries to create a table that already exists.
The issue occurs because a system versioned table type is SYSTEM VERSIONED, and not BASE TABLE. When collecting the tables to be dropped, the system versioned table is not selected. Selecting SYSTEM VERSIONED tables fixes the issue.
I am willing to provide a PR fixing the problem, but it seems like it might be a risky change, I don't know. Please advise.
Steps To Reproduce

Create base laravel project: composer create-project laravel/laravel example-app
Setup MariaDB 10.7+(ish?) and configure .env to connect to the database.
Run the default migrations with php artisan migrate
Create a new migration php artisan make:migration add_users_table_versioning
Execute the add versioning statement in the migration file:

public function up(): void
    {
        \Illuminate\Support\Facades\DB::statement("ALTER TABLE users ADD SYSTEM VERSIONING;");
    }

Run the migration. It works.
Run php artisan migrate:fresh
The command will fail with the message:

...
SQLSTATE[42S01]: Base table or view already exists: 1050 Table 'users' already exists
...

Navigate to MysqlGrammar:86.
Edit the query to

"from information_schema.tables where table_schema = %s and table_type = 'BASE TABLE' or table_type = 'SYSTEM VERSIONED' "

Execute php artisan migrate:fresh

The command runs successfully.

@fnzr @crynobone ,
Hey,
Please let me know. If I can assist you.


$this->db->update_batch() Returns False on Execution

I wrote the following code:
if($this->db->update_batch('table_name',$array, 'where_column'))
{
//run another query
}
However, the code inside the if statement was not executed. Which means upon execution, the function $this->db->update returned a false (0). I strongly feel it should return a true (1)

Thanks for the reply narfbg, however, check this out > http://codeigniter.com/user_guide/database/active_record.html#update
You'll find out that the syntax I used is correct:
$unsubscribes_array = array();
$to_unsubscribe = $this->db->get('subscribers');
foreach($to_unsubscribe->result() as $remove)
{
					$unsubscribe_array = array (
					    'email' => $remove->email,
						'subscription_status' => 0
					);

					$unsubscribes_array[] = $unsubscribe_array;
 }

if(!$this->db->update_batch('subscribers',$unsubscribes_array,'email'))
{
//EXECUTE SOME CODE
}

Thanks for the reply narfbg, however, check this out > http://codeigniter.com/user_guide/database/active_record.html#update
You'll find out that the syntax I used is correct:
$unsubscribes_array = array();
$to_unsubscribe = $this->db->get('subscribers');
foreach($to_unsubscribe->result() as $remove)
{
                $unsubscribe_array = array (
                    'email' => $remove->email,
                    'subscription_status' => 0
                );

                $unsubscribes_array[] = $unsubscribe_array;

}
if(!$this->db->update_batch('subscribers',$unsubscribes_array,'email'))
{
//EXECUTE SOME CODE
}

OK, that's ... unexpected at the least, but it indeed means that your example is correct. What driver are you using then?

I don't know exactly what you mean by driver. But I use MySQL DB: Server version: 5.5.8-log.
You know, the function $this->db->update in the code above actually works, and the update in the database is actually done. It's just that the return value is FALSE. That's why I had to use the ! operator.

I meant the value of $db['default']['dbdriver'] in your application/config/database.php file, but since you're using MySQL - that isn't the case.
There was indeed a bug in CodeIgniter 2.1, where update_batch() returned FALSE on failure and was otherwise void. So, if you put it in an if(), not that it would not catch it succeeding because it returned FALSE, but rather because it doesn't return anything. This is fixed in the current develop branch.

Got you nafbg. Thanks! Will be waiting for version 3.0 then. :)


Bug Report

Environment
Windows build number: 10.0.18362.175

Steps to reproduce

Even installing Ubuntu and Ubuntu 16LTS, the New Windows terminal does not recognize, does not appear the options to open the them in the tools menu, only appears PowerShell and the old Windows CMD.
Expected behavior
It is expected that the options of Ubuntu or any other system appear when choosing the terminal.

Actual behavior

The terminals of the systems simply do not appear in the new terminal.

Duplicate of #1449
please follow the solution provided. thank you


Lack of scrolling cuts off longer text.

It looks like the

 is set to overflow: hidden; which cuts off longer text closer to the right edge. See screenshot.

It would be nice if there was ability to scroll vertically and horizontally.
Great eye-candy project tho!

Oh snap. I hoped nobody would notice it ;)

Yes I do !!    :-D

I did worked on this and its fixed now.


Seg fault in call to InitAPI

As soon as my code calls InitAPI, it hits a seg fault. This is my code, which just tries to call InitAPI and ShutdownAPI with default options:
    std::shared_ptr<LoginExecutive>
    LoginExecutive::create(
        const LoginExecutiveContext & context
        )
    {
        Aws::SDKOptions options;
        Aws::InitAPI(options);
        Aws::ShutdownAPI(options);
        return std::make_shared<LoginExecutiveImpl>(context);
    }

This is the stack trace of the crash:
(lldb) bt
* thread #1: tid = 378, 0xf76c0d33 libc.so`pthread_mutex_lock + 35, name = 'doclily.doclily', stop reason = signal SIGSEGV: invalid address (fault address: 0x4)
    frame #0: 0xf76c0d33 libc.so`pthread_mutex_lock + 35
    frame #1: 0xe1a5cbb1 libc++_shared.so`std::__ndk1::mutex::lock() + 33
    frame #2: 0xe1624af8 libaws-cpp-sdk-core.so`Aws::Utils::Crypto::OpenSSL::locking_fn(int, int, char const*, int) + 88
    frame #3: 0xe17af9ba libaws-cpp-sdk-core.so`CRYPTO_lock + 266
    frame #4: 0xe1805735 libaws-cpp-sdk-core.so`___lldb_unnamed_symbol1057$$libaws-cpp-sdk-core.so + 101
    frame #5: 0xe1805944 libaws-cpp-sdk-core.so`___lldb_unnamed_symbol1059$$libaws-cpp-sdk-core.so + 68
    frame #6: 0xe1804db8 libaws-cpp-sdk-core.so`ERR_func_error_string + 120
    frame #7: 0xe17b4bd6 libaws-cpp-sdk-core.so`ERR_load_CRYPTO_strings + 38
    frame #8: 0xe16248fd libaws-cpp-sdk-core.so`Aws::Utils::Crypto::OpenSSL::init_static_state() + 29
    frame #9: 0xe16747b2 libaws-cpp-sdk-core.so`void std::__ndk1::__invoke_void_return_wrapper<void>::__call<void (*&)()>(void (*&&&)()) + 50
    frame #10: 0xe1674695 libaws-cpp-sdk-core.so`std::__ndk1::__function::__func<void (*)(), std::__ndk1::allocator<void (*)()>, void ()>::operator()() + 53
    frame #11: 0xe1611b19 libaws-cpp-sdk-core.so`std::__ndk1::function<void ()>::operator()() const + 41
    frame #12: 0xe1689876 libaws-cpp-sdk-core.so`Aws::Utils::GetTheLights::EnterRoom(std::__ndk1::function<void ()>&&) + 294
    frame #13: 0xe1672e2b libaws-cpp-sdk-core.so`DefaultMD5Factory::InitStaticState() + 107
    frame #14: 0xe166ecd1 libaws-cpp-sdk-core.so`Aws::Utils::Crypto::InitCrypto() + 705
    frame #15: 0xe15a765f libaws-cpp-sdk-core.so`Aws::InitAPI(Aws::SDKOptions const&) + 1231
  * frame #16: 0xe3f9013e libnative-lib.so`helloworld::LoginExecutive::create(context=0xfff92ec8) + 110 at LoginExecutiveImpl.cpp:18
    frame #17: 0xe3f8a618 libnative-lib.so`::Java_com_doclily_doclily_LoginExecutive_create(jniEnv=0xf434e2b0, (null)=0xfff92f0c, j_context=0xfff92f10) + 104 at NativeLoginExecutive.cpp:28
    frame #18: 0xe30633e7

Does your code initialize OpenSSL or libcurl anywhere else?
Sent from my iPhone

On Nov 18, 2016, at 2:28 AM, Chris Simmons<EMAIL_ADDRESS>wrote:
As soon as my code calls InitAPI, it hits a seg fault. This is my code, which just tries to call InitAPI and ShutdownAPI with default options:
std::shared_ptr<LoginExecutive>
LoginExecutive::create(
    const LoginExecutiveContext & context
    )
{
    Aws::SDKOptions options;
    Aws::InitAPI(options);
    Aws::ShutdownAPI(options);
    return std::make_shared<LoginExecutiveImpl>(context);
}

This is the stack trace of the crash:
(lldb) bt

thread #1: tid = 378, 0xf76c0d33 libc.sopthread_mutex_lock + 35, name = 'doclily.doclily', stop reason = signal SIGSEGV: invalid address (fault address: 0x4)   frame #0: 0xf76c0d33 libc.sopthread_mutex_lock + 35
frame #1: 0xe1a5cbb1 libc++_shared.sostd::__ndk1::mutex::lock() + 33   frame #2: 0xe1624af8 libaws-cpp-sdk-core.soAws::Utils::Crypto::OpenSSL::locking_fn(int, int, char const*, int) + 88
frame #3: 0xe17af9ba libaws-cpp-sdk-core.soCRYPTO_lock + 266   frame #4: 0xe1805735 libaws-cpp-sdk-core.so___lldb_unnamed_symbol1057$$libaws-cpp-sdk-core.so + 101
frame #5: 0xe1805944 libaws-cpp-sdk-core.so___lldb_unnamed_symbol1059$$libaws-cpp-sdk-core.so + 68   frame #6: 0xe1804db8 libaws-cpp-sdk-core.soERR_func_error_string + 120
frame #7: 0xe17b4bd6 libaws-cpp-sdk-core.soERR_load_CRYPTO_strings + 38   frame #8: 0xe16248fd libaws-cpp-sdk-core.soAws::Utils::Crypto::OpenSSL::init_static_state() + 29
frame #9: 0xe16747b2 libaws-cpp-sdk-core.sovoid std::__ndk1::__invoke_void_return_wrapper<void>::__call<void (*&)()>(void (*&&&)()) + 50   frame #10: 0xe1674695 libaws-cpp-sdk-core.sostd::__ndk1::__function::__func<void ()(), std::__ndk1::allocator<void ()()>, void ()>::operator()() + 53
frame #11: 0xe1611b19 libaws-cpp-sdk-core.sostd::__ndk1::function<void ()>::operator()() const + 41   frame #12: 0xe1689876 libaws-cpp-sdk-core.soAws::Utils::GetTheLights::EnterRoom(std::__ndk1::function<void ()>&&) + 294
frame #13: 0xe1672e2b libaws-cpp-sdk-core.soDefaultMD5Factory::InitStaticState() + 107   frame #14: 0xe166ecd1 libaws-cpp-sdk-core.soAws::Utils::Crypto::InitCrypto() + 705
frame #15: 0xe15a765f libaws-cpp-sdk-core.so`Aws::InitAPI(Aws::SDKOptions const&) + 1231

frame #16: 0xe3f9013e libnative-lib.sohelloworld::LoginExecutive::create(context=0xfff92ec8) + 110 at LoginExecutiveImpl.cpp:18 frame #17: 0xe3f8a618 libnative-lib.so::Java_com_doclily_doclily_LoginExecutive_create(jniEnv=0xf434e2b0, (null)=0xfff92f0c, j_context=0xfff92f10) + 104 at NativeLoginExecutive.cpp:28
frame #18: 0xe30633e7
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Not to my knowledge. Should it?

Actually, I now see that my code is getting called twice, which means that those libraries are getting initialized elsewhere, in the first call to InitAPI, which does succeed. I didn't expect my code to get called twice; there may be a bug on my end.

Would you mind sharing your build, execution, and debug (for the android app itself) setup?  Android debugging is something I've struggled with (pretty much log-only so far) and the fact that you're sitting in LLDB has me very jealous.

No, if you are, you have to tell the SDK not to via the SDKOptions.
Sent from my iPhone

On Nov 18, 2016, at 9:04 AM, Chris Simmons<EMAIL_ADDRESS>wrote:
Actually, I now see that my code is getting called twice, which means that those libraries are getting initialized elsewhere, in the first call to InitAPI, which does succeed. I didn't expect my code to get called twice; there may be a bug on my end.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

I now see that the documentation for ShutdownAPI states, "Do not call any other SDK methods after calling ShutdownAPI." I'm calling InitAPI after a call to ShutdownAPI; I think this is my bug.
@bretambrose I'm not sure exactly what information you're looking for regarding my development setup, but here's some info: My OS is OS X El Capitan. Android Studio 2.2.2 and the default tools it uses (Gradle, LLDB) handle most things related to build, execution, and debug. I'm running the app on a Genymotion 2.8.0 emulator.

It's complicated. As long as you can guarantee that all memory and modules are deleted before calling shutdownAPI, it should be fine. The other day, I actually accidentally had a segfault in my own code due to this.
In general non-trivial statics and long lived memory on the heap is your enemy, and will cause results like this.
Sent from my iPhone

On Nov 18, 2016, at 9:39 AM, Chris Simmons<EMAIL_ADDRESS>wrote:
I now see that the documentation for ShutdownAPI states, "Do not call any other SDK methods after calling ShutdownAPI." I'm calling InitAPI after a call to ShutdownAPI; I think this is my bug.
@bretambrose I'm not sure exactly what information you're looking for regarding my development setup, but here's some info: My OS is OS X El Capitan. Android Studio 2.2.2 and the default tools it uses (Gradle, LLDB) handle most things related to build, execution, and debug. I'm running the app on a Genymotion 2.8.0 emulator.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

I have a similar situation but only happen in a specific context. I'm trying to hook up S3 uploading feature in Unreal Engine 4 but segmentation fault always comes to me when executing Aws::InitAPI call. What confuses me is that the exact same code works on Unreal Engine 4 for Mac and Linux (using aws-doc-sdk-examples) except for Unreal Engine 4 for Linux. Also, when I compiled for -DNO_ENCRYPTION=ON, the unit test for sdk fails at AWSAuthV4SignerTest. Is there something you can point me that I should look into? Thanks in advance. @JonathanHenson
My configurations:

Ubuntu 16.04 LTS
libssl-dev: 1.0.2g-1ubuntu4.5
Unreal Engine 4.14

gdb output with backtrace
Thread 1 "UE4Editor" received signal SIGSEGV, Segmentation fault.
0x00000000029ccea0 in ?? ()
(gdb)
(gdb) bt
#0  0x00000000029ccea0 in ?? ()
#1  0x00007fff8468e39d in std::function<std::shared_ptr<Aws::Utils::Crypto::SymmetricCipherFactory> ()>::operator()() const ()
   from /usr/local/lib/libaws-cpp-sdk-core.so
#2  0x00007fff8468de04 in Aws::InitAPI(Aws::SDKOptions const&) ()
   from /usr/local/lib/libaws-cpp-sdk-core.so
#3  0x00007fff849c6edd in UAWSClient::UploadFileToS3 (FileName=...,
    FileAbsolutePath=...)
    at /home/ray/YSProjects/TestBed/Source/TestBed/Private/AWSClient.cpp:25
#4  0x00007fff849ca169 in UAWSClient::execUploadFileToS3 (
    this=<optimized out>, Stack=..., Z_Param__Result=<optimized out>)
    at ../../../YSProjects/TestBed/Source/TestBed/Public/AWSClient.h:19
#5  0x00007ffff66ecf22 in UFunction::Invoke (this=<optimized out>,
    Obj=<optimized out>, Stack=..., Z_Param__Result=<optimized out>)
    at /home/ray/UnrealEngine/Engine/Source/Runtime/CoreUObject/Private/UObject/Class.cpp:4474
#6  0x00007ffff6842237 in UObject::CallFunction (this=0x7fff89218fc0,
    Stack=..., Z_Param__Result=0x7fffffffa830, Function=<optimized out>)
    at /home/ray/UnrealEngine/Engine/Source/Runtime/CoreUObject/Private/UObject/ScriptCore.cpp:700
#7  0x00007ffff684b3f5 in FFrame::Step (this=0x7fffffffa950,
    Z_Param__Result=0x7fffffffa830, Context=<optimized out>)
    at /home/ray/UnrealEngine/Engine/Source/Runtime/CoreUObject/Private/UObject/---Typ------Type <return> to continu-----Type <r---Type <return> to continue, or q <return> to quit---

NO_ENCRYPTION means you are supplying your own crypto code via the factories in Aws::CryptoOptions and we will not attempt to compile the platform defaults (this is largely for playstation and xbox or other embedded platforms where they may want to do hardware level crypto). If libcrypto, commoncrypto or msft cng, are on the platform you are building against, leave that option on unless you have a very special use case. If you do turn it off, you are responsible for providing your own implementation via the CryptoOptions.
Sent from my iPhone

On Jan 29, 2017, at 11:03 AM, Ray Chen<EMAIL_ADDRESS>wrote:
I have a similar situation but only happen in a specific context. I'm trying to hook up S3 uploading feature in Unreal Engine 4 but segmentation fault always comes to me when executing Aws::InitAPI call. What confuses me is that the exact same code works on Unreal Engine 4 for Mac and Linux (using aws-doc-sdk-examples) except for Unreal Engine 4 for Linux. Also, when I compiled for -DNO_ENCRYPTION=ON, the unit test for sdk fails at AWSAuthV4SignerTest. Is there something you can point me that I should look into? Thanks in advance. @JonathanHenson
My configurations:
Ubuntu 16.04 LTS
libssl-dev: 1.0.2g-1ubuntu4.5
Unreal Engine 4.14
gdb output with backtrace
Thread 1 "UE4Editor" received signal SIGSEGV, Segmentation fault.
0x00000000029ccea0 in ?? ()
(gdb)
(gdb) bt
#0  0x00000000029ccea0 in ?? ()
#1  0x00007fff8468e39d in std::function<std::shared_ptrAws::Utils::Crypto::SymmetricCipherFactory ()>::operator()() const ()
from /usr/local/lib/libaws-cpp-sdk-core.so
#2  0x00007fff8468de04 in Aws::InitAPI(Aws::SDKOptions const&) ()
from /usr/local/lib/libaws-cpp-sdk-core.so
#3  0x00007fff849c6edd in UAWSClient::UploadFileToS3 (FileName=...,
FileAbsolutePath=...)
at /home/ray/YSProjects/TestBed/Source/TestBed/Private/AWSClient.cpp:25
#4  0x00007fff849ca169 in UAWSClient::execUploadFileToS3 (
this=, Stack=..., Z_Param__Result=)
at ../../../YSProjects/TestBed/Source/TestBed/Public/AWSClient.h:19
#5  0x00007ffff66ecf22 in UFunction::Invoke (this=,
Obj=, Stack=..., Z_Param__Result=)
at /home/ray/UnrealEngine/Engine/Source/Runtime/CoreUObject/Private/UObject/Class.cpp:4474
#6  0x00007ffff6842237 in UObject::CallFunction (this=0x7fff89218fc0,
Stack=..., Z_Param__Result=0x7fffffffa830, Function=)
at /home/ray/UnrealEngine/Engine/Source/Runtime/CoreUObject/Private/UObject/ScriptCore.cpp:700
#7  0x00007ffff684b3f5 in FFrame::Step (this=0x7fffffffa950,
Z_Param__Result=0x7fffffffa830, Context=)
at /home/ray/UnrealEngine/Engine/Source/Runtime/CoreUObject/Private/UObject/---Typ------Type  to continu-----Type <r---Type  to continue, or q  to quit---
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

as for your segfault, is anywhere else in your application using libcurl or openssl?
Sent from my iPhone

On Jan 29, 2017, at 11:03 AM, Ray Chen<EMAIL_ADDRESS>wrote:
I have a similar situation but only happen in a specific context. I'm trying to hook up S3 uploading feature in Unreal Engine 4 but segmentation fault always comes to me when executing Aws::InitAPI call. What confuses me is that the exact same code works on Unreal Engine 4 for Mac and Linux (using aws-doc-sdk-examples) except for Unreal Engine 4 for Linux. Also, when I compiled for -DNO_ENCRYPTION=ON, the unit test for sdk fails at AWSAuthV4SignerTest. Is there something you can point me that I should look into? Thanks in advance. @JonathanHenson
My configurations:
Ubuntu 16.04 LTS
libssl-dev: 1.0.2g-1ubuntu4.5
Unreal Engine 4.14
gdb output with backtrace
Thread 1 "UE4Editor" received signal SIGSEGV, Segmentation fault.
0x00000000029ccea0 in ?? ()
(gdb)
(gdb) bt
#0  0x00000000029ccea0 in ?? ()
#1  0x00007fff8468e39d in std::function<std::shared_ptrAws::Utils::Crypto::SymmetricCipherFactory ()>::operator()() const ()
from /usr/local/lib/libaws-cpp-sdk-core.so
#2  0x00007fff8468de04 in Aws::InitAPI(Aws::SDKOptions const&) ()
from /usr/local/lib/libaws-cpp-sdk-core.so
#3  0x00007fff849c6edd in UAWSClient::UploadFileToS3 (FileName=...,
FileAbsolutePath=...)
at /home/ray/YSProjects/TestBed/Source/TestBed/Private/AWSClient.cpp:25
#4  0x00007fff849ca169 in UAWSClient::execUploadFileToS3 (
this=, Stack=..., Z_Param__Result=)
at ../../../YSProjects/TestBed/Source/TestBed/Public/AWSClient.h:19
#5  0x00007ffff66ecf22 in UFunction::Invoke (this=,
Obj=, Stack=..., Z_Param__Result=)
at /home/ray/UnrealEngine/Engine/Source/Runtime/CoreUObject/Private/UObject/Class.cpp:4474
#6  0x00007ffff6842237 in UObject::CallFunction (this=0x7fff89218fc0,
Stack=..., Z_Param__Result=0x7fffffffa830, Function=)
at /home/ray/UnrealEngine/Engine/Source/Runtime/CoreUObject/Private/UObject/ScriptCore.cpp:700
#7  0x00007ffff684b3f5 in FFrame::Step (this=0x7fffffffa950,
Z_Param__Result=0x7fffffffa830, Context=)
at /home/ray/UnrealEngine/Engine/Source/Runtime/CoreUObject/Private/UObject/---Typ------Type  to continu-----Type <r---Type  to continue, or q  to quit---
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Not that I'm aware of. The AWS part is the only place that needs libcurl and openssl.

looks like you are segfaulting in initialization because you have not built the crypto layer. Is this running on a build that has NO_ENCRYPTION turned on? If so, that's your problem.
Sent from my iPhone

On Jan 29, 2017, at 11:36 AM, Ray Chen<EMAIL_ADDRESS>wrote:
Not that I'm aware of. The AWS part is the only place that needs libcurl and openssl.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Sorry I should have clarify that. The segfault wasn't resulting from enabling NO_ENCRYPTION. I was using cmake -DBUILD_ONLY="s3" <path-to-sdk-source> across platforms, which works for all the other except Unreal Linux. NO_ENCRYPTION is what I experiment to see if that's related.

any chance you can upload some valgrind output?
Sent from my iPhone

On Jan 29, 2017, at 12:40 PM, Ray Chen<EMAIL_ADDRESS>wrote:
Sorry I should have clarify that. The segfault wasn't resulting from enabling NO_ENCRYPTION. I was using cmake -DBUILD_ONLY="s3"  across platforms, which works for all the other except Unreal Linux. NO_ENCRYPTION is what I experiment to see if that's related.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Sure, I haven't used valgrind before but I'll give it a try. Unrelated question, sometime when I make some tests failed. Do you know why is that? credential is sitting in /.aws/ and never changed.
----------] Global test environment tear-down
[==========] 198 tests from 31 test cases ran. (1143 ms total)
[  PASSED  ] 195 tests.
[  FAILED  ] 3 tests, listed below:
[  FAILED  ] ProfileConfigFileAWSCredentialsProviderTest.TestDefaultConfig
[  FAILED  ] EnvironmentModifyingTest.ProfileConfigTestWithEnvVars
[  FAILED  ] EnvironmentModifyingTest.ProfileConfigTestWithEnvVarsButSpecifiedProfile

This valgrind output captures as soon as the Aws::InitAPI executes. Thanks for checking @JonathanHenson. I'm more than happy to provide any additional information you need.
~ valgrind ./UE4Editor "test.uproject"
...
==31245== Thread 1:
==31245== Conditional jump or move depends on uninitialised value(s)
==31245==    at 0x621CDC6: UScriptStruct::CompareScriptStruct(void const*, void const*, unsigned int) const (Class.cpp:2238)
==31245==    by 0x621CDC3: UScriptStruct::CompareScriptStruct(void const*, void const*, unsigned int) const (Class.cpp:2238)
==31245==    by 0x621CDC3: UScriptStruct::CompareScriptStruct(void const*, void const*, unsigned int) const (Class.cpp:2238)
==31245==    by 0x621CDC3: UScriptStruct::CompareScriptStruct(void const*, void const*, unsigned int) const (Class.cpp:2238)
==31245==    by 0x62F4CEF: UArrayProperty::Identical(void const*, void const*, unsigned int) const (PropertyArray.cpp:48)
==31245==    by 0x621483A: UStruct::SerializeTaggedProperties(FArchive&, unsigned char*, UStruct*, unsigned char*, UObject const*) const (Class.cpp:1129)
==31245==    by 0x62B5A76: UObject::SerializeScriptProperties(FArchive&) const (Obj.cpp:1130)
==31245==    by 0x62ABFDD: UObject::Serialize(FArchive&) (Obj.cpp:1060)
==31245==    by 0x7A0E429: AActor::Serialize(FArchive&) (Actor.cpp:538)
==31245==    by 0x8BC251E: AWorldSettings::Serialize(FArchive&) (WorldSettings.cpp:201)
==31245==    by 0x63A5D12: StaticDuplicateObjectEx(FObjectDuplicationParameters&) (UObjectGlobals.cpp:2028)
==31245==    by 0x63A5497: StaticDuplicateObject(UObject const*, UObject*, FName, EObjectFlags, UClass*, EDuplicateMode::Type, EInternalObjectFlags) (UObjectGlobals.cpp:1935)
==31245==
==31245== Conditional jump or move depends on uninitialised value(s)
==31245==    at 0x71AEFD92: Aws::InitAPI(Aws::SDKOptions const&) (in /usr/local/lib/libaws-cpp-sdk-core.so)
==31245==    by 0x71724EDC: UAWSClient::UploadFileToS3(FString const&, FString const&) (AWSClient.cpp:25)
==31245==    by 0x71728168: UAWSClient::execUploadFileToS3(FFrame&, void*) (AWSClient.h:19)
==31245==    by 0x6225F21: UFunction::Invoke(UObject*, FFrame&, void*) (Class.cpp:4474)
==31245==    by 0x637B236: UObject::CallFunction(FFrame&, void*, UFunction*) (ScriptCore.cpp:700)
==31245==    by 0x63843F4: Step (ScriptCore.cpp:265)
==31245==    by 0x63843F4: UObject::ProcessContextOpcode(FFrame&, void*, bool) (ScriptCore.cpp:2136)
==31245==    by 0x637C088: Step (ScriptCore.cpp:265)
==31245==    by 0x637C088: UObject::ProcessInternal(FFrame&, void*) (ScriptCore.cpp:952)
==31245==    by 0x637AD12: UObject::CallFunction(FFrame&, void*, UFunction*) (ScriptCore.cpp:821)
==31245==    by 0x637C088: Step (ScriptCore.cpp:265)
==31245==    by 0x637C088: UObject::ProcessInternal(FFrame&, void*) (ScriptCore.cpp:952)
==31245==    by 0x6225F21: UFunction::Invoke(UObject*, FFrame&, void*) (Class.cpp:4474)
==31245==    by 0x637F29C: UObject::ProcessEvent(UFunction*, void*) (ScriptCore.cpp:1301)
==31245==    by 0x7A22ED8: AActor::BeginPlay() (Actor.cpp:3046)
==31245==
==31245== Conditional jump or move depends on uninitialised value(s)
==31245==    at 0x71AF037C: std::function<std::shared_ptr<Aws::Utils::Crypto::SymmetricCipherFactory> ()>::operator()() const (in /usr/local/lib/libaws-cpp-sdk-core.so)
==31245==    by 0x71AEFDB4: Aws::InitAPI(Aws::SDKOptions const&) (in /usr/local/lib/libaws-cpp-sdk-core.so)
==31245==    by 0x71724EDC: UAWSClient::UploadFileToS3(FString const&, FString const&) (AWSClient.cpp:25)
==31245==    by 0x71728168: UAWSClient::execUploadFileToS3(FFrame&, void*) (AWSClient.h:19)
==31245==    by 0x6225F21: UFunction::Invoke(UObject*, FFrame&, void*) (Class.cpp:4474)
==31245==    by 0x637B236: UObject::CallFunction(FFrame&, void*, UFunction*) (ScriptCore.cpp:700)
==31245==    by 0x63843F4: Step (ScriptCore.cpp:265)
==31245==    by 0x63843F4: UObject::ProcessContextOpcode(FFrame&, void*, bool) (ScriptCore.cpp:2136)
==31245==    by 0x637C088: Step (ScriptCore.cpp:265)
==31245==    by 0x637C088: UObject::ProcessInternal(FFrame&, void*) (ScriptCore.cpp:952)
==31245==    by 0x637AD12: UObject::CallFunction(FFrame&, void*, UFunction*) (ScriptCore.cpp:821)
==31245==    by 0x637C088: Step (ScriptCore.cpp:265)
==31245==    by 0x637C088: UObject::ProcessInternal(FFrame&, void*) (ScriptCore.cpp:952)
==31245==    by 0x6225F21: UFunction::Invoke(UObject*, FFrame&, void*) (Class.cpp:4474)
==31245==    by 0x637F29C: UObject::ProcessEvent(UFunction*, void*) (ScriptCore.cpp:1301)
==31245==
==31245== Use of uninitialised value of size 8
==31245==    at 0x71AF039B: std::function<std::shared_ptr<Aws::Utils::Crypto::SymmetricCipherFactory> ()>::operator()() const (in /usr/local/lib/libaws-cpp-sdk-core.so)
==31245==    by 0x71AEFDB4: Aws::InitAPI(Aws::SDKOptions const&) (in /usr/local/lib/libaws-cpp-sdk-core.so)
==31245==    by 0x71724EDC: UAWSClient::UploadFileToS3(FString const&, FString const&) (AWSClient.cpp:25)
==31245==    by 0x71728168: UAWSClient::execUploadFileToS3(FFrame&, void*) (AWSClient.h:19)
==31245==    by 0x6225F21: UFunction::Invoke(UObject*, FFrame&, void*) (Class.cpp:4474)
==31245==    by 0x637B236: UObject::CallFunction(FFrame&, void*, UFunction*) (ScriptCore.cpp:700)
==31245==    by 0x63843F4: Step (ScriptCore.cpp:265)
==31245==    by 0x63843F4: UObject::ProcessContextOpcode(FFrame&, void*, bool) (ScriptCore.cpp:2136)
==31245==    by 0x637C088: Step (ScriptCore.cpp:265)
==31245==    by 0x637C088: UObject::ProcessInternal(FFrame&, void*) (ScriptCore.cpp:952)
==31245==    by 0x637AD12: UObject::CallFunction(FFrame&, void*, UFunction*) (ScriptCore.cpp:821)
==31245==    by 0x637C088: Step (ScriptCore.cpp:265)
==31245==    by 0x637C088: UObject::ProcessInternal(FFrame&, void*) (ScriptCore.cpp:952)
==31245==    by 0x6225F21: UFunction::Invoke(UObject*, FFrame&, void*) (Class.cpp:4474)
==31245==    by 0x637F29C: UObject::ProcessEvent(UFunction*, void*) (ScriptCore.cpp:1301)
==31245==
==31245== Jump to the invalid address stated on the next line
==31245==    at 0x0: ???
==31245==    by 0x71AEFDB4: Aws::InitAPI(Aws::SDKOptions const&) (in /usr/local/lib/libaws-cpp-sdk-core.so)
==31245==    by 0x71724EDC: UAWSClient::UploadFileToS3(FString const&, FString const&) (AWSClient.cpp:25)
==31245==    by 0x71728168: UAWSClient::execUploadFileToS3(FFrame&, void*) (AWSClient.h:19)
==31245==    by 0x6225F21: UFunction::Invoke(UObject*, FFrame&, void*) (Class.cpp:4474)
==31245==    by 0x637B236: UObject::CallFunction(FFrame&, void*, UFunction*) (ScriptCore.cpp:700)
==31245==    by 0x63843F4: Step (ScriptCore.cpp:265)
==31245==    by 0x63843F4: UObject::ProcessContextOpcode(FFrame&, void*, bool) (ScriptCore.cpp:2136)
==31245==    by 0x637C088: Step (ScriptCore.cpp:265)
==31245==    by 0x637C088: UObject::ProcessInternal(FFrame&, void*) (ScriptCore.cpp:952)
==31245==    by 0x637AD12: UObject::CallFunction(FFrame&, void*, UFunction*) (ScriptCore.cpp:821)
==31245==    by 0x637C088: Step (ScriptCore.cpp:265)
==31245==    by 0x637C088: UObject::ProcessInternal(FFrame&, void*) (ScriptCore.cpp:952)
==31245==    by 0x6225F21: UFunction::Invoke(UObject*, FFrame&, void*) (Class.cpp:4474)
==31245==    by 0x637F29C: UObject::ProcessEvent(UFunction*, void*) (ScriptCore.cpp:1301)
==31245==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==31245==
Signal 11 caught.
Malloc Size=131076 LargeMemoryPoolOffset=131092
Malloc Size=65535 LargeMemoryPoolOffset=196655
Malloc Size=112917 LargeMemoryPoolOffset=309589
==31245==
==31245== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==31245==    at 0x4E4A269: raise (pt-raise.c:35)
==31245==    by 0x53ECF2E: FLinuxCrashContext::GenerateCrashInfoAndLaunchReporter(bool) const (LinuxPlatformCrashContext.cpp:523)
==31245==    by 0x53F00B4: PlatformCrashHandler(int, siginfo_t*, void*) (LinuxPlatformCrashContext.cpp:576)
==31245==    by 0x4E4A38F: ??? (in /lib/x86_64-linux-gnu/libpthread-2.23.so)
==31245==
==31245== HEAP SUMMARY:
==31245==     in use at exit: 378,178,593 bytes in 66,590 blocks
==31245==   total heap usage: 390,248 allocs, 323,658 frees, 8,387,603,552 bytes allocated
==31245==
==31245== LEAK SUMMARY:
==31245==    definitely lost: 86,528 bytes in 689 blocks
==31245==    indirectly lost: 599,464 bytes in 5,344 blocks
==31245==      possibly lost: 332,731,704 bytes in 2,782 blocks
==31245==    still reachable: 44,760,897 bytes in 57,775 blocks
==31245==                       of which reachable via heuristic:
==31245==                         stdstring          : 192 bytes in 6 blocks
==31245==                         newarray           : 16 bytes in 1 blocks
==31245==                         multipleinheritance: 104 bytes in 1 blocks
==31245==         suppressed: 0 bytes in 0 blocks
==31245== Rerun with --leak-check=full to see details of leaked memory
==31245==
==31245== For counts of detected and suppressed errors, rerun with: -v
==31245== Use --track-origins=yes to see where uninitialised values come from
==31245== ERROR SUMMARY: 11074 errors from 121 contexts (suppressed: 0 from 0)
Segmentation fault (core dumped)

This should not be possible without you explicitly setting one of the callback functions in CryptoOptions, could you upload the full text of UAWSClient::UploadfileToS3 so I can have a look?

Sure
static const Aws::String BUCKET = "BUCKET_NAME";
static const Aws::String AWS_ACCESS_KEY_ID = "ACCESS_KEY_ID ";
static const Aws::String AWS_SECRET_ACCESS_KEY = "SECRET_ACCESS_KEY";
static const std::string DEFAULT_FILE_PATH = "PATH";

void UAWSClient::UploadFileToS3(const FString& FileName, const FString& FileAbsolutePath)
{
    const Aws::String key_name = TCHAR_TO_UTF8(*FileName);
    std::string filePATH = DEFAULT_FILE_PATH;

    if (!FileAbsolutePath.IsEmpty())
    {
        filePATH = TCHAR_TO_UTF8(*FileAbsolutePath);
    }

    Aws::SDKOptions options;

    // Where segfault happens
    Aws::InitAPI(options);

    Aws::Client::ClientConfiguration clientConfig;
    clientConfig.region = Aws::Region::AP_NORTHEAST_1;
    clientConfig.connectTimeoutMs = 30000;
    clientConfig.requestTimeoutMs = 600000;

    auto s3_client = Aws::MakeShared<Aws::S3::S3Client>("S3Client", Aws::Auth::AWSCredentials(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY), clientConfig);

    Aws::S3::Model::PutObjectRequest object_request;
    object_request.WithBucket(BUCKET).WithKey(key_name);

    std::shared_ptr<Aws::FStream> fileToUpload = Aws::MakeShared<Aws::FStream>("FileToUpload", filePATH, std::ios::in | std::ios::binary);
    object_request.SetBody(fileToUpload);
    object_request.SetContentType("image/jpeg"); // "text/plain"

    auto put_object_outcome = s3_client->PutObject(object_request);

    if(put_object_outcome.IsSuccess()) {
        UE_LOG(LogTemp, Warning, TEXT("S3 Uploaded Successfully"));
    }
    else {
        UE_LOG(LogTemp, Warning, TEXT("S3 Uploaded Failed"));
    }

    Aws::ShutdownAPI(options);
}

key_name is going to have its allocator swapped out from under it, when you call InitAPI(). You shouldn't use any thing from one of our header files before you call InitAPI().
Also InitAPI needs to go way way up the chain inside
int main(char** argv, int argv)
{
Aws::InitAPI()
//initialize the rest of the game and the engine.
//those components will presumably do AWS things.
Aws::CleanupAPI()
}
Also, as written, this will segfault at ShutdownAPI anyways, because ShutdownAPI will be called before clientConfig, s3_client, and the rest gets cleaned up. You would need scoping brackets to make that work.

For reference you could have a look at the AWS_Lmbr components in the Lumberyard Engine.

Just ran into this exact symptom (same stack as the OP), on the 2nd call to InitializeAPI.  Is it OK to call InitializeAPI after ShutdownAPI, or not?  That is, as long as every InitializeAPI has a corresponding ShutdownAPI that is called before the next InitializeAPI, should that work?

in theory, it would work, but you should not do it. Put InitAPI in the entry point of your application as early as possible. a.) Its expensive. b.) it manipulates global state. c.) on linux it alters state of openssl and libcurl. d.) Nothing guarantees the compiler does not change the ordering of your calls.
e.) That will not work in a multi-threaded scenario.
Sent from my iPhone

On Jan 30, 2017, at 5:36 PM, Andy Dennie<EMAIL_ADDRESS>wrote:
Just ran into this exact symptom (same stack as the OP), on the 2nd call to InitializeAPI. Is it OK to call InitializeAPI after ShutdownAPI, or not? That is, as long as every InitializeAPI has a corresponding ShutdownAPI that is called before the next InitializeAPI, should that work?
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub, or mute the thread.

also, that will shutdown the logging thread. It will likely not start back up on the next call to InitAPI()
Sent from my iPhone

On Jan 30, 2017, at 5:36 PM, Andy Dennie<EMAIL_ADDRESS>wrote:
Just ran into this exact symptom (same stack as the OP), on the 2nd call to InitializeAPI. Is it OK to call InitializeAPI after ShutdownAPI, or not? That is, as long as every InitializeAPI has a corresponding ShutdownAPI that is called before the next InitializeAPI, should that work?
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub, or mute the thread.

It turns out that the segfault is resulting from std::string for my case. Because UE4 is using libc++ for the engine while Linux uses libstdc++ by default. After switching to libc++ it's all working fine.

Closing

No, if you are, you have to tell the SDK not to via the SDKOptions.
Sent from my iPhone

On Nov 18, 2016, at 9:04 AM, Chris Simmons<EMAIL_ADDRESS>wrote:
Actually, I now see that my code is getting called twice, which means that those libraries are getting initialized elsewhere, in the first call to InitAPI, which does succeed. I didn't expect my code to get called twice; there may be a bug on my end.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

Hey @JonathanHenson : You mentioned in one of your post 'you have to tell the SDK not to via the SDKOptions.'. My application is getting crashed while InitAPI is getting called on Android. I also use licurl internally for other REST calls. How to tell AWS SDK not to initialize the libcurl again?? BTW the same code works fine on Windows.


Proper facades + lots of other PRs

Okay this approach is working pretty well. It doesn't do the big crazy restructure just yet as far as the paths to source code, package names, etc go. But it does properly hook up Android and even the GWT facade as separate Git repos you can fetch simply via Gradle: gradlew fetchAndroid

Android facade: https://github.com/MovingBlocks/DestSolAndroid
GWT facade: https://github.com/MovingBlocks/DestSolGwt

Haven't tried to automate the res copy into assets under android yet but if I do that and prep IntelliJ the Android version works on my phone just fine.
@SimonC4 - what did you do to get Android working via Gradle execution? I renamed the task so gradlew android will now attempt to run from the command line. I don't seem to have any devices set up so we need to add a quick bit to the instructions on how to do that. I did some initial readme updates.
Includes most other PRs: #19, #20, #21, and #24 so it should fix #10 and #23
Builds in Jenkins fine: http://jenkins.terasology.org/job/DestinationSolTest - since that builds without Android or GWT none of that stuff gets in the way.
However one problem remains: We've changed around the jars shipped in /libs and it looks like sol.exe isn't super happy for that. We may have to regenerate the exe. I tried renaming the plain GDX jar to its old name and got a Java error popup rather than nothing at all. But some of the other jars have been split out and we have more of them now.
After we figure that out I think this can finally be merged to master :-)
I'll leave around #22 just for review for when we do move more stuff around, but hopefully after merging / redoing #17

I am able to run this on an Android emulator following these steps. This will install the Destination Sol app and can then be run from the device.

gradlew fetchAndroid
create local.properties file with sdk.dir=[sdk path]
Copy res into assets
Make sure emulator is running - or Android device is connection
gradlew android:installDebug

I quickly tried using IntelliJ, but was unsuccessful. Maybe list your steps as I think I'm missing something.
I'm curious about the distribution issue. It looks like all the jars are there. I was able to get it to run on Mac by using my own installed java as there is no Mac dist yet. I essentially did what was stated in this thread: http://steamcommunity.com/app/342980/discussions/1/611698195147641156/ I'd be curious to see if you can get that to run.
I'm assuming that sol.exe doesn't do anything special and would be the same as the linux equivalent, which is one line:
lwjreLinux/bin/java -jar libs/solDesktop.jar
I'll try it out on Windows when I get a chance.

Quick update: I copied the DestinationSol.zip over to Windows and I was able to get it to work without any issues. Not sure what's different to your setup. I did do a git clean when I checked this branch out, to remove any leftover bits.

Here's a link to the zip file if you want to try it out.
https://www.dropbox.com/s/uip8cogwoa6z8qh/DestinationSol.zip?dl=0

Oh I can make it run with java -jar libs/solDesktop.jar just fine. It is only sol.exe that doesn't work for me. Does it work for you on Windows?

Sorry, I should have been more specific, sol.exe does work for me on
Windows.
On 2 June 2015 at 05:12, Rasmus Praestholm<EMAIL_ADDRESS>wrote:

Oh I can make it run with java -jar libs/solDesktop.jar just fine. It is
only sol.exe that doesn't work for me. Does it work for you on Windows?
—
Reply to this email directly or view it on GitHub
https://github.com/MovingBlocks/DestinationSol/pull/25#issuecomment-107642520
.

Thanks - huh that's weird. Was that the zip from Jenkins or one built locally? Not that it should matter. We might need to dig deeper but if more can confirm it as OK then at least we can merge.
Pinging @PrivateAlpha @theotherjay @LinusVanElswijk
As for the Android setup in IntelliJ I put the rough steps I use in the readme, but can make it more specific when I get a chance :-) It is weirdly inconsistent - sometimes I get it working first try, other times even with a valid key store it refuses to accept it as signed. I haven't tried entirely unsigned yet though.

That zip was built locally using "gradlew distZip". It was built on Mac, but I don't think it will matter.
When I get a a chance, I might create a thread post for the Android to remove it from this discussion and to get help working through the issues I'm having.
All dependent on having enough time though.

Yeah I hear you on the time bit - nearly 1:30 am here and I'm just getting to start on catching up on the day - or would, but I'm going to push it out and go to bed ;-)
I was able to test the .exe on my wife's PC and it worked fine. So I think that's a non-issue.
I'm ready to merge this unless you can think of any concerns :-) Then we can probably sort out the Android specifics later, weekend if needed for time zones. And yeah forum would be suitable too and might help invite others in to tinker.

I think it's good to merge.
I was able to do a clean checkout and following the instructions I was able to repeatedly run both the desktop and android app from both within IntelliJ and using gradle on the command line.
Some notes:

An alternative to running gradlew idea is from selecting Import Project from within IntelliJ. https://github.com/libgdx/libgdx/wiki/Gradle-and-Intellij-IDEA
I didn't need to do the steps of setting the android SDK or adding an artifact.
You don't need to sign the app for local dev, only on releases.
I couldn't run using gradlew android but installing using gradlew android:installDebug does work.
The recommended Android tools versions: http://libgdx.badlogicgames.com/versions.html

Done :-)
Yay for instructions!
The Android version stuff is pretty much arbitrary at this point. I spliced together what numbers I found in existing places with what I had handy locally.
We probably need to look at what the code base actually uses and how far back we want to support devices. API level 18 for instance is probably too new. But that takes some more thought and this week so far is bonkers :D

API 18 is the target SDK, not the minimum SDK. Currently the minimum is set to API 4 from memory. It's in the android manifest XML.
http://developer.android.com/guide/topics/manifest/uses-sdk-element.html

Ah yep. Lots of numbers to get used to :-)

Indeed. Nice job with all the Pull Requests. :-)

Not to get into a bad habit to comment on closed PRs but since it came up here I went through the Android setup a little more and tried gradlew android:installDebug - which worked, but just said it installed the app on my device, it didn't actually do anything.
I noticed the package (rather than the target class) in Gradle was still set to a placeholder LibGDX value and fixed it, now gradlew android works for me with my device connected, huzzah!
I dunno if that was in part because I did the installDebug first, but hey. It works :-)


Update What's changed docs https://github.com/celo-org/specs/ for SocialConnect

Nothing will change here


Redirects route

I was wondering if there were any reasons you didn't apply this to redirects as well?

We had no use-case yet, but we're open to pull requests.


LSTM gives unexpected output

What is wrong?
I’m completely new to neural networks so please correct me if I misunderstood something. Maybe this might be the expected outcome for this code.
I’m trying to train an LSTM network for detecting hate speech. When I run it I get some unexpected outcomes I can’t explain to myself.
    Pls just comit sucide'));

this gave me

toxic
happy
happy

it got the last one wrong lol and returned nothing for the first one which makes sense as there is nothing else like it but other than that it appears to be working correctly

I tried your code. The first one also returned me nothing. The next two are also fine only the last one is messed up again.
It looks like this

toxic
happy
 oe are amazinghappy

I noticed error rates were at 0.01 so it could simply be that the network needs longer to train.

I tried training the network with an iteration count of 200000 but still got weird outputs.

Please adjust hiddenLayers and learningRate for better results.


Coin control - spent coins still in list

V0.1.98-1.0
Spent coins still showing amounts and can be  selected while doing future coin control, although app does not allow to spend them again.
Soft & Hard refresh both do not fix this.
Originally had 2 frozen coins and did full sweep, then unfreezed one coin and sent remaining one with regular coin control.

Thanks for the report @JuanOfTheNorth !
We are looking into it.
Were you seeing the spent utxo while it was still unconfirmed, or does it still show up after confirmation?

Both. Coin was visible in coin control while awaiting confirmation and after confirmed.

fixed on main - should reflect on next release
thanks @JuanOfTheNorth

(Apologies if this is premature and the fix shouldn't be visible yet)
V0.1.98-1.1
Above wallet still showing spent coins in coin control.
Created new wallet and ran same tests as above, just to double-check.
Coins still showing, both while awaiting confirmation and after confirmed.

Hey, sorry we got to this late.
Taking a look now!


Add code to section "Excluding NAT interface from DNS"

Hi!
I want  you to add code to subj section.
Here my quick and dirty example of a code
Here the list what it proposed to do.

remove NAT adapters IP Resource records in DNS Server.
Uncheck option "Register this connection's addresses in dns"
Remove all RR from NAT if already registered. (unnessesary, if NAT adapter RR not exist)
restart DNS server service.

@dmi3mis can you give a bit more context about what problem this is attempting to solve?

Hi clang!
You added last line of code
...
Write-Host "Excluding NAT interface from DNS"
...
but you did not added a code to it.
this code removes NAT ip RR from domain dns zone.
3 things need to happen:

DNS server must stop listen  on NAT interface ip.
NAT interface ip must not register its connection specific ip in dns zone.
Remove NAT interface ip "A" resource records from domain dns zone, if it already added.

Looks good, no issues with testing


Rename this DSL

We need to come up with a new name for this DSL that will ideally tick these boxes:

not in use
easy to pronounce
distinct enough to search for
no negative connotations

I think points 1 and 3 are nice to have but not required as we are not thinking of this as a standalone product.

I liked @bobbinth's earlier suggestion, "abacus".

Alright, let's use Abacus! (Unless there are any final thoughts.) We can update all references as part of the v0.1 release this week.

One other option is to use "Air" in the name somehow. One option that comes to mind is "AirScript" (I've used this before, but that language is not maintained any more) - but there could be other options as well. So, if people prefer "AirScript" to "Abacus", I'm fine with that.

Sorry, I created a duplicate issue #59 for changing references to Air DSL.

Another suggestion by @bobbinth:

Thought for the future: I wonder if we should rename the executable to just air instead of air-script.

I agree about not using air-script for the executable name. We could also use airc as in "air compiler"

I like airc!

Closed by #62 and #69.


Cruft fails to update when the template is on a branch

Hello!
Thank you for your tool, it's an amazing idea!
Issue
While introducing it in a new project I noticed that cruft update will not honor the branch in the cruft.json, result being:
$ cruft update
Error: Was unable to locate a Cookiecutter template in `/var/folders/71/r1xvkqrs6qv1n1300d8gcy5m0000gn/T/tmpqpfexl6_/repo` !

Cruft Version: 2.10.1
Use-case
The use-case here is that i was running some tests with the template before merging the first version.  I don't know if it's a valid use-case, i don't know if people would keep templates on non-default branches.   If you feel this is a non-issue we can close it :), see the Workaround section at the end.
Reproduce
Here's how to reproduce

Create a template repo, leave default branch empty
Switch to a non-default branch
Commit template
Create a new project using cruft create <template repo> -c <non-default branch>
Commit a change in the template
Run cruft update
See the error above.

Reproduce env
To save you some trouble, I have the exact case reproduced here:

TEMPLATE: https://github.com/lindycoder/cruft-branch-issue-template

Branch is not-main

New project https://github.com/lindycoder/cruft-branch-issue

There are currently 2 commits on the not-main branch, cruft-branch-issue project's cruft points to the first one.  *Note the checkout: "not-main"in the cruft.json"
If you run cruft update in the new project you should get the error.
Work around
As a work around, cruft update -c not-main works.   But i would expect -c to be a way of switching branch, as the current branch information is in .cruft.json.

I can confirm I have experienced the exact issue described above.

Hello,
I am experiencing the same: I have the following .cruft.json

{
"template": "git@gitlab.com:blabla/blibli.git",
"commit": "3531eaa21dbd1deb931953c395ae5ce58e0ab708",
"context": {
"cookiecutter": {
...
}
},
"directory": null,
"checkout": "my-branch-name"
}

but the checkout parameter is ignored, I have to pass it explicitly to the command with
cruft check --checkout=my-branch-name

I have opened a PR but it is in draft as I have not written the tests yet https://github.com/cruft/cruft/pull/165


How to log in Azure in China with this extension.

Try to log into Chinese Azure Server with this. Does not see a clear configuration to do this.

you can use some other credentials to authenticate such as "connection string" if you are using iot hub or other service owning unique string which contains china azure endpoint


CHAOS-5887 Update chaos-engineering.md

Update release notes

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://66756926dd68c876d9fc258d--harness-developer.netlify.app


DEV: Create the New Container for 0.14.0

Deliver end of Sprint 14 docker image 0.14.0.  Please confirm that the trajectory return issue fix is captured in this container.
Hopefully this will address the open item c/o XiaoBin not being able to see it in 0.13.3.
For reference See Below FYI:
XiaoBin DENG [12:10 AM]
@Mike Sturm , thanks a lot. We'll try it later
XiaoBin DENG [3:20 AM]
@Mike Sturm checked newest available docker image is still 0.13.3. What we can check is against specific version, we cannot deploy Latest image. I will verify it on new image version

UPDATE: I have generated docker image 0.14.0 last night & per Chris, it should have included the Traj fix. Shehzad will be creating a Soap UI test suite for this, but it is involved. He would have to create tests that verify every property for the Traj.


K-Level Reasoning source: game, metric, result

Including two games, G08A and SAG, implemented Baseline and K-Level Reasoning methods. Additionally, experimental results (dialogue, win/loss information) are provided.

@enria please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
@microsoft-github-policy-service agree [company="{your company}"]

Options:

(default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

(when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.
@microsoft-github-policy-service agree company="Microsoft"


Add /ph rewrite

What kind of changes does this PR include?

Something else!

Description

What does this PR change? Adds a /ph rewrite to https://astro-houston-ph.pages.dev/

Looking at the preview, I'm not sure if this has worked

/ph/ worked for the page, but didn't work for subroutes. the wildcard doesn't seem to work for any of them, but does prevent the normal 404 from rendering :thinking:

Looks like this works, but I'll need to update the site to use a /ph base route

This works now! preview


Include project link in JSON and other data-specific formats

Project links are included in an HTML report, but not in JSON reports.  It should be pretty straightforward to include the project link in JSON and other data-specific formats.

Hey @eode ! Im not sure why it is in one and not the other. If you can make a PR I can take a look

Normally, I would, But I don't kniw Ruby and now isn't a good time for me to learn.  I can provide feedback and testing on this issue, though.


Azure.Data.Tables docs hard to find

Finding these docs was pretty difficult... CosmosDB Tables API kinda wins the search page rank if you are looking for examples where you can use Azure Storage Tables...)
Also having those names close is super confusing if anyone at Azure wants to improve it. Maybe there is a reason for it, but I would prefer discoverability over using the same name.
Maybe we are meant to find the Azure.Data.Tables docs ourselves but it would be nice if this page or  sdk samples was linked from https://docs.microsoft.com/en-us/azure/storage/tables/table-storage-overview page where you are learning about the resource itself

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 832d240a-84c0-c267-5f7c-2d319868b7bd
Version Independent ID: b2d297ee-89bb-0b34-e01e-87751658d17b
Content: Azure Tables client library for .NET - Azure for .NET Developers
Content Source: api/overview/azure/data.tables-readme.md
Service: tables
Product: azure
Technology: azure
GitHub Login: @maggiepint
Microsoft Alias: magpint

@maggiepint If the docs are open I'm happy to add a PR to link these docs to the overview page. Just point me in the right direction and I think that would solve this issue of discoverability

//cc: @scottaddie

Thank you for your feedback.  Tagging and routing to the team member best able to assist.

I believe @mario-guerra was working on this.

@mario-guerra is this still something you are actively looking at?

Hi @pallavit, @kyle-patterson owns Tables now.


Error when secDefinition does not exists

When you parse OAS3 files without secDefintion.scheme, the script will fail. Opeapi-to-har expects a scheme.
 authScheme = secDefinition.scheme.toLowerCase();
                                          ^
TypeError: Cannot read property 'toLowerCase' of undefined
at getHeadersArray (/node_modules/openapi-snippet/openapi-to-har.js:287:43)
    at Object.createHar [as getEndpoint] (//node_modules/openapi-snippet/openapi-to-har.js:44:14)
    at Object.getEndpointSnippets (/node_modules/openapi-snippet/index.js:31:28)
    at enrichSchema (/node_modules/snippet-enricher-cli/index.js:18:39)
    at fs.readFile (node_modules/snippet-enricher-cli/index.js:52:12)
    at FSReqWrap.readFileAfterClose [as oncomplete] (internal/fs/read_file_context.js:53:3)

True, although I'm not sure if this is something that we should be fixing inside this repo, as it's only a thin wrapper for ErikWittern/openapi-snippet.
We've hit a similar issue for API key schemes when integrating this into Flotiq's user API docs and filed a PR against ErikWittern/openapi-snippet. I believe you already traced that yours is a similar case and that there already is a PR for that in the respective repository.

True found that issue. Will close it.


Update payloads to using the ubuntu base image

enclave-cc: Fix preInstall image tag / hostPath / mountPath
We've updated the preInstall image to use Ubuntu 20.04 instead of
CentOS, but we didn't fully reflect the changes for Enclave CC.

payloads: Update to an image using ubuntu as base
Both Enclave CC and Kata Containers have switched to using a base image
for their payloads, and we should make sure it works on our side.
As part of the change, we need to update the hostPath / mountPath to
reflect what's needed in order to be able to call systemctl restart ... using the new Ubuntu base image.

/test

/test

LGTM. IIUC it is going to fix the operator that got problem by changes on kata-deploy (http://jenkins.katacontainers.io/blue/organizations/jenkins/confidential-containers-ci/detail/confidential-containers-ci/16/pipeline)

Yep, thanks @wainersm!  I still need to figure out why the enclave-cc test is not passing though.

The SEV job failed because it hit pull limit to docker.io:
13:28:22 TASK [Start a docker registry] *************************************************
13:28:24 fatal: [localhost]: FAILED! => {"changed": false, "msg": "Error pulling docker.io/library/registry - code: None message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: [https://www.docker.com/increase-rate-limit"}](https://www.docker.com/increase-rate-limit%22%7D)

Have you seen that problem often, @ryansavino , @UnmeshDeodhar ?

I've retriggered the sev job. Looks like the docker hub rate limit affects the operator ci. Maybe we should consider pulling from GHCR or quay?

The SEV job failed because it hit pull limit to docker.io:
13:28:22 TASK [Start a docker registry] *************************************************
13:28:24 fatal: [localhost]: FAILED! => {"changed": false, "msg": "Error pulling docker.io/library/registry - code: None message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: [https://www.docker.com/increase-rate-limit"}](https://www.docker.com/increase-rate-limit%22%7D)

Have you seen that problem often, @ryansavino , @UnmeshDeodhar ?

I've retriggered the sev job. Looks like the docker hub rate limit affects the operator ci. Maybe we should consider pulling from GHCR or quay?

I've opened an issue to track that issue: tests/e2e: hitting docker.io/library/registry pull rate limit and @UnmeshDeodhar will be working on a fix.

I debugged the enclave-cc failure and my conclusion is it will not work: kind nodes don't have dbus running so all systemctl commands are failing. Not sure why it worked with CentOS 7. It's using systemd-219 which is from 2015 so it could be systemctl was just very different then?
I managed to test a workaround but it has implications: when running without dbus, the pods deployed by the operator could run with hostPID: true to get access to PID 1. After that, SYSTEMD_IGNORE_CHROOT=1 systemctl works.
One observation while debugging this was that we restart systemd twice. I believe pre/post-install daemon could run unprivileged if it just copied containerd...

I debugged the enclave-cc failure and my conclusion is it will not work: kind nodes don't have dbus running so all systemctl commands are failing. Not sure why it worked with CentOS 7. It's using systemd-219 which is from 2015 so it could be systemctl was just very different then?
I managed to test a workaround but it has implications: when running without dbus, the pods deployed by the operator could run with hostPID: true to get access to PID 1. After that, SYSTEMD_IGNORE_CHROOT=1 systemctl works.
One observation while debugging this was that we restart systemd twice. I believe pre/post-install daemon could run unprivileged if it just copied containerd...

Hmm.  This ia bad, sincerely bad.

@fidencio I could try to ask kind to add dbus and explain our use-case (install custom runtimehandlers)

/test

/test

The TDX failures are do to a pre-existing error: https://github.com/kata-containers/kata-containers/issues/6529

/test

/test


Localization bug fix in localizeRoute method

Description
The CodeIgniter\Router\RouteCollection::localizeRoute method was in error in the routine that adjusts the URL for the isioma selected by the user through locale.
The bug was reported in issue #2921.
Checklist:

[x] Securely signed commits
[x] Component(s) with PHPdocs
[x] Unit testing, with >80% coverage
[ ] User guide updated
[x] Conforms to style guide

This makes sense to me. Since that wasn't caught by a test previously, could you work up a test or two to go with this?

@lonnieezell  I performed functional tests from the original code, without correction, and in fact there was no substitution for {locale} inserted in the route.
I checked the code and realized that in fact there was an error in the call to the method, and with this call it only occurred in the case of a reverse route, identifying this error could really take time.
I will work on unit tests and update soon.

@lonnieezell The requested tests have been added.


API base URL is an expert setting?

Why can I change the AOI base URL if there is deactivated "Customize expert settings"?

Happens when the preferences are opened at the first time.

Should be fixed in a recent push. Could try it out?

Thilo said it does not work on the current build

Strange.....
I can't reproduce it... However when we talked with Thi Lo, I could, and we together retried that.

Well, should work now...

I close. If it does not work for @ThiloteE we can reopen.

Works for me.


.Net: Update docs to remove references to SKFunction

What is the difference between the [SKFunction] and [KernelFunction] attributes?

On this document it is mentioned to use SKFunction attribute on ReadTextAsync method inside the DocumentPlugin - https://learn.microsoft.com/en-us/semantic-kernel/agents/plugins/?tabs=Csharp
While on this document it is mentioned to use KernelFunction attribute on SendEmailAsync method inside the EmailPlugin - https://learn.microsoft.com/en-us/semantic-kernel/agents/

So, when should I use which attribute? Please clarify, also provide any documentation link if available.
Thanks,
Jitendra

[SKFunction] was renamed to [KernelFunction]. Docs that still refer to "SKFunction" should be updated to "KernelFunction", as the former no longer exists.

Changed the name of the issue to reflect the need to update the docs.

Thanks for the clarification.

Why is this issue closed as completed? The docs are still referencing SKFunction, example below:
https://learn.microsoft.com/en-us/semantic-kernel/agents/plugins/?tabs=Csharp


Convert to Tables: Cached Activities & Local Activities

Is your feature request related to a problem? Please describe.
Feature Request/Enhancement.
Describe the solution you'd like
Converting the list of records on cached acitivities and local activities to display a table similar to how IAPP data is displayed in the app. Including pagination, sorting etc.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

@dogcomplex is this ticket a duplicate of what you're working on now?


Add Github Actions

Hi, I'm not a programmer (just do devops currently), but I'm familiar enough with Github Actions to contribute something.
Added:

A job to lint, using golangci-lint. It looks like it includes several different linters by default, and I also added golint to that list, since that's what was used in the Makefile.
A job to vet, test, then run the Makefile to build.
I also added a bit in the Makefile to set the GOPATH if it isn't set. I was having trouble with it not being set by default in the Action for some reason.

Should close issue #16.
Oh and thank you for making this. It's so so handy to move around Parameter Store like a filesystem, and the mv command has been helpful several times already. :+1:

Oh, this is fantastic. I tried it on a test branch and it found some linting issues that I need to correct. Thanks for doing this - most appreciated!


Should show alex who's turn to select clue on new rounds

Will require adding a textbox to base.ui for alex's screen.

It should also just show it in general which will be easy to implement.


feat: cairo enums in request and response parsers

Motivation and Resolution
Address issue #669, for request and response parsers.
Usage related changes
See guide
Development related changes
3 classes of enums have been created.
Checklist:

[x] Performed a self-review of the code
[x] Rebased to the last commit of the target branch (or merged it into my branch)
[x] Linked the issues which this PR resolves
[x] Documented the changes in code (API docs will be generated automatically)
[x] Updated the tests
[x] All tests are passing

Added Cairo Enums for request parser.
First message updated.

I have prepared a commit including Result tests.
Please inform me as soon as you have configured devnet with Cairo 2.1.0, then I will
commit the tests.


Clarification about the use_z argument

When we say that use_z=False when initializing a layer, does that layer behave like a Fully factored Gaussian layer? I saw that the logrm and logqm terms in the kldiv function are set to 0 when  use_z=False, but I still wanted to confirm if my understanding is correct.

Hi,
Yes that is indeed true. I will be closing this issue and the issue tracker as this repo is provided as is and not maintained any more.


feat Merge table sample to branch 2.0

Add table sample support like such SQL:
select * from test_table_sample_tbl tablesample(4 rows);
select * from test_table_sample_tbl t tablesample(20 percent);
select * from test_table_sample_tbl t tablesample(20 percent) repeatable 2;

This function has already been implemented in legacy planner, more detailed description could be found here: https://github.com/apache/doris/pull/10170
Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.28 seconds
stream load tsv:          569 seconds loaded<PHONE_NUMBER>9 Bytes, about 125 MB/s
stream load json:         21 seconds loaded<PHONE_NUMBER> Bytes, about 107 MB/s
stream load orc:          65 seconds loaded<PHONE_NUMBER> Bytes, about 16 MB/s
stream load parquet:          31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select:          29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size:<PHONE_NUMBER>7 Bytes


Bug in Code for inherited child of element with xs:string MaxLength restriction facet

Hi,
I'm not sure if you prefer pull request but I believe the following line should be:
if max_length_facet is not None and max_length > max_length_facet.value:
    raise XMLSchemaParseError("Child 'maxLength' has a greater value than parent", self)

I have attached sample xsd that results in error because min_length is None.
Code to test:
import xmlschema
path = "usage.xsd"
xs = xmlschema.XMLSchema(path)

results in:
TypeError: '>' not supported between instances of 'int' and 'NoneType'
usage.xsd.zip

Great! thanks.  My first contribution :)


Non-File FFMPEG output (rtmp etc)

Detailed Description

Context
Hi Abhishek! Your library is absolutely amazing, however, there is a feature that I think would be easy for you to implement that could make it even more powerful. This came up on stackoverflow a few days ago here. This person is pointing out that when one is trying to stream to an RTMP server using ffmpeg, the location of the server is put in the command where the name of the output file normally goes. Example:
ffmpeg -re -i file.flv -c copy -f flv rtmp://live.twitch.tv/app/<stream key>

I got this example from here.
It would be amazing if we could provide rtmp addresses instead of output file paths with WriteGear, like so:
writer = WriteGear(output_filename="rtmp://live.twitch.tv/app/<stream key>")

Your Environment

VidGear version: Latest PyPi
Branch: Master
Python version: 3
pip version: latest
Operating System and version: Ubuntu

Any Other Important Information
N/A

@esologic   Try #145 PR by installing development branch as follows:
  # clone the repository and get inside
  git clone https://github.com/abhiTronix/vidgear.git && cd vidgear

  # checkout the latest testing branch
  git checkout development

  # install normally
  pip install .

  # OR install with asyncio support
  pip install .[asyncio]

and check if it works as you expected or not? also share your result here.

@esologic Try following complete python example with WriteGear API instead:
# import required libraries
from vidgear.gears import CamGear
from vidgear.gears import WriteGear
import cv2

# Open live webcam video stream on first index(i.e. 0) device
stream = CamGear(source=0, logging=True).start()

# define required FFmpeg parameters for your writer
output_params = {"-vcodec":"libx264","-profile:v":"main","-preset:v":"medium","-r":30,"-g":60,"-keyint_min":60,"-sc_threshold":0,"-b:v":"2500k","-maxrate":"2500k","-bufsize":"2500k","-filter:v":"scale='trunc(oha/2)2:720'","-sws_flags":"lanczos+accurate_rnd","-acodec":"libfdk_aac","-b:a":"96k","-ar":"48000","-ac":2,"-f":"flv"}

# Define writer with defined parameters and change your STREAM_KEY below
writer = WriteGear(output_filename = 'rtmp://live.twitch.tv/app/STREAM_KEY', logging =True, **output_params)

# loop over
while True:

    # read frames from stream
    frame = stream.read()

    # check for frame if Nonetype
    if frame is None:
        break

    # {do something with the frame here}

    # write frame to writer
    writer.write(frame)

    # Show output window
    cv2.imshow("Output Frame", frame)

    # check for 'q' key if pressed
    key = cv2.waitKey(1) & 0xFF
    if key == ord("q"):
        break

# close output window
cv2.destroyAllWindows()

# safely close video stream
stream.stop()

# safely close writer
writer.close()

and revert results here. Goodluck!

@esologic Did you got the chance to test above example? Please let us know.

Successfully resolved and merged in commit d76c8be.

@esologic Closed and marked solved for now. Kindly share your results when you're ready. Good luck!

You must not copy the exact example, i meant to take ideas from it for building a RTMP stream pipeline. Btw you need to replace output_params with suitable FFmpeg parameters, in case if you wonder what's wrong, such as follows:
# define required FFmpeg parameters for your writer output_params = {"-profile:v":"main","-preset:v":"medium","-input_framerate":stream.framerate,"-r":stream.framerate,"-g":60,"-keyint_min":60,"-sc_threshold":0,"-b:v":"2500k","-maxrate":"2500k","-bufsize":"2500k","-f":"flv"}


snapshot during training

Hi, LP team
in DLC training, some snapshots would be saved every few iterations. and we can make use of certain snapshots for prediction.
Is there any function to do this in LP?
let's say I train the model for 500 epochs. and finally I found that 300 epochs would be the best. Then I would like to pick 300 epochs model for later use, instead of training the model from beginning to 300 epochs.
in the LP, the prediction py file has a line:
model = load_model_from_checkpoint(cfg=cfg, ckpt_file=ckpt_file, eval=True)

that means I can load the specific ckpt file, I think if we can save the ckpt file every few steps like 50/100/200, that would be better.

I updated the code to allow you to do this. To enable this feature, add the following to your config file:
training:
   ckpt_every_n_epoch: 50

This will save a checkpoint every 50 epochs; update to whatever value you'd like, but note that training.check_val_every_n_epoch should evenly divide whichever number you choose.
Please let me know if this works for you, and I'll close this issue.

Hi, @themattinthehatt
I tried it on LP 1.4.0 like:
  min_epochs: 30
  max_epochs: 30
  log_every_n_steps: 10
  check_val_every_n_epoch: 5
  gpu_id: 0
  rng_seed_data_pt: 0
  rng_seed_model_pt: 0
  lr_scheduler: multisteplr
  lr_scheduler_params:
    multisteplr:
      milestones:
      - 150
      - 200
      - 250
      gamma: 0.5
  ckpt_every_n_epochs: 10

but it did not work. only the final ckpt was saved. I listed all the ckpt files but only got one:
/root/autodl-tmp/DLC_LP/outputs/2024-06-18/12-43-09/tb_logs/DLC_LP/version_0/checkpoints/epoch=29-step=300.ckpt

any suggestion?

interesting, I checked this yesterday and it was working for me. How are you listing the checkpoints? Did you go into the actual checkpoints directory yourself to look? I'm not sure why that would've failed. I can test again on my side too.

Hi, @themattinthehatt
I run the code to list all the ckpt file:
import os

def find_ckpt_files(directory):
    ckpt_paths = []
    for root, _, files in os.walk(directory):
        for file in files:
            if file.endswith('.ckpt'):
                ckpt_paths.append(os.path.abspath(os.path.join(root, file)))
    return ckpt_paths

# Replace 'outputs' with your directory path
directory_path = 'outputs'
ckpt_paths = find_ckpt_files(directory_path)

# Print the absolute paths of all ckpt files
for path in ckpt_paths:
    print(path)

Hi, @themattinthehatt
I run the code to list all the ckpt file:
import os

def find_ckpt_files(directory):
    ckpt_paths = []
    for root, _, files in os.walk(directory):
        for file in files:
            if file.endswith('.ckpt'):
                ckpt_paths.append(os.path.abspath(os.path.join(root, file)))
    return ckpt_paths

# Replace 'outputs' with your directory path
directory_path = 'outputs'
ckpt_paths = find_ckpt_files(directory_path)

# Print the absolute paths of all ckpt files
for path in ckpt_paths:
    print(path)

@Wulin-Tan can you try again and watch the model directory during training? should be in <model_dir>/tb_logs/<model_name>/version_0/checkpoints. I was able to save the various checkpoints on my side using 1.4.0

Two questions:

can you send me the printout of your config file that is output to the terminal right before training starts? I want to make sure the checkpoint field is set properly in the config.
can you send me a screenshot of the function get_callbacks() in lightning-pose/lightning_pose/utils/scripts.py? Specifically I want to make sure you se the lines

if ckpt_model:
        if early_stopping:
            ckpt_callback = pl.callbacks.model_checkpoint.ModelCheckpoint(
                monitor="val_supervised_loss",
                mode="min",
            )
        else:
            # if ckpt_every_n_epochs is None, save after every validation step, but overwrite
            # if ckpt_every_n_epochs is not None, save separate checkpoint files
            ckpt_callback = pl.callbacks.model_checkpoint.ModelCheckpoint(
                monitor=None,
                every_n_epochs=ckpt_every_n_epochs,
                save_top_k=1 if ckpt_every_n_epochs is None else -1,
            )

hi, @themattinthehatt
here is the print out:
[2024-06-20 00:18:22,385][HYDRA] /root/miniconda3/lib/python3.8/site-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
  ret = run_job(

Our Hydra config file:
--------------------
data parameters
--------------------
image_orig_dims: {'height': 2160, 'width': 2160}
image_resize_dims: {'height': 256, 'width': 256}
data_dir: /root/autodl-tmp/DLC_LP
video_dir: /root/autodl-tmp/DLC_LP/videos
csv_file: CollectedData.csv
downsample_factor: 2
num_keypoints: 6
keypoint_names: ['snout', 'forepaw_L', 'forefaw_R', 'hindpaw_L', 'hindpaw_R', 'base']
mirrored_column_matches: None
columns_for_singleview_pca: [0, 1, 2, 3, 4, 5]

--------------------
training parameters
--------------------
imgaug: dlc
train_batch_size: 8
val_batch_size: 32
test_batch_size: 32
train_prob: 0.95
val_prob: 0.05
train_frames: 1
num_gpus: 1
num_workers: 16
early_stop_patience: 3
unfreezing_epoch: 20
min_epochs: 20
max_epochs: 20
log_every_n_steps: 10
check_val_every_n_epoch: 5
gpu_id: 0
rng_seed_data_pt: 0
rng_seed_model_pt: 0
lr_scheduler: multisteplr
lr_scheduler_params: {'multisteplr': {'milestones': [150, 200, 250], 'gamma': 0.5}}
ckpt_every_n_epochs: 5

--------------------
model parameters
--------------------
losses_to_use: ['pca_singleview', 'temporal']
backbone: resnet50_animal_ap10k
model_type: heatmap_mhcrnn
heatmap_loss_type: mse
model_name: DLC_LP

--------------------
dali parameters
--------------------
general: {'seed': 123456}
base: {'train': {'sequence_length': 32}, 'predict': {'sequence_length': 96}}
context: {'train': {'batch_size': 8}, 'predict': {'sequence_length': 48}}

--------------------
losses parameters
--------------------
pca_multiview: {'log_weight': 5.0, 'components_to_keep': 3, 'epsilon': None}
pca_singleview: {'log_weight': 5.0, 'components_to_keep': 0.99, 'epsilon': None}
temporal: {'log_weight': 5.0, 'epsilon': 20, 'prob_threshold': 0.05}

--------------------
callbacks parameters
--------------------
anneal_weight: {'attr_name': 'total_unsupervised_importance', 'init_val': 0.0, 'increase_factor': 0.01, 'final_val': 1.0, 'freeze_until_epoch': 0}

and the get_callbacks()
def get_callbacks(
    cfg: DictConfig,
    early_stopping=True,
    lr_monitor=True,
    ckpt_model=True,
    ckpt_every_n_epochs=None,
    backbone_unfreeze=True,
) -> List:

    callbacks = []

    if early_stopping:
        early_stopping = pl.callbacks.EarlyStopping(
            monitor="val_supervised_loss",
            patience=cfg.training.early_stop_patience,
            mode="min",
        )
        callbacks.append(early_stopping)
    if lr_monitor:
        lr_monitor = pl.callbacks.LearningRateMonitor(logging_interval="epoch")
        callbacks.append(lr_monitor)
    if ckpt_model:
        if early_stopping:
            ckpt_callback = pl.callbacks.model_checkpoint.ModelCheckpoint(
                monitor="val_supervised_loss",
                mode="min",
            )
        else:
            # if ckpt_every_n_epochs is None, save after every validation step, but overwrite
            # if ckpt_every_n_epochs is not None, save separate checkpoint files
            ckpt_callback = pl.callbacks.model_checkpoint.ModelCheckpoint(
                monitor=None,
                every_n_epochs=ckpt_every_n_epochs,
                save_top_k=1 if ckpt_every_n_epochs is None else -1,
            )
        callbacks.append(ckpt_callback)
    if backbone_unfreeze:
        transfer_unfreeze_callback = pl.callbacks.BackboneFinetuning(
            unfreeze_backbone_at_epoch=cfg.training.unfreezing_epoch,
            lambda_func=lambda epoch: 1.5,
            backbone_initial_ratio_lr=0.1,
            should_align=True,
            train_bn=True,
        )
        callbacks.append(transfer_unfreeze_callback)
    # we just need this callback for unsupervised models
    if (cfg.model.losses_to_use != []) and (cfg.model.losses_to_use is not None):
        anneal_weight_callback = AnnealWeight(**cfg.callbacks.anneal_weight)
        callbacks.append(anneal_weight_callback)

    return callbacks

can you share where get_callbacks() is called in train.py? should look like this (L73):
    callbacks = get_callbacks(
        cfg,
        early_stopping=False,
        ckpt_every_n_epochs=cfg.training.get("ckpt_every_n_epochs", None)
    )

hi, @themattinthehatt
my get_callbacks() is like:
    # early stopping, learning rate monitoring, model checkpointing, backbone unfreezing
    callbacks = get_callbacks(cfg, early_stopping=False)

and it does not include ckpt_every_n_epochs. So I follow your code:
    callbacks = get_callbacks(
        cfg,
        early_stopping=False,
        ckpt_every_n_epochs=cfg.training.get("ckpt_every_n_epochs", None)
    )

and it works!
I also tried your new version of train_hydra.py like:
"""Example model training script."""

import hydra
from omegaconf import DictConfig

from lightning_pose.train import train

@hydra.main(config_path="configs", config_name="config_mirror-mouse-example")
def train_model(cfg: DictConfig):
    """Main fitting function, accessed from command line.

    To train a model on the example dataset provided with the Lightning Pose package with this
    script, run the following command from inside the lightning-pose directory
    (make sure you have activated your conda environment):

    ```
    python scripts/train_hydra.py
    ```

    Note there are no arguments - this tells the script to default to the example data.

    To train a model on your own dataset, overwrite the default config_path and config_name args:

    ```
    python scripts/train_hydra.py --config-path=<PATH/TO/YOUR/CONFIGS/DIR> --config-name=<CONFIG_NAME.yaml>  # noqa
    ```

    For more information on training models, see the docs at
    https://lightning-pose.readthedocs.io/en/latest/source/user_guide/training.html

    """

    train(cfg)

if __name__ == "__main__":
    train_model()

It also works.
by the way, why do you set early_stopping false? I think it is useful and in the config file you also have early_stop_patience

@Wulin-Tan you should have those updates with the latest code - did you run git pull recently?
We have used early stopping in the past but actually the way it is implemented right now has some issues - the validation data (which is used for early stopping) uses the same image augmentation pipeline as the training data, which means that the validation data is actually different every time. So a low validation loss might mean that the model is doing better, or that batch just so happened to have some easier augmentations (by chance). This is on our list of things to fix, but for the time being we set the default to no early stopping. I should note that the early stopping still performs reasonably this way, but it's not perfect.

@themattinthehatt with the recent correction of the augmentation for the validation, is it safe to say that one should use early stopping by default? I am worried about over-trained models.
For example, a convergence at around 300 epochs with a "best" ckpt saved there and then a slight improvement at a single validation 300 epochs later. That is a bit worrisome, no?

You could turn on early stopping and it would definitely prevent that slight improvement 300 epochs later, but it's not clear to me if that model would be more overfit than the model checkpointed at epoch 300. Typically I set the max number of epochs to 300 to prevent the model from training too long anyways. You could try checkpointing a model at 300 and 600 epochs and see if their predictions are noticeably different.


Accelex, Glacien issue

When you run with Accelex and use Galcien, the game crashes
crash-2020-04-05_10.21.50-server.txt
crash-2020-04-04_12.38.24-server.txt

Fixed in the rewrite.

Fixed in the rewrite.


Remove border on hovered  elements in the docs

From the semantic awsm.css, hovered buttons get a one-pixel-wide black border when they're hovered. In the first example of the Usage section of the docs, this meant that hovering them would cause them to wiggle around slightly.
Given that this is a part of a separate library, not shareon itself, I figured that, while the wiggling around was almost certainly not intentional, neither was the black border. So, instead of keeping the border and removing the wiggling, this PR gets rid of both.
All this does is move the in-head <style> tag to underneath the awsm.css inclusion and add a copy of the same selector that caused the issue with a border: none.  Given there aren't any buttons elsewhere on the page, this should be fine. 😄

Hi Matthew! Thank you for this fix, an amazing job! :tada: I'll merge it right now. Ignore the CI fail — it doesn't have to do affect the library :)


Short solution needed: "Append data to array" (python-numpy)

Please help us write most modern and shortest code solution for this issue:
Append data to array (technology: python-numpy)
Fast way
Just write the code solution in the comments.
Prefered way

Create pull request with a new code file inside inbox folder.
Don't forget to use comments to make solution explained.
Link to this issue in comments of pull request.

Append data to array


Disabling first responder check

As application stops responding to any keycommand events after a focused input is rendered on the screen.

@parasharrajat @luacmartins needs to be reviewed and merged.


No 'do' method?

I've searched the documentation, I was trying to make the simplest rx stream just to test the library, it was supposed to be something like this:
observable.just(1).do(fmt.Print).subscribe()
Is there any documentation for the observable methods? I'm not that good in Go to be able to read the source

Hi @lucaszanella. Do is currently not implemented, but I'll look into it.
What you wanted to do can be achieved with

// wrap a side effect in an EventHandler
handler := handlers.NextFunc(func(v interface{}) {
        fmt.Println(v)
})
sc := observable.Just(1).Subscribe(handler)
<-sc

// or ...

// wrap the side effect in a MappableFunc and throw away the returned Observable
_ = observable.Just(1).Map(func(v interface{}) interface{} {
        fmt.Print(v)
        return nil
})

Go has no generic functions, so it is not possible for a method to accept any function outside of the typed ones declared in fx and handlers.

@jochasinga there's no possibility to pass a pointer to a generic function and execute it? By the way, thanks, I'm learning, Go is so different but I like it

@lucaszanella worth checking out this discussion on stackoverflow.
It might be interesting to have Do as a shortcut operator for a subscription in Connectable though, since it is a cold Observable that doesn't fire the stream on Subscribe:

printFunc := func(item interface{}) {
        fmt.Print(item)
}
c := connectable.Just(1).Subscribe(handlers.NextFunc(printFunc))
results := c.Connect()
<-results

// can be
results = connectable.Just(1).Do(printFunc).Connect()
<-results

Fixed -> Implement Do Operator


SelectDataset From with Error

Describe the bug
If you use SelectDataset From and pass it another SelectDataset that has an error already set on it, it seems to ignore the error on the subselect instead of propagating it.
To Reproduce
func TestSelectWithSubSelectError(t *testing.T) {

    ds := goqu.From("test").As("t").SetError(fmt.Errorf("An error"))
    assert.NotNil(t, ds.Error())

    ds2 := goqu.From(ds)
    assert.NotNil(t, ds2.Error())
}

If you run this test the first assertion passes, but the second assertion fails.
        	Error:      	Expected value not to be nil.
        	Test:       	TestSelectWithSubSelectError

Expected behavior
I would expect using a SelectDataset as a subselect on another SelectDataset would honor the error and propagate it rather than swallow the error. Obviously the caller can check this scenario with gs.Error() before using it in a subselect, but I think that puts too much burden on the caller IMO and this is something goqu should handle.
Dialect:

[X] postgres
[ ] mysql
[ ] sqlite3

Additional context
Add any other context about the problem here.

I've created a PR: https://github.com/doug-martin/goqu/pull/184 to address this issue in a way that made sense to me. @doug-martin please let me know what you think.

Thanks so much for getting this merged @doug-martin


[catalystproject] Increase size of nodepool disk to increase space in /tmp

For bioinformatics workloads on both the latam and africa catalyst hubs, i think we should provide more local disk space per node.  I'll suggest 400 Gb is a better number than the current 80Gb.
@jnywong and I can then work on guidance/tutorial for Catalyst communities that /tmp is the right place to store intermediate files which will be deleted automatically when the node shuts down.
The workflow I think we should recommend is
a) users should stage their 'input' datasets in object storage buckets
b) if workflows support reading directly from object storage then use that else make a local copy from object storage to /tmp
c) use /tmp for any intermediate files created during a workflow pipeline
d) push 'output' data sets to object storage for persistence
e) strongly encourage community users to keep home directory storage to under 1GB per user
f) discourage use of shared expect for smaller datasets (100Gb total per community)
Engineering Tasklist

[x] Increase disk size on GCP catalystproject-latam to 400 https://github.com/2i2c-org/infrastructure/pull/4215
[ ] Increase disk size on AWS catalystproject-africa to 400

I've done this for catalystproject-latam, but someone else would need to do it for catalystproject-af

The catalystproject-africa cluster has now been bumped too. There was a user on a node though, so I tainted it and brought up a new equivalent. I will wait for that to drain and delete it before closing this issue.