Meeting notes 5/31/2018

Attendees: @ajcvickers, @DamianEdwards, @divega, @sebastienros
We discussed today the next steps we can focus on:
Low hanging fruit

Platform versions for EF Core and Dapper: @sebastienros  to adapt @benaadams versions
Can we do better on Data Update, single and multiple query tests? @sebastienros can take a look. We wonder if @benaadams would be interested in these.
RavenDB : There is a PR, @sebastienros is tracking it

Longer term investments

Pipelining:

@divega to invite @roji and @bgrainger to next meeting to brainstorm.
We want to understand exactly what it is. For example, pipelining vs. multiplexing (out of order) vs. just batching queries from multiple public facing connections into the same physical connection?
Once we understand we can find out what we could do for each database.
Can have a mode in which we keep the same public API but with limitations (for example, no transactions)?
@sebastienros to check if we are already doing batching everywhere we can (we can ask for batching to be revisited if it is disallowed)

SqlClient:

@divega setting up something up to discuss future SqlClient development, performance issues, and the fact that TE wants to include SQL Server numbers.

Make our infra easy to access for the community: @sebastienros  to look into this.
Mongo (@divega to find contacts, @sebastienros to enable TE
EF Core:  We want to do more optimizations but no concrete plan ATM
EF6: Once on .NET Core it could be useful, if anything to compare with EF Core
ADO.NET improvements: lots of System.Data issues in Futures milestones
Features that cannot be used in benchmarks:

Coalescing: Probably a knob once we have pipelining

[TE] Add SQL Server (on linux) as a database type https://github.com/TechEmpower/FrameworkBenchmarks/pull/3807

Pipelining in MySQL
Background
The MySQL protocol is packet-oriented (details). (Note: multiple MySQL packets may be contained in a single TCP packet if they're small enough.) Each MySQL packet (up to 16MiB) has a one-byte sequence number. Client/server interactions are grouped into "conversations" where the first packet has sequence number 0 and each subsequent packet has the next sequence number (with wraparound). If a party in the conversation needs to send multiple packets, it numbers them i, i+1, ..., i+k and the other party replies (if necessary) with i+k+1.
Request/Response
Some conversations have multiple back-and-forth packets, e.g., log on:
<-- Initial handshake (0)
--> Handshake response (1)
<-- Authentication method switch (2)
--> Authentication data (3)
<-- OK (4)

Others (e.g., ping) are very simple, one packet from each side:
--> Ping (0)
<-- OK (1)

A query will result in multiple packets being sent back from the server; note that they're numbered consecutively:
--> "SELECT 1, 2;" Query (0)
<-- Column count (1)
<-- Column 1 (2)
<-- Column 2 (3)
<-- EOF (4)
<-- Row 1 (5)
<-- EOF (6)

Batching
I understand "batching" to mean putting multiple individual SQL statements into one COM_QUERY packet. The server will parse this into multiple SQL statements, then send back multiple result sets. This is easily accomplished in ADO.NET by setting DbCommand.CommandText to multiple semicolon-delimited SQL statements. This looks something like:
--> "SELECT 1; SELECT 2; SELECT 3;" Query (0)
<-- Column count (1)
<-- Column 1 (2)
<-- EOF (4)
<-- Row 1 (5)
<-- EOF + more results (6)
<-- Column count (7)
<-- Column 1 (8)
<-- EOF (9)
<-- Row 1 (10)
<-- EOF + more results (11)
<-- Column count (12)
<-- Column 1 (13)
<-- EOF (14)
<-- Row 1 (15)
<-- EOF (16)

This is done, for example, here: https://github.com/TechEmpower/FrameworkBenchmarks/blob/724a773096f403d149cfaf59b8257465bbf70103/frameworks/CSharp/aspnetcore/Benchmarks/Data/RawDb.cs#L130
Pipelining
By "pipelining" in MySQL, I mean starting multiple separate client conversations before the server response has been received. This is not an official part of the MySQL protocol and some servers (e.g., Amazon Aurora) will simply not respond to it. It looks something like:
--> Ping (0)
--> Reset Connection (0)
--> "SELECT 1;" Query (0)
<-- OK (1)
<-- OK (1)
<-- Column count (1)
<-- Column 1 (2)
<-- EOF (3)
<-- Row 1 (4)
<-- EOF (5)

It appears to work because MySQL Server reads the first packet with sequence ID 0, sends its response, then pulls the next packet off its receive buffer. It reads that as a new conversation with sequence ID 0, so it sends its response to that, and so on. One reason that it works is that very few conversations in the MySQL protocol have multiple replies from both sides; it's generally a client request followed by one or more server packets, then the conversation is implicitly over.
One core difference between batching and pipelining is that pipelining lets you send packets of different kinds; batching is just one outgoing packet with a larger payload.
The only place I've attempted to use it is in the library internals, e.g., sending one TCP packet that includes all the MySQL packets needed to send a "reset connection" request. (As hinted at earlier, this works great with MySQL Server, but not Amazon Aurora, so I reverted it and now all connection resets require multiple network roundtrips.)
This pipelining doesn't map easily on to the ADO.NET API. Conceivably one could let a client execute multiple ExecuteReaderAsync calls before consuming each reader, but because MySQL Server streams the result sets back in order, the client would have to read and dispose each reader in the order they were created. (Right now a client is simply forbidden from calling ExecuteXxx if there is still an open MySqlDataReader.)
Hope this helps; please let me know if anything's unclear.

By "pipelining" in MySQL, I mean starting multiple separate client conversations before the server response has been received.

As the authentication is the same, could connection pooling work to some degree down the same connection for multiplexing? (e.g. use less "physical connections" than the total "active connections" for fuller TCP packets and less chatter)
--> Ping (0)
--> Reset Connection (0)
--> "SELECT 1;" Query (0)
--> "SELECT 1;" Query (0)
--> "SELECT 1;" Query (0)
--> "SELECT 1;" Query (0)
<-- OK (1)
<-- OK (1)
<-- Column count (1)
<-- Column count (1)
<-- Column count (1)
<-- Column count (1)
<-- Column 1 (2)
<-- Column 1 (2)
<-- Column 1 (2)
<-- Column 1 (2)
<-- EOF (3)
<-- EOF (3)
<-- EOF (3)
<-- EOF (3)
<-- Row 1 (4)
<-- Row 1 (4)
<-- Row 1 (4)
<-- Row 1 (4)
<-- EOF (5)
<-- EOF (5)
<-- EOF (5)
<-- EOF (5)

The current model that is suggested by ADO.NET is to pool actual connections and associate them with a single ADO.NET connection. What other "fast" framework do is open a specific number of database connections, and not associate them with the ADO.NET connection itself. But when a command is sent, the ADO.NET connection will look for any currently open connection, or one by affinity, and then use it. It means that two requests, getting two ADO.NET connections, could be using the same database connection, and allowing pipelining. Even without using pipelining this could still improve perf as less database connections are active, and prepared statements are shared more efficiently.

What other "fast" framework do is open a specific number of database connections, and not associate them with the ADO.NET connection itself.

Base pooling around data stream rather than ADO connection; like?
-- Get from pool
--> "SELECT 1, 2;" Query (0)
<-- Column count (1)
<-- Column 1 (2)
<-- Column 2 (3)
<-- EOF (4)
<-- Row 1 (5)
<-- EOF (6)
-- Return to pool

Makes sense...

As the authentication is the same, could connection pooling work to some degree down the same connection for multiplexing?

I think this is theoretically possible, but haven't ever tested it. Presumably each MySqlDataReader would have to know its "order" in the queue (according to the sequence in which requests were sent) and if they were read "out of order" (e.g., on different threads) then it would pull responses off the socket and buffer them on the MySqlDataReader that they were for. Then you'd need to add locking around all I/O, or have a dedicated thread for each socket?

What other "fast" framework do is open a specific number of database connections, and not associate them with the ADO.NET connection itself. But when a command is sent, the ADO.NET connection will look for any currently open connection

The assumption of a dedicated connection is pretty common in client code: variables, temp tables, session settings, etc. are all associated with the current session, which is represented by a MySqlConnection. It would be a breaking change (and quite unexpected) to make a single MySqlConnection object use different physical connections behind the scenes; it could also break client code to multiplex multiple logical connections onto the same physical connection. The driver would have to detect if START TRANSACTION was ever executed, then put that connection into an "exclusive" mode that's no longer shared. It feels like there's a lot of edge cases here. (This could all be opt-in with an EnablePipelining=true connection string option, for clients who know they are using ADO.NET in a compatible way.)

It would be a breaking change (and quite unexpected) to make a single MySqlConnection object use different physical connections behind the scenes

I was suggesting the opposite, a single physical connection shared by multiple ado.net ones.

It still would cause problems if two non-pipeline-aware clients both execute CREATE TEMPORARY TABLE temp on different MySqlConnection objects that share the same physical server session.

A few comments from me...
First, the batching vs. pipelining distinction @bgrainger made above isn't very important IMHO. PostgreSQL does have two sub-protocols - simple and extended. The simple protocol has a single wire message (called Query), which may contain multiple SQL statements separated by semicolons, and which PostgreSQL parses on its end (this is what @bgrainger calls "batching"). Unfortunately the simple protocol is quite crippled - it does not support binary value transfers, parameters or prepared statements.
The extended protocol, on the other hand, requires a series of messages for each SQL statement. Executing an unprepared statements involves sending 5 protocol messages (Parse/Bind/Describe/Execute/Sync), while a prepared statement involves sending 3 (Bind/Execute/Sync). It is not permitted to include multiple SQL statements separated by semicolons, so Npgsql performs rudimentary SQL parsing client-side, splitting the CommandText on semicolons and sending the above set of messages for each parsed SQL.
I think that the difference between the above two types of batching is rather uninteresting - it's an internal protocol/implementation detail that doesn't affect much, and I tend to think of both of the above as "batching". Note that PostgreSQL messages do not have any sort of ID or running number, so we can send as many messages as we want up-front; PostgreSQL just reads and executes messages as they come in, and the client's responsibility is to keep track of ordering and parse the results accordingly.
I tend to think of pipelining as the ability of a single client (important!) to send a 2nd request before the 1st one has completed; in ADO.NET terms this would mean invoking DbCommand.Execute*() while another command is still running. This isn't currently supported, and AFAIK there's no TechEmpower test which potentially allows this to be exercised (but I may be mistaken).
Finally there's multiplexing, which to me describes the ability to combine queries from multiple clients into one or more physical connections - we're talking about something like a multiple-producer/multiple-consumer model.

As @bgrainger correctly pointed out, there's an ordering issue: since clients may attempt to consume query results in any order, it would require buffering entire resultsets in memory, killing the perf benefits of streaming (which I think can be quite important in many scenarios).
Transactions are obviously out of the question. In theory we could "degrade gracefully": the moment BeginTransaction() is invoked on a DbConnection, a physical connection is bound/affined to its producer in the same way that it is today. This could provide the best of both worlds.
As @bgrainger mentions above, physical connections are stateful objects (beyond transactions), so there can be other issues (temporary tables, various session parameters and settings...). FWIW, pgbouncer is a popular out-of-process connection pool which deals exactly with this kind of stuff: you can do session, transaction or statement pooling. I think that few usage scenarios out there actually make use of session state in this way, but who knows.
I think there's nothing preventing a multiplexing driver from implementing the ADO.NET interface (even if it wasn't the original intent). Nothing actually requires a DbConnection to correspond to a physical connection: Open() be a no-op, and the Execute*() methods would dispatch to one of the available physical connections (ideally the least loaded one). They would then await some TaskCompletionSource until the results are received etc.

And now for some perf analysis:

It's true that in a saturated scenario (all connections busy), multiplexing has its benefit. In the traditional implementation DbConnection.Open() blocks until a physical connection frees up. With multiplexing we can write our query to the server, at least until the TCP window is exhausted.
On the other hand, as long as we're not saturated (some physical connections are available), this may be a degradation. In the traditional model applications are responsible for keeping the connection open for as short a time as possible; as long as they don't hold open connections longer than they have to, multiplexing shouldn't have any advantage (unless I'm missing something) but only increased overhead (see below).
The plumbing for implementing multiplexing probably has some cost. The "synchronization point" is moved from DbConnection.Open() (now implemented in Npgsql by a lock-free pool) to DbCommand.Execute*(). For cases where multiple commands are being executed in series, that means we do synchronization more often when multiplexing. In other words, there may be an advantage to allocating your connection for multiple commands rather than allocating it again and again for each command.
I also suspect that the traditional model may simply have lower overhead, since Execute*() simply and directly awaits on the socket (rather than to a TaskCompletionSource behind plumbing etc.)

I understand that one (some?) of the top TechEmpower performers apparently multiplexes (have not actually looked into this), but do we have any justification for thinking that this is what makes it faster?

Unfortunately the simple protocol is quite crippled - it does not support binary value transfers, parameters or prepared statements.

Oh, that reminds me... MySQL also supports a binary protocol, which you have to use if you want to use prepared statements (which are not currently supported in MySqlConnector). Like PG, it doesn't support multiple statements, so MySqlConnector would have to parse the SQL and break them up. It has a three-phase Bind/Execute/Reset model.
Implementing prepared statements would provide an estimated 15% improvement for aspcore-*-my on the TFB benchmarks, so it's definitely a higher priority for me than hypothetical pipelining improvements.

Huh, interesting that the protocols are so similar in that way... I guess both started out from a simple text-based protocol and later added the advanced features as a new sub-protocol...
Implementing preparation definitely sounds like a high priority perf gain, you're welcome to help yourself to Npgsql's query parser if it can help you out.

It is not permitted to include multiple SQL statements separated by semicolons, so Npgsql performs rudimentary SQL parsing client-side, splitting the CommandText on semicolons and sending the above set of messages for each parsed SQL statement.

Interesting... as the final command in data updates is 20 update statements separated by semicolons. What would the best approach be to send this as a batch of commands? (From user api side)

In ADO.NET terms this would mean invoking DbCommand.Execute*() while another command is still running. This isn't currently supported, and AFAIK there's no TechEmpower test which potentially allows this to be exercised (but I may be mistaken).

Querys (20) would be such a test; also data updates (as that is queries + updates); and spawning them on multiple tasks. I tried it with Mono, the only safe way I saw at the time was to use multiple connections for the concurrency - however multiplying up the connection count made it very unhappy so I had to back it out https://github.com/TechEmpower/FrameworkBenchmarks/pull/3784/files

As things currently stand, you would pack them all into a single command's CommandText and execute. Npgsql splits them and send via separate protocol messages, which potentially fit into a single outgoing TCP packet, if the updates are small enough. https://github.com/dotnet/corefx/issues/3688 tracks adding a better API for batching which would possibly obviate client-side parsing.
In addition, preparation should be used, it has a significant impact for PostgreSQL.
I admit I haven't looked much at the update tests as the focus was on fortunes, I'll try to take ask a look and see where we are on the rest.

I'll take a look at Querys and the other db benchmarks this weekend to see what can be improved

dotnet/corefx#3688 tracks adding a better API for batching which would possibly obviate client-side parsing.

So that would be allowed by the updates part of data updates; however would be disallowed by the queries part as the queries need to be sent by the app as non-batched (though driver is allowed to then batch them)
However you could; validly do overlapping queries e.g.
var task1 =  cmd1.ExecuteReaderAsync();
var task2 =  cmd2.ExecuteReaderAsync();
var task3 =  cmd3.ExecuteReaderAsync();
var task4 =  cmd4.ExecuteReaderAsync();
var task = await Task.WhenAny(task1, task2, task3, task4);
using (var rdr = await task)
{
    // ....
}

Npgsql splits them and send via separate protocol messages, which potentially fit into a single outgoing TCP packet, if the updates are small enough.

Does it split the string and create more strings? Or just serialize in chunks to the wire?

So that would be allowed by the updates part of data updates; however would be disallowed by the queries part as the queries need to be sent by the app as non-batched (though driver is allowed to then batch them)
However you could; validly, do overlapping queries

If I understand you correctly, you're saying that this is disallowed:
var cmd = new NpgsqlCommand("SELECT 1; SELECT 2", conn);
var reader = cmd.ExecuteReader();

While the following is allowed:
var cmd1 = new NpgsqlCommand("SELECT 1", conn);
var cmd2 = new NpgsqlCommand("SELECT 2", conn);
var reader1 = cmd.ExecuteReader1();
var reader2 = cmd.ExecuteReader2();

If so, that seems a bit ridiculous - they're equivalent in almost any way, at least perf-wise (again, assuming a single producer, no multiplexing i.e. handling a single HTTP request)... The only user-facing difference would seem to be error handling (i.e. does the 2nd query get executed if the first fails). The first is already supported - Npgsql and PostgreSQL don't care if the batched statements are updates or selects - while the 2nd isn't. Can you point to the part in the TechEmpower docs which says this?

Npgsql splits them and send via separate protocol messages, which potentially fit into a single outgoing TCP packet, if the updates are small enough.

Does it split the string and create more strings? Or just serialize in chunks to the wire?

At the moment it splits, creating more strings (not ideal). Unfortunately there's another unrelated aspect that requires this: PostgreSQL parameter placeholders are positional ($1, $2), while historically ADO.NET (and therefore Npgsql) have allowed named placeholders (@p1, @p2). So in addition to splitting on semi-colons, the client-side SQL parsing phase also rewrites placeholders. https://github.com/npgsql/npgsql/issues/1042 is precisely about avoiding all this - if we allow applications to use native PostgreSQL positional placeholders, and in addition solve the batching API issue (https://github.com/dotnet/corefx/issues/3688), then Npgsql can stop parsing and rewriting the SQL entirely. There's also a chance that Span<char> can help reduce allocations now without such a big change.
There's one more aspect interacting with all this - prepared statements. If you call Prepare() on a command, all of the above parsing (and server preparation) happens at parse-time, and subsequent calls to Execute() avoid all of this. Unfortunately many applications (including the TE scenarios) have short-lived connection lifecycles, so you must instantiate a new command on each HTTP request. To still allow such applications to benefit from prepared statements, Npgsql persists prepared statements, which means that the server-side prepared statements doesn't get freed when the connection is returned to the pool. The next time NpgsqlCommand.Prepare() is called, we look up the each statement in an internal dictionary, and if we find that this particular SQL has already been prepared on this particular physical connection, we skip the preparation phase. This is a pretty major perf optimization (prepared statements are critical in PostgreSQL); however, it does require Prepare() to parse the CommandText to split on semicolons, so that each SQL can be looked up in the dictionary...
https://github.com/npgsql/npgsql/issues/1701 was about caching the raw SQL of the entire command (CommandText) rather than individual statements' SQLs, precisely to avoid having to parse CommandText when calling Prepare() on a command SQL that is already prepared (i.e. just do a single dictionary lookup on CommandText). This turned out to be quite complicated (two levels of caching - one for the commands, one for the statements) and the perf benefits simply didn't seem to justify it. And again, all this is basically a workaround for the fact that ADO.NET has no sane batching API and requires concatenating statements into CommandText (https://github.com/dotnet/corefx/issues/3688). So I ended up dropping it...
The above is quite a lot of Npgsql internals to take in :) Let me know if you want any more details, and looking forward to hearing any ideas.

Can you point to the part in the TechEmpower docs which says the 1st is disallowed while the 2nd is OK?

https://www.techempower.com/benchmarks/#section=code
“Test type 3: Multiple database queries”, point 6:

It is not acceptable to execute multiple SELECTs within a single complex query. It is not acceptable to retrieve all required rows using a SELECT ... WHERE id IN (...) clause. However, note General Requirement 7 which permits pipelining of network traffic between the application and database.

“General Requirements”, point 7:

… However, it is permissible to pipeline traffic between the application and database as a network-level optimization. That is, two or more separate queries may be delivered from the application to the database in a single transmission over the network, and likewise for query results from the database back to the application, as long as the queries themselves are executed separately on the database server and not combined into a single complex query.

@bgrainger hmm, thanks. However, I'm not sure that actually disallows SELECT 1; SELECT 2.
I think it can be claimed that SELECT 1; SELECT 2 isn't actually "multiple SELECTs within a single complex query": it's two different queries inside a single command (or batch); SELECT 1; SELECT 2 is nothing more than "piplining of network traffic between the application and database".
The restriction in point 6 seems to be more about using some SQL features to express the same thing in one query - see this comment. For example, it's forbidden to do select * from world where id in (2,3) instead of two selects. That restriction does make sense to me.
This seems supported by the following from the comment linked to above:

I don't know what Revenj does there.  If I enabled that commented-out code and I enabled postgres's query log, would the query log show a separate "SELECT * FROM world WHERE id = ?" statement for each query, or would it show a bulk query like "SELECT * FROM world WHERE id IN ...?"  If it's the latter, then no, that is still disallowed.

Sending a single NpgsqlCommand with SELECT 1; SELECT 2 will cause the PostgreSQL log to show two complete separate queries, which happened to be pipelined and combined into a single TCP packet.
So according to my interpretation this is allowed. I've posted a question to be absolutely sure.

 If they wanted separate DB queries (and network roundtrips), it would be nice if there were a clear data dependency in the tests, i.e., the second query depended on the results of executing the first query and thus couldn't be pipelined.

FWIW, I just looked at the source code of some of the fastest MySQL entrants (in the Multiple Queries and Data Updates categories); AFAICT each of them is executing a separate SELECT * FROM World WHERE id = ? command, repeatedly in a loop. These entrants all appear to be synchronous so it seems unlikely to me that an ORM layer (I did not look at how that is implemented for each framework) is combining them into a single network packet behind-the-scenes. The biggest differences I could see from aspcore-ado-my are a) synchronous, and b) prepared statements.

If I understand you correctly, you're saying that this is disallowed:
...
While the following is allowed:
...

Nah, wasn't the point I was making. The second example you gave is more or less the same as doing it in a loop (as they are sync queries)
So the first is disallowed because it is directly batching in application code, and that's only allowed as a network optimisation by the driver:

7 . Except where noted, all database queries should be delivered to the database servers as-is and not coalesced or deduplicated at the database driver. In all cases where a database query is required, it is expected that the query will reach and execute on the database server. However, it is permissible to pipeline traffic between the application and database as a network-level optimization. That is, two or more separate queries may be delivered from the application to the database in a single transmission over the network, and likewise for query results from the database back to the application, as long as the queries themselves are executed separately on the database server and not combined into a single complex query.

What I was suggesting is using async; stashing aside the Task, so it could make requests concurrently (i.e. not awaiting for the results of the first query until a few more queries had been individually submitted)

Even in async, if we compare this:
var cmd = new NpgsqlCommand("SELECT 1; SELECT 2", conn);
var reader = await cmd.ExecuteReaderAsync();

with this:
var cmd1 = new NpgsqlCommand("SELECT 1", conn);
var cmd2 = new NpgsqlCommand("SELECT 2", conn);
var task1 = cmd1.ExecuteReaderAsync();
var task2 = cmd2.ExecuteReaderAsync();
var task = await Task.WhenAny(task1, task2);

Once again, at the network level the above two fragments produce almost the exact same thing: a single roundtrip sending two queries, in almost identical wire protocol messages - there's no  advantage in the 2nd fragment compared to the first one. This is also why I can't really see any sane reason for the 1st fragment to be disallowed in a benchmark, while the 2nd fragment would be allowed. So I don't really see the sync/async aspect as being very important here.
The only advantage of the 2nd fragment, is if you don't have all the queries you want to send upfront (e.g. because they're gradually coming in via some HTTP connection or something). The 2nd method would allow you to send each command as it's coming in, whereas the 1st only allows you to batch several upfront, but then wait until that batch is complete.

One last comment, to reiterate on @bgrainger's earlier comment. If we look at this:
var task1 = cmd1.ExecuteReaderAsync();
var task2 = cmd2.ExecuteReaderAsync();
var task = await Task.WhenAny(task1, task2);

When does it make sense for task2 to complete? If it completes as soon as we're starting to receive the 2nd resultset (which is what the ADO.NET API suggests), then task2 can only complete after the 1st resultset has been consumed. In other words, the Task.WhenAny() above is totally useless, since we know task1 will always complete before task2.
There's nothing necessarily wrong with this, just pointing out that in the end there's a single TCP connection on which we're serially sending queries - there's no actual parallelism going on.

For the sync version
var cmd1 = new NpgsqlCommand("SELECT 1", conn);
var cmd2 = new NpgsqlCommand("SELECT 2", conn);
var reader1 = cmd.ExecuteReader1();
var reader2 = cmd.ExecuteReader2();

The statement var reader2 = cmd.ExecuteReader2(); cannot be run until the the preceding one is complete.
Whereas for the async version it can.

Once again, at the network level the above two fragments produce almost the exact same thing: a single roundtrip sending two queries, in almost identical wire protocol messages - there's no advantage in the 2nd fragment compared to the first one.

Which is the point. Batching the queries as one command in the app as SELECT 1; SELECT 2 allows any framework to do it; whereas requiring them as separate statements tests a frameworks ability to do concurrency; which not all do, and not all do well.
Rephrasing; the 20 query test as an http type test. If instead you had to download 20 images from a remote site - it becomes a test as to whether you can do it in parallel and how well you can do that as well as how your download works.
If the db driver then takes advantage of this an pipelines/batches them; that's great, but its an implementation detail outside of the code you are writing using the framework and the tests are about how code you can write with the framework performs (the db driver would count as part of framework rather than user code)
They are also looking to remove the database component to focus specifically on this concurrency:

Future: Tests that exercise requests made to external services and therefore must go idle until the external service provides a response. (High concurrency is desirable here.)

https://twitter.com/BrianHauerTSO/status/1000127582627545088

We are finding the variability in perfectly legitimate ways to communicate to databases (and variability of databases in general) make enforcing very strict uniformity of implementation unappealing. Perhaps better to do this with an intentionally rigid endpoint (e.g., jsonrpc).

There's nothing necessarily wrong with this, just pointing out that in the end there's a single TCP connection on which we're serially sending queries - there's no actual parallelism going on.

Whether its parallel, concurrent, pipelined or serial sends (request->response->request) is an implementation detail of how its achieved.

Hmm... reading the clarification: https://groups.google.com/forum/#!msg/framework-benchmarks/A78HrYsz4AQ/_zneyvo2CwAJ

For the kind of "pipelining" we do allow, it was my intent that
multiple queries do NOT get batched together in such a way that the
batching itself affects query semantics at all, especially with
respect to failed queries.  For example, what would happen if you
changed that Npgsql command to "SELECT 1; SELECT foo bar; SELECT 2",
where "foo bar" is nonsense that will cause that middle SELECT to
fail?  Will the "SELECT 2" query still be executed?  If not, then that
sort of batching violates requirements as I see them.  Why?  Because
that's exploiting simplifications in the test scenarios in a way that
goes against the spirit of the tests, where the same optimization
wouldn't necessarily be a good idea in the kind of real production
applications we're trying to simulate.  Meanwhile, as I understand the
pipelining that vertx-postgres is doing, there's no reason not to
enable that kind of pipelining.

ADO allows multiple active result sets, so you could do
var cmd = new NpgsqlCommand("SELECT 1; SELECT 2", conn);
var reader = await cmd.ExecuteReaderAsync();
reader.ReadAsync();
reader.NextResult(); // move to next query
reader.ReadAsync();

And that would be valid if the invalid query in the middle of SELECT 1; SELECT foo bar; SELECT 2 doesn't cause SELECT 2 not to run and return results.
Question, would it? How would that present via the api? (The invalid query)

@benaadams with the current behavior of Npgsql, if a statement in the middle of a batch fails then the rest of the statements are skipped by PostgreSQL. This is achieved via a feature of the PostgreSQL protocol, whereby the moment a protocol message generates an error, PostgreSQL will skip all subsequent messages until it sees a Sync message. So Npgsql will send SELECT 1; SELECT 2 as Parse1/Bind1/Describe1/Execute1/Parse2/Bind2/Describe2/Execute2/Sync (this is for unprepared queries, and all messages are potentially packed into a single TCP packet). If any of these messages generates an error, PostgreSQL will skip all the way to the sync.
In .NET API terms, you would be able to read resultsets and call reader.NextResult() until the problematic query, at which point calling reaader.NextResult() will generate an exception. But protocol sync will be maintained and the connection remains usable.
What I don't understand is why this failure semantics has anything to do with TechEmpower benchmark requirements - what possible interest is it of TechEmpower if Npgsql's batching logic skips remaining messages after error or not? In other words, should a database driver utilization be crippled in the benchmark just because it provides batching semantics which skip after error? What exactly would be the point of that?
Once again, I do understand the point of disallowing a single complex SQL statement such as SELECT x WHERE id IN (x, y, z) in place of three simple ones (SELECT x WHERE id=x...). But I don't understand the point of forbidden batching if it happens to have a certain skip-after-error behavior.
Out of curiosity, @benaadams, when executing two commands asynchronously with pipelining, would you expect the driver to retain the current streaming behavior, forcing querys' resultsets to be consumed in the same order that they were sent? Or would you expect resultsets to be buffered in memory, allowing the 2nd resultset to be consumed before the 1st? This question is irrelevant for multiplexing where buffering is clearly mandatory.
Finally, regardless of TechEmpower rules... Just to make sure I'm clear, I don't really have objections to pipelining or even multiplexing - they do have their perf benefits (although pipelining by a single client seems to be helpful only for a scenario which I find relatively rare - requests gradually coming in and being gradually pipelined on the same physical db connection). It definitely is a considerable departure from the current driver model, and I'm still not 100% sure I understand the perf impacts (i.e. what is the perf penalty of switching from the current streaming behavior to buffering?). But it's definitely an interesting direction.
At some point we may want to explore a design where a database driver has a low-level send/receive API (somewhat similar to libpq), on top of which we would be able to build either a classical ADO.NET model or a multiplexing layer.

What I don't understand is why this failure semantics has anything to do with TechEmpower benchmark requirements - what possible interest is it of TechEmpower if Npgsql's batching logic skips remaining messages after error or not?
Because the semantics of the test are a for a set of independent queries

That's exactly what I have a hard time accepting. So if Npgsql happened to implement its batching by sending Parse1/Bind1/Describe1/Execute1/Sync/Parse1/Bind1/Describe1/Execute1/Sync (note that additional Sync in the middle), then it would be eligible for using this optimizations in the benchmarks? How does that make sense, who benefits from this seemingly arbitrary exclusion? Would it make sense for Npgsql to have another batching mode without the skip-on-error logic so that batching can be used in TE? To be clear, I understand how what you're saying can be understood from the TE rules, but I'm questioning the sense of those rules.
The way I see it, forbidding the "complex query" case makes sense for preventing one benchmark implementation from things like using SQL IN for reducing multiple queries into one (especially since that may not be possible with some NoSQL databases). However, the fact that a batch skips-on-error is very different from that, and in my mind cannot be considered a single complex query in any way.
In any case, this discussion needs to happen with the TechEmpower folks rather than with you @benaadams (although it's interesting to hear your opinion). I'll post the arguments above in the Google discussion tomorrow.

I think we can leave the TechEmpower eligibility questions to the thread where Io posted, but it's still interesting to discuss batching/pipelining/multiplexing here...
I'm specifically interested in what people think about the impact of switching to buffering of resultsets as a result of multiplexing and possibly some forms of pipelining (since resultsets aren't consumed in order). I tend to think that streaming is very important when compared to buffering, but I know EF Core does buffer in some common cases. What are people's intuitions (or experience) about the degradation caused by buffering?

Most of my (micro) ORM experience is with Dapper and they buffer by default.


load data from url

It might be convenient to load data from a url instead of a file.
Something like:
def open_or_fetch(uri):
  if '://' in uri:
    req = requests.get(uri)
    req.raise_for_status()
    #print(req.content)
    fd = BytesIO(req.content)
    fd.name = uri
    return fd
  else:
    return open(uri, 'rb')
...
class FileReader:
    ...
    def read(self, filepath):
        ...
        if '://' in filepath or os.path.isfile(filepath):
            ...
            with open_or_fetch(filepath) as self.f_open:
                self.get_file_contents()

By monkey-patching I was able to get to load and plot using the following on colab::
from sasdata.dataloader import loader as loader_mod
from sasmodels.data import plot_data
from importlib import reload
reload(loader_mod) # need reload the module whenever FileReader changes.

root = 'https://raw.githubusercontent.com/SasView/sasview/main/src/sas/example_data/'
#uri = root + '1d_data/AOT_Microemulsion-Core_Contrast.xml'
#uri = root + '1d_data/FK403_0006_Nika.hdf'
#uri = root + '1d_data/ testdata_line1.txt'
uri = root + '1d_data/apoferritin.txt'
#uri = root + '1d_data/AUSANS_run3_2_no_buffer.ABS'
loader = loader_mod.Loader()
data = loader.load(uri)[0]
print(data)
data.mask = np.zeros_like(data.x, dtype='bool')
plot_data(data)

Here's the full patch cell I used to get it working:
#%pip install sasmodels sasdata matplotlib bumps periodictable
import os
from io import BytesIO
import requests
from sasdata.dataloader.filereader import FileReader, DEPRECATION_MESSAGE
from sasdata.data_util.loader_exceptions import NoKnownLoaderException, FileContentsException,\
    DataReaderException

def open_or_fetch(uri):
  if '://' in uri:
    req = requests.get(uri)
    req.raise_for_status()
    #print(req.content)
    fd = BytesIO(req.content)
    fd.name = uri
    return fd
  else:
    return open(uri, 'rb')

def read(self, filepath: str): #-> List[Union[Data1D, Data2D]]:
        """
        Basic file reader

        :param filepath: The full or relative path to a file to be loaded
        """
        self.filepath = filepath
        if '://' in filepath or os.path.isfile(filepath):
            basename, extension = os.path.splitext(os.path.basename(filepath))
            self.extension = extension.lower()
            # If the file type is not allowed, return nothing
            if self.extension in self.ext or self.allow_all:
                # Try to load the file, but raise an error if unable to.
                try:
                    print(f"++++++++++ trying {self.extension}")
                    with open_or_fetch(filepath) as self.f_open:
                        self.get_file_contents()
                except DataReaderException as e:
                    print(f"********** {self.extension} {str(e)}")
                    self.handle_error_message(str(e))
                except FileContentsException as e:
                    print(f"********** {self.extension} {str(e)}")
                    raise
                except OSError as e:
                    print(f"********** {self.extension} {str(e)}")
                    # If the file cannot be opened
                    msg = "Unable to open file: {}\n".format(filepath)
                    msg += str(e)
                    self.handle_error_message(msg)
                except Exception as e:
                    print(f"********** {self.extension} {str(e)}")
                    self.handle_error_message(str(e))
                finally:
                    print(f"Finally {self.extension}")
                    if any(filepath.lower().endswith(ext) for ext in
                           self.deprecated_extensions):
                        self.handle_error_message(DEPRECATION_MESSAGE)
                    if len(self.output) > 0:
                        # Sort the data that's been loaded
                        self.convert_data_units()
                        self.sort_data()
        else:
            msg = "Unable to find file at: {}\n".format(filepath)
            msg += "Please check your file path and try again."
            self.handle_error_message(msg)

        # Return a list of parsed entries that data_loader can manage
        final_data = self.output
        self.reset_state()
        return final_data

FileReader.read = read

HDF, xml and abs didn't work; not sure why.

Note: Should only fetch the datafile from the URI once, not once for each loader that it tries.

I like this suggestion and would be helpful for a number of reasons.
Notes:

The HDF reader has a separate file handler which is probably why it fails to load from URL since your implementation passes the as-opened file, not the raw file.
Note: Should only fetch the datafile from the URI once, not once for each loader that it tries. We could create two separate file handles one using open() and the other using h5py.File() and pass the open files to the readers.
I can't say why the XML and abs readers didn't work, but I would investigate that.

Potential implementation (untested):
sasdata/data/data_util/registry.py modifications:
# Outside the class
def open_or_fetch(uri):
  if '://' in uri:
    req = requests.get(uri)
    req.raise_for_status()
    #print(req.content)
    fd = BytesIO(req.content)
    fd.name = uri
    return fd, h5py.File(fd, 'r')
  else:
    freturn open(uri, 'rb'), h5py.File(uri, 'r')

# in ExtensionRegistry.load(), line 127 on
with open_or_fetch(path) as raw_file, h5_file:
    for load_function in loaders:
        try:
            return load_function(raw_file, h5_file)
        except Exception as e:
            last_exc = e
            pass  # give other loaders a chance to succeed
    # If we get here it is because all loaders failed
    raise last_exc
    else:
        msg = f"Unable to find file at: {path}\nPlease check your file path and try again."
        raise Exception(msg)

sasdata/data/dataloader/filereader.py modify lines 80 and on with:
def read(self, raw_file, hdf5_file) -> List[Union[Data1D, Data2D]]:
    """ documentation """
    self.filepath = raw_file.name
    self.f_open = raw_file
    self.hdf_open =hdf5_file
    basename, extension = os.path.splitext(os.path.basename(filepath))
    self.extension = extension.lower()
    if self.extension in self.ext or self.allow_all:
        try:
            self.get_file_contents()
...

sasdata/data/dataloader/readers/cansas_reader_HDF5.py get_file_contents() would simplify to:
try:
    # Read in all child elements of top level SASroot
    self.read_children(self.hdf_open, [])
    # Add the last data set to the list of outputs
    self.add_data_set()
except Exception as exc:
    raise FileContentsException(exc)
for data_set in self.output:
    if isinstance(data_set, Data1D):
        if data_set.x.size < 5:
            exception = FileContentsException("Fewer than 5 data points found.")
            data_set.errors.append(exception)


HoneyPot does not appear to be working

1.3.3
I have created a simple form. Added the config for honeypot, the field is displaying on front end in form (as expected till i hide it).
I submitted twice, once with it empty, once with content in the honey pot field.
Both went into DB and both arrived as emails?
I'll dig into it - but letting you know at this point. N

Okay, i believe the issue is the same as we have come across before for different PHP versions.
File: Formerly_SubmissionService.php
Line 69 was:
$honeyPotName =  craft()->config->get(Formerly_ConfigSettings::SettingsGroupName, Formerly_ConfigSettings::HoneyPotName);
Changed to:
$honeyPotName = craft()->config->get(Formerly_ConfigSettings::SettingsGroupName)[Formerly_ConfigSettings::HoneyPotName];
Honeypot now works for me...

I am confused, I thought we changed it to that because people were reporting the other way wasn't working in older versions of php. If someone can advise me on what code will work across all versions of php I'll push a fix up

I just had the same issue testing the new sendEmails and writeEmailBodyToFilePath settings. I've changed all settings code to use the format you specify

Yup, Sorry i know! Just throwing things up as i come across them on various different sites! It is a PHP version thing - solution, dunno tedious!
http://craftcms.stackexchange.com/questions/10213/getting-environmental-variable-via-plugin


@withalloc allocates but @withalloc1 does not

For the simple case of a single pre-allocation @withalloc is non-allocating. But for  more than one pre-allocation there is a very small allocation that I cannot track down and don't understand.
using WithAlloc, LinearAlgebra, Bumper, BenchmarkTools

mymul2!(A1, A2, B, C, D) = mul!(A1, B, C), mul!(A2, B, D)

function WithAlloc.whatalloc(::typeof(mymul2!), B, C, D)
   T1 = promote_type(eltype(B), eltype(C))
   T2 = promote_type(eltype(B), eltype(D))
   return ( (T1, size(B, 1), size(C, 2)),
            (T2, size(B, 1), size(D, 2)) )
end

function alloctest1(B, C, D)
   @no_escape begin
      a1, a2 = WithAlloc.whatalloc(mymul2!, B, C, D)
      A1 = @alloc(a1...)
      A2 = @alloc(a2...)
      mymul2!(A1, A2, B, C, D)
     # this change makes it again allocating:
     #    A12 = ( (@alloc(a1...)), (@alloc(a2...)) )
     #    mymul2!(A12..., B, C, D)
      sum(A1) + sum(A2)
   end
end

function alloctest2(B, C, D)
   @no_escape begin
      A1, A2 = @withalloc mymul2!(B, C, D)
      sum(A1) + sum(A2)
   end
end

B = randn(5,10); C = randn(10, 3); D = randn(10, 5)
@btime alloctest1($B, $C, $D)
@btime alloctest2($B, $C, $D)

fixed by commit e58c6171d11d9feb1ec1aae9fd938fd4d77e1cf2


rancher-compose does not discard build key if image key is set in local file (extends functionality)

@bymodude
docker-compose behaviour is like this, should rancher-compose not follow same behaviour?
https://docs.docker.com/compose/extends/

In the case of build and image, using one in the local service causes Compose to discard the other, if it was defined in the original service.

with the following files and using rancher-compose -f production.xml it is trying to upload local build context to s3, so its using build: . instead of image: test
docker-compose.yml:
test
  build: .

production.yml
test
  extends:
    file: docker-compose.yml
    service: test
  image: test

This is a bug in libcompose and will need to be fixed in libcompose.  https://github.com/docker/libcompose

Merged in libcompose. https://github.com/docker/libcompose/pull/66

Verified with v0.41.0-rc1 and rancher-compose v0.4.1-rc2.


Fix #19 - README.RST formatting errors

Still some missing content (and probably remaining formatting and English grammar errors), but at least it's readable.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

I signed it online.
--
Rich
+1.510.984.2571
On Tue, Nov 22, 2016 at 4:06 PM, CLAassistant<EMAIL_ADDRESS>wrote:

[image: CLA assistant check]
https://cla-assistant.io/aliyun/aliyun-cli?pullRequest=20
Thank you for your submission, we really appreciate it. Like many open
source projects, we ask that you sign our Contributor License Agreement
https://cla-assistant.io/aliyun/aliyun-cli?pullRequest=20 before we can
accept your contribution.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/aliyun/aliyun-cli/pull/20#issuecomment-262403358, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AACx_ERB9xbxTsp0lLf0S_fvxGeVQFpqks5rA4N_gaJpZM4K6F0x
.


PR pipelines are visible in the branches tab

Clear and concise description of the problem
If someone opens a PR to a branch (for example master), you can find the relevant pipeline under the target branch's section. For example:

This could cause some great confusion on which pipelines are run from commits verified by trusted people (people with push rights to the repo), and which are not. (I realize there's the PR icon on the pipeline, but most people won't notice it/won't know what it means.)
If something like #1577 merged, this could even lead to dangerous code being distributed to the target users.
Example of abuse:
Users are instructed to download the artifact from the latest pipeline on branch master.

Now users download the malicious binary when downloading the last artifact on branch master.
Suggested solution
Somehow separate PRs (not sure how exactly ATM).
Alternative
No response
Additional context
No response
Validations

[X] Checked that the feature isn't part of the next version already [https://woodpecker-ci.org/faq#which-version-of-woodpecker-should-i-use]
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

Probably would be good create separate PRs tab where to show list of all PR and be able to list specific PR builds

That was also my idea some time ago. To add an additional tab Pull Requests as well

I can confirm that this works correctly.
Thanks for the quick implementation!


Porting Bindings to Other Language?

Summary
I'm wondering if it would be feasible to take the JS bindings for a compiled wasm module and port them to another language if it has a library for running a wasm VM.
Additional Details
The swc project used wasm-bindgen to create an npm module.  The bindings are relatively straightforward.

let imports = {};
imports['__wbindgen_placeholder__'] = module.exports;
let wasm;
const { TextDecoder } = require(String.raw`util`);

let cachedTextDecoder = new TextDecoder('utf-8', { ignoreBOM: true, fatal: true });

cachedTextDecoder.decode();

let cachegetUint8Memory0 = null;
function getUint8Memory0() {
    if (cachegetUint8Memory0 === null || cachegetUint8Memory0.buffer !== wasm.memory.buffer) {
        cachegetUint8Memory0 = new Uint8Array(wasm.memory.buffer);
    }
    return cachegetUint8Memory0;
}

function getStringFromWasm0(ptr, len) {
    return cachedTextDecoder.decode(getUint8Memory0().subarray(ptr, ptr + len));
}

const heap = new Array(32).fill(undefined);

heap.push(undefined, null, true, false);

let heap_next = heap.length;

function addHeapObject(obj) {
    if (heap_next === heap.length) heap.push(heap.length + 1);
    const idx = heap_next;
    heap_next = heap[idx];

    heap[idx] = obj;
    return idx;
}

function getObject(idx) { return heap[idx]; }

let WASM_VECTOR_LEN = 0;

let cachegetNodeBufferMemory0 = null;
function getNodeBufferMemory0() {
    if (cachegetNodeBufferMemory0 === null || cachegetNodeBufferMemory0.buffer !== wasm.memory.buffer) {
        cachegetNodeBufferMemory0 = Buffer.from(wasm.memory.buffer);
    }
    return cachegetNodeBufferMemory0;
}

function passStringToWasm0(arg, malloc) {

    const len = Buffer.byteLength(arg);
    const ptr = malloc(len);
    getNodeBufferMemory0().write(arg, ptr, len);
    WASM_VECTOR_LEN = len;
    return ptr;
}

let cachegetInt32Memory0 = null;
function getInt32Memory0() {
    if (cachegetInt32Memory0 === null || cachegetInt32Memory0.buffer !== wasm.memory.buffer) {
        cachegetInt32Memory0 = new Int32Array(wasm.memory.buffer);
    }
    return cachegetInt32Memory0;
}

function dropObject(idx) {
    if (idx < 36) return;
    heap[idx] = heap_next;
    heap_next = idx;
}

function takeObject(idx) {
    const ret = getObject(idx);
    dropObject(idx);
    return ret;
}
/**
* @param {string} s
* @param {any} opts
* @returns {any}
*/
module.exports.parseSync = function(s, opts) {
    var ptr0 = passStringToWasm0(s, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc);
    var len0 = WASM_VECTOR_LEN;
    var ret = wasm.parseSync(ptr0, len0, addHeapObject(opts));
    return takeObject(ret);
};

/**
* @param {any} s
* @param {any} opts
* @returns {any}
*/
module.exports.printSync = function(s, opts) {
    var ret = wasm.printSync(addHeapObject(s), addHeapObject(opts));
    return takeObject(ret);
};

/**
* @param {string} s
* @param {any} opts
* @returns {any}
*/
module.exports.transformSync = function(s, opts) {
    var ptr0 = passStringToWasm0(s, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc);
    var len0 = WASM_VECTOR_LEN;
    var ret = wasm.transformSync(ptr0, len0, addHeapObject(opts));
    return takeObject(ret);
};

module.exports.__wbindgen_json_parse = function(arg0, arg1) {
    var ret = JSON.parse(getStringFromWasm0(arg0, arg1));
    return addHeapObject(ret);
};

module.exports.__wbindgen_json_serialize = function(arg0, arg1) {
    const obj = getObject(arg1);
    var ret = JSON.stringify(obj === undefined ? null : obj);
    var ptr0 = passStringToWasm0(ret, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc);
    var len0 = WASM_VECTOR_LEN;
    getInt32Memory0()[arg0 / 4 + 1] = len0;
    getInt32Memory0()[arg0 / 4 + 0] = ptr0;
};

module.exports.__wbindgen_string_new = function(arg0, arg1) {
    var ret = getStringFromWasm0(arg0, arg1);
    return addHeapObject(ret);
};

module.exports.__wbindgen_object_drop_ref = function(arg0) {
    takeObject(arg0);
};

module.exports.__wbg_new_59cb74e423758ede = function() {
    var ret = new Error();
    return addHeapObject(ret);
};

module.exports.__wbg_stack_558ba5917b466edd = function(arg0, arg1) {
    var ret = getObject(arg1).stack;
    var ptr0 = passStringToWasm0(ret, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc);
    var len0 = WASM_VECTOR_LEN;
    getInt32Memory0()[arg0 / 4 + 1] = len0;
    getInt32Memory0()[arg0 / 4 + 0] = ptr0;
};

module.exports.__wbg_error_4bb6c2a97407129a = function(arg0, arg1) {
    try {
        console.error(getStringFromWasm0(arg0, arg1));
    } finally {
        wasm.__wbindgen_free(arg0, arg1);
    }
};

module.exports.__wbindgen_rethrow = function(arg0) {
    throw takeObject(arg0);
};

const path = require('path').join(__dirname, 'wasm_bg.wasm');
const bytes = require('fs').readFileSync(path);

const wasmModule = new WebAssembly.Module(bytes);
const wasmInstance = new WebAssembly.Instance(wasmModule, imports);
wasm = wasmInstance.exports;
module.exports.__wasm = wasm;

I did see #1628 and #1634 and I understand that creating bindings for other languages is out of scope for this project.
My question is, on a basic level, is there anything preventing me from taking those specific JS bindings and porting them to a different language, to work with a wasm VM library?
It seems like the necessary steps are:

Implement the heap construct in native code.
Implement the helper methods natively and inject them into the VM so the wasm module can access them.
Create native wrappers that call into the VM to utilize the wasm module's exported functionality.

I gave this a shot but I can't seem to execute __wbindgen_malloc within the VM (it gets called but blows up the VM) so I'm wondering if there's some secret sauce that isn't obvious from the bindings, some JS-specific ingredient or detail about the wasm's compilation such that it won't work outside the browser.
I understand this may be a little out of the norm but any pointers are appreciated :)
Additional notes:

I tried this with both Life (which uses Wagon) and Gasm and was looking at WAKit next.
Yes, I did provide to the VM all methods that the wasm module wanted, with the correct signatures.
Once I understood what the bindings were doing and how they were getting data to and from the wasm module, I was quite impressed--very nice job!  :)

Thanks for the report! I unfortunately think, though, that wasm-bindgen literally-as-is is not a great choice for other languages. The idea of wasm-bindgen I think is a great one to take running for other languages, but the implementation as-is today is very JS-specific. Lots of runtime support in the wasm-bindgen crate as well as various options in the macros are all JS-specific and don't always have other clear meanings in other ecosystems
In essence I think wasm-bindgen is too ambitious about JS interop to be that successful in other languages. I think the general framework of what wasm-bindgen does is a great idea to try to mirror in other languages (and many Rust bindings to other languages already do this), however.

Hey @alexcrichton I appreciate you taking the time to swing by and drop a note!
It makes sense to me that wasm-bindgen won't output to other languages and how it's built very JS-specific.  I'm sure it would be a lot of work to refactor things to target a new language, and there's no incentive for anyone here to do that, and my Rust knowledge is far too small to attempt such a thing.
The thing that I really admire is the communication layer that this project creates.  Where normally you can only pass ints between the VM and the host, wasm-bindgen provides a way to communicate complex types between the two layers.  It was pretty mind-blowing when I grokked how that was happening.
Rather than trying to to change wasm-bindgen to target another language, I wanted to find out the feasibility of porting the generated bindings to another host environment besides Node or the browser.  Wasm-bindgen has injected certain vital functionality into the wasm itself, so in basic theory, all that's needed is to take that single JS bindings file and port it to another language/host environment.
I'm definitely aware that this would be one-off handwritten port, which gives it more of a science experiment status than anything else, but my curiosity has gotten the better of me.  There's a number of languages that have wasm runtimes, and by rewriting the JS bindings file to, say, Python, to work with py-wasm, you could then run that particular Rust program inside Python!
What I'm trying to find out is if this is entirely unfeasible or roadblocked in some way.  If I carefully rewrote a particular set of bindings to work with a different host environment, would it work?  You mention "runtime support" and "JS-specific macros" and I'm wondering if there's any magic going on besides the straight-up JS that's in the generated bindings file.  I can't think of anything myself, but I wanted to ask someone familiar with the wasm-bindgen project.
Thanks!

The language-neutral version of wasm-bindgen is "interface types", which is a proposal for WebAssembly. We're working in wasmtime to get this up and running and provide a way to compile down to interface types to get a wasm-bindgen-like-feeling in host runtimes too!

Hm, that is neat info, and something to look forward to, but you're not answering my actual question.  Maybe I haven't made myself clear, or maybe you're not reading my whole comment before responding, or maybe you don't know the answer.  The question was about feasibility of porting generated bindings to another host environment, not wasm-bindgen's capability for targeting other languages or host environments.
I appreciate that you took time out of your day to respond, and I acknowledge that the question is unusual, but it would have been nice to get an answer.

Sorry I don't mean to sound like I'm dodging the question. The question you're asking is pretty massive and could probably have at least an hour talk about simply it. I'm trying to be succinct and point you in the right direction.
My first point still stands, it will not work today in the limit. There are a lot of intrinsics that assume JS is the host language. Disregarding all that then you get to mostly the features defined by interface types. Wasm-bindgen currently does a bit more (like custom structs) but that's not going to be true in the long run.
The short answer is that it will be very difficult to port the JS file to Python, for example, and have it work there. That's not what wasm-bindgen was designed for.


After uninstalling 5.2 and installing 5.4 : can't start BCUninstaller. Error message screenshot attached

Can you copy and send me the value of "Stack Trace" from the second screenshot?


Make seed node list configurable

It'd useful to be able to configure seed node (list) to connect to, especially for people running node locally.

I use this approach to provide a default config, and will be override if some of them are provided by user: https://github.com/nknorg/nnet/blob/master/config/config.go
I agree that we shouldn't force ACK, that's why we should make it optional. But I think it's very useful to provide that option if user wants it to mimic the behavior of TCP to some degree. Actually the proto file contains the definition: https://github.com/nknorg/nkn-client-js/blob/master/lib/protocol/payloads.proto
I think it'd be good to use the same proto file for cross client communication.

Pushed configs for wallet/client.
Request-Response scheme WIP

I think this could be closed, because seed node list configuration was done long time ago
And as for req-resp it'll be implemented according to future NKP.


Help

please help how to decrypt url video twist via js node

Look at decrypt.py file and convert it to nodejs


syncthing: 0.14.0 -> 0.14.2

Motivation for this change

Update syncthing from 0.14.0 to 0.14.2
Use fetchFromGitHub instead of fetchgit
Add joko to package maintainers

Things done

[ ] Tested using sandboxing
(nix.useChroot on NixOS,
or option build-use-chroot in nix.conf
on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] OS X
[ ] Linux

[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

+1


python36Packages.django_2_1: 2.1.2 -> 2.1.3

Semi-automatic update generated by https://github.com/ryantm/nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/python3.6-django/versions.
meta.description for python36Packages.django_2_1 is: '"A high-level Python Web framework"'.

Checks done (click to expand)

built on NixOS
/nix/store/22ywx2wynshv2ywjl5mrdqm0xsk3hbz4-python3.6-Django-2.1.3/bin/django-admin.py passed the binary check.
/nix/store/22ywx2wynshv2ywjl5mrdqm0xsk3hbz4-python3.6-Django-2.1.3/bin/django-admin passed the binary check.
/nix/store/22ywx2wynshv2ywjl5mrdqm0xsk3hbz4-python3.6-Django-2.1.3/bin/..django-admin.py-wrapped-wrapped passed the binary check.
/nix/store/22ywx2wynshv2ywjl5mrdqm0xsk3hbz4-python3.6-Django-2.1.3/bin/.django-admin.py-wrapped passed the binary check.
/nix/store/22ywx2wynshv2ywjl5mrdqm0xsk3hbz4-python3.6-Django-2.1.3/bin/..django-admin-wrapped-wrapped passed the binary check.
/nix/store/22ywx2wynshv2ywjl5mrdqm0xsk3hbz4-python3.6-Django-2.1.3/bin/.django-admin-wrapped passed the binary check.
6 of 6 passed binary check by having a zero exit code.
6 of 6 passed binary check by having the new version present in output.
found 2.1.3 with grep in /nix/store/22ywx2wynshv2ywjl5mrdqm0xsk3hbz4-python3.6-Django-2.1.3
directory tree listing: https://gist.github.com/cec2fd9574c4cd1d6bf10bd8de669435
du listing: https://gist.github.com/67723ee21ce44aa2b3271c6e9043f0bd

Outpath report (click to expand)

Outpath difference report
4 total rebuild paths
1 package rebuilds
1 x86_64-linux rebuilds
1 i686-linux rebuilds
1 x86_64-darwin rebuilds
1 aarch64-linux rebuilds
First ten rebuilds by attrpath
python36Packages.django_2_1

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/22ywx2wynshv2ywjl5mrdqm0xsk3hbz4-python3.6-Django-2.1.3 \
  --option binary-caches 'https://cache.nixos.org/ https://r-ryantm.cachix.org/' \
  --option trusted-public-keys '
  r-ryantm.cachix.org-1:gkUbLkouDAyvBdpBX0JOdIiD2/DP1ldF3Z3Y6Gqcc4c=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(r-ryantm's Cachix cache is only trusted for this store-path realization.)
Or, build yourself:
nix-build -A python36Packages.django_2_1 https://github.com/r-ryantm/nixpkgs/archive/1330965fa588b479b104469d1c22796a263f37e8.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/22ywx2wynshv2ywjl5mrdqm0xsk3hbz4-python3.6-Django-2.1.3
ls -la /nix/store/22ywx2wynshv2ywjl5mrdqm0xsk3hbz4-python3.6-Django-2.1.3/bin

cc @georgewhewell for testing.

Success on x86_64-darwin (full log)
Attempted: python36Packages.django_2_1
Partial log (click to expand)
post-installation fixup
strip is /nix/store/g5r4apl0za012ffs6ladinwa5w0m1l3k-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/gc7xasgs6wkhvhpmvc7p36gz7ssg1fd1-python3.6-Django-2.1.3/lib  /nix/store/gc7xasgs6wkhvhpmvc7p36gz7ssg1fd1-python3.6-Django-2.1.3/bin
patching script interpreter paths in /nix/store/gc7xasgs6wkhvhpmvc7p36gz7ssg1fd1-python3.6-Django-2.1.3
/nix/store/gc7xasgs6wkhvhpmvc7p36gz7ssg1fd1-python3.6-Django-2.1.3/lib/python3.6/site-packages/django/bin/django-admin.py: interpreter directive changed from "/usr/bin/env python" to "/nix/store/0mha803yql9q7b6h93m8940z3sjbhiy8-python3-3.6.6/bin/python"
/nix/store/gc7xasgs6wkhvhpmvc7p36gz7ssg1fd1-python3.6-Django-2.1.3/lib/python3.6/site-packages/django/conf/project_template/manage.py-tpl: interpreter directive changed from "/usr/bin/env python" to "/nix/store/0mha803yql9q7b6h93m8940z3sjbhiy8-python3-3.6.6/bin/python"
wrapping `/nix/store/gc7xasgs6wkhvhpmvc7p36gz7ssg1fd1-python3.6-Django-2.1.3/bin/django-admin'...
wrapping `/nix/store/gc7xasgs6wkhvhpmvc7p36gz7ssg1fd1-python3.6-Django-2.1.3/bin/django-admin.py'...
wrapping `/nix/store/gc7xasgs6wkhvhpmvc7p36gz7ssg1fd1-python3.6-Django-2.1.3/bin/.django-admin-wrapped'...
wrapping `/nix/store/gc7xasgs6wkhvhpmvc7p36gz7ssg1fd1-python3.6-Django-2.1.3/bin/.django-admin.py-wrapped'...

Success on aarch64-linux (full log)
Attempted: python36Packages.django_2_1
Partial log (click to expand)
strip is /nix/store/p9akxn2sfy4wkhqdqa3li97pc6jaz3r1-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/w64z3nwdwyd069r1vn0wm74plg7wahps-python3.6-Django-2.1.3/lib  /nix/store/w64z3nwdwyd069r1vn0wm74plg7wahps-python3.6-Django-2.1.3/bin
patching script interpreter paths in /nix/store/w64z3nwdwyd069r1vn0wm74plg7wahps-python3.6-Django-2.1.3
/nix/store/w64z3nwdwyd069r1vn0wm74plg7wahps-python3.6-Django-2.1.3/lib/python3.6/site-packages/django/bin/django-admin.py: interpreter directive changed from "/usr/bin/env python" to "/nix/store/16wqq9ihrkdim3x8xs1lkxa3f4l3p3ir-python3-3.6.6/bin/python"
/nix/store/w64z3nwdwyd069r1vn0wm74plg7wahps-python3.6-Django-2.1.3/lib/python3.6/site-packages/django/conf/project_template/manage.py-tpl: interpreter directive changed from "/usr/bin/env python" to "/nix/store/16wqq9ihrkdim3x8xs1lkxa3f4l3p3ir-python3-3.6.6/bin/python"
checking for references to /build in /nix/store/w64z3nwdwyd069r1vn0wm74plg7wahps-python3.6-Django-2.1.3...
wrapping `/nix/store/w64z3nwdwyd069r1vn0wm74plg7wahps-python3.6-Django-2.1.3/bin/django-admin'...
wrapping `/nix/store/w64z3nwdwyd069r1vn0wm74plg7wahps-python3.6-Django-2.1.3/bin/django-admin.py'...
wrapping `/nix/store/w64z3nwdwyd069r1vn0wm74plg7wahps-python3.6-Django-2.1.3/bin/.django-admin.py-wrapped'...
wrapping `/nix/store/w64z3nwdwyd069r1vn0wm74plg7wahps-python3.6-Django-2.1.3/bin/.django-admin-wrapped'...

Success on x86_64-linux (full log)
Attempted: python36Packages.django_2_1
Partial log (click to expand)
stripping (with command strip and flags -S) in /nix/store/22ywx2wynshv2ywjl5mrdqm0xsk3hbz4-python3.6-Django-2.1.3/lib  /nix/store/22ywx2wynshv2ywjl5mrdqm0xsk3hbz4-python3.6-Django-2.1.3/bin
patching script interpreter paths in /nix/store/22ywx2wynshv2ywjl5mrdqm0xsk3hbz4-python3.6-Django-2.1.3
/nix/store/22ywx2wynshv2ywjl5mrdqm0xsk3hbz4-python3.6-Django-2.1.3/lib/python3.6/site-packages/django/bin/django-admin.py: interpreter directive changed from "/usr/bin/env python" to "/nix/store/bwfygfcdvis9wd1c1v51xwnwhw1hx0a0-python3-3.6.6/bin/python"
/nix/store/22ywx2wynshv2ywjl5mrdqm0xsk3hbz4-python3.6-Django-2.1.3/lib/python3.6/site-packages/django/conf/project_template/manage.py-tpl: interpreter directive changed from "/usr/bin/env python" to "/nix/store/bwfygfcdvis9wd1c1v51xwnwhw1hx0a0-python3-3.6.6/bin/python"
checking for references to /build in /nix/store/22ywx2wynshv2ywjl5mrdqm0xsk3hbz4-python3.6-Django-2.1.3...
wrapping `/nix/store/22ywx2wynshv2ywjl5mrdqm0xsk3hbz4-python3.6-Django-2.1.3/bin/django-admin'...
wrapping `/nix/store/22ywx2wynshv2ywjl5mrdqm0xsk3hbz4-python3.6-Django-2.1.3/bin/django-admin.py'...
wrapping `/nix/store/22ywx2wynshv2ywjl5mrdqm0xsk3hbz4-python3.6-Django-2.1.3/bin/.django-admin-wrapped'...
wrapping `/nix/store/22ywx2wynshv2ywjl5mrdqm0xsk3hbz4-python3.6-Django-2.1.3/bin/.django-admin.py-wrapped'...
/nix/store/22ywx2wynshv2ywjl5mrdqm0xsk3hbz4-python3.6-Django-2.1.3

The 50000 pull request!

Ok. Maybe not because ids are also given to issues.


Unable to record tracks on remote servers

Hi @murillo128
I have been using the Medooze Media Server package for my nodejs project.
It has been working fine when my recording initiator (client) and server are present on the same LAN. When I try to hit a remote server, acting as the media server, the Websocket connection is established with the exchange of offer and answer, an ICE candidate is also chosen for transmission, but on ending the recording the file size remains 1Kb, and no video or audio tracks are recorded.
I am using a Google Cloud VM instance to act as the remote server. The media server endpoint is the Google Cloud Instance's public IP, and the server has its own private IP.
All requests are sent to the public IP at port 8000, and forwarded to the private IP correctly, to create the answer and receive UDP packets, but the recording file dump is created without any video or audio content in it i.e. an empty file.
All paths from the Public IP to the Private IP are open in this scenario hence the Websocket Offer and Answer occurs successfully. I have exposed ports from 10000 to 55000 for the Media Server.
Yet the file dump created is empty.
The answer received by the client is shown below:
ANSWER:
v=0
o=-<PHONE_NUMBER>493 1 IN IP4 <IP_ADDRESS>
s=semantic-sdp
c=IN IP4 <IP_ADDRESS>
t=0 0
a=ice-lite
a=msid-semantic: WMS *
a=group:BUNDLE 0 1
m=audio 9 UDP/TLS/RTP/SAVPF 111
a=rtpmap:111 opus/48000/2
a=fmtp:111 minptime=10;useinbandfec=1
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:2 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=setup:passive
a=mid:0
a=sendrecv
a=ice-ufrag:cf01ba687a6fdfe0
a=ice-pwd:a8676c406ae16d1f53b1b9c0c8a01f5895405ce9e977d125
a=fingerprint:sha-256 A3:FB:6E:CE:E6:60:4A:EF:29:00:3E:FC:DD:A3:42:CC:82:82:14:8D:45:DD:B4:53:20:59:DE:A5:BD:E0:7A:4C
a=candidate:1 1 UDP<PHONE_NUMBER> <IP_ADDRESS> 50808 typ host
a=rtcp-mux
a=rtcp-rsize
m=video 9 UDP/TLS/RTP/SAVPF 98 99 100 101 102 122 125 107 124 120 123 119
a=rtpmap:98 VP9/90000
a=rtpmap:99 rtx/90000
a=rtpmap:100 VP9/90000
a=rtpmap:101 rtx/90000
a=rtpmap:102 H264/90000
a=rtpmap:122 rtx/90000
a=rtpmap:125 H264/90000
a=rtpmap:107 rtx/90000
a=rtpmap:124 H264/90000
a=rtpmap:120 rtx/90000
a=rtpmap:123 H264/90000
a=rtpmap:119 rtx/90000
a=fmtp:99 apt=98
a=fmtp:98 profile-id=0
a=fmtp:101 apt=100
a=fmtp:100 profile-id=2
a=fmtp:122 apt=102
a=fmtp:102 packetization-mode=1;level-asymmetry-allowed=1;profile-level-id=42001f
a=fmtp:107 apt=125
a=fmtp:125 packetization-mode=1;level-asymmetry-allowed=1;profile-level-id=42e01f
a=fmtp:120 apt=124
a=fmtp:124 packetization-mode=1;level-asymmetry-allowed=1;profile-level-id=4d0032
a=fmtp:119 apt=123
a=fmtp:123 packetization-mode=1;level-asymmetry-allowed=1;profile-level-id=640032
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=rtcp-fb:100 transport-cc
a=rtcp-fb:100 ccm fir
a=rtcp-fb:100 nack
a=rtcp-fb:100 nack pli
a=rtcp-fb:102 transport-cc
a=rtcp-fb:102 ccm fir
a=rtcp-fb:102 nack
a=rtcp-fb:102 nack pli
a=rtcp-fb:125 transport-cc
a=rtcp-fb:125 ccm fir
a=rtcp-fb:125 nack
a=rtcp-fb:125 nack pli
a=rtcp-fb:124 transport-cc
a=rtcp-fb:124 ccm fir
a=rtcp-fb:124 nack
a=rtcp-fb:124 nack pli
a=rtcp-fb:123 transport-cc
a=rtcp-fb:123 ccm fir
a=rtcp-fb:123 nack
a=rtcp-fb:123 nack pli
a=extmap:2 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=setup:passive
a=mid:1
a=sendrecv
a=ice-ufrag:cf01ba687a6fdfe0
a=ice-pwd:a8676c406ae16d1f53b1b9c0c8a01f5895405ce9e977d125
a=fingerprint:sha-256 A3:FB:6E:CE:E6:60:4A:EF:29:00:3E:FC:DD:A3:42:CC:82:82:14:8D:45:DD:B4:53:20:59:DE:A5:BD:E0:7A:4C
a=candidate:1 1 UDP<PHONE_NUMBER> <IP_ADDRESS> 50808 typ host
a=ssrc:279434614 cname:708efb21-d1eb-483f-813a-ac0cd62cc24b
a=ssrc:279434614 msid:708efb21-d1eb-483f-813a-ac0cd62cc24b video0
a=ssrc:362380999 cname:708efb21-d1eb-483f-813a-ac0cd62cc24b
a=ssrc:362380999 msid:708efb21-d1eb-483f-813a-ac0cd62cc24b video0
a=ssrc:961710332 cname:708efb21-d1eb-483f-813a-ac0cd62cc24b
a=ssrc:961710332 msid:708efb21-d1eb-483f-813a-ac0cd62cc24b video0
a=ssrc-group:FID 279434614 362380999
a=ssrc-group:FEC-FR 279434614 961710332
a=rtcp-mux
a=rtcp-rsize
Do you have any inputs on how to run the media server in an architecture where the server is hosted on a private IP masked by a public IP?
Are there any steps to take before we initiate recording in such an infrastructural setup?

You should just use the external ip address as ip parameter when creating the endpoint
https://github.com/medooze/media-server-node/blob/master/api.md#createendpoint

I am using the external IP address itself, to create the endpoint.
Even then it is not creating the file dump. The tracks are present in the peer connection object.
Everything is as demonstrated in your dummy project.

a=candidate:1 1 UDP<PHONE_NUMBER> <IP_ADDRESS> 50808 typ host

is this the correct ip address? check that with a network capture on both client and server if your browser is actually sending any data to that ip/port and if the firewalls are correctly opened so the server receives it.

Hi @murillo128,
Sorry for the delayed response.
I compared the logs of a locally hosted Medooze Server scenario where files were created and recording was dumped successfully, with the failed Google Instance scenario, as explained in the issue description.
I found that the DTLS handshake did not occur, due to which the UDP channels were not established for transmitting streams from my client.
The log screenshot of the Google Instance is shown below:

As you can see the DTLS handshake logs are not present, from which I can understand that the handshake has not occured.
The file that is created, is always of 444 bytes in size.
Is there any reason for WebRTC  to not perform the DTLS handshake automatically, as expected?
All required tracks are present in the Peer Connection object sent from the Client Side, and the endpoint is the external IP of the server. So necessary medooze server configurations are set as expected.

Can you enable the debug and ultra debug logs on the media server and send the new console log?
Also, can you check on chrome://webrtc-internals what are the ice events?

Hi @murillo128
I enabled the debug and ultra-debug as suggested.
This is the log:

The WebRTC Internals looks like this:

Please tell me what I am doing wrong.
The DTLS establishment is not happening successfully .

The ICE connection has failed, either provided ip address is not correct, or udp ports are not open on GCE firewall.
Take a network capture on client and server to compare traces. You would probably see that no ICE request are being received at server.

please reopen if you still have issues with this


Amazon Linux 2 requires pm-utils package to hibernate

When you install ec2-hibinit-agent to amzn2-ami-minimal, EC2 Hibernation doesn't work as expected.
Repro steps:

Create instance with the following configuration

Region: us-west-2
AMI: ami-09db9da37d31787f3 (amzn2-ami-minimal-hvm-2.0.20220606.1-x86_64-ebs)
Instance Type: t3.micro
EBS: 30GB gp2, Encrypted
Stop - Hibernate behavior: Enable

Install ec2-hibinit-agent

  $ sudo yum update -y
  $ sudo yum install ec2-hibinit-agent -y

Reboot

  $ sudo reboot

Hibernate the instance from Management console

Expected result:
The instance successfully hibernate.
We expect the following messages in /var/log/messages.
Jan  8 21:56:54 ip-172-31-1-206 kernel: PM: hibernation entry
...
Jan  8 21:56:54 ip-172-31-1-206 kernel: PM: hibernation exit

Actual result:
The instance keeps running.
You might see the following log in /var/log/messages.
Jan  8 21:44:22 ip-172-31-1-206 root: Hibernation failed, Sleeping 2 mins before retry

Note
Looking at /etc/acpi/actions/sleep.sh, it apparently depends on /usr/sbin/pm-hibernate included in pm-utils package.
( https://github.com/aws/amazon-ec2-hibinit-agent/blob/master/acpid.sleep.sh#L13 )
        swapon --priority=$swap_priority /swap && /usr/sbin/pm-hibernate

If you install pm-utils, hibernation works as expected.
$ sudo yum install pm-utils

Issue resolved in https://github.com/aws/amazon-ec2-hibinit-agent/commit/8ec340e71de2f4dbf38233fc295cf30cfd5dae32


[SPARK-15218][DEPLOY] replace any occurence of ':' in SPARK_HOME with…

What changes were proposed in this pull request?
Java could not find or load main org.apache.spark.launcher.Main when run from a directory containing colon ':'. Mesos used to use ':' in path generation.
This PR is proposed to create symlink with '_' instead of ':'.
How was this patch tested?
manual tests

Can one of the admins verify this patch?

cc @jerryshao @vanzin
please review

Can you please elaborate the problem you met, is it a Mesos specific problem?

@jerryshao it's seems that mesos creates directories with ':' (as date delimiter) which is unusual and put spark binaries into it.
I found this issue in issue list and like to resolve it.
https://issues.apache.org/jira/browse/SPARK-15218

Can you please print out the full command which has problem, you could simply set export SPARK_PRINT_LAUNCH_COMMAND=true in spark-env.sh.

Full launch command (though it's a toy example)
/Library/Java/JavaVirtualMachines/jdk1.8.0_31.jdk/Contents/Home/bin/java -cp /Users/slava/work/sandbox/test:colon/spark/assembly/target/scala-2.11/jars/* org.apache.spark.launcher.Main
folder named test:colon cause issue

Is there any way to tell Mesos to not do that? This is pretty horrible.

ok to test

can you at least add a comment in the code that links to the relevant Mesos issue?

Actually, the Mesos bug was fixed in 0.18.0 but we're on 0.21.0 already, so shouldn't the fix already be in?
@tnachen @mgummelt can you confirm?

Test build #58253 has started for PR 13026 at commit aecfa0a.

@sayevsky I'm not sure it's Mesos that's using colon as a seperator, what is in your stdout when the driver runs? I'm guessing perhaps that the command that we generate in the scheduler isn't able to join the classpath correctly with the colon in the folder name?

And what version of Mesos are you using? Also are you using docker?

@andrewor14 it is fixed in 0.18. But there are still people who use old mesos so that this workaround is proposed.

Test build #58253 has finished for PR 13026 at commit aecfa0a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58253/
Test FAILed.

I'm gonna propose we close this because (1) this is an old Mesos version being terribly broken and (2) spark-class is not expected to be able to write to $SPARK_HOME, much less its parent, so the fix in sketchy even if Mesos hadn't been fixed already.


TLS and cipher issues

encountered this issue, what to do?
### Sun Jul 16 20:32:47 2023 WARNING: Compression for receiving enabled. Compression has been used in the past to break encryption. Sent packets are not compressed unless "allow-compression yes" is also set.
Sun Jul 16 20:32:47 2023 DEPRECATED OPTION: --cipher set to 'AES-256-CBC' but missing in --data-ciphers (AES-128-CBC). OpenVPN ignores --cipher for cipher negotiations.
Sun Jul 16 20:32:47 2023 Note: '--allow-compression' is not set to 'no', disabling data channel offload.
Sun Jul 16 20:32:47 2023 OpenVPN 2.6.5 [git:v2.6.5/cbc9e0ce412e7b42] Windows-MSVC [SSL (OpenSSL)] [LZO] [LZ4] [PKCS11] [AEAD] [DCO] built on Jun 13 2023
Sun Jul 16 20:32:47 2023 Windows version 10.0 (Windows 10 or greater), amd64 executable
Sun Jul 16 20:32:47 2023 library versions: OpenSSL 3.1.1 30 May 2023, LZO 2.10
Sun Jul 16 20:32:47 2023 DCO version: v0
Sun Jul 16 20:32:47 2023 MANAGEMENT: TCP Socket listening on [AF_INET]<IP_ADDRESS>:25343
Sun Jul 16 20:32:47 2023 Need hold release from management interface, waiting...
Sun Jul 16 20:32:48 2023 MANAGEMENT: Client connected from [AF_INET]<IP_ADDRESS>:52359
Sun Jul 16 20:32:48 2023 MANAGEMENT: CMD 'state on'
Sun Jul 16 20:32:48 2023 MANAGEMENT: CMD 'log on all'
Sun Jul 16 20:32:48 2023 MANAGEMENT: CMD 'echo on all'
Sun Jul 16 20:32:48 2023 MANAGEMENT: CMD 'bytecount 5'
Sun Jul 16 20:32:48 2023 MANAGEMENT: CMD 'state'
Sun Jul 16 20:32:48 2023 MANAGEMENT: CMD 'hold off'
Sun Jul 16 20:32:48 2023 MANAGEMENT: CMD 'hold release'
Sun Jul 16 20:32:48 2023 TCP/UDP: Preserving recently used remote address: [AF_INET]<IP_ADDRESS>:1194
Sun Jul 16 20:32:48 2023 Socket Buffers: R=[65536->65536] S=[65536->65536]
Sun Jul 16 20:32:48 2023 UDPv4 link local: (not bound)
Sun Jul 16 20:32:48 2023 UDPv4 link remote: [AF_INET]<IP_ADDRESS>:1194
Sun Jul 16 20:32:48 2023 MANAGEMENT: >STATE:1689546768,WAIT,,,,,,
Sun Jul 16 20:32:48 2023 MANAGEMENT: >STATE:1689546768,AUTH,,,,,,
Sun Jul 16 20:32:48 2023 TLS: Initial packet from [AF_INET]<IP_ADDRESS>:1194, sid=7382e86c 1dec440f
Sun Jul 16 20:32:48 2023 VERIFY OK: depth=1, CN=ChangeMe
Sun Jul 16 20:32:48 2023 VERIFY KU OK
Sun Jul 16 20:32:48 2023 Validating certificate extended key usage
Sun Jul 16 20:32:48 2023 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Sun Jul 16 20:32:48 2023 VERIFY EKU OK
Sun Jul 16 20:32:48 2023 VERIFY OK: depth=0, CN=server
Sun Jul 16 20:33:48 2023 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Sun Jul 16 20:33:48 2023 TLS Error: TLS handshake failed
Sun Jul 16 20:33:48 2023 SIGUSR1[soft,tls-error] received, process restarting
Sun Jul 16 20:33:48 2023 MANAGEMENT: >STATE:1689546828,RECONNECTING,tls-error,,,,,
Sun Jul 16 20:33:48 2023 Restart pause, 1 second(s)
Sun Jul 16 20:33:49 2023 TCP/UDP: Preserving recently used remote address: [AF_INET]<IP_ADDRESS>:1194
Sun Jul 16 20:33:49 2023 Socket Buffers: R=[65536->65536] S=[65536->65536]
Sun Jul 16 20:33:49 2023 UDPv4 link local: (not bound)
Sun Jul 16 20:33:49 2023 UDPv4 link remote: [AF_INET]<IP_ADDRESS>:1194
Sun Jul 16 20:33:49 2023 MANAGEMENT: >STATE:1689546829,WAIT,,,,,,
Sun Jul 16 20:33:49 2023 MANAGEMENT: >STATE:1689546829,AUTH,,,,,,
Sun Jul 16 20:33:49 2023 TLS: Initial packet from [AF_INET]<IP_ADDRESS>:1194, sid=19d04519 56f98c1b
Sun Jul 16 20:33:49 2023 VERIFY OK: depth=1, CN=ChangeMe
Sun Jul 16 20:33:49 2023 VERIFY KU OK
Sun Jul 16 20:33:49 2023 Validating certificate extended key usage
Sun Jul 16 20:33:49 2023 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Sun Jul 16 20:33:49 2023 VERIFY EKU OK
Sun Jul 16 20:33:49 2023 VERIFY OK: depth=0, CN=server

replace cipher AES-256-CBC to data-ciphers AES-128-GCM or other OpenVPN Recommended  cipher:

AES-256-GCM
AES-128-GCM
CHACHA20-POLY1305

sudo sed -i 's|cipher AES-256-CBC|data-ciphers AES-128-GCM|' /etc/openvpn/server/client-common.txt
sudo sed -i 's|cipher AES-256-CBC|data-ciphers AES-128-GCM|' /etc/openvpn/server/client-common.txt
sudo sed -i 's|cipher AES-256-CBC|data-ciphers AES-128-GCM|' <youclient.ovpn>

replace cipher AES-256-CBC to data-ciphers AES-128-GCM or other OpenVPN Recommended cipher:
* AES-256-GCM

* AES-128-GCM

* CHACHA20-POLY1305

sudo sed -i 's|cipher AES-256-CBC|data-ciphers AES-128-GCM|' /etc/openvpn/server/server.conf
sudo sed -i 's|cipher AES-256-CBC|data-ciphers AES-128-GCM|' /etc/openvpn/server/client-common.txt
sudo sed -i 's|cipher AES-256-CBC|data-ciphers AES-128-GCM|' <youclient.ovpn>
sudo systemctl restart openvpn

This works for me. Thank you for sharing the solution.

This is a benign warning about a parameter which is there for legacy reasons. You can ignore the warning and there is no need to take any action.
The data-ciphers parameter is not required and there is absolutely no need to set it.


CSVMetadata parameters required for operator publication

ClusterServiceVersion requires additional parameters for publication. I have added them to CSVMetadata

I've made some modifications to your latest changes but I cannot update your PR: can you make it so that I can update the code?
My changes are basically removing the base64data field on Icon because I don't think that users are likely to use that, not to mention the fact that we probably want to avoid adding that data in the app if we can avoid it. To that effect, I also changed the location of where the icons should be put in order for the generator to pick them up: it will now look for icons using the name defined in Icon in src/main/kubernetes instead of simply checking for a default icon in src/main/resources.

Not sure what's going on but I still can't push changes to your branch… 😭
Will open a different PR with the additional changes and will close this one if you're OK with this.

See #432.

Yes. I'm ok. Thanks


⚠ Bofrost.be has degraded performance

In 3f3bfb7, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1979 ms

Resolved: Bofrost.be performance has improved in de5d3bf.


Delete Namespace sometimes failed

What happened:
Failed to delete the namespace generated according to the prefix
What you expected to happen:
Always can delete namespace
How to reproduce it (as minimally and precisely as possible):
namespace:   number: {{$namespacesNumber}}   prefix: test   deleteAutomanagedNamespaces: true   enableExistingNamespaces: true
Anything else we need to know?:
Environment: test

Kubernetes version (use kubectl version): 1.20.15
Cloud provider or hardware configuration:
OS (e.g: cat /etc/os-release):
Kernel (e.g. uname -a):
Install tools:
Network plugin and version (if this is a network-related bug):
Others:

This may be relevant. https://github.com/kubernetes/perf-tests/pull/1874
(try listing all the events in one of the failed namespaces, to see if there is lots of them)

This may be relevant. #1874
(try listing all the events in one of the failed namespaces, to see if there is lots of them)
Thx , It is not relevant.
I debugged, I found that sometimes delete the same namespace


[webview_flutter] Camera is not working

I am using webview_flutter's 4.2.0. Recently, the plugin started supporting camera permission. I am giving the camera permission in Android when asked but it does not open the camera. (Use case - I have a button that should open the camera).
iOS:

Click on a button, which opens the camera but on a view that asks for a selfie, does not open the camera. We see a black screen.
Android
Click on a button, which does the camera but on a view that asks for a selfie, does open the camera.

Here the behavior is quite the opposite in both iOS and android. What works in android does not work in iOS and vice versa.

Note: the same thing works fine with the in_appwebview_flutter plugin.

Flutter version - 3.3.9
webview_flutter - 4.2.0
Sample project - https://github.com/pushpendraKh/webviewdemo

Hi @dam-ease
Thanks for looking into it. I am able to reproduce the problem with the sample project itself.
unfortunately, It's not possible to directly load the URL where the camera does not work. I have updated the readme to reach the problem. Please have a look https://github.com/pushpendraKh/webviewdemo#readme

Thanks for responding. I am still unable to reproduce the issue as shown in the video below. Can you update to the latest Flutter version and the latest version of the package and attach a video showing the issue?
Video

https://github.com/flutter/flutter/assets/53122008/cf142069-15b1-449f-b76c-c8291fd32659


Included ng-http-cache module.

@davguij there are some conflicts that need to be resolved. Please pull latest and make a new PR with your changes.


Bypassing Captcha

Can this wrapper bypass captcha to get the contact information? By anyway is this possible to completely bypass captcha and get the contact info. I want to create a scrapper

Sorry I was wrong, the captcha is not shown while revealing contacts


[NPM] File import not found or unreadable: bootstrap.

When I try to import bootstrap into my sass file using @import "bootstrap" . I get an saying that the import is not found or unreadable. Any help?

I meet same problem

but when I import like this, it works:
@import "bootstrap-sass/assets/stylesheets/_bootstrap";

same here

same


What about multicolored svgs – should we support it?

Currently all colors are transformed to one color. Maybe this is done by svg-cleanup?

Currently all colors are transformed to one color. Maybe this is done by svg-cleanup?

If I'm not mistaken, fonts could still only include a single color definition; so the functional icons are meant to be single colored, whereas the illustrative aren't being transformed to icon fonts, as they are multi-colored.

As @mfranzke said, this won't be possible

As @mfranzke said, this won't be possible

to my knowledge it's only possible of you overlay the icon parts by having one of them within ::before and the other in ::after content-properties. But that's a little bit more complicated and we don't have the need at the moment.


Fix : Updated seed module to use "mscolab_settings.ARCHIVE_THRESHOLD"

Purpose of PR?
Fixes #2227
Updated the seed module to always use mscolab_settings.ARCHIVE_THRESHOLD instead of hardcoded dateutil.relativedelta.relativedelta(months=**2**).

@workaryangupta please look on this in develop. My failure to think we can fix this in stable.


set dollar-variable-colon-space-after to always-single-line

'scss/dollar-variable-colon-space-after': 'always' collides with the default formatting of Prettier when the value is broken into several lines.
For example, Prettier formats like this:
$ratios:
    1 1,
    2 1,
    5 4,
    4 3,
    3 2,
    16 9,
    4 5,
    3 4,
    2 3,
    9 16;

While the rule set to always formats the same like this:
$ratios: 1 1,
    2 1,
    5 4,
    4 3,
    3 2,
    16 9,
    4 5,
    3 4,
    2 3,
    9 16;

I propose to default to a setting that is compatible with Prettier.

Thanks @woylie !
Could you also add a test case for this changed to the valid.scss file?

Hey @kristerkari,
I added the example. I couldn't get the tests to run locally, so I hope this suffices.

fixes: #165

Sorry that I missed this one! Looks good now with the test added. 👍


[Request] libopenraw

Link to the package upstream website/repository
https://libopenraw.freedesktop.org/
https://gitlab.freedesktop.org/libopenraw/libopenraw
Utility this package has for you
It's needed to build tauri v2 appimages
Do you consider the package(s) to be useful for every Chaotic-Nyx user?
No, but for a few.
Do you consider the package to be useful for feature testing/preview?

[ ] Yes

Are you sure the package isn't available upstream already?

[x] Yes

Does the package's license allow redistributing it?
YES!
Have you searched the issues to ensure this request is unique?

[X] YES!

Have you read the README to ensure this package is not banned?

[X] YES!

More information
No response

I saw your PR at nixpkgs, and it looks almost ready to merge there.
Mind if instead I add a libopenraw_git here that updates daily from the latest commit in the master branch?

Sure, that's fine!

UPDATE: Current stuck due to: https://github.com/chaotic-cx/nyx/issues/403


Lack of timeout mechanism for service exit process.

In the server/server.go:201, some of the methods provided by user are executed but no timeout control applied.
This may cause the program to fail to exit properly include Deregister.
If refinement is required, what style should be used for this?
I would like to pr if possible.

Would you please provide a specific example to demonstrate the issue?

Would you please provide a specific example to demonstrate the issue?

func easyDemoConsumeSyncLogRegister(logChan <-chan logMessage){
	// user do something like emptying the log queue before 's.Stop()'
	// but do not any timeout mechanism or active withdrawal
	f := func(){
		for m := range logChan{
			Log(m)
		}
	}
	server.RegisterShutdownHook(f)
}

If you just want to do some clean-up after the main loop exit, you can simply place that code in the main.go after svr.Run(). And, of course, a defer statement is OK, too.

If you just want to do some clean-up after the main loop exit, you can simply place that code in the main.go after svr.Run(). And, of course, a defer statement is OK, too.
Thank for your replay.
Can you recommend any application scenarios for Hooks?

If you just want to do some clean-up after the main loop exit, you can simply place that code in the main.go after svr.Run(). And, of course, a defer statement is OK, too.
Thank for your replay.
Can you recommend any application scenarios for Hooks?

Hi, @dongzeXD.  The different scenarios between Hooks and svr.Run

ShutdownHook: the func you registered will be executed after receiving the exit signal but before deregister and the server stops.
Code after svr.Run(): will be executed after the server stops.

So if you want your logic to be executed before the server exit actually, use ShutdownHook. Or if you just want to do something after the server exit, you can place the code after svr.Run().

I am not actually sure about your need. Do you mean the timeout mechanism cannot be effective for ShutdownHooks? Yes, it is now, you can set ExitWaitTime but it only is effective when do underlying layer exit. So if your hook executes long time it may lead the process cannot be exited, for this case, you can choose to place code after svr.Run() or add timeout in your hook.


Why is GPU RAM not increasing when increasing batch size?

Hi,
I just came from axolotl and I'm impressed! I get 10x faster phi3-mini-4k and I can run the adapter in VLLM (couldn't with axolotl, vLLM says that lm_head module is not supported).
Question 1: Why doesn't increasing the batch size increase the GPU RAM? I was having batch size 50 and then 100 and still the same. On A100 40GB it started at 15GB and then it dropped to 13GB. Changing lora params does have an effect but it looks like batch size doesn't have any effect on the speed?
Question 2: since I have max tokens in my training data at 768, would SFTTrainer's packing = True make it any faster if I also set max_seq_length to 768 * 5?
My config:
max_seq_length = 768
dtype = None
load_in_4bit = True
model = FastLanguageModel.get_peft_model(
model,
r = 32,
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj",],
lora_alpha = 16,
lora_dropout = 0,
bias = "none",
use_gradient_checkpointing = "unsloth",
random_state = 3407,
use_rslora = False,
loftq_config = None
)
trainer = SFTTrainer(
model = model,
tokenizer = tokenizer,
train_dataset = dataset['train'],
dataset_text_field = "text",
max_seq_length = max_seq_length,
dataset_num_proc = 2,
callbacks=[CustomTrainerCallback],
packing = False,
args = TrainingArguments(
num_train_epochs=1,
per_device_train_batch_size = 100,
gradient_accumulation_steps = 4,
warmup_steps = 5,
#max_steps = 20,
learning_rate = 2e-4,
fp16 = not is_bfloat16_supported(),
bf16 = is_bfloat16_supported(),
logging_steps = 1,
optim = "adamw_8bit",
weight_decay = 0.01,
lr_scheduler_type = "linear",
seed = 3407,
output_dir = "outputs",
),
)

Thanks for the kind words :)

bsz does and should inc VRAM, however, in some cases, since the sequence lengths are minute, our RAM offloading "masks" it, hence you won't see any increase in VRAM.
Packing should make it faster