Addressing post-merge issues and comments

Signed-off-by: Andriy Redko<EMAIL_ADDRESS>Description
Addressing post-merge issues and comments
Issues Resolved
Addressing post-merge issues and comments from https://github.com/opensearch-project/spring-data-opensearch/pull/1
Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@rursprung please take a look, mostly addressing your comments, thank you


UI still freezes while using Isolates

Hello,
I am using Moor but I noticed that during a query the UI freezes. For this reason I switched to using Isolates, but the UI is still freezing and I don't know what I am doing wrong. I am using the code from the docs but adapted to use an existing database from the filesystem.
I fleshed out my project a bit to have a simpler example to show the problem.
One the FAB is clicked, I start showing a CircularProgressIndicator. This is spinning initially, but when the query is sent it freezes until the data is loaded...
Any help would be very appreciated! If needed I can make this whole test project available.
Thanks
main
import 'dart:io';
import 'dart:isolate';

import 'package:flutter/material.dart';
import 'package:flutter/services.dart';
import 'package:moor/ffi.dart';
import 'package:moor/isolate.dart';
import 'package:moor/moor.dart';
import 'package:path/path.dart';
import 'package:path_provider/path_provider.dart';

import 'infrastructure/database/flights_dao.dart';
import 'infrastructure/database/moor_database.dart';

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await _initDb();

  runApp(
    MaterialApp(
      title: 'Lazy Logbook',
      home: HomeWidget(),
    ),
  );
}

class HomeWidget extends StatefulWidget {
  const HomeWidget({
    Key key,
  }) : super(key: key);

  @override
  _HomeWidgetState createState() => _HomeWidgetState();
}

class _HomeWidgetState extends State<HomeWidget> {
  bool loading = false;
  List<FlightComposed> flights = [];

  @override
  Widget build(BuildContext context) {
    return Scaffold(
        floatingActionButton: FloatingActionButton(
          onPressed: () async {
            setState(() {
              loading = true;
            });
            MoorIsolate isolate = await _createMoorIsolate();
            var tempflights = await FlightsDao(MoorDatabase.connect(
                    DatabaseConnection.delayed(
                        isolate.connect(isolateDebugLog: true))))
                .getAllFlightsComposed();
            setState(() {
              flights = tempflights;
              loading = false;
            });
            isolate.shutdownAll();
          },
        ),
        appBar: AppBar(
          title: Text('test'),
        ),
        body: loading
            ? CircularProgressIndicator()
            : ListView.builder(
                itemCount: flights.length,
                itemBuilder: (context, index) {
                  return ListTile(
                    title: Text(flights[index].flight.flightCode.toString()),
                  );
                }));
  }
}

Future<void> _initDb() async {
  final dbFolder = await getApplicationDocumentsDirectory();
  var path = join(dbFolder.path, 'mcc2.db');

  await Directory(dirname(path)).create(recursive: true);
  ByteData data = await rootBundle.load(join("assets", "mcc.db"));
  List<int> bytes =
      data.buffer.asUint8List(data.offsetInBytes, data.lengthInBytes);
  await File(path).writeAsBytes(bytes, flush: true);
}

Future<MoorIsolate> _createMoorIsolate() async {
  // this method is called from the main isolate. Since we can't use
  // getApplicationDocumentsDirectory on a background isolate, we calculate
  // the database path in the foreground isolate and then inform the
  // background isolate about the path.
  final dir = await getApplicationDocumentsDirectory();
  final path = join(dir.path, 'mcc2.db');
  final receivePort = ReceivePort();

  await Isolate.spawn(
    _startBackground,
    _IsolateStartRequest(receivePort.sendPort, path),
  );

  // _startBackground will send the MoorIsolate to this ReceivePort
  return (await receivePort.first as MoorIsolate);
}

void _startBackground(_IsolateStartRequest request) {
  // this is the entry point from the background isolate! Let's create
  // the database from the path we received

  final executor = VmDatabase(File(request.targetPath), logStatements: true);
  // we're using MoorIsolate.inCurrent here as this method already runs on a
  // background isolate. If we used MoorIsolate.spawn, a third isolate would be
  // started which is not what we want!
  final moorIsolate = MoorIsolate.inCurrent(
    () => DatabaseConnection.fromExecutor(executor),
  );
  // inform the starting isolate about this, so that it can call .connect()
  request.sendMoorIsolate.send(moorIsolate);
}

// used to bundle the SendPort and the target path, since isolate entry point
// functions can only take one parameter.
class _IsolateStartRequest {
  final SendPort sendMoorIsolate;
  final String targetPath;

  _IsolateStartRequest(this.sendMoorIsolate, this.targetPath);
}

query in the DAO
Future<List<FlightComposed>> getAllFlightsComposed() async {
    final departureAirfields = alias(airfieldDtos, 'departureAirfields');
    final arrivalAirfields = alias(airfieldDtos, 'arrivalAirfields');

    var rows = await (select(flightDtos)
          ..orderBy([
            (t) => OrderingTerm(expression: t.flightDate),
            (t) => OrderingTerm(expression: t.departureTime),
          ]))
        .join([
      innerJoin(aircraftDtos,
          aircraftDtos.aircraftCode.equalsExp(flightDtos.aircraftCode)),
      innerJoin(
          departureAirfields,
          departureAirfields.airfieldCode
              .equalsExp(flightDtos.departureAirfieldCode)),
      innerJoin(
          arrivalAirfields,
          arrivalAirfields.airfieldCode
              .equalsExp(flightDtos.arrivalAirfieldCode)),
      innerJoin(pilotDtos, pilotDtos.pilotCode.equalsExp(flightDtos.picCode))
    ]).get();

    var result = rows.map((row) {
      var flightDetailed = FlightComposed(
        flight: row.readTable(flightDtos),
        aircraft: row.readTable(aircraftDtos),
        departureAirfield: row.readTable(departureAirfields),
        arrivalAirfield: row.readTable(arrivalAirfields),
        pic: row.readTable(pilotDtos),
      );
      return flightDetailed;
    }).toList();

    return result;
  }

How many flights are there, roughly speaking? There's some overhead that we can't avoid: Even if we're using isolates, we need to serialize and copy result data which takes time. For queries with huge result sets, isolates might even make things worse.
If you're loading lots of data at once, it might make sense to load data in chunks by introducing a LIMIT clause. On the ui side, packages like incrementally_loading_listview can help.

Dear Simon,
thanks a lot for your quick reply and suggestions.
There are only about 2300 flights, so I was a bit surprised that this would already be too much. I had also already tried limiting the flights to 100 per query, but this didn't make a big difference. However now I found out the culprit was probably that I was running the app in debug mode. If I run it in profile mode, the performance is already much better...
Brgds

However now I found out the culprit was probably that I was running the app in debug mode. If I run it in profile mode, the performance is already much better...

Is there documentation on this anywhere? Obviously I know that oftentimes with many frameworks/libraries debug performance can be worse, but hoping to understand reasoning specific to Drift. I am migrating from Isar (NoSQL) to Drift and that's for me the biggest thing I've noticed; very slow startup times. My use case is I have ~9000 finance transactions, each of which have about 15-20 properties. On app launch, these all are queried because the app does a bunch of calculations based on these transactions. There is probably room for optimization on my part, but I just want to be sure that there's nothing I'm missing.

Nothing immediately comes to mind, drift is not using a lot of costly assertions or anything else that would obviously be much slower on debug builds.
Are you loading the transactions one-by-one or in a single query? If anything interesting from drift pops out in the profiler I can also take a look at that. But this doesn't sound like anything drift shouldn't be able to handle.

Yeah I've checked the CPU Profiler and don't see anything obvious. Majority of the time was spent on isolate communication but that is about typical even before introducing drift because I make heavy use of isolates when crunching numbers on the transaction data. I'm new to drift so may be doing things incorrectly, but my transaction parsing logic is like this:
Stream<List<Transaction>> watchTransactions({
  required String budgetId,
}) {
  final query = select(dbTransactions).join([
    leftOuterJoin(
      dbSubTransactions,
      dbSubTransactions.transactionId.equalsExp(dbTransactions.uuid),
    ),
  ]);
  query.where(dbTransactions.budgetId.equals(budgetId) & dbTransactions.isDeleted.equals(false));
  query.orderBy([OrderingTerm.desc(dbTransactions.date)]);
  return query.watch().map((rows) {
    final collected = rows.parse(
      dbTransactions,
      (p0) => Transaction(
        id: p0.uuid,
        amount: p0.amount,
        date: p0.date,
        accountId: p0.accountId,
        isDeleted: p0.isDeleted,
        payeeName: p0.payeeName,
        payeeId: p0.payeeId,
        categoryId: p0.categoryId,
        categoryName: p0.categoryName,
        memo: p0.memo,
        transferAccountId: p0.transferAccountId,
        transferTransactionId: p0.transferTransactionId,
        matchedTransactionId: p0.matchedTransactionId,
        importId: p0.importId,
        flagColor: p0.flagColor?.name,
        subTransactions: List.empty(),
      ),
      dbSubTransactions,
      (p0) => SubTransaction(
        id: p0.uuid,
        transactionId: p0.transactionId,
        amount: p0.amount,
        isDeleted: p0.isDeleted,
        payeeName: p0.payeeName,
        payeeId: p0.payeeId,
        categoryId: p0.categoryId,
        categoryName: p0.categoryName,
        memo: p0.memo,
        transferAccountId: p0.transferAccountId,
        transferTransactionId: p0.transferTransactionId,
      ),
    );
    return collected.entries.map((e) => e.key.copyWith(subTransactions: e.value.toList())).toList();
  });
}

Parse is defined as an extension on the database instance:
// Parses the rows returned by a JOIN where each row
// is comprised of columns that can produce T and D when
// `readTable(orNull)` is called. The result is a map of
// T2 to a set of D2, which are produced when calling the
// `transformX` functions on T and D respectively.
Map<T2, Set<D2>> parse<T, D, T2, D2>(
  TableInfo<Table, T> tableA,
  T2 Function(T) transformA,
  TableInfo<Table, D> tableB,
  D2 Function(D) transformB,
) {
  final map = <T2, Set<D2>>{};
  for (final row in this) {
    final key = transformA(row.readTable(tableA));
    final value = row.readTableOrNull(tableB);
    final existing = map[key] ?? {};
    map[key] = value != null ? {...existing, transformB(value)} : existing;
  }
  return map;
}

It's very possible that collected.entries.map((e) => e.key.copyWith(subTransactions: e.value.toList())).toList(); is part of the lag here, but I don't recall seeing that in the profiler... I should look deeper.
That said, I do believe I've confirmed it IS for sure due to the number of transactions being processed, because it doesn't happen for smaller accounts with fewer transactions.

Just following up--I did a very rudimentary "Stopwatch test" and printed the elapsed time between when the query is invoked and when the function returns and it's about 5-6s which correlates to the initial lag I'm seeing during startup. Even worse, the vast majority of that time is the query execution itself. The parsing of rows is much less of a big deal which isn't what I expected. Here's the breakdown:
Total time: 5727ms/5.7s
Start query at 0ms
Received first rows at 5661ms: (98.84% of total time)
Parsed rows at 5708ms: (0.8% of total time)
Mapped to domain models at 5727ms (0.3% of total time)

So it looks to me like the actual query execution takes ~99% of the time, but I don't think I'm doing anything particularly interesting. FWIW the query returned 9371 rows in this case.

My use case is I have ~9000 finance transactions, each of which have about 15-20 properties. On app launch, these all are queried because the app does a bunch of calculations based on these transactions

What ends up happening with the result of the calculations? Is there any reason the calculations need to be done in dart instead of sqlite?
If you definitely need to load everything into dart memory, there are a couple things to keep in mind when optimizing.

Do the columns you're joining on have an index?
Do the columns in your WHERE clause have an index?
Your query selects the full Cartesian product of both tables - i.e if each of the 9000 Transaction has 2 SubTransactions, then your query would actually select 18000 rows of every column from both tables. That could be a lot of data - is all of it needed for processing? Could you query each table separately and do the "join" in dart?

In general though I'd recommend looking for ways to reduce the volume of data that's being touched: Can the UI be redesigned? No UI can show 9000 items. Can the "processing" be an SQL update? Get the database engine to do as much processing as possible.

What ends up happening with the result of the calculations? Is there any reason the calculations need to be done in dart instead of sqlite?

I actually had hopes of offloading a lot of the processing currently done in dart to sqlite as I think much of the logic can be done just as easily in SQL. Currently much of the logic I'm doing is done on a background Isolate due to the number of transactions. I'm utilizing package:actors for this and it works great. That said, I need to figure out why there's so much UI jank with a simple JOIN (likely my fault, I'm not blaming the package) before I feel confident enough to go forward with migrating logic to the query level.

Do the columns you're joining on have an index?

They didn't, but I added one and it did not help.

Do the columns in your WHERE clause have an index?

Same story as above. I will add that I haven't done anything database related since college, so I'm refreshing myself on everything. Here are the indexes I added:
// On Transactions
@TableIndex(name: 'idx_txn', columns: {#budgetId, #isDeleted})
// On SubTransactions
@TableIndex(name: 'idx_sub_txn', columns: {#budgetId, #transactionId, #isDeleted})

How many rows total are there in the Transactions table?

Since this was a select all it's safe to say it's very close to if not exactly the 9300 rows I mentioned above. I have not yet figured out how to dumb the database (I could obviously do this somewhat in dart code) from the Simulator and inspect it using some sqlite viewer but will do that eventually.

What’s the cardinality of the budgetId column?

Disclaimer: I had to look up cardinality, but the answer is "I believe extremely low". Of the ~9000 transactions, 99% of them are under a single budget id. The remainder are assigned to one of 3-4 other possible budget ids.

Your query selects the full Cartesian product of both tables - i.e if each of the 9000 Transaction has 2 SubTransactions, then your query would actually select 18000 rows of every column from both tables.

Fair point. The vast majority of transactions will not have sub-transactions, but still a valid point

That could be a lot of data - is all of it needed for processing?

I am sure I could optimize this a bit. Some calculations DO use all the transactions (to determine values since the "beginning of time"), but maybe I could change how this is done. To reiterate, I'm coming from Isar (NoSQL) and this was not an issue with Isar, even though it's not optimal. The UI freeze is only noticeable with drift.

Could you query each table separately and do the "join" in dart?

This is what I was doing with Isar, but the actual hang is happening in the query and not the "join". That's weird to me because AFAIK the default setup (createInBackground)  uses an Isolate for the query, so this feels odd to me.

Huh yeah that does seem odd - optimizing a slow query is one thing, but I overlooked that the primary issue is UI jank despite using isolates 😅
I'd be curious to see how the drift_sqflite backend performs in this scenario.

Huh yeah that does seem odd - optimizing a slow query is one thing, but I overlooked that the primary issue is UI jank despite using isolates 😅
I'd be curious to see how the drift_sqflite backend performs in this scenario. I think all you'd need to do is swap out your NativeDatabase constructor with a SqfliteQueryExecutor.

Gave this a shot this morning. Like you said, just swapped out NativeDatabase with return SqfliteQueryExecutor(path: file.path);
Unfortunately, no improvement 😞

I'll try to reproduce this as well, but it'll probably take me until the end of the week.

Mapped to domain models at 5727ms (0.3% of total time)

Do you define the mapping as the step where rows.parse is called? At that point drift will have already mapped the underlying rows to its own classes for the two tables. This step happens on the main isolate after the query ran and we have the raw results. So if that is how the stopwatch is set up, it might not report that the actual query is pretty fast and it takes drift too long to map the results.

My read on that was that the time between query.watch() being called and map being called (aka query execution) is the root of the slowdown here

I agree. But when map is called, drift will have mapped the query results into its own data classes generated from your table definitions already.
So the measurement unambiguously reveals that it takes a long time for drift to run the query and map results to drift-generated classes. But we don't know how much of that time is spent executing the query vs. drift doing its mapping. But I suspect the latter is taking most of the time.
You could measure the time for the actual execution with
class TimingInterceptor extends QueryInterceptor {
  @override
  Future<List<Map<String, Object?>>> runSelect(
      QueryExecutor executor, String statement, List<Object?> args) async {
    final sw = Stopwatch()..start();
    final rows = await super.runSelect(executor, statement, args);
    sw.stop();

    print('Ran $statement in ${sw.elapsed}');
    return rows;
  }
}

(you can apply this by passing NativeDatabase.createInBackground(...).interceptWith(TimingInterceptor()) to the database class)

OK that's super helpful, and has returned some interesting data. I slightly tweaked the print message because the elapsed time was getting pushed outside the debug message length:
if (sw.elapsed > 1.seconds) {
  print('BRANDON: ${sw.elapsed} (${rows.length}) : $statement');
}

Printed:
flutter: BRANDON: 0:00:06.431995 (1) : SELECT * FROM "db_spend_trackers" ORDER BY "name" ASC;

This table has 1 row in it 👀 So now I gotta figure out what I've done to cause this haha:
Here's the table definition:
class DbSpendTrackers extends Table {
  IntColumn get id => integer().autoIncrement()();
  TextColumn get name => text()();
  TextColumn get nickName => text().nullable()();
  TextColumn get sourceId => text()();
  TextColumn get budgetId => text()();
  BoolColumn get isDeleted => boolean()();
  TextColumn get type => textEnum<SpendTrackerType>()();
}

Yeah this is pretty wild. I used an sqlite viewer to view the DB and everything looks good. The same SQL statement run from that viewer runs in ~10ms.

OK Update: I commented out the query for the transactions table and the one for the above table began executing much faster, so it was a red herring that it took so long in my timings above.
Assuming a single Isolate (i.e that drift hasn't spun up a pool, which from my list of Isolates displayed by my IDE seems like the case), this just means the transactions query is blocking the execution of the other queries. That's fine, but doesn't explicitly explain the jank. So I'm still trying to understand that part.
Additionally, the timing/speed of the transactions table query is fast, so it's definitely something else related to that query that's causing the issue, and it must be happening on the main isolate.

this just means the transactions query is blocking the execution of the other queries

Ah true - the background isolate will serve requests in the order in which they come in, but can only serve on at the time. So if the main isolate is making requests concurrently, some queries will seemingly take longer than they should because they need to wait for their turn.
Installing the timing interceptor on the background isolate should yield more accurate measurements:
LazyDatabase(() async {
  final file = File(/*...*/);

  final isolate = await DriftIsolate.spawn(() {
    return NativeDatabase(file).interceptWith(TimingInterceptor());
  });

  return await isolate.connect();
});

But still, this doesn't explain why this only happens in debug builds - it has to be something happening in Dart.

Yeah I've checked the CPU Profiler and don't see anything obvious. Majority of the time was spent on isolate communication but that is about typical even before introducing drift because I make heavy use of isolates when crunching numbers on the transaction data.

Is there a way to start and stop the profile so that it primarily measures the transaction query, or is that impossible due to concurrent activity in the application? Do you see the time spent in JoinedSelectStatement._mapRow or $TransactionsTable.map, does anything stand out there?

OK I appreciate you pointing me back to the CPU Profiler, as there are some interesting things happening there:
JoinedSelectStatement._columnsWithName

JoinedSelectStatement._mapRow

InsertStatement.createContext

The last of these jumps out to me because I am probably doing something sub-optimal in how I'm handling updates to the transactions table. The API I'm hitting supports "delta" requests, so basically the process is:

Hit the API to fetch all transactions on first launch (on my account this equates to ~9k transactions). In this payload there is an int that can be provided in the next request and you will only receive transaction that have changed (modified or deleted) since that int
At each subsequent app launch (or every X minutes) the app hits the API again and provides the int. This generally only returns a handful of transactions, but obviously could return many if the user did some kind of bulk modification.

For both of these inserts/updates I'm using a transaction that calls insertOnConflictUpdate and passes a DbTransactionsCompanion.insert with all fields from the data returned by the API. I do this for both Transactions and SubTransactions. I wonder if this is not the right way to go about this. Should I insert only on the first bullet above and replace only on the second?

Thanks for providing the details from the profiler!  Spending so much time in _columnsWithName is ridiculous, there really is no reason to compute that over and over again for every row (which will have the same column names as all other rows) - fixed in 2d00149cb90da1187a36f1fe96e2e27e3ae30827.

I do this for both Transactions and SubTransactions. I wonder if this is not the right way to go about this.

I don't expect replace to have a different performance profile. At the moment, a source of inefficiency is that every insert creates its own SQL. When issuing many inserts with the same "shape" (columns appearing in the VALUES clause), we could potentially use that to not generate the SQL multiple times. But InsertStatement.createContext delegates most of its work to other functions not showing up in the profiler - nothing in that method stands out to me directly. And updates (through replace) will likely have the same issue.

I just did a test against that commit and here's what I saw:
BEFORE
Test 1
flutter: \^[[38;5;208m[W]\^[[0m  Slow query (1511ms)! SELECT "db_transactions"."uuid" AS "db_transac<…>
flutter: \^[[38;5;208m[W]\^[[0m  Slow query (1520ms)! SELECT * FROM "db_currency_formats" WHERE "bud<…>
flutter: \^[[38;5;208m[W]\^[[0m  Slow query (10932ms)! SELECT * FROM "db_spend_trackers" WHERE "budge<…>

Test 2
flutter: \^[[38;5;208m[W]\^[[0m  Slow query (1125ms)! SELECT "db_transactions"."uuid" AS "db_transac<…>
flutter: \^[[38;5;208m[W]\^[[0m  Slow query (1382ms)! SELECT * FROM "db_currency_formats" WHERE "bud<…>
flutter: \^[[38;5;208m[W]\^[[0m  Slow query (9809ms)! SELECT * FROM "db_spend_trackers" WHERE "budge<…>

AFTER
flutter: \^[[38;5;208m[W]\^[[0m  Slow query (556ms)! SELECT "db_category_groups"."uuid" AS "db_cate<…>
flutter: \^[[38;5;208m[W]\^[[0m  Slow query (1568ms)! SELECT "db_transactions"."uuid" AS "db_transac<…>
flutter: \^[[38;5;208m[W]\^[[0m  Slow query (1838ms)! SELECT * FROM "db_currency_formats" WHERE "bud<…>
flutter: \^[[38;5;208m[W]\^[[0m  Slow query (3733ms)! SELECT * FROM "db_spend_trackers" WHERE "budge<…>

Test 2
flutter: \^[[38;5;208m[W]\^[[0m  Slow query (836ms)! SELECT * FROM "db_payees" WHERE "budget_id" = <…>
flutter: \^[[38;5;208m[W]\^[[0m  Slow query (1018ms)! SELECT "db_category_groups"."uuid" AS "db_cate<…>
flutter: \^[[38;5;208m[W]\^[[0m  Slow query (1928ms)! SELECT "db_transactions"."uuid" AS "db_transac<…>
flutter: \^[[38;5;208m[W]\^[[0m  Slow query (1515ms)! SELECT * FROM "db_currency_formats" WHERE "bud<…>
flutter: \^[[38;5;208m[W]\^[[0m  Slow query (3700ms)! SELECT * FROM "db_spend_trackers" WHERE "budge<…>

As a reminder, the log logic is:
if (sw.elapsed > 500.milliseconds) {
  logger.w('Slow query (${sw.elapsedMilliseconds}ms)! ${statement.substring(0, 50)}');
}

The BEFORE has fewer queries over the 500 threshold BUT the AFTER definitely took a big chunk out of the slowest query (which was likely just held up due to the Transactions query). This was without the interceptor installed on the Isolate, I'll post another comment if that ends up making a difference in these timings.

Do you think the remaining slowness is due to the issue you mentioned above?

At the moment, a source of inefficiency is that every insert creates its own SQL. When issuing many inserts with the same "shape" (columns appearing in the VALUES clause), we could potentially use that to not generate the SQL multiple times...

Sorry to keep nagging you with the profiler, but is any change visible there? I'm actually surprised to see this impact the measured times on the isolate at all, since all of the result mapping is happening on the main isolate.
For the inserts, I agree that the duplicate query building could be a source of remaining slowness. If you're not using them already, I suggest taking a look at batches for bulk operations though. They won't fix the duplicate building problem, but they will bundle everything into a single message sent to the executor isolate, so there's less async overhead compared to many individual statements.

Sorry to keep nagging you with the profiler, but is any change visible there?

Here's what I'm seeing now:
BaseExecuter.runRequest

BaseExecuter.runRequest (again)

DriftCommunication.send

JoinedSelectStatement.mapRow

InsertStatement.createContext

I suggest taking a look at batches for bulk operations though

I'll take a look at batches, thanks for recommending them!

Here are timings after moving the heaviest insertion procedure to a batch (insertTransactions inserts ~9k entities, the rest are only inserting ~1-50 entities).

Unfortunately this is still an issue 😞 The orange bit in the middle is the insert codepath via drift. It would be amazing to get this fixed 🙏

Thanks for the follow-up! How large is the batch? There will always be some costs associated with generating lots of SQL statements (there's a potential optimization of caching SQL statements if most companions have the same columns present, but still).
From a quick search of this thread I haven't mentioned this yet, but you can use computeWithDatabase to spawn a temporary database that has access to the database - if you have large batches only depending on data that can be sent across isolates cheaply (or even making a network call on the background isolate if that's where the batch is coming from), that's a good way to offload the batch entirely. But still I'll look into optimization opportunities of caching statements if most companions have the same overall structure.


Stable release

Hello.
Sorry to be a burden by my request.
I'm simply interested to know if this is a stable project and whether it can be used in production.
I'm not expecting a definitive answer, but perhaps I'm not the only one with this question.

As a shared library replaceing zlib system-wide; no, zlib-ng has a few crash issues with some applications, presumably due to differences in alignment.
Statically linked or LD_PRELOAD or otherwise for a specific application it should be pretty stable, although you of course have to do your own testing before rolling this out into production systems.
Confidence-wise, I would not use zlib-ng for storage of important data at this point. For compression of html/etc to be delivered to a web-browser on-demand should not be a problem, since if you stumble upon a bug then you can just revert to stock zlib or a bugfixed zlib-ng without loosing important data.
We do need testers though, so any experiences both positive and negative would be appreciated. Currently we have no idea what applications crash with zlib-ng, and whether a recompile of that application to use zlib-ng would fix the problem.

@Dead2 any news about a release?

I'm not sure if this helps but I can explain our use case.
We are delivering images compressed using libpng. libpng uses libz for compression. We saw a significant improvement in performance using zlib-ng to generate png files. The png files are delivered to web browsers so they are not critical, but latency is very important. We haven't seen any issues or corruption, we compile the zlib-ng source directly into the application and link directly against it to produce a static binary.

@Adenilson I am planning to get back to zlib-ng soon. We had some big deadlines at work before and during the summer, so time has been very short. Hoping to get the remaining issues worked out this fall.
@ioquatix That is great feedback, thank you. Real-world testing beats narrow benchmarks any day 👍
I have been doing some smaller zlib-ng work since my last commit, mostly testing and validation related, such as static analysis, valgrind runs etc. Haven't had time to test it in any applications yet unfortunately.
I am wondering whether we should rename the library before any release is done though, so it can be parallel-installed with stock zlib. Could be made an optional thing, but I haven't looked at how much needs to change to do this properly. Could be done in relation to the zlib-compat compile option perhaps, so by default we use our new name, but compiling with zlib-compat would make it use zlib-naming as well. Thoughts and PRs on this would be much appreciated.

Given my limited experience...

zlib as maintained by Mark Addler seems to be very slow to evolve and seems to be very old code without much work to modernise it.

lots of libraries depend on that header and library specifically without much flexibility.

it's nice you can just build one library or the other, they both provide the same API, so it's very low cost to drop in, e.g. to replace compression in libpng.

I think it's good it's the same, because honestly, it makes my life much easier.

@ioquatix I am not suggesting that we remove the zlib-compat features, only that we make it optional. For you, this would make no difference since you are free to compile it the way you want to anyway.
For distros, this is a big deal, since most distros will not want to change to zlib-ng completely all in one go (and they should not, unless they are a very risky cutting-edge distro). Allowing them to package and provide zlib-ng without risking breakage in any other package will make it very easy for them to accept zlib-ng into their fold. That in turn will allow both their separate packages to choose to compile against zlib-ng when/if that application has been tested to work properly with our fork. This also lets end-users/developers compile against zlib-ng without having to compile and install zlib-ng itself.
We need to think about the big picture, but we also need to make sure we don't sacrifice too much. :)

On Friday August 11 2017 02:01:00 Hans Kristian Rosbach wrote:

I am wondering whether we should rename the library before any release is done though, so it can be parallel-installed with stock zlib. Could be made an optional thing, but I haven't looked at how much needs to change to do this properly. Could be done in relation to the zlib-compat compile option perhaps, so by default we use our new name, but compiling with zlib-compat would make it use zlib-naming as well.

+++ IMVHO the zlib-compat build option makes sense only if it results in a drop-in replacement for stock zlib -- and as shown here and elsewhere there's a real interest to that. That doesn't mean though that building for a different library name makes sense only in non-compat mode:
I am more sceptical about your view on introducing zlib-ng via a different library name. You're right that this could help, but it's probably still as dependent on being a drop-in replacement. Distros like Debian and Ubuntu have a mechanism in place where users chance chose between alternative packages to provide a given feature. That makes perfect sense for a central package like zlib: just about everything and its dog depend on it. So yes, making it possible to build for a different library name also in compat mode is a good idea because it makes the endeavour less risky in the sense that it becomes much easier to switch back and forth.
It shouldn't be hard nor unheard-of to introduce a library suffix that can be specified via configure and/or cmake (ffmpeg does, for instance).
Truth be told, for me the main interest of zlib-ng is really that (if) it can be a drop-in replacement for stock zlib . New features and support for platforms zlib doesn't support are iceing on the cake (on the latter the whole compatibility issue is moot of course as long as the resulting files can be exchanged).
R.

FYI: Chromium developers considered switching to zlib-ng recently, but they didn't in the end mainly due to the lack of a stable release https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAKx6XHPtwuq%2B3kSMM5tMt5uZZEEnwpnYKbLRb%2B0t5L45bRDG1w%40mail.gmail.com

@clopez Thanks for keeping us updated.
However, lets focus on doing it right and not rush something out just because someone wants to use it, they too will be better off if we do this right. Of course, doing things right requires time and energy, something that is unfortunately often a scarce resource in open-source. I am still hoping to release a release candidate soon, and hopefully get a proper stable out sometime this fall.
That said, I do view the hacknslash7 branch as beta quality, and most of what needs to change before release is build-system stuff and testing. The code itself seems to be in pretty good condition at this point.

We are now one step closer to a stable release. We are missing one feature though; minizip, see #156 .
The renamelib branch was tested for several months in production webservers statically compiled into nginx, that means a pretty limited scope, but there were no problems detected during testing. A lot of PRs was merged today though, so that might potentially have changed. We'll need to do more wide testing to make sure no new bugs have cropped up.
I do wish we had a good test framework though, the current tests are the bare minimum "Hello world" of testing, literally.

With the help of oss-fuzzers https://github.com/google/oss-fuzz/tree/master/projects/zlib-ng with more than 80% coverage I think zlib-ng now has pretty good testing in place.
@Dead2 what remains to be done to get a first stable release of zlib-ng?
A stable release would enable chromium.org and aosp to move to zlib-ng: see https://crbug.com/687631

I agree. I will try to reserve time tomorrow for doing final testing and hopefully merging of a few of the remaining PRs. When that is done, I think we can create a stable branch and perhaps put out a RC.
Remaining things would probably mainly be documenting the parts where zlib-ng differs from zlib the most, and simple end-user guidance. Such as how to build, showing the configure vs cmake methods, perhaps document the most important zlib-ng specific build options. Perhaps some documentation about why zlib-ng api differs and what programs needs to do to change existing zlib-compatible code.
I do wish someone other than me could have a look over zlib-ng.h, and see whether there are more things we should change before going stable. Think things like changing datatypes to stable stdint types instead of short/int/long for example. Mainly I think we want to stay pretty close to the zlib api, while updating the api to be more strict about datatypes. It has been a while since I last looked, so I don't really remember whether there were more things I was considering to change there, but I'll try to have a look at that too tomorrow.
@sebpop Great work on the test framework and fixes these past months, it has made me a lot more confident that we are indeed ready for a stable release.

@Dead2 any progress on a stable release?

We've been using it in production for several years now and it seems pretty solid!

We are getting there, slowly.
We want to get a couple new function into the API, for setting and getting settings beyond the scope of normal zlib functionality.
The zlib-ng.h file still needs to get a thorough review to check whether we want/should/need to make any changes before making a stable release.
Those are the API-level blockers for making a 1.0 version.
In addition there are some minor cleanups and changes we could/should do before stable, but nothing really critical.
Documentation of differences from stock zlib is also a major task I'd like to see some progress on before a stable release.
We have also been using zlib-ng in prod for a couple years now, and have not had any real problems during runtime use (compilation issues is another matter, but that is not as critical, and is getting fixed as we find them).

@Dead2 any news for stable release ?


Internal requests do NOT work properly

Add the following to your routes.php
$api = app('Dingo\Api\Routing\Router');
$api->version('v1', function ($api) {
    $api->resource('recipe', 'App\Http\Controllers\IndexController');
});

Add, edit or replace the following two methods in your IndexController
class IndexController extends Controller
{
    use Helpers;
    public function index()
    {
        $dispatcher = app('Dingo\Api\Dispatcher');

        try {
            $response = $dispatcher->with(
                [
                    'title'        => '!!!',
                    'description'  => '',
                    'lng_code'    => 'en'
                ])->post('recipe');
        }
        catch (Exception $e)
        {
            debug($e);
        }
    }

    public function store(StoreRecipeRequest $request)
    {
        return $request->get('title');
    }
}

Create the following file named StoreRecipeRequest.php in app\Http\Requests\
<?php
namespace App\Http\Requests;
use App\Http\Requests\Request;

class StoreRecipeRequest extends \Dingo\Api\Http\FormRequest
{
    /**
     * Determine if the user is authorized to make this request.
     *
     * @return bool
     */
    public function authorize()
    {
        return true;
    }

    /**
     * Get the validation rules that apply to the request.
     *
     * @return array
     */
    public function rules()
    {
        return [
            'title'         => 'required|string|max:128',
            'description'   => 'string',
            'lang_code'     => 'required|max:2'
        ];
    }
}

Now, using (for example) Postman to call the store() method with the same data being passed in as in the Controller's index() function, the results returned are as expected.
{
  "message": "422 Unprocessable Entity",
  "errors": {
    "lang_code": [
      "The lang code field is required."
    ]
  },
  "status_code": 422,.....

However, if you try to run the controller's index() function, you'll see this: ValidationHttpException in FormRequest.php line 22:
Removing the type hinting from the store() method (so that no more FormRequest is executed), and manually writing the validation logic in the store() method, like so:
    public function store(\Illuminate\Http\Request $request)
    {
        $this->validate($request, [
            'title'         => 'required|string|max:128',
            'description'   => 'string',
            'lang_code'     => 'required|max:2'
        ]);
    }

does NOT work either, this time throwing the following: InternalHttpException in Dispatcher.php line 543:
Am I doing it wrong?

Also watch #1014

Still no update or at least confirmation of this issue?

I would also like to know if this is getting fixed soon, it is keeping me from using this package.

@datune I think you have a typo in your IndexController:
       $response = $dispatcher->with(                 [                     'title'        => '!!!',                     'description'  => '',                     'lng_code'    => 'en'                 ])->post('recipe');
Should lng_code be lang_code ?

@jayhealey It's a typo on purpose, in order to trigger the validator...

still not fixed? :(

Any ideas how to fix this yet ?

+1

Gonna add this to the list of issues I need to address, once I finished getting caught up and going through the backlog of issues.

@datune So looked into this, is the issue you're having that calling a FormRequest validated route internally throws the ValidationHttpException exception? But when you call the endpoint externally (like through Postman), it correctly renders a JSON response?

@hskrasek
Thanks for looking into it, been waiting a long time. To answer your question, yes, externally calling the API works as expected, but internal requests throw exceptions.

@datune No worries. So the fact that internal requests throws an exception, in this case, during validation is by design. Since you're internally calling the API, this gives you the power to control what happens in an error case.
Think of it as using Guzzle to call your own API, you'd probably try/catch the Guzzle call, and handle the exception accordingly.
If you want the response from validation failures to be returned, you could remove the try/catch, which will let the exception be handled and rendered by Dingo's handler.

@hskrasek
This can't be by design, since the exception can not be catched! Give it a try and you'll see what I mean.

@datune If you look at the test here, you can see that any exception that extends HttpException does in fact fall through the router https://github.com/dingo/api/blob/master/tests/DispatcherTest.php#L160-L177. This is only for internal requests though.
Should also make sure your FormRequest objects are extending Dingo's and not Laravels.

Please see my comment at: https://github.com/dingo/api/issues/998#issuecomment-242742171


tensor size not match

Before saying the problems, I show my running environment:
pytorch version 1.0.1.post2
cuda version 8.0.61
cudnn version: 7102
python version: 3.7
I run the latest code, two problems bother me:
1.Traceback (most recent call last):
File "train_autodeeplab.py", line 20, in
import apex
File "/home/zhangjw/anaconda3/lib/python3.7/site-packages/apex/init.py", line 18, in
from apex.interfaces import (ApexImplementation,
File "/home/zhangjw/anaconda3/lib/python3.7/site-packages/apex/interfaces.py", line 10, in
class ApexImplementation(object):
File "/home/zhangjw/anaconda3/lib/python3.7/site-packages/apex/interfaces.py", line 14, in ApexImplementation
implements(IApex)
File "/home/zhangjw/anaconda3/lib/python3.7/site-packages/zope/interface/declarations.py", line 483, in implements
raise TypeError(_ADVICE_ERROR % 'implementer')
TypeError: Class advice impossible in Python3.  Use the @implementer class decorator instead.
Then I comment related code and set the APEX_AVAILABLE=False. The code continues to run， but the problems 2 occurs

Traceback (most recent call last):
File "train_autodeeplab.py", line 421, in
main()
File "train_autodeeplab.py", line 414, in main
trainer.training(epoch)
File "train_autodeeplab.py", line 176, in training
output = self.model(image)
File "/home/zhangjw/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/zhangjw/AutoML/auto_deeplab.py", line 282, in forward
normalized_alphas)
File "/home/zhangjw/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/zhangjw/AutoML/cell_level_search.py", line 138, in forward
s = sum(new_states)
RuntimeError: The size of tensor a (13) must match the size of tensor b (14) at non-singleton dimension 3
It's so weird, so I print the new_states[0].shape, as following:
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 16, 14, 14])
torch.Size([2, 16, 14, 14])
torch.Size([2, 16, 14, 14])
torch.Size([2, 16, 14, 14])
torch.Size([2, 16, 14, 14])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 16, 14, 14])
torch.Size([2, 16, 14, 14])
torch.Size([2, 16, 14, 14])
torch.Size([2, 16, 14, 14])
torch.Size([2, 16, 14, 14])
torch.Size([2, 16, 14, 14])
torch.Size([2, 16, 14, 14])
torch.Size([2, 16, 14, 14])
torch.Size([2, 16, 14, 14])
torch.Size([2, 16, 14, 14])
torch.Size([2, 32, 7, 7])
torch.Size([2, 32, 7, 7])
torch.Size([2, 32, 7, 7])
torch.Size([2, 32, 7, 7])
torch.Size([2, 32, 7, 7])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 4, 56, 56])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 8, 28, 28])
torch.Size([2, 16, 13, 13])

when the last line torch.Size([2, 16, 13, 13]) comes,  RuntimeError occurs.It's so weird.
 I would appreciate it if you tell me how to solve it.

@JingweiZhang12 hi, we’re running this code daily and haven’t seen this error yet. Can you give us the details of your run? Hyper parameter setting ect. Sorry about this, we’ll try to get it working again quick. And do let us know if you find anything.

I encountered something similar when I use: --crop_size 112.
I don't know if it helps..
One I got it back to 224 everything was ok

@JingweiZhang12
Hi, this first issue you described with apex seems like an apex installation issue and it's not related specifically to this repository. please see https://github.com/NVIDIA/apex/issues/116. could you try to uninstall apex and then reinstall and see it this solves the issue?
regarding the tensor size mis-match, i think @albert-ba might be right. the network can't accept any arbitrary input size, since it might cause size mis-match during one of the down-sampling \ up-sampling operations. could you please share the hyper-parameters you used in your run?

@iariav
Thank for your helpful information. I refer to NVIDIA/apex#116  and reinstall apex. The module apex can be used now.
As for the tensor size mis-match, I change the --crop-size from 224 to 256,  and it works now.


fix: build on arm macs that was broken after update of cppcheck

In new version of cpp check there is no with_z3 build option and we don't need any specific options for macos arm build

prnet cleaned-up successfully


When going to home web page, there are language latest releases missing.

When navigating to: https://azure.github.io/azure-sdk/
There is no data for C++, C, iOS and Android.
Also, the sidebar is all collapsed and top label latest is not highlighted.
User need to go to https://azure.github.io/azure-sdk/releases/latest
Then the data for C++, C, Android and iOS is populated.
And also the sidebar get sync
The root cause is that we have two different pages populating the language's latest releases data, and we need to maintain both.
Right now, they are not showing the same info.
The suggested solution is to replace the main root page for a direct redirect meta link which will redirect from home / to /releases/latest
Then, we need to create a new md file that group all the language's data and use it everywhere instead of listing all the data everywhere.

Nice catch!

There is no data for C++, C, iOS and Android.

Well, we can't have that! :)


Add copyright information to all source files

I'm not sure if this is necessary, but we might need to add the Sandia copyright information to all the source files in the repository.  It may be possible to automate this with the snlcopyright package.

Handled in #91.


[BUG]  Websockets API no longer sending data.

Lnbits v 0.12.8
Updated to 0.12.8 and websockets no longer sends data.  Reproduced using the demo site wss://demo.lnbits.com/api/v1/ws/2ec7f696f0854d88aed3279cdc08a907 and a websocket tester. (https://piehost.com/websocket-tester)
Websocket connects fine, but does not send any data regarding payments.  (Now all my websocket listeners are broken)

The websocket path was changed from wallet id to wallet inkey.
Wallet ID is public (can be seen in the URL) so it is considered unsafe (someone could listen to your wallet if it knows its ID)
https://github.com/lnbits/lnbits/pull/2516/files#r1615868255
Please try to use the wallet inkey and let me know if it works.

The websocket path was changed from wallet id to wallet inkey. Wallet ID is public (can be seen in the URL) so it is considered unsafe (someone could listen to your wallet if it knows its ID)
https://github.com/lnbits/lnbits/pull/2516/files#r1615868255
Please try to use the wallet inkey and let me know if it works.

Are you able to give me an example to how I could use websocket to listen to wallet activities(payment sent and payment received) please? What do i need to send in {data}?

On the right side of your wallet you can find the invoice key (inkey). Use that one for the wrbsocket connection instead of the wallet ID.

Thanks for the update.  The invoice key works great.

Here's one of the ways I'm using it:
`#!/usr/bin/env python3
import asyncio
import websockets
import json
import logging
import os
import httpx
import time
from typing import Dict, Any
from dataclasses import dataclass
from asyncio import Task
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logger = logging.getLogger(name)
@dataclass
class Config:
WS_CYBERHERD: str
WS_HERD: str
ENDPOINT_URL: str
IGNORE_NPUBS: list[str]
def load_config() -> Config:
return Config(
WS_CYBERHERD=os.getenv('WS_CYBERHERD', "wss://lnb.bolverker.com/api/v1/ws/"), # add invoice key
WS_HERD="wss://lnb.bolverker.com/api/v1/ws/", # add invoice key
ENDPOINT_URL="http://<IP_ADDRESS>:8090/messages/cyberherd_treats",
IGNORE_NPUBS=["Sat", "Unknown"]
)
config = load_config()
class WebSocketClient:
def init(self, uri: str, identifier: str):
self.uri = uri
self.identifier = identifier
self.websocket = None
self.http_client = httpx.AsyncClient(http2=True, limits=httpx.Limits(max_keepalive_connections=10))
self.reconnect_task: Task | None = None
async def connect(self):
    backoff = 1
    max_backoff = 60
    while True:
        try:
            self.websocket = await websockets.connect(self.uri)
            logger.info(f"Connected to WebSocket ({self.identifier}): {self.uri}")
            return
        except (websockets.exceptions.ConnectionClosedError, websockets.exceptions.InvalidURI) as e:
            logger.error(f"WebSocket connection error ({self.identifier}): {e}")
        except Exception as e:
            logger.error(f"Unexpected error during connection ({self.identifier}): {e}")

        logger.info(f"Reconnecting in {backoff} seconds ({self.identifier})...")
        await asyncio.sleep(backoff)
        backoff = min(backoff * 2, max_backoff)

async def send_to_endpoint(self, pubkey: str, amount: float):
    payload = {"pubkey": pubkey, "amount": amount}
    try:
        response = await self.http_client.post(config.ENDPOINT_URL, json=payload, timeout=10.0)
        response.raise_for_status()
        logger.info(f"Sent to endpoint: {payload}")
    except httpx.HTTPError as exc:
        logger.error(f"Error sending to endpoint: {exc}")

def extract_pubkey(self, memo: str) -> str | None:
    words = memo.split()
    if len(words) > 0 and words[-1] not in config.IGNORE_NPUBS:
        return words[-1]
    return None

async def process_herd_message(self, data: Dict[str, Any]):
    if "wallet_balance" in data:
        wallet_balance = data["wallet_balance"]
        amount = data["payment"]["amount"] / 1000
        wallet_fiat_rate = data["payment"]["extra"]["wallet_fiat_rate"]
        logger.info(f"[Herd] Wallet balance: {wallet_balance}, Amount: {amount}, Wallet Fiat Rate: {wallet_fiat_rate}")

async def process_cyberherd_message(self, data: Dict[str, Any]):
    if "payment" in data and data["payment"].get("amount", 0) < 0:
        amount = abs(data["payment"]["amount"]) / 1000
        memo = data["payment"]["memo"]
        pubkey = self.extract_pubkey(memo)
        if pubkey:
            await self.send_to_endpoint(pubkey, amount)

async def process_messages(self):
    while True:
        try:
            message = await self.websocket.recv()
            data = json.loads(message)

            if self.identifier == 'herd':
                await self.process_herd_message(data)
            elif self.identifier == 'cyberherd':
                await self.process_cyberherd_message(data)
        except websockets.exceptions.ConnectionClosed:
            logger.warning(f"WebSocket connection closed ({self.identifier}). Reconnecting...")
            break
        except json.JSONDecodeError:
            logger.error(f"Failed to decode JSON message ({self.identifier})")
        except Exception as e:
            logger.error(f"Error processing message ({self.identifier}): {e}")

async def run(self):
    while True:
        await self.connect()
        await self.process_messages()

async def start(self):
    self.reconnect_task = asyncio.create_task(self.run())

async def stop(self):
    if self.reconnect_task:
        self.reconnect_task.cancel()
        try:
            await self.reconnect_task
        except asyncio.CancelledError:
            pass
    if self.websocket:
        await self.websocket.close()
    await self.http_client.aclose()

async def main():
clients = [
WebSocketClient(config.WS_CYBERHERD, 'cyberherd'),
WebSocketClient(config.WS_HERD, 'herd')
]
for client in clients:
    await client.start()

try:
    await asyncio.gather(*(client.reconnect_task for client in clients))
except asyncio.CancelledError:
    logger.info("Main task cancelled. Shutting down...")
finally:
    await asyncio.gather(*(client.stop() for client in clients))

if name == "main":
try:
asyncio.run(main())
except KeyboardInterrupt:
logger.info("WebSocket listener stopped.")`

An html / javascript example:
`

js overlay


[BUG] Cannot create optional multi-valued positional argument

Describe the bug
First off, let me thank you for creating typer 😄 I absolutely love it, especially because click was my favorite CLI library and you went and made it even nicer 👍
Soooo, I was attempting to create a CLI with an optional multi-valued positional argument (i.e. taking zero or more values). I made two attempts to no avail. The first one didn't even run, and the second one didn't allow me to omit the argument (the "optional" part).
To Reproduce
Here is a minimal example that shows my first attempt at using a list-valued typer.Argument(). According to the docs, providing a default value makes the argument optional, so I provided an empty list.
import typer
import typing

app = typer.Typer()

@app.command()
def test(
    patterns: typing.List[str] = typer.Argument([]),
    untracked: bool = typer.Option(False, "-u", "--untracked"),
):
    typer.echo(locals())

if __name__ == "__main__":
    app()

Running this with python test.py gives me the following stack trace:
% python test.py
Traceback (most recent call last):
  File "test.py", line 14, in <module>
    app()
  File "/home/rui/miniconda3/envs/test/lib/python3.8/site-packages/typer/main.py", line 213, in __call__
    return get_command(self)()
  File "/home/rui/miniconda3/envs/test/lib/python3.8/site-packages/typer/main.py", line 238, in get_command
    click_command = get_command_from_info(typer_instance.registered_commands[0])
  File "/home/rui/miniconda3/envs/test/lib/python3.8/site-packages/typer/main.py", line 422, in get_command_from_info
    ) = get_params_convertors_ctx_param_name_from_function(command_info.callback)
  File "/home/rui/miniconda3/envs/test/lib/python3.8/site-packages/typer/main.py", line 403, in get_params_convertors_ctx_param_name_from_function
    click_param, convertor = get_click_param(param)
  File "/home/rui/miniconda3/envs/test/lib/python3.8/site-packages/typer/main.py", line 722, in get_click_param
    click.Argument(
  File "/home/rui/miniconda3/envs/test/lib/python3.8/site-packages/click/core.py", line 1986, in __init__
    raise TypeError(
TypeError: nargs=-1 in combination with a default value is not supported.

So in an attempt to fix this error, I replaced with argument's default value with ellipsis ..., which allows the program to run but requires at least one value for the argument.
% python test.py
Usage: test.py [OPTIONS] PATTERNS...
Try 'test.py --help' for help.

Error: Missing argument 'PATTERNS...'.

Expected behavior
I believe the error is due to typer marking the argument as required despite it being a list-valued argument. I would expect it to be possible (and even the default behavior!) to create list-valued arguments that take zero or more values, instead of one or more. Even better (if possible) would be a generalization where minimum and maximum number of values consumed by the argument are specified, defaulting to 0 and infinity, respectively.
Environment

OS: Debian 10
Python 3.8.2
Typer 0.2.1
Click 7.1.2

Hi,
To enable you to take in a list with zero or more values instead of using typer.Argument(...)  or typer.Argument([]) you can use typer.Argument(None).
Running the script below withpython3 main.py doesn't raise any errors.
import typer
import typing

app = typer.Typer()
app.command()

def test(
    patterns: typing.List[str] = typer.Argument(None),
    untracked: bool = typer.Option(False, "-u", "--untracked"),
):
    if patterns:
        typer.echo("Arguments for patterns provided")
    else:
        typer.echo("No arguments for patterns provided")

if __name__ == "__main__":
    app()

@IamCathal thank you for the response. I am very happy to know that it is possible to have optional multi-valued arguments in typer! 🥳
I seem to have missed the part of the docs where a similar example is given... 😅
I was probably misguided by this other example (https://typer.tiangolo.com/tutorial/multiple-values/multiple-options/#multiple-float) where an empty list is used as a default for a list-valued Option. Seeing this, I expected to be able to do the same with Argument.
Besides, to me at least, it is unintuitive to use None as a default for a list-valued argument, only to have it magically converted into an empty tuple before reaching the command.
Perhaps this should be fixed for consistency across Argument and Option. If not, at least the docs should mention this limitation/difference.
I am closing the issue since it is not a bug after all. Thanks again!

Thanks a lot for the help @IamCathal ! :cake: :bow:
Yeah, I agree @rrei it is not great. :sweat_smile: It is actually a bug in Click itself if I remember correctly.
Anyway, thanks for reporting back and closing the issue!


jshint not loading to syntastic abnormal termination

recently I switched to use node with 'nvm', installed jshint  set my vimrc  like this
set statusline+=%#warningmsg#
set statusline+=%{SyntasticStatuslineFlag()}
set statusline+=%*

let g:syntastic_always_populate_loc_list = 1
let g:syntastic_auto_loc_list = 1
let g:syntastic_check_on_open = 1
let g:syntastic_check_on_wq = 0
let g:syntastic_javascript_jshint_exec = '/home/user/.nvm/versions/node/v0.12.1/bin/jshint'

When opening vim now i get this and jshint is never loaded
syntastic: error: checker javascript/jshint: can't parse version string (abnormal termination?)

Please open a JavaScript file, set g:syntastic_debug to 32, run :SyntasticInfo, run :mes, and post the output.

yeah sorry i found a workaround and now it works. I tried to roll back to where it was before but it is not erroring out anymore.
The workaround is to install a copy of node's binaries for the nvm system.
i tried to remove the apt copy and just use the nvm one but now is not crashing anymore.
It might be a problem with the way/order i installed thing.
thanks, anyway

Once again, for people searching the archives: in the vast majority of cases, "abnormal termination" is a symptom of a problem outside syntastic.

I had the same problem using nvm and solved adding the following to my .vimrc:
if $PATH !~ "\.nvm"
    let $PATH="/home/services/.nvm/versions/node/v0.12.2/bin:" . $PATH
endif

I have the same probem. I am running ubuntu and installed npm.
When running jshint without arguments on the console, I get this exception:
/usr/bin/env: node: No such file or directory

When installing nodejs on ubuntu, the binary gets named nodejs not node
Locally, I fixed it by creating a node symlink:
sudo ln -s /usr/bin/node{js,}

I understand that this is not the fault of syntastic, but I did not find this hint anywhere else.

I also had this problem, and it seems to be because my jshint (version 2.9.2 from npm) outputs jshint v2.9.2 in response to --version. I made a wrapper as a temporary fix (obviously it would be better to actually parse the version than hard-code it).
#!/bin/sh

if [ "$1" = '--version' ]
then
  echo 2.9.2
else
  jshint "$@"
fi

I can't help but wonder if this is a common problem. Would it be a good idea to have an universal way to not try to parse the version string? Or make the version parser more lenient, and ignore everything except numbers and dots?

@hagabaka No, the output of v2.9.2 is not a problem for syntastic, I have it working fine here with the exact same version. Trying to run jshint not installed globally might be though.  Or it might be something completely unrelated.
Please open a new issue, and state (1) your Vim version, (2) what you did, (3) what did you expect to happen, and (4) what happened instead.  Open a JavaScript file, run :SyntasticInfo, and post the output.  Then set g:syntastic_debug to 3, run :SyntasticCheck jshint, run :mes, and post the output.

@lcd047 The output for me is jshint v2.9.2, not just v2.9.2.

@hagabaka Syntastic can still parse jshint v2.9.2:
:echo syntastic#util#parseVersion('jshint v2.9.2')
[2, 9, 2]

As I said, your problem is elsewhere.

I had the same problem with jsxhint,
Doing this solved my problem:
--- a/syntax_checkers/javascript/jsxhint.vim
+++ b/syntax_checkers/javascript/jsxhint.vim
@@ -23,7 +23,8 @@ function! SyntaxCheckers_javascript_jsxhint_IsAvailable() dict " {{{1
endif
 let version_output = syntastic#util#system(self.getExecEscaped() . ' --version')

let parsed_ver = !v:shell_error && (version_output =~# '\m^JSXHint>') ? syntastic#util#parseVersion(version_output) : []
+"    let parsed_ver = !v:shell_error && (version_output =~# '\m^JSXHint>') ? syntastic#util#parseVersion(version_output) : []

let parsed_ver = [2, 9, 2]
if len(parsed_ver)
call self.setVersion(parsed_ver)
else

Don't know why but bypassing version parsing did solved the problem.

@mannyhappenings That hides the problem rather than solve it. shrug


High CPU usage on start

Hello, I've just started using lsp-wl, and I've discovered that the WolframKernel process is using 100% of one core as soon as VSCode is launched, and pegs it there. I'm running Mathematica 11.3, and the latest pull of lsp-wl.

Hi, could you please send the wlserver.log under the repo folder? and what system are you using?

Closed due to no response for a long time. Hope you have resolved it in a newer version.

Ok, since @megatron0000 has the same issue as yours, I decided to reopen this issue. Although it works fine on Windows, I did reproduce the issue on Linux. Let me confirm whether it happens on MacOSX also. So if you, @kc9jud , see this message, please reply to me with your operating system, thanks.

Ok, I've confirmed that it works fine on MacOSX. I will start a hot-fix based on #7.

Hi @kc9jud and @megatron0000 , I believe that this has been solved in f93fb88ffe82f9de1e3068abec70f77cad508e19, please upgrade to the latest version, install the dependencies and check whether you still meet the problem.


Event series: design page for event lifecycle

Type
✨ Feature
Value

Makes it clear whether an event in a series is upcoming or past
Makes it clear when there are no upcoming events in this series
Gives an onward journey to event outputs, e.g. podcast

Recommendations

List fixed set of past events to provide impression of the event series.
Display all upcoming events and display 'No upcoming events' messaging if there are no upcoming events.
Include a link to output e.g. Podcast for Packed Lunch series

https://www.notion.so/wellcometrust/Events_Past-Events-cc2ff44298dd471dab4bbba7d74ac6f1
To Do

[x] Get copy for 'no upcoming events' (Jen S.) #2355
[x] Follow up with @silveroliver on how we want to reflect event outputs in the domain model
[x] Follow up with the events team regarding the branding and logo (Saturday Studio, Packed Lunch, etc.)
[x] Sketch wireframe @Heesoomoon

To Do Visual Design

[x] Follow up with the brand team about background pattern
[x] Update the template to match the design on Event schedule: background and page grid #2357
[ ] Create promo link for 'What's on' and 'Podcast' #2356
[x] Create promo for past events (include as part of Promo system discovery) #2354

@Heesoomoon I propose we close this because of the work happening on headers, unless you have a good reason to keep it open?

Sounds like a plan.


Update styles.css

Fix wonky stacking context

Thank you for your PR, someone threw in a media query though.
Happy coding!


Migrate off tumblrJS

tumblrJS is the official client library from Tumblr.  However, it looks like they are no longer actively investing or maintaining their library.  The library still depends on requestJS even after its deprecation years ago.  It is currently the only library that is causing deprecation and vulnerability warnings during our "npm install" step.
It also adds 7sec to the startup time of the app which is 33% of the load time for api.js (21.4s) and about 14.6% of the overall module load time (48.5s as measured by require-times).
We can replace the out of date library with some REST calls using Axios in our examples: https://www.tumblr.com/docs/en/api/v2

Ref: I created https://github.com/tumblr/tumblr.js/issues/91 in March 2020, over three years ago.
Other folks mentioning the lack of maintenance:
https://github.com/tumblr/tumblr.js/issues/96
https://github.com/tumblr/tumblr.js/issues/144
https://github.com/tumblr/tumblr.js/issues/101

Fixed by https://github.com/sahat/hackathon-starter/commit/b2a5a7dc630afdb14aadc33c81551d8d5765a4e3

I hope to have a new release of tumblr.js soon and it could be brought back then 🙂

@sirreal thanks for looking into it, but I am concerned about your team's resourcing and prioritization for maintenance of the package in the long run.  I am sure you and your colleagues are good developers, but it feels like there is a resourcing or prioritization issue from your management for monitoring and maintenance of the package.  What I like to see is:

A solid year+ trend and history of monitoring, patching, bug fixes, addressing git hub issues and pull request with a reasonable SLA.
Addition of the package and its dependency hygiene to tumblr's bug bounty program.  The hygiene may need to have a reasonable SLA, like if we don't fix something within 30 days and you report it to the bounty program ...
Addressing the performance issues.  I don't want to add 7 sec to the apps startup time when adding a single line const tumblr = require('tumblr.js'); .  The 7 sec was 15% of the require load times in hackathon starter during my last profiling.  The app loads up 15% now without the dependency. The startup time does matter when a dev has to restart the app over and over as they build feature, bug fix their app.

Ref/background: my and other git hub issues at https://github.com/tumblr/tumblr.js/issues

Tumblr.js v4 has been released, and I hope you see it's addressed the issues you've outlined.

Addressing the performance issues. I don't want to add 7 sec to the apps startup time when adding a single line const tumblr = require('tumblr.js'); .

 You should see it's no longer a problem with v4:
time node -e 'console.log(require("tumblr.js").Client.version)'
# 4.0.0
# 0.04s user 0.01s system 101% cpu 0.053 total

I'd be happy to contribute a PR bringing this up to date with tumblr.js v4.


Remove auto version update

Problem
The release process has auto version update code which updates and sets the version of next release, followed by committing and pushing the changes. Next version number cannot be decided at the time of current release.
Solution
Reverted automatic setting of next version.
Type of Change

[X] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update
[ ] Infrastructure change (CI configs, etc)
[ ] Non-code change (docs, etc)
[ ] None of the above: (explain here)

Test Plan
NA

I'm a little confused why we're downgrading the version in the version.sbt file as it doesn't appear like there were any workflows or anything that bumped this. What's the reasoning for that?
Is it related to the existing tag issue when running the release that you mentioned on Slack?

I tried releasing last time but the tag existed and caused the release to fail. But in the release process, it created a commit of updating the version to 0.1.3 and pushed the change. The change was merged as well since there are no merging rules currently setup.


Change the default for cascading

Is it possible to change the default cascading?
I saw #4213 that you changed the default to be cascade delete for all relationship.
I want the exact opposite, no cascading unless I override it, is it possible to customize it?
(BTW, our DBA, a Microsoft SQL Server MVP told me that almost no one is using cascade delete for various reasons.)

Drop this code in the start of your OnModelCreating to remove cascade from any FKs that got it by convention.
foreach (var entity in modelBuilder.Model.GetEntityTypes())
{
    foreach (var fk in entity.GetForeignKeys())
    {
        if (fk.DeleteBehavior == DeleteBehavior.Cascade)
        {
            fk.DeleteBehavior = DeleteBehavior.Restrict;
        }
    }
}

Or if you want to look cooler, you can write it like this.
modelBuilder.Model
    .GetEntityTypes()
    .SelectMany(e => e.GetForeignKeys()
    .Where(f => f.DeleteBehavior == DeleteBehavior.Cascade))
    .ToList()
    .ForEach(fk => fk.DeleteBehavior = DeleteBehavior.Restrict);

@rowanmiller is there a discussion thread about the reason(s) motivating the Cascade default? I think this is unwanted and potentially dangerous in most case but I guess I'm missing something.

It's the same behavior we've always had in EF - required relationships cascade by default, optional ones do not.


Support COSE detached content

This PR makes t_cose supporting "detached content", mentioned in Issue #33 ,

introduces new options T_COSE_OPT_DETACHED_CONTENT when signing and T_COSE_OPT_ALLOW_DETACHED_CONTENT when verifying in the first commit "add: options for detached content"
adds condition-branch to support "detached content" when signing and verifying, and modifies the rile of  t_cose_sign1_encode_parameters() and t_cose_sign1_encode_signature() in the second commit "change: support detached content" (see below in detail)
modifies existing test codes due to the second commit, and adds new test code short_circuit_self_detached_content_test() in the third commit "add&change: test for detached content"

Please squash when merging, because 2nd and 3rd commits are inseparable.
Changes in logic and interface when signing
Before
enum t_cose_err_t t_cose_sign1_encode_parameters(struct t_cose_sign1_sign_ctx *me,
                                                 QCBOREncodeContext           *cbor_encode_ctx)
{
    ...
    QCBOREncode_BstrWrap(cbor_encode_ctx);
}

enum t_cose_err_t t_cose_sign1_encode_signature(struct t_cose_sign1_sign_ctx *me
                                                QCBOREncodeContext           *cbor_encode_ctx)
{
    QCBOREncode_CloseBstrWrap2(cbor_encode_ctx, false, &signed_payload);
    ...
}

enum t_cose_err_t t_cose_sign1_sign()
{
    result = t_cose_sign1_encode_parameters(me, &encode_context);
    QCBOREncode_AddEncoded(&encode_context, payload);
    result = t_cose_sign1_encode_signature(me, &encode_context);
}

After
enum t_cose_err_t t_cose_sign1_encode_parameters(struct t_cose_sign1_sign_ctx *me,
                                                 QCBOREncodeContext           *cbor_encode_ctx)
{
    ...
    //QCBOREncode_BstrWrap(cbor_encode_ctx); /* moved to t_cose_sign1_sign() */
}

/* caused an !INTERFACE CHANGE! because I moved out the QCBOREncode_CloseBstrWrap2 */
enum t_cose_err_t t_cose_sign1_encode_signature(struct t_cose_sign1_sign_ctx *me
                                                QCBOREncodeContext           *cbor_encode_ctx,
                                                struct q_useful_buf_c         signed_payload) {
    //QCBOREncode_CloseBstrWrap2(cbor_encode_ctx, false, &signed_payload); /* moved to t_cose_sign1_sign() */
    ...
}

enum t_cose_err_t t_cose_sign1_sign() {
    ...
    result = t_cose_sign1_encode_parameters(me, &encode_context);

    if(me->option_flags & T_COSE_OPT_DETACHED_CONTENT) {
        /* add only nil (F6) */
        QCBOREncode_AddNULL(&encode_context);
    } else {
        QCBOREncode_BstrWrap(&encode_context); /* moved from t_cose_sign1_encode_parameters() */
        QCBOREncode_AddEncoded(&encode_context, payload); /* add the target payload */
        QCBOREncode_CloseBstrWrap2(&encode_context, false, &signed_payload); /* moved from t_cose_sign1_encode_signature() */
    }

    /* caused an !INTERFACE CHANGE! because I moved out the QCBOREncode_CloseBstrWrap2 */
    result = t_cose_sign1_encode_signature(me, &encode_context, signed_payload);
    ...
}

Change in logic when vefirying
Before
enum t_cose_err_t t_cose_sign1_verify(struct t_cose_sign1_verify_ctx *me,
                                      struct q_useful_buf_c           cose_sign1,
                                      struct q_useful_buf_c          *payload,
                                      struct t_cose_parameters       *returned_parameters)
{
    ...
    /* --- The payload --- */
    QCBORDecode_GetByteString(&decode_context, payload);
    ...
}

After
enum t_cose_err_t t_cose_sign1_verify(struct t_cose_sign1_verify_ctx *me,
                                      struct q_useful_buf_c           cose_sign1,
                                      struct q_useful_buf_c          *payload,
                                      struct t_cose_parameters       *returned_parameters)
{
    ...
    /* --- The payload --- */
    if(me->option_flags & T_COSE_OPT_ALLOW_DETACHED_CONTENT) {
        QCBORDecode_GetNext(&decode_context, &tmp_payload);
        if(tmp_payload.uDataType == QCBOR_TYPE_NULL) {
            /* "detached content": no need to overwrite the payload */
        } else if (tmp_payload.uDataType == QCBOR_TYPE_BYTE_STRING) {
            /* not "detached content": need to overwrite the payload */
            payload->ptr = tmp_payload.val.string.ptr;
            payload->len = tmp_payload.val.string.len;
        } else {
            /* ERROR! */
        }
    } else {
        QCBORDecode_GetByteString(&decode_context, payload);
    }
    ...
}

Hi Ken,
I'd like the interface to be different than you have it, partly to not break backwards compatibility and partly for design reasons. The interface change and backwards compatibility issue here is actually pretty big. It would require a lot of explaining for the many people that use t_cose now. Also, the new interface has an in/out parameter that doesn't respect const correctly.
On the encode side, I'd like there to be a new method that looks like this:
enum t_cose_err_t
t_cose_sign1_sign_detached(struct t_cose_sign1_sign_ctx *context,
                  struct q_useful_buf_c         detatched_payload,
                  struct q_useful_buf           out_buf,
                  struct q_useful_buf_c        *result);

The existing public interface of t_cose_sign1_sign(), t_cose_sign1_encode_parameters() and t_cose_sign1_encode_signature() would be the same.
Similarly there would be a new method
enum t_cose_err_t t_cose_sign1_verify_detatched(struct t_cose_sign1_verify_ctx *context,
                                      struct q_useful_buf_c           sign1,
                                      struct q_useful_buf_c          detached_payload,
                                      struct t_cose_parameters       *parameters);

and t_cose_sign1_verify() would be the same and backwards compatible.
T_COSE_OPT_ALLOW_DETACHED_CONTENT would be eliminated.
My first thought on how to implement this would be to rename the some existing functions and make them semi-private interfaces, t_cose_sign1_encode_parameters_internal() and t_cose_sign1_encode_signature_internal() that have an additional parameter or two to control how they behave for regular versus detached content. Then I'd wrap them with inline functions t_cose_sign1_encode_signature() and t_cose_sign1_encode_parameters() that select the non-detached behavior and provide backwards compatibility. t_cose_sign1_sign_detached() would invoke them for detached behavior.  Then do similar for t_cose_sign1_verify().
If you don't want to do this work, I will do it, but it will take me about a week as I have other things I have to work on too. I'd prefer if you did it (since I have other things to do). Thx :-)
In general, thanks for this. I'm happy that t_cose is getting to support detached content.
LL

I agree with you. I've misunderstood that the interfaces that I changed are almost private. Let me try to rewrite the commits. Maybe it takes also several days.
Thank you for your comments and the great library!

While we are at it, it seems good to add support for AAD which is very similar to detached content and which turns out to be easy to implement. I have a first PR here #35. I used inline implementations for backward compatibility. Will clean it up some and probably merge in the next few days.
So I think one more parameter should be added:
t_cose_sign1_sign_detached(struct t_cose_sign1_sign_ctx *context,
                  struct q_useful_buf_c         detatched_payload,
                  struct q_useful_buf_c         aad,
                  struct q_useful_buf           out_buf,
                  struct q_useful_buf_c        *result);

Same for verify detached.
No rush from my side. Let me know if you need something.
Thanks!

I was too late but now it can sign/verify the COSE binary with detached content.
Due to the conflict between the new functionality AAD, some functions such as t_cose_sign1_sign() and t_cose_sign1_verify() become ugly, but they remain backward compatible.
Can I improve my implementation?

I posted 2 comments with another account.
hisanak is my private one :)

I'm working on some comments. Will take a few days. Thanks again!


Fixed current location button not working on the first time the map is loaded

When the map is loaded the first time, the current location button is not working, until the device is rotated:

Looked into the code, noticed that in this PR https://github.com/iTwin/mobile-sdk-android/pull/97/files
in the trackPosition method 1 was changed to 0:

Not sure why it was done, maybe by mistake, but changing it back to 1 makes the current location work on the first time the Map is loaded in  android.

The PR where the change was made was 97, not 95: https://github.com/iTwin/mobile-sdk-android/pull/97/files. And @ValdasSorys did that in a bug fix PR to fix crashing. While I agree that 1 is the correct value here, I'd like to know if Valdas remembers why that specific change was made. If changing it back to 1 reintroduces intermittent crashes, that would be bad. (To be clear, I don't think it will.)

The crashes happened only because of a concurrency error which was fixed by using a lock. I didn't do that change intenionally (it.size == 0), seems like I accidentally changed it while refactoring.

@cezaris13 I'm guessing you want me to backport this so that it can make it into an iTwin 4-based release. Is this correct?

Yes, didn't see branches for different iTwin versions.

@Mergifyio backport release/itwin-4.x

@tcobbs-bentley do you know, when the next version of the mobile-sdk-android will be released?


Feature: faster refresh for argocd apps using gitea

Have you searched for this feature request?

[X] I searched but did not find similar requests

Problem Statement
idpbuilder uses the default config for argocd refresh timeout 3 minutes. This is too much time for the local development lifecycle loop, specially when running idpbuilder with --no-exit watch mode.
The sync of the manifest is fast from local folder to gitea in seconds, but then the argocd needs to be force the refresh manually if you don't want to wait the 3 minutes everytime to verify the change.
Possible Solution
There are a few possible solutions

change the default timeout of 3 minutes to something lower in the seconds time range
every time idpbuilder reconciles a change on git to refresh the argocd via adding refresh annotation argocd.argoproj.io/refresh for application and argocd.argoproj.io/application-set-refresh for applicationsets the docs states that it will refresh the appset it might not refresh the apps being generated.
allow and configure gitea to send a webhook to argocd. I tested this and currently is not possible as the repo URL don't match, the webhook sends https://gitea.cnoe.localtest.me:8443/giteaAdmin/idpbuilder-localdev-my-app-manifests but argocd app is configure with different one http://my-gitea-http.gitea.svc.cluster.local:3000/giteaAdmin/idpbuilder-localdev-my-app-manifests.git It might be possible to get this working if are able to have argocd use a valid https with a valid ssl cert for gitea something we are considering using cert-manager to have all communication to use https with valid certificates and internal cluster DNS and IP addresses.

Alternatives Considered
No response

For solution 3 using webhook gitea is allowed by default to send webhook to external IPs, not internal IPs it needs to be configure with ALLOWED_HOST_LIST here is a branch shows how to change  https://github.com/cnoe-io/idpbuilder/compare/main...csantanapr:idpbuilder:gitea-webhook-argocd?expand=1
Here are the docs on how to configure webhook in gitea
https://docs.gitea.com/administration/config-cheat-sheet?_highlight=allowed_hos#webhook-webhook

I am inclined to go with option 2 but we should also allow for 3 if end users choose to do so.
For 2:

We can probably just process them here: https://github.com/cnoe-io/idpbuilder/blob/cb0bd302caeddd0ee06a2733f8064ecaa1243e86/pkg/controllers/localbuild/controller.go#L128. Get argo apps or appsets, if annotations are not present, annotate.
If there is a sync operation already happening, we need to make sure they are either terminated properly  or wait (could take several minutes).
We may also need to be careful not to refresh apps too often. This means we should possibly check commit hash on gitea and what is reconciled by argocd.

For 3:

We need internal DNS to resolve correctly for core package endpoints. https://github.com/cnoe-io/idpbuilder/issues/243#issuecomment-2103879759. Likely we will need to import additional config files for CoreDNS so others can overwrite the main config file safely.
Valid certificate for ingress-nginx. https://github.com/cnoe-io/idpbuilder/issues/293. We likely will need certificate manager.

Option 1 is also a valid approach IMO. We can set it to lower by default to at least lower the feedback loop speed without code changes. I think we should take a PR on this. 90 seconds to start maybe? What do you think? @csantanapr

lets make it 60s that's the default for some k8s controllers


Ambient: fix incorrect dualstack EDS with tunnelling

Today, we set an address pointing to the internal listener, and an additionalAddress pointing to the raw IP. The raw IP is never used.
Per https://github.com/envoyproxy/envoy/issues/36318 - we cannot actually set the additional address with the internal listener.
In order to properly test this I also had to fix https://github.com/istio/istio/issues/53333

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

added tests. This will be the basis of testing for https://github.com/istio/istio/pull/53194 as well

@howardjohn: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

lint_istio
2db568471286841e190b09fc79e03378faafc671
link
true
/test lint

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.


create solr_index via REST-endpoint & update documentation (was: The solr_index field primary key "id" is not auto incrementing)

The solr_index field has the primary key id, however it is not auto incrementing.  When you follow the instructions in the README, you don't insert an id, which causes SMUI ui to report an error.
Changing
INSERT INTO solr_index (name, description) VALUES ('core_name1', 'Solr Search Index/Core');

to
INSERT INTO solr_index (id, name, description) VALUES (1, 'core_name1', 'Solr Search Index/Core');

fixes it.  However, should the ID field be autoincrementing?

See PR #20 for the fix!

thanks @epugh ,
to make SMUI compatible to standard SQL, I removed auto_increment and made ids GUIDs. SMUI is used on top of several DBMS installations rights now (HSQL, mariaDB / mySQL, Postgres), so the id definition should be fine as is.
But I'll definitively check the documentation here and will provide a REST-endpoint to create the solr_index entities, so that a proper interface is provided here.

as discussed in #20 REST interfaces exist.


Remove -mpower8-* as they are enabled by default on gcc and unrecognized by clang

Checklist

[ ] Used a fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[ ] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@conda-forge-admin, rerender


Install error(s)

As per your install instructions:
python -m pip install basemap throws an error with Python 3.9.13, and Clang 13.1.6 (clang-13<IP_ADDRESS>.5) on darwin with GEOS installed via homebrew.

src/_geoslib.c:745:10: fatal error: 'geos_c.h' file not found
#include "geos_c.h"
^~~~~~~~~~
220 warnings and 1 error generated.
error: command '/usr/bin/clang' failed with exit code 1

As a matter of fact, every single one of your installation instructions throws some error.

Hi @ktavabi. I have a couple of questions:

What value did you assign to the environment variable GEOS_DIR?
Where is the path to the GEOS installation through Homebrew?

Unfortunately, without more feedback there is not much more that I can do to help. The piece of traceback is typical from not setting GEOS_DIR to the appropriate folder. If that is not the case, feel free to re-open this issue.


Feature Reuqest: 3-way merge

Would you like to add a 3-way merge feature?

Can you provide more details on what you are looking for?

Released in 2.1.0


Proxy Support

Is this possible?, i believe almost all IPs from AWS servers is banned/limited by facebook, therefore your account will be banned or checkpoint.
One of many solution for this is to add proxy support.
sry for asking 2much, thanks.

It's okay that you ask. However, we are not planning to implement proxy support in the near future. Additionally, this API is not fully stable, because Facebook frequently changes its HTTP responses. (if you really want it to be implemented, do not hesitate to submit a pull request :-) )


something (google form?) prevents closing training tabs

If I just open a page, scroll a bit and then try to close it, I get an alert. This can become very intrusive very quickly I fear.

edit: this is from https://galaxyproject.github.io/training-material/topics/admin/tutorials/connect-to-compute-cluster/tutorial.html but it happens on other pages with the form too

Reopening this so I can close it finally :)


[BUG]Add additionalPrinterColumn to CRDs

Check https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#additional-printer-columns
As the first step, we need to define what fields should be exposed.
It might impact the upgrade, depends on if CRDs allow editing.

links seems to be no longer valid:
below might contain additional information about the additionalPrinterColumns for the crd spec.
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#customresourcedefinitionspec-v1beta1-apiextensions-k8s-io
https://kubernetes.io/docs/reference/using-api/api-concepts/#receiving-resources-as-tables

Thanks @joshimoo , link updated.

reproduce steps/test steps:
For each of the case below:

Fresh installation of Longhorn.
Upgrade from older version

Run:
kubectl get <LONGHORN-CRD> -n longhorn-system

Verify that the output contains information as specify in the additionalPrinerColumns at here

[leave a note for later]: we may consider automating the generation of CRDs yaml files as in https://book.kubebuilder.io/reference/generating-crd.html

Verified with longhorn master - 10/01/2020
Validation - Pass
The 'AdditionalPrinterColumn' are appearing for crds from https://book.kubebuilder.io/reference/generating-crd.html
e.g.


Vulnerability roundup 114: lua-5.4.3: 1 advisory [9.1]

search, files

[ ] CVE-2022-28805 CVSSv3=9.1 (nixos-21.11, nixos-unstable)

CVE details
CVE-2022-28805
singlevar in lparser.c in Lua through 5.4.4 lacks a certain luaK_exp2anyregup call, leading to a heap-based buffer over-read that might affect a system that compiles untrusted Lua code.

Scanned versions: nixos-21.11: 3c5ae9be1f1; nixos-unstable: f419dc5763c.

Addressed in #174094 & backported in #175581


Update KeepingYouAwake to latest release

1.3.1 has been released https://github.com/newmarcel/KeepingYouAwake/releases/tag/1.3.1

Thank you for the contribution. It was merged directly as 8c06f3711f09b4e7df26f42436a6b45f6b8c0d77 to keep commit history cleaner. Your contribution is still credited to you.

Ah, I had probably not pulled before committing. Thanks for the rebase and the merge!

No, you did everything fine. This is just so we don’t get the useless merge commits introduced by using github’s web interface, which make the history harder to see.


yarn run oi has cross-process dependencies

It is not possible to execute two processes of yarn run oi in parallel.
This is best reproduced by:

Step 1) Open a Console A, run yarn run oi help

This outputs
[...]
2024-12-20 17:23:27 [info]: Initialized plugin openibex.protocols.
2024-12-20 17:23:27 [info]: Initialized plugin openibex.usd-circle.
2024-12-20 17:23:27 [warn]: Plugins directory does not exist: commands
2024-12-20 17:23:27 [info]: Registered CLI command: init
2024-12-20 17:23:27 [info]: Registered CLI command: start
2024-12-20 17:23:27 [info]: Registered CLI command: protocol
2024-12-20 17:23:27 [info]: Registered CLI command: connect
2024-12-20 17:23:27 [info]: Registered CLI command: exec
2024-12-20 17:23:27 [info]: Registered CLI command: value
2024-12-20 17:23:27 [info]: Registered CLI command: watch
Usage: @openibex/cli [options] [command]

A CLI for OpenIbex applications

Options:
  -V, --version                                       output the version number
  --import <package>                                  Dynamically import a package from a path before executing commands
  --config <path>                                     Specify a custom configuration file (default: "config.yaml")
  --logdest <path>                                    Specify the log destination file (default: "logs/app.log")
  --help                                              Display help for the CLI

Commands:
  init [options]                                      Initialize a new app by creating a new core database.
  start [options]                                     Start a runner
  protocol [options] <name>                           Work with protocols
  connect [options] <artifact>                        Use connectors to retrieve smart contract events.
  exec [options] <artifact> <functionName> [args...]  Execute smart contract functions
  value [options] <action> <valname> [value]          Set, get and delete keys from the core key-value database.
  watch [options] <command>                           Watch executes the same command every n seconds or n blocks.
  help [command]                                      display help for command
Completed running 'packages/cli/src/index.ts help'

*Step 2) Open Console B *

Run  yarn run oi connect eip155:31337/erc20:0x5FbDB2315678afecb367f032d93F642f64180aa3
This starts an endline loop

*Step 3) Go back to Console A, run the same yarn run oi help again
Expected: Same output as in STep 1
Observed:
the following output:
 yarn run oi help                                                                                :(
yarn run v1.22.22
$ tsx --watch packages/cli/src/index.ts help
2024-12-20 17:26:57 [info]: Configuration loaded successfully.
2024-12-20 17:26:57 [info]: Starting Helia IPFS Server...
2024-12-20 17:26:57 [error]: Core not initialized. Run init and add the new CoreDB address to the config.
2024-12-20 17:26:57 [warn]: Plugins directory does not exist: commands
2024-12-20 17:26:57 [info]: Registered CLI command: init
Usage: @openibex/cli [options] [command]

A CLI for OpenIbex applications

Options:
  -V, --version       output the version number
  --import <package>  Dynamically import a package from a path before executing commands
  --config <path>     Specify a custom configuration file (default: "config.yaml")
  --logdest <path>    Specify the log destination file (default: "logs/app.log")
  --help              Display help for the CLI

Commands:
  init [options]      Initialize a new app by creating a new core database.
  help [command]      display help for command

It claims that core is not initialized. Presumably, this is because Helia-DB is locked as it is used by another process:
2024-12-20 17:26:57 [error]: Core not initialized. Run init and add the new CoreDB address to the config.
If you kill the process in Console B, it works again as intended.

We shall either print an Error saying that there is another process already connected to Helia-DB, but I rather think that multiple process should be able to connect to the same DB ?