Add GCC to runtime dependencies

Ruby 2.6.0 ships with a JIT (just in time compiler) that utilizes GCC (or any other cc such as clang) at runtime. This version of Ruby will be released on December 25th, 2018 and Heroku needs to provide support for this feature out of the box.
Background
The design alternatives can be found in this document
https://docs.google.com/document/d/1q5yEpRddUU2JPynESmffI_BpLDW4XaBgO7iyJE-mwio/edit?usp=sharing
This PR
This PR adds the gcc package to Heroku-16 and Heroku-18 runtime images.
Size cost
Adding the gcc package increases the runtime stack image by about 84mb:
Before adding gcc
-----> Size breakdown...
# ...
       ubuntu:18.04                      85.8MB
       heroku/heroku:18                  428MB
       heroku/heroku:18-build            805MB

After adding gcc
-----> Size breakdown...
# ...
       ubuntu:18.04                      85.8MB
       heroku/heroku:18                  512MB
       heroku/heroku:18-build            805MB

It looks like this works with only gcc, however if we end up needing build-essential then the cost would increase by 64MB to 576MB.
Testing JIT on the stack images testing
On a currently running Heroku-18 dyno with Ruby 2.6:
$ hs run bash -a infinite-everglades-58502
~ $ ruby -v
ruby 2.6.0preview3 (2018-11-06 trunk 65578) [x86_64-linux]
~ $ ruby -e "def a; end; def b; end; a; a; a; a; a; a; a;  puts 'done'" --jit --jit-verbose=1
MJIT: Error in execv: /usr/bin/gcc
MJIT warning: Making precompiled header failed on compilation. Stopping MJIT worker...
done

Note: The warnings in the output indicate gcc could not be found

On a local docker image with gcc available:
$ bin/build.sh heroku-18 heroku/heroku:18 heroku/heroku:18-build
# ...
$ docker run -i -t heroku/heroku:18 /bin/bash
root@4437f834d502:/# mkdir -p vendor/ruby-2.6.0
root@4437f834d502:/# cd vendor/ruby-2.6.0
root@4437f834d502:/vendor/ruby-2.6.0# curl https://heroku-buildpack-ruby.s3.amazonaws.com/heroku-18/ruby-2.6.0.tgz -s -o - | tar zxf -
root@4437f834d502:/vendor/ruby-2.6.0# bin/ruby -e "def a; end; def b; end; a; a; a; a; a; a; a;  puts 'done'" --jit --jit-verbose=1
done
Successful MJIT finish

Note: The output indicates that MJIT was successful

For more output you can also apply --jit-wait which will force mjit to run synchronously (instead of in a background thread).
cc/ @hone @hunterloftis

Also looks like you can get a return status code by trying to interact with MJIT internals if it's not active. On current Heroku 18 dyno:
$ hs run bash -a infinite-everglades-58502
~ $ ruby --jit -e "RubyVM::MJIT.pause(wait: false)" ; echo $?
Traceback (most recent call last):
	1: from -e:1:in `<main>'
-e:1:in `pause': MJIT is not enabled (RuntimeError)
1

For the record, @schneems and I have been discussing possibly using musl instead of GCC to reduce impact.
Richard has been poking the MJIT maintainers about that idea.

If you want to use musl for MJIT, of course ruby binary should be linked against musl as well. It may have difference in that layer (can musl's malloc be configured with MALLOC_ARENA_MAX? is it fast and memory efficient? etc). I would use Optcarrot to measure pure-VM performance and to compare it with JIT.

musl is not currently tested and the maintainer is worried about differences in performance and configuration
also Ruby would need to be recompiled with the same cc that mjit uses, so we would need to re-tool to build 2.6.0+ with musl.

Our thinking now is therefore that the slug size reduction isn't worth the fairly large effort it would be to switch to musl instead of gcc.

Thanks 🙏 ! When will this be released so I can test and verify everything works in prod?

Next week.

I shipped it this morning. https://devcenter.heroku.com/changelog-items/1531

Confirm that it works! Thanks ❤
~ $ ruby --jit -e "RubyVM::MJIT.pause(wait: false)" ; echo $?
0


Fix placeholder not being shown for select2

The placeholder is currently not being shown for the single-selection select2 theme. This pull request adds the missing placeholder attribute and tests that the placeholder is present on all themes.

Nevermind, this just duplicates the empty option text as a placeholder for the search text box in the select2 theme, which is not very helpful to be honest. Because of this, I'm closing this pull-request.
I would love for there to be a way to specify the search box's placeholder as a separate option for the select2 theme. Maybe that's a feature request instead.

I think this is actually a good patch. I use the select2 theme in my product but due to this placeholder problem, I have to overwrite the theme in my script.


PropertyGrid does not update when using keyboard to select ListView item

In SimpleDomEditor, if you add multiple Events to the XML and use the keyboard to change your selection, the property grid does not reflect that change. What is the ideal way to update the property grid? I'm building a tool based on this sample.

Thanks for reporting the bug.
It seems that the bug is in the class ListViewAdapter located at:  ..\Framework\Atf.Gui.WinForms\Applications\Listers
Look at the constructor istViewAdapter(ListView listView)
listView.SelectedIndexChanged  is not wired.
The above event should be wired and properly handled.
I will make a fix with next ATF update.
meanwhile you can fix it locally until next ATF update.
Alan

fixed,
Thanks for reporting the bug.
Alan


Add Javadoc, formId field, and move db impl (#19)

Move didn't keep diffs. I'll split this into two PRs.


[Enhancement] KCL needs to have API to list variables and attributes (including their name and type)

Enhancement
The KCL IDE extension needs many kinds of semantic information of the KCL files to provide visual and interactive support for users to modify their configurations quickly, without directly writing any code.

https://github.com/KusionStack/vscode-kusion/issues/10 describes image auto-update scenarios which requires listVariable and overrideFile API

https://github.com/KusionStack/vscode-kcl/issues/12 describes low-code KCL init/modify/view scenarios which requires listAttribute API

Related Issue: #174


Add missing

Add missing </disables>
Signed-off-by: Jason Feng<EMAIL_ADDRESS>

Thanks for fixing @JasonFengJ9 !  My bad for not running Grinders on my PR.

@smlambert The same thing happened to me here, I did run a grinder at another PR https://github.com/adoptium/aqa-tests/pull/5455 :)


Chris

Can you merge this one to add me into the project? This simply changes 2 lines in the readme

lgtm


[FLINK-31521][jdbc-driver] Initialize jdbc driver module in flink-table

What is the purpose of the change
This PR aims to initialize jdbc driver module in flink-table
Brief change log
Add flink-sql-jdbc-driver module in flink-table
Verifying this change
This change is already covered by existing tests, such as (please describe tests).
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no) no
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no) no
The serializers: (yes / no / don't know) no
The runtime per-record code paths (performance sensitive): (yes / no / don't know) no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) no
The S3 file system connector: (yes / no / don't know) no

Documentation

Does this pull request introduce a new feature? (yes / no) yes
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) not applicable

Thanks @libenchao Updated


Changed a word to its correct context.

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the master branch of freeCodeCamp.
[x] None of my changes are plagiarized from another source without proper attribution.
[x] My article does not contain shortened URLs or affiliate links.

If your pull request closes a GitHub issue, replace the XXXXX below with the issue number.
Closes #XXXXX

Closed and Reopened this PR to attempt to resolve a specific Travis build failure.

@JimmyJ4zz Thank you for  creating this PR.  However, since it was created, the underlying document has changed.  Can you please review and resolve the current merge conflicts?  It could be the changes to the underlying document are so significant that your changes are no longer relevant.  If the latter is the case, it might be easier to just close this PR and create a new PR with any changes you still feel are important to make.
If you need any assistance or have questions about resolving the merge conflicts, please reply back or contact someone in the Contributors Room on Gitter.
Thank you.


Provide an option for camel casing SQL Keywords (aduxr)

Issue Type: Feature Request
You can often create SQL code that vaguely resembles a ransom note. If your style for writing  SQL keywords does not match what the upper case style used by code completion. Code completion writes SQL keywords in all  upper case (SELECT), but users write these keywords in all lower case (select), title case (Select), or camel case (OpenRowset).
Let users choose how they want their SQL keywords written.

Azure Data Studio version: Azure Data Studio 1.5.2 (f74080c96310354bc92211f1826c0fdce78ca2f6, 2019-03-22T06:09:10.602Z)
OS version: Darwin x64 18.5.0

@gengelbms there is an setting named Mssql > Format: Keyword Casing that controls this behavior.  Currently we only support UPPERCASE and lowercase but the other options seem reasonable as well.

Thank you. Most of the participants I've seen seem to prefer lowercase, so having the setting is great. I have seen several camel-case user, so that option would be nice have. I would guess that current settings give us about 90% coverage.

@gengelbms there is an setting named Mssql > Format: Keyword Casing that controls this behavior. Currently we only support UPPERCASE and lowercase but the other options seem reasonable as well.

I changed this setting to lowercase, but the code completion list still provides me only uppercase items even after restarting the application.

@kazinad are you using Postgres extension?  This option is currently only supported for MSSQL.

@kazinad are you using Postgres extension? This option is currently only supported for MSSQL.

Yes I use MSSQL, but there are 3 options related to casing in settings, and I changed possibly the wrong one. Thx


Climate errors with Home Assistant

With latest HA version 0.99.x and latest CCU Firmware <IP_ADDRESS> (in my case a virtualized Debmatic VM) I get these errors with climate devices:
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 408, in _async_add_entity
    await entity.async_update_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 266, in async_update_ha_state
    self._async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 300, in _async_write_ha_state
    state = self.state
  File "/usr/src/homeassistant/homeassistant/components/climate/__init__.py", line 175, in state
    return self.hvac_mode
  File "/usr/src/homeassistant/homeassistant/components/homematic/climate.py", line 69, in hvac_mode
    if self.target_temperature <= self._hmdevice.OFF_VALUE + 0.5:
TypeError: '<=' not supported between instances of 'str' and 'float'
2019-09-10 16:16:36 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 408, in _async_add_entity
    await entity.async_update_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 266, in async_update_ha_state
    self._async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 300, in _async_write_ha_state
    state = self.state
  File "/usr/src/homeassistant/homeassistant/components/climate/__init__.py", line 175, in state
    return self.hvac_mode
  File "/usr/src/homeassistant/homeassistant/components/homematic/climate.py", line 69, in hvac_mode
    if self.target_temperature <= self._hmdevice.OFF_VALUE + 0.5:
TypeError: '<=' not supported between instances of 'str' and 'float'
2019-09-10 16:16:36 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 408, in _async_add_entity
    await entity.async_update_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 266, in async_update_ha_state
    self._async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 300, in _async_write_ha_state
    state = self.state
  File "/usr/src/homeassistant/homeassistant/components/climate/__init__.py", line 175, in state
    return self.hvac_mode
  File "/usr/src/homeassistant/homeassistant/components/homematic/climate.py", line 69, in hvac_mode
    if self.target_temperature <= self._hmdevice.OFF_VALUE + 0.5:
TypeError: '<=' not supported between instances of 'str' and 'float'

Is this pyhomematic related or HA related or even user related? :)

That's a change that has been made in the Home Assistant component, specifically the climate platform. It has to be fixed there. So the issue you have reopened there is the correct one.


use trait booting rather than overriding the entire boot

for some reason the traits overrode the whole boot instead of just using the trait boots.
This meant that if there was already a boot it would conflict.

Also, this will conflict if the #429 is merged first, but real easy to correct.

Can this get merged in please, it is outstanding and causing issues still locally for me as well as others.

Can we please merge this?
It is really annoying to spend hours debugging a issue to find out that it is once again the entrust trait messing things up.

Any updates guys? I've faced Trait method restore has not been applied, because there are collisions with other trait methods on ...\User


Service Scheduler is not attached to Service.

Describe the bug
It had created Service Scheduler, Template and Query Builder but didn't attached service Scheduler to the Service.
Screenshots

#532 is happening because of #533, both solved


missing option?

MYSERVER BEHIND NAT---> wsTunnelServer running on public server  <---  MY CLIENT BEHIND NAT
to allow any client to connect to "myserver" through a public server running wstunnel
This option is missing in most softwares.
And would allow for example two devices connected using a phone network (3g/4g/5g) to connect to each other services.

Take a look at this issue, it is kind a of the same answer in your case.
https://github.com/erebe/wstunnel/issues/84
You need to combine wstunnel and ssh -R

hmm no.. that does not work. I explain better:
I have a server on a local machine running on PORT XXX
I wish that to be reachable by "remotemachine" port 80
and "remote machine" can ONLY use port 80.

it's a feature missing from wstunnel.
what wstunnel should do is to serve a websocked AND html on the same port.. as "ttyd" does.
in this way it would be easy to map a firewalled port 80 to the same port wstunnel is running.

so, let's say:
on the public reachable server:
wstunnel --server ws://<IP_ADDRESS>:80
on the local machine:
wstunnel --client 8123 ws://publicserveraddress:80
on any other computer on internet:
browsing to http://publicserveraddress  will reflect port 80 of the local machine.

Hello,
If you want to share the same port to multiplex services, you can use a dedicated reverse proxy for that (i.e: nginx)
You can match on the http upgrade path  of websocket connection to proxy pass the request to wstunnel and let other request your webserver.
Take a look at this issue to get hint regarding how to do that https://github.com/erebe/wstunnel/issues/14

I was just suggesting a simple feature for wstunnel... so there is not need for other utilities like nginx

Ha, understood.
Sadly, this feature will not be part of wstunnel. The project aims to stay simple, advanced features require a real reverse proxy


fix: typescript TS2371 error

Encount compilation error TS2371 when build with typescript > 5.2.
TS2371: A parameter initializer is only allowed in a function or constructor implementation.

Oh, thank you. I forgot that function type definitions cannot have default parameters.


Shipping in Development Mode

Any reason you shipping the frontend in dev mode?

This probably explains why it is very slow to POST to www.jsonstore.io?

@peterhellberg I don't think so, the frontend is served static (I hope). POST should be handled dynamic by Proxy / Node.

@hmontazeri That is what I call cutting edge.

So should this issue be closed now?

@peterhellberg it's not fixed so far...

fixed


Flutter: Initialising variables on startup

Steps to Reproduce
See stackoverflow question here: https://stackoverflow.com/questions/50437687/flutter-initialising-variables-on-startup?noredirect=1#comment87889333_50437687
I'm trying to use the values saved in the SharedPreferences to initialise several variables in my app. In Flutter, the SharedPreferences are asynchronous so it results in the variables initialising later on in the code which is creating problems with my app as some of the variables are null when the build method is called.
Here is a small test Flutter app I wrote to demonstrate this issue:
import 'package:flutter/material.dart';
import 'package:flutter/cupertino.dart';
import 'package:path_provider/path_provider.dart';
import 'dart:io';
import 'dart:convert';
import 'package:shared_preferences/shared_preferences.dart';

class TestingApp extends StatefulWidget {

  TestingApp() {}

  @override
  State<StatefulWidget> createState() {
// TODO: implement createState
    return new _CupertinoNavigationState();
  }
}

class _CupertinoNavigationState extends State<TestingApp> {

  int itemNo;

  @override
  void initState() {
    super.initState();
//    SharedPreferences.getInstance().then((sp) {
//      sp.setInt("itemNo", 3);
//    });
    SharedPreferences.getInstance().then((sp)  {
      print("sp " + sp.getInt("itemNo").toString());
      setState(() {
        itemNo = sp.getInt("itemNo");
      });
    });
    print("This is the item number " + itemNo.toString());
  }

  @override
  Widget build(BuildContext context) {
    // TODO: implement build
    print("item number on build " + itemNo.toString());
    return new Text("Hello");
  }
}

This is the result in the console:
Performing full restart...
flutter: This is the item number null
flutter: item number on build null // build method being called and variable is null
Restarted app in 1 993ms.
flutter: sp 3
flutter: item number on build 3

You can see that when I tried to fetch the variable from SharedPreferences on startup, since SharedPreference is async, the itemNo is null. Then the app runs the build method and runs the build method on the itemNo = null which is resulting in crashes in the app.
Once it fetches the value from SharedPreferences, I call setState which then calls the build method again with the correct value. However, the initial call to build with the itemNo = null should not have happened.
I wish there was a synch method for SharedPreferences but it doesn't seem to exist. How do I run the app so that the variables are initialised properly in Flutter on startup?
I tried to solve this through using a synchronous method to initialise my variables by writing to a json file and then reading from it using the following short Flutter test app - to me, this appears to be overkill for saving a variable to initialise but I still gave it a try:
import 'package:flutter/material.dart';
import 'package:flutter/cupertino.dart';
import 'package:path_provider/path_provider.dart';
import 'dart:io';
import 'dart:convert';
import 'package:shared_preferences/shared_preferences.dart';

class TestingApp extends StatefulWidget {

  TestingApp() {}

  @override
  State<StatefulWidget> createState() {
// TODO: implement createState
    return new _CupertinoNavigationState();
  }
}

class _CupertinoNavigationState extends State<TestingApp> {

  int itemNo;
  File  jsonFile;
  String fileName = "items.json";
  Directory dir;
  bool fileExists = false;

void createFile(Map content, Directory dir, String fileName) {
//  print("Creating file for category " + dir.path);
  File file = new File(dir.path + "/" + fileName);
  file.createSync();
  fileExists = true;
  file.writeAsStringSync(json.encode(content));
}

void writeToFile(int itemNo) {
//  print("Writing to category file");
  Map itemMap = new Map();
  itemMap['item'] = itemNo;
  if (fileExists) {
    print("category file exists");
    Map jsonFileContent = json.decode(jsonFile.readAsStringSync());
    jsonFileContent.addAll(itemMap);
    jsonFile.writeAsStringSync(json.encode(itemMap));
  } else {
    print("category File does not exists");
    getApplicationDocumentsDirectory().then((Directory directory) {
      dir = directory;
      createFile(itemMap, dir, fileName);
    });

  }
}

fetchSavedItemNo() {
  //load the currency from the saved json file.
  getApplicationDocumentsDirectory().then((Directory directory) {
    dir = directory;
    jsonFile = new File(dir.path+ "/" + fileName);
    fileExists = jsonFile.existsSync();
    setState(() {
      if (fileExists)
        itemNo = json.decode(jsonFile.readAsStringSync())['item'];
      print("fetching saved itemNo " +itemNo.toString());
      if (itemNo == null) {
        itemNo = 0;
      }
    });

    return itemNo;
    //else the itemNo will just be 0
  });
}

  @override
  void initState() {
    super.initState();
    writeToFile(3);
    setState(() {
      itemNo = fetchSavedItemNo();
    });

  }

  @override
  Widget build(BuildContext context) {
    // TODO: implement build
    print("item number on build " + itemNo.toString());
    return new Text("Hello");
  }
}

I still have the results where the build method is being called before the variables are fully initialised which is causing app crashes.
Performing full restart...
flutter: category File does not exists
flutter: item number on build null   // the build method is called here
Restarted app in 1 894ms.
flutter: fetching saved itemNo 3
flutter: item number on build 3

How do I initialise variables in Flutter on app startup?

I'm running Flutter on Android Studios.
[✓] Flutter (Channel beta, v0.3.2, on Mac OS X 10.13.4 17E202, locale en-ZA)
• Flutter version 0.3.2 at /Users/WingHinChan/Documents/FlutterSDK/flutter
• Framework revision 44b7e7d3f4 (4 weeks ago), 2018-04-20 01:02:44 -0700
• Engine revision 09d05a3891
• Dart version 2.0.0-dev.48.0.flutter-fe606f890b
[✓] Android toolchain - develop for Android devices (Android SDK 27.0.3)
• Android SDK at /Users/WingHinChan/Library/Android/sdk
• Android NDK location not configured (optional; useful for native profiling support)
• Platform android-27, build-tools 27.0.3
• Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
• Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1024-b01)
• All Android licenses accepted.
[✓] iOS toolchain - develop for iOS devices (Xcode 9.3.1)
• Xcode at /Applications/Xcode.app/Contents/Developer
• Xcode 9.3.1, Build version 9E501
• ios-deploy 1.9.2
• CocoaPods version 1.4.0
[✓] Android Studio (version 3.1)
• Android Studio at /Applications/Android Studio.app/Contents
• Flutter plugin version 23.2.2
• Dart plugin version 173.4700
• Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1024-b01)
[!] IntelliJ IDEA Community Edition (version 2016.1.1)
• IntelliJ at /Applications/IntelliJ IDEA CE.app
✗ Flutter plugin not installed; this adds Flutter specific functionality.
✗ Dart plugin not installed; this adds Dart specific functionality.
• For information about installing plugins, see
https://flutter.io/intellij-setup/#installing-the-plugins
✗ This install is older than the minimum recommended version of 2017.1.0.
[!] VS Code (version 1.23.1)
• VS Code at /Applications/Visual Studio Code.app/Contents
• Dart Code extension not installed; install from
https://marketplace.visualstudio.com/items?itemName=Dart-Code.dart-code
[✓] Connected devices (2 available)
• Mom's iPhone 6s Plus • b8b365827eaf268714f5a2c44da4be813e26755e • ios • iOS 11.3.1
• iPhone X             • FE19AC9F-A4B4-4AE7-9D06-9DD21DDFEBAE     • ios • iOS 11.3 (simulator)

The linked SO question has an accepted answer.
I assume this issue is obsolete.


How to mute video programatically?

How can we mute the video programatically or on bootstrap? 😬

Nevermind, just noticed there's a controller.setVolume method. Would expect to mute with a Widget argument.


insert.mode UPSERT not working and same data would be inserted several times

Hi team,
I observed that when I restarted the connect-standalone.sh, the same full dataset from source were written again into the sink HANA.
It looks to me that the upsert mode is not working. I tried making the incrementing column (ID) as and as not PRIMARY KEY, but both scenarios would be the same result.
Moreover, if ID column is set as primary key, then restarting the connect-standalone would cause unique constraints exception and the sink worker would be killed.
The config files for source and sink are as below, the <topic_name>.insert.mode is set as upsert
connect-hana-sink-metadata-incr.properties:
auto.create=true
topics=topic_metadata_incr
topic_metadata_incr.table.name="HDB_METADATA"."SCHEMA_DISK_SIZE_1"
topic_metadata_incr.poll.interval.ms=5000
topic_metadata_incr.insert.mode=upsert
connect-hana-source-metadata-incr.properties:
topics=topic_metadata_incr
topic_metadata_incr.table.name="HDB_METADATA"."SCHEMA_DISK_SIZE"
topic_metadata_incr.incrementing.column.name=ID
topic_metadata_incr.poll.interval.ms=5000
mode=incrementing

At first glance I suspect it's because the ID column is not PRIMARY KEY as the code in AbstractHANAPartitionLoader prepare the UPSERT statement "WITH PRIMARY KEY". However, after setting the column as PK, it still does not work.
https://github.com/SAP/kafka-connect-sap/blob/645ab1ab00ce802a6753f02ba18022633c5862f6/src/main/scala/com/sap/kafka/client/hana/AbstractHANAPartitionLoader.scala#L259

At first glance I suspect it's because the ID column is not PRIMARY KEY as the code in AbstractHANAPartitionLoader prepare the UPSERT statement "WITH PRIMARY KEY". However, after setting the column as PK, it still does not work.
https://github.com/SAP/kafka-connect-sap/blob/645ab1ab00ce802a6753f02ba18022633c5862f6/src/main/scala/com/sap/kafka/client/hana/AbstractHANAPartitionLoader.scala#L259

After updating below config in connect-standalone.properties, truncating the sink table, deleting the topic and rm /tmp/connect.offsets, it seems that after restarting the connect, it can continue to work without re-reading all the data in the source table.
consumer.auto.offset.reset=latest
However, it can not explain why the UPSERT mode did not work.

After updating below config in connect-standalone.properties, truncating the sink table, deleting the topic and rm /tmp/connect.offsets, it seems that after restarting the connect, it can continue to work without re-reading all the data in the source table.
consumer.auto.offset.reset=latest
However, it can not explain why the UPSERT mode did not work.

@twonly Did you recreate the table after fixing the configuration? And how is your pk.fields and pk.mode are configured?
The incrementing column at the source should be used for the incrementing purpose (like using just an incremental index or timestamp) so that the source connector can pull only the newer records at each poll. For upsert at the sink connector, you should have another column that uniquely identifies the record so that the upsert will overwrite this record. I don't know what is not working. Could you describe the problem with some small concrete example? Thanks.

@twonly Did you recreate the table after fixing the configuration? And how is your pk.fields and pk.mode are configured?
The incrementing column at the source should be used for the incrementing purpose (like using just an incremental index or timestamp) so that the source connector can pull only the newer records at each poll. For upsert at the sink connector, you should have another column that uniquely identifies the record so that the upsert will overwrite this record. I don't know what is not working. Could you describe the problem with some small concrete example? Thanks.

@elakito After fixing the configuration (adding consumer.auto.offset.reset=latest), I truncated the sink table and left the source table as it was (no recreation, no truncation...), and since then, things look as expected, no matter I restarted the connect (both source and sink , only either of them), the sink can poll the incrementing data from source. However, from the HANA SQL Cache, I can only see INSERT but no UPSERT statements.
The source table has an incrementing column ID - BIGINT type, which is also defined as primary key and NOT NULL, and generated by default as IDENTITY. (Previously the UPDATE_TIME column - TIMESTAMP type - was used, but it's not strictly unique and the incrementing mode did not work as expected, so I added this new column ID)
By "For upsert at the sink connector, you should have another column that uniquely identifies the record so that the upsert will overwrite this record.", do you mean I need to specify the incrementing column as well in the sink property? (like .incrementing.column.name)
FYI, the sink table has same structure as source table, except that the ID column is not generated by default, but by being inserted/upserted by KAFKA.
I did not go through the source code yet, so it would be great if you can explain a little about how the incrementing work in source - like fetching the latest message in the topic and then retrieving all newer data from source by comparing the incrementing column (which also requires the incrementing column should be monotonically increasing?)
The source/sink properties files are as below:
SOURCE
name=topic_metadata_source_incr connector.class=com.sap.kafka.connect.source.hana.HANASourceConnector tasks.max=1 topics=topic_metadata_incr connection.url=jdbc:sap://localhost:31115/ connection.user=user connection.password="pass" topic_metadata_incr.table.name="HDB_METADATA"."SCHEMA_DISK_SIZE" topic_metadata_incr.incrementing.column.name=ID #queryMode=query topic_metadata_incr.poll.interval.ms=5000 #topic_metadata_incr.query=SELECT ID, UPDATE_TIME, SID, SCHEMA_NAME, DISK_SIZE_GB FROM HDB_METADATA.SCHEMA_DISK_SIZE batch.max.rows=100000 mode=incrementing
SINK
name=topic_metadata_sink_incr connector.class=com.sap.kafka.connect.sink.hana.HANASinkConnector tasks.max=1 topics=topic_metadata_incr connection.url=jdbc:sap://localhost:31115/ connection.user=user connection.password="pass" auto.create=true topic_metadata_incr.table.name="HDB_METADATA"."SCHEMA_DISK_SIZE1" topic_metadata_incr.poll.interval.ms=5000 topic_metadata_incr.insert.mode=upsert

@elakito After fixing the configuration (adding consumer.auto.offset.reset=latest), I truncated the sink table and left the source table as it was (no recreation, no truncation...), and since then, things look as expected, no matter I restarted the connect (both source and sink , only either of them), the sink can poll the incrementing data from source. However, from the HANA SQL Cache, I can only see INSERT but no UPSERT statements.
The source table has an incrementing column ID - BIGINT type, which is also defined as primary key and NOT NULL, and generated by default as IDENTITY. (Previously the UPDATE_TIME column - TIMESTAMP type - was used, but it's not strictly unique and the incrementing mode did not work as expected, so I added this new column ID)
By "For upsert at the sink connector, you should have another column that uniquely identifies the record so that the upsert will overwrite this record.", do you mean I need to specify the incrementing column as well in the sink property? (like .incrementing.column.name)
FYI, the sink table has same structure as source table, except that the ID column is not generated by default, but by being inserted/upserted by KAFKA.
I did not go through the source code yet, so it would be great if you can explain a little about how the incrementing work in source - like fetching the latest message in the topic and then retrieving all newer data from source by comparing the incrementing column (which also requires the incrementing column should be monotonically increasing?)
The source/sink properties files are as below:
SOURCE
name=topic_metadata_source_incr connector.class=com.sap.kafka.connect.source.hana.HANASourceConnector tasks.max=1 topics=topic_metadata_incr connection.url=jdbc:sap://localhost:31115/ connection.user=user connection.password="pass" topic_metadata_incr.table.name="HDB_METADATA"."SCHEMA_DISK_SIZE" topic_metadata_incr.incrementing.column.name=ID #queryMode=query topic_metadata_incr.poll.interval.ms=5000 #topic_metadata_incr.query=SELECT ID, UPDATE_TIME, SID, SCHEMA_NAME, DISK_SIZE_GB FROM HDB_METADATA.SCHEMA_DISK_SIZE batch.max.rows=100000 mode=incrementing
SINK
name=topic_metadata_sink_incr connector.class=com.sap.kafka.connect.sink.hana.HANASinkConnector tasks.max=1 topics=topic_metadata_incr connection.url=jdbc:sap://localhost:31115/ connection.user=user connection.password="pass" auto.create=true topic_metadata_incr.table.name="HDB_METADATA"."SCHEMA_DISK_SIZE1" topic_metadata_incr.poll.interval.ms=5000 topic_metadata_incr.insert.mode=upsert

@twonly One of the problems with your configuration is that you can't use the incrementing column that generates a unique value at the source connector as the primary key used at the sink connector because there won't be any upserts/updates in that case, as every record received will have a unique primary key. So to achieve the desired upsert effect at the sink connector, you need to choose the primary key appropriately. For the source connector, any column can be used as the incrementing column. This column is used in the select's where clause to load those records having only the newer/greater values.
This column could be a primary key for the source table. But in that case, any update on that record happening on the source table won't be recognised as a new record because it has a used old ID. So, I don't know what you want to transfer from the source table to the sink table in your scenario. One possible scenario for such incrementing source -> upsert-sink scenario would be to have the source table having some monotonically increasing column used as the incrementing column and some column like deviceID and the sink table having deviceID as the primary key so that the sink table has the device's current state.
Regarding the upsert statement not observed in the cache, are you sure you are using the current master version? The upsert handling is only in the master branch. By the way, if you use the INFO log level, you should see the created prepared statements in the log. The parameter names in your configuration seem to be correct.

@twonly One of the problems with your configuration is that you can't use the incrementing column that generates a unique value at the source connector as the primary key used at the sink connector because there won't be any upserts/updates in that case, as every record received will have a unique primary key. So to achieve the desired upsert effect at the sink connector, you need to choose the primary key appropriately. For the source connector, any column can be used as the incrementing column. This column is used in the select's where clause to load those records having only the newer/greater values.
This column could be a primary key for the source table. But in that case, any update on that record happening on the source table won't be recognised as a new record because it has a used old ID. So, I don't know what you want to transfer from the source table to the sink table in your scenario. One possible scenario for such incrementing source -> upsert-sink scenario would be to have the source table having some monotonically increasing column used as the incrementing column and some column like deviceID and the sink table having deviceID as the primary key so that the sink table has the device's current state.
Regarding the upsert statement not observed in the cache, are you sure you are using the current master version? The upsert handling is only in the master branch. By the way, if you use the INFO log level, you should see the created prepared statements in the log. The parameter names in your configuration seem to be correct.

@elakito You are right, I used to put the column ID in Sink Table as unique generated key but then realized the problem you just mentioned, so I just kept it as primary key but no longer auto generated as this value would also be coming from the Source.
In my scenario, just trying kafka-connect-sap with HANA integration and see how it work and will explore more scenarios.
If the current Source Connector can only capture the new inserted data by using the incrementing column (monotonically increasing) then the use case might be limited.
More specifically in the current configuration, I am replicating the source table to the sink table using incrementing mode, and wanna see if new data can be correctly inserted/upserted into the sink table. As I am using the same column (ID in this case) as the primary key and also the incrementing column, I think no the INSERT would have the same effect as UPSERT, as update on the existing record (PK also the auto generated column would not be updated) would not be captured.
previously you mentioned a CDC-like source connector is being developed, do you have release plan on this?
Regarding the LOG LEVEL, i did not change the configuration and did not see prepared statement from the log (printing in screen) either. Would you suggest which config properties should I modify to see this? The standalone connect itself or the connector's properties file? Thanks.

@elakito You are right, I used to put the column ID in Sink Table as unique generated key but then realized the problem you just mentioned, so I just kept it as primary key but no longer auto generated as this value would also be coming from the Source.
In my scenario, just trying kafka-connect-sap with HANA integration and see how it work and will explore more scenarios.
If the current Source Connector can only capture the new inserted data by using the incrementing column (monotonically increasing) then the use case might be limited.
More specifically in the current configuration, I am replicating the source table to the sink table using incrementing mode, and wanna see if new data can be correctly inserted/upserted into the sink table. As I am using the same column (ID in this case) as the primary key and also the incrementing column, I think no the INSERT would have the same effect as UPSERT, as update on the existing record (PK also the auto generated column would not be updated) would not be captured.
previously you mentioned a CDC-like source connector is being developed, do you have release plan on this?
Regarding the LOG LEVEL, i did not change the configuration and did not see prepared statement from the log (printing in screen) either. Would you suggest which config properties should I modify to see this? The standalone connect itself or the connector's properties file? Thanks.

@elakito You are right, I used to put the column ID in Sink Table as unique generated key but then realized the problem you just mentioned, so I just kept it as primary key but no longer auto generated as this value would also be coming from the Source.
In my scenario, just trying kafka-connect-sap with HANA integration and see how it work and will explore more scenarios.

That sounds good. I am looking forward to hearing your feedback.

If the current Source Connector can only capture the new inserted data by using the incrementing column (monotonically increasing) then the use case might be limited.

Yes. This point was raised several times.

More specifically in the current configuration, I am replicating the source table to the sink table using incrementing mode, and wanna see if new data can be correctly inserted/upserted into the sink table. As I am using the same column (ID in this case) as the primary key and also the incrementing column, I think no the INSERT would have the same effect as UPSERT, as update on the existing record (PK also the auto generated column would not be updated) would not be captured.

For table replication, the incrementing column approach will work partially (i.e supporting only updates but no deletes), if the table has a unique non-primary column (e.g., incremental number or timestamp) used as the incremental column and another column serving as the primary key column.

previously you mentioned a CDC-like source connector is being developed, do you have release plan on this?

This feature will require the use of another jar file which is currently not covered by SAP Developer License (under which ngdbc.jar is covered). Therefore, there are some questions remaining, regarding how we can provide this connector. We can update you as soon as we have more information.

Regarding the LOG LEVEL, i did not change the configuration and did not see prepared statement from the log (printing in screen) either. Would you suggest which config properties should I modify to see this? The standalone connect itself or the connector's properties file? Thanks.

The standard connect setting logs INFO entries and the version of the connector was supposed to be printed. But I noticed a few days ago that the version was printed and had to fix it with https://github.com/SAP/kafka-connect-sap/pull/67.
Where did you get kafka-connector-hana_xxxx.jar? As the upsert change is only in master, you need to build the jar on your own or get one from someone who built it from master.
regards, aki

@elakito You are right, I used to put the column ID in Sink Table as unique generated key but then realized the problem you just mentioned, so I just kept it as primary key but no longer auto generated as this value would also be coming from the Source.
In my scenario, just trying kafka-connect-sap with HANA integration and see how it work and will explore more scenarios.

That sounds good. I am looking forward to hearing your feedback.

If the current Source Connector can only capture the new inserted data by using the incrementing column (monotonically increasing) then the use case might be limited.

Yes. This point was raised several times.

More specifically in the current configuration, I am replicating the source table to the sink table using incrementing mode, and wanna see if new data can be correctly inserted/upserted into the sink table. As I am using the same column (ID in this case) as the primary key and also the incrementing column, I think no the INSERT would have the same effect as UPSERT, as update on the existing record (PK also the auto generated column would not be updated) would not be captured.

For table replication, the incrementing column approach will work partially (i.e supporting only updates but no deletes), if the table has a unique non-primary column (e.g., incremental number or timestamp) used as the incremental column and another column serving as the primary key column.

previously you mentioned a CDC-like source connector is being developed, do you have release plan on this?

This feature will require the use of another jar file which is currently not covered by SAP Developer License (under which ngdbc.jar is covered). Therefore, there are some questions remaining, regarding how we can provide this connector. We can update you as soon as we have more information.

Regarding the LOG LEVEL, i did not change the configuration and did not see prepared statement from the log (printing in screen) either. Would you suggest which config properties should I modify to see this? The standalone connect itself or the connector's properties file? Thanks.

The standard connect setting logs INFO entries and the version of the connector was supposed to be printed. But I noticed a few days ago that the version was printed and had to fix it with https://github.com/SAP/kafka-connect-sap/pull/67.
Where did you get kafka-connector-hana_xxxx.jar? As the upsert change is only in master, you need to build the jar on your own or get one from someone who built it from master.
regards, aki

@twonly I'm closing this ticket as the question originally asked has been clarified and it has been inactive. If you encounter other problems, please open a specific ticket. Thanks.


Add TLS 1.3 cipher suites

Constants were taken from OpenSSL master and https://tools.ietf.org/html/draft-ietf-tls-tls13-18#page-103
@Lukasa I don't have resources to test ST patch.

That's ok, Travis will test SecureTransport for us. :D

Codecov Report

Merging #1170 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #1170   +/-   ##
======================================
  Coverage     100%    100%
======================================
  Files          21      21
  Lines        1975    1975
======================================
  Hits         1975    1975

Impacted Files
Coverage Δ

urllib3/util/ssl_.py
100% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a68917a...814e1fb. Read the comment docs.


Design: Validate note commitment trees in state

Is your feature request related to a problem?
In order to maintain shielded note commitment trees, we will need to update our implementation of Zebra state for non-finalized chains, keep those trees up to date in the non-finalized state as we validate blocks, and save down the serialized trees to finalized state for the tip.
Questions:
Unanswered questions about the finalized and non-finalized state:
What keys do we need to be able to look up quickly? For transaction validation? For treestate updates?
This changes how we index treestates in the state.
The state already has sprout and sapling anchors, and sprout, sapling, and orchard nullifiers.
Are there any rules about which treestates can be used for which transactions?
This might change how we maintain (Sprout?) treestates in the non-finalized state.
For example, can Sprout, Sapling, and Orchard transactions use the previous block's treestate?
(There are a lot of references to "finalized" treestates in the rest of the RFC, but I'm not sure if they are the same as the "finalized" state.)
The rest of the RFC mentions that Sprout transactions can use the treestate of any previous transaction in the block.
Why do we need to store a separate Sprout treestate for every block? Can we skip blocks without Sprout transactions? Are there parts of the treestate data that don't change?
This impacts the indexes, storage, and lookups for those treestates.
What are the details of a Sprout treestate update?
This might change how we maintain (Sprout?) treestates in the non-finalized or finalized state.

Influences from the UTXO design

Referencing a root that we haven't seen before, because when we're verifying blocks in parallel, is that ok? Cryptographically ?

about using roots that we haven't produced ourselves yet, if you use an invalid root, the proof verification would fail, because the witness path in the tree would fail to match the root

Non-finalized state
Maintaining note commitment trees

[ ] Add (sprout|sapling|orchard)_note_commitment_tree: (sprout|sapling|orchard)::tree::NoteCommitmentTree data field to the Chain type

this stores the note commitment tree for each non-finalized tip

[ ] When adding a block to a non-finalized tip:

if there are any (Sprout|Sapling|Orchard) note commitments exposed by JoinSplits or Outputs or Actions in the transactions in the block, they must be appended to the respective note commitment tree in-order
append to trees using NoteCommitmentTree.append()

[ ] When forking a new side-chain:

clone the note commitment tree from the fork block
add from there, do not do any explicit rollback logic, just rebuild from known-good tree

Update Chain.update_chain_state_with()

Ensuring that transactions are processed in-block-order, then:
Update self.update_chain_state_with(joinsplit_data) to append any sprout::NoteCommitments
self.update_chain_state_with(sapling_shielded_data_per_spend_anchor) to append any sapling::NoteCommitments
self.update_chain_state_with(sapling_shielded_data_shared_anchor) to append any sapling::NoteCommitments
self.update_chain_state_with(orchard_shielded_data)  to append any orchard::NoteCommitments

Finalized state
FinalizedBlock

[ ] Add (sprout|sapling|orchard)_note_commitment_tree fields

These should be pre-computed by non-finalized state/ Chain, the output treestates as of the last transaction in the block

Database format:

[ ] Add a new RocksDB column family for each of Sprout, Sapling, and Orchard serialized note commitment trees for the finalized tip:

[ ]  | sprout_incremental | BE32(height)        | sprout::tree::NoteCommitmentTree            | Delete |
[ ]  | sapling_incremental | BE32(height)        | sapling::tree::NoteCommitmentTree            | Delete |
[ ] | orchard_incremental | BE32(height)        | orchard::tree::NoteCommitmentTree            | Delete |

[ ] Increment Zebra's state version

Service

[ ] Add a request type NoteCommitmentTree that can vary by each pool, that will fetch the deserialized tree from a chain tip
[ ] Add an anchor request type (Await)Anchor::{Sprout, Sapling, Orchard} -> Result<(), Err> to check if we've seen and validated the requested anchor before. Await

Additional context
See https://github.com/zcashfoundation/zebra/issues/2091#issuecomment-832311598 for further details

After studying this and https://github.com/ZcashFoundation/zebra/pull/2458 I see two options for the overall flow of the finalized state:

When finalizing a block, get the finalized tree from the DB (or a cached value?), add the note commitments from the block and then update the finalized tree and store the new anchor in the state. (This requires "redoing" the work on the tree since it was already done in the Chain, but we no longer have that treestate since the Chain only keeps the treestate of the chain tip)
Store each treestate (and matching anchor) in each PreparedBlock of the Chain, so that when a block is finalized we already have the data we need.

I think the first approach is better since it wastes less memory and the additional computation it requires is small. But it may lead to some code repetition / refactoring since we need to recalculate stuff that was already calculated in the Chain. I'll explore the first option for now.

After studying this and #2458 I see two options for the overall flow of the finalized state:
* When finalizing a block, get the finalized tree from the DB (or a cached value?), add the note commitments from the block and then update the finalized tree and store the new anchor in the state. (This requires "redoing" the work on the tree since it was already done in the Chain, but we no longer have that treestate since the Chain only keeps the treestate of the chain tip)

* Store each treestate (and matching anchor) in each PreparedBlock of the Chain, so that when a block is finalized we already have the data we need.

I think the first approach is better since it wastes less memory and the additional computation it requires is small. But it may lead to some code repetition / refactoring since we need to recalculate stuff that was already calculated in the Chain. I'll explore the first option for now.

I think the first option is fine - feel free to open a ticket for later optimisations!

Additional questions:
* Should the trees be stored only when applicable? E.g. when commiting the genesis block, should we store empty Sprout, Sapling and Orchard trees, or should we first store them when applicable?

I'm not sure, here are some benefits and drawbacks of waiting until Sprout, Sapling, or NU5 activation:
Benefits of waiting until activation:

empty trees and their anchors are not available until the shielded pool is activated, so Zebra can't accidentally approve invalid transactions that depend on empty anchors
less CPU and fewer database writes

Drawbacks of waiting until activation:

Zebra might panic if it tries to verify a pool transfer before that shielded pool is activated (but that code would be a bug)
return types are more complex - they'll need to add an Option or Result layer

Based on those tradeoffs, I would prefer waiting - because verification correctness is a high priority.
What do you think?
Am I missing any benefits or drawbacks?

I'm not sure, here are some benefits and drawbacks of waiting until Sprout, Sapling, or NU5 activation:
Benefits of waiting until activation:

empty trees and their anchors are not available until the shielded pool is activated, so Zebra can't accidentally approve invalid transactions that depend on empty anchors
less CPU and fewer database writes

Drawbacks of waiting until activation:

Zebra might panic if it tries to verify a pool transfer before that shielded pool is activated (but that code would be a bug)
return types are more complex - they'll need to add an Option or Result layer

Based on those tradeoffs, I would prefer waiting - because verification correctness is a high priority.
What do you think?
Am I missing any benefits or drawbacks?

The code becomes a bit awkward yes, and I think it will make testing a bit difficult since it will be more difficult to "fake" some stuff. For example, with the finalized state, we will need to create a full chain with Sapling and Orchard "fake" activations in order to test it properly. I'm not sure if the current tests already do something similar, I need to check.
Other than that, I think it's a good idea. I've started to use Option for the trees in Chain. @dconnolly can you provide any feedback on this, if you can?

For example, with the finalized state, we will need to create a full chain with Sapling and Orchard "fake" activations in order to test it properly. I'm not sure if the current tests already do something similar, I need to check.

It might be worth looking at the nullifier tests from #2497, and the UTXO tests in one of my upcoming PRs. (Which has been delayed because I have to work out how to make other tests pass the UTXO checks.)

For the UTXO tests, I'll need to add test methods to the finalized and non-finalized states which add fake UTXOs.
We should be able to do something similar for the anchors, and just have test methods which add fake state?
As long as the non-test anchor initialisation code waits for the network upgrade, we should be fine in production. (We have existing integration tests which test that code in production.)

@teor2345: my concern is, for example, if we only create the Sapling tree in the Sapling activation height, then in order to test it in the finalized state then we will need to create more than 100K blocks which is unfeasible. From what I understand in the nullifiers test this doesn't happen because there is no check anywhere the test flow regarding the block height.

Regarding using Option, the big question is: when to change it from None to Some(Default::default()) (i.e. the empty tree). I see the following options:

Don't use Option. Always create the empty trees when instantiating objects (FinalizedState, Chain). This is the easiest. Drawbacks:

As @teor2345 pointed out this could hide a possible bug that attempts to use the trees before they're allowed to. However I think that's easier to validate elsewhere (e.g. don't allow Orchard transactions before Orchard activation).
More DB writes. This can de dealt with separately, though (e.g. only write the tree if it has changed).

Use Option and change from None to Some lazily (i.e. when the first note is added to the tree). Drawbacks:

Transactions that anchor on the empty tree won't work because the tree doesn't exist yet and its anchor wasn't added before. This would require special-case handling.

Use Option and change from None to Some on the correct activation height. Drawbacks:

Can make testing difficult since the test will need to generate blocks with the expected heights (seems impossible), or we will need to change the activation heights only for the tests.

After trying to use Option in the PR I'm more inclined to just not using it (first option).

@teor2345: my concern is, for example, if we only create the Sapling tree in the Sapling activation height, then in order to test it in the finalized state then we will need to create more than 100K blocks which is unfeasible. From what I understand in the nullifiers test this doesn't happen because there is no check anywhere the test flow regarding the block height.

There is a check that the heights are in order, which is why all the tests that use the state service or finalized state start at a genesis block.
But you're right, there's no check for any absolute activation block height.

Regarding using Option, the big question is: when to change it from None to Some(Default::default()) (i.e. the empty tree). I see the following options:
* Don't use `Option`. Always create the empty trees when instantiating objects (FinalizedState, Chain). This is the easiest. Drawbacks:

  * As @teor2345 pointed out this could hide a possible bug that attempts to use the trees before they're allowed to. However I think that's easier to validate elsewhere (e.g. don't allow Orchard transactions before Orchard activation).

We already have this validation in the transaction::Verifier, so we don't need to repeat it in the state:
https://github.com/ZcashFoundation/zebra/blob/6df17ff78c4f78c156bbea3a9d2b4d5cf786ec72/zebra-consensus/src/transaction.rs#L336-L343
Zebra's V5 transactions are the only transaction version that contains orchard actions, so we won't be asking for orchard roots until the v5 transaction check passes.

  * More DB writes. This can de dealt with separately, though (e.g. only write the tree if it has changed).

The DB is cached in RAM, so we don't need to limit writes in this PR. (Unless it's really easy.)
We can just open a follow-up ticket.

* Use `Option` and change from `None` to `Some` lazily (i.e. when the first note is added to the tree). Drawbacks:

  * Transactions that anchor on the empty tree won't work because the tree doesn't exist yet and its anchor wasn't added before. This would require special-case handling.

I don't think special-case handling is a good idea - we'd have to get it right for both block validation and mempool validation. And test it for both cases.

* Use `Option` and change from `None` to `Some` on the correct activation height. Drawbacks:

  * Can make testing difficult since the test will need to generate blocks with the expected heights (seems impossible), or we will need to change the activation heights only for the tests.

The correct activation height is actually the commit of the block before the activation block, because the anchors need to be available to mempool transactions and transactions in the activation block:
https://discord.com/channels/809218587167293450/811004767043977288/867179258525515838
This needs a spec clarification.

After trying to use Option in the PR I'm more inclined to just not using it (first option).

I agree, I think we'd be effectively duplicating validation we've done already in the transaction::Verifier. And it gets really tricky handling mempool and block transactions.

@teor2345: my concern is, for example, if we only create the Sapling tree in the Sapling activation height, then in order to test it in the finalized state then we will need to create more than 100K blocks which is unfeasible. From what I understand in the nullifiers test this doesn't happen because there is no check anywhere the test flow regarding the block height.
Regarding using Option, the big question is: when to change it from None to Some(Default::default()) (i.e. the empty tree). I see the following options:

Don't use Option. Always create the empty trees when instantiating objects (FinalizedState, Chain). This is the easiest. Drawbacks:

As @teor2345 pointed out this could hide a possible bug that attempts to use the trees before they're allowed to. However I think that's easier to validate elsewhere (e.g. don't allow Orchard transactions before Orchard activation).
More DB writes. This can de dealt with separately, though (e.g. only write the tree if it has changed).

Use Option and change from None to Some lazily (i.e. when the first note is added to the tree). Drawbacks:

Transactions that anchor on the empty tree won't work because the tree doesn't exist yet and its anchor wasn't added before. This would require special-case handling.

Use Option and change from None to Some on the correct activation height. Drawbacks:

Can make testing difficult since the test will need to generate blocks with the expected heights (seems impossible), or we will need to change the activation heights only for the tests.

After trying to use Option in the PR I'm more inclined to just not using it (first option).

I'm fine with instantiating all the trees from the beginning, as teor said we have several checks in place to ensure they will not be populated (besides the Default::default() / empty tree) until needed, and I think that's fine, if any accidental queries to them come back with a result besides the root of the empty tree, that should correctly fail, unless that is the first sprout/sapling/orchard transaction correctly looking for the root of the empty tree.

It looks like we have implemented all tasks in this issue.

Done!


Timestamps on every line in output

Is it possible to get a timestamp for every line in the output from the vm? (at least after the clock has been initialized)
This was added to uplink, and is very useful, but it would be great to get it for every other line as well:
     [ Uplink ] [ 2018-08-23T09:23:19Z ] Sending auth request to https://<IP_ADDRESS>:9090/auth
     [ DHCPv4 ] Configuration complete (eth0)
[ NaCl plugin interface eth0 ] IP address updated: <IP_ADDRESS>
     [ Uplink ] [ 2018-08-23T09:23:19Z ] Auth success (token received)

It could be possible with a plugin, yes. But the timestamp would probably have to come first, or just work on the INFO macro (which is probably not what you're asking for).

Same principle as it's made in Acorn - just replace default std handler with one that appends timestamp.
  OS::add_stdout(
  [] (const char* data, size_t len) {
    // append timestamp
    auto entry = "[" + isotime::now() + "]" + std::string{data, len};
    logger_->log(entry);
  });

Did #1875 solve this?

We almost have it. Though when I tried it just now this is the output I got:
--------------------------------------------------------------------------------
================================================================================[2018-08-28T13:37:48Z]
 IncludeOS v0.12.1-rc.4-15-g3fa7519e0-dirty (x86_64 / 64[2018-08-28T13:37:48Z] -bit)
 +--> Running [ IncludeOS Starbase[2018-08-28T13:37:48Z]  ]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~[2018-08-28T13:37:48Z]
   [ Starbase[2018-08-28T13:37:48Z]  ] Running CPUID...
   [ Starbase ] Detected 47[2018-08-28T13:37:48Z]  CPU features:
TSC_INV , LM , SS , SSE , MMX , CLFLUSH , PSE_36 , PAT , CMOV , MCA , PGE , MTRR , SEP , APIC , CX8 , MCE , PAE , MSR , TSC , PSE , DE , VME , FPU , F16C , OSXSAVE , TSC_DEADLINE , PCID , POPCNT , MOVBE , X2APIC , SSE4.2 , SYSCALL , FMA , RDTSCP , PDPE1GB , CX16 , NX , AVX , SSE3 , AES , SSE4.1 , RDRAND , FXSR , PCLMULQDQ , XSAVE , SSE2 , SSSE3 [2018-08-28T13:37:48Z]
     [ Uplink ] [ 2018-08-28T13:38:05Z ] Auth failed - Request timed out[2018-08-28T13:38:05Z]
     [ Uplink ] Retry auth in 5[2018-08-28T13:38:05Z]  seconds...
     [ Uplink ] [ 2018-08-28T13:38:12Z ] Sending auth request to http://<IP_ADDRESS>:9090/auth[2018-08-28T13:38:12Z]
     [ Uplink ] [ 2018-08-28T13:38:16Z ] Auth failed - Connection closing[2018-08-28T13:38:16Z]
     [ Uplink ] Retry auth in 10[2018-08-28T13:38:16Z]  seconds...
     [ Uplink ] [ 2018-08-28T13:38:28Z ] Sending auth request to http://<IP_ADDRESS>:9090/auth[2018-08-28T13:38:28Z]

So the timestamp is attached at the end of the line, and the first timetamp appears with the pasting of the IncludeOS version tag.

Much better with https://github.com/hioa-cs/IncludeOS/pull/1881/commits/3f1227a8366b9aa5edc2dcf4a6c762b0289f6ebf but not perfect

https://github.com/hioa-cs/IncludeOS/pull/1883


Support for SELinux context based job banners

Version: -feature
CUPS.org User: mra
Utilizing the Classification setting, banner options were added mls, te,
and selinux.  This patch includes blank banner sheets for these banner types, but they are place holders so that the stored job context can be used to generate the appropriate label.
Choosing selinux would result in the entire SELinux context to be used
as the label: staff_u:sysadm_r:sysadm_lpr_t:SystemLow-SystemHigh
Using te would have a label of: sysadm_lpr_t
And mls would just be: SystemLow-SystemHigh
Also included in this patch is support for a boolean PerPageLabels which
defaults to yes.  When set to no the labels are left off the actual job
output, and only forced on the banner pages.
One concern that our evaluator had was that a significantly complex
SELinux context could extend beyond the width of the page.  For our
evaluation another patch for WriteLabelProlog() in filter/common.c
broke up long labels over multiple lines.  Since the 1.2 release used
in RHEL5 that filter code has changed, but it could be ported if there
is interest.

@michaelrsweet would you be able to edit your comments to encapsulate the patch file in a code block?  I'm interested in those patches. but it's parsed by markdown at the moment and I'm not sure how to recover the original text.

Should be fixed now...

On Jul 24, 2016, at 2:41 AM, ileyd<EMAIL_ADDRESS>wrote:
@michaelrsweet would you be able to edit your comments to encapsulate the patch file in a code block? I'm interested in those patches. but it's parsed by markdown at the moment and I'm not sure how to recover the original text.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Michael Sweet


Release Google.Cloud.ContactCenterInsights.V1 version 2.13.0

Changes in this release:
No API surface changes; just dependency updates.

Created release for Google.Cloud.ContactCenterInsights.V1-2.13.0


cannot provide options

there is no way to provide options in config and seems it doesn't do anything to my js code

im working on this now

Awesome @colek42 let me know if you drop it then I will pick it up

Published in 0.2.0
Big thanks to @colek42 for the PR!


Add adapter option from axios

It looks like with the move from axios to gaxios, googleapis lost the ability to define adapters using axios' adapter option. We've previously used this to instrument googleapi client requests and would like to continue that while upgrading to future versions of googleapis.

:tada: This issue has been resolved in version 1.4.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Automated Package Upgrade

:tada: This PR is included in version 6.0.130 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Compatibility with iOS 18

Hi,
First of all, thanks for this project.
I'm also experimenting this with iOS 18, however I found out some problems while tinkering with iOS 18 simulator:

The file and directory changes are not saved. (This is not #27 because nothing was saved under iOS 18).
The toolbar/navigation bar are broken (the image shown below is the demo in this repo).

iOS 17:

iOS 18:

Please do consider providing a fix or provide some work-around.
Best regards,

Thanks for reporting this. I haven't tried using the iOS 18 beta so far, but given that we already had a few betas, I agree, it is time to give it a shot. I'll let you know what I find.

Thanks for reporting this. I haven't tried using the iOS 18 beta so far, but given that we already had a few betas, I agree, it is time to give it a shot. I'll let you know what I find.

@mchakravarty
Thanks! I (sort of) solved the second UI issue by having my own toolbar setup, I will share it to you later when I've completed it.
I think for now #27 is the issue that should be taken care first because it seems that it influences the current stable OS (iPadOS 17). Please tell me if more information is needed.

Some update on this.
I found no options for iOS 18 to hide the document toolbar (i.e.: the return button and the "rename" text field).
My workaround is to keep it while hiding all the toolbar of the NavigationSplitView and use another added button to control what's shown on the NavigationSplitView.
I will make a PR for this later.

Some update on this.
I found no options for iOS 18 to hide the document toolbar (i.e.: the return button and the "rename" text field).
My workaround is to keep it while hiding all the toolbar of the NavigationSplitView and use another added button to control what's shown on the NavigationSplitView.
I will make a PR for this later.

Another problem is that the name of the root directory in the navigator won't change if the project "file" is renamed via the Apple Document toolbar. One of the solutions I can think of is just not show the root directory as a folder and directly show all the sub-folders and files.

Regarding the original issue (i.e., re the two screenshots on top). I simply added some dividers on iOS to be able to visually distinguish the different elements.  It's not pretty, but good enough for a demo app.


Cursor issues in documentation panel on Windows

This task is automatically imported from the old Task Issue Board and it was originally created by Ilya Bogdanov.
Original issue is here.

Described here: https://www.pivotaltracker.com/story/show/183970777/comments/234661381

Unable to interact with documentation panel - when I move cursor away of hovering over a component content of the documentation changes,
2. When I want to select text in the documentation panel, selection cursor disappears when close to left border rendering it impossible to select text without double clicking the first word,
3. Windows default cursor behavior is not consistant - from time to time selection doesn't work, cursor disappears close to edges.

@vitvakatu I remember you fixed something in mouse interactions with the documentation panel. Is this task still valid?

Yes, I think it should be fixed.

Works by design, you need to lock the selected entry with RMB
Was definitely fixed
Needs to be checked, but might be fixed as well.

@galin-enso could you check the point 3?

Hmm, I think the main problem is that clicking over the scroll bar should scroll, and it does not. Actually, I'm not able to scroll with mouse drag at all, only by mouse wheel.
Closing this task, will create a new one for the above issue.


Running Visual Studio Code as guest

I would like to request for feature to map Visual Studio Code running in guest Ubuntu OS to map all keyboard Keys to Host running MacOS. So that I can press Command + p in Ubuntu just like when in MacOS. I get confuse when switching between Host and Guest for ctrl+p and Command + p

We just announced remote development with VS Code, check out the blog post for details https://code.visualstudio.com/blogs/2019/05/02/remote-development


BUG: not refreshing the results using region filter after text input filter

When is being used the text input filter in conjunction with the region filter the latter will not filter the results if used last.
Tested with:

Chrome Version 124.0.6367.156 (Official Build) (64-bit)
Edge Version 124.0.2478.80 (Official Build) (64 bit)
Opera Version 109.0.5097.68 (64 bit)
Firefox Version 125.0.3 (64 bit)

Steps to reproduce the error:

Search any country by name (e.g. "ita")
The results are correctly filtered
Select from the option region input some region (e.g. "Africa")
The results are not filtered according to the region filter. Using the provided example the only country appearing in the page should be "Mauritania".
If you go back to the text input field and add or remove any character the results will be correctly filtered

Same behavior doing similar steps (i.e. switching filtering order):

Start filtering by region (e.g. "Africa")
The results are correctly filtered
Filter the countries by name (e.g. "rita")
The results are not filtered according to the region filter. Using the example "Africa" and "rita" should be appear a list of 2 countries: "Mauritania" and "United Kingdom of Great Britain and Northern Ireland"
If you go back to the text input field and add or remove any character the results will be correctly filtered

I did reproduce the first scenario but I can't the second one.
When I search the term in the already filtered scenario, writing "rita" filters countries and shows only "Mauritania" which is correct. "United Kingdom of Great Britain and Northern Ireland" is not Africa. At least according to the API and how I know :)))

You're right, I missed something in the second scenario. I will write it again below.
Same behavior doing similar steps (i.e. switching filtering order):

Start filtering by region (e.g. "Africa")
The results are correctly filtered
Filter the countries by name (e.g. "rita")
Again the results are correctly filtered
Remove the country filter (select "All", you will encounter the problem also changing the filter by country and selecting for example "Europe")
The results are not filtered according to the region filter. Using the example "Africa" and "rita" should be appear a list of 2 countries: "Mauritania" and "United Kingdom of Great Britain and Northern Ireland"
If you go back to the text input field and add or remove any character the results will be correctly filtered

Actually I noticed also an odd behavior that I'm not really capable to describe. I will send you a video about it for more clarity.


[hmac,doc] S&R reg changes when IDLE

a bug has been found via PR #25554, but this has been considered as a specification bug. This PR fixes this issue. Now the specification says that DIGEST and MSG_LENGTH registers can be updated only when HMAC is in IDLE state (instead of when SHA core is disabled).

I have tested to write while the HMAC is not IDLE status. And I have found some period where it's actually possible:

Here we can see that a write is occurring while we are in IDLE state, but the value is propagated to the internal logic and used.
So, the specification should be reworded to say that the SW should update the message length and the digest only when the core is in IDLE state. Any changes outside of this window can lead to an unexpected state.
I'll update this PR consequently.
Do you approve this planned change @andreaskurth ?

I have added the CherryPick:earlgrey_1.0.0 tag as this is also valid for the taped-out chip.


myLog after accepting the call

视频不是流，请给一些建议
VAdapt Frame: 0 / 600 Changes: 0 Input: 192x144 Scale: 1 Output: 192x144 Changed: false
Jingle:Port[audio:1:0::Net[en0:<IP_ADDRESS>/24]]: Port deleted
Jingle:Port[audio:1:0::Net[en0:<IP_ADDRESS>/24]]: Removed port from allocator (2 remaining)
Removed port from p2p socket: 2 remaining
VAdapt Frame: 0 / 900 Changes: 0 Input: 192x144 Scale: 1 Output: 192x144 Changed: false
Jingle:Port[audio:1:0:relay:Net[en0:<IP_ADDRESS>/24]]: Port deleted
Jingle:Port[audio:1:0:relay:Net[en0:<IP_ADDRESS>/24]]: Removed port from allocator (1 remaining)
Removed port from p2p socket: 1 remaining
Jingle:Port[audio:1:0:local:Net[en0:<IP_ADDRESS>/24]]: Port deleted
Jingle:Port[audio:1:0:local:Net[en0:<IP_ADDRESS>/24]]: Removed port from allocator (0 remaining)
Removed port from p2p socket: 0 remaining
VAdapt Frame: 0 / 1200 Changes: 0 Input: 192x144 Scale: 1 Output: 192x144 Changed: false
VAdapt Frame: 0 / 1500 Changes: 0 Input: 192x144 Scale: 1 Output: 192x144 Changed: false
VAdapt Frame: 0 / 1800 Changes: 0 Input: 192x144 Scale: 1 Output: 192x144 Changed: false
VAdapt Frame: 0 / 2100 Changes: 0 Input: 192x144 Scale: 1 Output: 192x144 Changed: false
VAdapt Frame: 0 / 2400 Changes: 0 Input: 192x144 Scale: 1 Output: 192x144 Changed: false

Have you solved this?

I also have this problem.
After change the two devices internet to LAN(may be the same wifi),it works!
you can have a try.

then add turn server

如果你没有可用的STUN/TURN服务器，建议你在局域网中（同一个WIFI）下进行测试。


command that redirects you to a rickroll

>:)

abandoning this idea to work on a text adventure game in the console instead


Request to update the Logo of GEND

Name of Company/Individual : Legendary coin / Gamie Arroyo
Project Description (Chinese and English editions): Legendary coin with legendary staking platform and exchange. (传奇硬币与传奇的赌注平台和交流。)
Official Website: https://legendarycoin.tech/
Official Contact Email<EMAIL_ADDRESS>Whitepaper: https://legendarycoin.tech/wp-content/uploads/2018/09/Legendary-Coin-Whitepaper.pdf
Smart Contract Address : 0x0140ac0f46cca38a6ebdf14fcf699fb6b5912c8d
Gas Recommended : 60000
Community Links :
Medium : https://medium.com/@legendarycoingend
Reddit: https://www.reddit.com/user/Legendary_coin
Bitcointalk : https://bitcointalk.org/index.php?topic=5025814.msg45493712#msg45493712
Telegram: https://t.me/LegendaryCoin
Github: https://github.com/LegendaryCoinTech
Youtube: https://www.youtube.com/channel/UCBqguFVSIK09t1GV8xx2Wjg
Twitter: https://twitter.com/Legendary_coin
https://www.linkedin.com/company/legendary-coin-technology/
Facebook: http://bit.ly/LegendaryCoinFacebook
Country : Philippines
Logo.png : http://bit.ly/GEND132x132LOGO

{
"symbol": "GEND",
"address": "0x0140ac0f46cca38a6ebdf14fcf699fb6b5912c8d",
"overview":{
"en": "Legendary coin with legendary staking platform and exchange",
"zh": ""
},
"email"<EMAIL_ADDRESS>"website": "https://legendarycoin.tech/",
"whitepaper": "https://legendarycoin.tech/wp-content/uploads/2018/09/Legendary-Coin-Whitepaper.pdf",
"state": "NORMAL",
"published_on": "2018-09-06",
"initial_price":{
"ETH":"0.0000002 ETH",
"USD":"0.00004 USD",
"BTC":"0.000000007 BTC"
},
"links": {
"blog": "https://medium.com/@legendarycoingend",
"twitter": "https://twitter.com/Legendary_coin",
"telegram": "https://t.me/LegendaryCoin",
"github": "https://github.com/LegendaryCoinTech",
"facebook": "http://bit.ly/LegendaryCoinFacebook",
"reddit": "https://www.reddit.com/user/Legendary_coin",
"slack": "https://legendarytechnology.slack.com",
"medium": "https://medium.com/@legendarycoingend"
}
}


Inference code

Hi, bravo for the great findings!
Do you have any inference script?
Thanks.

Hi, we have test.py for inference,
and currently, I am going to make demo.py for the simplified version of test.py.
I will upload demo.py in this weak.
Thanks.

Hello, great repo!
Here is my naive attempt.
https://colab.research.google.com/drive/1PHnc_QYyf9b1_KJ1r15wYXaOXkdm1Mrk

Thanks @sharavsambuu ! I changed it to best_accuracy.pth and it didn't work, is there any reason why?
And the model is pretty inaccurate with any type of images right? like this : https://drive.google.com/drive/folders/15SMmcdY9fzeKg6aHZjqlsfbqWNn2i6_0?usp=sharing

Hi @hasiftwistcode  if you want you use a best_accuracy.pth, please take an attention on the parser.character, it is using 36 characters '0123456789abcdefghijklmnopqrstuvwxyz'. Other one(case-sensitive version) is using 96 characters, string.printable[:-6]. I didn't evaluate an accuracy on my dataset yet.

The model can even predict my input image though, but it is alright. Next time, ill use this model to train first on my dataset. Thanks for your help!

I updated notebook for your 4 images. Result seems not so good.

Appreciate your effort! I've looked at it and yeah as expected. I am currently doing ocr for the identity card, I did crop the address to little pieces which the images are really small and surprisingly tesseract is pretty good at it (very image-sensitive). I tried kraken ocr but it wasn't good as tesseract.

Thank you for letting me know, I also working on similar project. Tried various repos and results were not so good, also tried google vision api and it is working perfect except their high api price. Also tried tesseract and it needs a lot of preprocessing in order to work well in my case. Currently scraping and collecting my own dataset in order to cope those problems :) I hope you will do well, cheers!

Sorry for the late update.
I just uploaded demo.py.
check the run demo part in the Readme.
Best regards.


Customizable Mini Cart height

#4 from Issue #12:
Have a way to increase the height of the product area of the minicart (Currently shows 3-4 products at most). The scrollbar works fine, but is not immediately obvious unless you look for it. I fear some customers may think nothing is getting added to the cart when they really just need to scroll down the minicart.

Hi, thank you again for a great script. I was looking for some way to set the height automatically, perhaps based on an item count. I'm curious if you're talking about a function that would dynamically size the height of the product area or is the plan to add a config file to set a preferred height?
Around line 218 in the minicart.js file is the line:
css.push('#' + name + ' ul { position:relative; overflow-x:hidden; overflow-y:auto; height:230px
I've increased my local copy to 230px and that works for now. Thanks again!

I just updated the README with instructions on customizing the look of the Mini Cart more via the page's CSS. Here's a link: https://github.com/jeffharrell/MiniCart#readme
Take a look at the example where I use new CSS to create a dynamic height effect. This should get you pointed in the correct direction.


FileBrowser: Shortcut Icons

Allow the user to select a local image instead of pasting a URL, it's already possible by inputting a file:///FILEPATH uri, but this could be made more convenient by letting the user pick a file.

I can take a look at this @sidequestlegend 😉

Nevermind, it feels like this needs to get changed on the Website 😥

on it

added! it will be in the next release.


Is the library thread-safe?

I read the following in the core gRPC header (grpc/grpc.h):

THREAD SAFETY: access to grpc_call_start_batch in multi-threaded environment
needs to be synchronized. As an optimization, you may synchronize batches
containing just send operations independently from batches containing just
receive operations

But I haven't noticed such precautions being taken while (briefly) reading the code of grpc-haskell. So my question is: is the library thread-safe, i.e. can all high-level functions (like making requests, etc.) be used without any synchronization?
Thanks.

Everything above the Unsafe level, except for Network.GRPC.LowLevel.CompletionQueue.Internal,  should be thread-safe. The only synchronization used before calling grpc_call_start_batch is for the completion queue, not the call to grpc_call_start_batch itself. So we might have a bug. I do remember seeing that warning and I remember doing something about it, but I can't remember what, if anything, I did. I suspect that the warning may only apply if you call grpc_call_start_batch on the same grpc_call from multiple threads, which our library never does.
If this is a bug, it hasn't been a problem in practice so far (though that's not very reassuring for a possible thread safety issue!). I've benchmarked the library by sending and receiving many calls in parallel on many threads, and the benchmark never exhibited any instability.

Reading through the C++ implementation, I actually can't find any synchronization on either grpc_call_start_batch, or on grpc_completion_queue_pluck, yet in a couple issues, they state that the C++ library is thread safe. Puzzling.

Thanks for your answers. Just wanted to check. I did have a thread-safety problem, but it turned out to be due to me, not to your library.

So should we close this issue, then?

Yes, sorry. I got what I wanted.


Feature/192

⚠ WIP please no merge!
IMPORTANT

This PR should not be merged before #187
This PR is not yet ready to merge; please check Backlog 👇
@ckarande @UlisesGascon @lirantal As this PR is going to be quite long, please start review it even if it is not yet finished and provide feedback 🙏
The scope of this PR is to cover just the feature Benefits without DB but including testing, mocking, Docker, CI, etc...so this PR will determine how we are going to develop new features in the React API Rest app sample
Note: CI still broken due #187 current situation (cc: @UlisesGascon), so that part is going to be solved in #187. Once that is ready, I'll pull the changes to this current PR.

PR Backlog (see #192)

[x] Add autogenerated React-app
[x] Add autogenerated Express-api
[ ] Connect React to Express
[ ] Add basic tests
[ ] Add Docker/Docker-componse
[ ] Add CI support

Notable changes
React api

Added autogenerated express project (0f3fb39, 9ba3401)
Upgraded dependencies (f7aebe7)

 cookie-parser   ~1.4.4  →   ~1.4.5
 debug           ~2.6.9  →   ~4.1.1
 express        ~4.16.1  →  ~4.17.1
 http-errors     ~1.6.3  →   ~1.7.3
 morgan          ~1.9.1  →  ~1.10.0

Added config files (8b1884d)

Added depedency<EMAIL_ADDRESS>

Updated gitignore rules (5034336)
ES6 migration (0efe7f0, 49493d9)
Added standard linter (67cefa0, d316b82 & bcaf6ef)

Added dev dependency<EMAIL_ADDRESS>

Added test environment (414e602, 83e6b58)

Added dev dependencies<EMAIL_ADDRESS>&<EMAIL_ADDRESS>

Update linting rules (00bc5f3)
Added initial tests (e591970)

React client

Added autogenerated react app (8fd44fb)
Upgrade testing dependencies (fed2529)

@testing-library/jest-dom    ^4.2.4  →   ^5.1.1
@testing-library/react       ^9.3.2  →  ^10.0.1
@testing-library/user-event  ^7.1.2  →  ^10.0.0

Added linter to project (a24718b, b77296b & 780c963)
Npm tasks for testing (fc07d62)
Added styled component & material UI (2f07bf1)
Added material ui icons (25addff)
Updated app component (5731cd9)

Changelog

8fd44fb feat(react client): added autogenerated react app by @KoolTheba
0f3fb39 feat(react api): added autogenerated express project by @KoolTheba
f7aebe7 chore(react api): upgraded dependencies by @KoolTheba
8b1884d feat(react api): added config files by @KoolTheba
5034336 chore(files): updated gitignore rules by @KoolTheba
9ba3401 fix(react api): project name by @KoolTheba
0efe7f0 refactor(react api): ES6 migration by @KoolTheba
49493d9 refactor(react api): ES6 migration by @KoolTheba
67cefa0 feat(react api): added standard linter by @KoolTheba
d316b82 feat(react api): added lint git hook by @KoolTheba
bcaf6ef chore(react api): files linted by @KoolTheba
414e602 feat(react api): added test environment by @KoolTheba
83e6b58 feat(react api): added test git hook by @KoolTheba
00bc5f3 chore(react api): update linting rules by @KoolTheba
e591970 feat(react api): added initial tests by @KoolTheba
fed2529 chore(react client): upgrade testing dependencies by @KoolTheba
a24718b feat(react client): added linter to project by @KoolTheba
b77296b feat(react client): added npm tasks for linting by @KoolTheba
780c963 chore(react client): files linted by @KoolTheba
fc07d62 WIP(react client): npm tasks for testing by @KoolTheba
2f07bf1 feat(react client): added styled component & material UI by @KoolTheba
25addff feat(react client): added material ui icons by @KoolTheba
5731cd9 feat(react client): updated app component by @KoolTheba

Hey @KoolTheba this is awesome!
I'm going to push in a bunch of small but needed changes and updates to the codebase and your PR would be great as another follow-up. I know we're like 2 years late here but would you like to keep working on pushing it in? :-)

@KoolTheba when and if we do that migration - will it come with the exact same set of vulnerabilities in the new stack?

Hi @lirantal !
I've sent you a couple of messages in Slack :-)
I've said that I wanted to clarify some details. When you mention the exact same set of vulnerabilities you mean that you are thinking about adding more to the top 10 or vulnerabilities of the libraries as dependencies? According to the second one, we can stick to the stack chosen 2 years ago. I'd just upgrade React to the latest and maybe use NextJS, and probably, but on this @UlisesGascon can confirm, we could use Turborepo instead of Lerna.

@KoolTheba apologies, hard to keep up with many slacks and GitHub notifications.
So what I mean is that we should maintain the current state of vulnerabilities as is demoed on the NodeGoat project right now. It is ok if we have different ways to exploit them, but the same set of vulnerabilities that are documented today in the project's tutorial should be kept.
Is that the case for this PR?

Hi @lirantal !
Thanks for the clarification!
Even if this PR is about changes in the stack, the same set of vulnerabilities will remain the same, as demoed in the past.

Thanks for replying. I wasn't participating at the time it was demo'ed. Any chance there's a recording that easily verifies that?

Hi @lirantal !
By demoed I meant the original video demos of the project that can be found here 😃 https://www.youtube.com/@owaspnodegoatproject910/videos

I'm going to push in a bunch of small but needed changes and updates to the codebase and your PR would be great as another follow-up. I know we're like 2 years late here but would you like to keep working on pushing it in? :-)

If you want to take leadership on it, feel free to add the changes you mentioned. And if you need a contributor/maintainer, let me know, very happy to collaborate with you if you guide the next iteration.


Do not reuse ambient module name resolution as resolving to unmodified source file as that might not be true or might not do augmentation

@typescript-bot test it

@typescript-bot test tsserver top100
@typescript-bot user test tsserver


Bars disappear

Expected behavior
When I open a mail the action bar, the 'header', the navigation bar at the bottom of the screen and the mail content should be loaded.
Actual behavior
The header and the content loads but the action bar and the navigation bar are buggy. They disappear and instead, there is some part of the mail content shown. (see picture).
The Buttons are klickable but invisible.
This behavior doesn't appear in every mail I open, but after opening 5-7 mail I got every time one thats buggy. It seems to be independently from the mail content, if I re-open a buggy mail the behavior isn't the same.
Some times it happens instantly after opening the mail, some times after scrolling.
Some times only the action bar disappears.
Here's the debug log. In my view there is nothing helpful documented. What you can see is, that I try to open some mails, and that's it, actually.
The screenshot is taken from the german version of the "New twitter ToS" mail.
Steps to reproduce

Open some mails
scroll through them and wait for it.

Environment
K-9 Mail version: 5.111 (screens from 5.111, logs from 5.112, but behavior exactly the same)
Android version: 7.0 on Nexus 5X
Account type (IMAP, POP3, WebDAV/Exchange): IMAP

I've seen this. I'm suspicious it might be a Nougat bug but we should look at it anyway.

I've seen this on Android 6.0, too. Maybe a WebView bug?

Android 6.0.1
Also to me.
In Mail view window it not happens if I set in settings "show pitures" to "always" or click the butten "show pictures" in the Mail view.
I have tested it with notification Mails from Github.

I can confirm that the bug only appears when there are external images that aren't loaded (In my test just now). If I load them through the button the bars are back as usual.

On Android 6.0, too. I'd hoped the latest beta would fix it, but no such luck.
Thanks, @levinus for connecting this to loading of external images. I hadn't made that connection, but it seems to be legitimate.

the V5.113 still has the problem

Can also confirm with 5.113 on Nexus 5 with Android 6.0.1

I've got the bug also in a mail without external images.
Screenshot is here, sorry that I had to censor so much, but I think you can see what the problem is.
It seems that the action bar shows somethink from the header of the mail, the bottom is empty.

the bug is still present, also in the v5.114

Yeah we've not really investigated this one yet. I've seen it myself but it's difficult to determine why :(

I'm not sure.
But it seems this problem is fixed in K-9 Mail Version: 5.114.
Mayby by chance?
Since I have isntalled the current Version it's not happens anymore for me.
Can someone confirm that?

It still occurs on 5.114
It's just sporadic.

Also is present on 5.115 (F-Droid) on my Android 6.0.1 (Official ROM)

I recently updated from K-9 Mail_5.006.apk to K-9 Mail_5.203.apk and see this new bug too on my Samsung (Unlocked/T-Mobile) Tab3 running KitKat [4.4.4].  Another device running K9v5.010 on 4.4.2 does not show this bug.
I have one message which seems to trigger this bug reliably.  The sequence appears to be:

The message is drawn as it should be when I tap it on the message list.
A scroll bar appears.
The scroll bar, message header and message toobar all go away.
One, both or neither of the message header and toolbar may reappear.

A workaround is to switch to another app (long-press Home) and switch back.
Another workaround is to press the button you want during the fraction of a second before it goes away.
As noted above, the invisible action buttons are still pressable, which can be destructive depending on what I might have in that bar.
Tried to submit screenshots of a message with either or both parts missing.  "Something went really wrong, and we can't process that file."  Haven't been able to read Android logs since my 2.3.4 handset died.  Hopefully the event sequence described above is helpful.

This has probably become more common but clearly has existed for a long time. I've closed the other one as dupe (though it may have useful information to a developer trying to fix it).

In the Release  notes 5.204:

Improved support for messages that didn't display correctly

Why this release and/or the following release 5.205 is still missing on Google Play?

This bug is still open. Neither 5.204 nor 5.205 will fix it.

f-droid 5.206 has it in my Motorola G Phone running 4.4.4  but I have never seen it in tablet Samsung galaxy 3 might be due to form factor?

Asked on GNU social and got a similar response,I think it's an Android 5+ issue.

Sorry to say this issue is still present in K9 5.400 on Android 4.4.

Yes I must confirm that the bar is still disappearing. The bug is repeatable I don't know why it's not corrected.
Jan Dudek
-------- Původní zpráva --------
Odesílatel: "Silas S. Brown"<EMAIL_ADDRESS>Odesláno: 29. prosince 2017 22:27:10 CET
Komu: k9mail/k-9<EMAIL_ADDRESS>Kopie: JDJADU<EMAIL_ADDRESS>Comment<EMAIL_ADDRESS>Předmět: Re: [k9mail/k-9] Bars disappear (#1671)
Sorry to say this issue is still present in K9 5.400 on Android 4.4.
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/k9mail/k-9/issues/1671#issuecomment-354502398

I have now changed to lineageOS android 7.1.2 and k9 5.403.
I can no longer replicate the issue.

I think that this problem is only on Android 4.4
Od: "Andres Muniz Piniella"<EMAIL_ADDRESS>Komu: "k9mail/k-9"<EMAIL_ADDRESS>Kopie: "JDJADU" dudek@jadu.cz,"Comment"<EMAIL_ADDRESS>Datum: Sun, 11 Feb 2018 17:18:11 -0800
Předmet: Re: [k9mail/k-9] Bars disappear (#1671)
I have now changed to lineageOS android 7.1.2 and k9 5.403.
I can no longer replicate the issue.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

Reproduced on Android 4.4 (Xperia Z Ultra) and Android 4.1 (Galaxy S2).  I have not been able to test on Android 5 or later.


Enhancement: Batch block document get requests

Description
Enables request batching for individual block document requests. Something to note is the isAnonymous: null part of the filter. This ensures that both anonymous and non anonymous blocks are returned.

In case you've not already seen, it looks like we need a type update for this.

In case you've not already seen, it looks like we need a type update for this.

I hadn't. Thank you!


Hook system tests to CI

Depends on #898
The only functional thing I changed is moving the Docker Hub login after the build&tag, that I do not think is relevant (login is still before push).
TODO:

[ ] Rebase on master once #898 merged

I arrived to saving system test logs on CI.
Tests fail in test case init (sample) with {'EXIT',{case_clause,{error,enoent}}}, not sure why.

Rebased on master.

Tracked as https://www.pivotaltracker.com/story/show/156683319