New optimizers API accepts 'lr' as argument, but ignores it

Note: this was originally posted on TF's Github, but identified as being a Keras-related bug: https://github.com/tensorflow/tensorflow/issues/58831
Click to expand!
Issue Type
Bug
Source
source
Tensorflow Version
TF 2.11
Custom Code
Yes
OS Platform and Distribution
All
Mobile device
No response
Python version
3.10.6
Bazel version
No response
GCC/Compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
No response
Current Behaviour?
The 'lr' argument for learning rate got deprecated in recent TF versions, and replaced by 'learning_rate'. Recent TF versions gave a warning when 'lr' was used, which got updated to:

"WARNING:absl:`lr` is deprecated, please use `learning_rate` instead, or use the legacy optimizer, e.g.,tf.keras.optimizers.legacy.Adagrad."

However, training still continuous, ignoring the learning rate set. See the code to reproduce: when 'lr=' is used, the final train loss is 0.1577; when 'learning_rate=' is used the final train loss is 0.5490 (higher loss due to much smaller learning rate).

I think the expected behavior here is that TF throwns an exception when 'lr' is used as an argument. The fact that the optimizer ignores it may easily go unnoticed, despite the warning, e.g. in an MLOps scenario, or when using a third-party module that still uses 'lr=' internally.

Standalone code to reproduce the issue
import tensorflow as tf
import tensorflow.keras as keras
from tensorflow.keras.datasets import mnist
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, Flatten, Activation
from tensorflow.keras.layers import Conv2D, MaxPooling2D
from tensorflow.keras import backend as K
from tensorflow.keras.layers import BatchNormalization
import timeit

print('TF version: {}'.format(tf.__version__))

batch_size = 128
num_classes = 10
epochs = 12
lr = 1e-4

# input image dimensions
img_rows, img_cols = 28, 28

# the data, shuffled and split between train and test sets
(x_train, y_train), (x_test, y_test) = mnist.load_data()

if K.image_data_format() == 'channels_first':
    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)
    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)
    input_shape = (1, img_rows, img_cols)
else:
    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
    input_shape = (img_rows, img_cols, 1)

x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255
print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')

# convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
                 input_shape=input_shape))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Conv2D(64, (3, 3)))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adagrad(lr=lr),
              metrics=['accuracy'])

tic = timeit.default_timer()
model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test, y_test))
toc = timeit.default_timer()
print('Total training time: %f sec' % (toc-tic))

score = model.evaluate(x_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])

Relevant log output
No response

@tilakraya and @jaspervandorp l I am interested. Could you elaborate more on it? Also, some guidance on solving it would be really helpful

@tilakraya and @jaspervandorp l I am interested. Could you elaborate more on it? Also, some guidance on solving it would be really helpful

Sure! Are there any details you'd like to know more on? The basic issue is really simple actually: up to recent TF versions, these two were equivalent:

AdaGrad(lr=...)
AdaGrad(learning_rate=...)

These two only differed in that the first one yielded a warning about 'lr' being deprecated.
In TF2.11, Adagrad() still accepts both keywords, but the learning rate is only set properly in option 2. You can try this out for yourself using the stand-alone example I posted. I would expect that either:
a) The use of 'lr' as a keyword yield an exception, or
b) The use of 'lr' is still equivalent to 'learning_rate' (with a warning)

@tilakraya and @jaspervandorp l I am interested. Could you elaborate more on it? Also, some guidance on solving it would be really helpful

Sure! Are there any details you'd like to know more on? The basic issue is really simple actually: up to recent TF versions, these two were equivalent:

AdaGrad(lr=...)
AdaGrad(learning_rate=...)

These two only differed in that the first one yielded a warning about 'lr' being deprecated.
In TF2.11, Adagrad() still accepts both keywords, but the learning rate is only set properly in option 2. You can try this out for yourself using the stand-alone example I posted. I would expect that either:
a) The use of 'lr' as a keyword yields an exception, or b) The use of 'lr' is still equivalent to 'learning_rate' (with a warning)

So, either it should throw an error or work (while giving deprecations warning), right?.

Exactly, either one of those two would be expected behavior as far as I'm concerned.

@jaspervandorp,
I can see that it is providing the warning UserWarning: The lrargument is deprecated, uselearning_rate instead. Kindly find the gist of it here.

@jaspervandorp, I can see that it is providing the warning UserWarning: The lrargument is deprecated, uselearning_rate instead. Kindly find the gist of it here.

Correct, and this warning has been there for a while now, with one major difference: in previous versions 'lr=' still worked as a keyword, despite the warning. I think that now 'lr=' no longer sets the learning rate appropriately, the warning should be an exception. As explained in the original report, the warning can be easily missed in a scenario where someone is running their ML within a MLOps environment, or using third-party scripts, for example.
But maybe I'm wrong in thinking this?

So, should we throw an exception or not?

If it has been there for a while, like for a few versions, maybe it is a good idea to start removing it with the initial steps of exceptions since it is intended to be removed eventually. It is just a personal thought. What do you think @tilakrayal ?

I would agree with this. Either remove it (with an exception) or keep it in place and working (with a warning)

@chenmoneygithub Would you please comment on this issue? Thanks.

@jaspervandorp What actually happens is lr is silently ignored, agree we should throw an exception. Will work on that.

@jaspervandorp What actually happens is lr is silently ignored, agree we should throw an exception. Will work on that.

@chenmoneygithub Can I work on it as a contributor?


Need to create postman doc for APIs

requirement:
need to create postman docs for better understanding of APIs and future use

https://github.com/bithyve/internal-docs

Look at the existing docs - make changes if needed
Look at the Drop Box created for the new joiners - add files in Internal Docs if needed
New documents as needed e.g Postman doc
Structure the documents in folders


Burn Mechanism

This burn mechanism is calculated based on the ratio of emission that is contributed by the modules. When the modules are fully out of stake, they are added to a PendingDeregisterUids Queue which is chipped off every block in the block_step.

@salvivona I've done an initial review
Very first question: why do you use u16 to store a value in the range [0, 100]?


Improve error messages

Handle :workspace and :prefix and print explanations
Handle filters not ending in .git

While I can see 422 returned for filters without .git, the body of the response isn't shown — not sure why.


export states

Usage: pummel export --states SORT_DIR
(Presumably I did something slightly wrong with git--what's up with Merge pull request 1 from ...?)

Not exactly sure, bit it probably has to do with you doing your PRs from master. Better practice is to do it on a separate branch off of my master for each PR.
Will look at these changes in a bit.

CLI for this added in 5ff9876


hanging with account command

With test pool, when try to connect to testnet type account command, it will hangup there.

It's fixed by @sgaragagghu. Not reproduced anymore.


SAS Online API endpoints changed - receiving error - unable to trade

Since today morning I am continuously receiving the following error message when I try to place the order. The automated TOTP is generated through pyotp as we know. I am not sure why it is not working properly. Earlier this issue was coming up randomly and it would work after trying multiple times. However, now it doesn't work at all.
{"data":{},"error_code":49008,"message":"Invalid TOTP, please try again","status":"error"}

Invalid totp error does not come while placing order. It may come at the time of login itself ,if any. Check your code.

Following is the code that generates the totp. Should I change anything? In the second line, for 'Totp' I pass the shortcode which was displayed below the barcode while configuring the Google Authenticator.
try:
pin = pyotp.TOTP(Totp)
totp = f"{int(pin.now()):06d}" if len(pin.now()) <=5 else pin.now()
sas = AlphaTrade(login_id=login_id,
password=password,
twofa=totp,
master_contracts_to_download=["NFO","NSE"],
 tokenPath = path)
It has been working fine so far. Not sure why it throws the error suddenly. I just followed whatever is mentioned in the documentation.

Sync your time. Let it auto adjust through internet. Make sure to have time zone as chennai,kolkata...

Time zone and auto sync is all active only. But still it was failing. I debugged the code and found that the profile URLs are not working anymore properly. For example, the profile login URL mentioned in the AlphaTrade.py file https://alpha.sasonline.in/api/v3/profile is returning 'Not Found' error.  Check the following line (link to the code).
https://github.com/algo2t/alphatrade/blob/326a0502ab8a06689ef6ba010b67de34090dba4b/alphatrade/alphatrade.py#L276
I think maybe they pulled down the old API since they appear to be having a new API with the documentation stored at https://drive.google.com/file/d/15HMf1ieewe2FgHxZJ-YIGeDtyNVEgCQ2/view

I dont think so.
May need to try two three times while login. Once logged in ,  no such totp issue must come.
Or are you logging in the web too simultaneously/patallely

Nope. I am not logging-in through Web or Mobile. Only through the python script. All these days it was working fine. Rarely it used to throw this error. But at-least that endpoint use to work. Not it is returning 'not found' error. So, creating the AlphaTrade object itself is not possible and because of that we are getting this error. It is not actually mainly related to invalid TOTP error, it is mainly that we session is not getting created since the profile URL is returning not found error.

Ok. Then others must havd reported too.
Lets wait..

@nevatia Were you able to run the code and test it out now?

I am outside. So cant check now. May check tomorrow night

All endpoints are now changed so I modified endpoint paths to v1 but no testing done. Kindly test and report bugs. Pull Requests are welcomed.

Time zone and auto sync is all active only. But still it was failing. I debugged the code and found that the profile URLs are not working anymore properly. For example, the profile login URL mentioned in the AlphaTrade.py file https://alpha.sasonline.in/api/v3/profile is returning 'Not Found' error. Check the following line (link to the code).
https://github.com/algo2t/alphatrade/blob/326a0502ab8a06689ef6ba010b67de34090dba4b/alphatrade/alphatrade.py#L276
I think maybe they pulled down the old API since they appear to be having a new API with the documentation stored at https://drive.google.com/file/d/15HMf1ieewe2FgHxZJ-YIGeDtyNVEgCQ2/view

@algo2t By v1 endpoints, do you mean the official API provided by SAS Online? Because, that is where I see v1 URIs.

@algo2t By v1 endpoints, do you mean the official API provided by SAS Online? Because, that is where I see v1 URIs.

Have you checked the new code? But it will no longer have the websocket implementation

@algo2t By v1 endpoints, do you mean the official API provided by SAS Online? Because, that is where I see v1 URIs.

Have you checked the new code? But it will no longer have the websocket implementation

Hi @rahulmr , have you checked the library https://github.com/TradelabTechnologiesIndia/pyoauthbridge ? It does have a websocket implementation as wsclient.py
This library is what SAS Online provides as their official API client library. Generating Oauth2 is the issue, but I think we can generate the token in a similar way we do in AlphaTrade.

I am closing this issue and waiting for people to come back with new bugs, because of my limited time. I have updated the endpoints and created a new release v1.0.0 as well. python -m pip install --upgrade alphatrade (before that make sure you go through the updated readme)


#6: Created Maven Central badge hyperlink to Sonatype Central

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

:exclamation: No coverage uploaded for pull request base (master@3f37423). Click here to learn what that means.

Additional details and impacted files
@@            Coverage Diff            @@
##             master       #7   +/-   ##
=========================================
  Coverage          ?   99.51%
  Complexity        ?      588
=========================================
  Files             ?      138
  Lines             ?     1451
  Branches          ?       37
=========================================
  Hits              ?     1444
  Misses            ?        6
  Partials          ?        1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


dart:ffi not working with Isolate.spawn

We implemented a flutter plugin which uses dart:ffi package to call c++ code.
It works well without Isolate.spawn. But it crashed after moving the code to spawned Isolate.
So, does Isolate.spawn not support dart:ffi?
The error message as following:
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0
Cause: null pointer dereference

My developer tools as following:
* Dart&Flutter SDK Version
Flutter 2.10.3 • channel stable • https://github.com/flutter/flutter.git
Framework • revision 7e9793dee1 (7 days ago) • 2022-03-02 11:23:12 -0600
Engine • revision bd539267b4
Tools • Dart 2.16.1 • DevTools 2.9.2

* Whether you are using Windows, MacOSX, or Linux (if applicable)
MacOSX

You need to provide more information. Right now the only thing we can see is that there is some null pointer dereference. There can be many different reasons why your code is crashing. dart:ffi should work fine with isolates - though you should keep in mind that isolates involve threads, which means that certain things might work differently (e.g. if your C++ library uses TLS storage, etc).

@mraleph thanks a lot for your quick response.
Basically, the data flow is as follows:
user code   <---dart object--->  lib (as a flutter plugin) which calls Isolate.spawn()   <---dart object--->  spawned isolate which uses dart:ffi to call c++ code <---json---> c++ code which runs heavy task, and also invokes callbacks from spawned isolate.
The code which calls Isolate.spawn() is as follows:
void testSpawn() async {
    Stopwatch stopwatch = Stopwatch()..start();
    final filename = 'json_01.json';
    final result = result = await _spawnAndReceive(filename);s
    print('result: $result');
    print('test spawn: in ${stopwatch.elapsedMilliseconds} ms');
  }

  // Spawns an isolate and sends a [filename] as the first message.
  // Waits to receive a message from the the spawned isolate containing the
  // parsed JSON.
  static Future<String> _spawnAndReceive(String fileName) async {
    final p = ReceivePort();
    await Isolate.spawn(_readAndParseJson, [p.sendPort, fileName]);
    return (await p.first) as String;
  }

  // The entrypoint that runs on the spawned isolate. Reads the contents of
  // fileName, decodes the JSON, and sends the result back the the main
  // isolate.
  static void _readAndParseJson(List<dynamic> args) async {
    SendPort responsePort = args[0];
    String fileName = args[1];
    // 🚀 use dart:ffi to call c++ code here 🚀
    Isolate.exit(responsePort, "1");
  }

If dart:ffi works fine with isolates,  I may need look more into c++ code.
BTW, my c++ code uses dart api to release dart object as follows:
  FinalizablePointer *peer = new FinalizablePointer();
  peer->pointer = pointer;
  peer->finalizer = finalizer;
  Dart_FinalizableHandle handle = Dart_NewFinalizableHandle_DL(
      object, (void *)peer, external_allocation_size, finalizer_callback);
  if (handle == NULL) {
    delete peer;
    return 1;
  }

//cc @aam

Isolate.spawn and ffi should work without any issues.
There are tests in dart sdk that ensure that, for example https://github.com/dart-lang/sdk/blob/main/tests/ffi/vmspecific_function_callbacks_exit_test.dart

One thing I could think about is that reusing something like a FFI trampoline between two isolates might result in a strange issues due to how this is implemented right now. e.g. the following will break in strange ways:
void callback() {
}

final callbackPtr = Pointer.fromFunction<Void Function()>(callback);

void isolateMain() {
  if (clibrary_needs_init())
    init_clibrary(callbackPtr);
}

e.g. if you spawn isolateMain multiple times but only initialize c library with a callbackPtr from the first isolate - things will go south later if library tries to call the callbackPtr it has. (we should probably fix this /cc @dcharkes @mkustermann)
But it is hard to say if this a problem or not without seeing stack trace from your SIGSEGV and/or poking around in the debugger to understand where exactly the crash occurs.

if you spawn isolateMain multiple times but only initialize c library with a callbackPtr from the first isolate - things will go south later if library tries to call the callbackPtr it has.

Callbacks into an isolate can only be invoked when being the mutator thread from that isolate (either by doing an ffi call from that isolate, or by using the Dart API to enter the isolate). This is a known limitation.
Asynchronous callbacks would not have this limitation, when we get around to implementing those:

https://github.com/dart-lang/sdk/issues/37022

The known workaround is to use ports for the callbacks rather than ffi callback trampolines. See documentation in the first post in the above issue.

we should probably fix this

@mraleph, how do you propose addressing this known limitation? Always calling to a trampoline which blocks until there's no more mutator running in the isolate and then run it? That is basically what we would do for async callbacks.

Maybe this is the key to this problem. I'm going to try long running isolate  solution, and will give feedback here soon.
Thank you. @mraleph @dcharkes

e.g. if you spawn isolateMain multiple times but only initialize c library with a callbackPtr from the first isolate - things will go south later if library tries to call the callbackPtr it has. (we should probably fix this /cc @dcharkes @mkustermann)

It still does NOT work stably, even if initialize the native code once in the spawned isolate and remove all callbacks.  @mraleph @dcharkes
I have created a minimal repo(isolate-spawn-demo)  to reproduce this:
1. clone the repo
2. cd example && flutter pub get && flutter run -v
3. click the button "call ffi in spawn isolate"
4. sometimes it works, sometimes it will cause an error: "############failed to call ffi: $e ################"

Meanwhile, It works stably if call ffi in the main isolate. Just show the button in example/lib/main.dart and run again(Don't hot reload):
ElevatedButton(
  child: Text('call ffi in main isolate'),
  onPressed: callInMainIsolate,
),
// ElevatedButton(
//   child: Text('call ffi in spawn isolate'),
//   onPressed: callInSpawnedIsolate,
// ),

@xuchaoqian My guess would be the following (without looking at this in the debugger). It's the same thing I warned about in my very first comment: V8 uses TLS (thread local storage) to associate v8::Isolate with native threads. However Dart does not guarantee that a particular isolate always runs on a particular thread: isolates run on top of a thread pool, so if Dart isolate suspends and resumes (e.g. due to waiting for some data to arrive via a port) it might end up on a different thread.
Your current code enters v8::Isolate when Runtime is created. However between creating Runtime and then using it for Script the underlying Dart isolate might have ended up on a different thread which does not have v8::Isolate.
You can try to change your code to enter isolate in Script instead, i.e. remove Enter/Exit from Runtime/~Runtime and instead in Script (https://github.com/xuchaoqian/isolate-spawn-demo/blob/main/cpp/Script.cpp#L7) do:
Script::Script() {
  auto isolate = Runtime::GetInstance()->GetIsolate();
  v8::Isolate::Scope isolate_scope(isolate);
  // ...
}

I think this will likely fix the issue.
There is a feature request to allow creating isolates pinned to a specific thread (https://github.com/dart-lang/sdk/issues/46943) but we have not yet have a chance to figure out how it would look like, so right now you have to be very careful using libraries like V8 which rely on TLS for certain things.

@mraleph Still does NOT work correctly. BUT you inspired me just now!
I should NOT use Isolate.spawn in my case, I should use thread in native code directly.
I will have a try in this way, and give feedback here soon.

Sorry for late reply.  It took me some time to completely rewrite my lib. Now it works perfectly to use this solution: combine the benefits of native threads and native port.
Thank you for your awesome works on dart vm and dart:ffi. @mraleph @dcharkes @aam
I still have some questions about resource management as follows:
1. Is it safe to malloc native memory on the [dart side]  (use String.toNativeUtf8() method) and call the free() function on the [cpp side]?
2. Is it safe to malloc native memory on the [cpp side] (use malloc() function) and call malloc.free() method on the [dart side]?
It's a little tricky to transfer pointer from cpp side to dart side:
Cpp side
Dart_CObject errmsg_obj;
errmsg_obj.type = Dart_CObject_kInt64;
errmsg_obj.value.as_int64 = reinterpret_cast<intptr_t>(errmsg); // cpp pointer, point to object allocated by malloc!!!

Dart_CObject payload_obj;
payload_obj.type = Dart_CObject_kInt64;
payload_obj.value.as_int64 = reinterpret_cast<intptr_t>(payload); // cpp pointer, point to object allocated by malloc!!!

Dart_CObject* result_values[] = {&errmsg_obj, &payload_obj};
Dart_CObject result_obj;
result_obj.type = Dart_CObject_kArray;
result_obj.value.as_array.values = result_values;
result_obj.value.as_array.length =
    sizeof(result_values) / sizeof(result_values[0]);
Dart_PostCObject_DL(send_port_, &result_obj);

Dart side
malloc.free(Pointer<Void>.fromAddress(errmsgPtr)); // from cpp side's errmsg_obj.value.as_int64
malloc.free(Pointer<Void>.fromAddress(payloadPtr)); // from cpp side's payload_obj.value.as_int64

(Note: this is becoming off topic, feel free to open new issues for things not imediately related to isolate.spawn and dart:ffi.)
RE 1 & 2: Yes. As long as you're not on Windows. On Windows package:ffi's malloc uses WinHeapAlloc and WinHeapFree:
https://github.com/dart-lang/ffi/blob/5e801e857662819d9bdb15df7dbb49d9ed60bb66/lib/src/allocation.dart#L90-L93
malloc and free on Windows cannot be paired from different dlls. WinHeapdoesn't have this problem. So make sure to use the WinHeap in your C code when targeting windows.

It's a little tricky to transfer pointers from cpp side to dart side

This is by design. Your code looks exactly as expected. We want to make Pointers disappear completely at runtime.

@dcharkes  I've learned a lot. Thank you!


Raspbian bullseye - Installation fails "Iek!"

I decided to do a bit of testing and while an in place upgrade from buster to bullseye does work[1]. a fresh install does not and bails with W: parser_rfc822: Iek! Don't find end of value!
Full log from telnet...
Installation started at Wed Sep 15 15:27:27 UTC 2021 (UTC).

Resulting installer configuration:
  preset = server
  packages = nano
  firmware_packages =
  mirror = http://raspbian.raspberrypi.org/raspbian/
  mirror_cache =
  release = bullseye
  hostname = rpi-e1
  domainname =
  user_ssh_pubkey = <REDACTED BY GCHQ>
  root_ssh_pubkey =
  root_ssh_pwlogin = 1
  ssh_pwlogin =
  username = mark
  usergpio =
  usergpu = 1
  usergroups =
  usersysgroups =
  userperms_admin = 1
  userperms_sound = 0
  cdebootstrap_cmdline = --flavour=minimal --include=locales,console-setup,kmod,libpam-systemd,rng-tools,cpufrequtils,openssh-server,dosfstools,ntp,fake-hwclock,ifupdown,net-tools,vim-tiny,iputils-ping,wget,ca-certificates,rsyslog,cron,dialog,locales,tzdata,less,man-db,logrotate,bash-completion,console-setup,apt-utils,systemd-sysv,sudo,isc-dhcp-client
  packages_postinstall = nano,raspberrypi-bootloader,raspberrypi-kernel,raspberrypi-sys-mods,libraspberrypi-bin,raspi-copies-and-fills
  boot_volume_label =
  root_volume_label =
  bootsize = +128M
  bootoffset = 8192
  rootsize =
  timeserver = time.nist.gov
  timezone = Europe/London
  rtc =
  dt_overlays =
  keyboard_layout = gb
  locales = en_GB.UTF-8
  system_default_locale = en_GB.UTF-8
  wlan_country =
  ip_ipv6 = 1
  cmdline = dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 elevator=deadline fsck.repair=yes
  drivers_to_load =
  gpu_mem = 32
  console_blank =
  hdmi_type =
  hdmi_tv_res = 1080p
  hdmi_monitor_res = 1024x768
  hdmi_disable_overscan = 0
  hdmi_system_only = 0
  usbroot = 0
  usbboot = 0
  rootdev = /dev/mmcblk0
  rootpartition = /dev/mmcblk0p2
  rootfstype = ext4
  rootfs_mkfs_options =
  rootfs_install_mount_options = noatime,data=writeback,nobarrier,noinit_itable
  rootfs_mount_options = errors=remount-ro,noatime
  final_action = reboot
  quiet_boot = 0
  disable_raspberries = 0
  disable_splash = 0
  cleanup = 0
  cleanup_logfiles = 0
  spi_enable = 0
  i2c_enable = 0
  i2c_baudrate =
  sound_enable = 0
  sound_usb_enable = 0
  sound_usb_first = 0
  camera_enable = 0
  camera_disable_led = 0
  use_systemd_services = 0

OTP dump:
  16:00280000
  17:1020000a
  18:1020000a
  27:00005050
  28:b9dd1201
  29:4622edfe
  30:00a01041
  45:472c0309

Waiting 5 seconds.....
Waiting for /dev/mmcblk0...
  Disk /dev/mmcblk0: 59.7 GiB,<PHONE_NUMBER>8 bytes, 125149184 sectors
  Disklabel type: dos
  Disk identifier: 0x30d03f0b
OK
Applying new partition table... OK
Initializing /boot as vfat... OK
Copying /boot files in... OK
Initializing / as ext4... mke2fs 1.44.5 (15-Dec-2018)
ext2fs_check_if_mount: Can't check if filesystem is mounted due to missing mtab file while determining whether /dev/mmcblk0p2 is mounted.
  Discarding device blocks: done
  Creating filesystem with 15609856 4k blocks and 3907584 inodes
  Filesystem UUID: 12614d1e-2ab8-4818-8616-d2802d8e47a2
  Superblock backups stored on blocks:
  	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
  	4096000, 7962624, 11239424

  Allocating group tables: done
  Writing inode tables: done
  Creating journal (65536 blocks): done
  Writing superblocks and filesystem accounting information: done

OK
Mounting new filesystems... OK
Starting HWRNG... OK

Starting install process...
  P: Retrieving Release
  P: Retrieving Release.gpg
  P: Validating Release
  I: Good signature from "Mike Thompson (Raspberry Pi Debian armhf ARMv6+VFP<EMAIL_ADDRESS>  P: Parsing Release
  P: Retrieving Packages.xz
  P: Validating Packages.xz
  P: Parsing Packages
  W: parser_rfc822: Iek! Don't find end of value!
  E: Internal error: download

  ERROR: 1, trying again (2/5)...
  P: Retrieving Release
  P: Retrieving Release.gpg
  P: Validating Release
  I: Good signature from "Mike Thompson (Raspberry Pi Debian armhf ARMv6+VFP<EMAIL_ADDRESS>  P: Parsing Release
  P: Validating Packages
  P: Parsing Packages
  W: parser_rfc822: Iek! Don't find end of value!
  E: Internal error: download

  ERROR: 1, trying again (3/5)...
  P: Retrieving Release
  P: Retrieving Release.gpg
  P: Validating Release
  I: Good signature from "Mike Thompson (Raspberry Pi Debian armhf ARMv6+VFP<EMAIL_ADDRESS>  P: Parsing Release
  P: Validating Packages
  P: Parsing Packages
  W: parser_rfc822: Iek! Don't find end of value!
  E: Internal error: download

  ERROR: 1, trying again (4/5)...
  P: Retrieving Release
  P: Retrieving Release.gpg
  P: Validating Release
  I: Good signature from "Mike Thompson (Raspberry Pi Debian armhf ARMv6+VFP<EMAIL_ADDRESS>  P: Parsing Release
  P: Validating Packages
  P: Parsing Packages
  W: parser_rfc822: Iek! Don't find end of value!
  E: Internal error: download

  ERROR: 1, trying again (5/5)...
  P: Retrieving Release
  P: Retrieving Release.gpg
  P: Validating Release
  I: Good signature from "Mike Thompson (Raspberry Pi Debian armhf ARMv6+VFP<EMAIL_ADDRESS>  P: Parsing Release
  P: Validating Packages
  P: Parsing Packages
  W: parser_rfc822: Iek! Don't find end of value!
  E: Internal error: download

  ERROR: 1

Error: The installation could not be completed!
  2 retries left.
  You have 10 seconds to press ENTER to get a shell or it will be retried automatically.

Would be awesome if we could get this fixed. Thanks
[1]: On an existing fresh install of buster - I just edited apt sources to point to bullseye from the raspberrypi.org mirror, and did the usual update/upgrade/dist-upgrade etc dance. Rebooted OK & came up no issues.

I'm interested in this aswell

I'm stuck with this. The workaround that works for me is using the installer with buster and then upgrade using #225 (don't forget to look at my last comment though!).

When are we having an official release for bullseye?

This is such a great project that has been useful to me for so many years, I really hope to see bullseye support soon. Cheers :+1:

Will be fixed in the upcoming release!


Way to use pd_zurg with Overseerr

First of all, thanks for this project I really love it!
Is your feature request related to a problem? Please describe.
As far as I know Plex Debrid is able to connect to Overseerr to request TV Shows and Movies so it's added to Debrid service
Describe the solution you'd like
Have switches to connect to Overseerr and configure Plex Debrid automatically
Describe alternatives you've considered
Configuring Plex Debrid manually within PD_Zurg, but I couldn't figure it out.

Nice! Thanks for your quick response!


Generate for video

Hello, it's me again :smile:
Here is a big PR which i have been working for a long time, I tried to decompose the important steps in meaningfull commit, although I haven't checked the build between each commit
The main point of this branch is to be able to work on big datasets with any images, and to work with point clouds that are more noisy and less dense than faro. I ensured that functionality regarding ETH3D initial dataset was not altered. I also changed the
As such here are the features that I added to improve speed :

Add Multithreading with openmp wherever possible : GroudtruthCreator, SplatCreator, and for occlusion boundaries
Prevent the program to filter out occlusion boundaries for splats : by onstruction they don't have any, so now the occlusion splats are a different parameter than occlusion mesh (i adapted the README accordingly)
Make the problem object only load images if necessary. For videos, you may have a memory error because of that. GroundtruthCreator is currently the only tool where there is a change. The rationale behind it is that you can run the image registrator with a small subset of the video frames, and interpolate the deformation after that only to create ground truth on the whole video.

Here are the features to deal with low quality occlusion meshes :

Occlusion boundaries does not assume the mesh to be on a perfect manifold. This is particuarly true for mesh constructed with unordered point clouds : Poisson Meshing is not possible anymore, and you might use PCL's greedy projection. As such, the code is robust to mesh edges with more than two faces. Besides, it automatically filters out very flat edges which have a very low chances of actually be an occlusion boundary.
The code now accepts meshlab projects as well as simple ply. This is interesting if you need to finetune the point cloud registration with respect to the colmap model, the same way as the lidar scans.
2D splats size are no longer a fixed size in pixels. Instead, a value in meters is given for splat radius and the 2D splat is the same size as if it was a 3D object of size splat radius oriented toward the camera. This is interesting for noisy meshes with many occlusion boundaries, where if far enough, would be considered only occlusion boundaries, and thus hide all possible GT points

Here are the remaining feature added :

Some log info for Ground truth Creator and Splat Creator
I changed the range of possible cloud transformations, and added scale to SO3, hence Sim3. This does not affect ICPScanAligner (although it would be nice) nor ImageRegistrator, where everything is kept SO3. The only diffrence is when there is scale information in the matrix given in the meshlab project. Note that for that, I had to update Sophus.
I added max occlusion depth options, I personally dealed with a 200m wide models, so I had to change it
Depth Maps and Occlusion Depth Maps can now be compressed on the fly before being saved in GroundTruthCreator
You can now launch DatasetInspector without multiscale pointcloud just to check depth maps. (it will warn you though)

Thanks for your help and for accepting my former PR.
I have been using your code for my own research on groundtruth depth for videos, I hope I will be able to share it with you soon, using your code was very instructive !
Best, Clément

Hi, thank you very much for the new PR! The improvements and additions sound very good to me, however I don't think I will have time to review the changes in depth now, I am sorry. I would thus suggest to merge these changes without review. Would it be fine for you if I'll give you admin rights for this repository with the ability for direct commits, so you can make any potential further changes directly?

hi, thanks for trusting me :)
I see no problem in that.
This PR is most likely my last one before sometime, because it now fits my needs in my current project. But I'll be sure to update it as soon as some improvement can be done !
Clément

Great. Thanks again for the PR and good luck with your project!


Stop block creators from knowingly creating blocks in the past

Should it be possible to create multiple blocks on the same exact timestamp?

@LeeSmet a problem in the consensus or just in the implementation of the blockcreator?

The specific problem being refered to:
The consensus allows blocks to be created some time in the future, and be immediately accepted. This prevents local forking behavior in case the local time is lagging a bit behind the rest of the network nodes. At the same time, the POBS protocol evaluates blocks up to 10 seconds in the future. So if a block is found by the block creator to be valid if created at time(now + 10 secs), it will create the block now, set the block timestamp to the aforementioned timestamp (which is in the future), and submit the block to the cs. The block is accepted, and propagated. Now other block creators (or the same block creator) try to create blocks again (having added the future block, which has a timestamp higher than the block creators current timestamp). If a block is now found at time(now + 1sec), then the block is once again created and submitted (and accepted), but with a lower timestamp than the previous block.
The reason the Cs allows this to happen is due to the mediantimestampwindow variable. This variable takes the median timestamp of the last few (in tfchain 11 or so iirc) blocks, and only verifies the new bock timestamp to be higher than the median of these few blocks

We do have to make sure that 1 badly behaving blockcreator ( might be on purpose) does not limit the blockcreation chances of the rest( and as such increases his).


Einthusan3 on kodi 19.1

I am getting error like “please check and update the base URL in Addon settings and restart the addon”
device : android 9 tv
kodi 19.1
eintamil einthusan3

Hi, please uninstall the add-on and it's data. Reinstall and try again.
Also, check there is no records in your hosts file. Please check add-on
settings in Expert mode and verify the Base_URL field value or reset it to
default. Please attach the screenshot.

Hello, Is this issue resolved. If yes, please close this ticket.