Binance: An existing connection was forcibly closed by the remote host

Hi, I think I need some help here...
I keep getting this error whenever I try to go live (it happens after around 7 min I start the bot):
2021-04-27 15:35:00 | XRPUSDT (BEAR) | 5m | Close: 1.4 | ^ EMA12/26: 1.39 > 1.39 ^ | ^ MACD: 0.0 > -0.01 ^ | ^ OBV: 59019460.33 (3.66%) ^ | ERI: sell | WAIT | Last Action: BUY | 0% (delta: 1.41) 2021-04-27 15:40:00 | XRPUSDT (BEAR) | 5m | Close: 1.42 | ^ EMA12/26: 1.4 > 1.39 ^ | ^ MACD: 0.0 > -0.01 ^ | ^ OBV: 61230004.87 (7.12%) ^ | ERI: buy | WAIT | Last Action: BUY | 0% (delta: 1.42) Traceback (most recent call last):   File "C:\Programs\Python\3.9\env\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen     httplib_response = self._make_request(   File "C:\Programs\Python\3.9\env\lib\site-packages\urllib3\connectionpool.py", line 445, in _make_request     six.raise_from(e, None)   File "<string>", line 3, in raise_from   File "C:\Programs\Python\3.9\env\lib\site-packages\urllib3\connectionpool.py", line 440, in _make_request     httplib_response = conn.getresponse()   File "C:\Users\al\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1345, in getresponse     response.begin()   File "C:\Users\al\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 307, in begin     version, status, reason = self._read_status()   File "C:\Users\al\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 268, in _read_status     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")   File "C:\Users\al\AppData\Local\Programs\Python\Python39\lib\socket.py", line 704, in readinto     return self._sock.recv_into(b)   File "C:\Users\al\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1241, in recv_into     return self.read(nbytes, buffer)   File "C:\Users\al\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1099, in read     return self._sslobj.read(len, buffer) ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
Here is my config file:
{     "binance" : {         "api_url" : "https://api.binance.com",         "api_key" : "",         "api_secret" : "",         "config" : {             "base_currency" : "XRP",             "quote_currency" : "BRL",             "granularity" : "15m",             "live" : 1,             "verbose" : 0         }     },     "telegram" : {         "token" : "",         "client_id" : ""     } }
I changed the quote_currenct to USDT but still got the same error.

I guess this could be the reason:
"Why are my WebSocket connections being dropped/disconnected?
Every 3 minutes, active WebSocket connections will receive a ping. If you do not respond with a properly framed pong, your connection will be terminated."
https://www.binance.com/en/support/faq/360004492232

I'm not using WebSockets, so I don't think this is related. I'n not sure if your config and your debug output are related. In your config you are trying 15 minute intervals but in your output you are trading 5 minute intervals. I personally would not trade anything less than 15 minute intervals (and only in really good market conditions). With such short intervals you probably would make losses from not covering your fees. The best option is to actually not specify the granularity at all. The bot will then "smart switch" between the 1 hour and 15 minute intervals automatically. This gives the best results.

Hi @whittlem ,  thanks for the hint with eh granularity and the smart switch.
I've changed it and I still got the error though.
I did some tests here, and it seems the poll request takes too long to happen. it's currently set to 300 sec. I just tested it for 180 seconds and now the code is not raising any run-time error.
            # poll every 5 minute
            list(map(s.cancel, s.queue))
            s.enter(180, 1, executeJob, (sc, app))
Log:
2021-04-28 11:00:00 | XRPBRL (BULL) | 1h | Close: 7.46 | v EMA12/26: 7.49 < 7.51 v | v MACD: -0.03 < 0.02 v | v OBV: -377454.6800 (-10.64%) v | ERI: buy | WAIT | Last Action: SELL 2021-04-28 07:27:51 | XRPBRL (BULL) | 1h | Current Price: 7.513 2021-04-28 07:30:56 | XRPBRL (BULL) | 1h | Current Price: 7.499
You may check the configuration and the error below: when running with 300 seconds for the poll
{     "binance" : {         "api_url" : "https://api.binance.com",         "api_key" : "",         "api_secret" : "",         "config" : {             "base_currency" : "XRP",             "quote_currency" : "BRL",             "live" : 1,             "verbose" : 0, 			"smartswitch" : 1         }     },     "telegram" : {         "token" : "",         "client_id" : ""     } }
Error:
`2021-04-28 11:00:00 | XRPBRL (BULL) | 1h | Close: 7.5 | v EMA12/26: 7.5 < 7.52 v | v MACD: -0.03 < 0.02 v | v OBV: -393008.3400 (-6.95%) v | ERI: buy | WAIT | Last Action: SELL
2021-04-28 07:23:36 | XRPBRL (BULL) | 1h | Current Price: 7.459
Traceback (most recent call last):
File "C:\Programs\Python\3.9\env\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "C:\Programs\Python\3.9\env\lib\site-packages\urllib3\connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "C:\Programs\Python\3.9\env\lib\site-packages\urllib3\connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "C:\Users\al\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1345, in getresponse
response.begin()
File "C:\Users\al\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 307, in begin
version, status, reason = self._read_status()
File "C:\Users\al\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 268, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "C:\Users\al\AppData\Local\Programs\Python\Python39\lib\socket.py", line 704, in readinto
return self._sock.recv_into(b)
File "C:\Users\al\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1241, in recv_into
return self.read(nbytes, buffer)
File "C:\Users\al\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1099, in read
return self._sslobj.read(len, buffer)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Programs\Python\3.9\env\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "C:\Programs\Python\3.9\env\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "C:\Programs\Python\3.9\env\lib\site-packages\urllib3\util\retry.py", line 532, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Programs\Python\3.9\env\lib\site-packages\urllib3\packages\six.py", line 734, in reraise
raise value.with_traceback(tb)
File "C:\Programs\Python\3.9\env\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "C:\Programs\Python\3.9\env\lib\site-packages\urllib3\connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "C:\Programs\Python\3.9\env\lib\site-packages\urllib3\connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "C:\Users\al\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 1345, in getresponse
response.begin()
File "C:\Users\al\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 307, in begin
version, status, reason = self._read_status()
File "C:\Users\al\AppData\Local\Programs\Python\Python39\lib\http\client.py", line 268, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "C:\Users\al\AppData\Local\Programs\Python\Python39\lib\socket.py", line 704, in readinto
return self._sock.recv_into(b)
File "C:\Users\al\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1241, in recv_into
return self.read(nbytes, buffer)
File "C:\Users\al\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1099, in read
return self._sslobj.read(len, buffer)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Programs\Python\cryptobot\pycryptobot\pycryptobot.py", line 1063, in
s.run()
File "C:\Users\al\AppData\Local\Programs\Python\Python39\lib\sched.py", line 151, in run
action(*argument, **kwargs)
File "C:\Programs\Python\cryptobot\pycryptobot\pycryptobot.py", line 1019, in executeJob
account.saveTrackerCSV(app.getMarket())
File "C:\Programs\Python\cryptobot\pycryptobot\models\TradingAccount.py", line 293, in saveTrackerCSV
df = self.getOrders(market, '', 'done')
File "C:\Programs\Python\cryptobot\pycryptobot\models\TradingAccount.py", line 94, in getOrders
resp = self.client.get_all_orders(symbol=market)
File "C:\Programs\Python\3.9\env\lib\site-packages\binance\client.py", line 1803, in get_all_orders
return self._get('allOrders', True, data=params)
File "C:\Programs\Python\3.9\env\lib\site-packages\binance\client.py", line 320, in _get
return self._request_api('get', path, signed, version, **kwargs)
File "C:\Programs\Python\3.9\env\lib\site-packages\binance\client.py", line 265, in _request_api
return self._request(method, uri, signed, **kwargs)
File "C:\Programs\Python\3.9\env\lib\site-packages\binance\client.py", line 259, in _request
self.response = getattr(self.session, method)(uri, **kwargs)
File "C:\Programs\Python\3.9\env\lib\site-packages\requests\sessions.py", line 555, in get
return self.request('GET', url, **kwargs)
File "C:\Programs\Python\3.9\env\lib\site-packages\requests\sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "C:\Programs\Python\3.9\env\lib\site-packages\requests\sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "C:\Programs\Python\3.9\env\lib\site-packages\requests\adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))`

Did this process get fixed, I'm having the same issue, I'm on UM_Futures and the same things is happening. I will get a buy trade go thru and then I will wait unitl a cross happens and then it will give me the connection reset error. Please help !


Mouse input support not working

Environment
Windows build number: 10.0.19041.423
Windows Terminal version (if applicable): Version: 1.4.3243.0

Steps to reproduce
Installed Version: 1.4.3243.0 from the Github release page. According to https://devblogs.microsoft.com/commandline/windows-terminal-preview-v0-10-release/, it should support mouse input, but my terminal doesn't seem to. How do I fix it?
Expected behavior
Should support mouse input (example: should be able to select panes in tmux via mouse click)
Actual behavior
Unable to switch panes in tmux using mouse.

Since you are not using "any other software" and have deleted it from the template, I can't help you.

If I had to guess, it's because you're using ssh.exe version 7.7 that comes with Windows. That version of SSH blocks mouse input.

yup
ssh -V
OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5.

What do you mean by "any other software" ? I was only using a WSL terminal. Does that use ssh underneath?

So, this terminal mostly runs Windows applications. The one you’re using is the version of SSH that was shipped with Windows :smile: which unfortunately has broken mouse support.
(Terminal can also run WSL, and it does so well. If you use SSH inside WSL, mouse support will work fine!)

I was trying to follow the instructions provided here https://github.com/PowerShell/Win32-OpenSSH/wiki/Install-Win32-OpenSSH to upgrade the ssh to release 8.1. I am stuck at step 3 - Make sure binary location has the Write permissions to just to SYSTEM, Administrator groups Would appreciate any help. (sorry, I am novice developer in Windows)


take allow to do one extra step

Hi,
I noticed strange behavior of the take function. It returns right result, but it doesn't end reduce immediately, so one extra step is executed:
var range = function(from, to) {
  return function*() {
    for (var i = from; i <= to; i++) { console.log('yield', i); yield i };
  }();
};

console.log(R.transduce(R.take(2), R.flip(R.append), [])(range(10, 2000000)));

yield 10
yield 11
yield 12 <--- this yield shouldn't be executed
[10, 11]

With two composed functions it does two extra useless steps:
var even = function(n) { console.log('even', n); return n % 2 === 0; };

console.log(R.transduce(R.compose(R.filter(even), R.take(2)), R.flip(R.append), [])(range(10, 2000000)));

yield 10
even 10
yield 11
even 11
yield 12
even 12
yield 13
even 13
yield 14
even 14
[10. 12]

This can be fixed by checking the remaining items earlier:
Take.prototype.step = function(value, item){
  if(this.n-- > 0){
    value = this.xf.step(value, item)
  }
  if(this.n <= 0){
    value = reduced(value)
  }
  return value
}

Probably a better way to write that so it's not dependent on the post-increment. This might have been a casualty  of a simplification of take during review of the original PR.


Update many with returning

PR Info

Closes https://github.com/SeaQL/sea-orm/issues/1669

New Features

[x] Added UpdateMany::exec_with_returning() method

I would need this if it can be merged :)

This is on my review pipeline


Allow targets to specify output of the rule

It's important that users of the rule can change the output.
For instance, we want to be able to generate "manual.html" and "manual.txt":
pandoc(
  name = "html",
  src " manual.md",
  from_format = "markdown",
  to_format = "html",
)
pandoc(
  name = "txt",
  src " manual.md",
  from_format = "markdown",
  to_format = "plain",
)

Today, the rule forces the output to be html.html and txt.txt.
Of course, you can argue that the name of the rule is used, and I could have "manual.html". However, two labels cannot be identical, so there is no easy way to generate "manual.txt".
A workaround could be to add a genrule that copies "txt.txt" to "manual.txt".
Instead, I propose to replace the pattern generated ctx.outputs by an output derived from ctx.attr.

I've put too many things in this PR. Sorry! Let's split it in more actionable smaller pull requests.

Please take another look.

PTAL


[SPARK-24234][SS] Reader for continuous processing shuffle

What changes were proposed in this pull request?
Read RDD for continuous processing shuffle, as well as the initial RPC-based row receiver.
https://docs.google.com/document/d/1IL4kJoKrZWeyIhklKUJqsW-yEN7V7aL05MmM65AYOfE/edit#heading=h.8t3ci57f7uii
How was this patch tested?
new unit tests

@HeartSaVioR @arunmahadevan @xuanyuanking

Can one of the admins verify this patch?

Test build #90656 has started for PR 21337 at commit 1d6b718.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90656/
Test FAILed.

Test build #90705 has started for PR 21337 at commit 46456dc.

@tdas

Test build #90706 has started for PR 21337 at commit 2ea8a6f.

Test build #90705 has finished for PR 21337 at commit 46456dc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90705/
Test FAILed.

Test build #90706 has finished for PR 21337 at commit 2ea8a6f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90706/
Test FAILed.

Test build #90735 has started for PR 21337 at commit 955ac79.

Test build #90735 has finished for PR 21337 at commit 955ac79.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90735/
Test PASSed.

Actually, now that I think about it, there's a lot of value in having that interface, both to make sure we don't accidentally leak across it and possibly allow debugging mode in the future. I'll add it.

Test build #90761 has started for PR 21337 at commit 97f7e8f.

Test build #90762 has started for PR 21337 at commit de21b1c.

Test build #90761 has finished for PR 21337 at commit 97f7e8f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait ContinuousShuffleReader

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90761/
Test FAILed.

Test build #90762 has finished for PR 21337 at commit de21b1c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90762/
Test PASSed.

Test build #90814 has started for PR 21337 at commit 154843d.

other than a few nits, LGTM.

Test build #90817 has started for PR 21337 at commit 00f910e.

LGTM. Will merge after tests pass.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90814/
Test PASSed.

Test build #90817 has finished for PR 21337 at commit 00f910e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90817/
Test PASSed.


creates script to run monthly stats email task

Kubernetes CronJob runs /app/bin/monthly_stats_mail, on a configurable schedule.

Totally agree. I think we can have the container setup the env and then maybe use positional arguments for the commands. That would require a bit more reworking of things but I'll certainly think on it


Error: No such method "code.set" for Redactor

Using Redactor version 10.2.3 along with Browserify and Gulp to include my front-end assets.
The error is coming from like 65, but goes away if you set
element.redactor('code.set', ngModel.$viewValue || '');

to
element.redactor('set', ngModel.$viewValue || '');

However, this then causes a new "Error: No such method "placeholder.toggle" for Redactor" error.

Something about Browserify and Gulp was causing my app to use an old version of Redactor - once it loaded the new version the error disappeared

I have same issue. I'm using Redactor version 10.2.4 and latest angular-redactor.

@arevutsky you have to take the angular-redactor-9.x.js ;)


Concepts work: how do users refine their search queries?

What is the problem to solve?
Researchers are unsure of what the Collection contains so they try progressively more specific queries
What stage of research is it? (discovery/general usability testing/post release)
discovery
What are the objectives?
If we can identify the areas of interest to researchers we can use this to inform the concepts offer.
What are the assumptions?
Most users of search are hobbyists who search once or twice, find something and leave.
Professional researchers will search multiple times.
What research technique is required?
Jupyter notebooks

I'm not going to push this Jupyter notebook to the data-science repo as it seems to be buried in a git stash. This analysis was to validate the uncontentious assumption that researchers use subject-related queries, so unlikely to be missed.


Update version number in readme

The most recent release is 0.3.0 but the readme still shows:
[dependencies]
jemallocator = "0.1.8"

Maybe add a comment next to the version number in Cargo.toml as a reminder to bump the version number in the readme when releasing breaking changes.

Sounds like a great idea!


Document how to wrap an eyre::Report with a different error type

Wrapping a source error type as the cause of an eyre::Report is very straightforward with the wrap_err function. But suppose I have an eyre::Result<()> and want to convert it into a Result<(), ()> without discarding the helpful backtrace/spantrace from eyre? Is that even possible?
If it is, can it be documented?

But suppose I have an eyre::Result<()> and want to convert it into a Result<(), ()> without discarding the helpful backtrace/spantrace from eyre? Is that even possible?

As stated, no, you definitely cannot convert an eyre::Result<()> to a Result<(), ()> without losing all relevant information. I think I may be misunderstanding what you mean, but based on what you wrote, you're essentially asking for Result<(), eyre::Report> -> Result<(), ()> which can be reduced to eyre::Report -> (), aka converting eyre::Report to the unit type. This a drop, and can only be accomplished by discarding the original eyre::Report. All relevant information to an error such as backtrace and spantrace are stored in the eyre::Report, not the Result. We cannot discard this Report type without losing all of that helpful information.

you're essentially asking for Result<(), eyre::Report> -> Result<(), ()> which can be reduced to eyre::Report -> (), aka converting eyre::Report to the unit type

Yeah that sums it up pretty well and it makes sense why that would not be viable. I guess what I'm really looking for is a way to continue to have whatever error type E my Result is supposed to use, but still propagate the helpful backtrace info that eyre contains. I can create a new Report but since its not a Report<E> I cannot access the wrapped error easily.

I'm not sure I understand. Can you give me an example of what you're trying to do?

Wow I left this for a full year lol.
I believe what I was asking for was a way to propagate the information about the chain of underlying errors to other error error libs like thiserror.
I wanted to be able to use eyre in a library for prototyping easy errors, and then have a parent library that uses thiserror that handles error from all subsystems, some of which may be using eyre. But I didn't want to lose the helpful error chain.


Profile photo modal not completely visible on mobile screen.

Describe the bug
The Profile photo modal doesn't completely visible on a mobile screen, the waving hand emoji icon overlaps the modal. (See screenshot)
To Reproduce
Steps to reproduce the behaviour:

Open any page of the website on mobile screen.
Click on the Avatar Modal (Please log in first)
See error

Expected behaviour
The profile photo modal should be on top of everything and completely visible when a user clicks on the profile photo.
Screenshots

Hi @rishipurwar1, I'd like to work on this

@drishit96 Assigned to you.
Note: Please make a PR to the development branch NOT in the main branch and test your changes on all the pages where we're using this component before making a PR.

Looking forward to seeing your PR! Feel free to give a star⭐ to this repo.


Remove Express dependency from GitHub API calls

Task

Remove Express dependency as discussed given that we do not have a backend

Deliverables

GitHub API calls without express. Related to Issue #42

Due 12/6

Removed Express dependency.


Adding Azure Data Studio Extension Tutorials

Adding tutorials for Dashboards, Notebooks, and Jupyter Notebook Wizard extensions
Creating section for tutorials in toc.yml, and moving Keymap Tutorial there

(collaborated with @laurajjiang )

@anjalia : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@anjalia
Thank you for this new content.
However, we are moving this PR to our private repo for review and further edits.
I will reach out for questions and clarity on a few things before we publish these changes/additions.
Thank you.
@yualan  for awareness.

@markingmyname for tracking

Changes moved to private repo - https://github.com/MicrosoftDocs/sql-docs-pr/pull/16777


Use ? instead of !!

It can cause KotlinNullPointerException when fragment's view was not created

This is intentional and indicates an ordering problem in your calling code.


CNV-48188: Fix networkpolicy list

fix bug adding unmountOnExit
remove the label filter as networkpolicy and multinetwrokpolicy do not need them

Before
https://github.com/user-attachments/assets/e3d9b41c-8175-47fe-b821-c93f24cfa9c5
After
https://github.com/user-attachments/assets/f30379da-3762-49b4-9c6f-a53ba053810f

/cherry-pick release-4.17


practical performence

Thank you for implementing the code in this article, but I would like to know what the three indicators (F1 value, recall, and precision) are generally in practice.

Hi, I haven't been able to get a satisfactory performance, you may try running the code on yourself. Since the code is not official, there might be some bugs.


traefik: 2.8.5 -> 2.8.8

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/traefik/traefik/releases.
meta.description for traefik is: "A modern reverse proxy"
meta.homepage for traefik is: "https://traefik.io"
meta.changelog for traefik is: "https://github.com/traefik/traefik/raw/v2.8.7/CHANGELOG.md"
Updates performed

Golang update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 2.8.7 with grep in /nix/store/8yzi4dz28zifwi47a5d7baflw7bw1qmp-traefik-2.8.7
found 2.8.7 in filename of file in /nix/store/8yzi4dz28zifwi47a5d7baflw7bw1qmp-traefik-2.8.7

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
traefik

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/8yzi4dz28zifwi47a5d7baflw7bw1qmp-traefik-2.8.7 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A traefik https://github.com/r-ryantm/nixpkgs/archive/efd6ef0762e2269502a434744e94dbc40d08d803.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/8yzi4dz28zifwi47a5d7baflw7bw1qmp-traefik-2.8.7
ls -la /nix/store/8yzi4dz28zifwi47a5d7baflw7bw1qmp-traefik-2.8.7/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

traefik

Maintainer pings
cc @vdemeester for testing.

Result of nixpkgs-review pr 192838 run on x86_64-linux 1

1 package built:

traefik

Result of nixpkgs-review pr 192838 run on x86_64-linux 1

1 package built:

traefik


Generation of conda.yaml from same.import(...)

Closing as I think this feature has fallen to the wayside.


ui: add fade animation in the setting dropdown-menu

This commit will add fade-in effect to the settings dropdown menu like we have in other ui elements(buddy info, stream popover)

Pr to fix #13916
GIFs or Screenshots:

I tried testing it on czo and I think these menus fadeout as well. This was just a few minutes of clicking around, but I think we might want to add that too.

I am personally in favor of going in the other direction, which is to remove fade-out effects for popup menus.  I think there's room for debate here (my opinion is mostly subjective).  I can also see the value of this fix in the short term--we can at least be consistent.

I also noticed that the settings menu has another odd behaviour. It does not close when other "menus" (stream popover, user info popover, etc) open nor do they close when I click on the settings menu.
This behaviour, although understandable, seems a little inconsistent, and sometimes even blocks the screen.

I agree with you @showell it's better to remove fade-out effects from popovers for making popovers close quickly and having fade-in effects only when opening the for better UX

Closing as I don't think this needs an animation.


Add links to example.rst to readthedocs

Add hyperlinks to readthedocs documentation on text that recommend using specific methods

Example
Change
[...] the :py:meth:`clean_names` method. Firstly, we pass [...]

to
[...] the `:py:meth:clean_names <https://pyjanitor.readthedocs.io/api.html#janitor.functions.clean_names>`_ method. Firstly, we pass [...]

Ok! Thanks for raising this issue, @jQwotos :) I will mark this as taken by you.

Taking a look more into sphinx and seeing if this can be done in a better way

@jQwotos just wanted to let you know, a number of RST files have been converted to ipynbs, and now they render on the docs page!

@jQwotos just wanted to let you know, a number of RST files have been converted to ipynbs, and now they render on the docs page!

Awesome! Shall I continue to look into this or shall we mark it is closed then

Let's mark this one as closed. However, because I've seen your effort in making this happen, let's consider your docs contribution done :smile:. Do feel free to tackle a code contribution at this point!


Create documentation on CAP-51, secp256r1 curve, webauthn, and passkey

What problem does your feature solve?
Coming back from offsite we need to document these concepts. Requested from ecosystem as well
Relevant link
https://github.com/stellar/stellar-protocol/blob/master/core/cap-0051.md

On it! I'm starting on this one next week. We'll have it ready by the 18th.

Thank you @briwylde08!

https://github.com/stellar/stellar-docs/pull/677


fix(vite): remove postcss-url and duplicate postcss-import

🔗 Linked issue
resolves https://github.com/nuxt/nuxt/issues/15765
❓ Type of change

[ ] 📖 Documentation (updates to the documentation, readme or JSdoc annotations)
[x] 🐞 Bug fix (a non-breaking change that fixes an issue)
[ ] 👌 Enhancement (improving an existing functionality like performance)
[ ] ✨ New feature (a non-breaking change that adds functionality)
[ ] 🧹 Chore (updates to the build process or auxiliary tools and libraries)
[ ] ⚠ Breaking change (fix or feature that would cause existing functionality to change)

📚 Description
This was a fun deep-dive.
Investigating the linked issue, I found that we are adding postcss-import - and so is vite. In addition, I believe the key features postcss-url (which was causing the bug) are in fact already covered by Vite's list of features:

Vite improves @import resolving for Sass and Less so that Vite aliases are also respected. In addition, relative url() references inside imported Sass/Less files that are in different directories from the root file are also automatically rebased to ensure correctness.
https://vitejs.dev/guide/features.html#css-pre-processors

Accordingly, this PR removes both plugins (or rather, doesn't add them).
I suspect removing these two plugins will improve our performance as well as fix the bug. I would also consider we reconsider whether to include cssnano or defer to vite's esbuild/lightningcss minificiation step in future (cc: antfu).
📝 Checklist

[ ] I have linked an issue or discussion.
[ ] I have updated the documentation accordingly.

Shouldn't this mention/link to #20533 since all the related issue were merged into that one ?

For CSS minification, I am not sure if users would want full control over which tool to use, maybe could be an option as 'cssnano' | 'esbuild' | 'lightningcss' where we built a thin layer of adapter?

I think that would be a very nice idea.

@danielroe @antfu

For CSS minification, I am not sure if users would want full control over which tool to use, maybe could be an option as 'cssnano' | 'esbuild' | 'lightningcss' where we built a thin layer of adapter?

Full control is good, but also with the most performant (sensible) default (likely lightningcss). I was recently looking at whether or not something like nuxt-modules/critters or nuxt-beastcss was even necessary, or if Nuxt 3 already accomplishes the same goals.
There's not much in the docs other than Nuxt is pre-configured to use cssnano for "minification and purge" (at the end of "Using Postcss") https://nuxt.com/docs/getting-started/styling#using-postcss - but nothing specifically about best practices of inlining critical css.
I know there's the inlineSSRStyles experimental flag defaulting to true, but unsure if that's doing the same thing that critters / beastcss are or not.
Anyway, ideally Nuxt 3 takes care of all of the above for us (inlined critical CSS, fastest minification and purging), and maybe explains the optimal pre-configured choice (lightning) - but then lets us opt for other less performant options if it's somehow a project requirement.


Defintion of baseline selectivity?

https://github.com/pacific-hake/hake-assessment/blob/1a62b9a85159656bb3aab4f85aa75ad3839ed255/doc/executive-summary/executive-summary.rnw#L797

Sorry I was trying to use VS Code to make an issue, so duplicate of #1017, so closing this one.


Need getCurrentPosition() method.

To get Current playing position is a default method in Android media player and I also see it implemented in the 'Player' Abstract class. And there is no method from the plugin. Not sure regarding iOS side. I guess its pretty straight forward to include this method in the plugin. Would be really helpful if added.
Thanks.

This is already implemented by the PositionEvent
https://github.com/luanpotter/audioplayers#position-event

Yes, I checked it already. But the problem with 'onAudioPositionChanged' Stream is that it fire roughly every 200 milliseconds. Since the project that i'm working with needs precise duration of the audio being played (For manipulating word by word text highlight with reference to the audio which needs exact audio timings and also for phrase repeat for learning the words kinda functionality), This 'onAudioPositionChanged' solution will not help.
Moreover, this is a stream, which is like a workaround. What i'm looking for is a Straight method that just does this as in MediaPlayer.getCurrentPosition() in Android and AVAudioPlayer.currentTime in iOS. Guess this is a meaning request and also easy to accomodate. Since there is no other plugin that does this(getCurrentPosition) and also 'AudioPlayers' only has muple audio playing option, i choosed this one.
Waiting for a positive response.
Ref from Code:
/// Stream for subscribing to audio position change events. Roughly fires
/// every 200 milliseconds. Will continuously update the position of the
/// playback if the status is [AudioPlayerState.PLAYING].
Stream get onAudioPositionChanged => _positionController.stream;

Fair enough, I will re open this and try to take a look, although I don't know how long will take to resolve since we are a bit overwhelmed, so if you can contribute with a PR, would be a lot a help.

Fixed by #220


Possible Codec Between SPARSE and DENSE: CHIMERA

Issue Outline
In the threshold between DENSE and SPARSE (5120 elements), compression is not great and select queries on SPARSE are relatively slow due to the binary_search.
Here is what a codec between sparse and dense could look like to increase performance and compression.
CHIMERA codec
Conceptually it is similar of what we do top level by creating blocks of size u16::MAX, but one level lower on the block level. It combines the index part of the DENSE (vals before) codec with the encoding of the SPARSE codec, therefore the name CHIMERA.
On a side note this actually a variation of datastructure I'm testing to speed up term aggregations.
A block is u16::MAX elements. From this value space we create 256 Blocks each with up to 256 elements.
Create an index with an entry for each block [VALS BEFORE;u16; ..x256].
Since we have only 256 elements in a chimera block we can use u8, as apposed to u16 in the SPARSE codec, to encode elements.
The binary search would have at most 8 steps as opposed to 13 steps with the SPARSE codec. We can also assume that a binary_search can on average be executed on ~2 cachelines, in contrast to ~11 cachelines (number of binary_search steps).
VALS BEFORE is the same as in the dense codec to speed up lookups. It also is used to compute the number of elements in a block and where they are encoded as every element is exactly 1 byte.
rank
direct access block + binary search in block
select
direct access value in data vec + binary_search on the metadata to find the block. We could have a select cursor similar to the existing one.
CHIMERA codec cost
2 bytes metadata per block: 256 * 2 = 512 bytes fixed codec + 1byte per element.
Thresholds
The current threshold is 5120 elements to switch between SPARSE and DENSE.
512 elements in a block before SPARSE is more compact than CHIMERA.
10240 bytes fixed cost for the DENSE codec (based on the threshold of 5120).
=> 9728 Elements threshold to DENSE codec after which DENSE is more compact than CHIMERA.
Selection range for maximum compression is therefore 512 - 9728 elements.
Conclusion
CHIMERA codec should provide higher compression and better performance compared to SPARSE codec.
The performance is probably lower compared to DENSE codec.
Further Considerations
A block of 256 could have different formats: FULL, BIZARRO, BITVEC.  But this would require extra metadata additional to VALS BEFORE
Related issue: https://github.com/quickwit-oss/tantivy/issues/2431

At the threshold we would have an average of 20 elements per chimera block. Maybe linear search should be considered too?

At the threshold we would have an average of 20 elements per chimera block. Maybe linear search should be considered too?

Yes, linear search would probably be better here.

Adding codecs could have some weird hidden switch-dispatch cost. It might be worth double checking how it goes.

In places where we fetch data in blocks (aggregation, sort in quickwit) we could do some optimization. The enum would have 3 instead of 2, so I think it should be fine.


feat: meteor mode

\
 \
  \
   \
    \
     🌟

            •
             •
              •
               ⭐

that's gonna be awesome :) You could use grayscale symbols to be in style: ⋆⭒˚｡⋆★

@ckaznable Would you consider something like that?
👇 (loading...)

@ckaznable Would you consider something like that? 👇

Ya, that's close to what I was thinking, but I've been a little busy lately and couldn't make it happen.

@ckaznable I will post my PR for what's on my gif, it's a little bit hacky though, cause I had to track the tail, and it doesn't easily fit to the existing code structure.

PR #12

Thanks for the PR, but I'd like to restructure the code first. I don't think the current structure is suitable for features expansion, especially in a mode like this one where you need to manage the status of the tail.

implemented


A text box doesn't pop up to ensure the filename when uploading

OS: All
Build Version: 20200911.6
Repro Steps:

Right click the blob container --> Select "Upload Files..." --> Select a file --> Click "Upload".
Check the result.

Expect:
A text box pops up to ensure the filename (may include a path).
Actual:
Upload the file directly without any text box.
More Info:

This issue also reproduces when uploading a folder.
This issue also reproduces for the file share.

Closing in favor of https://github.com/microsoft/vscode-azurestorage/issues/814.  We will use this logic for both single and multi-files that will overwrite this behavior.


Add a possible typo to a preset

From https://github.com/openstreetmap/iD/pull/4504

@peternewman Looks good, thank you!


Automatic PR from RestAPI-PR936

Generated from RestAPI PR: https://github.com/Azure/azure-rest-api-specs/pull/936

Done manually


Biome Tag list is either incomplete or wrong.

I'm not too knowledgeable about this topic, but your information on this page differs nearly 100% from the information we maintain on the wiki.
From speaking with members of our community, it seems out information is more up to date.
Your page: https://docs.microsoft.com/en-us/minecraft/creator/reference/content/entityreference/examples/definitions/biome_filter
Our page: https://wiki.bedrock.dev/world-generation/biome-tags.html#biome-per-biome-tag
Specific example of missing biome tag, found on our page: soulsand_valley

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b7fd0631-6230-0151-a8d1-6a9699ca3cb9
Version Independent ID: 0cb164e8-ad57-f1f1-a70d-19b1eb72f4bb
Content: Entity Documentation - minecraft:biome_filter
Content Source: creator/Reference/Content/EntityReference/Examples/Definitions/biome_filter.md
Product: gaming
GitHub Login: @v-josjones
Microsoft Alias: v-josjones

Thank you for your feedback; I apologize for the delayed response. Our documentation has been updated. Thanks again!


Provide the required ssh config needed to drive flutter driver test on Fuchsia

To run flutter driver tests on Fuchsia we need to wither have the FUCHSIA_BUILD_DIR or FUCHSIA_SSH_CONFIG file. Since we only have access to the SDK, we can not really have the build directory. Hence the need for tooling to generate the ssh_config file. An example is:
Inputs:

location to the private key.
path to the generated config file.

Output:
Generate a file like:
Host *
  CheckHostIP no
  StrictHostKeyChecking no
  ForwardAgent no
  ForwardX11 no
  GSSAPIDelegateCredentials no
  UserKnownHostsFile /dev/null
  User fuchsia
  IdentitiesOnly yes
  IdentityFile <path/to/.ssh/pkey>
  ControlPersist yes
  ControlMaster auto
  ControlPath /tmp/fuchsia--%r@%h:%p
  ConnectTimeout 10
  ServerAliveInterval 1
  ServerAliveCountMax 10
  LogLevel ERROR

If this is to enable the tests in luci the test beds are already provisioned with the private key and fuchsia_ctl can generate the authorized_keys at paving time.

This is already done in the runner script.


Replace workaround with Armeria's natively-supported context path

[x] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #. NO
[x] Update the CHANGES log.

Is CI / E2E test (Auth, test/e2e-v2/cases/simple/auth/e2e.yaml) case failing relative?


[dv] add send email option to dvsim.py

Add a send email options for dvsim.py. This only works when less secure
app is turned on. For google internal user, the script will use an
internal lib to route the email.
Signed-off-by: Cindy Chen<EMAIL_ADDRESS>

I agree with @imphil . The login of IMAP/SMTP is really messy so it is quite hard to be compliant with all the protocols. I also would suggest just dump the result in readable format that have subject, recipient, and body (or attachement) and use callback to call the email program whatever the user wants.

I see, thank you Eunchan and Philipp for the comments. I agree that SMTP has its limitation and the login is messy. I will bring it up in the dv meeting tomorrow and discuss what direction we wanted to move forward with.


Couldn't find Git Commit or Branch with name main

I'm unsure what I'm doing wrong here.
I more or less copied the example from the For building PRs section of the README. the only exception is that I use v4 of the checkout action and v4 of the setup dotnet action.
When I run the dotnet-affected action i get the following error:
Unhandled exception: System.InvalidOperationException: Couldn't find Git Commit or Branch with name main in repository /home/runner/work/itm_sursen4-backend/itm_sursen4-backend/.git/    at DotnetAffected.Core.GitChangesProvider.GetCommitOrThrow(Repository repo, String name) in /_/src/DotnetAffected.Core/GitChangesProvider.cs:line 152    at DotnetAffected.Core.GitChangesProvider.GetCommitOrHead(Repository repository, String name) in /_/src/DotnetAffected.Core/GitChangesProvider.cs:line [13](https://github.com/it-minds/itm_sursen4-backend/actions/runs/11343124706/job/31545025965#step:5:14)5    at DotnetAffected.Core.GitChangesProvider.ParseRevisionRanges(Repository repository, String from, String to) in /_/src/DotnetAffected.Core/GitChangesProvider.cs:line 82    at DotnetAffected.Core.GitChangesProvider.GetChangesForRange[T](Repository repository, String from, String to) in /_/src/DotnetAffected.Core/GitChangesProvider.cs:line 101    at DotnetAffected.Core.GitChangesProvider.GetChangedFiles(String directory, String from, String to) in /_/src/DotnetAffected.Core/GitChangesProvider.cs:line 24    at DotnetAffected.Core.Processor.AffectedProcessorBase.DiscoverChangedFiles(AffectedProcessorContext context) in /_/src/DotnetAffected.Core/Processor/AffectedProcessorBase.cs:line 57    at DotnetAffected.Core.Processor.AffectedProcessorBase.Process(AffectedProcessorContext context) in /_/src/DotnetAffected.Core/Processor/AffectedProcessorBase.cs:line 22    at DotnetAffected.Core.AffectedExecutor.Execute() in /_/src/DotnetAffected.Core/AffectedExecutor.cs:line 41    at Affected.Cli.Commands.InvocationContextExtensions.ExecuteAffectedExecutor(InvocationContext ctx) in /_/src/dotnet-affected/Commands/Binding/InvocationContextExtensions.cs:line 38    at Affected.Cli.Commands.AffectedRootCommand.<>c.<<-ctor>b__4_0>d.MoveNext() in /_/src/dotnet-affected/Commands/AffectedRootCommand.cs:line 38 --- End of stack trace from previous location ---    at System.CommandLine.Invocation.AnonymousCommandHandler.InvokeAsync(InvocationContext context)    at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext() --- End of stack trace from previous location ---    at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass17_0.<<UseParseErrorReporting>b__0>d.MoveNext() --- End of stack trace from previous location ---    at Affected.Cli.ErrorHandlingMiddleware.<>c__DisplayClass0_0.<<UseRenderingErrorHandler>b__0>d.MoveNext() in /_/src/dotnet-affected/CommandLine/ErrorHandlingMiddleware.cs:line 26 --- End of stack trace from previous location ---    at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass12_0.<<UseHelp>b__0>d.MoveNext() --- End of stack trace from previous location ---    at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<<UseVersionOption>b__0>d.MoveNext() --- End of stack trace from previous location ---    at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass19_0.<<UseTypoCorrections>b__0>d.MoveNext() --- End of stack trace from previous location ---    at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__18_0>d.MoveNext() --- End of stack trace from previous location ---    at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass[16](https://github.com/it-minds/itm_sursen4-backend/actions/runs/11343124706/job/31545025965#step:5:17)_0.<<UseParseDirective>b__0>d.MoveNext() --- End of stack trace from previous location ---    at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<RegisterWithDotnetSuggest>b__5_0>d.MoveNext() --- End of stack trace from previous location ---    at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass8_0.<<UseExceptionHandler>b__0>d.MoveNext()
I tried copying the command executed during my workflow and ran it lcoally with no issues.
My workflow can be seen here
name: PR Disaptcher

on:
  pull_request:
    branches: main

jobs:
  pr-dispatch:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Setup .NET
        uses: actions/setup-dotnet@v4
        with:
          dotnet-version: |
            8.0.x

      - uses: leonardochaia/dotnet-affected-action@v1
        id: dotnet_affected
        with:
          from: ${{ github.head_ref }}
          to: ${{ github.base_ref }}

      - run: echo ${{ steps.dotnet_affected.outputs.affected }}

I did some more research and found out that if i prefix both the to and from with origin/ it seems to work.
So the problem most likely comes from either checkout not checking out the base and head ref as heads during pr checkout, or the way LibGit2Sharp handles populating remote urls.
working workflow example, not sure if it's ideal though
name: PR Disaptcher

on:
  pull_request:
    branches: main

jobs:
  pr-dispatch:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Setup .NET
        uses: actions/setup-dotnet@v4
        with:
          dotnet-version: |
            8.0.x

      - uses: leonardochaia/dotnet-affected-action@v1
        id: dotnet_affected
        with:
          from: origin/${{ github.head_ref }}
          to: origin/${{ github.base_ref }}

      - run: echo ${{ steps.dotnet_affected.outputs.affected }}

hi @LordLyng ,I think we may need to review the example you've linked in order to make it work properly. Not sure if something changed in the tool that caused the breaking change, or if it is something else that changed.
Leo.


Encountering an issue with pdf.js truncating outline item titles when their text length exceeds 256 characters.

Encountering an issue with pdf.js truncating outline item titles when their text length exceeds 256 characters.

When opening an issue, please provide all of the information requested in https://github.com/mozilla/pdf.js/blob/master/.github/ISSUE_TEMPLATE.md since this contains no actionable information.


Fixed an issue with body's background color

Background color blue of the body didn't apply to the lower part of the body when options for I want were selected. Fixed the issue by removing h-screen class.

Footer alignment goes a bit off with the fix @ThulinaWickramasinghe


Show pending card when offline/ request fails (EXPOSUREAPP-4507)

When a request fails, the state should switch to pending and not stay in fetching as this suggests an ongoing process.
Another bug that was fixed: the error dialog (eg. when offline) was triggered multiple times when the screen was reopened, either by going back or putting the app to background

Branch name should be fix/4507-offline-test-result-pending instead of feature/4507-offline-test-result-pending, right?
But I don't know if it's worth fixing ...

Branch name should be fix/4507-offline-test-result-pending instead of feature/4507-offline-test-result-pending, right?
But I don't know if it's worth fixing ...

Branch name should be fix/4507-offline-test-result-pending instead of feature/4507-offline-test-result-pending, right?
But I don't know if it's worth fixing ...

Not worth changing once a PR is open. Just a pattern to help us devs when checking out branches.

Branch name should be fix/4507-offline-test-result-pending instead of feature/4507-offline-test-result-pending, right?
But I don't know if it's worth fixing ...

Not worth changing once a PR is open. Just a pattern to help us devs when checking out branches.

LGTM, Tested on Pixel 3A, Android 11

LGTM, Tested on Pixel 3A, Android 11

If I press "Zurück" on the dialog, I get back to the home screen.
If I press the back button on the device, I close the dialog and stay on an empty screen.

If I press "Zurück" on the dialog, I get back to the home screen.
If I press the back button on the device, I close the dialog and stay on an empty screen.

Yes, that is the default behavior and also what happens with all our dialogs. What would you expect?

Yes, that is the default behavior and also what happens with all our dialogs. What would you expect?

Yes, that is the default behavior and also what happens with all our dialogs., same for clicking outside the dialog. What would you expect?

I would have thought that if the dialogs only action is "Zurück", that dismissing the dialog would also act like "Zurück".

Yes, that is the default behavior and also what happens with all our dialogs., same for clicking outside the dialog. What would you expect?

I would have thought that if the dialogs only action is "Zurück", that dismissing the dialog would also act like "Zurück".


Inference tensors cannot be saved for backward.

I don't know if this is a bug. I'm trying to follow the official tutorial using pytorch engine.
Here is my code and exception.
String modelUrl = "https://resources.djl.ai/test-models/traced_distilbert_wikipedia_uncased.zip";
Criteria<NDList, NDList> criteria = Criteria.builder()
    .optApplication(Application.NLP.WORD_EMBEDDING)
    .setTypes(NDList.class, NDList.class)
    .optModelUrls(modelUrl)
    .optProgress(new ProgressBar())
    .build();

ZooModel<NDList, NDList> embedding = criteria.loadModel();
Predictor<NDList, NDList> embedder = embedding.newPredictor();
SequentialBlock classifier = new SequentialBlock()
    .add(
            ndList -> {
                NDArray data = ndList.singletonOrThrow();
                long batchSize = data.getShape().get(0);
                long maxLen = data.getShape().get(1);
                NDList inputs = new NDList();
                inputs.add(data.toType(DataType.INT64, false));
                inputs.add(data.getManager().full(data.getShape(), 1, DataType.INT64));
                inputs.add(data.getManager().arange(maxLen).toType(DataType.INT64, false).broadcast(data.getShape()));
                try {
                    return embedder.predict(inputs);
                } catch (TranslateException e) {
                    throw new RuntimeException(e);
                }
            }
    )
    .add(Linear.builder().setUnits(768).build())
    .add(Activation::relu)
    .add(Dropout.builder().optRate(0.2f).build())
    .add(Linear.builder().setUnits(5).build())
    .addSingleton(nd -> nd.get(":,0"));

Model model = Model.newInstance("review_classification");
model.setBlock(classifier);

DefaultVocabulary vocabulary = DefaultVocabulary.builder()
    .addFromTextFile(embedding.getArtifact("vocab.txt"))
    .optUnknownToken("[UNK]")
    .build();

int maxTokenLen = 64;
int batchSize = 8;
int limit = Integer.MAX_VALUE;

BertFullTokenizer tokenizer = new BertFullTokenizer(vocabulary, true);
CsvDataset awsDataset = getDataset(batchSize, tokenizer, maxTokenLen, limit);
RandomAccessDataset[] datasets = awsDataset.randomSplit(7, 3);
RandomAccessDataset trainDataset = datasets[0];
RandomAccessDataset evalDataset = datasets[1];

SaveModelTrainingListener listener = new SaveModelTrainingListener("build/model");
listener.setSaveModelCallback(
    trainer -> {
        TrainingResult result = trainer.getTrainingResult();
        Model trainerModel = trainer.getModel();
        float acc = result.getValidateEvaluation("Accuracy");
        trainerModel.setProperty("Accuracy", String.format("%.5f", acc));
        trainerModel.setProperty("Loss", String.format("%.5f", result.getValidateLoss()));
    }
);

DefaultTrainingConfig trainingConfig = new DefaultTrainingConfig(Loss.softmaxCrossEntropyLoss())
    .addEvaluator(new Accuracy())
    .addTrainingListeners(TrainingListener.Defaults.logging("build/model"))
    .addTrainingListeners(listener);

int epoch = 2;
Trainer trainer = model.newTrainer(trainingConfig);
trainer.setMetrics(new Metrics());
Shape shape = new Shape(batchSize, maxTokenLen);
trainer.initialize(shape);
EasyTrain.fit(trainer, epoch, trainDataset, evalDataset);
System.out.println(trainer.getTrainingResult());
model.save(Paths.get("build/model"), "aws-review-rank");

[main] INFO ai.djl.pytorch.engine.PtEngine - Number of inter-op threads is 6
[main] INFO ai.djl.pytorch.engine.PtEngine - Number of intra-op threads is 6
[main] INFO ai.djl.training.listener.LoggingTrainingListener - Training on: cpu().
[main] INFO ai.djl.training.listener.LoggingTrainingListener - Load PyTorch Engine Version 1.12.1 in 0.079 ms.
Exception in thread "main" ai.djl.engine.EngineException: Inference tensors cannot be saved for backward. To work around you can make a clone to get a normal tensor and use it in autograd.
	at ai.djl.pytorch.jni.PyTorchLibrary.torchNNLinear(Native Method)
	at ai.djl.pytorch.jni.JniUtils.linear(JniUtils.java:1189)
	at ai.djl.pytorch.engine.PtNDArrayEx.linear(PtNDArrayEx.java:390)
	at ai.djl.nn.core.Linear.linear(Linear.java:183)
	at ai.djl.nn.core.Linear.forwardInternal(Linear.java:88)
	at ai.djl.nn.AbstractBaseBlock.forwardInternal(AbstractBaseBlock.java:126)
	at ai.djl.nn.AbstractBaseBlock.forward(AbstractBaseBlock.java:91)
	at ai.djl.nn.SequentialBlock.forwardInternal(SequentialBlock.java:209)
	at ai.djl.nn.AbstractBaseBlock.forward(AbstractBaseBlock.java:91)
	at ai.djl.training.Trainer.forward(Trainer.java:175)
	at ai.djl.training.EasyTrain.trainSplit(EasyTrain.java:122)
	at ai.djl.training.EasyTrain.trainBatch(EasyTrain.java:110)
	at ai.djl.training.EasyTrain.fit(EasyTrain.java:58)
	at cn.amberdata.misc.djl.rankcls.Main.main(Main.java:114)

And here is my dependencies.
        <!-- https://mvnrepository.com/artifact/ai.djl/api -->
        <dependency>
            <groupId>ai.djl</groupId>
            <artifactId>api</artifactId>
            <version>0.19.0</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-simple -->
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-simple</artifactId>
            <version>1.7.36</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/ai.djl.pytorch/pytorch-engine -->
        <dependency>
            <groupId>ai.djl.pytorch</groupId>
            <artifactId>pytorch-engine</artifactId>
            <version>0.19.0</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/ai.djl/basicdataset -->
        <dependency>
            <groupId>ai.djl</groupId>
            <artifactId>basicdataset</artifactId>
            <version>0.19.0</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/ai.djl/model-zoo -->
        <dependency>
            <groupId>ai.djl</groupId>
            <artifactId>model-zoo</artifactId>
            <version>0.19.0</version>
        </dependency>

Thanks for pointing out this bug to us - this is an issue in the NoopTranslator when using the PyTorch engine. We will take a look and determine the best fix.
For now, you can adjust your code as follows to get around the exception:

Define your own custom translator that extends the NoopTranslator and overrides the processOutput method

public static final class MyTranslator extends NoopTranslator {

        @Override
        public NDList processOutput(TranslatorContext ctx, NDList input) {
            return new NDList(
                input.stream().map(ndArray -> ndArray.duplicate()).collect(Collectors.toList()));
        }
    }

When you are building the Criteria object, add the optTranslator builder method and pass in the customer translator

Criteria<NDList, NDList> criteria = Criteria.builder()
    .optApplication(Application.NLP.WORD_EMBEDDING)
    .setTypes(NDList.class, NDList.class)
    .optModelUrls(modelUrl)
    .optProgress(new ProgressBar())
    .optTranslator(new MyTranslator()) // Add this line
    .build();

@SuperMaskv
The problem comes from return embedder.predict(inputs);, the predict() assume inference only, it set the Inference mode. What you can do is instead of manually calling embedder.predict(), you can add the block into your classifer, something like:
SequentialBlock classifier = new SequentialBlock()
    .add(
            ndList -> {
                NDArray data = ndList.singletonOrThrow();
                long batchSize = data.getShape().get(0);
                long maxLen = data.getShape().get(1);
                NDList inputs = new NDList();
                inputs.add(data.toType(DataType.INT64, false));
                inputs.add(data.getManager().full(data.getShape(), 1, DataType.INT64));
                inputs.add(data.getManager().arange(maxLen).toType(DataType.INT64, false).broadcast(data.getShape()));
                return inputs;
            })
    .add(embedder.getBlock());

@frankfliu Thanks for helping. I have tried ur solution. And here comes out a new problem with initializing parameter.
Here is exception stack trace:
Exception in thread "main" java.lang.NullPointerException: No parameter shape has been set
	at java.util.Objects.requireNonNull(Objects.java:228)
	at ai.djl.nn.Parameter.initialize(Parameter.java:181)
	at ai.djl.nn.AbstractBaseBlock.initialize(AbstractBaseBlock.java:182)
	at ai.djl.nn.SequentialBlock.initializeChildBlocks(SequentialBlock.java:224)
	at ai.djl.nn.AbstractBaseBlock.initialize(AbstractBaseBlock.java:184)
	at ai.djl.training.Trainer.initialize(Trainer.java:117)
	at cn.amberdata.misc.djl.rankcls.Main.main(Main.java:72)

It seems that djl do not find a suitable implements for function prepare in AbstractBaseBlock.

@siddvenk Code in tutorial works fine with ur solution, thanks a lot.

I'm in exactly the same point. I cannot follow the fix for #2173. because it freezes the original distilbert model, while I need to finetune all the layers. @frankfliu setup seemed to me good, since it just put the model block in the sequence and nothing seems preventing it to be part of the backward. But @SuperMaskv  last NullPointerError blocks the initialize. It initializes correctly the input of the sequenceBlock, then it finds a PtSymbolBlock.
In AbstractBaseBlock.initialize() it checks:

if (!isInitialized()) {
// setShape for all params
prepare(inputShapes);
}

Here we have some issues:

isInitialized checks:

for (Parameter param : getParameters().values()) {
if (!param.isInitialized()) {
return false;
}
}

and they are initialized (but it checks for arrays == null, not shape == null - maybe it's a consequence, but this is a different check and shape == null will be the problem, as we'll see in a while).
So it doesn't execute prepare(inputShapes).
But later, in AbstractBaseBlock.initialize() in the loop:

for (Parameter parameter : getDirectParameters().values()) {
parameter.initialize(manager, dataType);
}

it finds that the first parameter has shape null, so it throws the error.
BTW: I don't know if getDirectParameters() and getParameters() return the same result.

let's suppose that isInitialized is false (I commented the check to simulate it): sadly prepare() is not abstract, but empty and PtSymbolBlock doesn't implement it. So it cannot set shape in parameters.

Since I understand that shapes should be the input/output of every layer, it seems to me that somewhere they should be set correctly if it's true (and it's true, as in the AmazonRating example) that the same block, if frozen, works.
Maybe is there a way to use that code (wherever it may be)?
I hope my comment could help in solving this issue without creating confusion.

@KexinFeng Sure, it's almost the same as the one provided before by @SuperMaskv , with the incremental corrections, but let's put it here to have a refreshed and summarized view:
package it.algaware.mrjvs.djl;

import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.Reader;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.stream.Stream;

import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVParser;
import org.apache.commons.csv.CSVRecord;
import org.apache.commons.io.input.BOMInputStream;

import ai.djl.Application;
import ai.djl.Device;
import ai.djl.MalformedModelException;
import ai.djl.Model;
import ai.djl.basicdataset.tabular.CsvDataset;
import ai.djl.basicdataset.tabular.utils.Feature;
import ai.djl.engine.Engine;
import ai.djl.huggingface.tokenizers.HuggingFaceTokenizer;
import ai.djl.inference.Predictor;
import ai.djl.metric.Metrics;
import ai.djl.modality.Classifications;
import ai.djl.modality.Classifications.Classification;
import ai.djl.modality.nlp.DefaultVocabulary;
import ai.djl.modality.nlp.bert.BertFullTokenizer;
import ai.djl.modality.nlp.qa.QAInput;
import ai.djl.ndarray.NDArray;
import ai.djl.ndarray.NDList;
import ai.djl.ndarray.types.DataType;
import ai.djl.ndarray.types.Shape;
import ai.djl.nn.Activation;
import ai.djl.nn.Block;
import ai.djl.nn.LambdaBlock;
import ai.djl.nn.SequentialBlock;
import ai.djl.nn.core.Linear;
import ai.djl.nn.norm.Dropout;
import ai.djl.repository.zoo.Criteria;
import ai.djl.repository.zoo.ModelNotFoundException;
import ai.djl.repository.zoo.ZooModel;
import ai.djl.training.DefaultTrainingConfig;
import ai.djl.training.EasyTrain;
import ai.djl.training.Trainer;
import ai.djl.training.TrainingResult;
import ai.djl.training.dataset.RandomAccessDataset;
import ai.djl.training.evaluator.Accuracy;
import ai.djl.training.listener.SaveModelTrainingListener;
import ai.djl.training.listener.TrainingListener;
import ai.djl.training.loss.Loss;
import ai.djl.training.optimizer.Adam;
import ai.djl.training.optimizer.Optimizer;
import ai.djl.training.tracker.LinearTracker;
import ai.djl.training.tracker.Tracker;
import ai.djl.training.tracker.WarmUpTracker;
import ai.djl.training.util.ProgressBar;
import ai.djl.translate.PaddingStackBatchifier;
import ai.djl.translate.TranslateException;

public class Test2 {

	public static void main(String[] args) throws ModelNotFoundException, MalformedModelException, IOException, TranslateException {

		System.out.println("You are using: " + Engine.getInstance().getEngineName() + " Engine");
		System.out.println("GPU count: " + Engine.getInstance().getGpuCount());
		Device[] pass1 = Engine.getInstance().getDevices();
		System.out.println("GPU: "+pass1[0]);

		getIntentAll(2, "C:\\clinc150_uci\\train",
				"C:\\clinc150_uci\\test", "feature", "label");

	}

	public static void getIntentAll(int epoch, String trainingPath, String validationPath, String feature, String label) throws ModelNotFoundException, MalformedModelException, IOException {
		// let's find intents, first
		ArrayList<String> intents = new ArrayList<>();

		File file = new File(trainingPath);
		Stream<String> stream = null;
		int trainingSetSize = 0;
		try {
			stream = Files.lines(file.toPath(), StandardCharsets.UTF_8);
			Iterator<String> iterator = stream.iterator();
			boolean headerFound = false;
			while (iterator.hasNext()) {
				trainingSetSize+=1;
			    String line = iterator.next();
				if (headerFound) {
				    String[] values = line.split("[|]");
				    if (!intents.contains(values[0].toLowerCase())) {
				    	intents.add(values[0].toLowerCase());
				    }
				} else {
					headerFound = true;
				}
			}
		} finally {
			if (stream != null) {
				stream.close();
				stream = null;
			}
		}

		file = new File(validationPath);
		try {
			stream = Files.lines(file.toPath());
			Iterator<String> iterator = stream.iterator();
			boolean headerFound = false;
			while (iterator.hasNext()) {
			    String line = iterator.next();
			    if (headerFound) {
				    String[] values = line.split("[|]");
				    if (!intents.contains(values[0].toLowerCase())) {
				    	intents.add(values[0].toLowerCase());
				    }
				} else {
					headerFound = true;
				}
			}
		} finally {
			if (stream != null) {
				stream.close();
				stream = null;
			}
		}

		System.out.println("found "+intents.size()+" intents");

		Criteria<NDList, NDList> criteria = Criteria.builder()
		        .optApplication(Application.NLP.WORD_EMBEDDING)
		        .setTypes(NDList.class, NDList.class)
		        //.optModelUrls(modelUrls)
		        .optModelPath(Paths.get("build/pytorch/traced_distilbert_wikipedia_uncased"))
		        .optProgress(new ProgressBar())
		        .optTranslator(new MyNoopTranslator())
		        .build();
		ZooModel<NDList, NDList> embedding = criteria.loadModel();

		System.out.println("Model Loaded");

		DefaultVocabulary vocabulary = DefaultVocabulary.builder()
		        .addFromTextFile(embedding.getArtifact("vocab.txt"))
		        .optUnknownToken("[UNK]")
		        .build();

		int maxTokenLength = 128;
		int batchSize = 32;
		int limit = Integer.MAX_VALUE;

		BertFullTokenizer tokenizer = new BertFullTokenizer(vocabulary, true);
		CsvDataset trainingDataset;
		CsvDataset validationDataset;
		try {
			trainingDataset = getIntentDataset(batchSize, tokenizer, maxTokenLength, limit, trainingPath, feature, label, intents, true);

			RandomAccessDataset trainingSet = trainingDataset;
			RandomAccessDataset validationSet = null;
			if (validationPath != null) {
				validationDataset = getIntentDataset(batchSize, tokenizer, maxTokenLength, limit, validationPath, feature, label, intents, false);
				validationSet = validationDataset;
			} else {
				RandomAccessDataset[] datasets = trainingDataset.randomSplit(7, 3);
				trainingSet = datasets[0];
				validationSet = datasets[1];
			}

			System.out.println("Vocabulary and dataset prepared");

			SaveModelTrainingListener listener = new SaveModelTrainingListener("build/model");
	        listener.setSaveModelCallback(
	            trainer -> {
	                TrainingResult result = trainer.getTrainingResult();
	                Model model2 = trainer.getModel();
	                // track for accuracy and loss
	                float accuracy = result.getValidateEvaluation("Accuracy");
	                model2.setProperty("Accuracy", String.format("%.5f", accuracy));
	                model2.setProperty("Loss", String.format("%.5f", result.getValidateLoss()));
	            });

	        int epochStep = trainingSetSize/batchSize+1;
	        LinearTracker lit = LinearTracker.builder()
	        		.setBaseValue(0.00115f)
	        		.optSlope(-0.00003f/epochStep)
	        		.optMinValue(0.00004f) // reached at epoch 32
	        		.build();

			WarmUpTracker warmuptr = Tracker.warmUp()
			        .optWarmUpSteps(epoch/10*epochStep)
			        .optWarmUpBeginValue(0.001f)
			        .setMainTracker(lit)
			        .build();

			Optimizer adam = Adam.builder().optLearningRateTracker(warmuptr).build();

	        Loss loss = Loss.softmaxCrossEntropyLoss();
	        DefaultTrainingConfig config = new DefaultTrainingConfig(loss) // loss type
	        .optOptimizer(adam) // Optimizer
	        .addEvaluator(new Accuracy())
	        .optDevices(Engine.getInstance().getDevices(1)) // train using single GPU
	        .addTrainingListeners(TrainingListener.Defaults.logging("build/model"))
	        .addTrainingListeners(listener);
	        System.out.println("Training config set up");

			Block classifier = new SequentialBlock()
					.add(embedding.getBlock())
			        // classification layer
			        .add(Dropout.builder().optRate(0.5f).build())
			        .add(Linear.builder().setUnits(728).build()) // pre classifier
			        .add(Activation::relu)
			        .add(Dropout.builder().optRate(0.5f).build())
			        .add(Linear.builder().setUnits(intents.size()).build()) // as many output as intents are
			        .addSingleton(nd -> nd.get(":,0")); // Take [CLS] as the head
			Model model = Model.newInstance("IntentClassification");
			model.setBlock(classifier);
			System.out.println("Embedding generated");

	        Trainer trainer = model.newTrainer(config);
	        trainer.setMetrics(new Metrics());
	        Shape encoderInputShape = new Shape(batchSize, maxTokenLength);

	        trainer.initialize(encoderInputShape);
	        EasyTrain.fit(trainer, epoch, trainingSet, validationSet);
	        System.out.println("Training ended: "+trainer.getTrainingResult());

	        model.save(Paths.get("build/model"), "bert-intent.param");
	        System.out.println("model saved");

		} catch (Exception e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
	}

	static CsvDataset getIntentDataset(int batchSize, BertFullTokenizer tokenizer, int maxLength, int limit, String path, String feature, String label, ArrayList<String> intents, boolean randomSample) {
	    float paddingToken = tokenizer.getVocabulary().getIndex("[PAD]");
	    return CsvDataset.builder()
	            .optCsvFile(Paths.get(path)) // load from File
	            .setCsvFormat(CSVFormat.Builder.create(CSVFormat.TDF)
	            		.setQuote(null)
	            		.setDelimiter("|")
	            		.setRecordSeparator("\n")
	            		.setHeader()
	            		.build()) // Setting loading format
	            .setSampling(batchSize, randomSample) // make sample size and random access
	            .optLimit(limit)
	            .addFeature(
	                    new Feature(feature, new BertFeaturizer(tokenizer, maxLength))
	                    )
	            .addLabel(
	                    new Feature(
	                            label, (buf, data) -> {
	                            	float intentNumber = intents.indexOf(data.toLowerCase());
	                            	buf.put(intentNumber);
	                            }))
	            .optDataBatchifier(
	                    PaddingStackBatchifier.builder()
	                            .optIncludeValidLengths(false)
	                            .addPad(0, 0, (m) -> m.ones(new Shape(1)).mul(paddingToken))
	                            .build()) // define how to pad dataset to a fix length
	            .build();
	}

}

I looked at your suggested example, but, sadly, I cannot spot logical differences, so I'm hoping in an help from you.

@KexinFeng Do we need your #2203 PR to succeed in fine tuning the model ?

I'm trying to repeat results from https://aclanthology.org/2021.naacl-industry.38.pdf for Distilbert and CLINC150: hyperparameters don't match (are really very different!) to those of the paper, but probably it's due to the fact I didn't understand that I was using a freezed block (as in the AmazonExample, also at the beginning of this post).
I expect that the parameters of the paper will match the final ones to be used if we can fine tune the Distilbert model (probably results will not, but only because they're using AdamW and not Adam - I opened #2185 enhancement for that).
Interesting, even with a frozen BERT you can reach about a 91% in accuracy, with just a simple classifier on top.
Well, for your knowledge, there could be probably another minor issue (I'll check it again before opening it and it is not so important for the current input, but it seems to me useful to highlight here for those reading) in the current DJL framework to correctly reach the results of the cited paper.
When you use batches you end up padding tokens, at least to align sentences to the same length (the maximum length of sentences in the batch) in the specific batch (while you can have different maximum length in different batches).
This modifies the input sentences lengths, while BERT model are not trained for fixed length, so I expect a possible degradation.
Indeed, trying a post-training workaround, consisting in padding till an predefined length (only if the sentence length is minor than the predefined one) the same input sentences used for test and running them (execution, not training) on the final model manually, you obtain a better accuracy than without the padding.

@KexinFeng In this paper the best accuracy for this task is 85.7%. Is it already better than: "with a frozen BERT you can reach about a 91% in accuracy"?

mmm, no, the 85.7% you mentioned appears in the difficult case (table 3), but Distilbert in the full case scores a 96.3% and that's what I'm trying to emulate now and where I reach a 91%, so I'm missing about a 5%.
Moreover, using few example tests (e.g. Curekart full, see table 7, where BERT reaches 83.6%) the gap is even greater (Distilbert frozen reaches about a 29%, absolutely not comparable and explainable probably only with the missing full training of all layers, i.e. with Distilbert not frozen).
So experimenting with the unfrozen model is a must.

By the way, you mentioned "hyperparameters don't match (are really very different!) to those of the paper,". Even though the unfreezing of the layer is enabled, it is still important to make sure hyperparamters, as well as the model and pre-processing agree to the paper, in order to reproduce or make sensible comparison of the results.

Obviously, it's important to use the same hyperparameters with the unfrozen case, but with the frozen one the paper hyperparameters perform absolutely worse than the values you see in the code. That's the reason why you see them in the code. My notice in the previous comment was only to explain that difference, it wasn't intended to highlight an issue.
I expect to find similar results with similar hyperparameters in the unfrozen case.
Still, I'm not so sure to understand what should I do to make the unfrozen case work

@KexinFeng Sorry, I'm not so sure to understand. Let's try point by point:

Yes, https://github.com/deepjavalibrary/djl/pull/2203 PR is about enabling fine tuning the embeddor layer, in this case the distilbert model.

So, I should wait for the next release or is it enough that I just take the 2 classes you modified and put this new version of them in place (before in the classpath, i.e., in my project) to test if this corrects the NullPointerException occurred to me and to @SuperMaskv  (as in his last comment)?
Or do you think that there is some error in the example code I put in my previous comment as you requested?

So you can take a look at it, and experiment with trainParam="true" when setting the Criteria.builder, used to load distilbert, to see if the metric is improved.

At this time, more then the improvement of the metric (I understand you're referring to the accuracy, let me know if I misunderstood, please), I'm interested in understanding how to finetune all layers (i.e. how to have an unfrozen Distilbert, if I understand correctly) and not to the metric itself (well, yes, but it's the final result: I'm trying to understand the correct setup to avoid the NullPointerError shown in the last comment of @SuperMaskv).

Another headup is that you might need to assign a smaller learning rate to the pre-trained layer (ie embedding layer), which makes sure it is "fine" tuning. This implementation is demonstrated again in examples/src/main/java/ai/djl/examples/training/transferlearning/TransferFreshFruit.java.

Ok, I see that there is a different learning rate tracker in the example, I'll try it, thank you, but I expect just it improves the results, not it avoids the NullPointerException.

By the way, the major difference in this https://github.com/deepjavalibrary/djl/pull/2203 PR is that the model now consists of a Lambda block, then the distilbert, then the following classifier layers. Previously Lambda block and the distilbert are together in one layer.

Sorry, here I don't understand anything at all about this point. Can you rephrase, please? Should I add a Lambda block before Distilbert?
I can't understand where should the Lambda block in the example (I think you refer to lines 83-89 of https://github.com/deepjavalibrary/djl/blob/master/examples/src/main/java/ai/djl/examples/training/transferlearning/TransferFreshFruit.java, isn't it? I cannot see anything before the original model: what am I missing? Which line adds the Lambda block?
Thank you again for your support

@KexinFeng  Thank you for your further effort in #2206 .
About the previous issues I:

substituted the latest version (0.20.0 in Maven since yesterday), hoping that it contained your PR
added .optOption("trainParam", "true") to the Criteria

No more NullPointerException (fine, thank you again).
But I got the following exception:

[main] INFO ai.djl.training.listener.LoggingTrainingListener - Load PyTorch Engine Version 1.13.0 in 0,045 ms.
ai.djl.engine.EngineException: forward() is missing value for argument 'argument_2'. Declaration: forward(torch.transformers.models.distilbert.modeling_distilbert.DistilBertModel self, Tensor input_ids, Tensor argument_2, Tensor input) -> ((Tensor))
at ai.djl.pytorch.jni.PyTorchLibrary.moduleForward(Native Method)
at ai.djl.pytorch.jni.IValueUtils.forward(IValueUtils.java:47)
at ai.djl.pytorch.engine.PtSymbolBlock.forwardInternal(PtSymbolBlock.java:144)
at ai.djl.pytorch.engine.PtSymbolBlock.getOutputShapes(PtSymbolBlock.java:224)
at ai.djl.nn.SequentialBlock.initializeChildBlocks(SequentialBlock.java:225)
at ai.djl.nn.AbstractBaseBlock.initialize(AbstractBaseBlock.java:184)
at ai.djl.training.Trainer.initialize(Trainer.java:118)
at it.algaware.mrjvs.djl.Test2.getIntentAll(Test2.java:230)
at it.algaware.mrjvs.djl.Test2.main(Test2.java:77)

I debugged and found that 0.20.0 not yet contains your merged updates. :(
Ok, I took the single files of your PR and put them in my project.
Same error.
Maybe should I fetch the whole SNAPSHOT, as you suggested? Is that different? Or is this another issue?
In this case, sorry for the silly question, but I'm not so used working with snapshots from GitHub: is there a way to get the snapshot version from a Maven Repo or should I download the zip of the master code and expand it in my project?

@KexinFeng Just trying the first hyperparameters setup, I already reached 94.17%, which is not the 96.3% of the paper, but I'm confident that I can find a better setup with more work and the code is using Adam and not AdamW (I expect a further minor enhancement from that).
Great work, thank you again

In my previous case everything worked with the last snapshot (even AdamW) with the Distilbert Model. I almost reached the results I expected (well, with different parameters, but I think the original paper I already mentioned isn't so clear: it leaves many possible interpretations - at least for me, who are not so good at interpreting all the models).
Anyway, I tried to change my model using a bert-base-uncased. Just saved it using this code:
import torch
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained("bert-base-uncased")

text = "Replace me by any text you'd like."
inputs = tokenizer(text, return_tensors='pt')
model.config.return_dict=False;
model.config.torchscript=True;
model.eval();
traced_model = torch.jit.trace(model, [inputs.data.get('input_ids'),inputs.data.get('attention_mask'),inputs.data.get('token_type_ids')])
torch.jit.save(traced_model, "C:\\bertBase\\bertBase.pt")

Then using the following code:
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.Reader;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.stream.Stream;

import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVParser;
import org.apache.commons.csv.CSVRecord;
import org.apache.commons.io.input.BOMInputStream;

import ai.djl.Application;
import ai.djl.Device;
import ai.djl.MalformedModelException;
import ai.djl.Model;
import ai.djl.basicdataset.tabular.CsvDataset;
import ai.djl.basicdataset.tabular.utils.Feature;
import ai.djl.engine.Engine;
import ai.djl.huggingface.tokenizers.HuggingFaceTokenizer;
import ai.djl.inference.Predictor;
import ai.djl.metric.Metrics;
import ai.djl.modality.Classifications;
import ai.djl.modality.Classifications.Classification;
import ai.djl.modality.nlp.DefaultVocabulary;
import ai.djl.modality.nlp.bert.BertFullTokenizer;
import ai.djl.modality.nlp.qa.QAInput;
import ai.djl.ndarray.NDArray;
import ai.djl.ndarray.NDList;
import ai.djl.ndarray.types.DataType;
import ai.djl.ndarray.types.Shape;
import ai.djl.nn.Activation;
import ai.djl.nn.Block;
import ai.djl.nn.Parameter;
import ai.djl.nn.LambdaBlock;
import ai.djl.nn.SequentialBlock;
import ai.djl.nn.core.Linear;
import ai.djl.nn.norm.Dropout;
import ai.djl.repository.zoo.Criteria;
import ai.djl.repository.zoo.ModelNotFoundException;
import ai.djl.repository.zoo.ZooModel;
import ai.djl.training.DefaultTrainingConfig;
import ai.djl.training.EasyTrain;
import ai.djl.training.ParameterStore;
import ai.djl.training.Trainer;
import ai.djl.training.TrainingResult;
import ai.djl.training.dataset.RandomAccessDataset;
import ai.djl.training.evaluator.Accuracy;
import ai.djl.training.listener.SaveModelTrainingListener;
import ai.djl.training.listener.TrainingListener;
import ai.djl.training.loss.Loss;
import ai.djl.training.optimizer.Adam;
import ai.djl.training.optimizer.AdamW;
import ai.djl.training.optimizer.Optimizer;
import ai.djl.training.tracker.FixedPerVarTracker;
import ai.djl.training.tracker.LinearTracker;
import ai.djl.training.tracker.Tracker;
import ai.djl.training.tracker.WarmUpTracker;
import ai.djl.training.util.ProgressBar;
import ai.djl.translate.PaddingStackBatchifier;
import ai.djl.translate.TranslateException;
import ai.djl.util.Pair;

public class Test2 {

	public static void main(String[] args) throws ModelNotFoundException, MalformedModelException, IOException, TranslateException {

		System.out.println("You are using: " + Engine.getInstance().getEngineName() + " Engine");
		System.out.println("GPU count: " + Engine.getInstance().getGpuCount());
		Device[] pass1 = Engine.getInstance().getDevices();
		System.out.println("GPU: "+pass1[0]);

		getIntentAll(5, "C:\\clinc150_uci\\train",
				"C:\\clinc150_uci\\test", "feature", "label");

	}

	public static void getIntentAll(int epoch, String trainingPath, String validationPath, String feature, String label) throws ModelNotFoundException, MalformedModelException, IOException {
		// let's find intents, first
		ArrayList<String> intents = new ArrayList<>();

		File file = new File(trainingPath);
		Stream<String> stream = null;
		int trainingSetSize = 0;
		try {
			stream = Files.lines(file.toPath(), StandardCharsets.UTF_8);
			Iterator<String> iterator = stream.iterator();
			boolean headerFound = false;
			while (iterator.hasNext()) {
				trainingSetSize+=1;
			    String line = iterator.next();
				if (headerFound) {
				    String[] values = line.split("[|]");
				    if (!intents.contains(values[0].toLowerCase())) {
				    	intents.add(values[0].toLowerCase());
				    }
				} else {
					headerFound = true;
				}
			}
		} finally {
			if (stream != null) {
				stream.close();
				stream = null;
			}
		}

		file = new File(validationPath);
		try {
			stream = Files.lines(file.toPath());
			Iterator<String> iterator = stream.iterator();
			boolean headerFound = false;
			while (iterator.hasNext()) {
			    String line = iterator.next();
			    if (headerFound) {
				    String[] values = line.split("[|]");
				    if (!intents.contains(values[0].toLowerCase())) {
				    	intents.add(values[0].toLowerCase());
				    }
				} else {
					headerFound = true;
				}
			}
		} finally {
			if (stream != null) {
				stream.close();
				stream = null;
			}
		}
		int avgExample = trainingSetSize/intents.size();
		System.out.println("found "+intents.size()+" intents - Avg example per intent: "+ avgExample);

		Criteria<NDList, NDList> criteria = Criteria.builder()
		        .optApplication(Application.NLP.WORD_EMBEDDING)
		        .setTypes(NDList.class, NDList.class)
		        .optModelPath(Paths.get("build/pytorch/bertBase"))
		        .optProgress(new ProgressBar())
		        .optOption("trainParam", "true") // this unlocks the finetuning on the loaded model
		        .build();

		ZooModel<NDList, NDList> embedding = criteria.loadModel();

		System.out.println("Model Loaded");

		DefaultVocabulary vocabulary = DefaultVocabulary.builder()
		        .addFromTextFile(embedding.getArtifact("vocab.txt"))
		        .optUnknownToken("[UNK]")
		        .build();

		int maxTokenLength = 128;
		int batchSize = 32;
		int limit = Integer.MAX_VALUE;

		BertFullTokenizer tokenizer = new BertFullTokenizer(vocabulary, true);
		CsvDataset trainingDataset;
		CsvDataset validationDataset;
		try {
			trainingDataset = getIntentDataset(batchSize, tokenizer, maxTokenLength, limit, trainingPath, feature, label, intents, true);

			RandomAccessDataset trainingSet = trainingDataset;
			RandomAccessDataset validationSet = null;
			if (validationPath != null) {
				validationDataset = getIntentDataset(batchSize, tokenizer, maxTokenLength, limit, validationPath, feature, label, intents, false);
				validationSet = validationDataset;
			} else {
				RandomAccessDataset[] datasets = trainingDataset.randomSplit(7, 3);
				trainingSet = datasets[0];
				validationSet = datasets[1];
			}

			System.out.println("Vocabulary and dataset prepared");

			SaveModelTrainingListener listener = new SaveModelTrainingListener("build/model");
	        listener.setSaveModelCallback(
	            trainer -> {
	                TrainingResult result = trainer.getTrainingResult();
	                Model model2 = trainer.getModel();
	                // track for accuracy and loss
	                float accuracy = result.getValidateEvaluation("Accuracy");
	                model2.setProperty("Accuracy", String.format("%.5f", accuracy));
	                model2.setProperty("Loss", String.format("%.5f", result.getValidateLoss()));
	            });

	        int epochStep = trainingSetSize/batchSize+1;
	        Tracker lrt = Tracker.fixed(0.00004f); // adam default: 0.001f, 4*10^-5 is HINT paper setup for BERT
	        LinearTracker lit = LinearTracker.builder()
	        		.setBaseValue(0.0001f)
	        		.optSlope(-0.000015f/epochStep)
	        		.optMinValue(0.00002f)
	        		.build();

	        Optimizer adam = AdamW.builder().optLearningRateTracker(lit).optWeightDecays(0.015f).build();

	        Loss loss = Loss.softmaxCrossEntropyLoss();
	        DefaultTrainingConfig config = new DefaultTrainingConfig(loss) // loss type
	        .optOptimizer(adam) // Optimizer
	        .addEvaluator(new Accuracy())
	        .optDevices(Engine.getInstance().getDevices(1)) // train using single GPU
	        .addTrainingListeners(TrainingListener.Defaults.logging("build/model"))
	        .addTrainingListeners(listener);
	        System.out.println("Training config set up");

		Model model = Model.newInstance("IntentClassification");
		model.setBlock(getBlock(embedding.getBlock(), intents, avgExample));
		System.out.println("Embedding generated");

	        Trainer trainer = model.newTrainer(config);
	        trainer.setMetrics(new Metrics());
	        Shape encoderInputShape = new Shape(batchSize, maxTokenLength);

	        trainer.initialize(encoderInputShape);
	        EasyTrain.fit(trainer, epoch, trainingSet, validationSet);
	        System.out.println("Training ended: "+trainer.getTrainingResult());

	        model.save(Paths.get("build/model"), "bert-intent.param");

		} catch (Exception e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
	}

	private static Block getBlock(Block embedder, ArrayList<String> intents, int avgExample) {
        SequentialBlock classifier = new SequentialBlock();
        // text embedding layer
        if ("PyTorch".equals(Engine.getDefaultEngineName())) {
            LambdaBlock lambda =
                    new LambdaBlock(
                            ndList -> {
                                NDArray data = ndList.singletonOrThrow();
                                NDList inputs = new NDList();
                                inputs.add(data.toType(DataType.INT64, false));
                                inputs.add(
                                        data.getManager().full(data.getShape(), 1, DataType.INT64));
                                inputs.add(
                                        data.getManager()
                                                .arange(data.getShape().get(1)) // maxLen
                                                .toType(DataType.INT64, false)
                                                .broadcast(data.getShape()));
                                return inputs;
                            });
            classifier.add(lambda);
            classifier.add(embedder);
        } else {
            // MXNet
            LambdaBlock lambda =
                    new LambdaBlock(
                            ndList -> {
                                NDArray data = ndList.singletonOrThrow();
                                long batchSize = data.getShape().get(0);
                                float maxLength = data.getShape().get(1);
                                return embedder.forward(
                                        new ParameterStore(),
                                        new NDList(
                                                data,
                                                data.getManager()
                                                        .full(new Shape(batchSize), maxLength)),
                                        true);
                            });
            classifier.add(lambda);
        }
        // Classification layers
        classifier
        	.add(Dropout.builder().optRate(0.05f).build())
                .add(Linear.builder().setUnits(768).build()) // pre classifier
                .add(Activation::relu)
                .add(Dropout.builder().optRate(0.18f).build())
                .add(Linear.builder().setUnits(intents.size()).build())
                .addSingleton(nd -> nd.get(":,0")); // follow HF classifier
        return classifier;
    }

	static CsvDataset getIntentDataset(int batchSize, BertFullTokenizer tokenizer, int maxLength, int limit, String path, String feature, String label, ArrayList<String> intents, boolean randomSample) {
	    float paddingToken = tokenizer.getVocabulary().getIndex("[PAD]");
	    return CsvDataset.builder()
	            .optCsvFile(Paths.get(path)) // load from File
	            .setCsvFormat(CSVFormat.Builder.create(CSVFormat.TDF)
	            		.setQuote(null)
	            		.setDelimiter("|")
	            		.setRecordSeparator("\n")
	            		.setHeader()
	            		.build()) // Setting loading format
	            .setSampling(batchSize, randomSample) // make sample size and random access
	            .optLimit(limit)
	            .addFeature(
	                    new Feature(feature, new BertFeaturizer(tokenizer, maxLength))
	                    )
	            .addLabel(
	                    new Feature(
	                            label, (buf, data) -> {
	                            	float intentNumber = intents.indexOf(data.toLowerCase());
	                            	buf.put(intentNumber);
	                            }))
	            .optDataBatchifier(
	                    PaddingStackBatchifier.builder()
	                            .optIncludeValidLengths(false)
	                            .addPad(0, 0, (m) -> m.ones(new Shape(1)).mul(paddingToken))
	                            .build()) // define how to pad dataset to a fix length
	            .build();
	}

}

I get the following error:

java.lang.IndexOutOfBoundsException: Incorrect number of elements in NDList.singletonOrThrow: Expected 1 and was 2
at ai.djl.ndarray.NDList.singletonOrThrow(NDList.java:218)
at ai.djl.nn.norm.Dropout.forwardInternal(Dropout.java:74)
at ai.djl.nn.AbstractBaseBlock.forwardInternal(AbstractBaseBlock.java:128)
at ai.djl.nn.AbstractBaseBlock.forward(AbstractBaseBlock.java:93)
at ai.djl.nn.SequentialBlock.forwardInternal(SequentialBlock.java:209)
at ai.djl.nn.AbstractBaseBlock.forward(AbstractBaseBlock.java:93)
at ai.djl.training.Trainer.forward(Trainer.java:189)
at ai.djl.training.EasyTrain.trainSplit(EasyTrain.java:122)
at ai.djl.training.EasyTrain.trainBatch(EasyTrain.java:110)
at ai.djl.training.EasyTrain.fit(EasyTrain.java:58)
at it.algaware.mrjvs.djl.Test2.getIntentAll(Test2.java:[different line: I cleaned the above code before posting])

I understand (well, probably...) from https://huggingface.co/docs/transformers/model_doc/bert#transformers.models.bert.modeling_bert.BertForPreTrainingOutput that it depends upon the fact that bert-base returns 2 outputs (prediction_logits and seq_relationship_logits, I suppose).
But how can I select the correct output, matching my following 768 Linear block, just after the BERT model?
I found it among the two NDList elements of the error, when debugging, but I don't know how to describe it in the getBlock method.

well, ok, in the previous code the modelDir seems not to match the phisical one where I saved the model, but it's just a not complete cleaning of the code: obviously they match, else I'd receive an error before.

It looks like it requires some model level design and debugging. Is this issue solved?

Regarding

But how can I select the correct output, matching my following 768 Linear block, just after the BERT model?

consider use the Lambda Block or other kinds of blocks in DJL to build it.

@KexinFeng

Is this issue solved?

Well... I didn't open this issue. It seems to me that the first reason for opening this issue has been solved, but I think @SuperMaskv  should say that.
About my last questions (raised only as continuations of previous comments), maybe it's better I'll open other issues, so to keep it cleaner.
Anyway, I tried the Lambda Block, but I got another issue. Opening apart.
Thank you again for your support