live candle results not accurate (tested bitfinex.com BTC/USD)

There are inconsistency while getting candle data from bitfinex.com
In some cases (occurs multiple times during a day) candle values differ when importing them live vs importing them later and saving to postgresql backend with --import switch.
This leads to inaccurate candle data in both ways live and importing them.
I launched live two gekko proceses:

1min. candles live importer with postgres save enabled (candleWriter enabled);
1min. candles live another gekko instance (candleWriter disabled, saved results to postgres with another method)
deleted data form db and imported the same timeframe with first method, but later with --import switch

Expected result: candle data match.
Actual result:
Case,	    start,     	open,	high,  	low,  	close,	vwp,	           volume,	   trades,

live to postgres:<PHONE_NUMBER>	6232.7	6242.6	6229.5	6238.9	6234.2724294016	32.15845437	99
live another process:<PHONE_NUMBER>	6232.6	6242.6	6229.5	6238.9	6234.206595785	33.56074073	108
reimport after:<PHONE_NUMBER>	6232.7	6242.6	6229.5	6241.9	6234.1964438444	25.28986376	60

This is not a single, but rather consistent issue - occurs in a constant basis.

If they are started even 1 second apart they will consist of a different 1min period which will lead to different results.

They could go theoretically to another 1min. period, but cannot disappear, like it happens. Notice different trades/volume count sometimes they shift to another time period, but mainly get lost.

If they are started even 1 second apart they will consist of a different 1min period which will lead to different results.
hey could go theoretically to another 1min. period

This should never happen: Gekko batches all trades together per clock minute (based on the "trade.date" property) see here:
https://github.com/askmike/gekko/blob/c48c85e8df4765f827ecc2269cfec152b280d532/core/budfox/candleCreator.js#L92-L104
Thinking more about this, it might be related to this: https://github.com/askmike/gekko/pull/2263#discussion_r199354415
EG. UTC vs non UTC.
@3pm would you be able to quickly test whether manually changing this line:
https://github.com/askmike/gekko/blob/c48c85e8df4765f827ecc2269cfec152b280d532/core/budfox/candleCreator.js#L95
Into:
var minute = trade.date.clone().utc().format('YYYY-MM-DD HH:mm');

fixes it?

Doesn't fix.

This I did with the fix, still bad records happen:
type;start;open;high;low;close;vwp;volume;trades
import;1531751880;6622.8144046;6623;6622.8144046;6623;6622.90439276175;6.53953563;8
live1;1531751880;6622.8;6623;6622.8;6623;6622.83625656969;11.11015384;13
live2;1531751880;6622.8;6623;6622.8;6623;6622.85288488848;12.90879502;17

As we can see in the last column, the amount of trades is different each
time. I am trying to reproduce this..
On Mon, Jul 16, 2018 at 11:36 PM, 3pm<EMAIL_ADDRESS>wrote:

This I did with the fix, still bad records happen:
type;start;open;high;low;close;vwp;volume;trades
import;1531751880;6622.8144046;6623;6622.8144046;6623;6622.90439276175;6.
53953563;8
live1;1531751880;6622.8;6623;6622.8;6623;6622.83625656969;11.11015384;13
live2;1531751880;6622.8;6623;6622.8;6623;6622.85288488848;12.90879502;17
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/askmike/gekko/issues/2307#issuecomment-405289112, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA7MD8i0rkKyM5W2LBqpKzmDSJj4n2cmks5uHLMAgaJpZM4VPM-6
.

--
PGP key at keybase.io/mikevanrossum
https://keybase.io/mikevanrossum/key.asc

Finally was able to do some testing:
Live trading
It appears that the first candle is not guaranteed to be full. Meaning I only see this happening on the very first candle, after that they are in sync. (If you see something else please let me know). I don't consider this a crucial issue right now - but Gekko should definitely not produce non full candles.
Importing
When importing not all trades are fetched, resulting in Gekko calculating candles that are not full, example:

logs of amount of candles per candle. Left is from importer, right is from a live gekko. As you can see not all candles have the same amount of trades.
I'll debug and fix the importer!

I have created a branch that has 2 minor fixes, but still has some differences in output. See #2354. Will try to pick it up asap. If someone else feel like helping out (on that branch) that would be great :)

Closing this for now, I consider this fixed enough for now.
Summary:

Live traders can have one different initial candle, after that they are all the same.
There was a bug in the importer: it would not fetch all trades, this is now fixed (in develop). HOWEVER there is still a tiny discrepancy caused by the importer using a different API than the liver trader. This may result in miniscule differences in calculates 1 minute candles (details are here: https://github.com/askmike/gekko/pull/2354#issuecomment-407047864). Since that discrepancy is so much smaller than the discrepancy described in this issue, I am considering this closed.


Repo.insert's on_conflict with an instruction set or {:replace, [:field]} is replacing timestamps (both updated_at and inserted_at)

Elixir version
1.16.2
Database and Version
PostgreSQL 14.8
Ecto Versions
3.11
Database Adapter and Versions (postgrex, myxql, etc)
ecto_sql 3.11
Current behavior
Run either of these with a few seconds between, note that the returned record has an altered inserted_at and updated_at:
Repo.insert(
  SomeSchemaWithTimestamps.changeset(%{email: email, remote_id: remote_id}),
  conflict_target: :remote_id,
  on_conflict: {:replace, [:email]}
)

Repo.insert(
  SomeSchemaWithTimestamps.changeset(%{email: email, remote_id: remote_id}),
  conflict_target: :remote_id,
  on_conflict: [
    set: [email: email]
  ]
)

Here is the Schema for the record where we're seeing this happen:
  defmodule ExternalUserSchema do
  alias Ecto.Changeset

  use Ecto.Schema

  @type t :: %__MODULE__{}

  @derive {Jason.Encoder, only: [:id, :email]}

  schema "external_users" do
    field(:deleted_at, :utc_datetime)
    field(:email, :string)
    field(:remote_id, :string)

    timestamps()
  end

  def changeset(params) do
    Changeset.cast(%__MODULE__{}, params, [:email, :remote_id])
    |> Changeset.validate_required([:email, :remote_id])
  end
end

Expected behavior
When providing an instruction set, the instruction set is carried out and timestamps remain unchanged.
When providing a {:replace, [:fields]}, only the specified fields are replaced (not timestamps).

updated_at changing makes sense here, but inserted_at changing seems like it can't be intentional.

@kybishop the struct is not updated with the information from the database. You would have to use read_after_writes or explicitly read it back instead, the semantics are database dependent and the docs cover it: https://hexdocs.pm/ecto/Ecto.Repo.html#c:insert/2-upserts :)


About the text annotation

Is the text annotation made by yourself? If I want to train my own dataset, do I need to make it？

The text annotation is made by the radiologist. And if you want to train your own dataset, maybe you can generate the text annotation through the mask.

你好，大佬，感谢您很有启发性的工作。我想问一下如果我想用其他的2D数据集，比如GLAS，BUSI，ISIC这些，如何用mask来生成text标注，可以提供一下对应的生成代码或者思路吗？期待您的回复！


Support organization secrets

Add support to manage organization secrets

@Alw3ys can you elaborate what is missing for organization secrets?

Hi @kneemaa,
Thanks for your reply! and sure!
Currently there's only support to update repositories actions secret.
It's the way the api is called, so if you look at your code at:
https://github.com/kneemaa/github-action-rotate-aws-secrets/blob/master/rotate_keys.py#L137
https://api.github.com/repos/{owner_repo}/actions/secrets/{key_name}
In order to support organization secrets another endpoint would need to be called:
https://api.github.com/orgs/{org}/actions/secrets/{key_name}
Here you can find the github documentation reference:
https://docs.github.com/en/rest/reference/actions#create-or-update-an-organization-secret
If you have any further questions, please feel free to ask!

For reference I've created a PR with the feature changes,
It's not tested but that should do it for what, Otherwise I hope it gets you a better understand of what feature request I'm asking.
https://github.com/kneemaa/github-action-rotate-aws-secrets/pull/10

Is there any chance this FEATURE gets implemented?
This works great, but AWS secret for the GitHub Actions user is normally stored on an organisation level instead of duplicating it into every repo with actions.
Would be a neat addition.


Provisioning error shows incorrect project number

When trying to bring up a stack that included a gcp.sql.DatabaseInstance, I got this error:
Plan apply failed: Error, failed to create instance unleash: googleapi: Error 403: Access Not Configured. Cloud SQL Admin API has not been used in project<PHONE_NUMBER>69 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/sqladmin.googleapis.com/overview?project=563584335869 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry., accessNotConfigured

I do have sqladmin.googleapis.com enabled, but<PHONE_NUMBER>69 is not my project.

After further investigation, I found<PHONE_NUMBER>69 is the project number for Firebase CLI, which I installed in the past.
I ran gcloud auth application-default login and now pulumi up works as expected.

Same thing happens in Terraform.. Thanks

Issue still apply when running an app locally through dev_appserver.py, even after running gcloud auth application-default login as suggested.


Fixed the null financial value issue. Used ruby exception handling coding pattern

helpers/project_helpers.rb - Handled exception for data existence.
views/partials/_project_list.html.erb - Handled null value exception.
views/projects/partners.html.erb - Handled null value exception.
views/projects/summary.html.erb - Handled null value exception.
views/projects/transactions.html.erb - Handled null value exception.
views/regions/region.html.erb - Handled null value exception.
views/search/search.html.erb - Handled null value exception.

Hi John,
I have passed all the possible activity_plus_child_aggregation through the exception handler. Did not find any error on the informed pages so far. Should be good to go now.


IWithdrawRestClient.WithdrawAsync response deserialization bug

When calling the WithdrawAsync method from the IWithdrawRestClient interface for the Mexc exchange, a JsonSerializationException occurs during response deserialization. The expected return type is IEnumerable, which requires a JSON array. However, the Mexc API returns a JSON object instead, causing the deserialization to fail. This issue only occurs with the Mexc exchange. I have shared logic that works correctly with other exchanges like Binance, Kucoin, and Gate, indicating that the issue is specific to Mexc’s API response format.
Deserialize JsonSerializationException: Cannot deserialize the current JSON object (e.g. {"name":"value"}) into type 'System.Collections.Generic.IEnumerable1[Mexc.Net.Objects.Models.Spot.MexcId]' because the type requires a JSON array (e.g. [1,2,3]) to deserialize correctly.
To fix this error either change the JSON to a JSON array (e.g. [1,2,3]) or change the deserialized type so that it is a normal .NET type (e.g. not a primitive type like integer, not a collection type like an array or List) that can be deserialized from a JSON object. JsonObjectAttribute can also be added to the type to force it to deserialize from a JSON object.
Path 'id', line 1, position 6.`
Version 1.13.1(newest)
The withdrawal proceeds perfectly fine otherwise.

Fixed in latest version


TS-4195: double free on exit

Rework on fixing crash on exit.
The previous commit https://github.com/apache/trafficserver/commit/e0e39fc5026f0f5bcd6c0884adde91bcccab6bf8 is reverted because it affects Asan report generating.
Tried self-pipe trick as @jpeach suggested, it still had a race condition between return from main() and exit() in the pipe handler. It seems exit is not necessary as we already have sleep loop waiting on shutdown_event_system before the end of main()
We can get the Asan report and get no crash with the current patch.
Thank @jaaju for helping.

[approve ci]

RAT check successful! https://ci.trafficserver.apache.org/job/RAT-github/67/

clang format successful! https://ci.trafficserver.apache.org/job/clang-format-github/54/

AU check failed! https://ci.trafficserver.apache.org/job/autest-github/51/

FreeBSD11 build successful! https://ci.trafficserver.apache.org/job/freebsd-github/1749/

Intel CC build successful! https://ci.trafficserver.apache.org/job/icc-github/180/

Linux build successful! https://ci.trafficserver.apache.org/job/linux-github/1643/

clang-analyzer build successful! https://ci.trafficserver.apache.org/job/clang-analyzer-github/312/

Once landed, I think we should contemplate this for a 7.1.0 cherry-pick.

@bryancall Ping?

@bryancall , @zwoop , ping you guys since this PR opened for a long while.
We landed this fix in Linkedin several weeks ago. It works great. No more annoying crashes when we do deployments.

Looks good

Did we want this for 7.1.0 as well? If so, can you make a PR please?

Sure. Thanks for your reminder.


Add TTL Value if Present to Check

It would be awesome see this on the drill-down for a check; Presently, I have to dig into the API to see that this is set on the most recent result string key for a check.  It would be even cooler to have a countdown timer that showed when the TTL is going to run out based on the last result somewhere.  Or perhaps a new dashboard page that showed checks with TTL's and their upcoming expiration time?
As my use case, I'm currently using TTL's on sensu checks submitted manually by cronjobs, and I like to know that they ran on the interval they are set to by the assurance that a TTL Event will tell me when something didn't run.  i.e. if I comment out a cronjob and forget to re-enable it, I don't get angry emails the next day..

I guess I should have waited longer; overnight, the TTL started showing up on the check.


SubQuery SQL Error

Your issue may already be reported! Please search on the issue track before creating one.
What version of Go are you using (go version)?
➜  awesomeProject go version
go version go1.11.1 darwin/amd64

Which database and its version are you using?
mysql> select version();
+-------------------------+
| version()               |
+-------------------------+
| 5.7.20-0ubuntu<IP_ADDRESS> |
+-------------------------+

Please provide a complete runnable program to reproduce your issue. IMPORTANT
Gorm version:
* commit 742154be9a26e849f02d296073c077e0a7c23828 (HEAD -> master, origin/master, origin/HEAD)
| Author: Iskander (Alex) Sharipov<EMAIL_ADDRESS>| Date:   Sun Oct 7 03:49:37 2018 +0300
|
|     rewrite if-else chain as switch statement (#2121)
|
|     From effective Go: https://golang.org/doc/effective_go.html#switch
|
|     > It's therefore possible—and idiomatic—to write an if-else-if-else chain as a switch.
|
|  association.go | 7 ++++---
|  1 file changed, 4 insertions(+), 3 deletions(-)
|

My code:
package main

import (
	"fmt"
	"github.com/jinzhu/gorm"
	_ "github.com/jinzhu/gorm/dialects/mysql"
	"time"
)

type User struct {
	gorm.Model
	Name string `gorm:"default:'g10guang'"`
	Age  int    `gorm:"default:100"`
}

func main() {
	url := fmt.Sprintf("%v:%v@(%v:%v)/%v?charset=utf8&parseTime=True&loc=Local",
		"username", "password", "<IP_ADDRESS>", "3306", "goweb")
	db, err := gorm.Open("mysql", url)
	if err != nil || db == nil {
		fmt.Println(err)
	}
	db.SingularTable(true)

	db.AutoMigrate(&User{})

	users := make([]*User, 0)

	fmt.Println(db.Where("age < ?", db.Table("user").Select("AVG(age)").QueryExpr()).Find(&users).QueryExpr())

}

output:
➜  awesomeProject go run main.go
&{SELECT * FROM `user`  WHERE `user`.`deleted_at` IS NULL AND ((age < SELECT AVG(age) FROM `user`  )) []}

I try it in mysql console. SQL Error happens.
I think the correct SQL is:
SELECT * FROM `user`  WHERE `user`.`deleted_at` IS NULL AND (age < (SELECT AVG(age) FROM `user` ))

Thanks.

On the docs site - http://gorm.io/docs/query.html#SubQuery
Result is also would be incorrect -
SELECT * FROM "branches"  WHERE "branches"."deleted_at" IS NULL AND ((amount > SELECT AVG(amount) FROM "orders"  WHERE (state = 'paid')))

:/

I have the same problem

@gq-tang also with mysql?

@gq-tang also with mysql?

yes

@gq-tang, @penguinpowernz:
I have sub queries working using this method...
var somevalue = 1
row := db.Select("something").Table("first_table").Where("exists(?)", db.Select("a_relationship_to_something").Model(&SecondTable{}).Where("id = ?", somevalue).QueryExpr()).Row()
	row.Scan(&id)

I've tested this with row(), rows(), first(), and find(). You can also use both .Table() and .Model() interchangeably as shown in the example.
May be worth updating the docs to this method if worthy

@g10guang can you try with Subquery() instead of QueryExpr()

@g10guang can you try with Subquery() instead of QueryExpr()

I quit my job. So I cannot replay this bug.

@g10guang can you try with Subquery() instead of QueryExpr()

Thanks, this worked for me. Query/SubAuery docs should be updated as the given example does not work and SubQuery() is not mentioned anywhere outside godoc.

docs
code:

db.Where("amount > ?", DB.Table("orders").Select("AVG(amount)").Where("state = ?", "paid").QueryExpr()).Find(&orders)

expected:

SELECT * FROM "orders"  WHERE "orders"."deleted_at" IS NULL AND (amount > (SELECT AVG(amount) FROM "orders"  WHERE (state = 'paid')));

got: (not work)

&{SELECT * FROM user  WHERE user.deleted_at IS NULL AND ((age < SELECT AVG(age) FROM user  )) []}

PART II
code:

db.Where("amount > (?)", DB.Table("orders").Select("AVG(amount)").Where("state = ?", "paid").QueryExpr()).Find(&orders)

got: (work)

SELECT * FROM user  WHERE user.deleted_at IS NULL AND ((age < (SELECT AVG(age) FROM user  )))


Spectator View : Cannot Build Compositor with  USE_OPENCV     TRUE

How it happened?

I changed USE_OPENCV    flag to TRUE (ofc, others are FALSE) in CompositorShared.h ;
In dependencies.props  added  <OpenCV_vc14>C:\apps\opencv_31\opencv\build\x64\vc14</OpenCV_vc14>

Ive downloaded the correct version for sure and as you can see 64 bit build has been pointed out. Also tried using 3.2 with changing lib includes, but that ofc failed.
What i receive when building with this configuration:
Severity	Code	Description	Project	File	Line	Suppression State
Error	LNK2001	unresolved external symbol "public: __thiscall cv::VideoCapture::VideoCapture(int)" (??0VideoCapture@cv@@QAE@H@Z)	CompositorDLL	C:\Users\girts.kesteris\Dev\unity\HoloLensCompanionKit\SpectatorView\Compositor\CompositorDLL\OpenCVFrameProvider.obj	1

Severity	Code	Description	Project	File	Line	Suppression State
Error	LNK2001	unresolved external symbol "void __cdecl cv::fastFree(void *)" (?fastFree@cv@@YAXPAX@Z)	CompositorDLL	C:\Users\girts.kesteris\Dev\unity\HoloLensCompanionKit\SpectatorView\Compositor\CompositorDLL\OpenCVFrameProvider.obj	1

Severity	Code	Description	Project	File	Line	Suppression State
Error	LNK2001	unresolved external symbol "public: void __thiscall cv::Mat::deallocate(void)" (?deallocate@Mat@cv@@QAEXXZ)	CompositorDLL	C:\Users\girts.kesteris\Dev\unity\HoloLensCompanionKit\SpectatorView\Compositor\CompositorDLL\OpenCVFrameProvider.obj	1

Severity	Code	Description	Project	File	Line	Suppression State
Error	LNK1120	3 unresolved externals	CompositorDLL	C:\Users\girts.kesteris\Dev\unity\HoloLensCompanionKit\SpectatorView\Compositor\Release\CompositorDLL.dll	1

build_log.txt

Are you building for x64 or x86?
x86 support for OpenCV will require you to build OpenCV yourself for x86.  If you download the OpenCV build, only x64 binaries are included.

Yes - it turned out to be the issue - i compiled to both x86 and x64. Sorry bout that!


Steps in "Publish To Environment" not using custom shell-property from deployment settings

The action "Publish To Environment invokes a few powershell actions with a static definition of the input parameter shell, even though most steps pass the value defined in the deployment settings:
https://github.com/microsoft/AL-Go/blob/17311608bf05bcf17459ab94f2120c97a2b024a7/Templates/Per Tenant Extension/.github/workflows/PublishToEnvironment.yaml#L149-L185
This results in errors on runners missing the correct keyword. Is there a reason why the parameter in the steps "Get Artifacts for deployment" and "Deploy To Power Platform" wasn't updated like the other steps?

Bumping this because I need an answer. If it's an oversight, I can create a new PR to fix this, but I need to know if it was left like this on purpose, and how we can fix this if yes.

Sorry, didn't see this.
You are welcome to replace the shell: powershell with the one from the matrix in a PR.
Please note that this needs to be done under Templates/Per Tenant Extension and Templates/AppSource App
Thanks


Read about fine-tuning strategies

Questions we want to answer:

Should we use pre-trained embeddings or the whole model?
Why or why not?
What are some major fine-tuning strategies and what are their benefits and drawbacks?

Related to this issue: https://github.com/lozadaa-chalmers/transformer-bio/issues/13

tyjtyj

Supervised full fine tuning:

Adjusts the weight parameters of the whole model
If not careful: Can lead to "catastrophic forgetting", i.e. "forgets" the pre-trained model.

Parameter Efficient Fine Tuning (PEFT):

Instead of adjusting weights in model, "freeze" most of pre-trained models parameters on only train a small subset.
Reduced computational cost
Minimal risk of catastrophic forgetting

Low Rank Adaptation (LoRA):

Doesn't touch the pre-trained model at all
Adds new parameters to train instead, in a lower dimension than the pre-trained model.
E.g. if pre-trained model is dimension NxN, LoRA adapter is NxM, where M<<N.
M is determined by SVD
LoRA adapters are modular
Can use quantization, which turns parameters from float32 into int4 (qLoRA).


Implement shortcut to close tinker console and output at the same time

Imported from JetBrains Plugin Repo review by Joao Alves
Is your feature request related to a problem? Please describe.

a shortcuts to close both windows at the same time

Describe the solution you'd like

a shortcuts to close both windows at the same time

Implemented, will be released in 1.3.0


same contents

Hi @colevandersWands,
There are two same contents in different links in prep work section.

thenodeway (article)
fredkschott (article)

🙇


cuitlazotlac/francais

Attention (CC)

Summary of changes
Description of the changes included in this pull request.
Steps to follow to confirm the feature is completed or the bug/issue is resolved.
Are there any dependencies?
Does this code break any existing functionality or introduce any risk?
Relevant Context

Feature: [ticket number](link to project management ticket)
Relevant Pull Request: #

How Has This Been Tested?

Screenshots (if appropriate):
Further comments

Types of changes

[x] Content edit, File upload, content addition
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] Technical Documentation Update
[ ] Other (please describe):

Does this introduce a breaking change?
Breaking change is defined as "A fix or feature that would cause existing functionality to change."

[ ] Yes
[x] No

Checklist:

[x] My code follows the code style of this project.
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have added/changed necessary documentation (if appropriate)
[ ] All new and existing tests passed.

Bonjour :heart:
I merged this because it was getting too big to readily keep track of. If you would finish the translations for the Research and Development pages, I will merge this into production :pray:


Deprecate support for torchtext.legacy dependencies

Proposed refactor
Restrict checks & support for torchtext.legacy in Lightning

Motivation
torchtext is retiring support for torchtext.legacy in upcoming versions: https://github.com/pytorch/text/issues/1422
Lightning uses the Batch from torchtext.legacy abstractions as part of this utility to move data to device:
https://github.com/PyTorchLightning/pytorch-lightning/blob/7f95fd04d77e3590c8073679e499eabb0c51f73c/pytorch_lightning/utilities/apply_func.py#L245-L255

Pitch
Restrict the checks for Batch from torchtext based on the last torchtext version which supports them (torchtext <= 0.11).
Might be as simple as updating this line:
https://github.com/PyTorchLightning/pytorch-lightning/blob/7f95fd04d77e3590c8073679e499eabb0c51f73c/pytorch_lightning/utilities/apply_func.py#L29
to
    if _compare_version("torchtext", operator.ge, "0.9.0") and _compare_version("torchtext", operator.le, "0.11"):

@parmeet @abhinavarora could you confirm?
Ultimately we could remove these checks as the torchtext team is not planning to support them. This would likely provide a performance boost on a critical path by removing a conditional check for torchtext when moving data to device.

Additional context
cc @justusschock @awaelchli @akihironitta @tchaton @parmeet @abhinavarora

If you enjoy Lightning, check out our other projects! ⚡

Metrics: Machine learning metrics for distributed, scalable PyTorch applications.

Lite: enables pure PyTorch users to scale their existing code on any kind of device while retaining full control over their own loops and optimization logic.

Flash: The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, fine-tuning, and solving problems with deep learning.

Bolts: Pretrained SOTA Deep Learning models, callbacks, and more for research and production with PyTorch Lightning and PyTorch.

Lightning Transformers: Flexible interface for high-performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.

Agree that we should eventually remove the legacy Batch logic. If users need the legacy version in a newer PL code base, they can always implement our hook to give special treatment to the legacy Batch object.


Optimize get_project_information DB calls

Coverage increased (+0.002%) to 93.829% when pulling c871c5b33a7e25bb33d4bae17e5a15c4c3f0cce2 on optimize_get_project_information into 31acdaec32ed7cdd8910f20ddd3cfec2ad8189a5 on master.


Support Ubuntu on ccportal

OS for ccportal is configurable via optional config.yml setting 'cyclecloud.linux_image'

regression tests passed

@xpillons how did playwright changes end up in this PR?


add node cli subcommand

@jsancio bump

@tamarrow what is our plan with this?  the CLI subcommand project repo is private.. we can't merge this that way.

We decided to punt on this until we decide what we want out of it.

Closing for now.


Cholesky fails

Traceback (most recent call last):
  File "test_from_init_pose_pickle.py", line 390, in <module>
    main()
  File "/usr/local/lib/python3.8/dist-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "test_from_init_pose_pickle.py", line 343, in main
    outputs = model(Gs=pose_input, images=images_input, depths_fullres=depths_input, \
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/gin/config.py", line 1605, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/usr/local/lib/python3.8/dist-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
    raise proxy.with_traceback(exception.__traceback__) from None
  File "/usr/local/lib/python3.8/dist-packages/gin/config.py", line 1582, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "/poselab/projects/bin-picking/gpose_cir/pose_models/raftse3.py", line 214, in forward
    Gs, _, _ = MoBA(target, weight, Gs, depths, intrinsics, num_solver_steps, ii, jj)
  File "/poselab/projects/bin-picking/gpose_cir/utils/geom/ba.py", line 217, in MoBA
    poses, depths, intrinsics = _step_moba(target, weight, poses, depths, intrinsics, ii, jj)
  File "/poselab/projects/bin-picking/gpose_cir/utils/geom/ba.py", line 203, in _step_moba
    dx = DenseSystemSolver.apply(_H, _v)
  File "/poselab/projects/bin-picking/gpose_cir/utils/geom/ba.py", line 128, in forward
    U = torch.cholesky(H)
RuntimeError: cholesky: (Batch element 0): The factorization could not be completed because the input is not positive-definite (the leading minor of order 4 is not positive-definite).

I got this problem using the given weights. I guess there might be something wrong with my data. Since I'm running on T-LESS dataset and there are not models directory,  I symlink it to 'models_reconst'. Am I using the right models for rendering or is there any way to find the cause of this problem?

Sorry I didn't meet this error before. Maybe you should check if the matrices are positive-definite.


Add metadata NAT helper

This is a short lived hack for the Folsom release, to be replaced
by authenticated proxies in Grizzly.
Signed-off-by: Stephen Gran<EMAIL_ADDRESS>

I think a 4 year old patch isn't useful


C:\Users\Lenovo\AppData\Local\Programs\Python\Python36\lib\site-packages\h5py_init_.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.

Please go to Stack Overflow for help and support:
https://stackoverflow.com/questions/tagged/tensorflow
If you open a GitHub issue, here is our policy:

It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).
The form below must be filled out.
It shouldn't be a TensorBoard issue. Those go here.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
TensorFlow installed from (source or binary):
TensorFlow version (use command below):
Python version:
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:
Exact command to reproduce:

You can collect some of this information using our environment capture script:
https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh
You can obtain the TensorFlow version with
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
Describe the problem
Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.
Source code / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.

what is that?
how can I try it?
On Mon, 28 May 2018 03:53 drpngx<EMAIL_ADDRESS>wrote:

I'm not sure there's much we can do. h5py is used by keras. Could you try
to file a bug upstream?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/19469#issuecomment-392381321,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AcHEoryy5trsvhXTxrzLgMNol3Qw7l37ks5t2ydYgaJpZM4UJG5U
.

Nagging Assignee @drpngx: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

You can close it.
On 12 June 2018 at 00:26, Alfred Sorten Wolf<EMAIL_ADDRESS>wrote:

Nagging Assignee @drpngx https://github.com/drpngx: It has been 14 days
with no activity and this issue has an assignee. Please update the label
and/or status accordingly.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/19469#issuecomment-396347264,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AcHEoh9UxUdxowi6xp2IaZchgQirHv5Cks5t7r1ugaJpZM4UJG5U
.


WFLY-18123 Remove use of deprecated VersionedURN subclasses

https://issues.redhat.com/browse/WFLY-18123
Requires https://github.com/wildfly/wildfly/pull/16952

Rebased on main (to incorporate wildfly-core upgrade)


closing parenthesis is not recognised in math formula in context block

Description
When a closing parenthesis is in a context block, it isn't detected by lr and appears small when it should be sized.
The bug is simple to reproduce:
#let rule(body) = {body}

#show ")" :rule

$
	(1/2)
$

Gives:

as expected  but
#let rule(body) = context{body}

#show ")" :rule

$
	(1/2)
$

Gives:

And I thing it's a bug because:
#let rule(body) = context{body}

#show "(" :rule

$
	(1/2)
$

Gives:

Reproduction URL
No response
Operating system
Linux
Typst version

[X] I am using the latest version of Typst

I can reproduce this issue on Typst v0.11.x, but cannot reproduce it on main branch. Using git bisect, I found #4212 fixed this issue.

I'm still able to reproduce the issue on main, only it happens with the left parenthesis instead of the right one:
#{
  show "(": it => context it
  $ (1 / 2) $
}
#{
  show ")": it => context it
  $ (1 / 2) $
}
#{
  show "(": it => context it
  show ")": it => context it
  $ (1 / 2) $
}

(sorry for the low quality, I think the problem is visible anyway).

Debugging (on main e90c30903d2c93a0d899ebd9326c5cd4ab11bf41) indicates that in the cases where ( is replaced with context("("), there is an extra MathFragment::Frame at the start of the lr element’s body containing a tag for the context.
This issue is similar to #3727 in that it’s related to ignorant or invisible elements affecting math layout. It still occurs on #4348, but it should provide the machinery to implement a similar fix for lr elems.

On main, the issue now happens with both left and right parentheses.


Use consistent naming for PHI-base releases

This pull request updates the names of the PHI-base release files so that they appear in the correct order when sorted by filename. This involved two changes:

always use leading zeroes on the minor version number (e.g. v4-6 is now v4-06), and
make all release filenames follow the format phi-base_v4-{NN}_{YYYY}-{MM}-{DD}.

Previously there were inconsistencies related to including a description of the format in the filename, and the fact that some names didn't start with "phi-base_".
The changes shouldn't affect the GLoBI integration, because I haven't renamed phi-base_current.csv. However, any URLs that reference the release files directly will presumably break if they aren't a permalink tied to a commit: for example, I expect the following URL will be broken after these changes are merged:
https://github.com/PHI-base/data/blob/master/releases/phi-base_v4-1_2016-05-01.csv
@martin2urban please let me know if you're happy with the changes.

Also, I don't know why GitHub is reporting additions or deletions for a few files that haven't changed. GitHub Desktop doesn't show these files as modified, so it's probably just a bug.


add unit tests for handling of LFNBase and forceMerged in workflow creation/assignment

Make sure that when a workflow is created with a non-standard LFNBase, that non-standard LFNBase survives assignment.
Make sure when a workflow has a task with forceMerged output, that output stays forceMerged after assignment.
These are basically the two things that were broken for the Tier0 and the same reason they broke now (these features aren't really used in WMCore) could cause them to be broken again later. A unit test specifically for this should prevent this.
I would look into it myself, but I have too much already on my plate and @amaltaro it seems to me you just recently looked at this code.

by "non-standard" you mean using the value specified during creation and not the default value?
Which spec files are we talking about?
Can you provide the dictionary passed during assignment for each T0 spec file that you want to have it tested?

Yes, LFNBase is set to something different than default during creation and the forceMerged = True is passed from within the spec (Express spec, maybe we can add a test in the Express spec unit test ?).
The calls the Tier0 uses to do the "assignment" are here:
https://github.com/dmwm/T0/blob/master/src/python/T0/RunConfig/RunConfigAPI.py#L676
https://github.com/dmwm/T0/blob/master/src/python/T0/RunConfig/RunConfigAPI.py#L1075
Pretty basic, just want to make sure that these type of calls don't automatically override LFNBase and the forceMerged setting.

Any news on this ?

Not yet, these output data checks are annoying and not a 15min task.
I might dedicate some time for it in the coming week though


docs: add default conf_target to new account cmd

the docs are manually mirrored in the builder's guide, i made the changes over there in this commit: https://github.com/lightninglabs/docs.lightning.engineering/commit/9fedf52389ffa3141372cd74a2ce2037f9b08204


open links in browser - not in a frameless chromium

https://github.com/atom/electron/blob/e0d0e7651fbeda2b84bad868ed159ffdaea32ab6/docs/api/shell.md#shellopenexternalurl


Fix extension grammar parsing

Extensions were not parsed correctly due to a typo in the Python script.

This needs to be merged after
https://github.com/KhronosGroup/SPIRV-Tools/pull/579

I've kicked off a new Travis-CI build.
I don't know how to restart the AppVeyor build.

Tested locally, inspected results manually too.
Rebased and pushed into master as 12f550973463


[Feature] Support for 100% stacked charts

What problem does this feature solve?
I am trying to emulate the functionality provided by Excel and echarts has no native support for 100% stacked charts.
I have tried to work around this by calculating the percentages myself but then you lose the original value tooltip.
There is an existing issue about this here:
https://github.com/apache/echarts/issues/8785
Apache superset has already implemented this feature, see here: (in the superset code it's called a 'contribution')
https://github.com/apache/superset/issues/13580
What does the proposed API look like?
My suggestion is that you extend the 'stack' syntax to support something like this:
stack: { name: 'male', type: '100%' }

The advised way to do so is to calculate percentage as you said. The tooltip can set formatter or valueFormatter to calculate the original value. If you or anyone from the Apache Superset community wish to implement this in Apache ECharts, we would appreciate it very much. Maybe an option like stackPercent: true should be added.

Hi @Ovilia, I tried implementing the tooltip in valueFormatter but I'm not given the params object (which is available a few levels up the call stack) so the only information I know is the percentage value. If I use the tooltip formatter instead then I lose the nice default tooltip containing the series colour and other information. Thanks for your help :)

You should probably save the sum to a variable that can be accessed by valueFormatter.

Hi. is this still updated

+1, Please implement this

Please checkout https://echarts.apache.org/examples/zh/editor.html?c=bar-stack-normalization and https://echarts.apache.org/examples/zh/editor.html?c=bar-stack-normalization-and-variation

Hi, I have looked at the two examples and these do work around part of the problem, but not fully.
By transforming the incoming data in to percentages we can no longer get access to the original values.
The result is that the tooltip can only show the percentage value, see below.

So the problem remains: I want to plot 100% stacked while keeping the source data as the original value?


Fix group with nearVector

What's being changed:
Currently a group and nearVector don't work together. For example this query should return a single value (the object with closest distance).
{
    Get {
        Test(
            nearVector: {
                vector: [1,2,3]
            }
            group: {
                type: closest
                force: 1.0
            }
        ) {
            name
        }
    }
}

In a similar way to https://github.com/semi-technologies/weaviate/issues/1958, group needs params.AdditionalProperties.Vector = true for getClassVectorSearch to work correctly.

Quick Q @trengrj – do you know if the docs assume this is possible? Or should we explicitly mention this?
(CC: @sebawita)

Quick Q @trengrj – do you know if the docs assume this is possible? Or should we explicitly mention this? (CC: @sebawita)

Our docs just have group by itself right now. We could do with more docs in this area.

We could do with more docs in this area.

Thanks @trengrj – @sebawita can somebody from your team help with this?

sure, @trengrj can you coordinate with JP on Slack?

Awesome, thank you! Could you rebase onto stable/v1.16? Then we could even release this right away instead of with v1.17 :-)

Branch has been changes to stable/v1.16 and I have cherry picked your go/x/net version bump @etiennedi.


Temperature control per 0.5 degrees

Is it possible to control the set target temperature per 0.5 degrees?

Or is this something I can fix in HA?

Implemented 0.1 degrees in V0.10

Wow you are so quick! Thanks! Much appreciated!
Just one small question:
Can you add type: thermostat to these element, so they can be used as thermostats?


[Content migration] - Asset Hub introduction, Register a Local Asset & Register a Foreign Asset

This PR aims to bring content from the polkadot-ecosystem-docs-draft repository
Migrated content:

Asset introduction
According to the spreadsheet, the location of this guide should be:
Polkadot Protocol > System Parachains > Asset Hub > Introduction to Asset Hub

Register a Local Asset
According to the spreadsheet, the location of this guide should be:
Develop > Parachain Developers > System Parachains > Register Your Native Asset on Asset Hub

Register a Foreign Asset
This one didn't exist on the spreadsheet, but I created a row there assuming that the location of this guide should be:
Develop > Parachain Developers > System Parachains > Register Your Foreign Asset on Asset Hub

@nhussein11 just some merge conflicts now

@nhussein11 just some merge conflicts now

Done! now, this needs one approval to be merged @dawnkelly09 @eshaben
Thanks!


fix Go SDK codegen for labservices

Package name can't be upper-case characters.
Make package name uniform with other SDKs.

Automation for azure-sdk-for-java

Automation for azure-sdk-for-js

Automation for azure-sdk-for-node

Automation for azure-sdk-for-ruby

Automation for azure-sdk-for-python

Can one of the admins verify this patch?

Automation for azure-sdk-for-go


Fix issue #1064

Remove the additional '_'.
Signed-off-by: David Geng<EMAIL_ADDRESS>

LGTM. Thanks


Strange behaviour for subscriber notifications

According to the documentation, subscribe(listener) should:

Adds a change listener. It will be called any time an action is dispatched, and some part of the state tree may potentially have changed. You may then call getState() to read the current state tree inside the callback.

Then one would assume notifications are actually delivered when getState() returns the state which was the result of the action dispatched.
But the current implementation of dispatch, actually lets a new action be dispatched, before all listeners are notified (if the action was dispatched by one of the listeners).
Therefore some listeners will get the notification of the second action, before the notification for the first action, but both with the latest state.
The current behavior makes the state.dirtyHandlerIds optimization in dnd-core end up missing updates to state.dragOperation.
When listeners do optimizations like dnd-core, will that not break the time travel in redux-devtool?
If the behavior is intentional, I think it should be documented, and the code should probably be optimized so that the new state is only notified once.

Good point. Can you raise this as an issue in dnd-core? The behavior is intentional; subscribers aren't guaranteed to get every state update, just the most recent one. If you have ideas how to optimize this please send a PR!

I have create a PR for documentation clarification.
I guess it would be possible to make sure that the listener is not called again with the same state.
It would be trivial if we added the non simplified fix for #1180 (https://github.com/slorber/redux/commit/ea8091b606efd31d416d1f04a63f9691b5d4ad9c), as:
     listeners.forEach(listener => {
-        if ( typeof listener !== 'undefined' ) {
+        if ( isDispatchingToListeners && typeof listener !== 'undefined' ) {
          listener()
        }

Would fix it.
But I agree with slorber that the non verified performance optimization is not worth the cost in code readability.


Fix broken locustfile.py

Description
This adds comments to ensure that imports are preserved in the locustfile.py. Without them, the linting tools treat the imports as unused and remove them.
Motivation and Context
Locust uses imports to locate test files. The locustfile.py contains imports that the linting tools don't realize are actually used.
How Has This Been Tested?
I ran the linting tools locally.
Checklist

[ ] I have updated the documentation accordingly.
[x] All new and existing tests passed.

Hm, something happened with my local repos here. Let me rebase and push again...


Minimal MDNS - start a generic server implementation

Problem
Minimal MDNS requires both client and server functionality (sending queries and receiving replies).
Summary of Changes
Started generic server support:

bind to all interfaces (important for IPV6, where link-local addresses are per interface)
parsing queries and replies
added command line options for a server and client app
fixed some code review comments that my previous PR missed.

Still need to add reply generation methods. Code is more clean though

@saurabhst @BroderickCarlin @hawk248 @jelderton ?


Can you add 1.18.2?

Hello,
the last app version that supports MC 1.18.2 is 1.2.3.
You can get it here https://github.com/crpmax/mc-bots/releases/tag/v1.2.3


File Version Missing

For the table data section and dialog, the field "File Version" is missing.

Added file version to the table settings section.


Move MUX Effects to private namespace

This is a backport of a fix for WinUI3. It has no specific impact for WinUI2, but porting it now will make future merges of WinUI2 into WinUI3 easier.
MUX has a set of Effects that it uses for its own internal implementation purposes. It is not intended for them to be used by other consumers.
These effects are in the Microsoft.UI.Composition.Effects namespace. In WinUI2, these types did not get included in the nuget package, since that only includes the Microsoft.UI.Xaml namespace. So these types were correctly filtered out.
However, WinUI3 publishes everything under Microsoft.UI, so we were unintentionally publishing these internal types. The fix is to move these types to a Microsoft.UI.Private namespace so that they get filtered out.

/azp run


Changes for State and test organization

Updated from clean base.

@HaoK @Eilon @tushargupta51 What is needed to pull this code?

https://github.com/aspnet/Security/issues/214

Reissued as:
https://github.com/aspnet/Security/pull/301


Support for the largePoints feature in Vulkan

Please support the largePoints feature in Vulkan,
Test case:
Compute N Bodies Simulation from Sascha Willems tests.
https://github.com/SaschaWillems/Vulkan/blob/master/examples/computenbody/computenbody.cpp
See also:
https://groups.google.com/forum/#!topic/swiftshader/LPZWJxIEoiQ

This is just copy of https://swiftshader.googlesource.com/SwiftShader, you should open issue there...

For future reference, it seems this was fixed in March 2020 https://swiftshader-review.googlesource.com/c/SwiftShader/+/41948


Delete session if secret id is not found

This loses the feature where if there is no id then ID of 1 is assumed. Was that the intent? As it is now, if the session cookie does not contain &id= then shouldDestroy is set to true and the first (most recent) secret is not tried.


Backport: Set content type if empty for gzipped responses

What kind of request is this (question/bug/enhancement/feature request):
bug
Steps to reproduce (least amount of steps as possible):

Open rancher using firefox browser
Enable monitoring for a cluster
Navigate to that cluster

Result:
Error messages caused by failure to parse JSON.
Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): 2.2.2-rc8

backport of https://github.com/rancher/rancher/issues/19507

in rc9

Replicated in v2.2.2-rc8
Validated in v2.2.2-rc9
Test with FireFox.

Create any cluster, enable monitoring

No longer get any errors as previously observed with rc8

Smoke test monitoring, ensure is working without any issues (all charts not just on main cluster page, include workloads, pods, containers).

Set up a pipeline, ensure this works as well without any issues.

For both monitoring and pipeline, ensured json response now appears correct.
Content-Type is as expected. For example pipeline configs query:

And response is proper json now.


Compatibility with Camful

Hi, I want to use your mod in my Modpack "Better Minecraft Modpack", 18K+ Downloads and I have a mod in it called Campful: https://www.curseforge.com/minecraft/mc-mods/campful. It seems that when I light the Campful campfires that they instantly go out. I was hoping you can make it Compatible so I can add it to my Modpack.
Thanks

Ok, I'll try to do it.

I added compatibility using a separate datapack. This isn't exactly the best solution for modpack AFAIK, so you could add the "data" folder of datapack directly to mod jar of Campful (it is loaded after my mod because of alphabetical load order) and it should work. I will look for a better solution, but for now that should do.


netlink: report neighbors

run scale-tests

nit:
s/nelink/netlink in commit msg

@masco-enovance Done


How to add image into rss-feed to "Data Output Agent" output  from URI?

How to add this?

Bonus points for making the image clickable, also works for custom youtube feeds with a clickable video thumbnail

In my case, I just add rss_content_type: application/xml in Options using the DataOutputAgent(the default value of rss_content_type is application/rss+xml), and then, '<' or '>' will not be escaped, it works.


pydeck: Update for new @deck.gl/json API and add additional tests

For #3640
Concludes feature work necessary to correspond to deck.gl 8.0 and is where I'll cut the pydeck 0.2.0 release.

Background

Change List

Add additional tests
Update pydeck API

Coverage decreased (-2.5%) to 80.601% when pulling d06e97617ae8de09d331402027c934624a4a671d on ajd/verify-pydeck into c44bf9a88eeedd3fd9bc5d267534f41f08269be6 on master.


Tokenizer redesign for better model-specific feature support

Motivation

Our tokenizers are not model specific and are rather tied to the base tokenizer, e.g., SentencePiece (SP) or TikToken (TT). These both implement tokenize_messages but this is not configurable for model-specific special tokens
Both of these are highly coupled with llama2 and llama3. For example, all the special tokens for TikToken are llama3 specific and it is difficult to customize
This has led to Phi3 having its own tokenizer file that reimplements SentencePieceTokenizer, but with custom special tokens. Ideally, this should be easily composable by configuring the phi3 specific tokens with SentencePiece as the base
This problem will only get worse as foundation models add more capabilities such as tool calling to their respective tokenizers. For example, Mistral v3 now uses their own special tokens on top of SentencePiece, and we have no way to onboard this besides copying and reimplementing SentencePieceTokenizer. Redundancy is normally ok but not if it can be easily addressed with composability.
If a new base tokenizer is used besides SentencePiece or TikToken, we should be flexible enough to quickly implement the underlying new encode and decode while still maintaining our higher level APIs (tokenize_messages, etc)

Design proposal
Intuition: separate the two core APIs (encode/decode and tokenize_messages) that operate at different levels of abstraction:

Encode/decode live in the base token encoding layer (SP or TT) and does not need model specific logic. This should be pulled out so any model can quickly use SP, TT, or any other base token encoding for encode/decode.
tokenize_messages requires model specific special tokens placed in particular locations. Thus, there should be a model specific class that implements tokenize_messages

We can achieve the above with BaseTokenizer and ModelTokenizer.

BaseTokenizer is the base abstract interface for any base tokenization model (SP or TT) that implements encode and decode
ModelTokenizer is the base abstract interface for any model-specific tokenizer that implements tokenize_messages. All models will implement their own Tokenizer class based on this interface so they can control tokenize_messages logic

class BaseTokenizer(Protocol):
    """Abstract token encoding model"""

    def encode(self, text: str, **kwargs) -> List[int]:
        """
        Given a string, return the a list of token ids.
        """

    def decode(
        self, token_ids: List[int], include_special: bool = False, **kwargs
    ) -> str:
        """
        Given a list of token ids, return the decoded text.
        """

class ModelTokenizer(Protocol):
    """
    Abstract tokenizer that implements model specific special token logic in
    ``tokenize_message`` and ``tokenize_messages`` methods.
    """

    def tokenize_messages(
        self, messages: List[Message], **kwargs
    ) -> Tuple[List[int], List[bool]]:
        """
        Given a list of messages, return a list of tokens and list of masks for
        the concatenated and formatted messages.
        """
        pass

This means the SentencePieceTokenizer and TikTokenTokenizer will be refactored to separate out encode/decode and tokenize_messages logic.
class TikTokenBaseTokenizer(BaseTokenizer):
    def __init__(
        self,
        path: str,
        pattern: str,
        name: str,
        special_tokens: Dict[str, int],
    ):
        mergeable_ranks = load_tiktoken_bpe(path)
        self.tt_model = Encoding(
            name=name,
            pat_str=pattern,
            mergeable_ranks=mergeable_ranks,
            special_tokens=special_tokens,
        )
        ...

    def encode(
        self,
        text: str,
        add_bos: bool,
        add_eos: bool,
        allowed_special: Set[str] = set(),
    ) -> List[int]:
        ...

    def decode(
        self,
        token_ids: List[int],
        truncate_at_eos: bool = True,
    ) -> str:
        ...

class SentencePieceBaseTokenizer(BaseTokenizer):
    def __init__(
        self,
        path: str,
    ):
        spm_model = SentencePieceProcessor()
        spm_model.load(path)
        self.spm_model = spm_model
        # Special tokens are defined in the protobuf itself

    def encode(
        self,
        text: str,
        add_bos: bool,
        add_eos: bool,
        allowed_special: Set[str] = set(),
        trim_leading_whitespace: bool = False,
        prefix: Optional[str] = None,
    ) -> List[int]:
        ...

    def decode(
        self,
        token_ids: List[int],
    ) -> str:
        ...

And any model tokenizers would compose with the above classes:
class Llama3Tokenizer(ModelTokenizer):
    """
    tiktoken tokenizer configured with Llama3 Instruct's special tokens, as described in
    https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3

    Args:
        path (str): Path to pretrained tiktoken tokenizer file.
    """

    def __init__(
        self,
        path: str,
    ):
        all_special_tokens_with_ids = self._get_all_special_tokens_with_ids()
        self.tt_model = TikTokenBaseTokenizer(
            path=path,
            name="llama3_tiktoken",
            pattern=CL100K_PATTERN,
            special_tokens=all_special_tokens_with_ids,
        )

    def tokenize_messages(
        self,
        messages: List[Message],
        max_seq_len: Optional[int] = None,
        tokenize_header: bool = True,
        add_eos: bool = True,
    ) -> Tuple[List[int], List[bool]]:
        ...

Changelog

Refactor SentencePieceTokenizer -> SentencePieceBaseTokenizer, TikTokenTokenizer -> TikTokenBaseTokenizer and only retain encode/decode logic
Move tokenize_messages logic to Llama2Tokenizer and Llama3Tokenizer for SP and TT, respectively
Create GemmaTokenizer and MistralTokenizer which leverage SentencePieceBaseTokenizer. The tokenize_messages logic is identical to Llama2Tokenizer (for now, Mistral needs to be updated with v3)
Refactor Phi3's tokenizer to use SentencePieceBaseTokenizer and retain special token logic
Revamp DummyTokenizer since it inherited previously from SentencePieceTokenizer
Update all tests and docstrings

Test plan

Added unit tests for all tokenizers, pytest tests runs successfully
Manual comparison of every updated tokenizer with the old version: https://gist.github.com/RdoubleA/a62855ae5440063d120de3c908c903e0

Planned follow-ups

Tokenizer live docs / API ref updates

Overall I like the proposal. Still going through all the code, but the division of encode and tokenize_messages APIs you've provided here definitely makes sense to me. One big question I have is whether we can do this in a way where we don't have to write an entirely new tokenizer class every time we onboard a new model. I get that if there's custom logic around special tokens we can't really avoid it, but for vanilla sentencepiece without much special formatting it'd be nice to make this process a bit more lightweight (e.g. Gemma and Llama2 tokenizers have the same tokenize_messages API, and imo copy-pasting this code is not as nice as than copy-pasting model builders, which are generally pretty straightforward to understand).
Also:

Move torchtune.modules.tokenizers to torchtune.data.tokenizers which is a more relevant location

I understand that this is a useful thing to do, but it makes reviewing this code harder and is kinda logically distinct from the important stuff in this PR (imo). With this move, I have to look at tokenizers in their entirety; without it I can actually see the diff.

Want to express my uncertainty on the TokenEncoding naming. This seems a bit unintuitive since I'd imagine the encoding to refer to something like UTF8.

Yeah I agree I don't like the TokenEncoding naming either, but naming SP and TT as Tokenizer and everything else as ModelTokenizer is more confusing imo... there should be a clear distinction between a model tokenizer and the base tokenizer. Maybe we could call it SentencePieceBaseTokenizer and TikTokenBaseTokenizer

@joecummings huh I guess Gemma does have special tokens, but our current tokenizer does not use them. what do you think about either punting the upgrade to later or parsing the special tokens from the HF json for now and then adding them in tokenize_messages as appropriate later? Same question for Mistral

Codecov Report
Attention: Patch coverage is 93.66516% with 28 lines in your changes missing coverage. Please review.

Project coverage is 67.45%. Comparing base (f200da5) to head (1d6e5e3).
Report is 1 commits behind head on main.

Files
Patch %
Lines

torchtune/modules/tokenizers/_utils.py
80.48%
8 Missing :warning:

torchtune/models/llama3/_tokenizer.py
88.33%
7 Missing :warning:

torchtune/models/gemma/_tokenizer.py
89.28%
3 Missing :warning:

torchtune/models/llama2/_tokenizer.py
89.28%
3 Missing :warning:

torchtune/models/mistral/_tokenizer.py
89.28%
3 Missing :warning:

torchtune/models/phi3/_tokenizer.py
84.61%
2 Missing :warning:

recipes/eleuther_eval.py
0.00%
1 Missing :warning:

torchtune/modules/tokenizers/_tiktoken.py
96.55%
1 Missing :warning:

Additional details and impacted files
@@             Coverage Diff             @@
##             main    #1082       +/-   ##
===========================================
+ Coverage   26.74%   67.45%   +40.71%
===========================================
  Files         183      191        +8
  Lines        8362     8498      +136
===========================================
+ Hits         2236     5732     +3496
+ Misses       6126     2766     -3360

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


embed code need longer char length than 32 chars

available in 0.6.343
thru migration m200714_140222_embed_longer


WIP: Hs editing provider front end

Some front-end changes as we implement the verification and provider updating processes.
This relates to this Trello card: https://trello.com/c/VdEJxyna/291-show-verification-requests-at-the-top-when-more-information-is-needed-and-tidy-up-homepage-generally

@mrchrisadams I'm assigning this to you to take a look at and merge in.
Because the new hosting provider functionality is hidden behind a flag, I don't believe this needs to wait. Deploying this now should deliver some improvements to the existing portal home, which will be helpful for Fish and users. And then when we deploy the provider editing stuff and lift the flag, the new edit listing button will become visible too.
So basically as long as I haven't done anything wild in the code I think we should merge now.


istio*.yaml -> istio-demo*.yaml

Istio files in v0.8.0 are now called istio-demo.yaml and istio-auth-demo.yaml.
This is a quick fix for issue: #5985 however, as mentioned there, for future releases we might want to install using helm.

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: ymesika
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: sdake
Assign the PR to them by writing /assign @sdake in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

install/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ymesika: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-pilot-e2e.sh
afd3a5b66f92f1959d0b79c61a51d33d95825535
link
/test istio-pilot-e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test istio-pilot-e2e

/test istio-pilot-e2e

@ymesika
I'm not a formal reviewer, but I can confirm this fixes #5985.
When you say using helm for future releases, is it deprecate Ansible installer in favor of helm ... or is it using helm commands on ansible playbooks?

@gbaufake I thought about the second option - use helm command within the playbooks.
Does it make sense?

@ymesika
it does.
Thanks for the clarification!

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: ymesika
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: mandarjog
Assign the PR to them by writing /assign @mandarjog in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: ymesika
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: sdake
Assign the PR to them by writing /assign @sdake in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

install/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Closing this one for a cleaner PR and will re-open a new one with master as the base.

@ymesika: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-simpleTests.sh
b5ccd7d6a641e51e5479a050720956899856d741
link
/test e2e-simple

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ymesika: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-simpleTests.sh
b5ccd7d6a641e51e5479a050720956899856d741
link
/test e2e-simple

prow/istio-pilot-e2e.sh
b5ccd7d6a641e51e5479a050720956899856d741
link
/test istio-pilot-e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


Question: Linux Kernel Keyring lost the saved password after reboot

I'm using this crate in headless mode with kernel keyring. However, my cosumer and myself have found that the saved password will be lost after reboot the system. I haven't found any information or documentation about that. Is it by design? Could I do someting to preserve the password?

cc'ing @landhb since he is more familiar with the kernel keyring.
My understanding (and what's documented in the keyring keystore itself) is that credentials are placed in the user's persistent keyring, which has a default persistence of 3 days.  This lifetime can be adjusted by administrators.  They keystore docs point you at the appropriate man pages for the underlying behaviors.

@Berrysoft @brotskydotcom That's correct, the kernel keystore is completely in-memory and will clear on a reboot.
The "persistent" keyring is persistent across sessions/logins but will still be wiped on a reboot.
Similar to how you need to unlock the Gnome Keyring with a password when you log-in, you'll need to re-prompt your users for a password after a reboot.
I noticed you're using a systems service, there is a systemd-ask-password utility that can be used to prompt the user when your service starts. manpage.

Thanks @landhb ! I think I need some other ways to save the password then:)

I am reopening this as a doc bug because it's the second time it's come up; I had totally forgotten about it, and it's not documented in the keystore docs.  So I need to fix that.  Thanks for the discussion.
@Berrysoft There is a headless credential keeper for Unix systems called pass that you might want to check out.  Here is a good article that explains how to run it headless and securely; here is another article that also talks about how to use it with GUI apps such as a browser.  I would certainly welcome the contribution of a pass-based keystore for this module.

To clarify. This issue is a limitation for all secure storage.

Backend
Memory Eviction
User Interaction

Gnome Keyring
On Logout
User must re-enter their account password to unlock the keyring after logging in again.   Unlocking the keyring with a master/account password allows Gnome to retrieve the encrypted credentials from disk.

Kernel Keyring
On Reboot
User must re-enter the credential or the application must prompt the user for a master password (to decrypt the contents from disk) or obtain the credential from a secure storage medium such as a Yubikey/TPM.

Pass
Typically every run
User must re-enter the master password to decrypt the contents from disk.

In general, to have your secrets accessible across reboots, something needs to be able to decrypt/access it securely from the filesystem.
That passrs crate is also highly insecure and stores the keys on disk in plaintext just in reverse order. So I wouldn't recommend using that crate. I also think password managers in general operate at a higher level of abstraction than this crate. A pass implementation could use this crate to store the master password between invocations of the CLI tool for instance.

Well, I think I need a beckend that is headless and unlocks on login. Gnome keyring unlocks on login, but it's not headless, and not all users install it. I choose to save the password directly on disk. It is a bit unsafe, but it is convenient.

@landhb Do the persistent credentials in the kernel keyring persist on disk between reboots or not?  In the case of secret-service and pass, the credentials are all persisted on disk and there is a single master password that must be provided to decrypt them.  My impression from your comments above is that the kernel keyring doesn't do any on-disk persistence, so if one has stored multiple credentials in it they are all lost on reboot.  That's quite a different discipline than the others.  That's what I think - if true - needs to be mentioned in the docs.
@Berrysoft The master password for pass is the password for the master GPG key used with the store.  Most pass users arrange for this to be prompted for at headless login.

@brotskydotcom It does not persist on reboot. I agree that it should be mentioned in the docs.

Thanks much to @landhb for the documentation fix (PR #138)!


Make pg instrumentation a no-op

Since we're publishing 2.0.0 we can do this for temporary relief for #254, as discussed privately.
/cc @MaxBittker @benvinegar

Sure, updated.


make gp3 default

@razvan-moj are we able to use https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/storage_class?


The stCN-generator_seed_pi branch is useless?

The stCN-generator_seed_pi branch is useless?It was deleted from your project

It is one of the experiments that was later found to be of no help to training stability.


debugTarget failed when debug_config.environment not present.

        debug_config.environment = debug_config.environment ? debug_config.environment.concat(util.makeDebuggerEnvironmentVars(configure_preset_environment)) : [];

means when debug_config have no environment  then  we can not debugging at all
even thout preset or toolkist are already configured it properly

The fix for this issue is available in 1.10 "pre-release".  You can help us validate it by:

Opening the extensions panel in VS Code
Selecting the CMake Tools extension
Clicking the "Switch to Pre-release Version" button


Added solve() function to all backends.

Added Solve class to keras/ops/math.py
Added solve function to keras/ops/math.py
Added solve function to all backends.
Added test cases.

Codecov Report
Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (080bca8) 78.91% compared to head (27dae3d) 56.26%.

Additional details and impacted files
@@             Coverage Diff             @@
##           master   #18744       +/-   ##
===========================================
- Coverage   78.91%   56.26%   -22.65%
===========================================
  Files         336      336
  Lines       33974    34004       +30
  Branches     6656     6657        +1
===========================================
- Hits        26811    19133     -7678
- Misses       5584    13422     +7838
+ Partials     1579     1449      -130

Flag
Coverage Δ

keras
56.26% <56.66%> (-22.55%)
:arrow_down:

keras-jax
?

keras-numpy
56.26% <56.66%> (+<0.01%)
:arrow_up:

keras-tensorflow
?

keras-torch
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files
Coverage Δ

keras/backend/numpy/math.py
83.11% <100.00%> (+0.45%)
:arrow_up:

keras/backend/jax/math.py
26.89% <25.00%> (-52.24%)
:arrow_down:

keras/backend/tensorflow/math.py
16.26% <25.00%> (-63.58%)
:arrow_down:

keras/ops/math.py
78.17% <78.57%> (+0.01%)
:arrow_up:

keras/backend/torch/math.py
0.00% <0.00%> (-87.50%)
:arrow_down:

... and 181 files with indirect coverage changes

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Update Docker build files to use branch release 2.0 of OpenImageIO

Thanks @thirdknife to improve the docker file with a better OIIO version. We may have notice that boost is now not used in the master branch. If you have some time, could take that opportunity to also remove the boost step in the docker files?


Crash "Apps may not schedule more than 100 distinct jobs"

Hi, I am testing an app since a few days I have been scheduling tasks with workmanager everyday. today I added a new one and I got a crash with the following log (all my tasks of the previous days were already finished).
Now the app can not start anymore, unless I delete all the app's data.
1-Does this mean I had 100 tasks during the last few days?
2-Or does this mean my last task was accidentally launched >100 times?
if 1, why are we limited to 100 tasks
Is it a known issue? any help would be appreciated. thanks
06-01 13:11:09.180 696-4700/system_process W/JobSchedulerService: Too many jobs for uid 10095
E/AndroidRuntime: FATAL EXCEPTION: pool-2-thread-1
Process: tv.ebox.freetv.debug, PID: 26258
java.lang.IllegalStateException: Apps may not schedule more than 100 distinct jobs
at android.os.Parcel.readException(Parcel.java:1691)
at android.os.Parcel.readException(Parcel.java:1636)
at android.app.job.IJobScheduler$Stub$Proxy.schedule(IJobScheduler.java:158)
at android.app.JobSchedulerImpl.schedule(JobSchedulerImpl.java:42)
at androidx.work.impl.background.systemjob.SystemJobScheduler.scheduleInternal(SystemJobScheduler.java:85)
at androidx.work.impl.background.systemjob.SystemJobScheduler.schedule(SystemJobScheduler.java:64)
at androidx.work.impl.Schedulers.scheduleInternal(Schedulers.java:98)
at androidx.work.impl.Schedulers.schedule(Schedulers.java:69)
at androidx.work.impl.utils.EnqueueRunnable.scheduleWorkInBackground(EnqueueRunnable.java:109)
at androidx.work.impl.utils.EnqueueRunnable.run(EnqueueRunnable.java:81)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
at java.lang.Thread.run(Thread.java:761)

@kvillnv i had this issue in my tests, then i started playing with uniqueWork and the problemas was gone.

@biliboss thank you for your fast response.
I am going to try beginUniqueWork() instead of enqueue()

@biliboss it seems it did not fix my issue.
here is how I use uniqueWork
val recorderWork = OneTimeWorkRequestBuilder()
.setInputData(data)
.addTag(mediaName)
.setConstraints(constraints)
.setInitialDelay(initialDelay, TimeUnit.SECONDS)
.build()
WorkManager.getInstance()
.beginUniqueWork(
mediaName,
ExistingWorkPolicy.REPLACE,
recorderWork).enqueue()
I tried unique tags with ExistingWorkPolicy.REPLACE
and I also tried with same tag and ExistingWorkPolicy.APPEND
it seems that sometimes work manager does create more than 100 hundred SystemJobScheduler at once

@kvillnv could you confirm that problem occurs only after open your app 100 times?
In your code, mediaName is a Constant?
Are you sure that your Worker isn't stay in a infinite loop?

@biliboss
-The problem can occur on a fresh install: Actually everytime I create one task, it seems that all the existing tasks are also re-created so when I have 8 tasks, when I create the 9th, the 8 others are also recreated, so I end up with 17 tasks. I can see that because there is a log from Android's SystemJobScheduler class that is repeating as many times ("Scheduling work ID %s Job ID %s"). I am able to recognize the WorkIds of all the previous tasks, and I can see the JobID incrementing  (to 17 in this case).
-When I used ExistingWorkPolicy.APPEND, mediaName was a constant, and when I tried ExistingWorkPolicy.REPLACE, mediaName was a variable with a different value everytime.
I confirm,there is no infinite loop, according to my logs, my method WorkManager.getInstance() .beginUniqueWork(...).enqueue() is called only once.
But I can see the method scheduleInternal() from Android's SystemJobScheduler is repeating as many times as the task is created.
What ExistingWorkPolicy TAG fo you use in the beginUniqueWork() ?
Do you use the method the same way I wrote in my previous message?

Hello @kvillnv.
I used KEEP in the start of program, and APPEND if it's called by itself (worker).
Do you already confirm if your Worker is returning a result? We saw this log here.
06-04 10:05:53.398 25529-25910/? D/WorkerWrapper: Worker result FAILURE for 417f62a3-0f24-4fb9-8884-cb440adae758

Hi @biliboss
The problem occurs before the result (the tasks are scheduled later in the day).
If I wait for my first time be executed, when I create the second one I won't have the repeating issue, and it seems that the duplicated tasks will be executed only once..
When my tasks are executed I always have result SUCCESS
Thanks for your help.
My mistake must be somewhere else. I'm going to recheck again all my implementation.

This is a bug.
When scheduling a new job in the future and already having several other job scheduled in the future -> the whole job list is duplicated every time.

I decided to remove workmanager from my project.
Finally I implemented Android JobService and I schedule with JobScheduler and JobInfo.
Now everything works fine.
Maybe workmanager is good for periodic tasks or short-term scheduled tasks in chronological order. But when you want to schedule at specific times differents jobs before or after another one it does not work.

I rolled back to the Alarm Manager after figured that workmanager isn't have a guaranteed timed execution, and so a little buggy yet.
I think that Work Manger isn't production ready.

@kvillnv  i used for periodic tasks and got the the bug mentioned before.

I got the same bug.

Got the same bug for periodic tasks

Got it too, here's how I reproduced it:

I scheduled 3 periodic (each day) task in my app
Using JobScheduler Api, we can see , enqueued task.
Then I use the ADB Idea plugin to restart my app. It force-kill it, then start it.

What happens is that each force-kill will reschedule tasks as seen in the ForceStopRunnable class from the WorkManager source.
So now I have 6 tasks scheduled. Doing so on and on will triggers the 100 limit.
Here's the only way I found to prevent crash without needing the user to clear data:
In your application class
        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP) {
            val jobScheduler = getSystemService(Context.JOB_SCHEDULER_SERVICE) as JobScheduler
            val jobCount = jobScheduler.allPendingJobs.count()
            if (jobCount > 50) {
                jobScheduler.cancelAll()
                val workManager = WorkManager.getInstance()
                AsyncTask.execute {
                    val dao = (workManager as WorkManagerImpl).workDatabase.workSpecDao()
                    val ids = dao.allWorkSpecIds
                    ids.forEach { dao.delete(it) }
                }
                Timber.e("job scheduled was above 50, cancelling all")
            }
        }

It will cancel all JobScheduler Tasks if there are more than 50 scheduled, and clear the WorkManager DB that will reschedule them. It's part from private API of WorkManager but hey, it's the only way I found ¯_(ツ)_/¯
I really think this bug is critical so I hope we will get some feedback from dev team soon.

Same here with WorkManager.beginUniqueWork("WORKER_CLASS_NAME", ExistingWorkPolicy.REPLACE, OneTimeWorkRequest).enqueue();
Fatal Exception: java.lang.IllegalStateException
Apps may not schedule more than 100 distinct jobs
android.os.Parcel.readException (Parcel.java:1691)
android.os.Parcel.readException (Parcel.java:1636)
android.app.job.IJobScheduler$Stub$Proxy.schedule (IJobScheduler.java:158)
android.app.JobSchedulerImpl.schedule (JobSchedulerImpl.java:42)
androidx.work.impl.background.systemjob.SystemJobScheduler.scheduleInternal (SystemJobScheduler.java:126)
androidx.work.impl.background.systemjob.SystemJobScheduler.schedule (SystemJobScheduler.java:95)
androidx.work.impl.Schedulers.scheduleInternal (Schedulers.java:104)
androidx.work.impl.Schedulers.schedule (Schedulers.java:73)
androidx.work.impl.utils.EnqueueRunnable.scheduleWorkInBackground (EnqueueRunnable.java:109)
androidx.work.impl.utils.EnqueueRunnable.run (EnqueueRunnable.java:81)
java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1133)
java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:607)
java.lang.Thread.run (Thread.java:761)

Same bug. I scheduled job only once, and Android 7.0 and 7.1 versions do get this bug:
`        if (!PersistentStorage.isTaskScheduled())
{
try
{
WorkManager.getInstance().cancelAllWorkByTag(TAG);
PeriodicWorkRequest request = new PeriodicWorkRequest.Builder(MyWorker.class, 15, TimeUnit.MINUTES)
.addTag(TAG)
.build();
            WorkManager.getInstance().enqueue(request);
            PersistentStorage.setTaskIsScheduled();
        }
        catch (Exception e)
        {
            e.printStackTrace();
        }
    }`

Sorry, but this is a bug which we addressed in alpha05 (coming soon). For more information look at: https://issuetracker.google.com/issues/111195153

Hi alpha05 is already release.
Please visit below link for more information on changes:
https://developer.android.com/jetpack/docs/release-notes

It’s the same bug. There are a few duplicates to the bug I linked as well. alpha05 should fix the problem.

I can confirm that the issue is not fixed for me either.

Same for me, is crashing on Android 8

There is a bug on the issue tracker bug around this. alpha07 should fix this.

@tikurahul you have the issue link?

Thanks for reporting this.
I'm closing this issue because this forum is for issues with the codelab.
If you think that this is a problem of the WorkManager library, please open a bug on the WorkManager public issue tracker: https://issuetracker.google.com/issues/new?component=409906&template=1094197
Please reopen if you still think this is an issue with the codelab.


Duplicate Refresh icon displaying on 3rd party extensions sidebars

Issue Type: Bug
Open VS Code an navigate to the sidebar for any of the extensions listed below :

Docker 1.0.0 - ms-azuretools.vscode-docker
Oracle Developer Tools for VS Code - oracle.oracledevtools
Test Explorer UI - benl.vscode-test-explorer

Also visible in Maven projects view in the file explorer.
 For example can you disable every extension except the docker extension?
I can't reproduce the issue, which leads me to think that you have an extension installed that is adding lots of refresh buttons in views it doesn't own.

@alexr00 I did as you advised and found the extension causing the issue, I logged an issue on there
https://github.com/ShawnZhong/vscode-system-task-manager/issues/1


[stable/percona-xtradb-cluster] Add maver1ck to OWNERS

Signed-off-by: Maciej Bryński<EMAIL_ADDRESS>What this PR does / why we need it:
Percona-xtradb-cluster is not maintained anymore. There are several PR waiting to merge.
Some of them get stale recently.
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[X] DCO signed
[X] Chart Version bumped

@paulczar ptal

/approve

@davidkarlsen lgtm please :)


The process has forked on macbook when running with python dataiter code

I am using my own data iter to train resnet, and encountered  error:
The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC() to debug.

My data iter code is like this:
`
class DataIter(mx.io.DataIter):
def init(self, images, image_pairs, image_similar_neg, batch_size, height, width, process_num):
assert process_num <= 40
super(DataIter, self).init()
self.batch_size = batch_size
self.conut = len(images)
self.height = height
self.width = width
self.images = images
self.image_pairs = image_pairs
self.image_similar_neg = image_similar_neg
self.cursor = -self.batch_size
self.provide_data = [("positive", (self.batch_size, 3, height, width)),
("negative", (self.batch_size, 3, height, width)),
("one", (self.batch_size, ))]
self.provide_label = [("anchor", (self.batch_size, 3, height, width))]
self.queue = multiprocessing.Queue(maxsize=4)
self.started = True
self.processes = [multiprocessing.Process(target=self.write) for i in range(process_num)]
for process in self.processes:
process.daemon = True
process.start()`

It seems that macos doesn't support fork reliablely. I am wandering how to implement a dataIter on macbook.

This issue is closed due to lack of activity in the last 90 days. Feel free to ping me to reopen if this is still an active issue. Thanks!


EVP: use evp_pkey_ctx_is_legacy() to find what implementation to use

We've had explicit checks for when to fall back to legacy code for
operations that use an EVP_PKEY.  Unfortunately, the checks were
radically different in different spots, so we refactor that into a
macro that gets used everywhere.

Merged
f21c9c64f53484d4abe25b76d29350ed683db855 EVP: use evp_pkey_ctx_is_legacy() to find what implementation to use


Explicitly declare the used but undeclared dependency.

Fix #2001 .

@stevehu Could you please take a look at this pr? Thx :)

This is a transitive dependency from the Undertow. Do you have a reason to add that explicitly?

Thanks for your quick reponse :).
The problem mainly lies in that light-4j use the classes in xnio-api directly as mentioned in #2001. This means that light-4j relys on xnio-api itself directly. It's certain that you don't need to declare all the transitive dependency used by your directly dependency because maven will do the resolution, but xnio-api here is actually your direct dependency.
If you uses the transitive dependnecy pulled in by undertow directly without declaring it, maven won't even know that your project relys on xnio-api, thus losing full control over your project's dependency. Let's assume the extreme cases (though it may not happen), if undertow abandons the use of xnio-api, light-4j won't be able to build. Or maybe undertow use upgraded xnio-api that removes certain classes, same thing will happen. The possible outcomes are induced by the used but undeclared dependency.
Actually, maven itself takes the act of using transitive dependency directly without declared as inappropriate. This can be proved by its official dependency managment plugin. You can take a look at https://maven.apache.org/plugins/maven-dependency-plugin/analyze-only-mojo.html if you are interested.  Hope that my reasons are sufficiently convincing, thanks for your patience :) @stevehu .

@HappyHacker123 I understand your concerns. We didn't put the xnio into the dependency because that is the hard dependency from Undertow, and it won't be changed in the feature confirmed by RedHat. By leaving xnio out of our dependency, we don't need to manage that library upgrade. At this moment, I think it

We didn't put the xnio into the dependency because that is the hard dependency from Undertow, and it won't be changed in the feature confirmed by RedHat.

@stevehu Thanks for your clarification :). I can see that optimizing the dependency structure may potentially introduce additional maintenance labor. So if it's confirmed by RedHat, then i understand your decision.

It is hard to draw the line for managing dependencies. If we include all transitive dependencies, we must add at least 30 more. So we must find a threshold between too few and too many. It was a good discussion and I never got a chance to think about it in depth. Thanks.

@HappyHacker123 Based on our conversation, I am closing this PR. Thanks.