API Functions for Exchange Implementation

Some additional features are needed for exchange implementation to be possible:

An API method that lists all transactions the wallet is involved with, including both sending and receiving. Each transaction listed should include at minimum the amount, the number of confirmations, the destination address, the transaction ID/hash, and the timestamp.

Methods for wallet encryption. This could be like bitcoin, where there is an unlock call and a lock call, or the encryption key/passphrase could be supplied as a param of the /wallet/send method.

@poloniex
Sorry it took so long to get here, but I'm working on the wallet right now. The new wallet should be coded up by Aug. 1st, and the first release with the changes should be on Aug. 15th.
Wanted to run the api calls by you though.
/wallet (GET) :: display the balance of the wallet, both siacoins and siafunds, potentially with additional information
I wasn't planning on having this call list all transactions related to the wallet, it seemed unwieldy.
/wallet/address (GET) :: list all active addresses (and maybe their respective balances)
/wallet/address (POST) :: returns a new address that can receive coins
/wallet/$(ADDR) ::  display the balance of an address, and return every txn related to the address. Each txn will also have a number of confirmations. The receiver will need to parse the transactions to figure out how many confirmations the balance as a whole has (or some subset of the balance). In the simplest case, this just means looking at the # of confirmations on each transaction and then taking the lowest number - that will give you the number of confirmations on the balance.
From the transaction you will be able to parse the amount, the confirmations, the source(s), the destination, the ids (both of the outputs and of the transaction), and the timestamp, as well as the actual raw transaction.
/wallet/siacoins/send (POST) :: requires a password, which will be established by the user when the wallet is created. The wallet will temporarily decrypt the private key, send the coins, and then wipe the decrypted key from memory.
/wallet/siafunds/send (POST) :: same as /wallet/siacoins/send, except it sends siafunds.
There will be a few other calls related to backups, setting passwords, and similar functions, but the above are the ones related to sending and receiving coins.

If /wallet/siacoins/send automatically selects inputs from all addresses in the wallet (rather than requiring a single source address to be specified), then an exchange will need a method to list all transactions in the wallet. If you provide only the methods proposed above, I would need to construct this list using /wallet/$(ADDR) by querying every deposit address, collecting the txns, and (if /wallet/$(ADDR) provides only txns and no further transaction information) fetching the amount, destination, and confirmations of each transaction. As more deposit addresses are generated, this could end up becoming very slow; it is preferable to minimize the number of API queries needed.
Unless sending funds must be done by specifying both a source and a destination address, knowing the balances of each address in the wallet is actually of little use to an exchange; because trades happen off the blockchain, the balance of a user's deposit address rarely matches that user's balance on the exchange or even the sum of that user's deposits.

I am concerned about the size of a /wallet/transactions call, it could potentially exceed the amount that you could safely send over an API.
I guess we could add pagination, so that if the size exceeds 1mb or something, it'd be broken up into several pages, and you could specify a page with each call.

Most wallet APIs provide a way to limit the number of transactions returned. Bitcoin's allows you to specify such a number, and will return the latest n transactions. Others allow you to specify a starting block, sometimes an ending block as well. The latter is ideal -- specifying a starting block would return all transactions between that block and the latest or a specified ending block.

Working on it now, I think I can have everything you've request ready in the next release.
Timestamps on the transactions are vague. Do you want the time that the transaction was first seen, or the timestamp on the block it appeared in? Transactions in Sia don't have a timestamp on their own. First block appearance should be pretty easy, but first seen at all will be harder.

First block appearance is fine.

For the most part, the implementation is complete. I still need to update the actually http api, but the backend is ready.
All wallet keys and seeds are encrypted on disk. After loading, the wallet needs to be unlocked. The wallet won't start until it has been unlocked. The wallet is unlocked using a password/key. The wallet can be relocked by calling 'Close', which will wipe all of the keys from memory (to the extent possible. Because of context switching, swap space, and other confounding variables, it's possible that keys will be in memory unreachable to siad) You can call unlock and close as many times as you like.
When requesting transaction history, you can either ask for all transactions, or you can ask for transactions between two block heights [start, end]. Instead of actual transactions being returned, a 'Wallet Transaction' will be returned, which contains a single siacoin or siafund input or output. It also contains the full transaction (which means there's high redundancy for large transactions - this could perhaps be replaced by using transaction ids).
As an example, transaction A might have 3 inputs, 1 output, and 1 refund. 5 wallet transactions will be created. One for each input, and one for each output. Each will specify the type of movement (input vs. output, and siacoin vs. siafund), the address involved (generally only useful for incoming coins (appearing as outputs in a transaction), as the wallet arbitrarily selects addresses when creating inputs to a transaction), and the value. Each wallet transaction contains a timestamp and a confirmation height (to get the number of confirmations, you'll need to compare to the current height).
All 5 are guaranteed to appear consecutively in the transaction history.
When requesting a balance, you can request 'ConfirmedBalance' and 'UnconfirmedBalance'. Confirmed balance currently will only look at 1 confirmation, other than parsing the transaction history yourself you can't get 6 confirmation balance right now. Unconfirmed balance will return the number of incoming and outgoing siacoins, which will include refunds.
For example, if you create an unconfirmed transaction that sends 5 coins, and a 500 coin input is used, the result reported will be 500 outgoing ,and 495 incoming.
Hope this is all okay. Let me know if there are changes you think should be made.

Queries:

/wallet                   [GET]
/wallet/close             [PUT]
/wallet/history           [GET]
/wallet/history/$(addr)   [GET]
/wallet/seed              [GET]
/wallet/seed              [PUT]
/wallet/seed              [POST]
/wallet/siacoins          [GET]
/wallet/siacoins          [PUT]
/wallet/siafunds          [GET]
/wallet/siafunds          [PUT]
/wallet/transaction/$(id) [GET]
/wallet/unlock            [PUT]

/wallet [GET]
Function: Returns basic information about the wallet, such as whether the
wallet is locked or unlocked.
Parameters: none
Response:
struct {
	encrypted bool
	unlocked  bool
}

'encrypted' indicates whether the wallet has been encrypted or not. If the
wallet has not been encrypted, then no data has been generated at all, and the
first time the wallet is unlocked, the password given will be used as the
password for encrypting all of the data. Encrypted will only be set to false if
the wallet has never been unlocked before (the unlocked wallet is still
encryped - but the encryption key is in memory).
'unlocked' indicates whether the wallet is currently locked or unlocked. Some
calls become unavailable when the wallet is locked.
/wallet/close [PUT]
Function: Locks and closes the wallet, preparing for shutdown. After being
closed, the keys are encrypted. Queries for the seed, to send siafunds, and
related queries become unavailable. Queries concerning transaction history and
balance are still available.
Parameters: none
Response: Standard.
/wallet/history [GET]
Function: Return a list of transactions related to the wallet.
Parameters:
struct {
	start int
	end   int
}

Response:
struct {
	transactions []WalletTransaction
}

'transactions' is a list of 'WalletTransactions'. Wallet transactions are
transactions that have been processed by the wallet and given more information,
such as a confirmation height and a timestamp. Each wallet transaction contins
information about only a single input or output. One network transaction with
many inputs an outputs can result in many wallet transactions. All of the
wallet transactions created by a network transaction are guaranteed to be
consecutive in history. Wallet transactions will always be returned in
chronological order.
A wallet transaction takes the
following form:
struct WalletTransaction {
	TransactionID         string
	ConfirmationHeight    int
	ConfirmationTimestamp uint64

	FundType       string
	OutputID       string
	RelatedAddress string
	Value          int
}

'TransactionID' is the id of the transaction from which the wallet transaction
was derived. The full transaction can be obtaied by calling
'/wallet/transaction/$(id)'
'ConfirmationHeight' is the height at which the transaction was confirmed. The
height will be set to 'uint64MAX' if the transaction has not been confirmed.
'ConfirmationTimestamp' is the time at which a transaction was confirmed. The
timestamp is a 64bit unix timestamp, and will be set to uint64MAX if the
transaction is unconfirmed.
'FundType' indicates what type of fund is represented by the wallet
transaction. The options are 'Siacoin Input', 'Siacoin Output', 'Siafund
Input', and 'Siafund Output', corresponding to whether the fund is related to
siacoins or siafunds, and to whether the fund is an input to the transaction or
an output of the transaction. When looking at transactions, it should be noted
that a 'Siacoin Input' actually represents outgoing siacoins, as they are an
input to the transaction; they are being spent. A 'Siacoin Output' represents
incoming coins, because the output is being created by the transaction and made
available to the wallet.
'OutputID' is the id of the output. OutputIDs will always be unique.
'RelatedAddress' is the address that is affected. For inputs (outgoing money),
the related address is usually not important because the wallet arbitrarily
selects which addresses will fund a transaction. For outputs (incoming money),
the related address field can be used to determine who has sent money to the
wallet.
'Value' indicates how much money has been moved in the input or output.
/wallet/history/$(addr) [GET]
Function: Return all of the transaction related to a specific address.
Parameters: none
Response:
struct {
	transactions []WalletTransaction
}

'transactions' is a list of 'WalletTransactions' that affect the input address.
See the documentation for '/wallet/history' for more information.
/wallet/seed [GET]
Function: Return the seed that is being used to generate addresses. This seed
can be used to derive secret keys, and must be kept safe. This call is
unavailable when the wallet is locked or closed.
Parameters:
struct {
	dictionary string
}

'dictionary' is the name of the dicitionary that should be used when encoding
the seed.
Response:
struct {
	primarySeed        string
	addressesRemaining int
	backupSeeds        []string
}

'primarySeed' is the seed that is actively being used to generate new addresses
for the wallet.
'addressesRemaining' is the number of addresses that remain in the primary seed
until exhaustion has been reached and no more addresses will be generated.
'backupSeeds' is a list of seeds that are no longer used to generate new
addresses, but are still tracked can have spendable outputs.
A seed is an encoded version of a 128 bit random seed. The output is 15 words
chosen from a small dictionary as indicated by the input. The most common
choice for the dictionary is going to be 'english'. The underlying seed is the
same no matter what dictionary is used for the encoding. The encoding also
contains a small checksum of the seed, to help catch simple mistakes when
copying. The library
entropy-mnemonics is used
when encoding.
/wallet/seed [PUT]
Function: Get a new address from the wallet generated by the primary seed. An
error will be returned if the seed has been exhausted (there is a limit on the
number of addresses that can be generated from one seed), or if the wallet is
locked/closed.
Parameters: none
Response:
struct {
	address string
}

'address' is the address that cna
/wallet/seed [POST]
Function: Fetch a new seed for the wallet. The old seed will be added to the
list of backup seeds. In general, this call should be avoided unless address
exhaustion is reached. Because making a new seed is considered a significant
action, there is a verification field that must be set to true. This is to
prevent mistakes e.g. calling POST instead of GET or PUT.
Parameters:
struct {
	verification bool
	dictionary   string
}

'verification' must be set to true, or an error is returned. This is to prevent
an implementation mistake where POST is called instead of GET or PUT.
'dictionary' is the name of the dictionary that should be used when encoding
the newly created seed. 'english' is the most common choice.
Response:
struct {
	newSeed string
}

'newSeed' is the new seed that will be used as the seed for generating new
addresses.
/wallet/siacoins [GET]
Function: get the siacoin balance (confirmed and unconfirmed) of the wallet.
Parameters: none
Response:
struct {
	confirmedSiacoinBalance     int
	unconfirmedOutgoingSiacoins int
	unconfirmedIncomingSiacoins int
}

'confirmedSiacoinBalance' is the number of siacoins available to the wallet as
of the most recent block in the blockchain.
'unconfirmedOutgoingSiacoins' is the number of siacoins that are leaving the
wallet according to the set of unconfirmed transactions. Often this number
appears inflated, because outputs are frequently larger than the number of
coins being sent, and there is a refund. These coins are counted as outgoing,
and the refund is counted as incoming. The difference in balance can be
claculated using 'unconfirmedIncomingSiacoins' - 'unconfirmedOutgoingSiacoins'
'unconfirmedIncomingSiacoins' is the number of siacoins are entering the wallet
according to the set of unconfirmed transactions. This number is often inflated
by outgoing siacoins, because outputs are frequently larger than the amount
being sent. The refund will be included in the unconfirmed incoming siacoins
balance.
/wallet/siacoins [PUT]
Function: Send siacoins to an address. The outputs are arbitrarily selected
from addresses in the wallet.
Parameters:
struct {
	amount      int
	destination string
}

'amount' is the number of siacoins being sent.
'destination' is the address that is receiving the coins.
Response: standard
/wallet/siafunds [GET]
Function: get the siafund balance of the wallet. Only the confirmed balance is
displayed for siafunds.
Parameters: none
Response:
struct {
	siafundBalance      int
	siacoinClaimBalance int
}

'siafundBalance' is the number of siafunds available to the wallet as
of the most recent block in the blockchain.
'siacoinClaimBalance' is the number of siacoins that can be claimed from the
siafunds as of the most recent block. Because the claim balance increases every
time a file contract is created, it is possible that the balance will increase
before any claim transaction is confirmed.
/wallet/siafunds [PUT]
Function: Send siafunds to an address. The outputs are arbitrarily selected
from addresses in the wallet. Any siacoins available in the siafunds being sent
(as well as the siacoins available in any siafunds that end up in a refund
address) will become available to the wallet as siacoins after 144
confirmations. To access all of the siacoins in the siacoin claim balance, send
all of the siafunds to an address in your control (this will give you all the
siacoins, while still letting you control the siafunds).
Parameters:
struct {
	amount      int
	destination string
}

'amount' is the number of siafunds being sent.
'destination' is the address that is receiving the funds.
Response: standard
/wallet/transaction/$(id) [GET]
Function: Get the transaction associated with a specific transaction id.
Parameters: none
Response:
struct {
	transaction Transaction
}

'transaction' is a 'types.Transaction'. The full transaction can be seen in
types.transaction.go. All hashes in the transaction are encoded as strings.
/wallet/unlock [PUT]
Function: Unlock the wallet. The wallet is capable of knowing whether the
correct password was provided. The first time that the wallet is unlocked ever,
the password used will become the permanent password. Any string can be used as
the key, though it is generally recommended that the encoded primary seed be
used.
Parameters:
struct {
	key string
}

Response: standard

@poloniex @agundy @Mingling94 @lukechampine
the above comment contains the rough draft of the specification for the wallet api. If there is anything that is unusual, unusable, or you think could be improved, please let me know.
@poloniex, let me know if this adds all of the features that you will need to add Sia to an exchange.
The majority of this has been implemented already.

The wallet transaction thing is a little cumbersome for exchange purposes. The API described is sufficient for an exchange implementation, but I will have to construct the list of transactions I described myself -- either by adding up all the wallet transactions, or fetching transactions individually by id (probably the former).

So if I'm understanding it correctly, you're intending seeds to serve a dual purpose of generating addresses and (optionally) being the passwords to wallets? Won't most people just want to use their own passwords schemes?

Also your description is incomplete for

/wallet/seed [PUT]

'address' is the address that cna

Lastly,

I think the API could be less repetitious and more logically consistent by combining /wallet/siacoins [PUT] and /wallet/siafunds [PUT] into one
/wallet/transaction [POST]
struct {
    amount      int
    destination string
    currency    string
}

while also combining /wallet/siacoins [GET] and /wallet/siafunds [GET] into
/wallet/balance [GET]
struct {
    confirmedSiacoinBalance     int
    unconfirmedOutgoingSiacoins int
    unconfirmedIncomingSiacoins int
    siafundBalance              int
    siacoinClaimBalance         int
}

People should not make their own passwords. People make grossly insecure passwords, and dictionary attacks are extremely powerful. While they could make their own password when locking the wallet, the vast majority of people make passwords (even at 24+ characters!) that can be guessed in a relatively short amount of time using a dictionary attack. That's why we recommend you use the seed password - it's got a full 128 bits of entropy and is immune to dictionary attacks.
I could put all of the balance stuff into the /wallet [GET] call.
I'm not sure that it makes sense to combine the /siacoins and /siafunds calls when sending money.

That's the advisable method, my concern is more of a use-case thing. Will we even give the option of a custom password or will we require (at least on the UI side) that passwords be this 15 word seed? If we give the custom password option, people will likely take it. Thus the UI will probably auto-use the primary seed as a password.

Balance stuff going into /wallet [GET] is fine. Nothing initially seems problematic about it.

Hmm, it's a different string whether in the call parameters or in the api url itself. I just feel it makes more sense semantically and RESTfully to be 'POST'ing a transaction than 'PUT'ing an update to the siacoin or siafund balances.

The new api functions have been implemented and merged into master. Documentation on the calls can be found in doc/API.md.
Further discussion/requests can be continued by opening a new issue.


Add a link to submitted users PRs

Or we just remove the issue part since we ask them to make a PR anyway
@thejameskyle

Yeah do you want to update this to just be "open a pr"

Updated - should I keep the or?

Oh no I meant to get rid of the "open an issue" part

Ahh sorry that makes sense too - Il update

Ok - https://github.com/babel/babel.github.io/commit/ea69f3d9181ad68768f0764f7f23d8353a6814ec


feat(types): add option to extend RouteRecordName

This change allows for editing the RouteRecordName in the a module augmentation.
TIP:
I have a route names map which I use to refer to names in router function.
export const routeNames = {
    public: {
        index: 'index',
        features: 'features',
        pricing: 'pricing',
        s: {
            index: 's-thing',
            edit: 's-thing-edit'
        }
    },
    app: {
        index: 'app',
    },
    fallback: 'all'
} as const;

Considering the above map.
In my app I created a little helper type:
type RouteNames<T extends Record<string, any>> = {
    [K in keyof T]: T[K] extends Record<string, any> ? RouteNames<T[K]> : T[K];
}[keyof T];

Which I could use in the module augmentation as if this PR gets merged
// eslint-disable-next-line @typescript-eslint/consistent-indexed-object-style
export interface RouteRecords extends Record<RouteNames<typeof routeNames>, never> {}

Thanks for the proposal! The direction we want to take is to take things a little bit further with params types as well 🙂 Take a look at https://github.com/posva/unplugin-vue-router and feel free to contribute there

take things a little bit further with params types as well

That's great to hear!
Will this be part of the this router eventually?
Or the users will have to install that linked package? (Is that package is an experimental now and will be merged into this?)

Yes, the idea is to launch something official

But wouldn't this solution benefit people

right now without having to wait for the experimental plugin development
that do not wish to use the plugin

Could you reconsider reopening this @posva as it would allow for greater flexibility, e.g. currently you cannot have a simple .startsWith() or .includes() check because TS complains of those methods not existing on Symbol, even if your project never uses Symbols for route names and only sticks to strings.


[Question] Failed to fetch onnx model when to use AutoModel.from_pretrained

The code here:
import { AutoModel, AutoTokenizer } from '@xenova/transformers';

const modelPath = 'Xenova/distilgpt2'
let tokenizer = await AutoTokenizer.from_pretrained(modelPath); // **successful to fetch model**

let model = await AutoModel.from_pretrained(modelPath);  // **failed to fetch model**
let inputs = await tokenizer('I love transformers!');
let { logits } = await model(inputs);

Error information:
file:///Users/xxx/Documents/github/transformers.js/examples/node/esm/node_modules/@xenova/transformers/src/utils/hub.js:223
throw Error(Could not locate file: "${remoteURL}".)
^
Error: Could not locate file: "https://huggingface.co/Xenova/distilgpt2/resolve/main/onnx/model_quantized.onnx".
at handleError (file:///Users/xxx/Documents/github/transformers.js/examples/node/esm/node_modules/@xenova/transformers/src/utils/hub.js:223:19)
at getModelFile (file:///Users/xxx/Documents/github/transformers.js/examples/node/esm/node_modules/@xenova/transformers/src/utils/hub.js:412:24)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async constructSession (file:///Users/xxx/Documents/github/transformers.js/examples/node/esm/node_modules/@xenova/transformers/src/models.js:88:18)
transformers.js version: 2.1.1

Hi there. The reason for this is because we have not exported the model without its LM head. Is feature extraction your intended use-case?

Hi there. The reason for this is because we have not exported the model without its LM head. Is feature extraction your intended use-case?
You are most likely looking to use AutoModelForCausalLM, right?

Thanks xenova. I want to compute perplexity with transformers.js. The origin python code is here:
https://huggingface.co/docs/transformers/perplexity#example-calculating-perplexity-with-gpt2-in-transformers.
I tried AutoModelForCausalLM and other model like gpt2, but it also got error.

Hi there. The reason for this is because we have not exported the model without its LM head. Is feature extraction your intended use-case?
You are most likely looking to use AutoModelForCausalLM, right?

Thanks xenova. I want to compute perplexity with transformers.js. The origin python code is here:
https://huggingface.co/docs/transformers/perplexity#example-calculating-perplexity-with-gpt2-in-transformers.
I tried AutoModelForCausalLM and other model like gpt2, but it also got error.

Thanks xenova. I want to compute perplexity with transformers.js. The origin python code is here:
https://huggingface.co/docs/transformers/perplexity#example-calculating-perplexity-with-gpt2-in-transformers.
I tried AutoModelForCausalLM and other model like gpt2, but it also got error.

Thank you @xenova .  I'm looking forward to it 👍 👍.

Will hopefully get this done this week! 👍 In the meantime, can you provide the equivalent python code (and its expected output) for doing this with the original transformers library, that would seriously help me out!

Sure, all the code is from https://huggingface.co/docs/transformers/perplexity#example-calculating-perplexity-with-gpt2-in-transformers, and pasted below:
  from transformers import GPT2LMHeadModel, GPT2TokenizerFast
  import torch
  from tqdm import tqdm

  device = "cpu"
  model_id = "gpt2"
  model = GPT2LMHeadModel.from_pretrained(model_id).to(device)
  tokenizer = GPT2TokenizerFast.from_pretrained(model_id)

  test = 'I love transformer'
  encodings = tokenizer(test, return_tensors="pt")
  max_length = model.config.n_positions
  stride = 512
  seq_len = encodings.input_ids.size(1)

  nlls = []
  prev_end_loc = 0
  for begin_loc in tqdm(range(0, seq_len, stride)):
      end_loc = min(begin_loc + max_length, seq_len)
      trg_len = end_loc - prev_end_loc  # may be different from stride on last loop
      input_ids = encodings.input_ids[:, begin_loc:end_loc].to(device)
      target_ids = input_ids.clone()
      target_ids[:, :-trg_len] = -100

      with torch.no_grad():
        outputs = model(input_ids, labels=target_ids)

        # loss is calculated using CrossEntropyLoss which averages over valid labels
        # N.B. the model only calculates loss over trg_len - 1 labels, because it internally shifts the labels
        # to the left by 1.
        neg_log_likelihood = outputs.loss

      nlls.append(neg_log_likelihood)

      prev_end_loc = end_loc
      if end_loc == seq_len:
          break

  ppl = torch.exp(torch.stack(nlls).mean())
  print('ppl result:', ppl)

Output:
ppl result: tensor(83531.9844)

Busy with this here. Should be ready soon :+1:

Okay I think I've got what you needed. These changes are currently present on this branch, and I would appreciate it if you tested it out before I merged into main.
Here's the example python and javascript code I used for testing:
Python
Adapted from https://huggingface.co/docs/transformers/perplexity#example-calculating-perplexity-with-gpt2-in-transformers
from transformers import GPT2LMHeadModel, GPT2TokenizerFast
import torch
from tqdm import tqdm

device = "cpu"
model_id = "gpt2"
model = GPT2LMHeadModel.from_pretrained(model_id).to(device)
tokenizer = GPT2TokenizerFast.from_pretrained(model_id)

test = 'Once upon a time, in a land shrouded by mist and mystery, there existed a small village nestled within a dense forest.'
encodings = tokenizer(test, return_tensors="pt")
max_length = 8 # model.config.n_positions
stride = 2 # 512
seq_len = encodings.input_ids.size(1)

nlls = []
prev_end_loc = 0
for begin_loc in range(0, seq_len, stride):
    end_loc = min(begin_loc + max_length, seq_len)
    trg_len = end_loc - prev_end_loc  # may be different from stride on last loop
    input_ids = encodings.input_ids[:, begin_loc:end_loc].to(device)
    target_ids = input_ids.clone()
    target_ids[:, :-trg_len] = -100

    with torch.no_grad():
      outputs = model(input_ids, labels=target_ids)

      # loss is calculated using CrossEntropyLoss which averages over valid labels
      # N.B. the model only calculates loss over trg_len - 1 labels, because it internally shifts the labels
      # to the left by 1.
      neg_log_likelihood = outputs.loss

    nlls.append(neg_log_likelihood)

    prev_end_loc = end_loc
    if end_loc == seq_len:
        break

ppl = torch.exp(torch.stack(nlls).mean())
print('ppl result:', ppl)

output:
ppl result: tensor(42.8545)

JavaScript
Above tutorial translated into JS
import { AutoModel, AutoTokenizer, log_softmax } from '../src/transformers.js';

// This code is normally handled inside the forward method of the model, but since
// we can't pass target_ids, we must calculate this separately using the logits.
const loss = (logits, labels, ignore_index = -100) => {
    let [batchSize, seqLength, vocabSize] = logits.dims;

    // Shift so that tokens < n predict n
    let shiftLogits = logits.slice(null, [0, seqLength - 1], null);
    let shiftLabels = labels.slice(null, [1, seqLength], null);

    // Flatten
    shiftLogits = shiftLogits.view(-1, shiftLogits.dims.at(-1));
    shiftLabels = shiftLabels.view(-1);

    // Actually calculate cross entropy loss
    let sum = 0;
    let count = 0;
    for (let i = 0; i < shiftLogits.dims[0]; ++i) {
        let label = shiftLabels[i].data;
        if (label == ignore_index) { // == to match int with BigInt
            continue;
        }

        let logLogits = log_softmax(shiftLogits[i].data);
        sum += logLogits[label];
        ++count;
    }
    return - sum / count;
};

// Choose model to use
let model_id = "Xenova/gpt2"

// Load model and tokenizer
let model = await AutoModel.from_pretrained(model_id, { quantized: false });
let tokenizer = await AutoTokenizer.from_pretrained(model_id);

// Tokenize input text
let text = 'Once upon a time, in a land shrouded by mist and mystery, there existed a small village nestled within a dense forest.';
let encodings = await tokenizer(text);

// Set parameters. NOTE: these values are just for testing, and should be replaced with the commented out values.
let max_length = 8; // model.config.n_positions;
let stride = 2; // Math.floor(max_length / 2);
let seq_len = encodings.input_ids.dims[1];

let nlls = [];
let prev_end_loc = 0;
for (let begin_loc = 0; begin_loc < seq_len; begin_loc += stride) {
    // Perform same indexing as the tutorial
    let end_loc = Math.min(begin_loc + max_length, seq_len);
    let trg_len = end_loc - prev_end_loc;

    let input_ids = encodings.input_ids.slice(null, [begin_loc, end_loc]);
    let target_ids = input_ids.clone();
    for (let tg of target_ids) { // target_ids[:, :-trg_len] = -100
        tg.data.fill(-100n, 0, -trg_len);
    }

    // Run model
    let output = await model({ input_ids });

    // Compute loss
    let neg_log_likelihood = loss(output.logits, target_ids);
    nlls.push(neg_log_likelihood);

    prev_end_loc = end_loc;
    if (end_loc === seq_len) {
        break;
    }
}

// Compute perplexity
const mean = nlls.reduce((acc, val) => acc + val, 0) / nlls.length;
const ppl = Math.exp(mean);
console.log(`ppl result: ${ppl}`);

output:
ppl result: 42.855085073203895

The JS code is a little longer, but that's mainly because I had to reimplement the missing torch functions (e.g., cross entropy loss calculation). Let me know if there's something missing, otherwise I'll merge into main. 👍

I tested it and it works now. Thank you so much for your professional and kind help. 👍👍


CI tests run time with IoT docker images

Following discussion at https://github.com/gusbrs/zref-clever/pull/15#issuecomment-1803661808
Since zref-clever moved to using the IoT TeX Live docker image for the CI tests GitHub workflow (https://github.com/gusbrs/zref-clever/commit/e1d2acfbc6c1b1321446cf3c5d813a27c889a6d2) my impression is that the workflow run was taking considerably more time to run than before. While I recalled workflow runs to take about 10-14 min, which is already considerable, with the use of the docker images, they were easily breaching the 20+ min mark.
It is true that workflows' run times tend to vary a lot, but at @muzimuzhi 's suggestion, I triggered workflows with IoT docker images and without it (using zauguin/install-texlive, as I used to before). Also true, this is just a one point sample for comparison, but better than nothing and better than my "impression".
Run with IoT docker image (https://github.com/gusbrs/zref-clever/actions/runs/6811932906):
Initialize containers + Update TeX Live: 1m27s + 30s
Run tests: 13m 52s
Total time: 15m56s
Run with zauguin/install-texlive (https://github.com/gusbrs/zref-clever/actions/runs/6812269840):
Install TeX Live: 4m24s
Run tests: 8m48s
Total time: 13m21s
So the overall difference seems to be smaller than granted by my previous impression. However, there really seems to be a difference in performance. I took more than twice as much to install TeX Live from scratch than to initialize the docker containers and update TeX Live. Despite that, the zauguin/install-texlive run was still faster overall, because the core "Run tests" task, which is exactly the same in both cases, and which is the most time consuming runs much faster in this case.
True, just 2.5min difference is hardly enough to revert to the previous approach, and to loose the convenience granted by the IoT docker image. But, as mentioned, this is just one point sample. I'll be keeping an eye on these run times, and if they remain being considerable, I may reconsider the use of the docker image.
In the meantime, if anyone has any ideas of why this performance difference occurs, and if there are ways to improve things, it'd be much appreciated.
Edit: These two workflow runs, from when the use of the IoT docker image was introduced, are from the same day and also comparable:
With IoT docker image (https://github.com/gusbrs/zref-clever/actions/runs/4212257093): 14m39s
With zauguin/install-texlive (https://github.com/gusbrs/zref-clever/actions/runs/4212132743): 8m27s
(But the logs are gone and we can no longer compare the parts).

Run with zauguin/install-texlive (https://github.com/gusbrs/zref-clever/actions/runs/6812269840):
Install TeX Live: 4m24s
Run tests: 8m48s
Total time: 13m21s

zauguin/install-texlive has built-in cache support, hence in following runs the "Install TeX Live" step should take much shorter. See for example workflow run #6811059931 in latex3/latex2e.
A separate workflow can be created to update TeX Live cache regularly.
Perhaps off-topic, zref-clever could also split its 79+1 tests into multiple jobs run in parallel. Currently there're only non-ideal l3build options --first and --last.

@muzimuzhi Thank you for your thoughts!

zauguin/install-texlive has built-in cache support, hence in following runs the "Install TeX Live" step should take much shorter. See for example workflow run #6811059931 in latex3/latex2e.
A separate workflow can be created to update TeX Live cache regularly.

These would presumably reduce further the run time for the zauguin/install-texlive alternative, and thus increase the difference from the IoT docker image, right?
Btw, I did previously use (some) caching for that, and it was dropped in that PR of yours. Do you remember the reason why you dropped it?

Perhaps off-topic, zref-clever could also split its 79+1 tests into multiple jobs run in parallel. Currently there're only non-ideal l3build options --first and --last.

I also don't see how l3build could handle that. Furthermore, I actually do have some qualms about splitting the checks from the docs, as I currently do, because (I think) I end up doing a costly operation twice (downloading image/initializing container/installing TL, etc). It spares me some time, but I'm unsure if this is an undue burden on someone else's server (I'm also not sure whose server that is). Do you happen to know if the current state of things implies the download and container initialization is being done in duplicity (checks / doc)? And on whose server this is done? (In particular, is the IoT unnecessarily burdened by it?)

BTW, action teatimeguest/setup-texlive-action supports installing historic TeX Live versions on all three OS. (Will tests remain passed in TeX Live 2022?)

Here I'm not convinced this is something to worry about. At least given how packages are distributed in the ecosystem. Any regular user who has access to a current version of zref-clever has it from their TeX distribution (TeX Live, MikTeX, etc.), so they also have access to a current TeX Live version in the same place and by the same means.
I had a similar discussion some time ago with daleif, related to how to decide on the required kernel version when we want to use some new feature.
All in all, I fail to see the need to support and test for an "old TeX distribution with a current zref-clever" because this would only ever happen to someone who installs zref-clever manually. And such kind of users are the exception, and presumably know what they are doing. Other than that the versions of the TeX distribution and that of the package will naturally match, because that's how the community handles the distribution of things.

Mhm, I tested again zauguin/install-texlive, now re-enabling caching (thanks for that latex3/latex2e workflow link, btw) and got the run time down to 6m25s (https://github.com/gusbrs/zref-clever/actions/runs/6816911621). That's quite the difference now... I'd say this is a keeper, unless someone has some better insight on why it takes longer with the docker image and how to improve it. (I'll keep the issue open for some time to see if anyone chimes in).

Btw, I did previously use (some) caching for that, and it was dropped in that PR of yours. Do you remember the reason why you dropped it?

Starting with v2, zauguin/install-texlive auto restores and caches the texlive installation, see https://github.com/zauguin/install-texlive/commit/0c87efffa145ce68b6c7baf65f639dab0c4bd3ba. That's why I dropped the caching steps in https://github.com/gusbrs/zref-clever/commit/4173c238a797e03f471764b29324fc53055364af.
But GitHub only retains caches for 7 days, hence to make sure there're always a hit cache to restore, you may need another workflow running on a regular basis, triggered by on.schedule.
As you already applied, latex3/latex2e uses a different strategy since its large test suite is split to 33 jobs. Note the root, first job "Update TeX Live" (or "Install TeX Live" in https://github.com/gusbrs/zref-clever/actions/runs/6816911621) may still take the time to do a clean installation if no cache is hit. Therefore the 6m25s is the best case, not the average case. You can emulate this situation by manually delete the caches listed in https://github.com/gusbrs/zref-clever/actions/caches.

Do you happen to know if the current state of things implies the download and container initialization is being done in duplicity (checks / doc)? And on whose server this is done? (In particular, is the IoT unnecessarily burdened by it?)

My guess is yes, since "Regression tests" and "Documentation" jobs are run in parallel.
Seems it's on GitLab's server, see https://gitlab.com/islandoftex/images/texlive/container_registry. The GitLab Container Registry is free for everyone (all tiers, all offerings).

BTW, action teatimeguest/setup-texlive-action supports installing historic TeX Live versions on all three OS. (Will tests remain passed in TeX Live 2022?)

Here I'm not convinced this is something to worry about.

I meant, teatimeguest/setup-texlive-action makes it easier to test if regression test running in texlive 2023 is than texlive 2022. But with your latest data, it seems unnecessary.

Starting with v2, zauguin/install-texlive auto restores and caches the texlive installation, see zauguin/install-texlive@0c87eff. That's why I dropped the caching steps in 4173c23.
[...]
As you already applied, latex3/latex2e uses a different strategy since its large test suite is split to 33 jobs.

That's what I get for approaching the problem in trial-and-error fashion, instead of actually studying how things work. ;-)
Let me save face and blame the "limited resources"...
Thanks for pointing that out. I might revert that commit then. Though handling the cache explicitly like that does have one benefit, which is the costly installation step is done only once for both checks and doc. Anyway, I'll test further.

But GitHub only retains caches for 7 days, hence to make sure there're always a hit cache to restore, you may need another workflow running on a regular basis, triggered by on.schedule.
[...] Note the root, first job "Update TeX Live" (or "Install TeX Live" in https://github.com/gusbrs/zref-clever/actions/runs/6816911621) may still take the time to do a clean installation if no cache is hit. Therefore 6m25s is the best case, not the average case. You can emulate the worst case by manually deleting the caches listed in https://github.com/gusbrs/zref-clever/actions/caches and then triggering a rerun.

Yes, I understand, of course, that that time depended on the cache being already there. But the "worst case" is still faster. And if I'm doing a batch of work in the repo I get the benefit of the cache on top of that. This seems like good enough, I'm not sure running a scheduled job just to keep the cache alive is granted for this (for my needs, that is).
I do think the use of the docker image is conceptually superior, it is arguably a more thorough and proper solution. The main practical benefit is not having to manually curate a list of packages to install, which is a pain, since the way to do it is through reiterated (failed) attempts to run the workflow on GitHub. But, once that's settled, it works too. And the current long CI run times have been bugging me.
Actually, another thing I'm considering is dropping the GitHub workflow altogether. When I added it, I did so because it seemed cool, and I wanted to learn about it. But, in hindsight, I've been getting little benefit from it. I tend to run the tests locally anyway. For releases, I must do so to prepare the package with l3build. The only difference here is that I get TeX Live versions of zref-vario and zref-check on GitHub and get my development versions locally. But that's really a further advantage of running locally, since occasionally there's a coordinated change between two of them, and the test fails on GitHub because of that. The other benefit of the workflow is to support a (very rare) PR. All in all, I'm not so sure if it's worth it.

My guess is yes, since "Regression tests" and "Documentation" jobs are run in parallel.
Seems it's on GitLab's server, see https://gitlab.com/islandoftex/images/texlive/container_registry. The GitLab Container Registry is free for everyone (all tiers, all offerings).

That's about what I suspected. Thanks for the explaination.
GitLab's servers than, makes sense. Better than burdening IoT's ones, but still.

I meant, teatimeguest/setup-texlive-action makes it easier to test if regression test running in texlive 2023 is than texlive 2022. But with your latest data, it seems unnecessary.

Then I'm afraid I had missed your point in this, and still do. Care to elaborate this a little further?

Perhaps off-topic, zref-clever could also split its 79+1 tests into multiple jobs run in parallel. Currently there're only non-ideal l3build options --first and --last.

I also don't see how l3build could handle that.

Sorry I missed this one in my last comment.
Currently all 79+1 tests were run in a single job named "Regression tests", but it don't have to. For the 79 tests in ./testfiles, one can use l3build check -c build to test them all, or can use
l3build check --last <name-of-nth-test>
l3build check --first <name-of-nth-test>

to check the tests in two steps, at the cost of running the nth test twice. Then the workflow can be set to start two parallel jobs for regression tests.
jobs:
  check:
    strategy:
      matrix:
        include:
          - id: 1
            command: |
              l3build check -c build --last zc-class-scrreprt01
          - id: 2
            command: |
              l3build check -c build -q --first zc-class-scrreprt01
              l3build check -c build-4runs -q

    name: Regression tests (${{ matrix.id }})
    runs-on: ubuntu-latest

    steps:
      - name: Install TeX Live
        uses: zauguin/install-texlive@v3
        with:
          packages: ${{ env.ZC_PACKAGE_LIST }}

      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Run tests
        run: ${{ matrix.command }}

      - name: Archive failed test output
        # ...

This provides another way to speed up l3build check. See also https://github.com/latex3/latex2e/issues/1073 and https://github.com/latex3/l3build/issues/314.

Actually, another thing I'm considering is dropping the GitHub workflow altogether. [...] The other benefit of the workflow is to support a (very rare) PR. All in all, I'm not so sure if it's worth it.

Another benefit: by setting a scheduled workflow run, it's easier to catch compatibility problems and uncover the need to update test files (.tlg) in time. It's most helpful when the base package is not in active development anymore, or its implementation and/or test suite is sensitive to changes in dependencies and/or packages used together.
For example CTeX-org/ctex-kit is relatively stable and its test suite compares output of \showbox and/or \showoutput a lot, hence sensitive to kernel changes. Thus I set its CI to run weekly and since then it did help to catch failed checks in time and make each adjustments small and clear.

I meant, teatimeguest/setup-texlive-action makes it easier to test if regression test running in texlive 2023 is than texlive 2022. But with your latest data, it seems unnecessary.

Then I'm afraid I had missed your point in this, and still do. Care to elaborate this a little further?

It's already outdated/useless.
In https://github.com/gusbrs/zref-clever/pull/15#issuecomment-1803729498 I speculated that

The workflow runs in current repo became slower since ~7 months ago, which seems to be the time when TeX Live 2023 released.

To check against it, one way is to set the workflow to install TeX Live 2022 then...

Sorry I missed this one in my last comment.
Currently all 79+1 tests were run in a single job named "Regression tests", but it don't have to. For the 79 tests in ./testfiles, one can use l3build check -c build to test them all, or can use
l3build check --last <name-of-nth-test>
l3build check --first <name-of-nth-test>

Interesting, thank you very much once again. This technique would also apply to the IoT docker images. Nice, but I think it is well beyond my "GitHub-actios-fu" so I'd be weary to add this complexity to the task. Still I'm happy to have this alternative as a reference here.
Also, I reduced further the run times by dropping testing on dev formats. Now we're at ~4min cached / ~8min no cache. Much better, and acceptable.
Btw, TIL that my regression tests were taking as much time as those of the LaTeX kernel...
Talk about self-taught rookies' insecurities and over-testing.  ;-)

Another benefit: by setting a scheduled workflow run, it's easier to catch compatibility problems and uncover the need to update test files (.tlg) in time. It's most helpful when the base package is not in active development anymore, or its implementation and/or test suite is sensitive to changes in dependencies and/or packages used together. (Of course this can be done locally too.)
For example CTeX-org/ctex-kit is relatively stable and its test suite compares output of \showbox and/or \showoutput a lot, hence sensitive to kernel changes. Thus I set its CI to run weekly and since then it did help to catch failed checks in time and make each adjustments small and clear.

That's a good point. I had been doing this kind of check to catch upstream changes locally. But, as long as do keep the CI workflow, you have me convinced on running things on schedule too.

In #15 (comment) I speculated that

The workflow runs in current repo became slower since ~7 months ago, which seems to be the time when TeX Live 2023 released.

To check against it, one way is to set the workflow to install TeX Live 2022 then...

Ah! I finally understood what you meant. I had totally lost this train of thought. Though I think this would be a little too much for this check.
All in all, and again, thank you very very much for your comments! Insightful and useful as usual.

I think this is a fair time to close this one. I'll be glad to reopen it, though, if any other thoughts or ideas come up.


Possibility for new Release-Tag

Hello,
a great library :-)
I wanted to ask for a new release tag, since I have issues with the timeout operator.
I'm using tag 4.0.0 but there is still this bug in:
https://github.com/Reactive-Extensions/RxCpp/commit/5aea424898994b3d409db268486202244cfe5053
Is there any chance that there will be a new tag in the foreseeable future?
Thx,
Juergen

Any news on this?

I missed my deadline, I will schedule time for this soon. Thanks for the prompt!

The new tag is 4.1.0


Benchmark and optimize code

(Moved from https://github.com/falconry/falcon/issues/1183)
Optimizations may include:

[ ] Cython support (simply compiling and/or *.pyx modules)
[ ] Algorithms
[ ] Micro-optimizations

See also: https://github.com/falconry/python-mimeparse/issues/40


Random failures of test.aria.core.environment.Customizations

This PR fixes random failures of test.aria.core.environment.Customizations, by waiting for the load of the first customization descriptor to be finished before loading another one.
To be sure that this PR fixes the issue, I have run several times in a loop a short campaign containing test.aria.core.environment.Customizations on travis-ci until the campaign fails.

Without this fix, there is a failing campaign in less than 5 min, after 41 iterations in this travis build
With this fix, there is no failing campaign after more than 30 min and 600 iterations in this travis build

@divdavem you were right, the initial implementation could not work properly for sure. Seems :ok: now.
You can merge

@benouat Thank you for your review


Any example for side bar navigation like netflix?

I'm trying to develop an OTT platform like netflix, I need the sidebar navigation

Sorry, this really isn't the place to ask these types of questions. This is a simple template repo for getting started writing Roku apps using brighterscript. Feel free to ask this type of question in the Roku developers slack

Sorry, this really isn't the place to ask these types of questions. This is a simple template repo for getting started writing Roku apps using brighterscript. Feel free to ask this type of question in the Roku developers slack

Ok, used the hero grid example but it miss the navigation


cherry-pick constant format changes

Looks like you picked one more commit [bump up snapshot version for 6.7 release] from develop?


water draws above the shittle

Description
yes
Reproduction

ftl to planet with rivers
get (un)lucky

Screenshots

happens for lava too

Dupe https://github.com/space-wizards/RobustToolbox/issues/2191


🐛 Bug Report: JSDocstrings for functions not appearing in VSCode Intellisense thingy when using Web SDK

👟 Reproduction steps

Open HTML file.
Add Web SDK CDN.
Initialize Appwrite Client using:

const client = new Appwrite.Client()

Set Projects, Endpoint:

client.setProject("myProject");
client.setEndpoint("https://cloud.appwrite.io/v1")

Hover over client.setProject("myProject"); or client.setEndpoint("https://cloud.appwrite.io/v1") and see no JSDocstring.

👍 Expected behavior
When I hover over the client.setProject("myProject"); or client.setEndpoint("https://cloud.appwrite.io/v1") part, it should show the JSDoc using this Docstring provided in your source code:
/**
     * Set Endpoint
     *
     * Your project endpoint
     *
     * @param {string} endpoint
     *
     * @returns {this}
*/

or
/**
     * Set Project
     *
     * Your project ID
     *
     * @param value string
     *
     * @return {this}
*/

👎 Actual Behavior
They actually don't. I am not sure if this is a problem by the IDE or not.
🎲 Appwrite version
Version 0.10.x
💻 Operating system
Windows
🧱 Your Environment
Vercel for hosting. VSCode 1.70.3 as IDE, Windows 7 for OS, HP Probook 4520s for Machine
👀 Have you spent some time to check if this issue has been raised before?

[X] I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

[X] I have read the Code of Conduct

@RyanBaig, thanks for raising this issue! 🙏🏼 Does this work with any other package from CDN? Do you have a code sample of it working with another CDN? I tried with lodash and didn't get any typing info:

From my experience, your IDE knows about packages when you have a package.json and install the package so it's in your node_modules. When you add a script tag for a package, the IDE has no idea about the types and whatnot about the package.

Closing as stale.


How to force Annotator poly_stream population

With the annotators, the poly_stream (hv.streams.PolyDraw) doesn't get populated until its rendered on the screen. I'm trying to write some tests for pytest that will be programmatic and non-UI. Is it possible to force poly_stream to populate without rendering?

@philippjfr will publish some pytests for annotators to use as an example

@philippjfr Did examples ever get pushed? I don't see any.

test are published

To be precise they now live in ./earthsim/tests/test_annotators.py


Feature request! Open Model Therad Group

It is awesome cool. And real open model!
https://jmeter.apache.org/usermanual/component_reference.html#Open_Model_Thread_Group

Super fresh request considering that JMeter 5.5 was released today! Thank you for proposing it!
Do you see any particular value on it compared to the current solution the DSL provides aside from the dependency to a plugin that it now provides? I see it as replacement the existing usage of throughput shaping timer + concurrency thread group. I think we may incorporate it into existing rpsThreadGroup without any alterations to end user API and just replacing the underlying implementation. Any thoughts?
Regards

I was really looking forward to this feature!
I've looked at the new implementation a lot.  I even ran some tests.  This implementation is slightly more demanding on resources.  But at the moment I can't say with certainty that it replaces the current implementation of RpsThreadGroup.  A safe option in my opinion is to expand the implementation and support two thread groups in one implementation.  Maybe with the .useOpenModel() method.  All the same, in their model, these thread groups are slightly different.  But it seems that this difference does not prevent us from starting to add a new thread group.  The key feature of the new thread group is the amazing accuracy and reproducibility of results.

I wonder if makes sense to keep both thread group implementations. When would you use one and the other?
Do you have any link/article that compares the accuracy/reproducibility of results compared to existing Throughput Shaping Timer + Concurrency Thread Group? Would be nice to have it to include it in user guide, release notes or any place we want to give users information why did we do a change or when to use each alternative.
Thank you again for bringing this into attention.

I tested it locally.  So the result has not been published.  There is an overview https://qainsights.com/open-model-thread-group-in-jmeter/ , but the problem is that I don't know the deep characteristics of this thread group.  How it works with connections, how it utilizes resources.  I know its functionality and features, they are similar to RPS thread group, but the implementation is completely different.  In my opinion, the implementation is important and you cannot replace one thread group with another only on the basis of functionality.


refactor settings into own class and save window state

It looks great :-) and +1 for own settings class... please push it into original repo too, as in previous pull request - we will see what happens next ;-)


Set avatar is broken or docs are incorrect

andrewhill@~/Textile/tag/gist(master)$ textile profile set avatar --help
usage: textile profile set avatar <value>

Sets the profile avatar of the peer

Flags:
      --help              Show context-sensitive help (also try --help-long and
                          --help-man).
      --api="http://<IP_ADDRESS>:40600"
                          API Address to use
      --api-version="v0"  API version to use
      --debug             Set the logging level to debug
  -n, --name=NAME         Set the peer's display name
  -a, --avatar=AVATAR     Set the peer's avatar from an image path (JPEG, PNG,
                          or GIF)

Args:
  <value>  The value (as an image path: JPEG, PNG, GIF) to set the profile
           avatar to

andrewhill@~/Textile/tag/gist(master)$ textile profile set avatar -a ../../../Desktop/avatar.png
textile: error: required argument 'value' not provided, try --help
andrewhill@~/Textile/tag/gist(master)$ textile profile set avatar --avatar=../../../Desktop/avatar.png
textile: error: required argument 'value' not provided, try --help

textile profile set avatar ../../../Desktop/avatar.png
File target: QmPn5kyMMzx2cdb9PyVwyrqgoj9ycoANxXzEbhecZASJiF
Added 1 file in 515.035517ms
ok

okay, got it. looks like the flag -a or --avatar is misleading here in the docs

Update could be added to #821 before it lands

You can set your whole profile:
textile profile set --name=... --avatar=...

Or each individually,
textile profile set name ...
textile profile set avatar ...

closing this for now, please reopen if you think of a way to make it less confusing


Failed to update table schema

I have Postgres source connector and Bigquery sink connector, also Schema Registry. I add new field into a table in Postgres and insert a row into the table. Postgres connector detects the changes, but Bigquery failed to update schema (autoUpdateSchamas is true):
[2020-07-03 02:35:58,704] ERROR WorkerSinkTask{id=bq-core-field-connector-1} Commit of offsets threw an unexpected exception for sequence number 96048: null (org.apache.kafka.connect.runtime.WorkerSinkTask:259)
com.wepay.kafka.connect.bigquery.exception.BigQueryConnectException: Some write threads encountered unrecoverable errors: com.wepay.kafka.connect.bigquery.exception.BigQueryConnectException; See logs for more detail
at com.wepay.kafka.connect.bigquery.write.batch.KCBQThreadPoolExecutor.awaitCurrentTasks(KCBQThreadPoolExecutor.java:97)
at com.wepay.kafka.connect.bigquery.BigQuerySinkTask.flush(BigQuerySinkTask.java:119)
at org.apache.kafka.connect.sink.SinkTask.preCommit(SinkTask.java:125)
at org.apache.kafka.connect.runtime.WorkerSinkTask.commitOffsets(WorkerSinkTask.java:379)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:208)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:192)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2020-07-03 02:35:58,713] INFO Putting 19 records in the sink. (com.wepay.kafka.connect.bigquery.BigQuerySinkTask:170)
[2020-07-03 02:35:58,874] INFO Putting 29 records in the sink. (com.wepay.kafka.connect.bigquery.BigQuerySinkTask:170)
[2020-07-03 02:35:59,229] INFO Attempting to update table GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=BQ_CORE_FIELD_ALL_SCHEMAS, tableId=core_yga_ed_core_data_device}} with schema Schema{fields=[Field{name=before, type=RECORD, mode=NULLABLE, description=null}, Field{name=after, type=RECORD, mode=NULLABLE, description=null}, Field{name=source, type=RECORD, mode=REQUIRED, description=null}, Field{name=op, type=STRING, mode=REQUIRED, description=null}, Field{name=ts_ms, type=INTEGER, mode=NULLABLE, description=null}, Field{name=transaction, type=RECORD, mode=NULLABLE, description=null}]} (com.wepay.kafka.connect.bigquery.SchemaManager:60)
[2020-07-03 02:35:59,915] ERROR Task failed with com.wepay.kafka.connect.bigquery.exception.BigQueryConnectException error: Failed to update table schema for: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=BQ_CORE_FIELD_ALL_SCHEMAS, tableId=core_yga_ed_core_data_device}} (com.wepay.kafka.connect.bigquery.write.batch.KCBQThreadPoolExecutor:68)

@mtthien-tma what's the name of the field you added to your Postgres table?

@C0urante I added 'priority' column to the table.

@mtthien-tma that doesn't appear to be a name of a column in the BigQuery schema in the log you provided. Are you sure that's the only column? The ones listed are before, after, source,op, ts_ms, and transaction.

I am sure that is the field. It should be nested in before and after RECORD. It seems that the log does not show all the fields in RECORD type.

Ah, gotcha. You mentioned you're using Schema Registry; can you provide the Avro schema for the data that's breaking on the connector?

Hi @C0urante Sorry for so late reply. You are right that adding non-optional field seems to be the root cause. I tried adding nullable field in Postgres and could see that BQ connector could create the column in BQ. And it failed when I added required field in Postgres.
I am using BQ connector 1.3. Is there any workaround? Do I have to manually update BQ schema?


Template updates are not propagated to existing jobs.

This can be accomplished by deleting feature-branch jobs and letting them get recreated from the new template, but you lose your build history.

This is actually intended. Even though master may evolve over time, a given branch may require that its job configs be consistent with when the branch was initially created and not update to latest master.
In addition, this would imply on some setups that hundreds or even thousands of jobs get deleted and then recreated every time a change is made to the master job, which in turn would cause a flood of jobs being executed. We've been down this road, and it's no fun.
Instead, what we did locally was setup a parameterized job that takes a regex for determining what jobs to delete, and then kicks off jenkins-build-per-branch to recreate them. This allows us to be much more selective.
Closing since we don't consider this a bug.


Reading directly from s3 paths fails until s3fs package is installed

Was easy fix :
pip install s3fs
(Worked - pip installed version 0.1.6)
Happy to PR the docker image

Closed via Add S3fs


spark: Unable to find submodules when adding spark integration to other projects via gradle

I am trying to add openlineage-spark to my project with a similar line in build.gradle:
dependencies {
    api 'io.openlineage:openlineage-spark:0.11.0'
}

But it gives this error when I run gradle shadowJar or gradle build:
* What went wrong:
Could not determine the dependencies of task ':shadowJar'.
> Could not resolve all dependencies for configuration ':runtimeClasspath'.
   > Could not find spark:app:0.11.0.
     Searched in the following locations:
       - https://repo.maven.apache.org/maven2/spark/app/0.11.0/app-0.11.0.pom
     If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
     Required by:
         project : > io.openlineage:openlineage-spark:0.11.0
   > Could not find spark:shared:0.11.0.
     Searched in the following locations:
       - https://repo.maven.apache.org/maven2/spark/shared/0.11.0/shared-0.11.0.pom
     If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
     Required by:
         project : > io.openlineage:openlineage-spark:0.11.0
   > Could not find spark:spark2:0.11.0.
     Searched in the following locations:
       - https://repo.maven.apache.org/maven2/spark/spark2/0.11.0/spark2-0.11.0.pom
     If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
     Required by:
         project : > io.openlineage:openlineage-spark:0.11.0
   > Could not find spark:spark3:0.11.0.
     Searched in the following locations:
       - https://repo.maven.apache.org/maven2/spark/spark3/0.11.0/spark3-0.11.0.pom
     If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.
     Required by:
         project : > io.openlineage:openlineage-spark:0.11.0

This used to work fine with version 0.9.0 and older.

Hi, thank you for the speedy solution.
I don't think #937 will solve the issue of dependencies not showing up with gradle dependencies command. Does it also come under this issue?

@varuntestaz the internal dependencies shouldn't be seen there - or do you mean something else?

@mobuchowski No, I mean the external dependencies like org.slf4j:slf4j-api:1.7.25 in version 0.8.2. This don't show up in the new version.

We explicitely do not want to expose them because the jar we produce is ment to be self-contained and used in Spark deployments. That's why we include and relocate all the dependencies.
If you want to do something different, I think better way would be to work with source code directly rather than jar.


Forward bridge networks not supported

Hi. I was trying to set up a simple bridge network (see below), but I ran into the following error:
[ERROR]   stderr) okt 01 22:29:54 flat nixvirt-start[12725]: NixVirt: libvirt error: unsupported configuration: Unsupported <mac> element in network bridged-network with forward mode='bridge'

Now this is weird because my definition does not contain any <mac> element:
<network>
  <name>bridged-network</name>
  <uuid>cc7439ed-36af-4696-a6f2-1d0c4474d87e</uuid>
  <forward mode="bridge" />
  <bridge name="br0" />
</network>

The nix side of things just imports the file as-is:
networks = [
  {
    definition = ./network.xml;
    active = true;
  }
];

I think it might be because of this snippet? It seems to expect a <mac> entry to exist in every network, and adds it if not found. This should probably not apply for bridge networks.
https://github.com/AshleyYakeley/NixVirt/blob/1d5c17c1f3a03f5d7e4dacc1025db350190f8f6c/tool/nixvirt.py#L175-L185

This is surprising. As you can see from the snippet, the code already checks the "mode" attribute and should not futz around with the MAC address if it's set to "bridge".


enable visualization of 3 canonical planes like knossos does

@brainmaps wants this.

Moved to new repo -- https://github.com/openconnectome/NeuroDataViz/issues/1
I'll move the other issues soon.


Add Twitch and Gitlab

Just a couple copy and paste jobs to add things I use. Thanks for the awesome theme!

Looks good, thanks for the contribution!


question of multi-cam with D415s

Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

Consider checking out SDK examples.
Have you looked in our documentations?
Is you question a frequently asked one?
Try searching our GitHub Issues (open and closed) for a similar issue.

All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

Required Info

Camera Model
{ {D400}

Firmware Version
(not the newest one)

Operating System & Version
{Win (10)}

Kernel Version (Linux Only)
(not using)

Platform
PC

SDK Version
{}

Language
{C }

Segment
{others }

Issue Description
<Describe your issue / question / feature request / etc..>
Hi dear sir
I have two D415 cams needed to be synced. I am wondering that belows:
in paper of using the intel realsence depths camera D4xx in multi-camera configurations,
the paper writes "rs2_Set_Option with argument “rs2_option_inter_cam_sync_mode” "
is it can be used in the latest firmware and SDK now?
any response will be appreciated

Hi sbnn123456,
Multi-camera sync of the D415 has been available since June 2018.

@MartyG-RealSense thanks a lots.
i was seen two way to sync multi-cam ,one is HW Sync.
you means it also can be used by HW sync now?
maybe same question asked again. thank you very much.

Yes, there is software sync and hardware sync.  HW sync is available now.

@MartyG-RealSense  thank you very much.
we really appreciate you help.


Explicitly exclude encrypted files from backups (EncryptedSharedPreferences, encrypted DBs)

Task/Issue URL: https://app.asana.com/0/0/1206667832334038/f
Description
Steps to test this PR
Feature 1

[ ]
[ ]

UI changes

Before
After

!(Upload before screenshot)
(Upload after screenshot)

#4204  👈
develop

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @CDRussell and the rest of your teammates on  Graphite


Please Add: Parent and Educator Ally

Compassionate assistant for parents and educators, focusing on health and emotional development parenting.
https://chat.openai.com/g/g-2HzecDNbj-parent-and-educator-ally

Please raise a pull request for the same


Add extra config options

Adds two new configuration options:

validation.failUnrecognisedHeaders which will fail when unrecognised headers have been detected in the samplesheet instead of emitting a warning
validation.summary.hideParams which takes a list of parameters to exclude from the summary created by the plugin

Fixes #45, #31 and #7


#1: add support for horses in database initialization

/paulo - can you please review

/paulo - is the pony ok?

/paulo - can you check now?


Input 0 is incompatible with layer model: expected shape=(None, 256, 256, 1), found shape=(None, 128, 128, 3)

i am trying to evaluate my model but when i run model.evaluate(test_image), it showed this error:
ValueError: in user code:
/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py:1323 test_function  *
    return step_function(self, iterator)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py:1314 step_function  **
    outputs = model.distribute_strategy.run(run_step, args=(data,))
/usr/local/lib/python3.7/dist-packages/tensorflow/python/distribute/distribute_lib.py:1285 run
    return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/distribute/distribute_lib.py:2833 call_for_each_replica
    return self._call_for_each_replica(fn, args, kwargs)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/distribute/distribute_lib.py:3608 _call_for_each_replica
    return fn(*args, **kwargs)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py:1307 run_step  **
    outputs = model.test_step(data)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py:1266 test_step
    y_pred = self(x, training=False)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/base_layer.py:1013 __call__
    input_spec.assert_input_compatibility(self.input_spec, inputs, self.name)
/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/input_spec.py:270 assert_input_compatibility
    ', found shape=' + display_shape(x.shape))

ValueError: Input 0 is incompatible with layer model: expected shape=(None, 256, 256, 1), found shape=(None, 128, 128, 3)

How do i resolve it?

@vjamishi ,
In order to expedite the trouble-shooting process, could you please provide the complete code and dataset to reproduce the issue reported here.


Add Cancel pipelinerun option in the plr list and details view

Fixes

https://issues.redhat.com/browse/HACBS-1174
Description

user should be able to cancel the Pipelineruns. Description of the actions are shown in the tooltip (Similar to Openshift console).
Type of change

[x] Feature

Screen shots / Gifs for design review

Tooltip:

https://user-images.githubusercontent.com/9964343/204297073-b3ff5439-d1e3-4ac3-85c8-7c8b3875cbbe.mov
https://user-images.githubusercontent.com/9964343/204297120-df99a777-c6a0-4ec7-9cce-0776535d6fe0.mov
How to test or reproduce?

Stop and Cancel action will be available in following places.

Kebab menu in pipelineruns list page
PipelineRun details page dropdown

Unit Tests
  PipelineRunDetailsView
    ✓ should render Stop and Cancel button under the Actions dropdown (134 ms)
    ✓ should enable Stop and Cancel button if the pipeline is running (107 ms)
    ✓ should disable Stop and Cancel button if the pipelinerun is succeeded (106 ms)
    ✓ should disable Stop and Cancel button if the pipelinerun is in cancelled state (103 ms)
    ✓ should disable Stop and Cancel button if the pipelinerun is in stopped state (102 ms)

Browser conformance:

[x] Chrome
[x] Firefox
[x] Safari
[ ] Edge

/lgtm
/approve


update prompt for starcoder/magicoder

Type of Change
update prompt=2012 for starcoder and magicoder
Description
error log:
starcoder: https://inteltf-jenk.sh.intel.com/blue/rest/organizations/jenkins/pipelines/nlp_toolkit_cpp_graph_test/runs/2667/nodes/77/steps/80/log/?start=0
magicoder: https://inteltf-jenk.sh.intel.com/blue/rest/organizations/jenkins/pipelines/neural_speed_cpp_graph_test/runs/106/nodes/77/steps/80/log/?start=0

https://inteltf-jenk.sh.intel.com/view/nlp-toolkit-validation/job/neural_speed_extension/27/

https://inteltf-jenk.sh.intel.com/job/neural_speed_cpp_graph_test/126/


Remove version: TeamViewer.TeamViewer version 15.53.6

Same as previous version
Microsoft Reviewers: Open in CodeFlow

URL: https://download.teamviewer.com/download/TeamViewer_Setup.exe
OK
(Automated message - build 674)

Hi @PckgrBot,
This manifest has the highest version number for this package. Is it available from another location? (This might be in error if the version is switching from semantic to string, or string to semantic.)
(Automated response - build 877.)

Remove as it has the same installer hash as previous

Hi @PckgrBot,
This manifest has the highest version number for this package. Is it available from another location? (This might be in error if the version is switching from semantic to string, or string to semantic.)
(Automated response - build 877.)


Update volume description

Design is for catalog but will be the same for volume

I can take this one.


Rectangle.Box coords are invalid

It seems that GetBoundingBox on a Result (from TessBaseApi) returns box coords as left, top, right, bottom whereas when creating the Rectangle, coords are used as left, top, width, height...

@gwyohm thanks! Fixed.


Azure.android

Apologies, but this does not appear to be a change we would accept. Please reach out by filing an issue if you are experiencing any problem with the SDK. For future code contributions, make sure to follow the Azure SDK Design Guidelines for Android.


add ElementType.LOCAL_VARIABLE to Target for @Nullable/NonNull

When dealing with unannotated code this would be useful for annotating "after the fact".
for example rev can be null here, but distance does not allow it
    var rev = this.git.get().tryGit(git -> git.getRepository().resolve(revString)).getOrElse(() -> null);
    return this.distance(rev);

since distance doesn't allow null, this would force the compile error.
    @Nullable var rev = this.git.get().tryGit(git -> git.getRepository().resolve(revString)).getOrElse(() -> null);
    return this.distance(rev);

could be a simple path forward for a local fixup. this is not as good as providing a definition, but still useful.

(I know that this isn't what you're asking, but for anyone who might come across this thread and wonder why we don't require such annotations on local variables: See this FAQ entry.)
There's some discussion around this in https://github.com/jspecify/jspecify/issues/251, which is nominally about @NonNull but also includes at least one post on @Nullable. To recap two points from there:

We'd likely want to support both @Nullable and @NonNull (as you suggest) or neither, so we'd want to have a good story for what both are useful for.
The Checker Framework (and perhaps other tools; I haven't checked) won't actually treat the value of rev as nullable in your example: It will continue to treat it as whatever type it thinks tryGet returned (which would be "non-null" for it currently and "unspecified" under JSpecify semantics). Other tools could do different things, and that's relatively OK for them to do for implementation code, but it's not ideal. If we end up having JSpecify get into this, that would probably happen as part of https://github.com/jspecify/jspecify/issues/661.

I don't know if this came up in the thread, but there's a potential workaround, which is to declare a method like this:
<T> @Nullable T castToNullable(T t) {
  return t;
}

(And you could likewise declare a castToNonNull method if you wanted to use it to "localize" errors, though we'd generally recommend requireNonNull instead. There is one occasional edge case in which employ a similar but slightly different trick.)
Then you can write:
    var rev = castToNullable(this.git.get().tryGit(git -> git.getRepository().resolve(revString)).getOrElse(() -> null));

(Bonus: If you later reassign a non-null value to rev, the checker can recognize that the value is non-null after that. Under the @Nullable var approach, I'd expect it to stay nullable.)

Oh, one other thought here: We can separate two possible features:

@Target(..., LOCAL_VARIABLE)
permitting usages on types of local variables

I think that (1), while convenient for the var case, ends up as a non-starter because of the non-var case: It means that a declaration like @Nullable String[] s; would mean both "this variable is nullable" and "this array's elements are nullable," thanks to Java's rules for what I like to call "hybrid" annotations (i.e., those that are applicable both to types and to "elements" like local variables).
My earlier response was written more from the perspective of (2), though probably all of those points would carry over to (1), as well.


Fix panic on failed load balancer key parsing

Problem
In metrics.go on line 519 we do not check the length of the slice before accessing by index, causing panics in some cases.
Proposed solution
Check the length of the slice before accessing

https://github.com/kubecost/cost-model/pull/579/files fixes, but I'm not sure this was ever a real issue, given that we should be doing the nil check somewhere else? Better to be safe though so we can merge #579


Handling of empty properties

It seems like empty properties are (generally? often?) handled as non-existent. Example in which I noticed this is Martijn's h-card with Martijn van der Ven, which gets parsed as url: ["Martijn van der Ven", …], uid: ["Martijn van der Ven"] instead of correctly resolving the url to the current page.
(This was noticed using microformats-node, but happens using the -shiv chrome extension as well)

There was a similar php-mf2 bug recently: https://github.com/indieweb/php-mf2/issues/118

Important to note that this also applies to implied properties.


Do not mandate direct bytes in SslHandler queue

Motivation:
Currently when the SslHandler coalesces outbound bytes it always
allocates a direct byte buffer. This does not make sense if the JDK
engine is being used as the bytes will have to be copied back to heap
bytes for the engine to operate on them.
Modifications:
Inspect engine type when coalescing outbound bytes and allocate heap
buffer if heap bytes are preferred by the engine.
Result:
Improved performance for JDK engine. Better performance in environments
without direct buffer pooling.

@netty-bot test this please

@tbrooks8 out of interest, do you have any benchmark results you can share?

@johnou unfortunately not. The reason this came to my attention was not because of an obvious performance regression, but because I was seeing direct buffers being allocated in an environment without direct buffer pooling enabled. Netty normal does its best to ensure pooled direct buffers are enabled before using them.
As I investigated more I saw this was only happening in the SslHandler. The reasoning behind my decision to always use a heap buffer for the JDK engine, is that if you pass the JDK engine a direct buffer it will internally allocate a byte array and copy the data to that for wrap. I just presume that is worse than using a heap buffer in the first place.

@tbrooks8 great catch!

@tbrooks8 btw are you sure the copy to heap byte array is still a thing in java11 and later ?

@normanmaurer Obviously it is pretty complicated to read through the JDK code with all the indirection. But here is what I currently see (JDK 12):
The SSLEngine will eventually call to Cipher#doFinal(ByteBuffer input, ByteBuffer output) or Ciper#update(ByteBuffer input, ByteBuffer output).
    public final int doFinal(ByteBuffer input, ByteBuffer output)
            throws ShortBufferException, IllegalBlockSizeException,
            BadPaddingException {
        checkCipherState();

        if ((input == null) || (output == null)) {
            throw new IllegalArgumentException("Buffers must not be null");
        }
        if (input == output) {
            throw new IllegalArgumentException("Input and output buffers must "
                + "not be the same object, consider using buffer.duplicate()");
        }
        if (output.isReadOnly()) {
            throw new ReadOnlyBufferException();
        }

        chooseFirstProvider();
        return spi.engineDoFinal(input, output);
    }

This calls The CipherSpi:
protected int engineDoFinal(ByteBuffer input, ByteBuffer output)
            throws ShortBufferException, IllegalBlockSizeException,
            BadPaddingException {
        return bufferCrypt(input, output, false);
    }

bufferCrypt does the copying.
    /**
     * Implementation for encryption using ByteBuffers. Used for both
     * engineUpdate() and engineDoFinal().
     */
    private int bufferCrypt(ByteBuffer input, ByteBuffer output,
            boolean isUpdate) throws ShortBufferException,
            IllegalBlockSizeException, BadPaddingException {
        if ((input == null) || (output == null)) {
            throw new NullPointerException
                ("Input and output buffers must not be null");
        }
        int inPos = input.position();
        int inLimit = input.limit();
        int inLen = inLimit - inPos;
        if (isUpdate && (inLen == 0)) {
            return 0;
        }
        int outLenNeeded = engineGetOutputSize(inLen);

        if (output.remaining() < outLenNeeded) {
            throw new ShortBufferException("Need at least " + outLenNeeded
                + " bytes of space in output buffer");
        }

        boolean a1 = input.hasArray();
        boolean a2 = output.hasArray();
        int total = 0;
        byte[] inArray, outArray;
        if (a2) { // output has an accessible byte[]
            outArray = output.array();
            int outPos = output.position();
            int outOfs = output.arrayOffset() + outPos;

            if (a1) { // input also has an accessible byte[]
                inArray = input.array();
                int inOfs = input.arrayOffset() + inPos;
                if (isUpdate) {
                    total = engineUpdate(inArray, inOfs, inLen, outArray, outOfs);
                } else {
                    total = engineDoFinal(inArray, inOfs, inLen, outArray, outOfs);
                }
                input.position(inLimit);
            } else { // input does not have accessible byte[]
                inArray = new byte[getTempArraySize(inLen)];
                do {
                    int chunk = Math.min(inLen, inArray.length);
                    if (chunk > 0) {
                        input.get(inArray, 0, chunk);
                    }
                    int n;
                    if (isUpdate || (inLen > chunk)) {
                        n = engineUpdate(inArray, 0, chunk, outArray, outOfs);
                    } else {
                        n = engineDoFinal(inArray, 0, chunk, outArray, outOfs);
                    }
                    total += n;
                    outOfs += n;
                    inLen -= chunk;
                } while (inLen > 0);
            }
            output.position(outPos + total);
        } else { // output does not have an accessible byte[]
            if (a1) { // but input has an accessible byte[]
                inArray = input.array();
                int inOfs = input.arrayOffset() + inPos;
                if (isUpdate) {
                    outArray = engineUpdate(inArray, inOfs, inLen);
                } else {
                    outArray = engineDoFinal(inArray, inOfs, inLen);
                }
                input.position(inLimit);
                if (outArray != null && outArray.length != 0) {
                    output.put(outArray);
                    total = outArray.length;
                }
            } else { // input also does not have an accessible byte[]
                inArray = new byte[getTempArraySize(inLen)];
                do {
                    int chunk = Math.min(inLen, inArray.length);
                    if (chunk > 0) {
                        input.get(inArray, 0, chunk);
                    }
                    int n;
                    if (isUpdate || (inLen > chunk)) {
                        outArray = engineUpdate(inArray, 0, chunk);
                    } else {
                        outArray = engineDoFinal(inArray, 0, chunk);
                    }
                    if (outArray != null && outArray.length != 0) {
                        output.put(outArray);
                        total += outArray.length;
                    }
                    inLen -= chunk;
                } while (inLen > 0);
            }
        }
        return total;
    }

It looks to me that the P11Cipher overrides engineDoFinal and engineUpdate with implementations that do support direct byte buffers. But that is the only Cipher. For example ChaCha20Cipher and AESCipher appear to only operate on byte arrays to me.
So I think my understanding is correct. And I have seen in debugging allocations happening when working with AES. But I can investigate more this week if you think there is something I am missing.

Slightly related, but Conscrypt had a very similar issue a few years back, which never ended up getting resolved: https://github.com/google/conscrypt/issues/317


feat: implement disk-based inverted index

Codecov Report
Attention: Patch coverage is 66.75676% with 246 lines in your changes missing coverage. Please review.

Project coverage is 79.38%. Comparing base (9535186) to head (921c421).

Files
Patch %
Lines

rust/lance-index/src/scalar/inverted/index.rs
67.08%
109 Missing and 22 partials :warning:

rust/lance-index/src/scalar/inverted/builder.rs
64.65%
74 Missing and 14 partials :warning:

rust/lance-index/src/scalar/inverted/wand.rs
74.07%
16 Missing and 5 partials :warning:

rust/lance-index/src/scalar/lance_format.rs
60.00%
4 Missing :warning:

rust/lance-index/src/scalar/bitmap.rs
0.00%
1 Missing :warning:

rust/lance/src/io/exec/fts.rs
0.00%
1 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2643      +/-   ##
==========================================
- Coverage   79.44%   79.38%   -0.07%
==========================================
  Files         221      222       +1
  Lines       64683    64849     +166
  Branches    64683    64849     +166
==========================================
+ Hits        51388    51480      +92
- Misses      10335    10410      +75
+ Partials     2960     2959       -1

Flag
Coverage Δ

unittests
79.38% <66.75%> (-0.07%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Error when generating graph invoking programl

When I try to generate graph using programl, say
graph_generator(kernel, path, BENCHMARK, generate_programl = True)
the using of commands llvm2graph and graph2json in programl_script.sh meets some troubles.
I have already installed programl 0.3.2. How can I use these two commands of it?
####################
now processing aes
/home/qinaox/GNN-DSE/GNN-DSE/dse_database/programl_script.sh: line 3: llvm2graph: command not found
/home/qinaox/GNN-DSE/GNN-DSE/dse_database/programl_script.sh: line 4: graph2json: command not found
Traceback (most recent call last):
File "graph_gen.py", line 439, in
graph_generator(kernel, path, BENCHMARK, generate_programl = True)
File "graph_gen.py", line 390, in graph_generator
g_nx = read_json_graph(join(path, name))
File "graph_gen.py", line 78, in read_json_graph
js_graph=json.load(f)
File "/usr/lib/python3.6/json/init.py", line 299, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib/python3.6/json/init.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Thanks for your kind reply. It works with the PrograML Python APIs.


[Codegen] Bail TileAndDistributeToWorkgroups if the workgroup count is lowered

If workgroup IDs are already present in the IR, infer that it has already been distributed and skip tile + distribute.

I guess this is fine, but the expectation is that if the distribution has already been done the export op region also has been specified properly. So this check has a bit of a smell.

Makes sense, is the new check more reasonable? Today if there is no flow.workgroup_count_from* op it will still try to tile and distribute and then just not touch the workgroup count region, so this does change the behavior if someone for whatever reason wants to pre-lower the workgroup count region.


Bug: Keybinding configuration is not accessible

Keybindings for Outfitter are not showing up in the Options:Keybindings panel.

Will have a look in how to show it in the new options

Next version which should be released soon will have a fix for the keybindings they will be under Outfitter! added question mark because I couldn't use addon name for it and couldn't find a way to use translations for that.

thank you again Nulian


Request to contribute a supported react template.

Preflight Checklist

[x] I have read the contribution documentation for this project.
[x] I agree to follow the code of conduct that this project follows, as appropriate.
[x] I have searched the issue tracker for a feature request that matches the one I want to file, without success.

Problem Description

I can't seem to find any supported templates for electron forge with react.
Proposed Solution

I have read the decisions made by the maintainers to not maintain framework specific templates, which I am totally onboard with. So I would be happy to contribute and make a template for it.
Would like to know your thoughts on it.

@kanitsharma I'm looking for exactly this.  Need to upgrade an Electron-forge 5.x app that started with the 5.x React/TS template to 6.x.  Don't want to reinvent the wheel!

Electron Forge is designed to use third party plugins and templates. Use one of the existing first-party templates as a guide and you can publish your own NPM package that you can use in the --template option of init.


lightdm-gtk-greeter: 2.0.8 -> 2.0.9

https://github.com/Xubuntu/lightdm-gtk-greeter/compare/lightdm-gtk-greeter-2.0.8...lightdm-gtk-greeter-2.0.9
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

@ofborg test lightdm


Submarines of Deleted and Ignored Characters Reappearing

Repro Steps #1.

Delete Character (s) from the Manage menu, and add character(s) to Ignore
Enter the house of the deleted character.
Enter the Company Workshop
Previously deleted character(s) as well as ignored character(s) subs will appear in the Submarine Tracker Overlay.

Repro Steps #2.

Delete Character (s) from the Manage menu, and add character(s) to Ignore
Enter the house of a separate character.
Enter the Company Workshop
Previously deleted character(s) as well as ignored character(s) subs will appear in the Submarine Tracker Overlay.

Deletion is currently "disabled" , it will come back at some point in the near future

Fixed


Add new Embeddings endpoint

Adds ability to create embeddings.
See more:
https://beta.openai.com/docs/guides/embeddings/what-are-embeddings
https://beta.openai.com/docs/api-reference/embeddings/create

Thank you for the PR 🙌


can't get data when overriding apps

Please try to fill all questions below before submitting an issue.

[ ] On which android versions do you have this issue? : every version
[ ] On which phones do you have this issue? : xiaomi 8 lite
[ ] Which hawk version are you using? : 2.0.1
[ ] Does this issue happen always or is it flaky? : no
[ ] Can you write failing test? : no
[ ] If it is Hawk.put and Hawk.get issue, can you write down which data you were trying to save and get it back?no
[ ] Is the data you are trying to save it huge or small? small

i am using Hawk in a project (A), it's publishing in google play. Now, i created another project (B), I let the appId of 2 projects overlap, I increased the version code and version name on B, B can override A but B can't get data in A.
in project A, i saved String by Hawk, when B override A, B can read this data, but when i save object in A, B can't read it.
What have i do?
Thanks

I have exactly the same problem
when i try to update app to new version، that's how it was
On which android versions do you have this issue? : every version
On which phones do you have this issue? : every phones
Which hawk version are you using? : 2.0.1
Does this issue happen always or is it flaky? : in latest update of app happend
Can you write failing test? : no
If it is Hawk.put and Hawk.get issue, can you write down which data you were trying to save and get it back? simple pojo
Is the data you are trying to save it huge or small? small

this is my pojo
public class TokenRes {
@SerializedName("token_type")
@Expose
public String tokenType;
@SerializedName("expires_in")
@Expose
public String expiresIn;
@SerializedName("access_token")
@Expose
public String accessToken;
@SerializedName("refresh_token")
@Expose
public String refreshToken;

}
When will this problem be solved? Or is there at least one solution right now?
I encountered this strange problem just when the new version was delivered, and I have very little time

I fixed this problem. my solution was to clone the hawk project and copy all the code into my project then i edited the code in it. I found the problem where the hawk library decryption key, I skipped the encryption and decryption step.

I have exactly the same problem when i try to update app to new version، that's how it was
On which android versions do you have this issue? : every version On which phones do you have this issue? : every phones Which hawk version are you using? : 2.0.1 Does this issue happen always or is it flaky? : in latest update of app happend Can you write failing test? : no If it is Hawk.put and Hawk.get issue, can you write down which data you were trying to save and get it back? simple pojo Is the data you are trying to save it huge or small? small
this is my pojo
public class TokenRes {
@SerializedName("token_type")
@Expose
public String tokenType;
@SerializedName("expires_in")
@Expose
public String expiresIn;
@SerializedName("access_token")
@Expose
public String accessToken;
@SerializedName("refresh_token")
@Expose
public String refreshToken;

}
When will this problem be solved? Or is there at least one solution right now? I encountered this strange problem just when the new version was delivered, and I have very little time

I fixed this problem. my solution was to clone the hawk project and copy all the code into my project then i edited the code in it. I found the problem where the hawk library decryption key, I skipped the encryption and decryption step.


Error after latest update

Same here with AttributeError: 'float' object has no attribute 'find'