High file size increase of PDF when editing with many pages

First of all thanks to everyone collaborating in this awesome library. 🎉
I am working on a tool that must me able to reorder an existing PDF in the browser without any server intervention, for that i am using Mozilla's PDF.js library to do the render part and this library to do the actual file edition. This is the code that i'm using to do such task:
/**
 *
 * @param {Uint8Array} file Raw PDF file bytes
 * @param {Array} newOrder array of the new page order, example: [3,1,2] means page 3 is now page 1 and so on
 */
const reorderPdf = (rawPdf, newOrder) => {
  const pdfDoc = PDFLib.PDFDocumentFactory.load(rawPdf);
  const pdfDocPages = pdfDoc.getPages();
  const outPdf = PDFLib.PDFDocumentFactory.create();

  for (let currentPage = 0; currentPage < newOrder.length; currentPage++) {
    outPdf.addPage(pdfDocPages[newOrder[currentPage] - 1]);
  }

  const outPdfBytes = PDFLib.PDFDocumentWriter.saveToBytes(outPdf, { useObjectStreams: true });
  bytes2File(outPdfBytes);
};

While testing with various files i notice that there is always a small increment in the size, this was without any reordering being done. After all of the testing i concluded that when creating a PDF there is a significant overhead in size attached to the creation of a page, take for example this table (will provide files at the end):

File Name
Original Size
Pages
New Size
Increment

document A
75.8 KB
4
87 KB
+14%~

document B
224 KB
38
1,841 KB
+800%~

document C
2,238 KB
7
3,331 KB
+48%~

document D
1,898 KB
229
44,354 KB
+2200%~

Document A, which has a small size only gets increased a bit after passing it to my function.
Document B which has a considerable number of pages get increased a lot with respect to the original size, still not a problem since the resultant size is manageable.
Document C and D is where things get wild, notice how they both are about the same size (2 MB), but with much larger page difference. The result of document B is noticeable big but still, manageable, but the result of document D is really a problem, with an increment of 2200%.

The behaviour of those last 2 documents gets worse the more pages the original document has, with a case of a document of 500 pages with only 7MB in sice increasing up to 150MB.
Thanks for your time.

Files:
document A.pdf
document B.pdf
document C.pdf
document D.pdf

Hello @AldrinSalazar!
You've stumbled upon a flaw in the design of pdf-lib's page copying API. The version of this API that exists in v0.x.x will often create duplicate objects whenever more than one page is copied between different PDFDocument objects.
This flaw will be fixed in v1.0.0. This version is currently in beta (1.0.0-beta.3). I'd recommend you try it out. I'm hoping to do a full release of v1.0.0 within 2-3 weeks. Most of the remaining work is just documenting things.
That being said, there is a better way to implement page reordering. This approach is faster and also avoids creating duplicate objects. Basically, instead of creating two PDFDocument objects and copying pages between them, you can create a single PDFDocument and reorder pages within it. Here are some examples to clarify things:
Bad Page Reversal
This implementation is slower than necessary, and also ends up bloating the resulting file. It's slow due to the nature of the implementation (and it would also be slow in v1.0.0). It bloats the file size due to an API flaw in v0.x.x:
const badReversePages = rawPdf => {
  const pdfDoc = PDFDocumentFactory.load(rawPdf);
  const newDoc = PDFDocumentFactory.create();
  pdfDoc
    .getPages()
    .reverse()
    .forEach(page => {
      newDoc.addPage(page);
    });
  return PDFDocumentWriter.saveToBytes(newDoc);
};

const fs = require('fs');
const rawPdf = fs.readFileSync('document D.pdf');

console.time('badReversePages');
const pdfBytes = badReversePages(rawPdf);
console.timeEnd('badReversePages');
console.log('pdfBytes.length ===', pdfBytes.length);

// => badReversePages: 878.281ms
// => pdfBytes.length === 45418620

Good Page Reversal
This implementation is much faster (~57%) and avoids bloating the file:
const goodReversePages = (rawPdf: any) => {
  const pdfDoc = PDFDocumentFactory.load(rawPdf);
  const pages = pdfDoc.getPages();
  pdfDoc
    .getPages()
    .reverse()
    .forEach((page, idx) => {
      pdfDoc.removePage(idx);
      pdfDoc.insertPage(idx, page);
    });
  return PDFDocumentWriter.saveToBytes(pdfDoc);
};

const fs = require('fs');
const rawPdf = fs.readFileSync('document D.pdf');

console.time('goodReversePages');
const pdfBytes = goodReversePages(rawPdf);
console.timeEnd('goodReversePages');
console.log('pdfBytes.length ===', pdfBytes.length);

// => goodReversePages: 372.451ms
// => pdfBytes.length === 1883468

Comparison
The original size of the input document (document D.pdf) is 1943532 bytes (1.94 MB).

Implementation
Speed
Output Size
Difference

Bad
878ms
~45 MB
+43 MB

Good
372ms
1.88 MB
-0.06 MB

As you can see, the good implementation is actually able to decrease the size of the document (despite not having removed any pages).
Reimplemented reorderPdf
Here's a verion of the reorderPdf function with the "good" implementation:
/**
 *
 * @param {Uint8Array} file Raw PDF file bytes
 * @param {Array} newOrder array of the new page order, example: [3,1,2] means page 3 is now page 1 and so on
 */
const reorderPdf = (rawPdf, newOrder) => {
  const pdfDoc = PDFDocumentFactory.load(rawPdf);

  const pages = pdfDoc.getPages();
  for (let currentPage = 0; currentPage < newOrder.length; currentPage++) {
    pdfDoc.removePage(currentPage);
    pdfDoc.insertPage(currentPage, pages[newOrder[currentPage] - 1]);
  }

  const pdfBytes = PDFDocumentWriter.saveToBytes(pdfDoc);
  return bytes2File(pdfBytes);
};

I hope this helps! Please let me know if you have any additional questions.

Thanks for your very detailed answer! it helped a lot and indeed it makes PDF a little smaller and is faster. I dont know if this is a common use case but i think it would be a good idea to have that snippet of code as one of the examples in the lib.
Thanks!

Sorry for resurrecting a months-old closed thread but I wanted to say thanks for this thread and for the detailed comment above; this specific page came up when I was searching for how to do this and I'm glad to have learned of this project/library... it helped me scratch a minor itch and write a trivial (given pdf-lib) tool. :-)

@shreevatsa That's a really neat tool you've built! Thanks for sharing. I really enjoy seeing the cool things people use pdf-lib for.


implement feature layer shapefile sample

shapefile should already be present in samples-data

@john0005 Please review when you get a chance


Jade and normal multisig compatibility

Are there plans to add support for standard multisig wallets?
I see that you have support for Green-specific 2-of-2 or 2-of-3 or 2-of-2+csv, but would be nice to be able to use Jade in standard multisig with other hardware wallets.
For example, software wallet could pass receiving and change descriptors to the HW and it could verify that all scriptpubkeys in inputs and change outputs are derived from these descriptors.

This is something in the pipeline for an upcoming release.
The current thinking is that you would have to upload/inspect/confirm a descriptor for the multisig on Jade.  Once registered, you can then get receive addresses, validate change outputs and sign multisigs based on that descriptor.

Merged to master, for the next fw release, 0.1.28

fyi: the next release of the Jade fw should add support for bip67/sortedmulti(), and a PR to enable that on HWI will follow.
Then hopefully this item should be fully addressed.  Cheers.

fyi: Jade fw 0.1.32 is now available, which supports sortedmulti().  An HWI PR will be opened early in the new year.

Awesome, thank you!

Interesting article: https://shiftcrypto.ch/blog/the-pitfalls-of-multisig-when-using-hardware-wallets/

Thanks.
So with Jade we opted to be quite strict in this regard -  in order to use multisig the user first needs to register the setup on Jade - ie. as a separate registration step the user inspects/confirms:

all co-signer xpubs (the hw checks that at least one belongs the Jade)
threshold
script type
(and now) whether bip67 key sorting applies

Once that has been confirmed, future requests to get receive addresses or change addresses will use the registered setup - the only additional data passed per request is the bip32 path suffixes.  The full key derivation paths are validated against bip45/48/87, and that the final  element is within a reasonable range.  If this validation fails, a warning is displayed and the user must explicitly check/confirm the path.
We believe this should cover the known attack scenarios - but obviously are happy to hear if you think otherwise!  ;-)

I am trygin to implement that and currently have a problem registering multisig on the device, here are the parameters for register_multisig json-rpc request:
{'network': 'testnet', 'multisig_name': 'hwi41bdc647542b', 'descriptor': {'variant': 'wsh(multi(k))', 'sorted': True, 'threshold': 2, 'signers': [{'fingerprint': '396779c9', 'derivation': [2147483696,<PHONE_NUMBER>,<PHONE_NUMBER>,<PHONE_NUMBER>], 'xpub': 'tpubDCGbvM6UyTcRHu4rXgSPnMhaHuNec86ME1Ap9CuqBbhNpwdR28a7MRQoeKeqizAcSb3MKFD47jHg2bYUR6u95tD3K5grTAFXHvhWwELX6C2', 'path': []}, {'fingerprint': 'fb7c1f11', 'derivation': [2147483696,<PHONE_NUMBER>,<PHONE_NUMBER>,<PHONE_NUMBER>], 'xpub': 'tpubDExnGppazLhZPNadP8Q5Vgee2QcvbyAf9GvGaEY7ALVJREaG2vdTqv1MHRoDtPaYP3y1DGVx7wrKKhsLhs26GY263uE6Wi3qNbi71AHZ6p7', 'path': []}, {'fingerprint': '47fc1ba1', 'derivation': [2147483696,<PHONE_NUMBER>,<PHONE_NUMBER>,<PHONE_NUMBER>], 'xpub': 'tpubDFa3rRFxivQmR3dcgfStqsry6QXHrjpV1FjyXKE8BroySBRTL1SD8ddh2sWXUZxF6wRZufhpkaiR9qqmc7mzEUr4ddvpN7ZHkvJ7vTbgRJD', 'path': []}]}}

The same happens with the current master of HWI when I am trying to display address for unsorted multisig descriptor:
hwi --chain regtest -f 396779c9 displayaddress --desc "wsh(multi(2,[396779c9/48h/1h/0h/2h]tpubDCGbvM6UyTcRHu4rXgSPnMhaHuNec86ME1Ap9CuqBbhNpwdR28a7MRQoeKeqizAcSb3MKFD47jHg2bYUR6u95tD3K5grTAFXHvhWwELX6C2/0/0,[fb7c1f11/48h/1h/0h/2h]tpubDExnGppazLhZPNadP8Q5Vgee2QcvbyAf9GvGaEY7ALVJREaG2vdTqv1MHRoDtPaYP3y1DGVx7wrKKhsLhs26GY263uE6Wi3qNbi71AHZ6p7/0/0,[47fc1ba1/48h/1h/0h/2h]tpubDFa3rRFxivQmR3dcgfStqsry6QXHrjpV1FjyXKE8BroySBRTL1SD8ddh2sWXUZxF6wRZufhpkaiR9qqmc7mzEUr4ddvpN7ZHkvJ7vTbgRJD/0/0))#5a3f60y9"
{"error": "Failed to validate multisig co-signers", "code": -7}

Is one of the cosigners the Jade key/fingerprint - and it is correct ?  ie. the jade root key plus the derivation given, definitely yields that xpub ?  Jade itself and HWI have tests which [should] be covering the above.

OK, I loaded the test seed (from the jade tests) into jade and got the xpub we want to use:
TEST_SEED = bytes.fromhex('b90e532426d0dc20fffe01037048c018e940300038b165c211915c672e07762c')
jade.set_seed(TEST_SEED, temporary_wallet=True)
print(jade.get_xpub('localtest', [2147483696,<PHONE_NUMBER>,<PHONE_NUMBER>,<PHONE_NUMBER>]))

to get:
tpubDEAjmvwVDj4aTdNLyLdoVGdEuRf8ZMsZEun6Aa2uMHa3DFNSzXQFt8DY62gi37RuaPTcpoNFage2h6dPHABssCnWcp2j6srQTpWbGEKHgU1
I then switched out the first signer fingerprint/xpub of your multisig to my jade xpub, and got the following:
(HWI)$ hwi enumerate
[{"type": "jade", "model": "jade", "path": "/dev/ttyUSB0", "needs_pin_sent": false, "needs_passphrase_sent": false, "fingerprint": "1273da33"}]

(HWI)$ hwi --chain regtest -f 1273da33 displayaddress --desc "wsh(multi(2,[1273da33/48h/1h/0h/2h]tpubDEAjmvwVDj4aTdNLyLdoVGdEuRf8ZMsZEun6Aa2uMHa3DFNSzXQFt8DY62gi37RuaPTcpoNFage2h6dPHABssCnWcp2j6srQTpWbGEKHgU1/0/0,[fb7c1f11/48h/1h/0h/2h]tpubDExnGppazLhZPNadP8Q5Vgee2QcvbyAf9GvGaEY7ALVJREaG2vdTqv1MHRoDtPaYP3y1DGVx7wrKKhsLhs26GY263uE6Wi3qNbi71AHZ6p7/0/0,[47fc1ba1/48h/1h/0h/2h]tpubDFa3rRFxivQmR3dcgfStqsry6QXHrjpV1FjyXKE8BroySBRTL1SD8ddh2sWXUZxF6wRZufhpkaiR9qqmc7mzEUr4ddvpN7ZHkvJ7vTbgRJD/0/0))"

{"address": "bcrt1qegfvejxlklt72peyhjyg9f0zm579ep0wv9lqqqg4ek75zk8yhpasp5vr8t"}

With your first example, I added my Jade test key, so the multisig became a 2-of-4:
    multisig = {
      'network': 'testnet',
      'multisig_name': 'hwi41bdc647542b',
      'descriptor': {
        'variant': 'wsh(multi(k))',
        'sorted': True,
        'threshold': 2,
        'signers': [
          {'fingerprint': bytes.fromhex('1273da33'), 'derivation': [2147483696,<PHONE_NUMBER>,<PHONE_NUMBER>,<PHONE_NUMBER>], 'xpub': 'tpubDEAjmvwVDj4aTdNLyLdoVGdEuRf8ZMsZEun6Aa2uMHa3DFNSzXQFt8DY62gi37RuaPTcpoNFage2h6dPHABssCnWcp2j6srQTpWbGEKHgU1', 'path': []},
          {'fingerprint': bytes.fromhex('396779c9'), 'derivation': [2147483696,<PHONE_NUMBER>,<PHONE_NUMBER>,<PHONE_NUMBER>], 'xpub': 'tpubDCGbvM6UyTcRHu4rXgSPnMhaHuNec86ME1Ap9CuqBbhNpwdR28a7MRQoeKeqizAcSb3MKFD47jHg2bYUR6u95tD3K5grTAFXHvhWwELX6C2', 'path': []},
          {'fingerprint': bytes.fromhex('fb7c1f11'), 'derivation': [2147483696,<PHONE_NUMBER>,<PHONE_NUMBER>,<PHONE_NUMBER>], 'xpub': 'tpubDExnGppazLhZPNadP8Q5Vgee2QcvbyAf9GvGaEY7ALVJREaG2vdTqv1MHRoDtPaYP3y1DGVx7wrKKhsLhs26GY263uE6Wi3qNbi71AHZ6p7', 'path': []},
          {'fingerprint': bytes.fromhex('47fc1ba1'), 'derivation': [2147483696,<PHONE_NUMBER>,<PHONE_NUMBER>,<PHONE_NUMBER>], 'xpub': 'tpubDFa3rRFxivQmR3dcgfStqsry6QXHrjpV1FjyXKE8BroySBRTL1SD8ddh2sWXUZxF6wRZufhpkaiR9qqmc7mzEUr4ddvpN7ZHkvJ7vTbgRJD', 'path': []}
        ]
      }
    }
    print(jade._jadeRpc('register_multisig', params=multisig))

(Note the fingerprints are sent as bytes, not hex strings)
Sending: {'method': 'register_multisig', 'id': '512432', 'params': {'network': 'testnet', 'multisig_name': 'hwi41bdc647542b', 'descriptor': {'variant': 'wsh(multi(k))', 'sorted': True, 'threshold': 2, 'signers': [{'fingerprint': '1273da33', 'derivation': [2147483696,<PHONE_NUMBER>,<PHONE_NUMBER>,<PHONE_NUMBER>], 'xpub': 'tpubDEAjmvwVDj4aTdNLyLdoVGdEuRf8ZMsZEun6Aa2uMHa3DFNSzXQFt8DY62gi37RuaPTcpoNFage2h6dPHABssCnWcp2j6srQTpWbGEKHgU1', 'path': []}, {'fingerprint': '396779c9', 'derivation': [2147483696,<PHONE_NUMBER>,<PHONE_NUMBER>,<PHONE_NUMBER>], 'xpub': 'tpubDCGbvM6UyTcRHu4rXgSPnMhaHuNec86ME1Ap9CuqBbhNpwdR28a7MRQoeKeqizAcSb3MKFD47jHg2bYUR6u95tD3K5grTAFXHvhWwELX6C2', 'path': []}, {'fingerprint': 'fb7c1f11', 'derivation': [2147483696,<PHONE_NUMBER>,<PHONE_NUMBER>,<PHONE_NUMBER>], 'xpub': 'tpubDExnGppazLhZPNadP8Q5Vgee2QcvbyAf9GvGaEY7ALVJREaG2vdTqv1MHRoDtPaYP3y1DGVx7wrKKhsLhs26GY263uE6Wi3qNbi71AHZ6p7', 'path': []}, {'fingerprint': '47fc1ba1', 'derivation': [2147483696,<PHONE_NUMBER>,<PHONE_NUMBER>,<PHONE_NUMBER>], 'xpub': 'tpubDFa3rRFxivQmR3dcgfStqsry6QXHrjpV1FjyXKE8BroySBRTL1SD8ddh2sWXUZxF6wRZufhpkaiR9qqmc7mzEUr4ddvpN7ZHkvJ7vTbgRJD', 'path': []}]}}} as cbor of size 848

<pause while confirmed on hw>

Received msg: {'id': '512432', 'result': True}

fyi: I do have 'Add api/cbor-message documentation' as an outstanding task!

Thank you for the clarification, Jamie. I had one more doubt, but I created a new issue not to go off-topic on this issue: https://github.com/Blockstream/Jade/issues/29

@JamieDriver this is very strange, I tried again, the same issue. Firmware version 0.1.32.
Here is the python code I use - it gets xpub from the device, adds it to cosigners and tries to register multisig.
Can you try running it yourself? Could it be that production and debug firmware behave differently?
from jadepy import JadeAPI
from hwilib import _base58 as base58

NETWORK = "testnet"
HARD = 0x80000000
PATH = [HARD+48, HARD+1, HARD, HARD+2]

params = {
'network': NETWORK,
'multisig_name': 'tmp',
'descriptor': {
  'variant': 'wsh(multi(k))',
  'sorted': True,
  'threshold': 2,
  'signers': [
    {'fingerprint': 'fb7c1f11', 'derivation': PATH, 'xpub': 'tpubDExnGppazLhZPNadP8Q5Vgee2QcvbyAf9GvGaEY7ALVJREaG2vdTqv1MHRoDtPaYP3y1DGVx7wrKKhsLhs26GY263uE6Wi3qNbi71AHZ6p7', 'path': []},
    {'fingerprint': '47fc1ba1', 'derivation': PATH, 'xpub': 'tpubDFa3rRFxivQmR3dcgfStqsry6QXHrjpV1FjyXKE8BroySBRTL1SD8ddh2sWXUZxF6wRZufhpkaiR9qqmc7mzEUr4ddvpN7ZHkvJ7vTbgRJD', 'path': []}
  ]}
}

with JadeAPI.create_serial("/dev/ttyUSB0") as d:
    d.auth_user(NETWORK)
    xpub = d.get_xpub(NETWORK, PATH)
    print(xpub)
    child0 = d.get_xpub(NETWORK, [0])
    fgp = base58.get_xpub_fingerprint(child0)
    print(fgp.hex())
    params["descriptor"]["signers"].append({
        "fingerprint": fgp.hex(),
        "derivation": PATH,
        "xpub": xpub,
        "path": [],
    })
    print(d._jadeRpc('register_multisig', params=params))

As I say above (and as in my example) fingerprint should be bytes, not hex-string.
ie:
@@ -16,6 +16,6 @@
   'threshold': 2,
   'signers': [
-    {'fingerprint': 'fb7c1f11', 'derivation': PATH, 'xpub': 'tpubDExnGppazLhZPNadP8Q5Vgee2QcvbyAf9GvGaEY7ALVJREaG2vdTqv1MHRoDtPaYP3y1DGVx7wrKKhsLhs26GY263uE6Wi3qNbi71AHZ6p7', 'path': []},
-    {'fingerprint': '47fc1ba1', 'derivation': PATH, 'xpub': 'tpubDFa3rRFxivQmR3dcgfStqsry6QXHrjpV1FjyXKE8BroySBRTL1SD8ddh2sWXUZxF6wRZufhpkaiR9qqmc7mzEUr4ddvpN7ZHkvJ7vTbgRJD', 'path': []}
+    {'fingerprint': bytes.fromhex('fb7c1f11'), 'derivation': PATH, 'xpub': 'tpubDExnGppazLhZPNadP8Q5Vgee2QcvbyAf9GvGaEY7ALVJREaG2vdTqv1MHRoDtPaYP3y1DGVx7wrKKhsLhs26GY263uE6Wi3qNbi71AHZ6p7', 'path': []},
+    {'fingerprint': bytes.fromhex('47fc1ba1'), 'derivation': PATH, 'xpub': 'tpubDFa3rRFxivQmR3dcgfStqsry6QXHrjpV1FjyXKE8BroySBRTL1SD8ddh2sWXUZxF6wRZufhpkaiR9qqmc7mzEUr4ddvpN7ZHkvJ7vTbgRJD', 'path': []}
   ]}
 }
@@ -29,5 +29,5 @@
     print(fgp.hex())
     params["descriptor"]["signers"].append({
-        "fingerprint": fgp.hex(),
+        "fingerprint": fgp,
         "derivation": PATH,
         "xpub": xpub,

with these changes, your script works for me.
I know I need to generate some documentation so totally my bad.  It is high on my list for the new year!

Sorry, I missed it. Works now, thanks a lot!

Thanks. So with Jade we opted to be quite strict in this regard - in order to use multisig the user first needs to register the setup on Jade - ie. as a separate registration step the user inspects/confirms:
* all co-signer xpubs (the hw checks that at least one belongs the Jade)

* threshold

* script type

* (and now) whether bip67 key sorting applies

Once that has been confirmed, future requests to get receive addresses or change addresses will use the registered setup - the only additional data passed per request is the bip32 path suffixes. The full key derivation paths are validated against bip45/48/87, and that the final element is within a reasonable range. If this validation fails, a warning is displayed and the user must explicitly check/confirm the path.
We believe this should cover the known attack scenarios - but obviously are happy to hear if you think otherwise! ;-)

I think I thought of a scenario.
I noticed that to get an address, the computer sends the following to the Jade:
INFO:jade:Sending: {'method': 'get_receive_address', 'id': '468131', 'params': {'network': 'mainnet', 'paths': [[0, 2], [0, 2], [0, 2]], 'multisig_name': 'hwi'}} as cbor of size 108

However, I see that the Jade accepts paths that don't match like:
INFO:jade:Sending: {'method': 'get_receive_address', 'id': '865589', 'params': {'network': 'mainnet', 'paths': [[0, 1], [0, 2], [0, 3]], 'multisig_name': 'hwi'}} as cbor of size 108

which I think is non-standard in most companion wallets.
Could an attacker pass something like [[0, 2333], [0,3999], [0,4383]] and perform a type of ransomware attack on get_address or sign_tx, where they will only reveal the paths in return for some BTC?
I was happy to see there's actually some validation on paths here: https://github.com/Blockstream/Jade/blob/34a1814bbde5486e08600de701619fb7779b5455/main/wallet.c#L343 , but I don't think this case is covered. I think this wouldn't be a problem for 2 of 3 multisig, as the victim could easily brute force the paths because there's a check on MAX_PATH_PTR of 10000, so 10000^3=10^12 combinations to try (for comparison, bitcoin network hashrate is in the order of 10^20 Hashes/second), but this might be a problem for larger multisigs (I see up to 8 signers are supported)? Should we be checking that the paths are all equal to each other for multisig wallets?

Might be a good idea.  Let's continue discussion on #30 - cheers.


Added name & Profile link

@OmkarK45  thnx for contributing, try this year, and make sure you follow me and star this repo for updates about special swags I will be giving.
Contribute in October for Hacktoberfest 2020

I want to say sorry for the PR I created last year. As a maintainer I can understand it's tough to manage spam PRs. Hopefully this year I will contribute in some meaningful way.


Unable to remove Liquid Syntax warning

+1

+1

No idea if it will get merged in given that there still pull requests open from 2015 but I fixed this issue in the pull request #24
Two stray '(' were causing the issue


Update searching-for-repositories.md

The search jquery in:name will only find repositories that start with the search term jquery, rather than repos that contain this search term in their name.
https://github.com/search?q=jquery+in%3Aname   (122,150 repository results)
For example if one changes the search to query in:name, i.e. drop the j, then if the original description in this file was correct it should find more repos. However as one can see by running this query
https://github.com/search?q=query+in%3Aname
you get less results, 81,119. This is because the examination is not actually searching if the word "query" is in the name but rather does the repo start with the word "query".
Just to prove the point lets search for repos that have "uery" in their names, uery in:name. If the code really was searching for this search term IN the name it should at least find as many results as in  jquery in:name.
https://github.com/search?q=uery+in%3Aname
We only get 8.

Why:
Closes [issue link]

What's being changed (if available, include any code snippets, screenshots, or gifs):

Check off the following:

[ ] I have reviewed my changes in staging (look for the "Automatically generated comment" and click the links in the "Preview" column to view your latest changes).
[ ] For content changes, I have completed the self-review checklist.

@screig Thanks so much for opening a PR and providing your examples! I'll get this triaged for review :zap:

Hi @screig, thanks again for bringing this to our attention and providing so much detail!
I followed your testing, and technically, it looks like searches for query in:name will return things like blueimp/jQuery-File-Upload and camsong/You-Dont-Need-jQuery, so it does successfully return some repositories, but you're right that the number of results decreases significantly from jquery in:name, and it isn't functioning as described. This is a little bit too detailed of an update to add to the docs as of now, but we have started work on it internally 👍🏼 thanks again for your help maintaining our docs! ✨

Bit of a disappointing response, you are leaving docs live that suggest features which in practise don't exist. I agree changing the underlying functionality is a big job, however by refusing to say in the docs that it does not work as described is very much like ignoring the problem.


[Question] Phrase plugin development

Hi Decathlon team,
I really like Phrase Gradle plugin but to be honest it's a little bit risky to use it on some bigger or important projects because it seems that you're not actively working on it.
So I have a simple question for you. Do you plan to do some development on it? Or is this project dying?
Thank you.

Hello @pstambrecht sorry for the delay, this project is used every day on our projects. So it is not dying at all. We was completely absorbed by other priorities. I will take the lead from today on this project. Don't hesitate to open an issue or to suggest a pull request if you find some issues !

Hello @adrienbusin,
thank you for your reply. It's perfect to read that this project is still alive. We like it. Meanwhile we've prepared some pull requests with a logic we were missing (https://github.com/Decathlon/gradle-plugin-phraseapp/pull/13,  https://github.com/Decathlon/gradle-plugin-phraseapp/pull/14). I'm happy that this was approved and is the part of official plugin.


Fix setting sandbox on Lua instance without strlib

Currently calling luaL_sandbox on Lua instance without loaded strlib causes crash (assertion).
It happens because inside luaL_sandbox there is no check that metatable  for strings is present.

Thank you.


[linq2db] DbConnection.GetSchema API support

Right now this API is not implemented and throws NotSupportedException from base class.
linq2db use following schema tables for ASE:

DataTypes
Procedures
ProcedureParameters

for other metadata we query system tables directly.
Why this feature is important for linq2db - schema provider used to generate data model from database using T4 templates.
Another important thing is support for CommandBehavior.SchemaOnly for procedures. Native provider (or in worst case ASE itself) has bug in this area, where it executes procedure for real which is very bad. See https://github.com/linq2db/linq2db/issues/792. We fixed it for native provider on our side by wrapping such calls in transactions with rollback.
linq2db test: SchemaProviderTest.IncludeExcludeCatalogTest

I would say, it has lower priority compared to bugs, as we can try to replace those calls on our side with queries to system tables for time being

On your point about CommandBehaviour.SchemaOnly and using a transaction to work around the SAP driver's bug, ASE does have support for getting a procedure's (or any command's) schema via SET FMTONLY [ON|OFF]

Also, it might be a while before DbConnection.GetSchema is implemented completely.
However, if all you rely on is DataTypes, Procedures, and ProcedureParameters then we could push out a partial implementation...

Thanks for fmtonly, will try it later, if it works, it meant that native provider doesn't set it...
As for API, as I said - we can workaround it on our side, so no rush here

FYI, this API is not required for linq2db anymore, as I implemented required functionality on our side. So if nobody else needs it ATM, you can postpone it


Using the OpenSaveFileDialog will now default to the directory of the…

This should solve issue #164.
It's caused because the filename is included when setting the InitialDirectory property which as of Vista causes it to use a different path. FileDialog.InitialDirectory - MSDN

Cool! Thanks!
Will test soon.

nice enhancement! Thanks!


Inserting decorated emojis while using an IME crashes editor

Do you want to request a feature or report a bug?
Bug.
What is the current behavior?
Typing in a language which uses an IME, inserting an emoji directly after it (i.e. no spaces), then typing in the IME-using language again gives the following error if you've applied a decorator to the emojis:

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem. You can use this jsfiddle to get started: https://jsfiddle.net/gmertk/e61z7nfa/.
Fiddle: https://jsfiddle.net/36tvhmce/
For example, in Chinese on Mac:

Switch your input method to "Pinyin - Traditional".
Type in the Draft.js field "yldx" (Yale University) and hit spacebar to insert the Chinese characters.
Type "hhh" (for laughing face emoji) and hit spacebar to select the laughing face emoji.
Type "yldx" and hit spacebar again to insert more Chinese characters.
Notice the above error in the console. Also notice that you can't backspace anything you've typed.

There's similar behavior inserting Japanese:

Switch your input method to Hiragana.
Type in the Draft.js field "watashi" (I) then hit Enter to select 私.
Type "emoji" and scroll down by hitting spacebar until you land on any emoji, then hit Enter to select it.
Type "watashi" and select 私 with Enter again.
Notice the above error in the console. Also notice that you can't backspace anything you've typed.

What is the expected behavior?
No error when inserting characters via an IME, an emoji, and then more characters, and you can backspace everything. Here is a fiddle where I use Draft.js v0.10.5 instead and this issue is no longer present. This is what it looks like:

Which versions of Draft.js, and which browser / OS are affected by this issue? Did this work in previous versions of Draft.js?
Draft.js version 0.11.0
Chrome v76.0.3809.132, Firefox v69.0.1, and Safari v12.1.2 on Mac OS 10.14.6
This previously worked in Draft.js v0.10.5. The PR that changed this is #2035. Reverting the changes made to resolveComposition in DraftEditorCompositionHandler.js in that patch fixes this issue.

Hi @jdecked, I tried your scenarios in my fork of Draft, and they both work correctly. I suspect that this commit is the one that fixes it. Would you have time to give it a try and see if it solves your issue? If so, feel free to "borrow" the commit to open a pull request.

Thanks for reporting this issue @jdecked, the repro steps are very well written and helped me understand the problem!
Thanks for chiming in @robbertbrak, I was able to verify that the suggested check does indeed prevent the error reported by Justine. While I'm not fond of silently ignoring errors, recovering from the error returning the previous selection seems safe enough to unbreak the scenario of this issue.
I'll prepare a PR shortly


Search suggestion for Ahk & Ahp

need kodadot/stick#30 first

Closing because #7976
and  - https://github.com/kodadot/stick/issues/141


Add note about where e2e tests currently located

Adds a note about where e2e tests are currently located in the
zalando-incubator/kubernetes-on-aws repository and fixes up a broken
link in README.

/lgtm

/approve


fix issue SCREENSHOT is not defined when running test

Hello, and thanks for contributing to Appium Driver Codeception!
Description
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes # (issue)
Type of change
Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update
[ ] Fixing typos

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[ ] Test A
[ ] Test B

Test Configuration:

PHP Version:
Appium Version:
Codeception Version:

Checklist:

[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] Any dependent changes have been merged and published in downstream modules

Thanks


Checkbox handling

Improved handling of checkbox and radio states

apply selected class when selected(!)
ensure that choices are displayed when page loads

Go!


Build error sample test related to pedantic

I'm getting an error when Mimick is build:
$ g last -1
commit b9a277dbfe149bda0c50472f862a1394c6349cc1
Author: Snaipe<EMAIL_ADDRESS>Date:   Sun May 1 00:02:08 2016 +0200

    readme: Added coverity badge [ci skip]

$ mkdir build && cd build && cmake ../
-- The C compiler identification is GNU 6.1.1
-- Check for working C compiler: /usr/lib/ccache/bin/cc
-- Check for working C compiler: /usr/lib/ccache/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- The ASM compiler identification is GNU
-- Found assembler: /usr/lib/ccache/bin/cc
-- Looking for _r_debug
-- Looking for _r_debug - found
-- Looking for _DYNAMIC
-- Looking for _DYNAMIC - found
-- Performing Test HAVE_ELF_AUXV_T
-- Performing Test HAVE_ELF_AUXV_T - Success
-- Performing Test HAVE_ELF_AUXINFO
-- Performing Test HAVE_ELF_AUXINFO - Failed
-- Looking for mmap
-- Looking for mmap - found
-- Looking for MAP_ANONYMOUS
-- Looking for MAP_ANONYMOUS - found
-- Looking for MAP_ANON
-- Looking for MAP_ANON - found
-- Configuring done
-- Generating done
-- Build files have been written to: /home/tiamarin/repos/Mimick/build

$ make
Scanning dependencies of target mimick
[  5%] Building ASM object CMakeFiles/mimick.dir/src/asm/trampoline-x86_64-systemv.S.o
[ 10%] Building C object CMakeFiles/mimick.dir/src/core.c.o
[ 15%] Building C object CMakeFiles/mimick.dir/src/matcher.c.o
[ 20%] Building C object CMakeFiles/mimick.dir/src/mock.c.o
[ 25%] Building C object CMakeFiles/mimick.dir/src/plt.c.o
[ 30%] Building C object CMakeFiles/mimick.dir/src/plt-elf.c.o
[ 35%] Building C object CMakeFiles/mimick.dir/src/stub.c.o
[ 40%] Building C object CMakeFiles/mimick.dir/src/trampoline.c.o
[ 45%] Building C object CMakeFiles/mimick.dir/src/verify.c.o
[ 50%] Building C object CMakeFiles/mimick.dir/src/vitals.c.o
[ 55%] Building C object CMakeFiles/mimick.dir/src/when.c.o
[ 60%] Linking C static library libmimick.a
[ 60%] Built target mimick
Scanning dependencies of target strdup
[ 65%] Building C object sample/strdup/CMakeFiles/strdup.dir/strdup.c.o
[ 70%] Linking C shared library libstrdup.so
[ 70%] Built target strdup
Scanning dependencies of target strdup_test
[ 75%] Building C object sample/strdup/CMakeFiles/strdup_test.dir/test.c.o
In file included from /home/tiamarin/repos/Mimick/include/mimick/mock.h:32:0,
                 from /home/tiamarin/repos/Mimick/include/mimick.h:389,
                 from /home/tiamarin/repos/Mimick/sample/strdup/test.c:1:
/home/tiamarin/repos/Mimick/include/mimick/matcher.h:45:22: error: enumerator value for ‘MMK_MATCHER_EQ’ is not an integer constant expression [-Werror=pedantic]
     MMK_MATCHER_EQ  = MMK_MATCHER_BIT_CMP | MMK_MATCHER_BIT_EQ,
                      ^
/home/tiamarin/repos/Mimick/include/mimick/matcher.h:46:23: error: enumerator value for ‘MMK_MATCHER_NEQ’ is not an integer constant expression [-Werror=pedantic]
     MMK_MATCHER_NEQ = MMK_MATCHER_BIT_CMP | MMK_MATCHER_BIT_LT
                       ^
/home/tiamarin/repos/Mimick/include/mimick/matcher.h:48:22: error: enumerator value for ‘MMK_MATCHER_LT’ is not an integer constant expression [-Werror=pedantic]
     MMK_MATCHER_LT  = MMK_MATCHER_BIT_CMP | MMK_MATCHER_BIT_LT,
                      ^
/home/tiamarin/repos/Mimick/include/mimick/matcher.h:49:23: error: enumerator value for ‘MMK_MATCHER_LEQ’ is not an integer constant expression [-Werror=pedantic]
     MMK_MATCHER_LEQ = MMK_MATCHER_BIT_CMP | MMK_MATCHER_BIT_EQ
                       ^
/home/tiamarin/repos/Mimick/include/mimick/matcher.h:51:22: error: enumerator value for ‘MMK_MATCHER_GT’ is not an integer constant expression [-Werror=pedantic]
     MMK_MATCHER_GT  = MMK_MATCHER_BIT_CMP | MMK_MATCHER_BIT_GT,
                      ^
/home/tiamarin/repos/Mimick/include/mimick/matcher.h:52:23: error: enumerator value for ‘MMK_MATCHER_GEQ’ is not an integer constant expression [-Werror=pedantic]
     MMK_MATCHER_GEQ = MMK_MATCHER_BIT_CMP | MMK_MATCHER_BIT_EQ
                       ^
cc1: all warnings being treated as errors
sample/strdup/CMakeFiles/strdup_test.dir/build.make:62: recipe for target 'sample/strdup/CMakeFiles/strdup_test.dir/test.c.o' failed
make[2]: *** [sample/strdup/CMakeFiles/strdup_test.dir/test.c.o] Error 1
CMakeFiles/Makefile2:325: recipe for target 'sample/strdup/CMakeFiles/strdup_test.dir/all' failed
make[1]: *** [sample/strdup/CMakeFiles/strdup_test.dir/all] Error 2
Makefile:94: recipe for target 'all' failed
make: *** [all] Error 2

If a remove -pedantic from sample/CMakeLists.txt and test/CMakeLists.txt files. Build and tests pass:
$ make test
Running tests...
Test project /home/tiamarin/repos/Mimick/build
    Start 1: mmk_test
1/2 Test #1: mmk_test .........................   Passed    0.00 sec
    Start 2: strdup_test
2/2 Test #2: strdup_test ......................   Passed    0.00 sec

100% tests passed, 0 tests failed out of 2

Total Test time (real) =   0.00 sec

OS: Arch Linux
Compiler: gcc 6.1

Fixed -- this was caused by the value of MMK_MATCHER_BIT_CMP setting the sign bit in signed integers, which, as a positive value, is technically invalid for an enum value.


Explorer: drag and drop of files and folders to other window explorer only copies files

I think I would expect the same behaviour when I drag and drop files and folders from the finder where I am asked if the folders are added to the workspace or copied into.
We might not be able to do this given Electron, but I still wonder why files copy while folders do not.
Steps:

open 2 windows
drag and drop a selection of files and folders to the other window explorer
only files get copied in

The issue here is that we simply do not allow DnD of folders between VS Code windows currently. So files have no influence here. Simply take one folder in Win A and drag it over to Win B and it will not work.
The issue is that we currently disable it here
And the original drag event does not contain the data needed to do the folder copy. Thus extractResources here does not handle it.
Maybe we could change the dragged data such that it just contains the uri of dragged folders and when they get dropped we simply do fileService.copy?

@isidorn yeah I think this code maybe originates from a time where we did not ask the user to copy or add the folder, so it was potentially a dangerous operation with lots of files getting copied. Now that we have this check I would think we can remove it and get the dialog, no?
Given it already works with files, I would expect it would just work for folders too?

@bpasero I removed the check and it did not just work. Though assigning to August to investigate more.

Moving to debt week.

Pushed a fix for this, it should now work nicely.
Bonus: I handle the explorer drop as a potential adding of a workspace root only if it dropped top level. Which I think makes this flow nicer. Anyways try it out and let me know how it goes


Create optionally-available __int128 typedefs and use them for ARM64 definitions.

Potentially fixes #2524, see the comments in the patch for details.

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @Amanieu (or someone else) soon.
Please see the contribution instructions for more information.

See https://github.com/rust-lang/rust/issues/54341#issuecomment-1062498329 for details on my investigation into this.
I will happily fill in more OS/arches if anyone can run https://github.com/Gankra/abi-checker on that target and get full passes. I only have access to aarch64 Android (phone) and macOS (coworker). I'm kinda blindly assuming iOS is the same as macOS here.

For context, properly defining these types is important for saving a crashing process' registers (i.e. when generating a breakpad-style minidump, which I am currently working on porting to Rust for Android).

In the upstream issue nagisa notes that rust uniformly has this repr on all aarch64 targets -- but I'd still like to verify that the primary C compilers for those targets agree.

So since putting up this PR I spent a bit more time digging into what can be done to polyfill this type on x64 (since SysV actually passes structs the same as ints, in general). Unfortunately on x64 (linux, at least) clang and gcc actually disagree on the pass-on-stack convention for __int128:
https://twitter.com/Gankra_/status/1502320041315946499
x64 SysV fails to specify what "pushing" a value actually means and so clang packs __int128 while gcc properly aligns it. This can introduce padding and differences and completely desynchronize the offsets to args.
In discussing this on Twitter some Apple engineers also pointed out that clang actually has macos-specific ABI differences for backcompat with itself (details vague, no documentation for this in general). Similarly a coworker noted that stuff like this is a problem for clang plugins when the plugin and compiler weren't both consistently built by clang or gcc.
This now has me increasingly paranoid about claiming u128/i128 are ABI-compatible on ARM64 without any way for me to properly compare gcc and clang both targetting these platforms. And also makes me less inclined to willy-nilly declare that Rust agrees with C compilers for "all non-windows OSes". Even asserting they agree on size and align isn't sufficient because the compilers can still disagree on how the value is pushed to the stack. :(
I'm not sure what libc's policy is on divergent impls for one platform, and how to declare one "canonical" over the others.
Unfortunately I can't really test this stuff out better -- the "gcc" on my phone's termux is just an alias for clang, and similarly I expect that will be the case for m1 macbooks.

If anyone has better access to arm64 platforms and can setup clang/gcc properly, they can test this out by checking out:
https://github.com/Gankra/abi-checker
and running
cargo run -- --tests ui128

This will compare CC to rustc and CC to itself. That is a good start for rust matching the "canonical" implementation.
To exhaustively try out all the combos of clang+gcc+rust to check if the ABI is actually consistent between all impls, you can do:
cargo run -- --tests ui128 --pairs clang_calls_gcc gcc_calls_clang rustc_calls_gcc gcc_calls_rustc rustc_calls_clang clang_calls_rustc clang_calls_clang gcc_calls_gcc rustc_calls_rustc

Otherwise you can manually do something like this godbolt.

I ran your ABI checker on AArch64 Linux, all the tests pass:
Final Results:
ui128::c::clang_calls_gcc        all 226/226 passed
ui128::c::gcc_calls_clang        all 226/226 passed
ui128::c::rustc_calls_gcc        all 226/226 passed
ui128::c::gcc_calls_rustc        all 226/226 passed
ui128::c::rustc_calls_clang      all 226/226 passed
ui128::c::clang_calls_rustc      all 226/226 passed
ui128::c::clang_calls_clang      all 226/226 passed
ui128::c::gcc_calls_gcc          all 226/226 passed
ui128::c::rustc_calls_rustc      all 226/226 passed
ui128::cdecl::rustc_calls_rustc  all 226/226 passed
ui128::stdcall::rustc_calls_rustc all 226/226 passed
ui128::fastcall::rustc_calls_rustc all 226/226 passed

2712 passed, 0 failed, 0 completely failed, 33 skipped

Oh nice!! that helps my confidence with going forward with this a lot. I'll clean this up on monday.

Also lmao @ rustc actually managing to do something on linux when you ask it to fastcall/stdcall with itself, I hadn't actually every tested that.

Applied review requests and squashed.

@bors r+

:pushpin: Commit 3741d52dcba0d782cd27b1680e9d3dd481f33470 has been approved by Amanieu

:hourglass: Testing commit 3741d52dcba0d782cd27b1680e9d3dd481f33470 with merge ca04d0981e9f44b4cad00b1a21850852895f2ec0...

:broken_heart: Test failed - checks-actions

:pushpin: Commit c0d7485876f65785df4877b1db74e701dfa8a879 has been approved by Amanieu

:hourglass: Testing commit c0d7485876f65785df4877b1db74e701dfa8a879 with merge cda106ab42245f3936cdbde322785c2af2d74391...

:broken_heart: Test failed - checks-actions

error: constant is never used: `SIZE_128`
  --> src/fixed_width_ints.rs:65:9
   |
65 |         const SIZE_128: usize = 16;
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
note: the lint level is defined here
  --> src/lib.rs:18:38
   |
18 | #![cfg_attr(libc_deny_warnings, deny(warnings))]
   |                                      ^^^^^^^^
   = note: `#[deny(dead_code)]` implied by `#[deny(warnings)]`

error: constant is never used: `ALIGN_128`
  --> src/fixed_width_ints.rs:66:9
   |
66 |         const ALIGN_128: usize = 16;
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: could not compile `libc` due to 2 previous errors

??????? how ???????

Maybe const _ASSERT_128_COMPAT: () = { ... } doesn't mark them as used because it isn't used itself? Would const _: () = { ... } work?

It looks like prefixing them with _'s is the best approach.
pushed fix

:pushpin: Commit 80b951d67ab995da61b18588ba37d56dd3351f6e has been approved by Amanieu

:hourglass: Testing commit 80b951d67ab995da61b18588ba37d56dd3351f6e with merge c048d5be95f68c1b5f328396f566b8c2621a92b4...

uhhh looks like the last remaining issue is... rust 1.13 support(????)
not sure what to do about that...

proc_macro::is_available() was stabilized on rustc 1.57 (https://github.com/rust-lang/rust/commit/09dd213cd2253015b80605e374d624fcf3c9aea0). CI uses a nightly from a couple of weeks before the moment it was stabilized, but because it is still 1.57, proc_macro2 thinks it was stabilized and tries to use it. 2021-10-11 is the first nightly with is_available() stabilized I think. You can either upgrade to it or pin proc_macro2 to 1.0.35 which doesn't use is_available() yet.

For the 1.13 issue you could conditionally compile it. The build script already detects a couple of rustc versions.

@bors r+

:pushpin: Commit 957e97b91d2f8e572853b49ccc8b246a874313eb has been approved by Amanieu

I think I added the cfgs?
Since these defs are conditional I added a fallback for the old macos definition. That seems like the right thing to do?

:hourglass: Testing commit 957e97b91d2f8e572853b49ccc8b246a874313eb with merge 7bf5e77e2a66a24b50a30122c3361a66caf30f89...

i128 was added in Rust 1.26, you might want to add a check for that in build.rs.
For the unreachable in garando_syntax I have no idea. cc @JohnTitor
thread 'main' panicked at 'internal error: entered unreachable code', /cargo/registry/src/github.com-1ecc6299db9ec823/garando_syntax-0.1.0/src/ext/expand.rs:337:21

I don't have enough time to investigate, but the code is quite old and I suspect that a newer thing here triggers that panic.
FWIW here's the repo: https://github.com/JohnTitor/garando

:broken_heart: Test failed - checks-actions

sorry i am unfortunately stuck in the situation of compile-by-ci here because I don't have direct access to these platforms on my dev machine.

macos freaked out from the way i cfg'd, went with cfg_if with the s!'s being internal
musl aarch64 complained it didn't have fpsimd; added it since it's a kernel API so it should be portable?

arghh nope macros bad style now 😭

Is there a "right" way to express what I'm doing with macos here? Should I just drop updating the macos def? Or not make it backcompat?

I would just drop the type on old rustc which don't support i128. The type definition is incorrect in any case.

@bors delegate=Gankra

:v: @Gankra can now approve this pull request

@bors try

@bors try

:hourglass: Trying commit 49400c79f7071c1cfa809684f2b9af27f9b83b4a with merge ab286d503d1b2a79db3cfba496bfa1881eaea521...

:broken_heart: Test failed - checks-actions

You can ignore the failure on sparc, it's spurious, just retry the build.

@bors try

:hourglass: Trying commit e07de0ee9e264b262c90ead5e049ff05d13d8ab2 with merge 3e9c0c0a3a53e139935c6c143731293fd85080fb...

const asserts only exist as of 1.57 so I added another cfg for that

had to make a bunch more types optional behind int128 support in macos since the type is transitively used by several other ones.

:broken_heart: Test failed - checks-actions

@bors try

:hourglass: Trying commit 127542f58cd2e2e370062a5ff434c7c8d1749d68 with merge 1577ee48b59b1c08ebdfa1787a7f172da6891219...

trying to instead cfg the fields and keep the struct around, since the fallout from removing it seems kinda nasty

ok!!! this one actually seems to work and satisfy the nightmare cfg macro rube-goldberg machine (remaining failures are the pre-existing ones in the repo itself).

:broken_heart: Test failed - checks-actions

@bors r+

:pushpin: Commit 127542f58cd2e2e370062a5ff434c7c8d1749d68 has been approved by Gankra

:hourglass: Testing commit 127542f58cd2e2e370062a5ff434c7c8d1749d68 with merge aeb1a4e0b8b793db36065e40d0bab3118b6532b1...

:broken_heart: Test failed - checks-actions

ok coming back to this with a fresh mind and the design of libc i think now has congealed into a coherent form for me. Hopefully this works now...

@bors try

:hourglass: Trying commit adfa38507c1aa4adce7010a58825af35a1f5fb99 with merge 245c710f19b7fa29a1cad6bf9a3afc5000b69de0...

@bors r+

:pushpin: Commit adfa38507c1aa4adce7010a58825af35a1f5fb99 has been approved by Gankra

:hourglass: Testing commit adfa38507c1aa4adce7010a58825af35a1f5fb99 with merge f00ea005769ba76610002b3652a5c6272ddc3633...

:sunny: Test successful - checks-actions, checks-cirrus-freebsd-11, checks-cirrus-freebsd-12, checks-cirrus-freebsd-13
Approved by: Gankra
Pushing f00ea005769ba76610002b3652a5c6272ddc3633 to master...


Ag en network copy

Why:
update network copy to be a broader message to check the user has internet

Why:
update network copy to be a broader message to check the user has internet

Looks good @artgibson


ag searches through "tags" files by default

I would love to use ag, but I don't think it should search through my tags file by default.

For anyone looking at this issue in the future, if you're trying to exclude the file tags (from ctags) from search, but to include tags/ as a directory in your searches
(does not work, tags/ directory is still ignored)
tags
!tags/

But this workaround does work - the file is ignored but the files in the directory aren't.
[t]ags
!tags/


Dexie not working with TypeScript

Hi!
I am trying to use Dexie for a web application, which uses TypeScript for backend and JS for front end. I installed it using "npm install dexie" and now I have the 'dexie' module inside node_modules folder.
One of the simplest versions I tried (from the Internet) so far to create a database and open it is like:
   export class DbConnection extends Dexie {
    constructor() {
    super("MyAppFirstDb");

    this.version(1).stores({
        friends: 'name, age'
    });
}

}
and then try to use this class as:
    var db:DbConnection = new DbConnection();
    db.table("friends").add({name: "Josephine", age: 21}).then(()=>{
        return db.table("friends").where("age").below(25).toArray();
    }).then(youngFriends => {
        console.log("My young friends: " + JSON.stringify(youngFriends));
    }).catch(e => {
        console.log("error: " + e.stack || e);
    });

but every time I get this error:
MissingAPIError: indexedDB API not found. If using IE10+, make sure to run your code on a server URL (not locally). If using old Safari versions, make sure to include indexedDB polyfill.
Can you please help me?

That error usually happens when not running in a web browser, such as in nodejs for example.

Dexie is a wrapper for indexedDB but indexedDB is a part of the W3C Dom, so you need to run Dexie in an environment that supports indexedDB.

So, is there a way I can use Dexie with TypeScript, in this situation?

So, is there a way I can use Dexie with TypeScript, in this situation?

This is not a Typescript issue. Your code compiles, right? But you get an exception when you run it.
Are you running this on Node or in  a browser?

You are right, the title may not be the proper one, I'm sorry. The code compiles, indeed. It is a Node project, but I'm using express, in order to run it in browser.

Do you think it is a Node related issue, or?

@leoc5 to use Dexie.js in Node.js environment you need a shim for IndexedDB. There are a few available like fake-indexeddb or IndexedDBShim. With fake-indexeddb it could look somewhat like:
import Dexie from 'dexie';
import IDBKeyRange from 'fake-indexeddb/lib/FDBKeyRange';
import indexedDB from 'fake-indexeddb';

class DexieForNode extends Dexie {
  constructor(name) {
    super(name, {
      indexedDB,
      IDBKeyRange,
    });
  }
}

This will keep all the data in memory only but there are options to have a shim which persists data in sqlite on disk for example.


imageData.data from a bitmapData does not contain pixel values

This issue comes from a post on the forum.
I would like to apply a filter on a picture with a bitmapData as a proxy to apply pixel transformations.
The problem comes from the array imageData.data of a bitmapData which is filled with zeros instead of being filled with correct values.
Here is a piece of code to illustrate my issue:
var originalPic = game.add.sprite(0, 0, 'originalPic');
var bmd = game.add.bitmapData(originalPic.width, originalPic.height);
bmd.draw(originalPic);

console.log(bmd.imageData.data); // ==> [0, 0, 0, 0, ...]

var newPic = game.add.image(600, 0, bmd); // Display on screen is okay

Obviously, if I cannot access those data, I would not be able to make any computation on pixels. :/
I came up with a solution by using the context attribute of a bitmapData, instead of using directly the imageData attribute.
var originalPic = game.add.sprite(0, 0, 'originalPic');
var bmd = game.add.bitmapData(originalPic.width, originalPic.height);
bmd.draw(originalPic);
var imageData = bmd.context.getImageData(0, 0, originalPic.width, originalPic.height);

console.log(imageData.data); // ==> now contains correct values
// Here I can call my image processing function
// ...

var newPic = game.add.image(600, 0, bmd); // Display the processed image

So, BitmapData.imageData is quite buggy. I am going to report an issue about this on GitHub.
Thank you for having read me.

If you don't call BitmapData.update after manipulating it, it won't update the imageData property. This isn't done automatically because it's expensive, so you have to call it yourself.

It works. Thank you for your answer.
The call to BitmapData.update() should be mentionned in the documentation so, under the imageData entry.
This would have saved me couple of hours understanding where was the problem. ^_^


ctrl + alt + t shortcut conflicts with terminal on Linux

On most Linux systems ctr + alt + t is set as the default shortcut for opening the terminal, this prevents the default shortcut for toggling the diff view from working.
I suggest changing it to something else.

As all split-diff commands are ctr + alt based, how about ctr + alt + d, as for diff?
In the past this combination was used on some systems to show the desktop, but I think not anymore. Definitely better than the very common ctr + alt + t.


Backing up linked folders not in account

It may be useful to some to back up contents of folders the user has links to in their inventory. And possibly of folders linked in there too.
It would be really cool to have a maximum recursion depth setting one could put in to back up these folders too, with a warning that this can easily balloon in size.

okay So what would you do if a folder linked to itself?
This would result in it just grabbing itself infinitely

I'm not the dev of this program, but I imagine it shouldn't be hard to keep track of which folders were already visited and store just the link itself if it points to one such folder?
That, or at least rely on the maximum recursion depth setting provided there. Neos assets are uniquely identified as far as I understand, so a few recursions into the same folder shouldn't be too much of a problem besides resulting in a bunch of repeated metadata.

This won't be supported sorry.


Arduino and ESP32 demo: fix typos and improve Pico kit wiring picture

Fixed  @ashvayka and @vkukhtyn comments.


Error rendering code search page

Description
Hello,
Not sure when did this start, but I'm getting the following error when using code search:
An error occurred:

Render failed, failed to render template: explore/code, error: template error: builtin(bindata):code/searchresults:34:83 : executing "code/searchresults" at <Safe>: wrong type for value; expected string; got template.HTML
----------------------------------------------------------------------
								{{.FormattedLines | Safe}}
								                                                                        ^
----------------------------------------------------------------------

Looks like the search does work but rendering the results fails:

I'm using bleve indexer with the following settings:
[indexer]
REPO_INDEXER_ENABLED = true
REPO_INDEXER_PATH = indexers/repos.bleve
MAX_FILE_SIZE = 1048576
REPO_INDEXER_INCLUDE =
REPO_INDEXER_EXCLUDE = resources/bin/**

The above config worked for me until I upgraded to one of the recent version (not sure which update, I haven't used code search for a bit).
I will appreciate any help - thanks very much!
Gitea Version
1.21.3
Can you reproduce the bug on the Gitea demo site?
No
Log Gist
No response
Screenshots
No response
Git Version
No response
Operating System
ubuntu
How are you running Gitea?
Running gitea binary as a systemd service on Ubuntu 22.04
Database
MySQL/MariaDB

duplicated with #28575 ?

Please use 1.21 nightly build.
->  Fix 500 error of searching commits (#28576) #28579


Ensure telemetry pipe is openable before starting loop to read it.

Attempts to avoid a race condition between creating the FIFO and the name appearing in the file system so it can be opened.

lgtm


build dev 1.10 pr12

Partial implementation of build for core-test

*Issue #1888 *
Description of changes:
Changes to consume CRT form its own FindPackage construction available since 10.18.1, this BREAKs the build, need to rebase our branch to new version of SDK. I will do that in follow up PR.
Check all that applies:

[x] Did a review by yourself.
[ ] Added proper tests to cover this PR. (If tests are not applicable, explain.)
[ ] Checked if this PR is a breaking (APIs have been changed) change.
[ ] Checked if this PR will not introduce cross-platform inconsistent behavior.
[ ] Checked if this PR would require a ReadMe/Wiki update.

Check which platforms you have built SDK on to verify the correctness of this PR.

[ ] Linux
[ ] Windows
[ ] Android
[ ] MacOS
[ ] IOS
[ ] Other Platforms

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

please fix cspell and looks good
/home/runner/work/aws-sdk-cpp/aws-sdk-cpp/aws-sdk-cpp/CMakeLists.txt:46:22 - Unknown word (DNON)
/home/runner/work/aws-sdk-cpp/aws-sdk-cpp/aws-sdk-cpp/CMakeLists.txt:88:20 - Unknown word (COMMONCRYPTO)


document the duck array integration status

With the initial (experimental) release of pint-xarray getting closer (see xarray-contrib/pint-xarray#25), I figured it might make sense to document the parts of the API of xarray which right now won't work with duck arrays, and to start maintaining a list of extensions like pint-xarray.
The remaining issues are taken from #3594, but I'm sure I missed something (e.g. plotting).

[x] Passes isort . && black . && mypy . && flake8
[x] User visible changes (including notable bug fixes) are documented in whats-new.rst

I'm not sure "plotting" should be included in that list since converting to numpy is intentional.
Other than that, this should be ready for review.

I just noticed that we have to wrap chunk calls with a dequantify / quantify cycle if we want to have it work correctly, but maybe that's something that belongs into the pint-xarray docs?

if there are no further comments, should we merge this? I'm almost certain the list of missing features is not complete, but I would open other PRs once new items are found.


List models feel clunky

Things such as AgentList, AgentGroupList, etc. make working with that data feel clunky.
For example, to get a list of agents, you need to run:
agents = client.agent_api.list().agents

Instead if the response simply returned a list of Agent objects instead of a meta-list object:
agents = client.agent_api.list()

This feels more intuitive and feels like the expected behavior.

Thanks for the feedback, as we've spoken offline. This is designed to stay truthful to the API actually response, which is a list wrapped in an object. It's possible that we can unwrap the list by itself, however, I'd need confirmation from the API team that there will never be any metadata be sent in the root object. Otherwise we will have to revert back to this thus create backward compatibility issue.
Feel free to reopen this if there is any new development on this topic.

After this is validated by Andrew Scott, let's please make returning as a list happen

@paragbaxi @tenableAndrew I am happy to do that, but please consider the following points that makes me prefer the current implementation.

If pagination ever comes around, would it be wrapped in an object with metadata (page size, page, total... etc)? That's usually the implementation and in that case we would be reverting back to this.

By design, I have been making these API wrappers to stay truthful to the API. Meaning to minimize reinterpretations of the request and response. "Feeling clunky" is a UX concern that is the priority of the helpers but not at the API level. Making this change would prioritize UX over staying true to the API.

These might not be moot points but I want to make sure these have a chance to be considered before moving forward. Let me know, thanks!

Good points. Let's stay true to API until these questions are answered
On Thu, Apr 20, 2017 at 12:48 PM alanfung<EMAIL_ADDRESS>wrote:

@paragbaxi https://github.com/paragbaxi @tenableAndrew
https://github.com/tenableAndrew I am happy to do that, but please
consider the following points that makes me prefer the current
implementation.

If pagination ever comes around, would it be wrapped in an object with
metadata (page size, page, total... etc)? That's usually the implementation
and in that case we would be reverting back to this.
2.
By design, I have been making these API wrappers to stay truthful to
the API. Meaning to minimize reinterpretations of the request and response.
"Feeling clunky" is a UX concern that is the priority of the helpers but
not at the API level. Making this change would prioritize UX over staying
true to the API.
These might not be moot points but I want to make sure these have a chance
to be considered before moving forward. Let me know, thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tenable/Tenable.io-SDK-for-Python/issues/22#issuecomment-295878341,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA6R8UXXj09vuRJCwl-qX2TQkNhKtI61ks5rx7aDgaJpZM4M3lrW
.

--
Parag


More info about the LCD i80_controller example (IDFGH-6902)

Hello,
I'm trying to figure out whether or not an ESP32 can be used as an LCD driver. I'm looking for more info about this example:
LVGL porting example (based on i80 interfaced LCD controller)
What is the FPS you were able to achieve when using the 8080 interface? It looks like only 10 FPS? And what size screen were you using? If you increase the PSRAM frequency, are you able to get anywhere near 60 FPS? What about if you just use the internal SRAM, rather than the PSRAM?

Hi @ZaneL As you didn't mention which branch you're working on, let's take the master branch as an example.

ESP32 can generate the 8080 timing by I2S peripheral, the esp_lcd component will take care of it, and provide a general user API, naming i80 bus and i80 io device driver. You can find the driver in esp_lcd_panel_io_i2s.c
ESP32's I2S DMA can't use PSRAM, so you can see EXAMPLE_LCD_I80_COLOR_IN_PSRAM is only visible for ESP32S3, where the LCD DMA can fetch data from PSRAM
In the example, we locked down the FPS by a little for ESP32S3, by this Kconfig CONFIG_LV_DISP_DEF_REFR_PERIOD. This is because the example is default to enable the PSRAM if the target is select to ESP32S3. (you can check the sdkconfig.defaults.esp32s3 file). And 10FPS should work on most of PSRAM frequency.
For higher FPS, you can either i) adjust the PSRAM mode (like frequency, mode) ii) Using internal SRAM
This example uses a 240*280 LCD, because the author only has that LCD at hand, no saying the resolution limitation is that.

I tried to run it on an ESP32_DevkitC_V4 with 8M Flash ESP32-Wrover-E and 8M PSRAM. The module was not connected to any thing. With the example from espidf release/5.0 example from i80_example, the target was set to esp32, flash/ram were set accordingly based on the module used, GPIO pins were set based what are available from the module. The build was success, but the code crashed during heap allocation occurred in esp_lcd_new_panel_io_i80 from file components/esp_lcd/src/esp_lcd_panel_io_i2s.c. See below


STORM-2705: DRPCSpout sleeps twice

When the DRPCSpout cannot fetch a request it puts itself to sleep at the end of nextTuple(). After leaving this method the case that the spout did not emit any tuple is handled by ISpoutWaitStrategy - which in case of SleepSpoutWaitStrategy (the default) issues another sleep.
This causes unnecessary context switches in my opinion, especially with a high parallelism setting and many different DRPCSpouts.
Only the configured / active ISpoutWaitStrategy should decide the action.

+1 to master branch

Ok, I will update the PR so it can be applied to master easily.

+1 again.


fix(nuxt): use routeRules to hint pages to prerender

🔗 Linked issue
https://github.com/nuxt/nuxt/issues/22661#issuecomment-2376237042
📚 Description
This addresses an overlooked area - when a user is building a site and wants to prerender some routes but has just specified the route rule with a wildcard and not added prerender.routes.
This won't cover every area (such as routes changed with app/router.options.ts) but it should be safe as it requires an overlap between pages/ and routeRules.
If you have a pages/bing/foo.vue, then the following config will ensure it is prerendered when running nuxi build:
export default defineNuxtConfig({
  routeRules: {
    '/bing/**': { prerender: true },
  },
})

However, note that this still won't help with non-deterministic pages, so we still can't know the paths to prerender if you have pages/bing/[slug].vue - you'll still need to provide a starting point in this case.
🚧 TODO

[x] add a similar route-rule-aware approach to the runtime hinting.

However, note that this still won't help with non-deterministic pages, so we still can't know the paths to prerender if you have pages/bing/[slug].vue - you'll still need to provide a starting point in this case.

Good point. When we use nitro.prerender.crawlLinks: true it does the following (from the docs):
1. Load the HTML of your application's root route (/), any non-dynamic pages in your ~/pages directory, and any other routes in the nitro.prerender.routes array.
2. Save the HTML and payload.json to the ~/.output/public/ directory to be served statically.
3.Find all anchor tags (<a href="...">) in the HTML to navigate to other routes.
4. Repeat steps 1-3 for each anchor tag found until there are no more anchor tags to crawl.

Is there the capability for something like '/bing/**': { prerender: true }, to use the same workings as crawlLinks so it would run through these steps but just for that route so it would pick up on the non-deterministic pages?
Ultimately if I keep nitro.prerender.crawlLinks set to true, which is fine, but any routeRules set to make one of those routes ssr or swr then it shouldn't prerender those overrides, right?

I have a query about whether this fix impacts the following:
nitro: {
	prerender: {
		crawlLinks: true
	}
},

routeRules: {

	'/making-public/reading-room-library': { ssr: true },

	// Exclude API routes from prerendering
	'/api/**': { prerender: false },
	// API routes that fetch Are.na content use SWR (cached for 1 hour)
	'/api/arena/**': { swr: 3600 }, // 3600 seconds = 1 hour

	// Redirects
	'/program/labs': {
		redirect: {
			to: '/program',
			statusCode: 301
		},
	},
	'/community/profiles': {
		redirect: {
			to: '/community',
			statusCode: 301
		},
	},

},

When I run npm run build it prerenders the '/making-public/reading-room-library' route rather than skipping and running it as ssr. Any thoughts?

ssr doesn't mean not to pre-render. it means that when prerendered, it will have the full html of the page present, not an empty div

Thanks, @danielroe
Can I confirm that if I need a route to always be up to date (always fetching "live" data) what's the best route rule to use? prerender: false?
I think I'm slightly confused because if you were to run nuxi build without any route rules then you'd have no static prerendered files, right? That's ssr out of the box. So when setting ssr: true I'd assume it would yield the same results and therefore on said route it wouldn't produce a .html file. Am I missing something? Am I mad? 😂
The /server/api route that I'm fetching would be set to swr: 3600 so it's cached every hour(ish)

If we set nitro.prerender.crawlLinks: true then setting a route (via routeRules) to prerender: false is essentially making the route server side rendered. Correct? However, you're suggesting there seems to be an "in between state" where setting ssr: true with crawlLinks: true on a route prerenders that route but then what's server side rendered?

ssr: true is default
prerender: false is default
If you set crawlLinks: true then it's like enabling prerender: true for every route.
SSR refers to whether full HTML is rendered by the server, at whatever time that render happens. It's not related to whether that render happens at build time or per-request.

Thanks. That totally clears things up. Now I also see why /** didn't work without crawlLinks.
This is an example of what I ended up using with nitro.prerender.crawlLinks: true
routeRules: {

    // Exclude specific routes from pre-rendering (fresh data on each request)
    '/events': { prerender: false },
    '/page-that-uses-api-for-example': { prerender: false },
    ...

    // Disable pre-rendering for all API routes
    '/api/**': { prerender: false },

    // Apply SWR caching (1 hour) for API routes
    '/api/some-api/**': { swr: 3600 },

    // Redirects
    ...

},


Android release build doesn't work with<EMAIL_ADDRESS>(AndroidX support is needed)

Steps to reproduce

Init new fresh project with latest<EMAIL_ADDRESS>Install latest react-native-share (with autolinking)
Run RELEASE Android build assembling

Expected behaviour
Release Android build is assembled.
Actual behaviour
Release Android build is failed.
Error
error: package android.support.v4.content does not exist [05:45:01]: ▸ import android.support.v4.content.FileProvider; [05:45:01]: ▸ ^ [05:45:01]: ▸ /Users/distiller/mobile-web-app/node_modules/react-native-share/android/src/main/java/cl/json/RNShareFileProvider.java:5: error: cannot find symbol [05:45:01]: ▸ public class RNShareFileProvider extends FileProvider { [05:45:01]: ▸ ^ [05:45:01]: ▸ symbol: class FileProvider [05:45:01]: ▸ /Users/distiller/mobile-web-app/node_modules/react-native-share/android/src/main/java/cl/json/RNShareModule.java:7: error: package android.support.annotation does not exist [05:45:01]: ▸ import android.support.annotation.Nullable; [05:45:01]: ▸ ^ [05:45:01]: ▸ /Users/distiller/mobile-web-app/node_modules/react-native-share/android/src/main/java/cl/json/RNSharePathUtil.java:12: error: package android.support.annotation does not exist [05:45:01]: ▸ import android.support.annotation.NonNull; [05:45:01]: ▸ ^ [05:45:01]: ▸ /Users/distiller/mobile-web-app/node_modules/react-native-share/android/src/main/java/cl/json/RNSharePathUtil.java:13: error: package android.support.v4.content does not exist [05:45:01]: ▸ import android.support.v4.content.CursorLoader; [05:45:01]: ▸ ^ [05:45:01]: ▸ /Users/distiller/mobile-web-app/node_modules/react-native-share/android/src/main/java/cl/json/RNSharePathUtil.java:14: error: package android.support.v4.content does not exist [05:45:01]: ▸ import android.support.v4.content.FileProvider; [05:45:01]: ▸ ^ [05:45:01]: ▸ /Users/distiller/mobile-web-app/node_modules/react-native-share/android/src/main/java/cl/json/RNShareModule.java:126: error: cannot find symbol [05:45:01]: ▸ public void open(ReadableMap options, @Nullable Callback failureCallback, @Nullable Callback successCallback) { [05:45:01]: ▸ ^ [05:45:01]: ▸ symbol:   class Nullable [05:45:01]: ▸ location: class RNShareModule [05:45:01]: ▸ /Users/distiller/mobile-web-app/node_modules/react-native-share/android/src/main/java/cl/json/RNShareModule.java:126: error: cannot find symbol [05:45:01]: ▸ public void open(ReadableMap options, @Nullable Callback failureCallback, @Nullable Callback successCallback) { [05:45:01]: ▸ ^ [05:45:01]: ▸ symbol:   class Nullable [05:45:01]: ▸ location: class RNShareModule [05:45:01]: ▸ /Users/distiller/mobile-web-app/node_modules/react-native-share/android/src/main/java/cl/json/RNShareModule.java:143: error: cannot find symbol [05:45:01]: ▸ public void shareSingle(ReadableMap options, @Nullable Callback failureCallback, @Nullable Callback successCallback) { [05:45:01]: ▸ ^ [05:45:01]: ▸ symbol:   class Nullable [05:45:01]: ▸ location: class RNShareModule [05:45:01]: ▸ /Users/distiller/mobile-web-app/node_modules/react-native-share/android/src/main/java/cl/json/RNShareModule.java:143: error: cannot find symbol [05:45:01]: ▸ public void shareSingle(ReadableMap options, @Nullable Callback failureCallback, @Nullable Callback successCallback) { [05:45:01]: ▸ ^ [05:45:01]: ▸ symbol:   class Nullable [05:45:01]: ▸ location: class RNShareModule [05:45:01]: ▸ /Users/distiller/mobile-web-app/node_modules/react-native-share/android/src/main/java/cl/json/RNShareModule.java:169: error: cannot find symbol [05:45:01]: ▸ public void isPackageInstalled(String packagename, @Nullable Callback failureCallback, @Nullable Callback successCallback) { [05:45:01]: ▸ ^ [05:45:01]: ▸ symbol:   class Nullable [05:45:01]: ▸ location: class RNShareModule [05:45:01]: ▸ /Users/distiller/mobile-web-app/node_modules/react-native-share/android/src/main/java/cl/json/RNShareModule.java:169: error: cannot find symbol [05:45:01]: ▸ public void isPackageInstalled(String packagename, @Nullable Callback failureCallback, @Nullable Callback successCallback) { [05:45:01]: ▸ ^ [05:45:01]: ▸ symbol:   class Nullable [05:45:01]: ▸ location: class RNShareModule [05:45:01]: ▸ /Users/distiller/mobile-web-app/node_modules/react-native-share/android/src/main/java/cl/json/RNShareModule.java:180: error: cannot find symbol [05:45:01]: ▸ public void isBase64File(String url, @Nullable Callback failureCallback, @Nullable Callback successCallback) {
Environment

React Native version: 0.60.3
React Native platform + platform version: iOS 12.2, Android 9.0 (any 6.0+)

react-native-share
Version: 1.2.1
Notes
Main breaking change in<EMAIL_ADDRESS>is AndroidX libraries support.
Upgrade of libraries is needed for react-native-share to avoid errors.
As example we can see same changes were merged in https://github.com/react-native-community/react-native-camera/pull/2306

I didn't have any problem using this plugin with ReactNative 0.60. Do you use the "jetifier" tool as advised here?

@guillaume-tgl Hello, I have problem only in release build mode. Dbug works fine. I have a lot of another react-native-... dependencies, project fully migrated to<EMAIL_ADDRESS>"jetifier" is enabled in Android gradle.properties by default now.
Do I need to do some custom steps to use react-native with jetifier?

I have only tried to run my app in debug. Sorry, I read your issue a bit too quickly, I din't see it was specifically in release mode 🙄

@guillaume-tgl No problems, 👍 yes, problem is present only in release mode

Not sure if any other errors pop up but to fix that
AndroidManifest.xml
<provider         android:name="android.support.v4.content.FileProvider"         ...
to
<provider         android:name="androidx.core.content.FileProvider"         ...
Check: https://developer.android.com/jetpack/androidx/migrate

@arivanandan Hello, could you please specify do we need to modify "AndroidManifest.xml" file in react-native-share project or in our own project?

You can also follow example: https://github.com/react-native-community/react-native-share/blob/master/example/android/app/src/main/AndroidManifest.xml

@guillaume-tgl @arivanandan @jgcmarins @heracek
Hello guys, I have fixed AndroidX issues in release build and added<EMAIL_ADDRESS>support.
Please check PR:
https://github.com/react-native-community/react-native-share/pull/550
I have checked production release builds on CI with my fork, everything is assembling correctly.
Thanks

Awesome

Just to say that the jetifier npm library should handle this for you with current code, but the linked PR is also correct as a migration - just that it is a breaking change (nothing wrong with that! just bump the version + add docs and go) and then people that are stuck on RN0.59 should be advised to use jetifier -r as a package.json postinstall. Cheers

@mikehardy I have checked previously, jetifier handle AndroidX Support fine only for Debug build.
In my production project I saw error in release build, so we need to migrate to androidx packages anyway.
For react-native0.59 people just can use<EMAIL_ADDRESS>it is working fine.
But to support react-native0.50 they need to use<EMAIL_ADDRESS>I reflected that in README in my PR.

The CI for jetifier checks release build specifically because that's where you see resource errors. Something is wrong, it should work.

@mikehardy If you check different react-native community PRs, you will find a lot of PRs to migrate to AndroidX libs, because Jetifier is temporary solution, it will be enabled for transition period only. We need to migrate anyway, like other react-native-... repos

@oleksandr-dziuban I am the jetifier author, I spent a month discussing it with RN-community in the related issue and testing solutions - I know this :-). What I'm getting at is that your temporary is a different length of time than others temporary. jetifier is supposed to work regardless. And libraries are supposed to migrate regardless
I am absolutely :+1: on the AndroidX pull request (and am on record with the same, on this pull request I think?) - I think it is great and thank you for doing it.
But jetifier should still work. If you can figure out exactly why it is not working, so we can fix it, that would be great.

Jetifier missed this :angry-face:
Up and running now though, thanks @oleksandr-dziuban

@jordangrant that si fascinating - the only instance of the string "CursorLoader" in jetifier is here: https://github.com/mikehardy/jetifier/blob/master/mapping/androidx-class-mapping.csv#L1403
It maps to androidx.loader.content.CursorLoader, which is apparently correct: https://developer.android.com/reference/androidx/loader/content/CursorLoader
If by "Jetifier missed this" you mean the javascript jetifier package that you run in postinstall in package.json did not transform correctly, I would love to see a reproduction that demonstrated this so we could fix it. The name of a react-native library I could use in the test suite perhaps?
This is maybe unrelated to this current PR (which I like! this library should migrate) but if jetifier has a probem it will cause pain for lots of our fellow developers, I'd love to know and make it work correctly.

@mikehardy thanks for authoring jetifier! I lost some context on this, but from what I recall: went through https://github.com/react-native-community/react-native-share/pull/550, and went through the diff. The only thing I didn’t already have was CursorLoader. Now, it’s possible I only did Refactor > Migate to AndroidX..
But I did have jetifier in my post-install script so I’m pretty sure it ran npx jetify.

@mikehardy Here's the verdict:

Default import statement was import android.support.v4.content.CursorLoader;
I manually migrated to AndroidX by: Refactor > Migrate to AndroidX...
This improperly changed the import statement to import androidx.core.content.CursorLoader;
When npx jetify ran it wasn't able to change it to import androidx.loader.content.CursorLoader;

However, if I run npx jetify first instead of manually Migrating it works as expected.
https://youtu.be/tmxeFpuYgE8

@mikehardy @jordangrant
I described my issue previously, I have react native project with all files migrated to support<EMAIL_ADDRESS>braking changes. That means I have enabled jetifier by default in gradle settings.
So, I noticed that debug build is working correctly, so jetifier can find and replace all libraries before build automatically. But when I deploy Android app on CircleCI and assemble Release build I see the error from first message in this Issue: package android.support.v4.content does not exist.
That means jetifier didn't update libraries automatically for production build.
You can reproduce that easily:
Just create new fresh<EMAIL_ADDRESS>project and add<EMAIL_ADDRESS>and try to build debug and release Android apps. You will be able to see success debug build and failed release build.
That's why I created this PR and migrated to androidx libraries manually.
New<EMAIL_ADDRESS>will conatin androidx support breaking change.
If you need to use previousl android libs with<EMAIL_ADDRESS>just please use<EMAIL_ADDRESS>which is reflected in updated README file.
I'm not sure what we are looking here now in this discussion?  Do we want to leave repo as is and just understand why Jetifier doesn't work in Android release build?

@oleksandr-dziuban we definitely want to update the library for AndroidX, no question there
However, the issue here is still conflating java-jetifier and javascript-jetifier. The status of your android/gradle.properties file is related yes, but separate. It is for java-jetifier
I'm not sure how any support class references could remain in node_modules after javascript-jetifier has run, so this actually appears to be a failure in setup somehow
Anyway, it's odd. I'll add react-native-share to the rn-androidx-demo CI suite for jetifier and investigate, but my general take is still that react-native-share 1.2.x should work just fine with react-native 0.60. I'll prove that assertion though

@oleksandr-dziuban we definitely want to update the library for AndroidX, no question there
However, the issue here is still conflating java-jetifier and javascript-jetifier. The status of your android/gradle.properties file is related yes, but separate. It is for java-jetifier
I'm not sure how any support class references could remain in node_modules after javascript-jetifier has run, so this actually appears to be a failure in setup somehow
Anyway, it's odd. I'll add react-native-share to the rn-androidx-demo CI suite for jetifier and investigate, but my general take is still that react-native-share 1.2.x should work just fine with react-native 0.60. I'll prove that assertion though

Ok, thanks

Just tested on a fresh project using:
  "dependencies": {
    "react": "16.8.6",
    "react-native": "0.60.3",
    "react-native-share": "^1.2.1"
  },

And it works for both dev/release mode. However, i needed to manually link react-native-share.

For anyone having problems with Android X or react-native >= 0.60, we Just Merge a PR that update react-native-share to 0.60.3 and fix some problems related to Android X. #550
Please ler us know If you find any problem related to this update. :smiley:

I just updated the example to RN60 and it seems to work #567 - but yes, in order to use the java object 'ShareApplication' in your MainApplication you have to manually link it for Android, auto-linking doesn't work with that style.
However, with the implementation of an internal FileProvider via #490 it is not necessary to import or implement anything in MainApplication and auto-linking should work. I demonstrate this in #570


Current daily budget ~= 360K including witness pay?

Current hourly budget ~= 1BB * 3600 * 17 / 2^32, related code is here.
Why is that?
I remember it's said that we have 400K budget for workers every day.

Yes, that's correct.
The original motivation was to set an upper bound on dilution per second to approximately the same as a full slate of 100% witnesses in BTS 0.x.  The initial float and dilution rate are inherited from BTS 0.x, the denominator 2**32 is chosen as it's a nice round binary value for bitshifting, and the single free parameter (the numerator 17) is actually function of the other parameters.
1.25 billion * 17 / 2**32 = 4.948
which is the number of BTS per second.  17 is simply the largest integer value which results in LHS is less than 5.
I don't remember whether we used 1.2 or 1.25 billion for the initial float when we did this calculation, (the infographic says 1.2), but I do remember that 18 gave something over 5 BTS per second, so we went with 17.
The actual genesis ended up having 1.07 billion, so everything is less; maybe that 17 should have been 20 based on the actual genesis.  But raising it at this point is going to generate a political discussion I don't want to get into.
>>> get_object 2.13.2912
[{
    "id": "2.13.2912",
    "time": "2016-02-12T00:00:00",
    "record": {
      "time_since_last_budget": 3600,
      "from_initial_reserve": "105939578948024",
      "from_accumulated_fees": 18325207,
      "from_unused_witness_budget": 450000,
      "requested_witness_budget": 180000000,
      "total_budget":<PHONE_NUMBER>,
      "witness_budget": 180000000,
      "worker_budget":<PHONE_NUMBER>,
      "leftover_worker_funds": 0,
      "supply_delta":<PHONE_NUMBER>
    }
  }
]

The amounts are all in satoshis, so this means total_budget is 15096 per hour or about 362,304 per day.  I'm not sure where you get 400K; it's probably just rounding (362,304 turns into 400,000 when rounded to a single digit of precision) or giving a loose upper bound (even if witness pay is 0, the most workers would be able to get is less than 400K per day).
This nifty infographic was created from my whiteboard drawing of this back when we were designing Graphene.

I think the number 5 BTS/second is typically quoted as the "dilution" rate of BitShares. That gives a total of 432,000 BTS/day.
What people are missing with that quote is that the rate reduces over time, because it is defined as a percentage of the reserves. (I have to admit that I was surprised that it is already 15% below its original value.)

Thanks for the clarification.


Spring Cloud Netflix Hystrix (Finchley.M2) is incompatible with Spring Boot Actuator (2.0.0.M4)

As soon as I add org.springframework.boot:spring-boot-starter-actuator in build.gradle I'm getting the following log output when I start a microservice based on Spring Boot, Spring Webflux, Spring Data MongoDB, Spring Security Webflux etc.
[...]
INFO  [o.s.c.a.AnnotationUtils] [main] Failed to introspect annotations on [class org.springframework.cloud.netflix.hystrix.HystrixAutoConfiguration$$EnhancerBySpringCGLIB$$51330790]: java.lang.ArrayStoreException: sun.reflect.annotation.TypeNotPresentExceptionProxy
INFO  [o.s.c.a.AnnotationUtils] [main] Failed to introspect annotations on [class org.springframework.cloud.netflix.hystrix.HystrixAutoConfiguration$$EnhancerBySpringCGLIB$$51330790]: java.lang.ArrayStoreException: sun.reflect.annotation.TypeNotPresentExceptionProxy
WARN  [o.s.b.w.r.c.ReactiveWebServerApplicationContext] [main] Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Unable to start reactive web server; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'httpHandler' defined in class path resource [org/springframework/boot/autoconfigure/web/reactive/HttpHandlerAutoConfiguration$AnnotationConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.http.server.reactive.HttpHandler]: Factory method 'httpHandler' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'webHandler' defined in class path resource [org/springframework/boot/autoconfigure/web/reactive/WebFluxAutoConfiguration$EnableWebFluxConfiguration.class]: Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'requestMappingHandlerMapping' defined in class path resource [org/springframework/boot/autoconfigure/web/reactive/WebFluxAutoConfiguration$EnableWebFluxConfiguration.class]: Invocation of init method failed; nested exception is java.lang.ArrayStoreException: sun.reflect.annotation.TypeNotPresentExceptionProxy

Finchley.M2 was meant to work with Boot 2.0.0.M3.  We recently just started looking at upgrading to the latest Boot milestone and realize there are some breaking changes.  We will address these soon.

@ryanjbaxter any luck / progress so far ? spring-boot 2.0.0.RC1 is out recently

@n0mer We've had 4 more milestones since. M6(announcement pending) works with RC1. Next time please ask on gitter, not a closed issue.

@spencergibb ok
Starters for Eureka still on M2 (http://repo.spring.io/milestone/org/springframework/cloud/spring-cloud-starter-eureka/ , http://repo.spring.io/milestone/org/springframework/cloud/spring-cloud-starter-eureka-server/), hence was my question


localforage

Can I use localForage instead of localstorage?
What part of code needs to be changed to support localForage?

Your goal is to use IndexedDB or WebSQL, right? Check out the async branch (and #73), which does that already. It uses sticky.js instead of localForage for the adapter, but if that matters to you, it would be trivial to write one in backbone.dualstorage.adapters.coffee.
The async branch has some failing tests related to how promises act in some situations, but it should otherwise work fine. If you think you'd like to address those issues before I get to it, have at it.


tedge-mapper-c8y panics on startup if an invalid operations file exists

Describe the bug

thin-edge.io tedge-mapper-c8y panics on startup if there is an invalid toml file defined under the operations/c8y/ folder. If the invalid operations file is added whilst the tedge-mapper-c8y is running then an error is printed but the mapper keeps on functioning, but if the mapper is restarted, then it goes into an infinite start -> panic -> shutdown loop.
Below shows the panic during the tedge-mapper-c8y service startup:
Nov 07 13:58:18 tedge tedge-mapper[4922]: 2024-11-07T13:58:18.68308912Z  INFO C8Y-REST: start initialisation
Nov 07 13:58:18 tedge tedge-mapper[4922]: thread 'main' panicked at /home/runner/work/thin-edge.io/thin-edge.io/crates/core/tedge_actors/src/builders.rs:87:26:
Nov 07 13:58:18 tedge tedge-mapper[4922]: called `Result::unwrap()` on an `Err` value: ActorError(OperationsError(TomlError("/etc/tedge/operations/c8y/invalidfile", Error { inner: Error { inner: TomlError { message: "expected `.`, `=`", original: Some("sleep 10\n"), keys: [], span: Some(6..7) } } })))
Nov 07 13:58:18 tedge tedge-mapper[4922]: note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Nov 07 13:58:18 tedge tedge-mapper[4922]: 2024-11-07T13:58:18.683160412Z  INFO mqtt_channel::connection: MQTT connecting to broker: host=<IP_ADDRESS>:8883, session_name=None
Nov 07 13:58:18 tedge tedge-mapper[4922]: 2024-11-07T13:58:18.683445496Z  INFO mqtt_channel::connection: MQTT connecting to broker: host=<IP_ADDRESS>:8883, session_name=Some("tedge-mapper-c8y")
Nov 07 13:58:18 tedge tedge-mapper[4922]: 2024-11-07T13:58:18.683461704Z  INFO HTTP Server: Cumulocity proxy has HTTPS enabled (configured in `c8y.proxy.cert_path`/`c8y.proxy.key_path`) and certificate authentication enabled (2 certificates found) (configured in `c8y.proxy.ca_path`)
Nov 07 13:58:18 tedge tedge-mapper[4922]: 2024-11-07T13:58:18.683543954Z  INFO c8y_auth_proxy::server: Launching on port 8001 with HTTPS
Nov 07 13:58:18 tedge systemd[1]: tedge-mapper-c8y.service: Main process exited, code=exited, status=101/n/a
Nov 07 13:58:18 tedge systemd[1]: tedge-mapper-c8y.service: Failed with result 'exit-code'.

To Reproduce

Assuming you have a functional thin-edge.io installation connected to Cumulocity, then run:

Create an invalid operations file (with invalid toml syntax)
echo "sleep 10" > /etc/tedge/operations/c8y/invalidfile

Restart the tedge-mapper-c8y
systemctl restart tedge-mapper-c8y

If the bug is present, then the tedge-mapper-c8y will continuously start then crash. You can monitor via journald:
journalctl -fu tedge-mapper-c8y -n 20

Expected behavior

The tedge-mapper-c8y should be resistent to invalid user configuration and not crash/panic. Instead it should:

log a warning that an invalid operations file has been detected. The message should include the file path and the error
ignore the invalid operation file

Screenshots

Environment (please complete the following information):

Property
Value

OS [incl. version]
Debian GNU/Linux 12 (bookworm)

Hardware [incl. revision]
unknown

System-Architecture
Linux tedge 6.8.0-39-generic #39-Ubuntu SMP PREEMPT_DYNAMIC Sat Jul  6 02:50:39 UTC 2024 aarch64 GNU/Linux

thin-edge.io version
tedge 1.3.1

Additional context

Resolved. The cloud operation definition file is handled nicely and an error is printed out to the user so they know why the operation file is being rejected, below is the output showing the new handling.
Dec 05 07:09:39 tedge tedge-mapper[3693]: 2024-12-05T07:09:39.172554464Z ERROR c8y_mapper_ext::supported_operations::operation: Failed to load an operation from file path=/etc/tedge/operations/c8y/invalidfile error=Error while parsing operation file: '/etc/tedge/operations/c8y/invalidfile': TOML parse error at line 1, column 7
Dec 05 07:09:39 tedge tedge-mapper[3693]:   |
Dec 05 07:09:39 tedge tedge-mapper[3693]: 1 | sleep 10
Dec 05 07:09:39 tedge tedge-mapper[3693]:   |       ^
Dec 05 07:09:39 tedge tedge-mapper[3693]: expected `.`, `=`
Dec 05 07:09:39 tedge tedge-mapper[3693]: .
Dec 05 07:09:39 tedge tedge-mapper[3693]: 2024-12-05T07:09:39.272615995Z  INFO mqtt_channel::connection: MQTT connecting to broker: host=<IP_ADDRESS>:8883, session_name=None


Controller.RenderJSON throws error

Hello,
i did some search in issues and in code, but I didnt find anythink what could help me.
This is my code
func (this *Foo) Boo() {

	data := map[string]interface{}{
		"yaya": "blabla",
		"blabla": "yaya",
	}

	return this.RenderJson(data)
}

I think, it should be ok.. I even found similar code in some issue here, but my code still throws me this error
ERROR 2017/07/03 22:31:56 build.go:119: # remembrall/app/controllers
app/controllers/app.go:19: too many arguments to return
	have (revel.Result)
	want ()

Can anyone help me, please?
Thanks,
Tobias

The first line should be
func (this *Foo) Boo() *revel.Result {
On Mon, Jul 3, 2017, 2:04 PM Huholoman<EMAIL_ADDRESS>wrote:

Hello,
i did some search in issues and in code, but I didnt find anythink what
could help me.
This is my code
func (this *Foo) Boo() {
mp := map[string]interface{}{
"yaya": "blabla",
"blabla": "yaya",
}
return this.RenderJson(mp)
}
I think, it should be ok.. I even found similar code in some issue here,
but my code still throws me this error
ERROR 2017/07/03 22:31:56 build.go:119: # remembrall/app/controllers
app/controllers/app.go:19: too many arguments to return
have (revel.Result)
want ()
Can anyone help me, please?
Thanks,
Tobias
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/revel/revel/issues/1190, or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUsBgXFaFvk2ndNM5_hMI-ZrULEEIzZks5sKVdcgaJpZM4OMrte
.

Yay, of course.. Thanks. :D


The results of baselines are not reasonable and not correct

For the ogbn-Arxiv supervised setting, the results of GCN, Graphsage, and other GNN methods on ogbn-arxiv should be higher than 70. This is well-known in the community.

For the ogbn-Arxiv supervised setting, the results of GCN, Graphsage, and other GNN methods on ogbn-arxiv should be higher than 70. This is well-known in the community.

Hi, we have reply this issue in the issue #2 . Please refer to that issue.

Learning

BERT have many versions of models. We use this version https://huggingface.co/google/bert_uncased_L-2_H-128_A-2.

Thank you for your reply.

I still believe that even with the used BERT, the performance would not be so low. I will check this by running the code in [1,2].
If GNNs with other BERT models in [1,2] can outperform your methods, how can we say GraphGPT can beat simple GNNs?

[1] SimTG: A Frustratingly Simple Approach For Textual Graph Representation Learning. [2] Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs.

We use the same node features processed by the same bert model https://huggingface.co/google/bert_uncased_L-2_H-128_A-2 into baselines and our GraphGPT. So the comparisons are fair and it's only the case of initial node features. The methods you mentioned tends to enhance the node representations use language models or others. The bert models they used may be more powerful than ours. But it's only different data pre-processing and initial experimental settings. You could use our data to run the baselines like GAT, GCN ... to verify our experiments. Btw, the results could further prove that our GraphGPT are not that sensitive to the quality of node features than traditional GNNs.

This is not the case. Your input for GraphGPT also includes the abstract and title, i.e., raw text .
For fairness, one can actually use any BERT model to embed this raw text for GNN. If we input the raw text into GNN, we can achieve an accuracy higher than 70%

Thanks for your questions. But traditional GNNs could not directly leverage raw text into them. They could only leverage node features. GraphGPT could leverage the raw text because the superiority of LLMs.

a

Thanks for your questions. We'd like to clarify that we use the same node features into both baseline GNNs and GraphGPT and in this academic experiment, it's the commonly used control variable method to validate the effectiveness of the model architecture. In our experiments, we keep the same data-preprocessing, i.e., the same node features all the time.

is

Thanks for your question! But actually, in Language is All a Graph Needs, it leverage Skip-gram rather than bag-of-words for arxiv.

And refer to original OGB website, thay run the skip-gram model over the MAG corpus. So skip-gram requires training but we use the pre-trained BERT (that is training-free) for data pre-processing. So to some extent, the features of the OGBN-ARXIV are optimized for MAG corpus.

And we clarify again that our experiments are fair and reasonable enough because we use the same node features all the time.

Thank you for your reply.
But you chose a very weak BERT model for GNN. Could you explain why you didn't choose the BERT methods that were widely used in previous work? My point is that you utilize raw text for GraphGPT. Thus, we can actually utilize any pre-trained BERT model (which are all training-free) for GNN.

Thanks for your questions! The original reason why we choose this model is very straightforward. Because we want keep the same dimension of our data (i.e. 128) with original ogb data. Most dimensions of features you mentioned above are 768. So the acc are higher is understandable.

@tjb-tech You still haven’t answered @AGTSAAA  ’s question about why you’re using a weaker BERT model for GNNs. In my experiments, a GNN with a ‘standard’ BERT can achieve much better performance than you reported. You reported significantly lower performance for the GNN baselines, so how can you claim your model has a ‘significant’ improvement over GNNs?